Skip to content

feat: add SQL (SQLite) source reader - #285

Open
ArdaTkn wants to merge 1 commit into
SkyeAv:mainfrom
ArdaTkn:feat/sql-source-reader
Open

ArdaTkn wants to merge 1 commit into
SkyeAv:mainfrom
ArdaTkn:feat/sql-source-reader

Conversation

@ArdaTkn

@ArdaTkn ArdaTkn commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #263 — SQLite-first, per the issue's "Questions for Reviewers" (stdlib sqlite3, zero new dependencies; multi-engine connection strings left as a documented follow-up rather than shipping connectorx blind).

  • kind: sql with local carrying the SQLite database file (same as the file kinds) plus exactly one of table / query, enforced in the model (sql-table-and-query-conflict).
  • Deterministic row order by contract: table scans as SELECT * FROM "table" ORDER BY rowid; a query must carry its own ORDER BY — a missing one fails fast with sql-query-needs-order-by so row indices and row:<N> StudyResult ids stay reproducible across runs.
  • The table identifier is validated as a plain name (alnum + underscore, no leading digit) before quoted interpolation, so config-injected SQL cannot break out of the identifier position.
  • Columns rename positionally to column_1.. in result order per the reader contract; a zero-row result keeps its column schema (unlike schemaless NDJSON, a SQL cursor always knows its columns).
  • Secrets posture: the issue's "credentials never logged or stamped into provenance" holds structurally — no connection string exists in this design; url stays the declared provenance list, format-validated only, never fetched.

Changes

  • Files.SQL enum member + SQL source model in the Section.source union
  • lib.sql reader (stdlib sqlite3), dispatch in Tcode._source_ops, runcache op-cost entry
  • Tests: table rowid order, query rename, empty-result schema, ORDER BY enforcement, identifier hardening, selector exclusivity, diagnostic label, and an end-to-end six-stage build smoke over a SQLite source
  • Docs: "SQL (SQLite) Source" section in docs/configuration/table.md + docs source-of-truth binding

Local gate

  • ruff check / ruff format --check: clean
  • pyright: 0 errors; vulture / deptry: clean
  • Full pytest suite: 1381 passed; the 3 failures are pre-existing on clean main (two macOS errno-mapping tests + test_real_repo_files_round_trip, covered by your follow-up issue)
  • mkdocs build --strict: clean

@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 52a2edae-231b-47e1-b3cf-6b140918a1d4
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@SkyeAv

SkyeAv commented Nov 19, 2026

Copy link
Copy Markdown
Owner

@ArdaTkn, could you rebase this branch onto the latest main and add an Unreleased changelog entry for the SQLite reader? That will bring it up to date and satisfy the new changelog check so I can rerun CI. Thanks!

@ArdaTkn
ArdaTkn force-pushed the feat/sql-source-reader branch from 0a9a8b2 to 4b60a04 Compare November 19, 2026 22:09
@ArdaTkn

ArdaTkn commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

@SkyeAv Rebased onto latest main (ef65a36), resolved merge conflicts, and added the Unreleased changelog entry for the SQLite reader. All local checks (check_changelog.py, ruff, pyright, mkdocs build --strict, pytest) and CI checks are updated.

@SkyeAv SkyeAv left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution, @ArdaTkn! The integration and tests look good overall. I found a few edge cases worth addressing before merging:

  • Query safety: src/tablassert/lib.py:1016-1031 allows write queries containing order by. I confirmed that DELETE ... RETURNING * -- order by deleted rows. Could we open SQLite read-only, reject writes with an authorizer, and add regression tests?
  • src/tablassert/lib.py:1016: the substring check rejects valid ORDER\nBY and accepts comment-only or subquery-only matches. A top-level clause check would be more reliable.
  • src/tablassert/lib.py:1031: duplicate result names crash non-empty queries and collapse empty schemas. Using positional names when constructing the frame should avoid this.
  • src/tablassert/lib.py:1027: missing databases are silently created. mode=ro and explicit connection cleanup would help here too.
  • src/tablassert/lib.py:1024: invalid table names currently use the wrong error code.
  • docs/configuration/table.md:262-268: the SQL example has a few leftover NDJSON lines, creating duplicate YAML keys.

The table identifier validation looks safe against injection; the raw query path needs similar protection. CI passes, but these cases are not covered yet. Thanks again!

@ArdaTkn

ArdaTkn commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

Addressed all review items in ba353f8:

  1. Read-only SQLite enforcement & cleanup: lib.sql now verifies p.is_file() first (raising SourceFileError / source-file-unreadable without creating an empty DB), opens the database with ?mode=ro URI, wraps the connection in contextlib.closing, and installs a set_authorizer allowlist permitting only SQLITE_SELECT, SQLITE_READ, and SQLITE_FUNCTION (raising sql-query-not-read-only on DML/DDL).
  2. Top-level ORDER BY check: Added _has_top_level_order_by, which strips SQL comments (-- / /* */) and quoted/bracketed literals/identifiers while tracking parenthesis depth, matching \bORDER\s+BY\b only at depth == 0 (handles ORDER\nBY, rejects comment-only -- order by and subquery-only ORDER BY).
  3. Duplicate result column names: Constructs pl.DataFrame directly with positional schema=["column_1", ...] instead of building with duplicate SQL column names and renaming afterwards (verified on both populated and empty SELECT a, a ... queries).
  4. Invalid table identifier error code: Now raises sql-invalid-table-name instead of reusing sql-query-needs-order-by.
  5. Docs cleanup: Removed the stray ndjson lines from the sql YAML example in docs/configuration/table.md.

Local gates re-run and passing: check_changelog.py, ruff check, ruff format --check, pyright, vulture, deptry, mkdocs build --strict, and pytest.

@ArdaTkn
ArdaTkn force-pushed the feat/sql-source-reader branch from 4b60a04 to ba353f8 Compare November 19, 2026 23:04
@ArdaTkn
ArdaTkn force-pushed the feat/sql-source-reader branch from ba353f8 to 6a12a4e Compare November 19, 2026 23:18

@SkyeAv SkyeAv left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @ArdaTkn! The previous fixes look good, and CI is green. Two small items remain:

  • Recursive CTEs: src/tablassert/lib.py:1087 rejects valid read-only recursive CTEs because sqlite3.SQLITE_RECURSIVE is not allowlisted. I confirmed that SQLite executes the query directly, while lib.sql raises sql-query-not-read-only. Please allow this action and add a regression test.
  • Docs: docs/configuration/table.md:252 marks table as required, but exactly one of table/query is required, so either may be omitted. Please mark both optional while retaining the exactly-one explanation.

Once those are addressed, this should be ready to merge.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a SQL database source reader (SQLite and similar)

2 participants