diff --git a/CHANGELOG.md b/CHANGELOG.md index 6a64043d..cd141cce 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ All notable changes to this project are documented in this file. - **NDJSON (JSONL) is a fifth table source kind.** `kind: ndjson` scans newline-delimited JSON lazily with `pl.scan_ndjson` and renames its columns positionally to `column_1..` in first-seen key order, matching the `parquet` reader contract so `method: column` encodings keep using Excel-style letters. Ragged records scan missing keys as nulls instead of dropping rows; an empty file fails fast with a coded `ndjson-empty` error instead of polars' opaque `Cannot infer NDJSON types` compute error. ([#283](https://github.com/SkyeAv/Tablassert/pull/283), [#262](https://github.com/SkyeAv/Tablassert/issues/262)) - **Every pull request must carry a new changelog entry.** A stdlib-only `scripts/check_changelog.py` requires at least one new bullet under the `## Unreleased` heading of `CHANGELOG.md` relative to the pull request's merge base, failing with remediation guidance when whitespace, subheadings, or released-history edits are the only additions; a CI job runs it on every pull request including forks, release-bump and dependency-bump pull requests are exempt by their `chore(release):` / `chore(deps):` title or commit subject, documentation-only pull requests (every changed path is markdown or under `docs/`) are exempt via `--exempt-docs-only`, and a pre-push hook mirrors the same check locally. ([#284](https://github.com/SkyeAv/Tablassert/pull/284), [#282](https://github.com/SkyeAv/Tablassert/issues/282)) - **Arrow IPC (Feather v2) is a fourth table source kind.** `kind: ipc` scans a `.arrow` file lazily with `pl.scan_ipc` and renames its columns positionally to `column_1..` in schema order, matching the `parquet` reader contract so `method: column` encodings keep using Excel-style letters. Only IPC file layout is supported; a stream-layout file fails fast with a coded `ipc-stream-unsupported` error naming the conversion fix instead of polars' opaque `InvalidFooter` compute error. ([#281](https://github.com/SkyeAv/Tablassert/pull/281), [#264](https://github.com/SkyeAv/Tablassert/issues/264)) +- **SQL (SQLite) is a sixth table source kind.** `kind: sql` reads a table or query result from a SQLite database via the stdlib `sqlite3` driver (zero new dependencies), with `local` carrying the database file and exactly one of `table` / `query` selecting the rows. `table` scans in `rowid` order and a `query` must carry its own `ORDER BY` (coded `sql-query-needs-order-by` otherwise), so row indices and `row:` StudyResult ids stay reproducible across runs; columns rename positionally to `column_1..` per the reader contract. ([#285](https://github.com/SkyeAv/Tablassert/pull/285), [#263](https://github.com/SkyeAv/Tablassert/issues/263)) ### Fixed diff --git a/docs/configuration/table.md b/docs/configuration/table.md index 6cc43134..258f4c90 100644 --- a/docs/configuration/table.md +++ b/docs/configuration/table.md @@ -111,8 +111,7 @@ source: row_slice: [1, auto] # Start at the second physical row, read to end ``` -> **Specify `kind` explicitly.** Tablassert selects the reader purely from the declared `kind`: `excel` reads a workbook (`sheet`), `text` scans delimited text (`delimiter`), `parquet` scans a parquet file (columns renamed positionally), `ipc` scans an Arrow IPC file (columns renamed positionally); the file on disk is never inspected to infer its format. Because `kind` carries a default, a source whose `kind` is omitted or does not match the actual file is still accepted and fed to the wrong reader, surfacing only later as a read error or garbled rows. Stating `kind` explicitly makes a mis-declared source fail fast. -> **Specify `kind` explicitly.** Tablassert selects the reader purely from the declared `kind`: `excel` reads a workbook (`sheet`), `text` scans delimited text (`delimiter`), `parquet` scans a parquet file (columns renamed positionally), `ndjson` scans newline-delimited JSON (columns renamed positionally); the file on disk is never inspected to infer its format. Because `kind` carries a default, a source whose `kind` is omitted or does not match the actual file is still accepted and fed to the wrong reader, surfacing only later as a read error or garbled rows. Stating `kind` explicitly makes a mis-declared source fail fast. +> **Specify `kind` explicitly.** Tablassert selects the reader purely from the declared `kind`: `excel` reads a workbook (`sheet`), `text` scans delimited text (`delimiter`), `parquet` scans a parquet file (columns renamed positionally), `ipc` scans an Arrow IPC file (columns renamed positionally), `ndjson` scans newline-delimited JSON (columns renamed positionally), `sql` queries a SQLite database (`table` or `query`); the file on disk is never inspected to infer its format. Because `kind` carries a default, a source whose `kind` is omitted or does not match the actual file is still accepted and fed to the wrong reader, surfacing only later as a read error or garbled rows. Stating `kind` explicitly makes a mis-declared source fail fast. #### Text Source (CSV/TSV) @@ -232,6 +231,42 @@ source: row_slice: [1, auto] ``` +#### SQL (SQLite) Source + +Reads a table or query result from a SQLite database via the stdlib `sqlite3` driver (zero +extra dependencies; Postgres/MySQL/DuckDB connection strings are a documented follow-up). +`local` carries the database file, as with the file kinds. Columns are addressed +positionally: the reader renames whatever names the result carries to `column_1`..`column_n` +in result order (the same convention the `parquet` reader produces), so `method: column` +encodings keep using Excel-style letters and the table's own column names are never +referenced from configs. Row order is deterministic by contract: `table` scans as +`SELECT * FROM "table" ORDER BY rowid`, and a `query` must carry its own `ORDER BY` (a +missing one fails fast with `sql-query-needs-order-by`) so row indices and `row:` +StudyResult ids are reproducible across runs. Exactly one of `table` / `query` is required. + +| Field | Type | Required | Description | +| ------- | ------ | ---------- | ------------- | +| `kind` | String | No | Source kind. Model default is `"sql"`, but specify it explicitly in configs. | +| `local` | Path | Yes | Local path of the SQLite database file. The file must already exist here; Tablassert does not download it. | +| `url` | List[URL] | Yes | One or more source URLs recorded as provenance (derived sections emit them as the upstream repository supporting entry's `source_record_urls` list, infores:pubmed-central for PMC sections, and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement instead). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. | +| `table` | String | No | Table to scan, in `rowid` order. Exactly one of `table` / `query`. Plain identifiers only (letters, digits, underscore, not starting with a digit). | +| `query` | String | No | Full `SELECT` query; must carry an `ORDER BY` clause. Exactly one of `table` / `query`. | +| `row_slice` | List[NonNegativeInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a non-negative integer or `"auto"`. Mutually exclusive with `rows`. | +| `rows` | List[NonNegativeInt] | No | Zero-based row indices to keep after any `row_slice` crop. Mutually exclusive with `row_slice`. | +| `reindex` | List[Reindex] | No | Conditional row filtering | + +**Example:** + +```yaml +source: + kind: sql + local: ./data/curated.db + url: + - https://example.com/curated.db + table: genes + row_slice: [1, auto] +``` + #### Reindexing (Conditional Filtering) Filter rows based on column values. diff --git a/src/tablassert/enums.py b/src/tablassert/enums.py index e5dd920b..3cbc924a 100644 --- a/src/tablassert/enums.py +++ b/src/tablassert/enums.py @@ -59,6 +59,7 @@ class Files(str, Enum): PARQUET = "parquet" IPC = "ipc" NDJSON = "ndjson" + SQL = "sql" class EncodingMethods(str, Enum): diff --git a/src/tablassert/errors.py b/src/tablassert/errors.py index 978659dd..c0931498 100644 --- a/src/tablassert/errors.py +++ b/src/tablassert/errors.py @@ -48,6 +48,10 @@ def redact_secrets(text: str, secrets: Sequence[str] = ()) -> str: "source-file-unreadable", "ipc-stream-unsupported", "ndjson-empty", + "sql-table-and-query-conflict", + "sql-query-needs-order-by", + "sql-invalid-table-name", + "sql-query-not-read-only", "resolve-bad-specs", "taxon-from-conflict", "taxon-from-unknown-column", diff --git a/src/tablassert/lib.py b/src/tablassert/lib.py index 0910a27a..3d1c5900 100644 --- a/src/tablassert/lib.py +++ b/src/tablassert/lib.py @@ -1,5 +1,6 @@ from __future__ import annotations +import contextlib import json import operator import re @@ -46,7 +47,7 @@ study_size_target, ) from tablassert.enums import Comparisons, EncodingMethods, Files, Functions, InformationResources, Repositories, Tokens -from tablassert.errors import TablassertError +from tablassert.errors import SourceFileError, TablassertError from tablassert.fullmap import ResolveSpec, fullmap_db_path, resolve, resolve_batch from tablassert.log import cat from tablassert.models import EDGE_ID_PLACEHOLDER, Encoding, NodeEncoding, Qualifier, Reindex, RIGConfig, Section @@ -986,6 +987,125 @@ def parquet(p: Path) -> pl.LazyFrame: return lf.rename(dict(zip(names, (f"column_{i + 1}" for i in range(len(names))), strict=True))) +def _has_top_level_order_by(query: str) -> bool: + """Return True if ``query`` contains an ``ORDER BY`` clause at parenthesis depth 0 outside comments and literals.""" + tokens: list[str] = [] + depth: int = 0 + i: int = 0 + n: int = len(query) + while i < n: + ch: str = query[i] + nxt: str = query[i + 1] if i + 1 < n else "" + if ch == "-" and nxt == "-": + end: int = query.find("\n", i + 2) + i = n if end == -1 else end + 1 + tokens.append(" ") + continue + if ch == "/" and nxt == "*": + end = query.find("*/", i + 2) + i = n if end == -1 else end + 2 + tokens.append(" ") + continue + if ch in ("'", '"', "`"): + quote: str = ch + i += 1 + while i < n: + if query[i] == quote: + if i + 1 < n and query[i + 1] == quote: + i += 2 + continue + i += 1 + break + i += 1 + tokens.append(" ") + continue + if ch == "[": + end = query.find("]", i + 1) + i = n if end == -1 else end + 1 + tokens.append(" ") + continue + if ch == "(": + depth += 1 + i += 1 + continue + if ch == ")": + depth = max(0, depth - 1) + i += 1 + continue + if depth == 0: + tokens.append(ch) + i += 1 + top_level: str = "".join(tokens) + return re.search(r"\bORDER\s+BY\b", top_level, re.IGNORECASE) is not None + + +def sql(p: Path, table: str | None, query: str | None) -> pl.LazyFrame: + """Read a SQLite source as a LazyFrame with positional ``column_`` names. + + Args: + p: Path to the SQLite database file. + table: Table to scan as ``SELECT * FROM "table" ORDER BY rowid``. + query: Full SELECT query; must carry its own top-level ``ORDER BY`` clause. + + Returns: + LazyFrame whose columns are named ``column_1..`` in result order. + + Raises: + SourceFileError: ``source-file-unreadable`` when ``p`` does not exist as a file. + TablassertError: ``sql-invalid-table-name`` when ``table`` is not a plain identifier. + TablassertError: ``sql-query-needs-order-by`` when a ``query`` has no top-level ORDER BY — + row indices and ``row:`` StudyResult ids must be reproducible across runs. + TablassertError: ``sql-query-not-read-only`` when a ``query`` attempts writes or non-read-only operations. + + Notes: + The result carries its own column names, but the config contract addresses columns + positionally (Excel-style ``A-ZZ`` encoding letters resolve to ``column_``), so + the reader names result columns ``column_1..`` directly at DataFrame construction -- + the same convention the ``parquet`` reader produces. Opens the database in read-only + mode (``?mode=ro``) with an authorizer permitting only ``SELECT`` / ``READ`` / + ``FUNCTION``. Uses stdlib ``sqlite3``: zero new dependencies, single-engine by design; + multi-engine ``connection`` strings are a documented follow-up. ``table`` quoting uses + bound-parameter-style double quotes with the identifier validated as a plain name first. + """ + import sqlite3 + + if not p.is_file(): + raise SourceFileError(p, "SQLite database file does not exist.") + + if query is not None: + if not _has_top_level_order_by(query): + raise TablassertError( + f"`sql` query for {p} has no top-level ORDER BY clause; row order (and row: ids) would not be reproducible across runs.", + code="sql-query-needs-order-by", + ) + statement: str = query + else: + if not table or not table.replace("_", "a").isalnum() or table[0].isdigit(): + raise TablassertError(f"`sql` table name {table!r} for {p} is not a plain identifier.", code="sql-invalid-table-name") + statement = f'SELECT * FROM "{table}" ORDER BY rowid' + + allowed_actions: frozenset[int] = frozenset({sqlite3.SQLITE_SELECT, sqlite3.SQLITE_READ, sqlite3.SQLITE_FUNCTION, sqlite3.SQLITE_RECURSIVE}) + + def _authorizer(action: int, _arg1: str | None, _arg2: str | None, _db_name: str | None, _trigger: str | None) -> int: + return sqlite3.SQLITE_OK if action in allowed_actions else sqlite3.SQLITE_DENY + + uri: str = f"{p.resolve().as_uri()}?mode=ro" + with contextlib.closing(sqlite3.connect(uri, uri=True)) as conn: + conn.set_authorizer(_authorizer) + try: + cur: sqlite3.Cursor = conn.execute(statement) + if cur.description is None: + raise TablassertError(f"`sql` query for {p} did not return a result set.", code="sql-query-not-read-only") + col_names: list[str] = [f"column_{i + 1}" for i in range(len(cur.description))] + rows: list[tuple[Any, ...]] = cur.fetchall() + except sqlite3.DatabaseError as exc: + if "not authorized" in str(exc).lower(): + raise TablassertError(f"`sql` query for {p} must be read-only: {exc}", code="sql-query-not-read-only") from exc + raise + df: pl.DataFrame = pl.DataFrame(rows, schema=col_names, orient="row") if rows else pl.DataFrame({n: [] for n in col_names}) + return df.lazy() + + def ipc(p: Path) -> pl.LazyFrame: """Read an Arrow IPC (Feather v2 / `.arrow`) source as a LazyFrame with positional ``column_`` names. @@ -1573,6 +1693,9 @@ def _source_ops(self: Self) -> list[Any]: (parquet, (self.source.local,)) if self.source.kind == Files.PARQUET else None, # pyright: ignore (ipc, (self.source.local,)) if self.source.kind == Files.IPC else None, # pyright: ignore (ndjson, (self.source.local,)) if self.source.kind == Files.NDJSON else None, # pyright: ignore + (sql, (self.source.local, getattr(self.source, "table", None), getattr(self.source, "query", None))) + if self.source.kind == Files.SQL + else None, # pyright: ignore (idx, ()), (crop, (self.source.row_slice,)) if self.source.row_slice else None, (pick, (self.source.rows,)) if self.source.rows else None, diff --git a/src/tablassert/models.py b/src/tablassert/models.py index f0aecc03..a8fb9fbb 100644 --- a/src/tablassert/models.py +++ b/src/tablassert/models.py @@ -76,7 +76,7 @@ def _shown(name: str, target: str) -> str: return f"`{name}` (coerced to `{target}`)" if target != name else f"`{name}`" -def _section_source_label(source: Excel | Text | Parquet | IPC | NDJSON) -> str: +def _section_source_label(source: Excel | Text | Parquet | IPC | NDJSON | SQL) -> str: """Render a section's source for a diagnostic message: the local path, plus the sheet for Excel. Args: @@ -300,6 +300,28 @@ class NDJSON(BaseSource): kind: Literal[Files.NDJSON] = Field(Files.NDJSON, description="Source kind; must be 'ndjson'.") +class SQL(BaseSource): + """SQLite database source: ``local`` carries the database file (as with the file kinds). + + Exactly one of ``table`` / ``query`` selects the rows. ``table`` scans the whole table + in ``rowid`` order; a ``query`` must carry its own ``ORDER BY`` so row indices and + ``row:`` StudyResult ids stay stable across runs. + """ + + kind: Literal[Files.SQL] = Field(Files.SQL, description="Source kind; must be 'sql'.") + table: str | None = Field(None, description="Table to scan; scanned in rowid order. Mutually exclusive with `query`.") + query: str | None = Field(None, description="Full SELECT query; must carry an ORDER BY clause. Mutually exclusive with `table`.") + + @model_validator(mode="after") + def exactly_one_selector(self: Self) -> Self: + if bool(self.table) == bool(self.query): + raise TablassertValidationError( + "A `sql` source needs exactly one of `table` or `query` (mutually exclusive).", code="sql-table-and-query-conflict" + ) + + return self + + class Regex(TablaBase): pattern: int | float | str = Field(..., description="Regex pattern passed to string replacement.", examples=["\\s+", "\\.$"]) @@ -994,7 +1016,7 @@ def warn_when_the_slot_cannot_reach_the_edge(self) -> Self: class Section(TablaBase): """Pydantic section model and coercion target for a single table configuration.""" - source: Excel | Text | Parquet | IPC | NDJSON = Field(..., description="Input source definition for reading tabular rows.") + source: Excel | Text | Parquet | IPC | NDJSON | SQL = Field(..., description="Input source definition for reading tabular rows.") statement: Statement = Field(..., description="Subject-object statement mapping for this section.") provenance: Provenance = Field(..., description="Provenance metadata applied to all produced edges.") annotations: list[Annotation] | None = Field(None, description="Optional extra encoded columns added to each row.") diff --git a/src/tablassert/runcache.py b/src/tablassert/runcache.py index 149d4e30..5ab96ae7 100644 --- a/src/tablassert/runcache.py +++ b/src/tablassert/runcache.py @@ -255,6 +255,7 @@ def prefix_digest(ops: list[tuple[Callable, tuple[Any, ...]]]) -> str: lib.parquet: 2, lib.ipc: 2, lib.ndjson: 2, + lib.sql: 2, # Filter: single lazy row predicates/slices. lib.idx: 1, lib.crop: 1, diff --git a/tests/test_cover_lib.py b/tests/test_cover_lib.py index 6ed200fe..baf0fb43 100644 --- a/tests/test_cover_lib.py +++ b/tests/test_cover_lib.py @@ -173,6 +173,131 @@ def test_ipc_rename_tracks_schema_order_not_name_order(tmp_path: Path) -> None: assert df["column_2"].to_list() == ["x", "y"] +def test_sql_reads_table_in_rowid_order(tmp_path: Path) -> None: + """``sql`` scans a table via stdlib sqlite3 and renames columns positionally. + + The whole-table path appends ``ORDER BY rowid`` so row indices and ``row:`` ids are + reproducible across runs. + """ + import sqlite3 + + db: Path = tmp_path / "genes.db" + conn: sqlite3.Connection = sqlite3.connect(db) + conn.execute("CREATE TABLE genes (gene_a TEXT, gene_b TEXT, p REAL)") + conn.executemany("INSERT INTO genes VALUES (?, ?, ?)", [("brca1", "mapk1", 0.5), ("brca1", "mapk1", 0.01)]) + conn.commit() + conn.close() + + lf: pl.LazyFrame = lib.sql(db, "genes", None) + assert isinstance(lf, pl.LazyFrame) + df: pl.DataFrame = lf.collect() + assert df.columns == ["column_1", "column_2", "column_3"] + assert df["column_1"].to_list() == ["brca1", "brca1"] + assert df["column_3"].to_list() == [0.5, 0.01] + + +def test_sql_query_rename_and_empty_result(tmp_path: Path) -> None: + """A ``query`` renames in result order; a zero-row result keeps its column schema.""" + import sqlite3 + + db: Path = tmp_path / "genes.db" + conn: sqlite3.Connection = sqlite3.connect(db) + conn.execute("CREATE TABLE genes (gene_a TEXT, gene_b TEXT)") + conn.executemany("INSERT INTO genes VALUES (?, ?)", [("brca1", "mapk1")]) + conn.commit() + conn.close() + + df: pl.DataFrame = lib.sql(db, None, "SELECT gene_b, gene_a FROM genes ORDER BY gene_a").collect() + assert df.columns == ["column_1", "column_2"] + assert df["column_1"].to_list() == ["mapk1"] + + empty: pl.LazyFrame = lib.sql(db, None, "SELECT gene_a FROM genes WHERE 0 ORDER BY gene_a") + assert empty.collect_schema().names() == ["column_1"] + assert empty.collect().shape == (0, 1) + + +def test_sql_query_requires_order_by(tmp_path: Path) -> None: + """An ORDER BY-less query would give non-reproducible row order; fail coded.""" + import sqlite3 + + db: Path = tmp_path / "genes.db" + conn: sqlite3.Connection = sqlite3.connect(db) + conn.execute("CREATE TABLE genes (gene_a TEXT)") + conn.commit() + conn.close() + + with pytest.raises(TablassertError, match="sql-query-needs-order-by"): + lib.sql(db, None, "SELECT * FROM genes") + + +def test_sql_rejects_non_plain_identifier_table(tmp_path: Path) -> None: + """The table name is interpolated as a quoted identifier; only plain names pass.""" + import sqlite3 + + db: Path = tmp_path / "genes.db" + conn: sqlite3.Connection = sqlite3.connect(db) + conn.execute("CREATE TABLE genes (gene_a TEXT)") + conn.commit() + conn.close() + + with pytest.raises(TablassertError, match="sql-invalid-table-name"): + lib.sql(db, 'genes"; DROP TABLE genes; --', None) + + +def test_sql_missing_db_raises_source_file_error(tmp_path: Path) -> None: + """A missing SQLite file fails with ``source-file-unreadable`` without creating an empty DB.""" + missing: Path = tmp_path / "missing.db" + with pytest.raises(TablassertError, match="source-file-unreadable"): + lib.sql(missing, "genes", None) + assert not missing.exists() + + +def test_sql_rejects_write_query_and_fake_order_by(tmp_path: Path) -> None: + """Read-only authorizer blocks DML/DDL even with a trailing ``-- order by`` comment.""" + import sqlite3 + + db: Path = tmp_path / "genes.db" + conn: sqlite3.Connection = sqlite3.connect(db) + conn.execute("CREATE TABLE genes (gene_a TEXT)") + conn.execute("INSERT INTO genes VALUES ('brca1')") + conn.commit() + conn.close() + + with pytest.raises(TablassertError, match="sql-query-needs-order-by"): + lib.sql(db, None, "DELETE FROM genes RETURNING * -- order by") + with pytest.raises(TablassertError, match="sql-query-needs-order-by"): + lib.sql(db, None, "SELECT * FROM (SELECT gene_a FROM genes ORDER BY gene_a)") + with pytest.raises(TablassertError, match="sql-query-not-read-only"): + lib.sql(db, None, "INSERT INTO genes SELECT gene_a FROM genes ORDER BY gene_a") + with pytest.raises(TablassertError, match="sql-query-not-read-only"): + lib.sql(db, None, "CREATE TABLE evil AS SELECT gene_a FROM genes ORDER BY gene_a") + + df: pl.DataFrame = lib.sql(db, None, "SELECT gene_a, gene_a FROM genes ORDER\nBY gene_a").collect() + assert df.columns == ["column_1", "column_2"] + assert df.rows() == [("brca1", "brca1")] + + empty_dup: pl.DataFrame = lib.sql(db, None, "SELECT gene_a, gene_a FROM genes WHERE 0 ORDER BY gene_a").collect() + assert empty_dup.columns == ["column_1", "column_2"] + assert empty_dup.shape == (0, 2) + + +def test_sql_allows_read_only_recursive_cte(tmp_path: Path) -> None: + """Read-only recursive CTEs (``SQLITE_RECURSIVE``) are permitted by the authorizer.""" + import sqlite3 + + db: Path = tmp_path / "cte.db" + conn: sqlite3.Connection = sqlite3.connect(db) + conn.execute("CREATE TABLE genes (gene_a TEXT)") + conn.commit() + conn.close() + + df: pl.DataFrame = lib.sql( + db, None, "WITH RECURSIVE cnt(x) AS (SELECT 1 UNION ALL SELECT x + 1 FROM cnt WHERE x < 3) SELECT x FROM cnt ORDER BY x" + ).collect() + assert df.columns == ["column_1"] + assert df["column_1"].to_list() == [1, 2, 3] + + def test_ipc_reads_empty_frame(tmp_path: Path) -> None: """An empty IPC file still yields its (renamed) schema, so downstream ops see the columns.""" ap: Path = tmp_path / "empty.arrow" diff --git a/tests/test_cover_models.py b/tests/test_cover_models.py index 54bb1e70..e88b4259 100644 --- a/tests/test_cover_models.py +++ b/tests/test_cover_models.py @@ -16,7 +16,7 @@ from pydantic import BaseModel, ValidationError from tablassert.biolink import _build_str_enum, _category_name -from tablassert.models import IPC, NDJSON, Excel, Graph, ManualProvenance, Parquet, Section, Text, _section_source_label +from tablassert.models import IPC, NDJSON, SQL, Excel, Graph, ManualProvenance, Parquet, Section, Text, _section_source_label def test_manual_provenance_explicit_none_publications_short_circuits() -> None: @@ -128,8 +128,11 @@ def test_section_accepts_all_source_kinds() -> None: "statement": {"subject": {"method": "value", "encoding": "BRCA1"}, "object": {"method": "value", "encoding": "TP53"}}, "provenance": {"repo": "PMC", "publication": "PMC0000000"}, } - for kind, model in [("text", Text), ("excel", Excel), ("parquet", Parquet), ("ipc", IPC), ("ndjson", NDJSON)]: - payload: dict[str, Any] = {**base, "source": {"kind": kind, "local": f"./table.{kind}", "url": [f"https://example.com/table.{kind}"]}} + for kind, model in [("text", Text), ("excel", Excel), ("parquet", Parquet), ("ipc", IPC), ("ndjson", NDJSON), ("sql", SQL)]: + source: dict[str, Any] = {"kind": kind, "local": f"./table.{kind}", "url": [f"https://example.com/table.{kind}"]} + if kind == "sql": + source["table"] = "t" # `sql` requires exactly one of table/query. + payload: dict[str, Any] = {**base, "source": source} section: Section = Section.model_validate(payload) assert section.source.kind == kind assert type(section.source) is model @@ -147,5 +150,19 @@ def test_section_source_label_per_kind() -> None: assert _section_source_label(Parquet.model_validate({"kind": "parquet", "local": "./t.parquet", "url": urls})) == "`t.parquet`" assert _section_source_label(IPC.model_validate({"kind": "ipc", "local": "./t.arrow", "url": urls})) == "`t.arrow`" assert _section_source_label(NDJSON.model_validate({"kind": "ndjson", "local": "./t.ndjson", "url": urls})) == "`t.ndjson`" + assert _section_source_label(SQL.model_validate({"kind": "sql", "local": "./t.db", "url": urls, "table": "t"})) == "`t.db`" excel: Excel = Excel.model_validate({"kind": "excel", "local": "./t.xlsx", "url": urls, "sheet": "S1"}) assert _section_source_label(excel) == "`t.xlsx` (sheet `S1`)" + + +def test_sql_source_requires_exactly_one_selector() -> None: + """``table`` and ``query`` are mutually exclusive and one is required.""" + base: dict[str, Any] = {"kind": "sql", "local": "./data.db", "url": ["https://example.com/data.db"]} + selected: SQL = SQL.model_validate({**base, "table": "t"}) + assert selected.table == "t" + queried: SQL = SQL.model_validate({**base, "query": "SELECT 1 ORDER BY 1"}) + assert queried.query == "SELECT 1 ORDER BY 1" + with pytest.raises(ValidationError): + SQL.model_validate(base) + with pytest.raises(ValidationError): + SQL.model_validate({**base, "table": "t", "query": "SELECT 1 ORDER BY 1"}) diff --git a/tests/test_docs_source_of_truth.py b/tests/test_docs_source_of_truth.py index 792472ee..4372ba58 100644 --- a/tests/test_docs_source_of_truth.py +++ b/tests/test_docs_source_of_truth.py @@ -74,6 +74,7 @@ from tablassert.models import ( IPC, NDJSON, + SQL, Annotation, Encoding, Excel, @@ -589,6 +590,7 @@ def location(self) -> str: ModelTableBinding(Parquet, TABLE_CONFIGURATION, ("Parquet Source",)), ModelTableBinding(IPC, TABLE_CONFIGURATION, ("Arrow IPC Source",)), ModelTableBinding(NDJSON, TABLE_CONFIGURATION, ("NDJSON Source",)), + ModelTableBinding(SQL, TABLE_CONFIGURATION, ("SQL (SQLite) Source",)), ModelTableBinding(Reindex, TABLE_CONFIGURATION, ("Reindexing (Conditional Filtering)",)), ModelTableBinding(Statement, TABLE_CONFIGURATION, ("Statement (Triple Definition)",)), ModelTableBinding(NodeEncoding, TABLE_CONFIGURATION, ("NodeEncoding",)), diff --git a/tests/test_e2e_smoke.py b/tests/test_e2e_smoke.py index e22a4195..e83c77f0 100644 --- a/tests/test_e2e_smoke.py +++ b/tests/test_e2e_smoke.py @@ -359,6 +359,73 @@ def test_build_pipeline_over_ndjson_source(tmp_path: Path, monkeypatch: pytest.M assert "HGNC:6871" in edge_text +def test_build_pipeline_over_sql_source(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, rig_factory: Any) -> None: + """SMOKE: the same six-stage build reads a ``kind: sql`` source end to end. + + Mirrors the parquet/ipc smokes exactly except for the source kind: the declared ``kind`` + must select the SQL reader, its positional ``column_1..`` convention must drive the + column encodings (the same ``A``/``B`` letters), and the rows must reach the real redb + and come back as resolved CURIEs in the KGX output. + """ + import sqlite3 + + monkeypatch.chdir(tmp_path) + (tmp_path / ".tablassert" / "store").mkdir(parents=True) + + fullmap: Path = _build_real_redb(tmp_path / "fullmap") + + # Two-row SQLite source: first column = subject term, second column = object term. + data: Path = tmp_path / "data.db" + conn: sqlite3.Connection = sqlite3.connect(data) + conn.execute("CREATE TABLE genes (gene_a TEXT, gene_b TEXT)") + conn.executemany("INSERT INTO genes VALUES (?, ?)", [("brca1", "mapk1"), ("brca1", "mapk1")]) + conn.commit() + conn.close() + + table: Path = tmp_path / "table.yaml" + table_config: dict[str, Any] = { + "template": { + "source": {"kind": "sql", "local": str(data), "url": ["https://example.com/data.db"], "table": "genes"}, + "statement": { + "subject": {"method": "column", "encoding": "A"}, + "predicate": "associated_with", + "object": {"method": "column", "encoding": "B"}, + }, + "provenance": {"repo": "PMC", "publication": "PMC0000000"}, + } + } + to_yaml(table, table_config) + + graph: Path = tmp_path / "graph.yaml" + graph_config: dict[str, Any] = { + "name": "SQL_KG", + "version": "1.0.0", + "tables": [str(table)], + "fullmap": str(fullmap), + "rig": rig_factory(tmp_path, infores_id="infores:sql-kg", source_info={"description": "sql source smoke graph"}), + } + to_yaml(graph, graph_config) + + build_pipeline(graph, PipelineProgress(total_stages=6)) + + nodes: Path = tmp_path / "SQL_KG_1.0.0.nodes.ndjson" + edges: Path = tmp_path / "SQL_KG_1.0.0.edges.ndjson" + assert nodes.is_file() + assert edges.is_file() + + node_text: str = nodes.read_text() + edge_text: str = edges.read_text() + assert node_text.strip() + assert edge_text.strip() + + # Only the real redb knows these CURIEs, so their presence proves the SQL rows flowed + # through load, encode, and resolve under the positional column contract. + assert "HGNC:1100" in node_text + assert "HGNC:6871" in node_text + assert "HGNC:1100" in edge_text + assert "HGNC:6871" in edge_text + + def test_validate_command_happy_path(tmp_path: Path) -> None: """SMOKE (ii): ``validate`` (cli.py wrapper) + ``validate_pipeline`` accept a valid table. diff --git a/tests/test_enums.py b/tests/test_enums.py index 78496351..b0e0e0f5 100644 --- a/tests/test_enums.py +++ b/tests/test_enums.py @@ -87,6 +87,7 @@ def test_files_values() -> None: assert Files.PARQUET == "parquet" assert Files.IPC == "ipc" assert Files.NDJSON == "ndjson" + assert Files.SQL == "sql" def test_comparisons_membership() -> None: