Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ All notable changes to this project are documented in this file.
- **NDJSON (JSONL) is a fifth table source kind.** `kind: ndjson` scans newline-delimited JSON lazily with `pl.scan_ndjson` and renames its columns positionally to `column_1..` in first-seen key order, matching the `parquet` reader contract so `method: column` encodings keep using Excel-style letters. Ragged records scan missing keys as nulls instead of dropping rows; an empty file fails fast with a coded `ndjson-empty` error instead of polars' opaque `Cannot infer NDJSON types` compute error. ([#283](https://github.com/SkyeAv/Tablassert/pull/283), [#262](https://github.com/SkyeAv/Tablassert/issues/262))
- **Every pull request must carry a new changelog entry.** A stdlib-only `scripts/check_changelog.py` requires at least one new bullet under the `## Unreleased` heading of `CHANGELOG.md` relative to the pull request's merge base, failing with remediation guidance when whitespace, subheadings, or released-history edits are the only additions; a CI job runs it on every pull request including forks, release-bump and dependency-bump pull requests are exempt by their `chore(release):` / `chore(deps):` title or commit subject, documentation-only pull requests (every changed path is markdown or under `docs/`) are exempt via `--exempt-docs-only`, and a pre-push hook mirrors the same check locally. ([#284](https://github.com/SkyeAv/Tablassert/pull/284), [#282](https://github.com/SkyeAv/Tablassert/issues/282))
- **Arrow IPC (Feather v2) is a fourth table source kind.** `kind: ipc` scans a `.arrow` file lazily with `pl.scan_ipc` and renames its columns positionally to `column_1..` in schema order, matching the `parquet` reader contract so `method: column` encodings keep using Excel-style letters. Only IPC file layout is supported; a stream-layout file fails fast with a coded `ipc-stream-unsupported` error naming the conversion fix instead of polars' opaque `InvalidFooter` compute error. ([#281](https://github.com/SkyeAv/Tablassert/pull/281), [#264](https://github.com/SkyeAv/Tablassert/issues/264))
- **SQL (SQLite) is a sixth table source kind.** `kind: sql` reads a table or query result from a SQLite database via the stdlib `sqlite3` driver (zero new dependencies), with `local` carrying the database file and exactly one of `table` / `query` selecting the rows. `table` scans in `rowid` order and a `query` must carry its own `ORDER BY` (coded `sql-query-needs-order-by` otherwise), so row indices and `row:<N>` StudyResult ids stay reproducible across runs; columns rename positionally to `column_1..` per the reader contract. ([#285](https://github.com/SkyeAv/Tablassert/pull/285), [#263](https://github.com/SkyeAv/Tablassert/issues/263))

### Fixed

Expand Down
39 changes: 37 additions & 2 deletions docs/configuration/table.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,8 +111,7 @@ source:
row_slice: [1, auto] # Start at the second physical row, read to end
```

> **Specify `kind` explicitly.** Tablassert selects the reader purely from the declared `kind`: `excel` reads a workbook (`sheet`), `text` scans delimited text (`delimiter`), `parquet` scans a parquet file (columns renamed positionally), `ipc` scans an Arrow IPC file (columns renamed positionally); the file on disk is never inspected to infer its format. Because `kind` carries a default, a source whose `kind` is omitted or does not match the actual file is still accepted and fed to the wrong reader, surfacing only later as a read error or garbled rows. Stating `kind` explicitly makes a mis-declared source fail fast.
> **Specify `kind` explicitly.** Tablassert selects the reader purely from the declared `kind`: `excel` reads a workbook (`sheet`), `text` scans delimited text (`delimiter`), `parquet` scans a parquet file (columns renamed positionally), `ndjson` scans newline-delimited JSON (columns renamed positionally); the file on disk is never inspected to infer its format. Because `kind` carries a default, a source whose `kind` is omitted or does not match the actual file is still accepted and fed to the wrong reader, surfacing only later as a read error or garbled rows. Stating `kind` explicitly makes a mis-declared source fail fast.
> **Specify `kind` explicitly.** Tablassert selects the reader purely from the declared `kind`: `excel` reads a workbook (`sheet`), `text` scans delimited text (`delimiter`), `parquet` scans a parquet file (columns renamed positionally), `ipc` scans an Arrow IPC file (columns renamed positionally), `ndjson` scans newline-delimited JSON (columns renamed positionally), `sql` queries a SQLite database (`table` or `query`); the file on disk is never inspected to infer its format. Because `kind` carries a default, a source whose `kind` is omitted or does not match the actual file is still accepted and fed to the wrong reader, surfacing only later as a read error or garbled rows. Stating `kind` explicitly makes a mis-declared source fail fast.

#### Text Source (CSV/TSV)

Expand Down Expand Up @@ -232,6 +231,42 @@ source:
row_slice: [1, auto]
```

#### SQL (SQLite) Source

Reads a table or query result from a SQLite database via the stdlib `sqlite3` driver (zero
extra dependencies; Postgres/MySQL/DuckDB connection strings are a documented follow-up).
`local` carries the database file, as with the file kinds. Columns are addressed
positionally: the reader renames whatever names the result carries to `column_1`..`column_n`
in result order (the same convention the `parquet` reader produces), so `method: column`
encodings keep using Excel-style letters and the table's own column names are never
referenced from configs. Row order is deterministic by contract: `table` scans as
`SELECT * FROM "table" ORDER BY rowid`, and a `query` must carry its own `ORDER BY` (a
missing one fails fast with `sql-query-needs-order-by`) so row indices and `row:<N>`
StudyResult ids are reproducible across runs. Exactly one of `table` / `query` is required.

| Field | Type | Required | Description |
| ------- | ------ | ---------- | ------------- |
| `kind` | String | No | Source kind. Model default is `"sql"`, but specify it explicitly in configs. |
| `local` | Path | Yes | Local path of the SQLite database file. The file must already exist here; Tablassert does not download it. |
| `url` | List[URL] | Yes | One or more source URLs recorded as provenance (derived sections emit them as the upstream repository supporting entry's `source_record_urls` list, infores:pubmed-central for PMC sections, and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement instead). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. |
| `table` | String | No | Table to scan, in `rowid` order. Exactly one of `table` / `query`. Plain identifiers only (letters, digits, underscore, not starting with a digit). |
| `query` | String | No | Full `SELECT` query; must carry an `ORDER BY` clause. Exactly one of `table` / `query`. |
| `row_slice` | List[NonNegativeInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a non-negative integer or `"auto"`. Mutually exclusive with `rows`. |
| `rows` | List[NonNegativeInt] | No | Zero-based row indices to keep after any `row_slice` crop. Mutually exclusive with `row_slice`. |
| `reindex` | List[Reindex] | No | Conditional row filtering |

**Example:**

```yaml
source:
kind: sql
local: ./data/curated.db
url:
- https://example.com/curated.db
table: genes
row_slice: [1, auto]
```

#### Reindexing (Conditional Filtering)

Filter rows based on column values.
Expand Down
1 change: 1 addition & 0 deletions src/tablassert/enums.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ class Files(str, Enum):
PARQUET = "parquet"
IPC = "ipc"
NDJSON = "ndjson"
SQL = "sql"


class EncodingMethods(str, Enum):
Expand Down
4 changes: 4 additions & 0 deletions src/tablassert/errors.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,10 @@ def redact_secrets(text: str, secrets: Sequence[str] = ()) -> str:
"source-file-unreadable",
"ipc-stream-unsupported",
"ndjson-empty",
"sql-table-and-query-conflict",
"sql-query-needs-order-by",
"sql-invalid-table-name",
"sql-query-not-read-only",
"resolve-bad-specs",
"taxon-from-conflict",
"taxon-from-unknown-column",
Expand Down
125 changes: 124 additions & 1 deletion src/tablassert/lib.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
from __future__ import annotations

import contextlib
import json
import operator
import re
Expand Down Expand Up @@ -46,7 +47,7 @@
study_size_target,
)
from tablassert.enums import Comparisons, EncodingMethods, Files, Functions, InformationResources, Repositories, Tokens
from tablassert.errors import TablassertError
from tablassert.errors import SourceFileError, TablassertError
from tablassert.fullmap import ResolveSpec, fullmap_db_path, resolve, resolve_batch
from tablassert.log import cat
from tablassert.models import EDGE_ID_PLACEHOLDER, Encoding, NodeEncoding, Qualifier, Reindex, RIGConfig, Section
Expand Down Expand Up @@ -986,6 +987,125 @@ def parquet(p: Path) -> pl.LazyFrame:
return lf.rename(dict(zip(names, (f"column_{i + 1}" for i in range(len(names))), strict=True)))


def _has_top_level_order_by(query: str) -> bool:
"""Return True if ``query`` contains an ``ORDER BY`` clause at parenthesis depth 0 outside comments and literals."""
tokens: list[str] = []
depth: int = 0
i: int = 0
n: int = len(query)
while i < n:
ch: str = query[i]
nxt: str = query[i + 1] if i + 1 < n else ""
if ch == "-" and nxt == "-":
end: int = query.find("\n", i + 2)
i = n if end == -1 else end + 1
tokens.append(" ")
continue
if ch == "/" and nxt == "*":
end = query.find("*/", i + 2)
i = n if end == -1 else end + 2
tokens.append(" ")
continue
if ch in ("'", '"', "`"):
quote: str = ch
i += 1
while i < n:
if query[i] == quote:
if i + 1 < n and query[i + 1] == quote:
i += 2
continue
i += 1
break
i += 1
tokens.append(" ")
continue
if ch == "[":
end = query.find("]", i + 1)
i = n if end == -1 else end + 1
tokens.append(" ")
continue
if ch == "(":
depth += 1
i += 1
continue
if ch == ")":
depth = max(0, depth - 1)
i += 1
continue
if depth == 0:
tokens.append(ch)
i += 1
top_level: str = "".join(tokens)
return re.search(r"\bORDER\s+BY\b", top_level, re.IGNORECASE) is not None


def sql(p: Path, table: str | None, query: str | None) -> pl.LazyFrame:
"""Read a SQLite source as a LazyFrame with positional ``column_<n>`` names.

Args:
p: Path to the SQLite database file.
table: Table to scan as ``SELECT * FROM "table" ORDER BY rowid``.
query: Full SELECT query; must carry its own top-level ``ORDER BY`` clause.

Returns:
LazyFrame whose columns are named ``column_1..`` in result order.

Raises:
SourceFileError: ``source-file-unreadable`` when ``p`` does not exist as a file.
TablassertError: ``sql-invalid-table-name`` when ``table`` is not a plain identifier.
TablassertError: ``sql-query-needs-order-by`` when a ``query`` has no top-level ORDER BY —
row indices and ``row:<N>`` StudyResult ids must be reproducible across runs.
TablassertError: ``sql-query-not-read-only`` when a ``query`` attempts writes or non-read-only operations.

Notes:
The result carries its own column names, but the config contract addresses columns
positionally (Excel-style ``A-ZZ`` encoding letters resolve to ``column_<n>``), so
the reader names result columns ``column_1..`` directly at DataFrame construction --
the same convention the ``parquet`` reader produces. Opens the database in read-only
mode (``?mode=ro``) with an authorizer permitting only ``SELECT`` / ``READ`` /
``FUNCTION``. Uses stdlib ``sqlite3``: zero new dependencies, single-engine by design;
multi-engine ``connection`` strings are a documented follow-up. ``table`` quoting uses
bound-parameter-style double quotes with the identifier validated as a plain name first.
"""
import sqlite3

if not p.is_file():
raise SourceFileError(p, "SQLite database file does not exist.")

if query is not None:
if not _has_top_level_order_by(query):
raise TablassertError(
f"`sql` query for {p} has no top-level ORDER BY clause; row order (and row:<N> ids) would not be reproducible across runs.",
code="sql-query-needs-order-by",
)
statement: str = query
else:
if not table or not table.replace("_", "a").isalnum() or table[0].isdigit():
raise TablassertError(f"`sql` table name {table!r} for {p} is not a plain identifier.", code="sql-invalid-table-name")
statement = f'SELECT * FROM "{table}" ORDER BY rowid'

allowed_actions: frozenset[int] = frozenset({sqlite3.SQLITE_SELECT, sqlite3.SQLITE_READ, sqlite3.SQLITE_FUNCTION, sqlite3.SQLITE_RECURSIVE})

def _authorizer(action: int, _arg1: str | None, _arg2: str | None, _db_name: str | None, _trigger: str | None) -> int:
return sqlite3.SQLITE_OK if action in allowed_actions else sqlite3.SQLITE_DENY

uri: str = f"{p.resolve().as_uri()}?mode=ro"
with contextlib.closing(sqlite3.connect(uri, uri=True)) as conn:
conn.set_authorizer(_authorizer)
try:
cur: sqlite3.Cursor = conn.execute(statement)
if cur.description is None:
raise TablassertError(f"`sql` query for {p} did not return a result set.", code="sql-query-not-read-only")
col_names: list[str] = [f"column_{i + 1}" for i in range(len(cur.description))]
rows: list[tuple[Any, ...]] = cur.fetchall()
except sqlite3.DatabaseError as exc:
if "not authorized" in str(exc).lower():
raise TablassertError(f"`sql` query for {p} must be read-only: {exc}", code="sql-query-not-read-only") from exc
raise
df: pl.DataFrame = pl.DataFrame(rows, schema=col_names, orient="row") if rows else pl.DataFrame({n: [] for n in col_names})
return df.lazy()


def ipc(p: Path) -> pl.LazyFrame:
"""Read an Arrow IPC (Feather v2 / `.arrow`) source as a LazyFrame with positional ``column_<n>`` names.

Expand Down Expand Up @@ -1573,6 +1693,9 @@ def _source_ops(self: Self) -> list[Any]:
(parquet, (self.source.local,)) if self.source.kind == Files.PARQUET else None, # pyright: ignore
(ipc, (self.source.local,)) if self.source.kind == Files.IPC else None, # pyright: ignore
(ndjson, (self.source.local,)) if self.source.kind == Files.NDJSON else None, # pyright: ignore
(sql, (self.source.local, getattr(self.source, "table", None), getattr(self.source, "query", None)))
if self.source.kind == Files.SQL
else None, # pyright: ignore
(idx, ()),
(crop, (self.source.row_slice,)) if self.source.row_slice else None,
(pick, (self.source.rows,)) if self.source.rows else None,
Expand Down
26 changes: 24 additions & 2 deletions src/tablassert/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ def _shown(name: str, target: str) -> str:
return f"`{name}` (coerced to `{target}`)" if target != name else f"`{name}`"


def _section_source_label(source: Excel | Text | Parquet | IPC | NDJSON) -> str:
def _section_source_label(source: Excel | Text | Parquet | IPC | NDJSON | SQL) -> str:
"""Render a section's source for a diagnostic message: the local path, plus the sheet for Excel.

Args:
Expand Down Expand Up @@ -300,6 +300,28 @@ class NDJSON(BaseSource):
kind: Literal[Files.NDJSON] = Field(Files.NDJSON, description="Source kind; must be 'ndjson'.")


class SQL(BaseSource):
"""SQLite database source: ``local`` carries the database file (as with the file kinds).

Exactly one of ``table`` / ``query`` selects the rows. ``table`` scans the whole table
in ``rowid`` order; a ``query`` must carry its own ``ORDER BY`` so row indices and
``row:<N>`` StudyResult ids stay stable across runs.
"""

kind: Literal[Files.SQL] = Field(Files.SQL, description="Source kind; must be 'sql'.")
table: str | None = Field(None, description="Table to scan; scanned in rowid order. Mutually exclusive with `query`.")
query: str | None = Field(None, description="Full SELECT query; must carry an ORDER BY clause. Mutually exclusive with `table`.")

@model_validator(mode="after")
def exactly_one_selector(self: Self) -> Self:
if bool(self.table) == bool(self.query):
raise TablassertValidationError(
"A `sql` source needs exactly one of `table` or `query` (mutually exclusive).", code="sql-table-and-query-conflict"
)

return self


class Regex(TablaBase):
pattern: int | float | str = Field(..., description="Regex pattern passed to string replacement.", examples=["\\s+", "\\.$"])

Expand Down Expand Up @@ -994,7 +1016,7 @@ def warn_when_the_slot_cannot_reach_the_edge(self) -> Self:
class Section(TablaBase):
"""Pydantic section model and coercion target for a single table configuration."""

source: Excel | Text | Parquet | IPC | NDJSON = Field(..., description="Input source definition for reading tabular rows.")
source: Excel | Text | Parquet | IPC | NDJSON | SQL = Field(..., description="Input source definition for reading tabular rows.")
statement: Statement = Field(..., description="Subject-object statement mapping for this section.")
provenance: Provenance = Field(..., description="Provenance metadata applied to all produced edges.")
annotations: list[Annotation] | None = Field(None, description="Optional extra encoded columns added to each row.")
Expand Down
1 change: 1 addition & 0 deletions src/tablassert/runcache.py
Original file line number Diff line number Diff line change
Expand Up @@ -255,6 +255,7 @@ def prefix_digest(ops: list[tuple[Callable, tuple[Any, ...]]]) -> str:
lib.parquet: 2,
lib.ipc: 2,
lib.ndjson: 2,
lib.sql: 2,
# Filter: single lazy row predicates/slices.
lib.idx: 1,
lib.crop: 1,
Expand Down
Loading