diff --git a/CHANGELOG.md b/CHANGELOG.md index 4bb24097..0c893141 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,7 @@ All notable changes to this project are documented in this file. ### Changed +- **BREAKING: Single `Source` discriminated union replaces repeated source unions in `models.py` and requires `kind`.** `Source` is now defined as `Annotated[Excel | Text | Parquet | IPC | NDJSON, Field(discriminator="kind")]` and used by both `Section.source` and `_section_source_label`. Configs that omit `source.kind` (which previously fell back to `Excel`) now fail validation with `union_tag_not_found`. ([#289](https://github.com/SkyeAv/Tablassert/pull/289), [#286](https://github.com/SkyeAv/Tablassert/issues/286)) - **CI runs only when a release is cut or on manual dispatch.** `.github/workflows/ci.yml` no longer triggers on pull requests; it runs on a push to main that changes `project.version` in `pyproject.toml` (the `chore(release):` commit, detected by the same `detect-pyproject-version-change` action the tag and PyPI workflows use) or via `workflow_dispatch`. The PR-only `changes` path filter, the push-back `autofix` job, the `python-smoke` subset, and the CI `changelog` job are removed; formatting is now a check-only `format` job, and the changelog requirement stays enforced by the pre-push hook. Run `make check` locally before opening a pull request. - **The vulture allowlist lives in `scripts/` next to the other developer tooling.** `vulture_whitelist.py` moves to `scripts/vulture_whitelist.py` unchanged; the `[tool.vulture]` `paths` list drops the now-redundant root entry because `scripts` is already scanned, and the CI `python` path filter, the pre-commit hook comment, and the CONTRIBUTING dead-code section follow the new location. The gate is behaviorally identical: same vulture pin, same config, same allowlist content. ([#287](https://github.com/SkyeAv/Tablassert/pull/287)) diff --git a/docs/configuration/table.md b/docs/configuration/table.md index 6cc43134..93f9df1b 100644 --- a/docs/configuration/table.md +++ b/docs/configuration/table.md @@ -91,7 +91,7 @@ Defines the data file location and format. | Field | Type | Required | Description | | ------- | ------ | ---------- | ------------- | -| `kind` | String | No | Source kind. Model default is `"excel"`, but specify it explicitly in configs. | +| `kind` | String | Yes | Source kind (`"excel"`). Required as the discriminated-union tag. | | `local` | Path | Yes | Local file path the source is read from. The file must already exist here; Tablassert does not download it. | | `url` | List[URL] | Yes | One or more source URLs recorded as provenance (derived sections emit them as the upstream repository supporting entry's `source_record_urls` list, infores:pubmed-central for PMC sections, and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement instead). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. | | `sheet` | String | No | Sheet name. Defaults to `"Sheet1"`. | @@ -111,14 +111,13 @@ source: row_slice: [1, auto] # Start at the second physical row, read to end ``` -> **Specify `kind` explicitly.** Tablassert selects the reader purely from the declared `kind`: `excel` reads a workbook (`sheet`), `text` scans delimited text (`delimiter`), `parquet` scans a parquet file (columns renamed positionally), `ipc` scans an Arrow IPC file (columns renamed positionally); the file on disk is never inspected to infer its format. Because `kind` carries a default, a source whose `kind` is omitted or does not match the actual file is still accepted and fed to the wrong reader, surfacing only later as a read error or garbled rows. Stating `kind` explicitly makes a mis-declared source fail fast. -> **Specify `kind` explicitly.** Tablassert selects the reader purely from the declared `kind`: `excel` reads a workbook (`sheet`), `text` scans delimited text (`delimiter`), `parquet` scans a parquet file (columns renamed positionally), `ndjson` scans newline-delimited JSON (columns renamed positionally); the file on disk is never inspected to infer its format. Because `kind` carries a default, a source whose `kind` is omitted or does not match the actual file is still accepted and fed to the wrong reader, surfacing only later as a read error or garbled rows. Stating `kind` explicitly makes a mis-declared source fail fast. +> **`kind` is required.** Tablassert selects the reader purely from the declared `kind`: `excel` reads a workbook (`sheet`), `text` scans delimited text (`delimiter`), `parquet` scans a parquet file (columns renamed positionally), `ipc` scans an Arrow IPC file (columns renamed positionally), `ndjson` scans newline-delimited JSON (columns renamed positionally); the file on disk is never inspected to infer its format. Omitting `kind` fails config validation (`union_tag_not_found`). #### Text Source (CSV/TSV) | Field | Type | Required | Description | | ------- | ------ | ---------- | ------------- | -| `kind` | String | No | Source kind. Model default is `"text"`, but specify it explicitly in configs. | +| `kind` | String | Yes | Source kind (`"text"`). Required as the discriminated-union tag. | | `local` | Path | Yes | Local file path the source is read from. The file must already exist here; Tablassert does not download it. | | `url` | List[URL] | Yes | One or more source URLs recorded as provenance (derived sections emit them as the upstream repository supporting entry's `source_record_urls` list, infores:pubmed-central for PMC sections, and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement instead). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. | | `delimiter` | String | No | Field delimiter. Defaults to `","`. | @@ -150,7 +149,7 @@ as-is. | Field | Type | Required | Description | | ------- | ------ | ---------- | ------------- | -| `kind` | String | No | Source kind. Model default is `"parquet"`, but specify it explicitly in configs. | +| `kind` | String | Yes | Source kind (`"parquet"`). Required as the discriminated-union tag. | | `local` | Path | Yes | Local file path the source is read from. The file must already exist here; Tablassert does not download it. | | `url` | List[URL] | Yes | One or more source URLs recorded as provenance (derived sections emit them as the upstream repository supporting entry's `source_record_urls` list, infores:pubmed-central for PMC sections, and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement instead). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. | | `row_slice` | List[NonNegativeInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a non-negative integer or `"auto"`. Mutually exclusive with `rows`. | @@ -181,7 +180,7 @@ Only IPC **file layout** is supported; a stream-layout file fails with a clear | Field | Type | Required | Description | | ------- | ------ | ---------- | ------------- | -| `kind` | String | No | Source kind. Model default is `"ipc"`, but specify it explicitly in configs. | +| `kind` | String | Yes | Source kind (`"ipc"`). Required as the discriminated-union tag. | | `local` | Path | Yes | Local file path the source is read from. The file must already exist here; Tablassert does not download it. | | `url` | List[URL] | Yes | One or more source URLs recorded as provenance (derived sections emit them as the upstream repository supporting entry's `source_record_urls` list, infores:pubmed-central for PMC sections, and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement instead). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. | | `row_slice` | List[NonNegativeInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a non-negative integer or `"auto"`. Mutually exclusive with `rows`. | @@ -214,7 +213,7 @@ from no records. | Field | Type | Required | Description | | ------- | ------ | ---------- | ------------- | -| `kind` | String | No | Source kind. Model default is `"ndjson"`, but specify it explicitly in configs. | +| `kind` | String | Yes | Source kind (`"ndjson"`). Required as the discriminated-union tag. | | `local` | Path | Yes | Local file path the source is read from. The file must already exist here; Tablassert does not download it. | | `url` | List[URL] | Yes | One or more source URLs recorded as provenance (derived sections emit them as the upstream repository supporting entry's `source_record_urls` list, infores:pubmed-central for PMC sections, and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement instead). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. | | `row_slice` | List[NonNegativeInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a non-negative integer or `"auto"`. Mutually exclusive with `rows`. | diff --git a/src/tablassert/models.py b/src/tablassert/models.py index f0aecc03..f6ff09e0 100644 --- a/src/tablassert/models.py +++ b/src/tablassert/models.py @@ -3,7 +3,7 @@ import re import warnings from pathlib import Path -from typing import TYPE_CHECKING, Any, Literal, Self +from typing import TYPE_CHECKING, Annotated, Any, Literal, Self, TypeAlias from pydantic import BaseModel, ConfigDict, Field, HttpUrl, NonNegativeInt, PositiveInt, field_validator, model_validator @@ -76,7 +76,7 @@ def _shown(name: str, target: str) -> str: return f"`{name}` (coerced to `{target}`)" if target != name else f"`{name}`" -def _section_source_label(source: Excel | Text | Parquet | IPC | NDJSON) -> str: +def _section_source_label(source: Source) -> str: """Render a section's source for a diagnostic message: the local path, plus the sheet for Excel. Args: @@ -300,6 +300,9 @@ class NDJSON(BaseSource): kind: Literal[Files.NDJSON] = Field(Files.NDJSON, description="Source kind; must be 'ndjson'.") +Source: TypeAlias = Annotated[Excel | Text | Parquet | IPC | NDJSON, Field(discriminator="kind")] + + class Regex(TablaBase): pattern: int | float | str = Field(..., description="Regex pattern passed to string replacement.", examples=["\\s+", "\\.$"]) @@ -994,7 +997,7 @@ def warn_when_the_slot_cannot_reach_the_edge(self) -> Self: class Section(TablaBase): """Pydantic section model and coercion target for a single table configuration.""" - source: Excel | Text | Parquet | IPC | NDJSON = Field(..., description="Input source definition for reading tabular rows.") + source: Source = Field(..., description="Input source definition for reading tabular rows.") statement: Statement = Field(..., description="Subject-object statement mapping for this section.") provenance: Provenance = Field(..., description="Provenance metadata applied to all produced edges.") annotations: list[Annotation] | None = Field(None, description="Optional extra encoded columns added to each row.") diff --git a/tests/test_cover_models.py b/tests/test_cover_models.py index 54bb1e70..6b700341 100644 --- a/tests/test_cover_models.py +++ b/tests/test_cover_models.py @@ -149,3 +149,16 @@ def test_section_source_label_per_kind() -> None: assert _section_source_label(NDJSON.model_validate({"kind": "ndjson", "local": "./t.ndjson", "url": urls})) == "`t.ndjson`" excel: Excel = Excel.model_validate({"kind": "excel", "local": "./t.xlsx", "url": urls, "sheet": "S1"}) assert _section_source_label(excel) == "`t.xlsx` (sheet `S1`)" + + +def test_section_source_requires_kind() -> None: + """The discriminated ``Source`` union rejects payloads omitting ``kind``.""" + from pydantic import ValidationError + + payload: dict[str, Any] = { + "source": {"local": "./table.csv", "url": ["https://example.com/table.csv"]}, + "statement": {"subject": {"method": "value", "encoding": "BRCA1"}, "object": {"method": "value", "encoding": "TP53"}}, + "provenance": {"repo": "PMC", "publication": "PMC0000000"}, + } + with pytest.raises(ValidationError, match="union_tag_not_found"): + Section.model_validate(payload)