Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ All notable changes to this project are documented in this file.

### Changed

- **BREAKING: Single `Source` discriminated union replaces repeated source unions in `models.py` and requires `kind`.** `Source` is now defined as `Annotated[Excel | Text | Parquet | IPC | NDJSON, Field(discriminator="kind")]` and used by both `Section.source` and `_section_source_label`. Configs that omit `source.kind` (which previously fell back to `Excel`) now fail validation with `union_tag_not_found`. ([#289](https://github.com/SkyeAv/Tablassert/pull/289), [#286](https://github.com/SkyeAv/Tablassert/issues/286))
- **CI runs only when a release is cut or on manual dispatch.** `.github/workflows/ci.yml` no longer triggers on pull requests; it runs on a push to main that changes `project.version` in `pyproject.toml` (the `chore(release):` commit, detected by the same `detect-pyproject-version-change` action the tag and PyPI workflows use) or via `workflow_dispatch`. The PR-only `changes` path filter, the push-back `autofix` job, the `python-smoke` subset, and the CI `changelog` job are removed; formatting is now a check-only `format` job, and the changelog requirement stays enforced by the pre-push hook. Run `make check` locally before opening a pull request.
- **The vulture allowlist lives in `scripts/` next to the other developer tooling.** `vulture_whitelist.py` moves to `scripts/vulture_whitelist.py` unchanged; the `[tool.vulture]` `paths` list drops the now-redundant root entry because `scripts` is already scanned, and the CI `python` path filter, the pre-commit hook comment, and the CONTRIBUTING dead-code section follow the new location. The gate is behaviorally identical: same vulture pin, same config, same allowlist content. ([#287](https://github.com/SkyeAv/Tablassert/pull/287))

Expand Down
13 changes: 6 additions & 7 deletions docs/configuration/table.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ Defines the data file location and format.

| Field | Type | Required | Description |
| ------- | ------ | ---------- | ------------- |
| `kind` | String | No | Source kind. Model default is `"excel"`, but specify it explicitly in configs. |
| `kind` | String | Yes | Source kind (`"excel"`). Required as the discriminated-union tag. |
| `local` | Path | Yes | Local file path the source is read from. The file must already exist here; Tablassert does not download it. |
| `url` | List[URL] | Yes | One or more source URLs recorded as provenance (derived sections emit them as the upstream repository supporting entry's `source_record_urls` list, infores:pubmed-central for PMC sections, and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement instead). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. |
| `sheet` | String | No | Sheet name. Defaults to `"Sheet1"`. |
Expand All @@ -111,14 +111,13 @@ source:
row_slice: [1, auto] # Start at the second physical row, read to end
```

> **Specify `kind` explicitly.** Tablassert selects the reader purely from the declared `kind`: `excel` reads a workbook (`sheet`), `text` scans delimited text (`delimiter`), `parquet` scans a parquet file (columns renamed positionally), `ipc` scans an Arrow IPC file (columns renamed positionally); the file on disk is never inspected to infer its format. Because `kind` carries a default, a source whose `kind` is omitted or does not match the actual file is still accepted and fed to the wrong reader, surfacing only later as a read error or garbled rows. Stating `kind` explicitly makes a mis-declared source fail fast.
> **Specify `kind` explicitly.** Tablassert selects the reader purely from the declared `kind`: `excel` reads a workbook (`sheet`), `text` scans delimited text (`delimiter`), `parquet` scans a parquet file (columns renamed positionally), `ndjson` scans newline-delimited JSON (columns renamed positionally); the file on disk is never inspected to infer its format. Because `kind` carries a default, a source whose `kind` is omitted or does not match the actual file is still accepted and fed to the wrong reader, surfacing only later as a read error or garbled rows. Stating `kind` explicitly makes a mis-declared source fail fast.
> **`kind` is required.** Tablassert selects the reader purely from the declared `kind`: `excel` reads a workbook (`sheet`), `text` scans delimited text (`delimiter`), `parquet` scans a parquet file (columns renamed positionally), `ipc` scans an Arrow IPC file (columns renamed positionally), `ndjson` scans newline-delimited JSON (columns renamed positionally); the file on disk is never inspected to infer its format. Omitting `kind` fails config validation (`union_tag_not_found`).

#### Text Source (CSV/TSV)

| Field | Type | Required | Description |
| ------- | ------ | ---------- | ------------- |
| `kind` | String | No | Source kind. Model default is `"text"`, but specify it explicitly in configs. |
| `kind` | String | Yes | Source kind (`"text"`). Required as the discriminated-union tag. |
| `local` | Path | Yes | Local file path the source is read from. The file must already exist here; Tablassert does not download it. |
| `url` | List[URL] | Yes | One or more source URLs recorded as provenance (derived sections emit them as the upstream repository supporting entry's `source_record_urls` list, infores:pubmed-central for PMC sections, and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement instead). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. |
| `delimiter` | String | No | Field delimiter. Defaults to `","`. |
Expand Down Expand Up @@ -150,7 +149,7 @@ as-is.

| Field | Type | Required | Description |
| ------- | ------ | ---------- | ------------- |
| `kind` | String | No | Source kind. Model default is `"parquet"`, but specify it explicitly in configs. |
| `kind` | String | Yes | Source kind (`"parquet"`). Required as the discriminated-union tag. |
| `local` | Path | Yes | Local file path the source is read from. The file must already exist here; Tablassert does not download it. |
| `url` | List[URL] | Yes | One or more source URLs recorded as provenance (derived sections emit them as the upstream repository supporting entry's `source_record_urls` list, infores:pubmed-central for PMC sections, and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement instead). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. |
| `row_slice` | List[NonNegativeInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a non-negative integer or `"auto"`. Mutually exclusive with `rows`. |
Expand Down Expand Up @@ -181,7 +180,7 @@ Only IPC **file layout** is supported; a stream-layout file fails with a clear

| Field | Type | Required | Description |
| ------- | ------ | ---------- | ------------- |
| `kind` | String | No | Source kind. Model default is `"ipc"`, but specify it explicitly in configs. |
| `kind` | String | Yes | Source kind (`"ipc"`). Required as the discriminated-union tag. |
| `local` | Path | Yes | Local file path the source is read from. The file must already exist here; Tablassert does not download it. |
| `url` | List[URL] | Yes | One or more source URLs recorded as provenance (derived sections emit them as the upstream repository supporting entry's `source_record_urls` list, infores:pubmed-central for PMC sections, and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement instead). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. |
| `row_slice` | List[NonNegativeInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a non-negative integer or `"auto"`. Mutually exclusive with `rows`. |
Expand Down Expand Up @@ -214,7 +213,7 @@ from no records.

| Field | Type | Required | Description |
| ------- | ------ | ---------- | ------------- |
| `kind` | String | No | Source kind. Model default is `"ndjson"`, but specify it explicitly in configs. |
| `kind` | String | Yes | Source kind (`"ndjson"`). Required as the discriminated-union tag. |
| `local` | Path | Yes | Local file path the source is read from. The file must already exist here; Tablassert does not download it. |
| `url` | List[URL] | Yes | One or more source URLs recorded as provenance (derived sections emit them as the upstream repository supporting entry's `source_record_urls` list, infores:pubmed-central for PMC sections, and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement instead). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. |
| `row_slice` | List[NonNegativeInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a non-negative integer or `"auto"`. Mutually exclusive with `rows`. |
Expand Down
9 changes: 6 additions & 3 deletions src/tablassert/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
import re
import warnings
from pathlib import Path
from typing import TYPE_CHECKING, Any, Literal, Self
from typing import TYPE_CHECKING, Annotated, Any, Literal, Self, TypeAlias

from pydantic import BaseModel, ConfigDict, Field, HttpUrl, NonNegativeInt, PositiveInt, field_validator, model_validator

Expand Down Expand Up @@ -76,7 +76,7 @@ def _shown(name: str, target: str) -> str:
return f"`{name}` (coerced to `{target}`)" if target != name else f"`{name}`"


def _section_source_label(source: Excel | Text | Parquet | IPC | NDJSON) -> str:
def _section_source_label(source: Source) -> str:
"""Render a section's source for a diagnostic message: the local path, plus the sheet for Excel.

Args:
Expand Down Expand Up @@ -300,6 +300,9 @@ class NDJSON(BaseSource):
kind: Literal[Files.NDJSON] = Field(Files.NDJSON, description="Source kind; must be 'ndjson'.")


Source: TypeAlias = Annotated[Excel | Text | Parquet | IPC | NDJSON, Field(discriminator="kind")]


class Regex(TablaBase):
pattern: int | float | str = Field(..., description="Regex pattern passed to string replacement.", examples=["\\s+", "\\.$"])

Expand Down Expand Up @@ -994,7 +997,7 @@ def warn_when_the_slot_cannot_reach_the_edge(self) -> Self:
class Section(TablaBase):
"""Pydantic section model and coercion target for a single table configuration."""

source: Excel | Text | Parquet | IPC | NDJSON = Field(..., description="Input source definition for reading tabular rows.")
source: Source = Field(..., description="Input source definition for reading tabular rows.")
statement: Statement = Field(..., description="Subject-object statement mapping for this section.")
provenance: Provenance = Field(..., description="Provenance metadata applied to all produced edges.")
annotations: list[Annotation] | None = Field(None, description="Optional extra encoded columns added to each row.")
Expand Down
13 changes: 13 additions & 0 deletions tests/test_cover_models.py
Original file line number Diff line number Diff line change
Expand Up @@ -149,3 +149,16 @@ def test_section_source_label_per_kind() -> None:
assert _section_source_label(NDJSON.model_validate({"kind": "ndjson", "local": "./t.ndjson", "url": urls})) == "`t.ndjson`"
excel: Excel = Excel.model_validate({"kind": "excel", "local": "./t.xlsx", "url": urls, "sheet": "S1"})
assert _section_source_label(excel) == "`t.xlsx` (sheet `S1`)"


def test_section_source_requires_kind() -> None:
"""The discriminated ``Source`` union rejects payloads omitting ``kind``."""
from pydantic import ValidationError

payload: dict[str, Any] = {
"source": {"local": "./table.csv", "url": ["https://example.com/table.csv"]},
"statement": {"subject": {"method": "value", "encoding": "BRCA1"}, "object": {"method": "value", "encoding": "TP53"}},
"provenance": {"repo": "PMC", "publication": "PMC0000000"},
}
with pytest.raises(ValidationError, match="union_tag_not_found"):
Section.model_validate(payload)