diff --git a/CHANGELOG.md b/CHANGELOG.md index b13f2edd..2c75eee5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,7 @@ All notable changes to this project are documented in this file. ### Added +- **`identified` passthrough skips fullmap resolution for pre-identified CURIE columns.** `subject` and `object` encodings can declare `identified: {name: , category: }` to emit CURIEs, names, and Biolink categories directly without touching a fullmap database (mutually exclusive with resolution options such as `taxon`, `prioritize`, `avoid`, `exclude_prefixes`, and `exclude_regex`). When every node column in a graph uses `identified`, `Graph.fullmap` may be omitted. ([#272](https://github.com/SkyeAv/Tablassert/issues/272)) - **NDJSON (JSONL) is a fifth table source kind.** `kind: ndjson` scans newline-delimited JSON lazily with `pl.scan_ndjson` and renames its columns positionally to `column_1..` in first-seen key order, matching the `parquet` reader contract so `method: column` encodings keep using Excel-style letters. Ragged records scan missing keys as nulls instead of dropping rows; an empty file fails fast with a coded `ndjson-empty` error instead of polars' opaque `Cannot infer NDJSON types` compute error. ([#283](https://github.com/SkyeAv/Tablassert/pull/283), [#262](https://github.com/SkyeAv/Tablassert/issues/262)) - **Every pull request must carry a new changelog entry.** A stdlib-only `scripts/check_changelog.py` requires at least one new bullet under the `## Unreleased` heading of `CHANGELOG.md` relative to the pull request's merge base, failing with remediation guidance when whitespace, subheadings, or released-history edits are the only additions; a CI job runs it on every pull request including forks, release-bump and dependency-bump pull requests are exempt by their `chore(release):` / `chore(deps):` title or commit subject, documentation-only pull requests (every changed path is markdown or under `docs/`) are exempt via `--exempt-docs-only`, and a pre-push hook mirrors the same check locally. ([#284](https://github.com/SkyeAv/Tablassert/pull/284), [#282](https://github.com/SkyeAv/Tablassert/issues/282)) - **Arrow IPC (Feather v2) is a fourth table source kind.** `kind: ipc` scans a `.arrow` file lazily with `pl.scan_ipc` and renames its columns positionally to `column_1..` in schema order, matching the `parquet` reader contract so `method: column` encodings keep using Excel-style letters. Only IPC file layout is supported; a stream-layout file fails fast with a coded `ipc-stream-unsupported` error naming the conversion fix instead of polars' opaque `InvalidFooter` compute error. ([#281](https://github.com/SkyeAv/Tablassert/pull/281), [#264](https://github.com/SkyeAv/Tablassert/issues/264)) diff --git a/docs/configuration/graph.md b/docs/configuration/graph.md index 2abfcd40..a7409c4b 100644 --- a/docs/configuration/graph.md +++ b/docs/configuration/graph.md @@ -22,7 +22,6 @@ QC auditing and verbose logging are controlled at build time via the `build-kg - | `name` | String | Knowledge graph name (used in output filenames and the RIG) | | `version` | String | Knowledge graph version (used in output filenames) | | `tables` | List[Path] | Paths to table configuration YAML files | -| `fullmap` | Path | Path to the fullmap redb file, or a base directory containing it | | `rig` | RIGConfig | Resource Ingest Guide metadata (see below) | The legacy top-level RIG fields (`description`, `contributions`, `ui_explanation`, `infores`) are **rejected** with a migration pointer; they now live under `rig:`. @@ -31,6 +30,7 @@ The legacy top-level RIG fields (`description`, `contributions`, `ui_explanation | Field | Type | Description | | ------- | ------ | ------------- | +| `fullmap` | Path | Path to the fullmap redb file, or a base directory containing it. Required whenever any table section uses fullmap resolution; optional when every subject/object uses `identified` passthrough | | `uuid_fields` | List[str] | Edge fields that constitute edge identity. Only these feed the derived edge `id` (see [Stable edge ids](#stable-edge-ids)) | | `uuid_domain` | String | Explicit UUID namespace. Defaults to `rig.source_info.infores_id` when `uuid_fields` is set, `TABLASSERT` otherwise | | `uuid_on_collision` | `error` \| `merge` | What to do when two different edges derive one id. `error` (default) aborts; `merge` folds them into one edge. Requires `uuid_fields` (see [Merging collisions instead](#merging-collisions-instead)) | diff --git a/docs/configuration/table.md b/docs/configuration/table.md index 6cc43134..3ff7bcca 100644 --- a/docs/configuration/table.md +++ b/docs/configuration/table.md @@ -372,6 +372,7 @@ Defines how to extract and resolve entities. | `avoid` | List[String] | No | Excluded Biolink categories (must be valid `Categories` enum values). Setting `avoid` makes the column an allow-list by complement: categories the `Categories` enum cannot name (fullmap-emitted Biolink mixins; see `biolink.CATEGORY_OVERRIDES`) are dropped as well, so the guard has no silent hole | | `exclude_prefixes` | List[String] | No | CURIE namespace prefixes excluded during entity resolution: the prefix is the text before the first `:` of a resolved CURIE, and every candidate whose prefix is listed is dropped. Matching is exact and case-sensitive (`OMIM` drops `OMIM:100100` but not `OMIMPS:100` or `omim:100100`). Optional; defaults to null (no prefix filtering). See [Resolution Filters](#resolution-filters). | | `exclude_regex` | List[String] | No | Case-sensitive regex patterns; any resolved CURIE matching one is dropped during entity resolution. Polars-compatible patterns only (no backreferences or lookarounds); an empty or whitespace-only pattern is rejected at config time (`regex-bad-pattern`) because an empty pattern would match every CURIE and silently drop all candidates. Optional; defaults to null (no regex filtering). See [Resolution Filters](#resolution-filters). | +| `identified` | IdentifiedSpec | No | Passthrough mode for pre-identified CURIE columns (`{name: , category: }`); skips fullmap resolution and is mutually exclusive with resolution options (`taxon`, `taxon_from`, `prioritize`, `avoid`, `exclude_prefixes`, `exclude_regex`). | | `regex` | List[Regex] | No | Pattern replacements | | `fill` | String | No | Null-filling strategy: `"forward"`, `"backward"`, `"min"`, `"max"`, `"mean"`, `"zero"`, `"one"` | | `remove` | List[Int\|Float\|String] | No | Regex patterns to remove (replaced with empty string) | @@ -605,6 +606,13 @@ Available functions: `copysign`, `pow` Use the `"values"` token to reference column values in transformations. +### IdentifiedSpec + +| Field | Type | Required | Description | +| ------- | ------ | ---------- | ------------- | +| `name` | Encoding | Yes | Encoding supplying the node's human-readable name. | +| `category` | Encoding | Yes | Encoding supplying the node's Biolink category (e.g. `Gene` or `biolink:Gene`). | + ### Qualifiers Add context to edges (anatomical location, disease context, etc.). diff --git a/src/tablassert/cli.py b/src/tablassert/cli.py index 2fc798b2..8a3687b4 100644 --- a/src/tablassert/cli.py +++ b/src/tablassert/cli.py @@ -403,7 +403,16 @@ def build_graph_pipeline( raise SectionValidationError(configuration_file, h, flatten_pydantic_error(e)) from e advance() - db: Path = fullmap_db_path(g.fullmap) + db: Path | None = None + if g.fullmap is not None: + db = fullmap_db_path(g.fullmap) + elif any(x.requires_fullmap() for x in tcode): + from tablassert.errors import TablassertError + + raise TablassertError( + "fullmap is required because at least one table section uses entity resolution; omit `fullmap` only when every subject/object uses `identified` passthrough.", + code="fullmap-required", + ) # Stage 4/6: collect instructions. progress.stage("Collecting Instructions") @@ -605,15 +614,24 @@ def validate_graph_pipeline(configuration_file: Path, progress: PipelineProgress table_sections.append((table, section)) n: int = len(table_sections) start, advance, _ = progress.section_loop(n, "Validate") + needs_fullmap: bool = False for table, s in table_sections: h: str = mkhash(s) start(f"{Path(str(s['config'])).stem} ยท {h[:8]}") try: - Tcode.model_validate({**s, "store": (STORE / f"{h}.parquet")}) + tc: Tcode = Tcode.model_validate({**s, "store": (STORE / f"{h}.parquet")}) except pydantic.ValidationError as e: raise SectionValidationError(table, h, flatten_pydantic_error(e)) from e + if tc.requires_fullmap(): + needs_fullmap = True advance() + if g.fullmap is None and needs_fullmap: + raise GraphValidationError( + configuration_file, + "fullmap is required because at least one table section uses entity resolution; omit `fullmap` only when every subject/object uses `identified` passthrough.", + ) + logger.info("Validated graph {name}: {n} sections across {tables} tables", name=g.name, n=n, tables=len(g.tables)) diff --git a/src/tablassert/errors.py b/src/tablassert/errors.py index 978659dd..4ddad97d 100644 --- a/src/tablassert/errors.py +++ b/src/tablassert/errors.py @@ -87,6 +87,11 @@ def redact_secrets(text: str, secrets: Sequence[str] = ()) -> str: "qualifier-nullable-literal", "qualifier-duplicated", "qualifier-vocabulary-violation", + "qualifier-identified-unsupported", + "identified-resolution-conflict", + "identified-bad-category", + "identified-bad-curie", + "fullmap-required", "rig-bad-infores", "rig-bad-artifact-url", "rig-bad-access-location", diff --git a/src/tablassert/lib.py b/src/tablassert/lib.py index 0910a27a..c7e682b9 100644 --- a/src/tablassert/lib.py +++ b/src/tablassert/lib.py @@ -1597,32 +1597,28 @@ def _source_ops(self: Self) -> list[Any]: (drop_low_number_of_cases, ()) if self.release and self.statement.predicate == "applied_to_treat" else None, ] - def _node_ops(self: Self, db: Path) -> list[Any]: + def _node_ops(self: Self, db: Path | None) -> list[Any]: """Collect the per-node encoding/resolve/QC ops shared across subject/object/qualifiers. Args: - db: Path to the fullmap redb used for entity resolution. + db: Path to the fullmap redb used for entity resolution, or ``None`` when + every node column in this section is passthrough (``identified``). Returns: - Raw op list: one ``node_prep`` block per node column, ``trim`` to drop - the spent raw ``column_`` columns, the single shared - ``resolve_batch`` op, then per-column ``fullmap_audit`` ops when QC is - enabled. + Raw op list: ``node_prep`` / passthrough encoding per node column, ``trim`` to + drop spent raw ``column_`` columns, ``resolve_batch`` ops for resolved + columns only, then per-column QC ops when QC is enabled. """ - # Subject/object/qualifiers share one resolve_batch call instead of one per column. - # Enum-ranged qualifiers are excluded from resolution: their range is a closed - # Biolink vocabulary, so sending them through the fullmap would turn the required - # token `increased` into the CURIE `UMLS:C0205217`, which the slot rejects. qualifiers: list[Qualifier] = self.statement.qualifiers or [] node_columns: list[tuple[NodeEncoding, str]] = [ (self.statement.subject, "subject"), (self.statement.object, "object"), *[(x, x.qualifier) for x in qualifiers if x.resolved], ] + resolved_columns: list[tuple[NodeEncoding, str]] = [(x, col) for x, col in node_columns if x.identified is None] + passthrough_columns: list[tuple[NodeEncoding, str]] = [(x, col) for x, col in node_columns if x.identified is not None] literals: list[Qualifier] = [x for x in qualifiers if not x.resolved] - # A nullable qualifier keeps its edge when the cell is blank or unresolvable (the - # column stays null and the null-stripper omits the key); subject/object and - # strict qualifiers drop the row as before. + specs: list[ResolveSpec] = [ ResolveSpec( col, @@ -1634,13 +1630,14 @@ def _node_ops(self: Self, db: Path) -> list[Any]: x.nullable if isinstance(x, Qualifier) else False, x.taxon_from, ) - for x, col in node_columns + for x, col in resolved_columns ] - # Per-row taxon scoping (ResolveSpec.taxon_from): a scoped column reads the - # taxon metadata its source column's resolution emits, so sources must resolve - # first. Layer the specs by dependency (config validation has already rejected - # cycles and unknown targets); with no taxon_from anywhere this is one batch - # and the op list is exactly what it always was. + if specs and db is None: + cols_list: str = ", ".join(s.col for s in specs) + raise TablassertError( + f"fullmap is required to resolve node column(s): {cols_list}. Set `fullmap` on the graph or use `identified` passthrough.", + code="fullmap-required", + ) spec_cols: set[str] = {spec.col for spec in specs} batches: list[list[ResolveSpec]] = [] placed: set[str] = set() @@ -1658,21 +1655,22 @@ def _node_ops(self: Self, db: Path) -> list[Any]: placed.update(s.col for s in ready) pending = [s for s in pending if s.col not in placed] resolve_ops: list[Any] = [(resolve_batch, (batch, db, self.log, self.store.stem, self.config.name, True, "_two")) for batch in batches] - return [ - [self.node_prep(x, col) for x, col in node_columns], - # Encode only: no pre-resolution snapshot and no NLP normalization, both of - # which exist to feed entity resolution these columns never undergo. Each enum- - # ranged column ends with a vocabulary audit: nothing else validates its cell - # values against the closed Biolink range before KGX emission. - [[*self.encoding(x, x.qualifier), (qualifier_vocabulary_audit, (str(x.qualifier), sorted(x.vocabulary or ())))] for x in literals], - # Raw ``column_`` columns are dead weight once the encodings above have - # copied them into named slots; trim before resolution so the frame that - # resolve_batch materializes and joins stays narrow. - (trim, ()), - *resolve_ops, - # QC audits only the strict columns: a nullable qualifier's nulls are expected - # (blank cell / no match), not resolution errors for the audit to delete. - [ + + passthrough_ops: list[list[Any]] = [] + for x, col in passthrough_columns: + assert x.identified is not None + passthrough_ops.append( + [ + *self.encoding(x, col, table_literal=True), + *self.encoding(x.identified.name, f"{col}_name"), + *self.encoding(x.identified.category, f"{col}_category"), + (passthrough_audit, (col,)), + ] + ) + + qc_ops: list[Any] | None = None + if self.qc: + qc_ops = [ ( fullmap_audit, ( @@ -1697,11 +1695,17 @@ def _node_ops(self: Self, db: Path) -> list[Any]: ) ), ) - for x, col in node_columns + for x, col in resolved_columns if not (isinstance(x, Qualifier) and x.nullable) - ] - if self.qc - else None, + ] + [(passthrough_qc_audit, (col, self.store.stem, self.config.name, self.log)) for _, col in passthrough_columns] + + return [ + [self.node_prep(x, col) for x, col in resolved_columns], + passthrough_ops, + [[*self.encoding(x, x.qualifier), (qualifier_vocabulary_audit, (str(x.qualifier), sorted(x.vocabulary or ())))] for x in literals], + (trim, ()), + *resolve_ops, + qc_ops, ] def _provenance_ops(self: Self) -> list[Any]: @@ -1790,7 +1794,16 @@ def _provenance_ops(self: Self) -> list[Any]: (to_store, (self.store, self.config.name)), ] - def collect(self: Self, db: Path) -> list[tuple[Callable, tuple[Any]]] | Path: + def requires_fullmap(self: Self) -> bool: + """Return whether any node column in this section requires fullmap entity resolution.""" + qualifiers: list[Qualifier] = self.statement.qualifiers or [] + return ( + self.statement.subject.identified is None + or self.statement.object.identified is None + or any(q.resolved and q.identified is None for q in qualifiers) + ) + + def collect(self: Self, db: Path | None) -> list[tuple[Callable, tuple[Any]]] | Path: """Build the ordered operation list that drives section transformation. Args: @@ -1810,6 +1823,51 @@ def collect(self: Self, db: Path) -> list[tuple[Callable, tuple[Any]]] | Path: return self.clean(tcode) +def passthrough_audit(lf: pl.LazyFrame, col: str) -> pl.LazyFrame: + """Validate pre-identified CURIE, name, and category columns without fullmap lookup.""" + from tablassert.biolink import Categories + from tablassert.rig import curie_prefix + + name_col: str = f"{col}_name" + cat_col: str = f"{col}_category" + filtered: pl.LazyFrame = lf.filter( + pl.col(col).is_not_null() + & (pl.col(col).cast(pl.String).str.strip_chars() != "") + & pl.col(name_col).is_not_null() + & (pl.col(name_col).cast(pl.String).str.strip_chars() != "") + & pl.col(cat_col).is_not_null() + & (pl.col(cat_col).cast(pl.String).str.strip_chars() != "") + ) + distinct_pairs: pl.DataFrame = filtered.select(col, cat_col).unique().collect() + for val in distinct_pairs.get_column(col).to_list(): + text: str = str(val).strip() + if curie_prefix(text) is None: + raise TablassertError( + f"Passthrough `{col}` value {val!r} is not a valid CURIE (expected `PREFIX:LOCAL_ID`).", code="identified-bad-curie" + ) + for cat_val in distinct_pairs.get_column(cat_col).to_list(): + raw: str = str(cat_val).strip().removeprefix("biolink:") + try: + Categories(raw) + except ValueError as e: + raise TablassertError(f"Passthrough `{col}` category {cat_val!r} is not a valid Biolink category.", code="identified-bad-category") from e + return filtered + + +def passthrough_qc_audit(lf: pl.LazyFrame, col: str, section_hash: str, config_file: str, log: bool = True) -> pl.LazyFrame: + """Record passthrough status in QC logs for pre-identified columns.""" + if log: + from tablassert.log import cat + + cat("QC").info( + "QC passthrough ({section_hash} {config_file}): column {col!r} skipped fullmap resolution (pre-identified CURIE)", + section_hash=section_hash, + config_file=config_file, + col=col, + ) + return lf + + PHASE_OF: dict[Callable, str] = { # Maps each Tcode op callable to a short lowercase phase label for progress UX. csv: "load", @@ -1828,7 +1886,9 @@ def collect(self: Self, db: Path) -> list[tuple[Callable, tuple[Any]]] | Path: level_two: "resolve", resolve: "resolve", resolve_batch: "resolve", + passthrough_audit: "resolve", fullmap_audit: "qc", + passthrough_qc_audit: "qc", column: "encode", qualifier_vocabulary_audit: "encode", edge_category: "edge", @@ -2247,8 +2307,8 @@ def _collect_subframes( # Phase: normalize. Only subject and object become nodes; qualifier columns stay as edge attributes. if on_phase is not None: on_phase("normalize") - originals: list[str] = [c.removesuffix("_pre_resolution") for c in lf.collect_schema().names() if c.endswith("_pre_resolution")] - node_cols: list[str] = [c for c in originals if c in ("subject", "object")] + schema_names: list[str] = lf.collect_schema().names() + node_cols: list[str] = [c for c in ("subject", "object") if f"{c}_pre_resolution" in schema_names or f"{c}_name" in schema_names] for col in node_cols: partial, lf = normalize(lf, col, infores_id=infores_id) subnodes.append(partial) diff --git a/src/tablassert/models.py b/src/tablassert/models.py index f0aecc03..f5011a4b 100644 --- a/src/tablassert/models.py +++ b/src/tablassert/models.py @@ -408,6 +408,25 @@ def polars_compatible_replacement(cls, remove: list[int | float | str] | None) - ) +class IdentifiedSpec(TablaBase): + """Passthrough metadata for a pre-identified CURIE column (skips fullmap resolution).""" + + name: Encoding = Field(..., description="Encoding supplying the node's human-readable name.") + category: Encoding = Field(..., description="Encoding supplying the node's Biolink category (e.g. `Gene` or `biolink:Gene`).") + + @model_validator(mode="after") + def validate_literal_category(self: Self) -> Self: + if self.category.method == EncodingMethods.VALUE: + raw: str = str(self.category.encoding).strip().removeprefix("biolink:") + try: + Categories(raw) + except ValueError as e: + raise TablassertValidationError( + f"`identified.category` literal must be a valid Biolink category, got {self.category.encoding!r}.", code="identified-bad-category" + ) from e + return self + + class NodeEncoding(Encoding): taxon: PositiveInt | None = Field( 9606, @@ -461,6 +480,24 @@ def taxon_from_excludes_explicit_taxon(cls, data: Any) -> Any: description="Regex patterns; any resolved CURIE matching one is dropped during entity resolution (case-sensitive).", examples=[["^OMIM:\\d+$"]], ) + identified: IdentifiedSpec | None = Field( + None, description="Passthrough mode for pre-identified CURIE columns; skips fullmap resolution when `name` and `category` are supplied." + ) + + @model_validator(mode="before") + @classmethod + def identified_excludes_resolution_options(cls, data: Any) -> Any: + """Reject `identified` combined with fullmap resolution options.""" + if not isinstance(data, dict) or data.get("identified") is None: + return data + resolution_keys: tuple[str, ...] = ("taxon", "taxon_from", "prioritize", "avoid", "exclude_prefixes", "exclude_regex") + present: list[str] = [k for k in resolution_keys if k in data and (k == "taxon" or data.get(k) is not None)] + if present: + raise TablassertValidationError( + f"`identified` passthrough is mutually exclusive with fullmap resolution options ({', '.join(present)}); remove them.", + code="identified-resolution-conflict", + ) + return data @field_validator("exclude_regex", mode="after") @classmethod @@ -587,6 +624,15 @@ def reject_nullable_literal_qualifiers(self: Self) -> Self: ) return self + @model_validator(mode="after") + def reject_identified_on_qualifier(self: Self) -> Self: + """Reject `identified` on qualifiers; passthrough applies only to subject/object nodes.""" + if self.identified is not None: + raise TablassertValidationError( + "`identified` only applies to `subject` and `object`, not qualifiers.", code="qualifier-identified-unsupported" + ) + return self + class Statement(TablaBase): subject: NodeEncoding = Field(..., description="Subject node encoding and mapping configuration.") @@ -1324,7 +1370,11 @@ class Graph(TablaBase): name: str = Field(..., description="Graph name written into output metadata.") version: str = Field(..., description="Graph version label.") tables: list[Path] = Field(..., description="Paths to table YAML files included in this graph.", examples=[["tables/tutorial-table.yaml"]]) - fullmap: Path = Field(..., description="Base fullmap directory or fullmap redb file for entity resolution.", examples=[".fullmap"]) + fullmap: Path | None = Field( + None, + description="Base fullmap directory or fullmap redb file for entity resolution. Optional when every subject/object uses `identified` passthrough.", + examples=[".fullmap"], + ) rig: RIGConfig = Field(..., description="Resource Ingest Guide metadata emitted as _.RIG.yaml.") uuid_fields: list[str] | None = Field( default=None, diff --git a/src/tablassert/runcache.py b/src/tablassert/runcache.py index 149d4e30..41d8f210 100644 --- a/src/tablassert/runcache.py +++ b/src/tablassert/runcache.py @@ -296,8 +296,10 @@ def prefix_digest(ops: list[tuple[Callable, tuple[Any, ...]]]) -> str: # was not itemized in the cost spec; it is costed here to keep the table exhaustive.) lib.resolve: 60, lib.resolve_batch: 60, + lib.passthrough_audit: 1, # QC: fuzzy match plus a SapBERT (sentence-transformers) cascade when enabled. lib.fullmap_audit: 30, + lib.passthrough_qc_audit: 1, # Edge/provenance: single lazy struct/column emissions. lib.edge_category: 1, lib.publications: 1, diff --git a/tests/test_curie_passthrough.py b/tests/test_curie_passthrough.py new file mode 100644 index 00000000..e500d453 --- /dev/null +++ b/tests/test_curie_passthrough.py @@ -0,0 +1,241 @@ +from __future__ import annotations + +import json +from pathlib import Path + +import polars as pl +import pydantic +import pytest + +from tablassert.cli import APP +from tablassert.errors import GraphValidationError, TablassertError +from tablassert.lib import Tcode, passthrough_audit +from tablassert.models import Graph, NodeEncoding, Qualifier + + +def _rig_block(out_dir: Path) -> str: + return f""" source_info: + infores_id: infores:passthrough-test + terms_of_use_info: + terms_of_use_description: CC0 + data_access_locations: + - https://example.org/data.tsv + source_status: unknown + ingest_info: + utility: Test passthrough build + scope: Minimal fixture + provenance_info: + contributions: + - Test Author - author + artifact_base_url: https://example.org/out + artifact_base_path: {out_dir} +""" + + +def test_identified_rejects_resolution_options_and_bad_literal_category() -> None: + with pytest.raises(pydantic.ValidationError, match="identified-resolution-conflict"): + NodeEncoding.model_validate( + { + "method": "column", + "encoding": "A", + "taxon": 9606, + "identified": {"name": {"method": "column", "encoding": "B"}, "category": {"method": "value", "encoding": "Gene"}}, + } + ) + + with pytest.raises(pydantic.ValidationError, match="identified-bad-category"): + NodeEncoding.model_validate( + { + "method": "column", + "encoding": "A", + "identified": {"name": {"method": "column", "encoding": "B"}, "category": {"method": "value", "encoding": "NotACategory"}}, + } + ) + + with pytest.raises(pydantic.ValidationError, match="qualifier-identified-unsupported"): + Qualifier.model_validate( + { + "qualifier": "anatomical_context_qualifier", + "method": "column", + "encoding": "A", + "identified": {"name": {"method": "column", "encoding": "B"}, "category": {"method": "value", "encoding": "AnatomicalEntity"}}, + } + ) + + +def test_passthrough_audit_validates_curie_and_category() -> None: + bad_curie = pl.LazyFrame({"subject": ["not-a-curie"], "subject_name": ["TP53"], "subject_category": ["Gene"]}) + with pytest.raises(TablassertError, match="identified-bad-curie"): + passthrough_audit(bad_curie, "subject").collect() + + bad_cat = pl.LazyFrame({"subject": ["HGNC:11998"], "subject_name": ["TP53"], "subject_category": ["FakeCategory"]}) + with pytest.raises(TablassertError, match="identified-bad-category"): + passthrough_audit(bad_cat, "subject").collect() + + +def test_full_passthrough_builds_without_fullmap(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.chdir(tmp_path) + (tmp_path / ".tablassert" / "store").mkdir(parents=True, exist_ok=True) + data = tmp_path / "data.tsv" + data.write_text("HGNC:11998\tTP53\tGene\tMONDO:0007254\tbreast cancer\tDisease\n", encoding="utf-8") + + table = tmp_path / "table.yaml" + table.write_text( + f"""template: + source: + kind: text + local: {data} + delimiter: "\\t" + url: + - https://example.org/data.tsv + statement: + subject: + method: column + encoding: A + identified: + name: + method: column + encoding: B + category: + method: column + encoding: C + predicate: associated_with + object: + method: column + encoding: D + identified: + name: + method: column + encoding: E + category: + method: value + encoding: Disease + provenance: + repo: PMID + publication: "12345678" +sections: + - {{}} +""", + encoding="utf-8", + ) + + out_dir = tmp_path / "out" + out_dir.mkdir() + graph_file = tmp_path / "graph.yaml" + graph_file.write_text( + f"""name: passthrough-kg +version: 0.1.0 +tables: + - {table} +rig: +{_rig_block(out_dir)}""", + encoding="utf-8", + ) + + Graph.model_validate( + { + "name": "passthrough-kg", + "version": "0.1.0", + "tables": [table], + "rig": { + "source_info": { + "infores_id": "infores:passthrough-test", + "terms_of_use_info": {"terms_of_use_description": "CC0"}, + "data_access_locations": ["https://example.org/data.tsv"], + "source_status": "unknown", + }, + "ingest_info": {"utility": "Test", "scope": "Test"}, + "provenance_info": {"contributions": ["Test - author"]}, + "artifact_base_url": "https://example.org/out", + "artifact_base_path": out_dir, + }, + } + ) + + APP(["validate", str(graph_file), "--schema", "graph"], exit_on_error=False, result_action="return_value") + APP(["build-kg", str(graph_file)], exit_on_error=False, result_action="return_value") + + nodes_file = out_dir / "passthrough-kg_0.1.0.nodes.ndjson" + edges_file = out_dir / "passthrough-kg_0.1.0.edges.ndjson" + assert nodes_file.is_file() + assert edges_file.is_file() + + nodes = {row["id"]: row for row in (json.loads(line) for line in nodes_file.read_text(encoding="utf-8").splitlines() if line.strip())} + assert nodes["HGNC:11998"]["name"] == "TP53" + assert nodes["HGNC:11998"]["category"] == ["biolink:Gene"] + assert nodes["MONDO:0007254"]["name"] == "breast cancer" + assert nodes["MONDO:0007254"]["category"] == ["biolink:Disease"] + + +def test_mixed_section_without_fullmap_is_rejected(tmp_path: Path) -> None: + table = tmp_path / "table.yaml" + table.write_text( + """template: + source: + kind: text + local: data.tsv + delimiter: "\\t" + url: + - https://example.org/data.tsv + statement: + subject: + method: column + encoding: A + identified: + name: + method: column + encoding: B + category: + method: value + encoding: Gene + predicate: associated_with + object: + method: column + encoding: C + provenance: + repo: PMID + publication: "12345678" +sections: + - {} +""", + encoding="utf-8", + ) + + out_dir = tmp_path / "out" + out_dir.mkdir() + graph_file = tmp_path / "graph.yaml" + graph_file.write_text( + f"""name: mixed-kg +version: 0.1.0 +tables: + - {table} +rig: +{_rig_block(out_dir)}""", + encoding="utf-8", + ) + + with pytest.raises(GraphValidationError, match="fullmap is required"): + APP(["validate", str(graph_file), "--schema", "graph"], exit_on_error=False, result_action="return_value") + + +def test_tcode_requires_fullmap_reflects_mixed_vs_full_passthrough(tmp_path: Path) -> None: + mixed = Tcode.model_validate( + { + "config": tmp_path / "t.yaml", + "store": tmp_path / "s.parquet", + "source": {"kind": "text", "local": "d.tsv", "url": ["https://example.org/d.tsv"]}, + "statement": { + "subject": { + "method": "column", + "encoding": "A", + "identified": {"name": {"method": "column", "encoding": "B"}, "category": {"method": "value", "encoding": "Gene"}}, + }, + "predicate": "associated_with", + "object": {"method": "column", "encoding": "C"}, + }, + "provenance": {"repo": "PMID", "publication": "12345678"}, + } + ) + assert mixed.requires_fullmap() is True + with pytest.raises(TablassertError, match="fullmap-required"): + mixed.collect(None) diff --git a/tests/test_docs_source_of_truth.py b/tests/test_docs_source_of_truth.py index 792472ee..4364d28c 100644 --- a/tests/test_docs_source_of_truth.py +++ b/tests/test_docs_source_of_truth.py @@ -78,6 +78,7 @@ Encoding, Excel, Graph, + IdentifiedSpec, ManualProvenance, NodeEncoding, Parquet, @@ -113,7 +114,7 @@ # Keys whose presence makes a parsed block a COMPLETE config for one live schema. Mirrors the # required fields of `models.Section` (source/statement/provenance) and `models.Graph`. TABLE_KEYS: frozenset[str] = frozenset({"provenance", "source", "statement"}) -GRAPH_KEYS: frozenset[str] = frozenset({"fullmap", "name", "rig", "tables", "version"}) +GRAPH_KEYS: frozenset[str] = frozenset({"name", "rig", "tables", "version"}) # Elision markers. A block carrying one is an excerpt by construction and cannot validate. PLACEHOLDER_TOKENS: tuple[str, ...] = ("{...}", "...") @@ -592,6 +593,7 @@ def location(self) -> str: ModelTableBinding(Reindex, TABLE_CONFIGURATION, ("Reindexing (Conditional Filtering)",)), ModelTableBinding(Statement, TABLE_CONFIGURATION, ("Statement (Triple Definition)",)), ModelTableBinding(NodeEncoding, TABLE_CONFIGURATION, ("NodeEncoding",)), + ModelTableBinding(IdentifiedSpec, TABLE_CONFIGURATION, ("IdentifiedSpec",)), ModelTableBinding(Qualifier, TABLE_CONFIGURATION, ("Qualifiers",)), ModelTableBinding(Provenance, TABLE_CONFIGURATION, ("Provenance",)), ModelTableBinding(ManualProvenance, TABLE_CONFIGURATION, ("Manual provenance override",)),