Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ All notable changes to this project are documented in this file.

### Added

- **`identified` passthrough skips fullmap resolution for pre-identified CURIE columns.** `subject` and `object` encodings can declare `identified: {name: <Encoding>, category: <Encoding>}` to emit CURIEs, names, and Biolink categories directly without touching a fullmap database (mutually exclusive with resolution options such as `taxon`, `prioritize`, `avoid`, `exclude_prefixes`, and `exclude_regex`). When every node column in a graph uses `identified`, `Graph.fullmap` may be omitted. ([#272](https://github.com/SkyeAv/Tablassert/issues/272))
- **NDJSON (JSONL) is a fifth table source kind.** `kind: ndjson` scans newline-delimited JSON lazily with `pl.scan_ndjson` and renames its columns positionally to `column_1..` in first-seen key order, matching the `parquet` reader contract so `method: column` encodings keep using Excel-style letters. Ragged records scan missing keys as nulls instead of dropping rows; an empty file fails fast with a coded `ndjson-empty` error instead of polars' opaque `Cannot infer NDJSON types` compute error. ([#283](https://github.com/SkyeAv/Tablassert/pull/283), [#262](https://github.com/SkyeAv/Tablassert/issues/262))
- **Every pull request must carry a new changelog entry.** A stdlib-only `scripts/check_changelog.py` requires at least one new bullet under the `## Unreleased` heading of `CHANGELOG.md` relative to the pull request's merge base, failing with remediation guidance when whitespace, subheadings, or released-history edits are the only additions; a CI job runs it on every pull request including forks, release-bump and dependency-bump pull requests are exempt by their `chore(release):` / `chore(deps):` title or commit subject, documentation-only pull requests (every changed path is markdown or under `docs/`) are exempt via `--exempt-docs-only`, and a pre-push hook mirrors the same check locally. ([#284](https://github.com/SkyeAv/Tablassert/pull/284), [#282](https://github.com/SkyeAv/Tablassert/issues/282))
- **Arrow IPC (Feather v2) is a fourth table source kind.** `kind: ipc` scans a `.arrow` file lazily with `pl.scan_ipc` and renames its columns positionally to `column_1..` in schema order, matching the `parquet` reader contract so `method: column` encodings keep using Excel-style letters. Only IPC file layout is supported; a stream-layout file fails fast with a coded `ipc-stream-unsupported` error naming the conversion fix instead of polars' opaque `InvalidFooter` compute error. ([#281](https://github.com/SkyeAv/Tablassert/pull/281), [#264](https://github.com/SkyeAv/Tablassert/issues/264))
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration/graph.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@ QC auditing and verbose logging are controlled at build time via the `build-kg -
| `name` | String | Knowledge graph name (used in output filenames and the RIG) |
| `version` | String | Knowledge graph version (used in output filenames) |
| `tables` | List[Path] | Paths to table configuration YAML files |
| `fullmap` | Path | Path to the fullmap redb file, or a base directory containing it |
| `rig` | RIGConfig | Resource Ingest Guide metadata (see below) |

The legacy top-level RIG fields (`description`, `contributions`, `ui_explanation`, `infores`) are **rejected** with a migration pointer; they now live under `rig:`.
Expand All @@ -31,6 +30,7 @@ The legacy top-level RIG fields (`description`, `contributions`, `ui_explanation

| Field | Type | Description |
| ------- | ------ | ------------- |
| `fullmap` | Path | Path to the fullmap redb file, or a base directory containing it. Required whenever any table section uses fullmap resolution; optional when every subject/object uses `identified` passthrough |
| `uuid_fields` | List[str] | Edge fields that constitute edge identity. Only these feed the derived edge `id` (see [Stable edge ids](#stable-edge-ids)) |
| `uuid_domain` | String | Explicit UUID namespace. Defaults to `rig.source_info.infores_id` when `uuid_fields` is set, `TABLASSERT` otherwise |
| `uuid_on_collision` | `error` \| `merge` | What to do when two different edges derive one id. `error` (default) aborts; `merge` folds them into one edge. Requires `uuid_fields` (see [Merging collisions instead](#merging-collisions-instead)) |
Expand Down
8 changes: 8 additions & 0 deletions docs/configuration/table.md
Original file line number Diff line number Diff line change
Expand Up @@ -372,6 +372,7 @@ Defines how to extract and resolve entities.
| `avoid` | List[String] | No | Excluded Biolink categories (must be valid `Categories` enum values). Setting `avoid` makes the column an allow-list by complement: categories the `Categories` enum cannot name (fullmap-emitted Biolink mixins; see `biolink.CATEGORY_OVERRIDES`) are dropped as well, so the guard has no silent hole |
| `exclude_prefixes` | List[String] | No | CURIE namespace prefixes excluded during entity resolution: the prefix is the text before the first `:` of a resolved CURIE, and every candidate whose prefix is listed is dropped. Matching is exact and case-sensitive (`OMIM` drops `OMIM:100100` but not `OMIMPS:100` or `omim:100100`). Optional; defaults to null (no prefix filtering). See [Resolution Filters](#resolution-filters). |
| `exclude_regex` | List[String] | No | Case-sensitive regex patterns; any resolved CURIE matching one is dropped during entity resolution. Polars-compatible patterns only (no backreferences or lookarounds); an empty or whitespace-only pattern is rejected at config time (`regex-bad-pattern`) because an empty pattern would match every CURIE and silently drop all candidates. Optional; defaults to null (no regex filtering). See [Resolution Filters](#resolution-filters). |
| `identified` | IdentifiedSpec | No | Passthrough mode for pre-identified CURIE columns (`{name: <Encoding>, category: <Encoding>}`); skips fullmap resolution and is mutually exclusive with resolution options (`taxon`, `taxon_from`, `prioritize`, `avoid`, `exclude_prefixes`, `exclude_regex`). |
| `regex` | List[Regex] | No | Pattern replacements |
| `fill` | String | No | Null-filling strategy: `"forward"`, `"backward"`, `"min"`, `"max"`, `"mean"`, `"zero"`, `"one"` |
| `remove` | List[Int\|Float\|String] | No | Regex patterns to remove (replaced with empty string) |
Expand Down Expand Up @@ -605,6 +606,13 @@ Available functions: `copysign`, `pow`

Use the `"values"` token to reference column values in transformations.

### IdentifiedSpec

| Field | Type | Required | Description |
| ------- | ------ | ---------- | ------------- |
| `name` | Encoding | Yes | Encoding supplying the node's human-readable name. |
| `category` | Encoding | Yes | Encoding supplying the node's Biolink category (e.g. `Gene` or `biolink:Gene`). |

### Qualifiers

Add context to edges (anatomical location, disease context, etc.).
Expand Down
22 changes: 20 additions & 2 deletions src/tablassert/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -403,7 +403,16 @@ def build_graph_pipeline(
raise SectionValidationError(configuration_file, h, flatten_pydantic_error(e)) from e
advance()

db: Path = fullmap_db_path(g.fullmap)
db: Path | None = None
if g.fullmap is not None:
db = fullmap_db_path(g.fullmap)
elif any(x.requires_fullmap() for x in tcode):
from tablassert.errors import TablassertError

raise TablassertError(
"fullmap is required because at least one table section uses entity resolution; omit `fullmap` only when every subject/object uses `identified` passthrough.",
code="fullmap-required",
)

# Stage 4/6: collect instructions.
progress.stage("Collecting Instructions")
Expand Down Expand Up @@ -605,15 +614,24 @@ def validate_graph_pipeline(configuration_file: Path, progress: PipelineProgress
table_sections.append((table, section))
n: int = len(table_sections)
start, advance, _ = progress.section_loop(n, "Validate")
needs_fullmap: bool = False
for table, s in table_sections:
h: str = mkhash(s)
start(f"{Path(str(s['config'])).stem} · {h[:8]}")
try:
Tcode.model_validate({**s, "store": (STORE / f"{h}.parquet")})
tc: Tcode = Tcode.model_validate({**s, "store": (STORE / f"{h}.parquet")})
except pydantic.ValidationError as e:
raise SectionValidationError(table, h, flatten_pydantic_error(e)) from e
if tc.requires_fullmap():
needs_fullmap = True
advance()

if g.fullmap is None and needs_fullmap:
raise GraphValidationError(
configuration_file,
"fullmap is required because at least one table section uses entity resolution; omit `fullmap` only when every subject/object uses `identified` passthrough.",
)

logger.info("Validated graph {name}: {n} sections across {tables} tables", name=g.name, n=n, tables=len(g.tables))


Expand Down
5 changes: 5 additions & 0 deletions src/tablassert/errors.py
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,11 @@ def redact_secrets(text: str, secrets: Sequence[str] = ()) -> str:
"qualifier-nullable-literal",
"qualifier-duplicated",
"qualifier-vocabulary-violation",
"qualifier-identified-unsupported",
"identified-resolution-conflict",
"identified-bad-category",
"identified-bad-curie",
"fullmap-required",
"rig-bad-infores",
"rig-bad-artifact-url",
"rig-bad-access-location",
Expand Down
142 changes: 101 additions & 41 deletions src/tablassert/lib.py
Original file line number Diff line number Diff line change
Expand Up @@ -1597,32 +1597,28 @@ def _source_ops(self: Self) -> list[Any]:
(drop_low_number_of_cases, ()) if self.release and self.statement.predicate == "applied_to_treat" else None,
]

def _node_ops(self: Self, db: Path) -> list[Any]:
def _node_ops(self: Self, db: Path | None) -> list[Any]:
"""Collect the per-node encoding/resolve/QC ops shared across subject/object/qualifiers.

Args:
db: Path to the fullmap redb used for entity resolution.
db: Path to the fullmap redb used for entity resolution, or ``None`` when
every node column in this section is passthrough (``identified``).

Returns:
Raw op list: one ``node_prep`` block per node column, ``trim`` to drop
the spent raw ``column_<n>`` columns, the single shared
``resolve_batch`` op, then per-column ``fullmap_audit`` ops when QC is
enabled.
Raw op list: ``node_prep`` / passthrough encoding per node column, ``trim`` to
drop spent raw ``column_<n>`` columns, ``resolve_batch`` ops for resolved
columns only, then per-column QC ops when QC is enabled.
"""
# Subject/object/qualifiers share one resolve_batch call instead of one per column.
# Enum-ranged qualifiers are excluded from resolution: their range is a closed
# Biolink vocabulary, so sending them through the fullmap would turn the required
# token `increased` into the CURIE `UMLS:C0205217`, which the slot rejects.
qualifiers: list[Qualifier] = self.statement.qualifiers or []
node_columns: list[tuple[NodeEncoding, str]] = [
(self.statement.subject, "subject"),
(self.statement.object, "object"),
*[(x, x.qualifier) for x in qualifiers if x.resolved],
]
resolved_columns: list[tuple[NodeEncoding, str]] = [(x, col) for x, col in node_columns if x.identified is None]
passthrough_columns: list[tuple[NodeEncoding, str]] = [(x, col) for x, col in node_columns if x.identified is not None]
literals: list[Qualifier] = [x for x in qualifiers if not x.resolved]
# A nullable qualifier keeps its edge when the cell is blank or unresolvable (the
# column stays null and the null-stripper omits the key); subject/object and
# strict qualifiers drop the row as before.

specs: list[ResolveSpec] = [
ResolveSpec(
col,
Expand All @@ -1634,13 +1630,14 @@ def _node_ops(self: Self, db: Path) -> list[Any]:
x.nullable if isinstance(x, Qualifier) else False,
x.taxon_from,
)
for x, col in node_columns
for x, col in resolved_columns
]
# Per-row taxon scoping (ResolveSpec.taxon_from): a scoped column reads the
# taxon metadata its source column's resolution emits, so sources must resolve
# first. Layer the specs by dependency (config validation has already rejected
# cycles and unknown targets); with no taxon_from anywhere this is one batch
# and the op list is exactly what it always was.
if specs and db is None:
cols_list: str = ", ".join(s.col for s in specs)
raise TablassertError(
f"fullmap is required to resolve node column(s): {cols_list}. Set `fullmap` on the graph or use `identified` passthrough.",
code="fullmap-required",
)
spec_cols: set[str] = {spec.col for spec in specs}
batches: list[list[ResolveSpec]] = []
placed: set[str] = set()
Expand All @@ -1658,21 +1655,22 @@ def _node_ops(self: Self, db: Path) -> list[Any]:
placed.update(s.col for s in ready)
pending = [s for s in pending if s.col not in placed]
resolve_ops: list[Any] = [(resolve_batch, (batch, db, self.log, self.store.stem, self.config.name, True, "_two")) for batch in batches]
return [
[self.node_prep(x, col) for x, col in node_columns],
# Encode only: no pre-resolution snapshot and no NLP normalization, both of
# which exist to feed entity resolution these columns never undergo. Each enum-
# ranged column ends with a vocabulary audit: nothing else validates its cell
# values against the closed Biolink range before KGX emission.
[[*self.encoding(x, x.qualifier), (qualifier_vocabulary_audit, (str(x.qualifier), sorted(x.vocabulary or ())))] for x in literals],
# Raw ``column_<n>`` columns are dead weight once the encodings above have
# copied them into named slots; trim before resolution so the frame that
# resolve_batch materializes and joins stays narrow.
(trim, ()),
*resolve_ops,
# QC audits only the strict columns: a nullable qualifier's nulls are expected
# (blank cell / no match), not resolution errors for the audit to delete.
[

passthrough_ops: list[list[Any]] = []
for x, col in passthrough_columns:
assert x.identified is not None
passthrough_ops.append(
[
*self.encoding(x, col, table_literal=True),
*self.encoding(x.identified.name, f"{col}_name"),
*self.encoding(x.identified.category, f"{col}_category"),
(passthrough_audit, (col,)),
]
)

qc_ops: list[Any] | None = None
if self.qc:
qc_ops = [
(
fullmap_audit,
(
Expand All @@ -1697,11 +1695,17 @@ def _node_ops(self: Self, db: Path) -> list[Any]:
)
),
)
for x, col in node_columns
for x, col in resolved_columns
if not (isinstance(x, Qualifier) and x.nullable)
]
if self.qc
else None,
] + [(passthrough_qc_audit, (col, self.store.stem, self.config.name, self.log)) for _, col in passthrough_columns]

return [
[self.node_prep(x, col) for x, col in resolved_columns],
passthrough_ops,
[[*self.encoding(x, x.qualifier), (qualifier_vocabulary_audit, (str(x.qualifier), sorted(x.vocabulary or ())))] for x in literals],
(trim, ()),
*resolve_ops,
qc_ops,
]

def _provenance_ops(self: Self) -> list[Any]:
Expand Down Expand Up @@ -1790,7 +1794,16 @@ def _provenance_ops(self: Self) -> list[Any]:
(to_store, (self.store, self.config.name)),
]

def collect(self: Self, db: Path) -> list[tuple[Callable, tuple[Any]]] | Path:
def requires_fullmap(self: Self) -> bool:
"""Return whether any node column in this section requires fullmap entity resolution."""
qualifiers: list[Qualifier] = self.statement.qualifiers or []
return (
self.statement.subject.identified is None
or self.statement.object.identified is None
or any(q.resolved and q.identified is None for q in qualifiers)
)

def collect(self: Self, db: Path | None) -> list[tuple[Callable, tuple[Any]]] | Path:
"""Build the ordered operation list that drives section transformation.

Args:
Expand All @@ -1810,6 +1823,51 @@ def collect(self: Self, db: Path) -> list[tuple[Callable, tuple[Any]]] | Path:
return self.clean(tcode)


def passthrough_audit(lf: pl.LazyFrame, col: str) -> pl.LazyFrame:
"""Validate pre-identified CURIE, name, and category columns without fullmap lookup."""
from tablassert.biolink import Categories
from tablassert.rig import curie_prefix

name_col: str = f"{col}_name"
cat_col: str = f"{col}_category"
filtered: pl.LazyFrame = lf.filter(
pl.col(col).is_not_null()
& (pl.col(col).cast(pl.String).str.strip_chars() != "")
& pl.col(name_col).is_not_null()
& (pl.col(name_col).cast(pl.String).str.strip_chars() != "")
& pl.col(cat_col).is_not_null()
& (pl.col(cat_col).cast(pl.String).str.strip_chars() != "")
)
distinct_pairs: pl.DataFrame = filtered.select(col, cat_col).unique().collect()
for val in distinct_pairs.get_column(col).to_list():
text: str = str(val).strip()
if curie_prefix(text) is None:
raise TablassertError(
f"Passthrough `{col}` value {val!r} is not a valid CURIE (expected `PREFIX:LOCAL_ID`).", code="identified-bad-curie"
)
for cat_val in distinct_pairs.get_column(cat_col).to_list():
raw: str = str(cat_val).strip().removeprefix("biolink:")
try:
Categories(raw)
except ValueError as e:
raise TablassertError(f"Passthrough `{col}` category {cat_val!r} is not a valid Biolink category.", code="identified-bad-category") from e
return filtered


def passthrough_qc_audit(lf: pl.LazyFrame, col: str, section_hash: str, config_file: str, log: bool = True) -> pl.LazyFrame:
"""Record passthrough status in QC logs for pre-identified columns."""
if log:
from tablassert.log import cat

cat("QC").info(
"QC passthrough ({section_hash} {config_file}): column {col!r} skipped fullmap resolution (pre-identified CURIE)",
section_hash=section_hash,
config_file=config_file,
col=col,
)
return lf


PHASE_OF: dict[Callable, str] = {
# Maps each Tcode op callable to a short lowercase phase label for progress UX.
csv: "load",
Expand All @@ -1828,7 +1886,9 @@ def collect(self: Self, db: Path) -> list[tuple[Callable, tuple[Any]]] | Path:
level_two: "resolve",
resolve: "resolve",
resolve_batch: "resolve",
passthrough_audit: "resolve",
fullmap_audit: "qc",
passthrough_qc_audit: "qc",
column: "encode",
qualifier_vocabulary_audit: "encode",
edge_category: "edge",
Expand Down Expand Up @@ -2247,8 +2307,8 @@ def _collect_subframes(
# Phase: normalize. Only subject and object become nodes; qualifier columns stay as edge attributes.
if on_phase is not None:
on_phase("normalize")
originals: list[str] = [c.removesuffix("_pre_resolution") for c in lf.collect_schema().names() if c.endswith("_pre_resolution")]
node_cols: list[str] = [c for c in originals if c in ("subject", "object")]
schema_names: list[str] = lf.collect_schema().names()
node_cols: list[str] = [c for c in ("subject", "object") if f"{c}_pre_resolution" in schema_names or f"{c}_name" in schema_names]
for col in node_cols:
partial, lf = normalize(lf, col, infores_id=infores_id)
subnodes.append(partial)
Expand Down
Loading