Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ All notable changes to this project are documented in this file.

### Changed

- **The `rig:` graph section is now optional.** Graphs without `rig:` validate, `build-kg` emits KGX only (skipping `compile_rig`) with a build-time warning, and two optional top-level fallback fields, `infores` (defaulting to `infores:tablassert`) and `output_dir` (defaulting to the current directory), supply provenance and output paths when `rig:` is omitted, while being rejected when `rig:` is present so publication metadata remains authoritative. ([#288](https://github.com/SkyeAv/Tablassert/pull/288), [#274](https://github.com/SkyeAv/Tablassert/issues/274))
- **CI runs only when a release is cut or on manual dispatch.** `.github/workflows/ci.yml` no longer triggers on pull requests; it runs on a push to main that changes `project.version` in `pyproject.toml` (the `chore(release):` commit, detected by the same `detect-pyproject-version-change` action the tag and PyPI workflows use) or via `workflow_dispatch`. The PR-only `changes` path filter, the push-back `autofix` job, the `python-smoke` subset, and the CI `changelog` job are removed; formatting is now a check-only `format` job, and the changelog requirement stays enforced by the pre-push hook. Run `make check` locally before opening a pull request.
- **The vulture allowlist lives in `scripts/` next to the other developer tooling.** `vulture_whitelist.py` moves to `scripts/vulture_whitelist.py` unchanged; the `[tool.vulture]` `paths` list drops the now-redundant root entry because `scripts` is already scanned, and the CI `python` path filter, the pre-commit hook comment, and the CONTRIBUTING dead-code section follow the new location. The gate is behaviorally identical: same vulture pin, same config, same allowlist content. ([#287](https://github.com/SkyeAv/Tablassert/pull/287))

Expand Down
14 changes: 8 additions & 6 deletions docs/configuration/graph.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ A graph configuration file specifies:
- Output knowledge graph name and version
- List of table configurations to process
- Database location for entity resolution
- The **required `rig:` section**: all Resource Ingest Guide (RIG) metadata emitted as `<name>_<version>.RIG.yaml`
- The **optional `rig:` section**: all Resource Ingest Guide (RIG) metadata emitted as `<name>_<version>.RIG.yaml`; omit it for drafts and internal pipelines (KGX artifacts still build)

QC auditing and verbose logging are controlled at build time via the `build-kg --qc` and `build-kg --log` flags: they are **not** graph-config fields.

Expand All @@ -23,22 +23,24 @@ QC auditing and verbose logging are controlled at build time via the `build-kg -
| `version` | String | Knowledge graph version (used in output filenames) |
| `tables` | List[Path] | Paths to table configuration YAML files |
| `fullmap` | Path | Path to the fullmap redb file, or a base directory containing it |
| `rig` | RIGConfig | Resource Ingest Guide metadata (see below) |

The legacy top-level RIG fields (`description`, `contributions`, `ui_explanation`, `infores`) are **rejected** with a migration pointer; they now live under `rig:`.

### Optional Fields

| Field | Type | Description |
| ------- | ------ | ------------- |
| `rig` | RIGConfig | Resource Ingest Guide metadata emitted as `<name>_<version>.RIG.yaml` (see below). Omit it for draft/internal builds: KGX artifacts still build, a build-time warning notes the absent publication metadata, and no `.RIG.yaml` is emitted |
| `infores` | String | RIG-less builds only: graph-level primary knowledge source infores recorded as node `provided_by` (must be a valid infores CURIE). Mutually exclusive with `rig:`; defaults to `infores:tablassert` |
| `output_dir` | Path | RIG-less builds only: local output directory for the KGX artifacts (`rig.artifact_base_path` owns it otherwise). Mutually exclusive with `rig:`; defaults to the current directory |
| `uuid_fields` | List[str] | Edge fields that constitute edge identity. Only these feed the derived edge `id` (see [Stable edge ids](#stable-edge-ids)) |
| `uuid_domain` | String | Explicit UUID namespace. Defaults to `rig.source_info.infores_id` when `uuid_fields` is set, `TABLASSERT` otherwise |
| `uuid_domain` | String | Explicit UUID namespace. Defaults to the graph infores (`rig.source_info.infores_id`, else the `infores` fallback) when `uuid_fields` is set, `TABLASSERT` otherwise |
| `uuid_on_collision` | `error` \| `merge` | What to do when two different edges derive one id. `error` (default) aborts; `merge` folds them into one edge. Requires `uuid_fields` (see [Merging collisions instead](#merging-collisions-instead)) |
| `qc_similarity_threshold` | Float 0-1 | SapBERT cosine threshold used by QC (default 0.5) |
| `qc_fuzzy_ratio_threshold` | Float 0-100 | RapidFuzz ratio threshold used by QC (default 70) |
| `qc_fuzzy_partial_threshold` | Float 0-100 | RapidFuzz partial/token-set threshold used by QC (default 80) |
| `qc_aliases` | Mapping[String, String] | Source-text aliases applied before QC similarity stages (default empty) |

The legacy top-level RIG fields (`description`, `contributions`, `ui_explanation`, `infores`) are **rejected when `rig:` is set** with a migration pointer; they live under `rig:`. (`infores` at the top level without `rig:` is the RIG-less fallback field, not a legacy key.)

## Stable edge ids

Every edge gets a deterministic `id`: a UUID v3 derived from the edge itself. By default it is
Expand Down Expand Up @@ -153,7 +155,7 @@ opt-in. The default `error` behavior is unchanged in every respect.
### Namespacing

Because a narrow field set no longer distinguishes graphs by accident, declaring `uuid_fields`
moves the UUID namespace onto the graph's own `rig.source_info.infores_id`. Two graphs asserting
moves the UUID namespace onto the graph's own infores (`rig.source_info.infores_id`, or the `infores` fallback on RIG-less graphs). Two graphs asserting
the same triple from the same publication then still derive different ids, structurally.

Set `uuid_domain` only when graphs must deliberately **share** an id space: a KG compiled in
Expand Down
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ expectations.

### Reference

- **[Configuration](configuration/graph.md)**: graph-level schema, the required `rig:` section, and stable edge ids
- **[Configuration](configuration/graph.md)**: graph-level schema, the optional `rig:` section, and stable edge ids
- **[Table Configuration](configuration/table.md)**: sources, statements, entity-resolution rules, provenance, annotations
- **[Advanced Example](configuration/advanced-example.md)**: a fully-annotated real-world configuration
- **[API Reference](api/fullmap.md)**: `resolve()` and `quick_map()`
Expand Down
2 changes: 1 addition & 1 deletion llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Current CLI behavior queries an embedded redb database whose primary file is `fu

## YAML Authoring
- [Table Configuration Reference](docs/configuration/table.md): the table (`Section`) schema, template/sections patterns, and field semantics.
- [Graph Configuration Reference](docs/configuration/graph.md): the Graph config fields (`name`/`version`/`tables`/`fullmap`/`rig`, plus optional `uuid_fields`/`uuid_domain`/`uuid_on_collision` and QC controls `qc_similarity_threshold`/`qc_fuzzy_ratio_threshold`/`qc_fuzzy_partial_threshold`/`qc_aliases`) and orchestration model.
- [Graph Configuration Reference](docs/configuration/graph.md): the Graph config fields (`name`/`version`/`tables`/`fullmap`, optional `rig` plus RIG-less fallbacks `infores`/`output_dir`, and `uuid_fields`/`uuid_domain`/`uuid_on_collision` with QC controls `qc_similarity_threshold`/`qc_fuzzy_ratio_threshold`/`qc_fuzzy_partial_threshold`/`qc_aliases`) and orchestration model.
- [Use Case Gallery](docs/examples.md): real-world table-configuration patterns for common data types.
- [Tutorial Table Example](docs/examples/tutorial-table.yaml): minimal table configuration.
- [Tutorial Graph Example](docs/examples/tutorial-graph.yaml): minimal graph configuration.
Expand Down
17 changes: 12 additions & 5 deletions src/tablassert/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -391,7 +391,7 @@ def build_graph_pipeline(
"head": head,
"linkouts": linkouts,
"name": g.name,
"infores": g.rig.source_info.infores_id,
"infores": g.infores_id,
"qc_similarity_threshold": g.qc_similarity_threshold,
"qc_fuzzy_ratio_threshold": g.qc_fuzzy_ratio_threshold,
"qc_fuzzy_partial_threshold": g.qc_fuzzy_partial_threshold,
Expand Down Expand Up @@ -466,6 +466,13 @@ def build_graph_pipeline(
progress.stage("Compiling Graph")
start, advance, sub_step = progress.section_loop(len(subgraphs), "Graph")
start(f"{g.name} · v{g.version}")
if g.rig is None:
logger.warning(
"Graph {name} v{version} has no `rig:` section; emitting KGX artifacts only (provided_by falls back to {infores}). Add `rig:` when the graph is headed for Translator publication.",
name=g.name,
version=g.version,
infores=g.infores_id,
)
# on_phase drives the phase tag (scan → normalize → write-nodes → write-edges → dedup → rig);
# on_subgraph ticks the bar once per subgraph, so the total is len(subgraphs).
compile_graph(
Expand All @@ -479,6 +486,8 @@ def build_graph_pipeline(
uuid_fields=g.uuid_fields,
uuid_domain=g.uuid_namespace,
uuid_on_collision=g.uuid_on_collision,
output_dir=g.output_dir,
infores=g.infores if g.rig is None else None,
)

# Stage 7/7 (only with --qc): assert over the final NDJSON files.
Expand All @@ -490,9 +499,7 @@ def build_graph_pipeline(
# Association is the author's accepted default. Graphs where every section is pinned
# are unaffected; a mixed graph should split or accept the stricter gate. An empty
# ``category_override: {}`` pins nothing and counts as undeclared.
study_final_ndjson(
g.name, g.version, Path(g.rig.artifact_base_path), category_override_declared=any(x.statement.category_override for x in tcode)
)
study_final_ndjson(g.name, g.version, g.artifact_path, category_override_declared=any(x.statement.category_override for x in tcode))

logger.info("Built graph {name} v{version}: {n} sections", name=g.name, version=g.version, n=n)

Expand Down Expand Up @@ -953,7 +960,7 @@ def build_kg(
cached under its own ``.head`` store, so it never reuses or replaces a full build.
linkouts: Append the EdgeLinkouts per-edge record URL
(``https://linkouts.skyelanegoetz.com/edges/{edge_id}``, resolved to the final
edge id after dedup) to the graph infores entry (``rig.source_info.infores_id``)
edge id after dedup) to the graph infores entry (``rig.source_info.infores_id``, or the ``infores`` fallback on RIG-less graphs)
in every emitted edge's ``sources`` list, so linkouts.skyelanegoetz.com can
deep-link each edge. Sections declaring ``provenance.override`` are untouched
(their ``sources`` emission is author-controlled). Cached under its own
Expand Down
41 changes: 27 additions & 14 deletions src/tablassert/lib.py
Original file line number Diff line number Diff line change
Expand Up @@ -2268,7 +2268,7 @@ def _write_ndjson(
edges_tmp: Path,
name: str,
version: str,
rig: RIGConfig,
rig: RIGConfig | None,
section_sources: list[dict[str, object]] | None,
on_phase: Callable[[str], None] | None = None,
domain: str = "TABLASSERT",
Expand All @@ -2294,7 +2294,8 @@ def _write_ndjson(
edges_tmp: Working ``.edges.ndjson.tmp`` output path.
name: Graph name (output stems and RIG).
version: Graph version string.
rig: Validated ``rig:`` graph-config section driving the RIG.
rig: Optional validated ``rig:`` graph-config section driving the RIG;
``None`` (RIG-less build) skips the ``rig`` phase.
section_sources: Per-section source descriptors for the RIG
relevant-file cross-check.
on_phase: Optional callback fired with ``"write-nodes"``,
Expand Down Expand Up @@ -2331,23 +2332,27 @@ def _write_ndjson(
dedup_stream(nodes_tmp, is_edges=False)

# Phase: rig. Summaries come from the FINAL deduplicated KGX files, and the
# assembled document is audited before anything is written.
if on_phase is not None:
on_phase("rig")
compile_rig(name, version, rig, section_sources, nodes_tmp.with_suffix(""), edges_tmp.with_suffix(""))
# assembled document is audited before anything is written. RIG-less builds
# skip this phase entirely (#274): only the KGX artifacts are emitted.
if rig is not None:
if on_phase is not None:
on_phase("rig")
compile_rig(name, version, rig, section_sources, nodes_tmp.with_suffix(""), edges_tmp.with_suffix(""))


def compile_graph(
subgraphs: list[Path],
name: str,
version: str,
rig: RIGConfig | dict[str, Any],
rig: RIGConfig | dict[str, Any] | None,
section_sources: list[dict[str, object]] | None = None,
on_phase: Callable[[str], None] | None = None,
on_subgraph: Callable[[], None] | None = None,
uuid_fields: list[str] | None = None,
uuid_domain: str | None = None,
uuid_on_collision: str = "error",
output_dir: Path | None = None,
infores: str | None = None,
) -> None:
"""Aggregate subgraph parquets for NDJSON KGX export using a lazy scan.

Expand All @@ -2359,6 +2364,8 @@ def compile_graph(
``.RIG.yaml`` (source metadata, artifact bases, provenance).
A plain dict is validated as :class:`RIGConfig` first, so
programmatic callers get the same config-time errors as YAML.
``None`` builds KGX only: no ``.RIG.yaml`` is emitted and the
``rig`` phase is skipped.
section_sources: Per-section source descriptors (``local``, ``urls``)
used to cross-check configured RIG relevant-file entries.
on_phase: Optional callback fired with the current phase label
Expand All @@ -2375,11 +2382,16 @@ def compile_graph(
provided -- and to ``TABLASSERT`` otherwise.
uuid_on_collision: ``error`` aborts when two different edges derive one id;
``merge`` folds them into one edge instead (``Graph.uuid_on_collision``).
output_dir: RIG-less output directory for the KGX artifacts (``Graph.output_dir``);
ignored when ``rig`` is set.
infores: RIG-less graph-level infores recorded as ``provided_by``
(``Graph.infores_id``); ignored when ``rig`` is set.

Returns:
``None``; writes ``<name>_<version>.nodes.ndjson``,
``<name>_<version>.edges.ndjson``, and ``<name>_<version>.RIG.yaml``
into ``rig.artifact_base_path`` (created when missing).
``None``; writes ``<name>_<version>.nodes.ndjson`` and
``<name>_<version>.edges.ndjson`` into the output directory
(``rig.artifact_base_path``, else ``output_dir``, else the current
directory), plus ``<name>_<version>.RIG.yaml`` when ``rig`` is set.

Raises:
TablassertError: With code ``rig-validation-failed`` when the
Expand All @@ -2388,8 +2400,8 @@ def compile_graph(
different edges derive one id under the declared ``uuid_fields``
and ``uuid_on_collision`` is ``error``.
"""
rig_cfg: RIGConfig = rig if isinstance(rig, RIGConfig) else RIGConfig.model_validate(rig)
out_dir: Path = Path(rig_cfg.artifact_base_path)
rig_cfg: RIGConfig | None = None if rig is None else (rig if isinstance(rig, RIGConfig) else RIGConfig.model_validate(rig))
out_dir: Path = Path(rig_cfg.artifact_base_path) if rig_cfg is not None else (output_dir or Path("."))
out_dir.mkdir(parents=True, exist_ok=True)
p: Path = out_dir / f"{name}_{version}.tmp"

Expand All @@ -2401,11 +2413,12 @@ def compile_graph(
if n.exists():
n.unlink()

domain: str = uuid_domain or (rig_cfg.source_info.infores_id if uuid_fields else "TABLASSERT")
graph_infores: str = rig_cfg.source_info.infores_id if rig_cfg is not None else (infores or "infores:tablassert")
domain: str = uuid_domain or (graph_infores if uuid_fields else "TABLASSERT")

subnodes: list[pl.LazyFrame]
subedges: list[pl.LazyFrame]
subnodes, subedges = _collect_subframes(subgraphs, on_phase, on_subgraph, rig_cfg.source_info.infores_id)
subnodes, subedges = _collect_subframes(subgraphs, on_phase, on_subgraph, graph_infores)
_write_ndjson(subnodes, subedges, n, e, name, version, rig_cfg, section_sources, on_phase, domain, uuid_fields, uuid_on_collision)


Expand Down
Loading