Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ All notable changes to this project are documented in this file.

### Added

- **`tablassert init` scaffolds a new graph project directory.** Given an optional project directory (`NAME`, default `.`) and `--kind` (`text`, `excel`, `parquet`, `ipc`, or `ndjson`), it writes schema-valid `graph.yaml` (with `DRAFT` RIG placeholders) and `tables/table.yaml` (with a placeholder `source.local` path), refuses to overwrite without `--force`, and prints next-step commands. ([#276](https://github.com/SkyeAv/Tablassert/issues/276))
- **NDJSON (JSONL) is a fifth table source kind.** `kind: ndjson` scans newline-delimited JSON lazily with `pl.scan_ndjson` and renames its columns positionally to `column_1..` in first-seen key order, matching the `parquet` reader contract so `method: column` encodings keep using Excel-style letters. Ragged records scan missing keys as nulls instead of dropping rows; an empty file fails fast with a coded `ndjson-empty` error instead of polars' opaque `Cannot infer NDJSON types` compute error. ([#283](https://github.com/SkyeAv/Tablassert/pull/283), [#262](https://github.com/SkyeAv/Tablassert/issues/262))
- **Every pull request must carry a new changelog entry.** A stdlib-only `scripts/check_changelog.py` requires at least one new bullet under the `## Unreleased` heading of `CHANGELOG.md` relative to the pull request's merge base, failing with remediation guidance when whitespace, subheadings, or released-history edits are the only additions; a CI job runs it on every pull request including forks, release-bump and dependency-bump pull requests are exempt by their `chore(release):` / `chore(deps):` title or commit subject, documentation-only pull requests (every changed path is markdown or under `docs/`) are exempt via `--exempt-docs-only`, and a pre-push hook mirrors the same check locally. ([#284](https://github.com/SkyeAv/Tablassert/pull/284), [#282](https://github.com/SkyeAv/Tablassert/issues/282))
- **Arrow IPC (Feather v2) is a fourth table source kind.** `kind: ipc` scans a `.arrow` file lazily with `pl.scan_ipc` and renames its columns positionally to `column_1..` in schema order, matching the `parquet` reader contract so `method: column` encodings keep using Excel-style letters. Only IPC file layout is supported; a stream-layout file fails fast with a coded `ipc-stream-unsupported` error naming the conversion fix instead of polars' opaque `InvalidFooter` compute error. ([#281](https://github.com/SkyeAv/Tablassert/pull/281), [#264](https://github.com/SkyeAv/Tablassert/issues/264))
Expand Down
23 changes: 22 additions & 1 deletion docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Tablassert extracts knowledge assertions from tabular data into KGX NDJSON. The `tablassert` app
requires the `[cli]` extra (`pip install "tablassert[cli]"` or `uv tool install "tablassert[cli]"`) and
exposes **five subcommands**: `build-fullmap`, `build-kg`, `quick-map`, `validate`,
exposes **six subcommands**: `build-fullmap`, `build-kg`, `init`, `quick-map`, `validate`,
and `validate-kgx`, plus an app-level `--version` flag. Run `tablassert --help` (or `<command> --help`)
for the live surface.

Expand All @@ -12,6 +12,7 @@ for the live surface.
| --- | --- |
| [`build-fullmap`](#build-fullmap) | Build the embedded fullmap redb used for entity resolution |
| [`build-kg`](#build-kg) | Build a KGX NDJSON knowledge graph from a YAML configuration |
| [`init`](#init) | Scaffold a new graph project directory with schema-valid configs |
| [`quick-map`](#quick-map) | Show what fullmap entity resolution does with one or more terms |
| [`validate`](#validate) | Validate a graph or table configuration without executing it |
| [`validate-kgx`](#validate-kgx) | Validate built KGX NDJSON against the Biolink Model and the Translator registry |
Expand Down Expand Up @@ -242,6 +243,26 @@ tablassert validate graph.yaml --schema graph

---

## init

Scaffold a new Tablassert project directory containing `graph.yaml` (with `DRAFT` RIG placeholders)
and `tables/table.yaml` for the selected source `--kind` (with a placeholder `source.local` path).

```bash
tablassert init demo
tablassert init demo --kind parquet
```

| Option | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `NAME` | Path | No | `.` | Target project directory to create or populate |
| `--kind`, `-k` | `excel` \| `text` \| `parquet` \| `ipc` \| `ndjson` | No | `text` | Source format for the scaffolded table config |
| `--force`, `-f` | Flag | No | `false` | Overwrite existing `graph.yaml` or `tables/table.yaml` files |

Refuses to overwrite existing `graph.yaml` or `tables/table.yaml` unless `--force` is passed.

---

## validate-kgx

Use this to check that a completed build is actually Biolink-compliant and that the knowledge-source
Expand Down
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ expectations.

### How-to guides

- **[CLI Reference](cli.md)**: every command and flag: `build-kg`, `build-fullmap`, `validate`,
- **[CLI Reference](cli.md)**: every command and flag: `build-kg`, `build-fullmap`, `init`, `validate`,
`validate-kgx`, `quick-map`
- **[Fullmap](fullmap.md)**: obtain and build the entity-resolution database

Expand Down
3 changes: 2 additions & 1 deletion docs/tutorial.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ gene-disease associations**: nodes and edges with standardized CURIEs, biolink c
and statistical annotations, ready for NCATS Translator.

The input files ship under `docs/examples/` and are validated against the live schema by the test suite;
the configurations below reproduce them exactly.
the configurations below reproduce them exactly. To scaffold a fresh project directory from scratch
instead of copying files manually, run `tablassert init my-graph` (see [CLI Reference](cli.md#init)).

**Time:** 5-10 minutes once the fullmap database is present. Obtaining it is a separate, larger
step: `tablassert build-fullmap` downloads a multi-GB prebuilt archive when one is published for
Expand Down
2 changes: 1 addition & 1 deletion llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Current CLI behavior queries an embedded redb database whose primary file is `fu
- [README](README.md): high-level overview, install snippets, and one-command graph build.
- [Documentation Site Home](docs/index.md): docs landing page and navigation context.
- [Installation Guide](docs/installation.md): three install paths (PyPI, GitHub main, development from source), platform and Rust-toolchain notes, and the missing-extra behavior. Optional extras: `cli`, `rt`, `aria2`, `qc`, and `log`.
- [CLI Reference](docs/cli.md): the five commands (`build-fullmap`, `build-kg`, `quick-map`, `validate`, and `validate-kgx`) plus the app-level `--version` flag.
- [CLI Reference](docs/cli.md): the six commands (`build-fullmap`, `build-kg`, `init`, `quick-map`, `validate`, and `validate-kgx`) plus the app-level `--version` flag.
- [Tutorial](docs/tutorial.md): first end-to-end run from CSV input to KGX NDJSON output.

## YAML Authoring
Expand Down
4 changes: 4 additions & 0 deletions scripts/vulture_whitelist.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@

from typing import Any

from tablassert.cli import init
from tablassert.enums import ContentCategories, DataFormats, FillMethods, IngestCategories, ModelingCategories, ProvisionMechanisms, SourceStatuses
from tablassert.models import (
RIGFilteredContent,
Expand Down Expand Up @@ -105,3 +106,6 @@
RIGFutureModelingConsideration.consideration
RIGSupportingDataSourceInfo.terms_of_use_info
RIGProvenanceInfo.artifacts

# cli.py: Cyclopts @APP.command callbacks registered dynamically at import time.
init
102 changes: 102 additions & 0 deletions src/tablassert/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -987,6 +987,108 @@ def validate(
run(3, validate_pipeline, configuration_file)


@APP.command(name="init")
def init(
name: Annotated[Path, cyclopts.Parameter(allow_leading_hyphen=False, negative="")] = Path("."),
kind: Annotated[Literal["excel", "text", "parquet", "ipc", "ndjson"], cyclopts.Parameter(name=["--kind", "-k"])] = "text",
force: Annotated[bool, cyclopts.Parameter(name=["--force", "-f"], negative="")] = False,
) -> None:
"""Scaffold a new Tablassert graph project directory.

Creates ``graph.yaml`` and ``tables/table.yaml`` for the chosen source ``--kind``
so ``tablassert validate <dir>/graph.yaml --schema graph`` passes immediately.

Args:
name: Target project directory (created if missing; default: current directory).
kind: Source format for the scaffolded table config (``excel``, ``text``,
``parquet``, ``ipc``, or ``ndjson``).
force: Overwrite existing ``graph.yaml`` or ``tables/table.yaml`` files.
"""
from tablassert import ingests
from tablassert.enums import IngestCategories, SourceStatuses
from tablassert.models import Graph, Section

target_dir: Path = name
graph_path: Path = target_dir / "graph.yaml"
tables_dir: Path = target_dir / "tables"
table_path: Path = tables_dir / "table.yaml"

if not force and (graph_path.exists() or table_path.exists()):
existing: Path = graph_path if graph_path.exists() else table_path
print(f"tablassert init: {existing} already exists (use --force to overwrite).", file=sys.stderr)
raise SystemExit(1)

source_specs: dict[str, tuple[str, dict[str, Any]]] = {
"text": ("sample.csv", {"kind": "text", "local": "data/sample.csv", "delimiter": ",", "row_slice": [1, "auto"]}),
"excel": ("sample.xlsx", {"kind": "excel", "local": "data/sample.xlsx", "sheet": "Sheet1", "row_slice": [1, "auto"]}),
"parquet": ("sample.parquet", {"kind": "parquet", "local": "data/sample.parquet"}),
"ipc": ("sample.arrow", {"kind": "ipc", "local": "data/sample.arrow"}),
"ndjson": ("sample.ndjson", {"kind": "ndjson", "local": "data/sample.ndjson"}),
}
sample_filename, source_block = source_specs[kind]
source_block["url"] = [f"https://example.org/{sample_filename}"]

table_data: dict[str, Any] = {
"template": {
"source": source_block,
"statement": {
"subject": {"method": "column", "encoding": "A", "prioritize": ["Gene"]},
"predicate": "associated_with",
"object": {"method": "column", "encoding": "B", "prioritize": ["Disease"]},
},
"provenance": {"repo": "PMID", "publication": "12345678"},
"annotations": [{"annotation": "p_value", "method": "column", "encoding": "C"}],
},
"sections": [{}],
}
Section.model_validate(table_data["template"])

graph_name: str = target_dir.resolve().name or "my-graph"
graph_data: dict[str, Any] = {
"name": graph_name,
"version": "0.1.0",
"tables": ["tables/table.yaml"],
"fullmap": ".fullmap",
"rig": {
"source_info": {
"infores_id": "infores:tablassert-draft",
"name": f"DRAFT: {graph_name}",
"description": "DRAFT: describe the upstream source, its scope, and how its knowledge is produced.",
"terms_of_use_info": {"terms_of_use_description": "DRAFT: assess upstream source terms of use or license before publishing."},
"data_access_locations": ["https://example.org/DRAFT-replace-with-source-url"],
"source_status": SourceStatuses.UNKNOWN.value,
},
"ingest_info": {
"ingest_categories": [IngestCategories.TRANSLATOR_KNOWLEDGE_CREATOR.value],
"utility": "DRAFT: describe why this source was ingested and its utility for Translator use cases.",
"scope": "DRAFT: describe what knowledge is included and excluded in this ingest.",
},
"provenance_info": {"contributions": ["DRAFT: Your Name - code author, data modeling"]},
"artifact_base_url": "https://example.org/DRAFT-replace-with-artifact-url",
"artifact_base_path": "./output",
},
}
Graph.model_validate(graph_data)

target_dir.mkdir(parents=True, exist_ok=True)
tables_dir.mkdir(parents=True, exist_ok=True)

ingests.to_yaml(table_path, table_data)
ingests.to_yaml(graph_path, graph_data)
header: str = "# DRAFT graph configuration generated by `tablassert init`.\n# Replace every DRAFT placeholder before publishing.\n"
graph_path.write_text(header + graph_path.read_text(encoding="utf-8"), encoding="utf-8")

print(
f"Scaffolded Tablassert project in {target_dir} (kind={kind}).\n"
"Next steps:\n"
f" 1. Replace source.local ({source_block['local']}) and DRAFT placeholders in {table_path} / {graph_path}\n"
f" 2. tablassert build-fullmap -o {target_dir / '.fullmap'}\n"
f" 3. tablassert validate {graph_path} --schema graph\n"
f" 4. tablassert build-kg {graph_path}",
file=sys.stderr,
)


def _print_kgx_section(label: str, section: dict[str, Any], path: Path) -> None:
"""Print one ``validate-kgx`` per-file section (shared by the plain and ``--prune`` paths)."""
if section["missing"]:
Expand Down
2 changes: 2 additions & 0 deletions tests/test_docs_cli_coverage.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,15 @@
COMMAND_NAME_DOCS: dict[str, tuple[str, ...]] = {
"build-fullmap": ("cli.md", "fullmap.md"),
"build-kg": ("cli.md",),
"init": ("cli.md",),
"quick-map": ("cli.md", "fullmap.md"),
"validate": ("cli.md",),
"validate-kgx": ("cli.md",),
}
COMMAND_FLAG_DOCS: dict[str, tuple[str, ...]] = {
"build-fullmap": ("cli.md", "fullmap.md"),
"build-kg": ("cli.md",),
"init": ("cli.md",),
"quick-map": ("cli.md", "fullmap.md"),
"validate": ("cli.md",),
"validate-kgx": ("cli.md",),
Expand Down
48 changes: 48 additions & 0 deletions tests/test_init.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
from __future__ import annotations

from pathlib import Path

import pytest

from tablassert import ingests
from tablassert.cli import APP
from tablassert.models import Graph


@pytest.mark.parametrize("kind", ["text", "excel", "parquet", "ipc", "ndjson"])
def test_init_scaffolds_valid_project(tmp_path: Path, kind: str, monkeypatch: pytest.MonkeyPatch) -> None:
project_dir = tmp_path / f"demo-{kind}"
APP(["init", str(project_dir), "--kind", kind], exit_on_error=False, result_action="return_value")

graph_path = project_dir / "graph.yaml"
table_path = project_dir / "tables" / "table.yaml"
assert graph_path.is_file()
assert table_path.is_file()
assert not (project_dir / "data").exists()

raw_table = ingests.from_yaml(table_path)
assert isinstance(raw_table, dict)
assert raw_table["template"]["source"]["kind"] == kind
assert str(raw_table["template"]["source"]["local"]).startswith("data/sample.")

raw_graph = ingests.from_yaml(graph_path)
assert isinstance(raw_graph, dict)
graph = Graph.model_validate(raw_graph)
assert graph.name == f"demo-{kind}"
assert graph.rig is not None
assert graph.rig.source_info.infores_id == "infores:tablassert-draft"

monkeypatch.chdir(project_dir)
APP(["validate", "graph.yaml", "--schema", "graph"], exit_on_error=False, result_action="return_value")


def test_init_refuses_overwrite_without_force(tmp_path: Path) -> None:
project_dir = tmp_path / "demo"
APP(["init", str(project_dir)], exit_on_error=False, result_action="return_value")

with pytest.raises(SystemExit) as exc:
APP(["init", str(project_dir)], exit_on_error=False, result_action="return_value")
assert exc.value.code == 1

APP(["init", str(project_dir), "--force"], exit_on_error=False, result_action="return_value")
assert (project_dir / "graph.yaml").is_file()