diff --git a/CHANGELOG.md b/CHANGELOG.md index 6a64043d..f729679f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,7 @@ All notable changes to this project are documented in this file. ### Added +- **`tablassert init` scaffolds a new graph project directory.** Given an optional project directory (`NAME`, default `.`) and `--kind` (`text`, `excel`, `parquet`, `ipc`, or `ndjson`), it writes schema-valid `graph.yaml` (with `DRAFT` RIG placeholders) and `tables/table.yaml` (with a placeholder `source.local` path), refuses to overwrite without `--force`, and prints next-step commands. ([#276](https://github.com/SkyeAv/Tablassert/issues/276)) - **NDJSON (JSONL) is a fifth table source kind.** `kind: ndjson` scans newline-delimited JSON lazily with `pl.scan_ndjson` and renames its columns positionally to `column_1..` in first-seen key order, matching the `parquet` reader contract so `method: column` encodings keep using Excel-style letters. Ragged records scan missing keys as nulls instead of dropping rows; an empty file fails fast with a coded `ndjson-empty` error instead of polars' opaque `Cannot infer NDJSON types` compute error. ([#283](https://github.com/SkyeAv/Tablassert/pull/283), [#262](https://github.com/SkyeAv/Tablassert/issues/262)) - **Every pull request must carry a new changelog entry.** A stdlib-only `scripts/check_changelog.py` requires at least one new bullet under the `## Unreleased` heading of `CHANGELOG.md` relative to the pull request's merge base, failing with remediation guidance when whitespace, subheadings, or released-history edits are the only additions; a CI job runs it on every pull request including forks, release-bump and dependency-bump pull requests are exempt by their `chore(release):` / `chore(deps):` title or commit subject, documentation-only pull requests (every changed path is markdown or under `docs/`) are exempt via `--exempt-docs-only`, and a pre-push hook mirrors the same check locally. ([#284](https://github.com/SkyeAv/Tablassert/pull/284), [#282](https://github.com/SkyeAv/Tablassert/issues/282)) - **Arrow IPC (Feather v2) is a fourth table source kind.** `kind: ipc` scans a `.arrow` file lazily with `pl.scan_ipc` and renames its columns positionally to `column_1..` in schema order, matching the `parquet` reader contract so `method: column` encodings keep using Excel-style letters. Only IPC file layout is supported; a stream-layout file fails fast with a coded `ipc-stream-unsupported` error naming the conversion fix instead of polars' opaque `InvalidFooter` compute error. ([#281](https://github.com/SkyeAv/Tablassert/pull/281), [#264](https://github.com/SkyeAv/Tablassert/issues/264)) diff --git a/docs/cli.md b/docs/cli.md index f92989b8..fff76af3 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -2,7 +2,7 @@ Tablassert extracts knowledge assertions from tabular data into KGX NDJSON. The `tablassert` app requires the `[cli]` extra (`pip install "tablassert[cli]"` or `uv tool install "tablassert[cli]"`) and -exposes **five subcommands**: `build-fullmap`, `build-kg`, `quick-map`, `validate`, +exposes **six subcommands**: `build-fullmap`, `build-kg`, `init`, `quick-map`, `validate`, and `validate-kgx`, plus an app-level `--version` flag. Run `tablassert --help` (or ` --help`) for the live surface. @@ -12,6 +12,7 @@ for the live surface. | --- | --- | | [`build-fullmap`](#build-fullmap) | Build the embedded fullmap redb used for entity resolution | | [`build-kg`](#build-kg) | Build a KGX NDJSON knowledge graph from a YAML configuration | +| [`init`](#init) | Scaffold a new graph project directory with schema-valid configs | | [`quick-map`](#quick-map) | Show what fullmap entity resolution does with one or more terms | | [`validate`](#validate) | Validate a graph or table configuration without executing it | | [`validate-kgx`](#validate-kgx) | Validate built KGX NDJSON against the Biolink Model and the Translator registry | @@ -242,6 +243,26 @@ tablassert validate graph.yaml --schema graph --- +## init + +Scaffold a new Tablassert project directory containing `graph.yaml` (with `DRAFT` RIG placeholders) +and `tables/table.yaml` for the selected source `--kind` (with a placeholder `source.local` path). + +```bash +tablassert init demo +tablassert init demo --kind parquet +``` + +| Option | Type | Required | Default | Description | +| --- | --- | --- | --- | --- | +| `NAME` | Path | No | `.` | Target project directory to create or populate | +| `--kind`, `-k` | `excel` \| `text` \| `parquet` \| `ipc` \| `ndjson` | No | `text` | Source format for the scaffolded table config | +| `--force`, `-f` | Flag | No | `false` | Overwrite existing `graph.yaml` or `tables/table.yaml` files | + +Refuses to overwrite existing `graph.yaml` or `tables/table.yaml` unless `--force` is passed. + +--- + ## validate-kgx Use this to check that a completed build is actually Biolink-compliant and that the knowledge-source diff --git a/docs/index.md b/docs/index.md index ae64a6d9..9cee8736 100644 --- a/docs/index.md +++ b/docs/index.md @@ -32,7 +32,7 @@ expectations. ### How-to guides -- **[CLI Reference](cli.md)**: every command and flag: `build-kg`, `build-fullmap`, `validate`, +- **[CLI Reference](cli.md)**: every command and flag: `build-kg`, `build-fullmap`, `init`, `validate`, `validate-kgx`, `quick-map` - **[Fullmap](fullmap.md)**: obtain and build the entity-resolution database diff --git a/docs/tutorial.md b/docs/tutorial.md index a49e906c..efd67cfd 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -5,7 +5,8 @@ gene-disease associations**: nodes and edges with standardized CURIEs, biolink c and statistical annotations, ready for NCATS Translator. The input files ship under `docs/examples/` and are validated against the live schema by the test suite; -the configurations below reproduce them exactly. +the configurations below reproduce them exactly. To scaffold a fresh project directory from scratch +instead of copying files manually, run `tablassert init my-graph` (see [CLI Reference](cli.md#init)). **Time:** 5-10 minutes once the fullmap database is present. Obtaining it is a separate, larger step: `tablassert build-fullmap` downloads a multi-GB prebuilt archive when one is published for diff --git a/llms.txt b/llms.txt index 3091f560..b10e48fa 100644 --- a/llms.txt +++ b/llms.txt @@ -10,7 +10,7 @@ Current CLI behavior queries an embedded redb database whose primary file is `fu - [README](README.md): high-level overview, install snippets, and one-command graph build. - [Documentation Site Home](docs/index.md): docs landing page and navigation context. - [Installation Guide](docs/installation.md): three install paths (PyPI, GitHub main, development from source), platform and Rust-toolchain notes, and the missing-extra behavior. Optional extras: `cli`, `rt`, `aria2`, `qc`, and `log`. -- [CLI Reference](docs/cli.md): the five commands (`build-fullmap`, `build-kg`, `quick-map`, `validate`, and `validate-kgx`) plus the app-level `--version` flag. +- [CLI Reference](docs/cli.md): the six commands (`build-fullmap`, `build-kg`, `init`, `quick-map`, `validate`, and `validate-kgx`) plus the app-level `--version` flag. - [Tutorial](docs/tutorial.md): first end-to-end run from CSV input to KGX NDJSON output. ## YAML Authoring diff --git a/scripts/vulture_whitelist.py b/scripts/vulture_whitelist.py index 2d7b3485..478947d0 100644 --- a/scripts/vulture_whitelist.py +++ b/scripts/vulture_whitelist.py @@ -20,6 +20,7 @@ from typing import Any +from tablassert.cli import init from tablassert.enums import ContentCategories, DataFormats, FillMethods, IngestCategories, ModelingCategories, ProvisionMechanisms, SourceStatuses from tablassert.models import ( RIGFilteredContent, @@ -105,3 +106,6 @@ RIGFutureModelingConsideration.consideration RIGSupportingDataSourceInfo.terms_of_use_info RIGProvenanceInfo.artifacts + +# cli.py: Cyclopts @APP.command callbacks registered dynamically at import time. +init diff --git a/src/tablassert/cli.py b/src/tablassert/cli.py index 2fc798b2..76e42429 100644 --- a/src/tablassert/cli.py +++ b/src/tablassert/cli.py @@ -987,6 +987,108 @@ def validate( run(3, validate_pipeline, configuration_file) +@APP.command(name="init") +def init( + name: Annotated[Path, cyclopts.Parameter(allow_leading_hyphen=False, negative="")] = Path("."), + kind: Annotated[Literal["excel", "text", "parquet", "ipc", "ndjson"], cyclopts.Parameter(name=["--kind", "-k"])] = "text", + force: Annotated[bool, cyclopts.Parameter(name=["--force", "-f"], negative="")] = False, +) -> None: + """Scaffold a new Tablassert graph project directory. + + Creates ``graph.yaml`` and ``tables/table.yaml`` for the chosen source ``--kind`` + so ``tablassert validate /graph.yaml --schema graph`` passes immediately. + + Args: + name: Target project directory (created if missing; default: current directory). + kind: Source format for the scaffolded table config (``excel``, ``text``, + ``parquet``, ``ipc``, or ``ndjson``). + force: Overwrite existing ``graph.yaml`` or ``tables/table.yaml`` files. + """ + from tablassert import ingests + from tablassert.enums import IngestCategories, SourceStatuses + from tablassert.models import Graph, Section + + target_dir: Path = name + graph_path: Path = target_dir / "graph.yaml" + tables_dir: Path = target_dir / "tables" + table_path: Path = tables_dir / "table.yaml" + + if not force and (graph_path.exists() or table_path.exists()): + existing: Path = graph_path if graph_path.exists() else table_path + print(f"tablassert init: {existing} already exists (use --force to overwrite).", file=sys.stderr) + raise SystemExit(1) + + source_specs: dict[str, tuple[str, dict[str, Any]]] = { + "text": ("sample.csv", {"kind": "text", "local": "data/sample.csv", "delimiter": ",", "row_slice": [1, "auto"]}), + "excel": ("sample.xlsx", {"kind": "excel", "local": "data/sample.xlsx", "sheet": "Sheet1", "row_slice": [1, "auto"]}), + "parquet": ("sample.parquet", {"kind": "parquet", "local": "data/sample.parquet"}), + "ipc": ("sample.arrow", {"kind": "ipc", "local": "data/sample.arrow"}), + "ndjson": ("sample.ndjson", {"kind": "ndjson", "local": "data/sample.ndjson"}), + } + sample_filename, source_block = source_specs[kind] + source_block["url"] = [f"https://example.org/{sample_filename}"] + + table_data: dict[str, Any] = { + "template": { + "source": source_block, + "statement": { + "subject": {"method": "column", "encoding": "A", "prioritize": ["Gene"]}, + "predicate": "associated_with", + "object": {"method": "column", "encoding": "B", "prioritize": ["Disease"]}, + }, + "provenance": {"repo": "PMID", "publication": "12345678"}, + "annotations": [{"annotation": "p_value", "method": "column", "encoding": "C"}], + }, + "sections": [{}], + } + Section.model_validate(table_data["template"]) + + graph_name: str = target_dir.resolve().name or "my-graph" + graph_data: dict[str, Any] = { + "name": graph_name, + "version": "0.1.0", + "tables": ["tables/table.yaml"], + "fullmap": ".fullmap", + "rig": { + "source_info": { + "infores_id": "infores:tablassert-draft", + "name": f"DRAFT: {graph_name}", + "description": "DRAFT: describe the upstream source, its scope, and how its knowledge is produced.", + "terms_of_use_info": {"terms_of_use_description": "DRAFT: assess upstream source terms of use or license before publishing."}, + "data_access_locations": ["https://example.org/DRAFT-replace-with-source-url"], + "source_status": SourceStatuses.UNKNOWN.value, + }, + "ingest_info": { + "ingest_categories": [IngestCategories.TRANSLATOR_KNOWLEDGE_CREATOR.value], + "utility": "DRAFT: describe why this source was ingested and its utility for Translator use cases.", + "scope": "DRAFT: describe what knowledge is included and excluded in this ingest.", + }, + "provenance_info": {"contributions": ["DRAFT: Your Name - code author, data modeling"]}, + "artifact_base_url": "https://example.org/DRAFT-replace-with-artifact-url", + "artifact_base_path": "./output", + }, + } + Graph.model_validate(graph_data) + + target_dir.mkdir(parents=True, exist_ok=True) + tables_dir.mkdir(parents=True, exist_ok=True) + + ingests.to_yaml(table_path, table_data) + ingests.to_yaml(graph_path, graph_data) + header: str = "# DRAFT graph configuration generated by `tablassert init`.\n# Replace every DRAFT placeholder before publishing.\n" + graph_path.write_text(header + graph_path.read_text(encoding="utf-8"), encoding="utf-8") + + print( + f"Scaffolded Tablassert project in {target_dir} (kind={kind}).\n" + "Next steps:\n" + f" 1. Replace source.local ({source_block['local']}) and DRAFT placeholders in {table_path} / {graph_path}\n" + f" 2. tablassert build-fullmap -o {target_dir / '.fullmap'}\n" + f" 3. tablassert validate {graph_path} --schema graph\n" + f" 4. tablassert build-kg {graph_path}", + file=sys.stderr, + ) + + def _print_kgx_section(label: str, section: dict[str, Any], path: Path) -> None: """Print one ``validate-kgx`` per-file section (shared by the plain and ``--prune`` paths).""" if section["missing"]: diff --git a/tests/test_docs_cli_coverage.py b/tests/test_docs_cli_coverage.py index 6c802d6a..15a2908a 100644 --- a/tests/test_docs_cli_coverage.py +++ b/tests/test_docs_cli_coverage.py @@ -23,6 +23,7 @@ COMMAND_NAME_DOCS: dict[str, tuple[str, ...]] = { "build-fullmap": ("cli.md", "fullmap.md"), "build-kg": ("cli.md",), + "init": ("cli.md",), "quick-map": ("cli.md", "fullmap.md"), "validate": ("cli.md",), "validate-kgx": ("cli.md",), @@ -30,6 +31,7 @@ COMMAND_FLAG_DOCS: dict[str, tuple[str, ...]] = { "build-fullmap": ("cli.md", "fullmap.md"), "build-kg": ("cli.md",), + "init": ("cli.md",), "quick-map": ("cli.md", "fullmap.md"), "validate": ("cli.md",), "validate-kgx": ("cli.md",), diff --git a/tests/test_init.py b/tests/test_init.py new file mode 100644 index 00000000..4abf1f74 --- /dev/null +++ b/tests/test_init.py @@ -0,0 +1,48 @@ +from __future__ import annotations + +from pathlib import Path + +import pytest + +from tablassert import ingests +from tablassert.cli import APP +from tablassert.models import Graph + + +@pytest.mark.parametrize("kind", ["text", "excel", "parquet", "ipc", "ndjson"]) +def test_init_scaffolds_valid_project(tmp_path: Path, kind: str, monkeypatch: pytest.MonkeyPatch) -> None: + project_dir = tmp_path / f"demo-{kind}" + APP(["init", str(project_dir), "--kind", kind], exit_on_error=False, result_action="return_value") + + graph_path = project_dir / "graph.yaml" + table_path = project_dir / "tables" / "table.yaml" + assert graph_path.is_file() + assert table_path.is_file() + assert not (project_dir / "data").exists() + + raw_table = ingests.from_yaml(table_path) + assert isinstance(raw_table, dict) + assert raw_table["template"]["source"]["kind"] == kind + assert str(raw_table["template"]["source"]["local"]).startswith("data/sample.") + + raw_graph = ingests.from_yaml(graph_path) + assert isinstance(raw_graph, dict) + graph = Graph.model_validate(raw_graph) + assert graph.name == f"demo-{kind}" + assert graph.rig is not None + assert graph.rig.source_info.infores_id == "infores:tablassert-draft" + + monkeypatch.chdir(project_dir) + APP(["validate", "graph.yaml", "--schema", "graph"], exit_on_error=False, result_action="return_value") + + +def test_init_refuses_overwrite_without_force(tmp_path: Path) -> None: + project_dir = tmp_path / "demo" + APP(["init", str(project_dir)], exit_on_error=False, result_action="return_value") + + with pytest.raises(SystemExit) as exc: + APP(["init", str(project_dir)], exit_on_error=False, result_action="return_value") + assert exc.value.code == 1 + + APP(["init", str(project_dir), "--force"], exit_on_error=False, result_action="return_value") + assert (project_dir / "graph.yaml").is_file()