Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ All notable changes to this project are documented in this file.

### Added

- **`tablassert publish-hf` exports validated KGX outputs and RIG metadata to Hugging Face Hub.** Behind the opt-in `[hf]` extra (`huggingface-hub`), it gates on `validate_kgx`, generates a RIG-derived `README.md` dataset card with `nodes`/`edges` Parquet configs, attaches raw NDJSON and `.RIG.yaml`, and supports `--dry-run`. ([#269](https://github.com/SkyeAv/Tablassert/issues/269))
- **NDJSON (JSONL) is a fifth table source kind.** `kind: ndjson` scans newline-delimited JSON lazily with `pl.scan_ndjson` and renames its columns positionally to `column_1..` in first-seen key order, matching the `parquet` reader contract so `method: column` encodings keep using Excel-style letters. Ragged records scan missing keys as nulls instead of dropping rows; an empty file fails fast with a coded `ndjson-empty` error instead of polars' opaque `Cannot infer NDJSON types` compute error. ([#283](https://github.com/SkyeAv/Tablassert/pull/283), [#262](https://github.com/SkyeAv/Tablassert/issues/262))
- **Every pull request must carry a new changelog entry.** A stdlib-only `scripts/check_changelog.py` requires at least one new bullet under the `## Unreleased` heading of `CHANGELOG.md` relative to the pull request's merge base, failing with remediation guidance when whitespace, subheadings, or released-history edits are the only additions; a CI job runs it on every pull request including forks, release-bump and dependency-bump pull requests are exempt by their `chore(release):` / `chore(deps):` title or commit subject, documentation-only pull requests (every changed path is markdown or under `docs/`) are exempt via `--exempt-docs-only`, and a pre-push hook mirrors the same check locally. ([#284](https://github.com/SkyeAv/Tablassert/pull/284), [#282](https://github.com/SkyeAv/Tablassert/issues/282))
- **Arrow IPC (Feather v2) is a fourth table source kind.** `kind: ipc` scans a `.arrow` file lazily with `pl.scan_ipc` and renames its columns positionally to `column_1..` in schema order, matching the `parquet` reader contract so `method: column` encodings keep using Excel-style letters. Only IPC file layout is supported; a stream-layout file fails fast with a coded `ipc-stream-unsupported` error naming the conversion fix instead of polars' opaque `InvalidFooter` compute error. ([#281](https://github.com/SkyeAv/Tablassert/pull/281), [#264](https://github.com/SkyeAv/Tablassert/issues/264))
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,7 @@ Python API to your project. Other extras are opt-in:
| `aria2` | bundled aria2c downloader, used automatically by `build-fullmap` when installed (Linux/Windows wheels only) | `pip install "tablassert[aria2]"` |
| `qc` | four-stage QC audit (exact match, fuzzy match, abbreviation expansion, and SapBERT embeddings) | `pip install "tablassert[qc]"` |
| `log` | loguru-backed file/progress logging (rotation, enqueue) | `pip install "tablassert[log]"` |
| `hf` | Hugging Face Hub dataset export (`publish-hf`) | `pip install "tablassert[hf]"` |

Reaching a feature whose extra is missing never produces a bare `ModuleNotFoundError`: the failure
names the missing package and the install command that fixes it. QC is opt-in at build time
Expand Down
23 changes: 22 additions & 1 deletion docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Tablassert extracts knowledge assertions from tabular data into KGX NDJSON. The `tablassert` app
requires the `[cli]` extra (`pip install "tablassert[cli]"` or `uv tool install "tablassert[cli]"`) and
exposes **five subcommands**: `build-fullmap`, `build-kg`, `quick-map`, `validate`,
exposes **six subcommands**: `build-fullmap`, `build-kg`, `publish-hf`, `quick-map`, `validate`,
and `validate-kgx`, plus an app-level `--version` flag. Run `tablassert --help` (or `<command> --help`)
for the live surface.

Expand All @@ -12,6 +12,7 @@ for the live surface.
| --- | --- |
| [`build-fullmap`](#build-fullmap) | Build the embedded fullmap redb used for entity resolution |
| [`build-kg`](#build-kg) | Build a KGX NDJSON knowledge graph from a YAML configuration |
| [`publish-hf`](#publish-hf) | Validate and publish KGX outputs and RIG metadata to Hugging Face Hub |
| [`quick-map`](#quick-map) | Show what fullmap entity resolution does with one or more terms |
| [`validate`](#validate) | Validate a graph or table configuration without executing it |
| [`validate-kgx`](#validate-kgx) | Validate built KGX NDJSON against the Biolink Model and the Translator registry |
Expand Down Expand Up @@ -242,6 +243,26 @@ tablassert validate graph.yaml --schema graph

---

## publish-hf

Validate built KGX artifacts with `validate-kgx` and publish them (Parquet-converted `nodes` and
`edges` configs, raw NDJSONs, `.RIG.yaml`, and a RIG-derived `README.md` dataset card) to the
Hugging Face Hub (requires the `[hf]` extra).

```bash
tablassert publish-hf graph.yaml --repo-id org/my-kg --dry-run
tablassert publish-hf -f graph.yaml -r org/my-kg
```

| Option | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `GRAPH-CONFIGURATION-FILE` (`--configuration-file`, `-f`) | Path | Yes | n/a | Graph YAML configuration file |
| `--repo-id`, `-r` | String | Yes | n/a | Target Hugging Face dataset repository ID (`org/dataset`) |
| `--private` | Flag | No | `false` | Create the dataset repository as private if it does not exist |
| `--dry-run` | Flag | No | `false` | Print the upload plan without contacting the Hugging Face Hub |

---

## validate-kgx

Use this to check that a completed build is actually Biolink-compliant and that the knowledge-source
Expand Down
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ expectations.

### How-to guides

- **[CLI Reference](cli.md)**: every command and flag: `build-kg`, `build-fullmap`, `validate`,
- **[CLI Reference](cli.md)**: every command and flag: `build-kg`, `build-fullmap`, `publish-hf`, `validate`,
`validate-kgx`, `quick-map`
- **[Fullmap](fullmap.md)**: obtain and build the entity-resolution database

Expand Down
2 changes: 2 additions & 0 deletions docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ uv tool install "tablassert[cli]" # or: pip install "tablassert[cli]"
| `aria2` | Bundled aria2c downloader, used automatically by `build-fullmap` when installed (Linux/Windows wheels only) | `aria2==0.0.1b0` (imports as `aria2c`, bundles aria2c) |
| `qc` | QC runtime (exact match, fuzzy match, abbreviation expansion, then a SapBERT embedding audit) | `scikit-learn`, `sentence-transformers` (`torch` + `numpy` arrive transitively; `rapidfuzz` is a core dependency) |
| `log` | loguru-backed file/progress logging (rotation, enqueue) | `loguru` |
| `hf` | Hugging Face Hub dataset export (`publish-hf`) | `huggingface-hub` |

Install any extra the same way: `uv tool install "tablassert[<extra>]"` or
`pip install "tablassert[<extra>]"`, and combine them as `"tablassert[cli,qc]"`. Wheels ship for
Expand Down Expand Up @@ -72,6 +73,7 @@ fails, not a failure report):
| `tablassert` | `[cli]` | Before importing the command application, so a base Python-API install reports the exact install command instead of a bare module error |
| `build-kg --qc` | `[qc]` | Before the build starts: the QC audit runs at the very end of the build, so a late failure would cost the entire entity-resolution pass |
| `build-fullmap` | `[aria2]` | Before the first download, to pick the downloader: bundled aria2c when the extra is installed, Python downloader otherwise (announced on stderr and logged either way; never a missing-extra failure) |
| `publish-hf` | `[hf]` | Before validating and uploading KGX artifacts to the Hugging Face Hub |

A partially installed extra names every package it is still missing, so installing them is one step
rather than a retry loop. Library calls that reach an optional import directly (for example
Expand Down
6 changes: 3 additions & 3 deletions llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ Current CLI behavior queries an embedded redb database whose primary file is `fu
## Quickstart
- [README](README.md): high-level overview, install snippets, and one-command graph build.
- [Documentation Site Home](docs/index.md): docs landing page and navigation context.
- [Installation Guide](docs/installation.md): three install paths (PyPI, GitHub main, development from source), platform and Rust-toolchain notes, and the missing-extra behavior. Optional extras: `cli`, `rt`, `aria2`, `qc`, and `log`.
- [CLI Reference](docs/cli.md): the five commands (`build-fullmap`, `build-kg`, `quick-map`, `validate`, and `validate-kgx`) plus the app-level `--version` flag.
- [Installation Guide](docs/installation.md): three install paths (PyPI, GitHub main, development from source), platform and Rust-toolchain notes, and the missing-extra behavior. Optional extras: `cli`, `rt`, `aria2`, `qc`, `log`, and `hf`.
- [CLI Reference](docs/cli.md): the six commands (`build-fullmap`, `build-kg`, `publish-hf`, `quick-map`, `validate`, and `validate-kgx`) plus the app-level `--version` flag.
- [Tutorial](docs/tutorial.md): first end-to-end run from CSV input to KGX NDJSON output.

## YAML Authoring
Expand All @@ -35,7 +35,7 @@ Current CLI behavior queries an embedded redb database whose primary file is `fu
## Contributor Development
- [Contributing Guide](CONTRIBUTING.md): setup, style conventions, checks, and PR expectations.
- [Development](docs/development.md): docs-site contributor setup and the local `uv run mkdocs build --strict` docs gate.
- [Project Metadata](pyproject.toml): dependencies, tooling config, and CLI entrypoint. Optional extras: `cli`, `rt`, `aria2`, `qc`, and `log`.
- [Project Metadata](pyproject.toml): dependencies, tooling config, and CLI entrypoint. Optional extras: `cli`, `rt`, `aria2`, `qc`, `log`, and `hf`.
- [Tests](tests/): pytest coverage and fixtures.
- [MkDocs Configuration](mkdocs.yml): docs navigation and published pages.
- [GitHub Workflows](.github/workflows/): `ci.yml` runs lint, format, type, dead-code (vulture, deptry), test, rust (clippy, machete), and mkdocs checks when a release changes the project version or on manual dispatch; `docs.yml` builds and deploys MkDocs; `pipy.yml` checks the project version, builds sdist and wheel artifacts, and publishes to PyPI; `tag-version.yml` tags changed project versions; `paper.yml` tests the JOSS paper compile and uploads the PDF artifact; `security.yml` scans commits for leaked secrets with gitleaks and audits dependencies against advisory databases with pip-audit and cargo-audit for the Rust lockfile.
Expand Down
3 changes: 3 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,9 @@ qc = [
log = [
"loguru>=0.7.3",
]
hf = [
"huggingface-hub>=0.25.0",
]

[dependency-groups]
test = [
Expand Down
4 changes: 4 additions & 0 deletions scripts/vulture_whitelist.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@

from typing import Any

from tablassert.cli import publish_hf
from tablassert.enums import ContentCategories, DataFormats, FillMethods, IngestCategories, ModelingCategories, ProvisionMechanisms, SourceStatuses
from tablassert.models import (
RIGFilteredContent,
Expand Down Expand Up @@ -105,3 +106,6 @@
RIGFutureModelingConsideration.consideration
RIGSupportingDataSourceInfo.terms_of_use_info
RIGProvenanceInfo.artifacts

# cli.py: Cyclopts @APP.command callbacks registered dynamically at import time.
publish_hf
110 changes: 110 additions & 0 deletions src/tablassert/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -987,6 +987,116 @@ def validate(
run(3, validate_pipeline, configuration_file)


def _hf_dataset_card(g: Any) -> str:
"""Render a Hugging Face dataset card README.md from Graph and RIG metadata."""
rig = getattr(g, "rig", None)
if rig is not None:
src = rig.source_info
terms = src.terms_of_use_info
license_line: str = terms.license_name or terms.terms_of_use_description or "See source terms of use"
contributions: str = "\n".join(f"- {c}" for c in rig.provenance_info.contributions)
description: str = src.description or rig.ingest_info.utility
infores_id: str = src.infores_id
else:
license_line = "See source terms of use"
contributions = "- Tablassert"
description = f"Knowledge graph {g.name} ({g.version}) built with Tablassert."
infores_id = getattr(g, "infores_id", "infores:tablassert")
return (
"---\n"
"configs:\n"
" - config_name: nodes\n"
" data_files:\n"
" - split: train\n"
" path: nodes/*.parquet\n"
" - config_name: edges\n"
" data_files:\n"
" - split: train\n"
" path: edges/*.parquet\n"
"---\n\n"
f"# {g.name} ({g.version})\n\n"
f"{description}\n\n"
f"- **Infores:** `{infores_id}`\n"
f"- **Terms of use:** {license_line}\n\n"
"## Contributions\n\n"
f"{contributions}\n"
)


@APP.command(name="publish-hf")
def publish_hf(
graph_configuration_file: Annotated[Path, cyclopts.Parameter(name=["--configuration-file", "-f"])],
repo_id: Annotated[str, cyclopts.Parameter(name=["--repo-id", "-r"])],
private: Annotated[bool, cyclopts.Parameter(name=["--private"], negative="")] = False,
dry_run: Annotated[bool, cyclopts.Parameter(name=["--dry-run"], negative="")] = False,
) -> None:
"""Validate and publish built KGX outputs and RIG metadata as a Hugging Face dataset.

Args:
graph_configuration_file: Path to the Graph YAML configuration file.
repo_id: Target Hugging Face dataset repository ID (e.g. ``org/my-kg``).
private: Create the dataset repository as private if it does not exist.
dry_run: Print the upload plan without contacting the Hugging Face Hub.
"""
from tempfile import TemporaryDirectory

extras.require("hf", required_by="publish-hf")
from tablassert import biolink, ingests
from tablassert.models import Graph

raw: object = ingests.from_yaml(graph_configuration_file)
g: Graph = Graph.model_validate(raw)
stem: str = f"{g.name}_{g.version}"
rig = getattr(g, "rig", None)
configured_dir: Path = g.artifact_path if hasattr(g, "artifact_path") else (rig.artifact_base_path if rig is not None else Path(".")) # pyright: ignore[reportAttributeAccessIssue]
base_dir: Path = configured_dir if (configured_dir / f"{stem}.nodes.ndjson").is_file() else Path(".")
nodes_path: Path = base_dir / f"{stem}.nodes.ndjson"
edges_path: Path = base_dir / f"{stem}.edges.ndjson"
rig_path: Path = base_dir / f"{stem}.RIG.yaml"

if not nodes_path.is_file() or not edges_path.is_file():
print(f"tablassert publish-hf: built KGX artifacts not found ({nodes_path}, {edges_path}).", file=sys.stderr)
raise SystemExit(1)

report: dict[str, Any] = biolink.validate_kgx(nodes_path, edges_path)
if not report.get("ok", report["nodes"]["failures"] == 0 and report["edges"]["failures"] == 0):
print(
f"tablassert publish-hf: refusing to publish invalid KGX "
f"(nodes failures={report['nodes']['failures']}, edges failures={report['edges']['failures']}).",
file=sys.stderr,
)
raise SystemExit(1)

staged_files: list[str] = ["README.md", "nodes/data.parquet", "edges/data.parquet", nodes_path.name, edges_path.name]
if rig_path.is_file():
staged_files.append(rig_path.name)

if dry_run:
print(f"Dry run for Hugging Face dataset {repo_id} (private={private}):")
for item in staged_files:
print(f" - {item}")
return

from huggingface_hub import HfApi # pyright: ignore[reportMissingImports]

with TemporaryDirectory() as tmp:
stage: Path = Path(tmp)
(stage / "nodes").mkdir()
(stage / "edges").mkdir()
pl.read_ndjson(nodes_path).write_parquet(stage / "nodes" / "data.parquet")
pl.read_ndjson(edges_path).write_parquet(stage / "edges" / "data.parquet")
(stage / nodes_path.name).write_bytes(nodes_path.read_bytes())
(stage / edges_path.name).write_bytes(edges_path.read_bytes())
if rig_path.is_file():
(stage / rig_path.name).write_bytes(rig_path.read_bytes())
(stage / "README.md").write_text(_hf_dataset_card(g), encoding="utf-8")

api = HfApi()
api.create_repo(repo_id=repo_id, repo_type="dataset", private=private, exist_ok=True)
api.upload_folder(folder_path=str(stage), repo_id=repo_id, repo_type="dataset")
print(f"Published {g.name} ({g.version}) to Hugging Face dataset {repo_id}.", file=sys.stderr)


def _print_kgx_section(label: str, section: dict[str, Any], path: Path) -> None:
"""Print one ``validate-kgx`` per-file section (shared by the plain and ``--prune`` paths)."""
if section["missing"]:
Expand Down
2 changes: 2 additions & 0 deletions src/tablassert/extras.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@
"aria2": {"aria2c": "aria2"},
"qc": {"sklearn": "scikit-learn", "sentence_transformers": "sentence-transformers"},
"log": {"loguru": "loguru"},
"hf": {"huggingface_hub": "huggingface-hub"},
}

# Import name -> owning extra, derived so the two can never drift apart.
Expand All @@ -46,6 +47,7 @@
"aria2": "the bundled aria2c downloader (used automatically by build-fullmap when installed)",
"qc": "the QC audit",
"log": "file and progress logging",
"hf": "Hugging Face Hub dataset export",
}


Expand Down
2 changes: 2 additions & 0 deletions tests/test_docs_cli_coverage.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,15 @@
COMMAND_NAME_DOCS: dict[str, tuple[str, ...]] = {
"build-fullmap": ("cli.md", "fullmap.md"),
"build-kg": ("cli.md",),
"publish-hf": ("cli.md",),
"quick-map": ("cli.md", "fullmap.md"),
"validate": ("cli.md",),
"validate-kgx": ("cli.md",),
}
COMMAND_FLAG_DOCS: dict[str, tuple[str, ...]] = {
"build-fullmap": ("cli.md", "fullmap.md"),
"build-kg": ("cli.md",),
"publish-hf": ("cli.md",),
"quick-map": ("cli.md", "fullmap.md"),
"validate": ("cli.md",),
"validate-kgx": ("cli.md",),
Expand Down
3 changes: 2 additions & 1 deletion tests/test_extras.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ def test_registry_covers_every_declared_extra() -> None:
assert set(extras.EXTRA_PACKAGES) == declared - {"rt"}


@pytest.mark.parametrize("extra", ["cli", "qc", "aria2", "log"])
@pytest.mark.parametrize("extra", ["cli", "qc", "aria2", "log", "hf"])
def test_registered_packages_are_actually_shipped_by_their_extra(extra: str) -> None:
"""Each import name maps to a distribution that its extra really installs.

Expand All @@ -65,6 +65,7 @@ def test_module_lookup_is_derived_and_unambiguous() -> None:
assert extras.EXTRA_FOR_MODULE["sklearn"] == "qc"
assert extras.EXTRA_FOR_MODULE["sentence_transformers"] == "qc"
assert extras.EXTRA_FOR_MODULE["loguru"] == "log"
assert extras.EXTRA_FOR_MODULE["huggingface_hub"] == "hf"
assert len(extras.EXTRA_FOR_MODULE) == sum(len(packages) for packages in extras.EXTRA_PACKAGES.values())


Expand Down
Loading