Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ concurrency:
env:
# The Python-syncing jobs install this EXACT set, so they share one warm uv cache instead
# of racing to save mutually-useless ones. See the `ci` group in pyproject.toml.
SYNC_ARGS: --no-default-groups --group ci --extra cli --extra qc --extra log
SYNC_ARGS: --no-default-groups --group ci --extra cli --extra qc --extra log --extra server

jobs:
# Release gate: a pyproject.toml push that does not change project.version (a dependency
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/security.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ concurrency:

env:
# Keep in sync with SYNC_ARGS in ci.yml so this workflow restores the same warm uv cache.
SYNC_ARGS: --no-default-groups --group ci --extra cli --extra qc --extra log
SYNC_ARGS: --no-default-groups --group ci --extra cli --extra qc --extra log --extra server

jobs:
secrets:
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ All notable changes to this project are documented in this file.

### Added

- **Localhost Web UI (`tablassert serve`) behind the optional `[server]` extra.** Installs `fastapi` and `uvicorn`, binds strictly to loopback (`127.0.0.1` by default), walks users through source selection, column mapping, provenance, and RIG metadata, validates inline against `Section`/`Graph`, emits plain editable YAML, and streams `build-kg` progress. ([#261](https://github.com/SkyeAv/Tablassert/issues/261))
- **NDJSON (JSONL) is a fifth table source kind.** `kind: ndjson` scans newline-delimited JSON lazily with `pl.scan_ndjson` and renames its columns positionally to `column_1..` in first-seen key order, matching the `parquet` reader contract so `method: column` encodings keep using Excel-style letters. Ragged records scan missing keys as nulls instead of dropping rows; an empty file fails fast with a coded `ndjson-empty` error instead of polars' opaque `Cannot infer NDJSON types` compute error. ([#283](https://github.com/SkyeAv/Tablassert/pull/283), [#262](https://github.com/SkyeAv/Tablassert/issues/262))
- **Every pull request must carry a new changelog entry.** A stdlib-only `scripts/check_changelog.py` requires at least one new bullet under the `## Unreleased` heading of `CHANGELOG.md` relative to the pull request's merge base, failing with remediation guidance when whitespace, subheadings, or released-history edits are the only additions; a CI job runs it on every pull request including forks, release-bump and dependency-bump pull requests are exempt by their `chore(release):` / `chore(deps):` title or commit subject, documentation-only pull requests (every changed path is markdown or under `docs/`) are exempt via `--exempt-docs-only`, and a pre-push hook mirrors the same check locally. ([#284](https://github.com/SkyeAv/Tablassert/pull/284), [#282](https://github.com/SkyeAv/Tablassert/issues/282))
- **Arrow IPC (Feather v2) is a fourth table source kind.** `kind: ipc` scans a `.arrow` file lazily with `pl.scan_ipc` and renames its columns positionally to `column_1..` in schema order, matching the `parquet` reader contract so `method: column` encodings keep using Excel-style letters. Only IPC file layout is supported; a stream-layout file fails fast with a coded `ipc-stream-unsupported` error naming the conversion fix instead of polars' opaque `InvalidFooter` compute error. ([#281](https://github.com/SkyeAv/Tablassert/pull/281), [#264](https://github.com/SkyeAv/Tablassert/issues/264))
Expand Down
8 changes: 4 additions & 4 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,12 @@ Thank you for helping make Tablassert reliable. Reliability here includes the de
```bash
git clone https://github.com/SkyeAv/Tablassert.git
cd Tablassert
uv sync --group dev --extra cli --extra qc --extra log
uv sync --group dev --extra cli --extra qc --extra log --extra server
uv run maturin develop --manifest-path rust/Cargo.toml
uv run tablassert --help
```

`uv sync --group dev --extra cli --extra qc --extra log` installs the development tools plus the CLI runtime, optional QC runtime, and loguru logging. `maturin develop` builds the PyO3 extension from `rust/` and installs it into the uv-managed environment.
`uv sync --group dev --extra cli --extra qc --extra log --extra server` installs the development tools plus the CLI runtime, optional QC runtime, and loguru logging. `maturin develop` builds the PyO3 extension from `rust/` and installs it into the uv-managed environment.

Shortcut:

Expand Down Expand Up @@ -49,7 +49,7 @@ The `Makefile` is intentionally small and mirrors the underlying commands:

| Target | Runs |
| --- | --- |
| `make setup` | `uv sync --group dev --extra cli --extra qc --extra log` and debug `maturin develop` |
| `make setup` | `uv sync --group dev --extra cli --extra qc --extra log --extra server` and debug `maturin develop` |
| `make dev` | Debug editable extension build |
| `make build` | Release editable extension build |
| `make test` | Python tests with coverage |
Expand Down Expand Up @@ -118,7 +118,7 @@ What the gates cover:

- **Ruff linting and formatting.** The current tree enforces core pycodestyle/pyflakes safety checks plus stale-suppression detection. It also enforces an expanded rule set covering common bug patterns (bugbear), simplifications, Python-version upgrades, pytest style, import order, and comprehensions. Treat `uv run ruff check .` and `uv run ruff format --check .` as the stable interface rather than relying on individual rule codes.
- **Pyright.** Type checking runs through `uv run pyright`; the project is tightening this as a strict-inference ratchet over time.
- **Python tests.** The suite is offline and runs in parallel by default via [pytest-xdist](https://pypi.org/project/pytest-xdist/) (`-n auto` in `pyproject.toml`'s addopts, which also enable `--cov` so coverage is reported inline); Disable parallelism for a single serial run with `pytest -n 0`. CI installs the `ci` dependency group with the `cli`, `qc`, and `log` extras (`--no-default-groups --group ci --extra cli --extra qc --extra log`) and runs the suite as a single job rather than sharding it across a matrix of runners: most of a run is fixed overhead (interpreter start, imports, xdist worker spin-up, coverage init) rather than test execution, so splitting the suite would cost more in per-runner setup than it recovers.
- **Python tests.** The suite is offline and runs in parallel by default via [pytest-xdist](https://pypi.org/project/pytest-xdist/) (`-n auto` in `pyproject.toml`'s addopts, which also enable `--cov` so coverage is reported inline); Disable parallelism for a single serial run with `pytest -n 0`. CI installs the `ci` dependency group with the `cli`, `qc`, `log`, and `server` extras (`--no-default-groups --group ci --extra cli --extra qc --extra log --extra server`) and runs the suite as a single job rather than sharding it across a matrix of runners: most of a run is fixed overhead (interpreter start, imports, xdist worker spin-up, coverage init) rather than test execution, so splitting the suite would cost more in per-runner setup than it recovers.
- **Rust tests.** `cargo test --manifest-path rust/Cargo.toml` runs the extension's Rust unit tests.
- **Rust style and lints.** `cargo fmt --check` enforces formatting; clippy runs all targets with warnings denied.
- **Dead code.** `uv run vulture` finds unused Python functions, methods, classes, and attributes that ruff and pyright cannot see; its scope and suppressions live in `[tool.vulture]` and `scripts/vulture_whitelist.py` (names used dynamically, like config-vocabulary enum members and pydantic fields). `uv run deptry .` finds unused declared dependencies and imports that only work through a transitive dependency; its ignores live in `[tool.deptry]`. On the Rust side, clippy already denies `dead_code`, and cargo-machete (CI `rust` job and pre-push hook, no local install required) rejects unused crate dependencies.
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

# Install Python dev dependencies, CLI/QC/log extras, and the editable Rust extension.
setup:
uv sync --group dev --extra cli --extra qc --extra log
uv sync --group dev --extra cli --extra qc --extra log --extra server
uv run maturin develop --manifest-path rust/Cargo.toml

# Build and install the Rust extension in release mode for local performance checks.
Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,7 @@ Python API to your project. Other extras are opt-in:
| `aria2` | bundled aria2c downloader, used automatically by `build-fullmap` when installed (Linux/Windows wheels only) | `pip install "tablassert[aria2]"` |
| `qc` | four-stage QC audit (exact match, fuzzy match, abbreviation expansion, and SapBERT embeddings) | `pip install "tablassert[qc]"` |
| `log` | loguru-backed file/progress logging (rotation, enqueue) | `pip install "tablassert[log]"` |
| `server` | localhost web UI for YAML-free config authoring and builds (`tablassert serve`) | `pip install "tablassert[server]"` |

Reaching a feature whose extra is missing never produces a bare `ModuleNotFoundError`: the failure
names the missing package and the install command that fixes it. QC is opt-in at build time
Expand Down Expand Up @@ -191,7 +192,7 @@ no LazyFrame setup or NLP preprocessing required. See the
## Developing

```bash
uv sync --group dev --extra cli --extra qc --extra log
uv sync --group dev --extra cli --extra qc --extra log --extra server
uv run maturin develop --manifest-path rust/Cargo.toml
make check
```
Expand Down
21 changes: 20 additions & 1 deletion docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Tablassert extracts knowledge assertions from tabular data into KGX NDJSON. The `tablassert` app
requires the `[cli]` extra (`pip install "tablassert[cli]"` or `uv tool install "tablassert[cli]"`) and
exposes **five subcommands**: `build-fullmap`, `build-kg`, `quick-map`, `validate`,
exposes **six subcommands**: `build-fullmap`, `build-kg`, `quick-map`, `serve`, `validate`,
and `validate-kgx`, plus an app-level `--version` flag. Run `tablassert --help` (or `<command> --help`)
for the live surface.

Expand All @@ -13,6 +13,7 @@ for the live surface.
| [`build-fullmap`](#build-fullmap) | Build the embedded fullmap redb used for entity resolution |
| [`build-kg`](#build-kg) | Build a KGX NDJSON knowledge graph from a YAML configuration |
| [`quick-map`](#quick-map) | Show what fullmap entity resolution does with one or more terms |
| [`serve`](#serve) | Start the localhost-only Web UI for YAML-free config authoring and builds |
| [`validate`](#validate) | Validate a graph or table configuration without executing it |
| [`validate-kgx`](#validate-kgx) | Validate built KGX NDJSON against the Biolink Model and the Translator registry |

Expand Down Expand Up @@ -357,6 +358,24 @@ once a model release drops the requirement.

---

## serve

Start a localhost-only Web UI (`[server]` extra) to author `tables/table.yaml` and `graph.yaml`
without writing YAML by hand, run inline `validate`, and stream `build-kg`.

```bash
tablassert serve
tablassert serve --port 9000 --output-dir ./my-project
```

| Option | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `--host` | String | No | `127.0.0.1` | Loopback interface to bind (`127.0.0.1`, `localhost`, or `::1`) |
| `--port`, `-p` | Integer | No | `8765` | TCP port to listen on |
| `--output-dir`, `-o` | Path | No | `.` | Directory where generated `graph.yaml` and `tables/table.yaml` are written |

---

## Typical workflow

1. Author a table config, then a graph config that references it.
Expand Down
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ expectations.

### How-to guides

- **[CLI Reference](cli.md)**: every command and flag: `build-kg`, `build-fullmap`, `validate`,
- **[CLI Reference](cli.md)**: every command and flag: `build-kg`, `build-fullmap`, `serve`, `validate`,
`validate-kgx`, `quick-map`
- **[Fullmap](fullmap.md)**: obtain and build the entity-resolution database

Expand Down
4 changes: 3 additions & 1 deletion docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ uv tool install "tablassert[cli]" # or: pip install "tablassert[cli]"
| `aria2` | Bundled aria2c downloader, used automatically by `build-fullmap` when installed (Linux/Windows wheels only) | `aria2==0.0.1b0` (imports as `aria2c`, bundles aria2c) |
| `qc` | QC runtime (exact match, fuzzy match, abbreviation expansion, then a SapBERT embedding audit) | `scikit-learn`, `sentence-transformers` (`torch` + `numpy` arrive transitively; `rapidfuzz` is a core dependency) |
| `log` | loguru-backed file/progress logging (rotation, enqueue) | `loguru` |
| `server` | Localhost web UI for YAML-free config authoring and builds (`tablassert serve`) | `fastapi`, `uvicorn` |

Install any extra the same way: `uv tool install "tablassert[<extra>]"` or
`pip install "tablassert[<extra>]"`, and combine them as `"tablassert[cli,qc]"`. Wheels ship for
Expand Down Expand Up @@ -72,6 +73,7 @@ fails, not a failure report):
| `tablassert` | `[cli]` | Before importing the command application, so a base Python-API install reports the exact install command instead of a bare module error |
| `build-kg --qc` | `[qc]` | Before the build starts: the QC audit runs at the very end of the build, so a late failure would cost the entire entity-resolution pass |
| `build-fullmap` | `[aria2]` | Before the first download, to pick the downloader: bundled aria2c when the extra is installed, Python downloader otherwise (announced on stderr and logged either way; never a missing-extra failure) |
| `serve` | `[server]` | Before starting the localhost web server |

A partially installed extra names every package it is still missing, so installing them is one step
rather than a retry loop. Library calls that reach an optional import directly (for example
Expand Down Expand Up @@ -101,7 +103,7 @@ Editable install plus the development tools; needs `uv` and a Rust toolchain.
```bash
git clone https://github.com/SkyeAv/Tablassert.git
cd Tablassert
uv sync --group dev --extra cli --extra qc --extra log
uv sync --group dev --extra cli --extra qc --extra log --extra server
uv run maturin develop --manifest-path rust/Cargo.toml
uv run tablassert --help
```
Expand Down
7 changes: 4 additions & 3 deletions llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ Current CLI behavior queries an embedded redb database whose primary file is `fu
## Quickstart
- [README](README.md): high-level overview, install snippets, and one-command graph build.
- [Documentation Site Home](docs/index.md): docs landing page and navigation context.
- [Installation Guide](docs/installation.md): three install paths (PyPI, GitHub main, development from source), platform and Rust-toolchain notes, and the missing-extra behavior. Optional extras: `cli`, `rt`, `aria2`, `qc`, and `log`.
- [CLI Reference](docs/cli.md): the five commands (`build-fullmap`, `build-kg`, `quick-map`, `validate`, and `validate-kgx`) plus the app-level `--version` flag.
- [Installation Guide](docs/installation.md): three install paths (PyPI, GitHub main, development from source), platform and Rust-toolchain notes, and the missing-extra behavior. Optional extras: `cli`, `rt`, `aria2`, `qc`, `log`, and `server`.
- [CLI Reference](docs/cli.md): the six commands (`build-fullmap`, `build-kg`, `quick-map`, `serve`, `validate`, and `validate-kgx`) plus the app-level `--version` flag.
- [Tutorial](docs/tutorial.md): first end-to-end run from CSV input to KGX NDJSON output.

## YAML Authoring
Expand All @@ -35,7 +35,7 @@ Current CLI behavior queries an embedded redb database whose primary file is `fu
## Contributor Development
- [Contributing Guide](CONTRIBUTING.md): setup, style conventions, checks, and PR expectations.
- [Development](docs/development.md): docs-site contributor setup and the local `uv run mkdocs build --strict` docs gate.
- [Project Metadata](pyproject.toml): dependencies, tooling config, and CLI entrypoint. Optional extras: `cli`, `rt`, `aria2`, `qc`, and `log`.
- [Project Metadata](pyproject.toml): dependencies, tooling config, and CLI entrypoint. Optional extras: `cli`, `rt`, `aria2`, `qc`, `log`, and `server`.
- [Tests](tests/): pytest coverage and fixtures.
- [MkDocs Configuration](mkdocs.yml): docs navigation and published pages.
- [GitHub Workflows](.github/workflows/): `ci.yml` runs lint, format, type, dead-code (vulture, deptry), test, rust (clippy, machete), and mkdocs checks when a release changes the project version or on manual dispatch; `docs.yml` builds and deploys MkDocs; `pipy.yml` checks the project version, builds sdist and wheel artifacts, and publishes to PyPI; `tag-version.yml` tags changed project versions; `paper.yml` tests the JOSS paper compile and uploads the PDF artifact; `security.yml` scans commits for leaked secrets with gitleaks and audits dependencies against advisory databases with pip-audit and cargo-audit for the Rust lockfile.
Expand All @@ -59,6 +59,7 @@ Current CLI behavior queries an embedded redb database whose primary file is `fu
- [Quality Control](src/tablassert/qc.py): exact, fuzzy, abbreviation, and SapBERT validation stages.
- [RIG Helpers](src/tablassert/rig.py): Resource Ingest Guide normalization and serialization helpers.
- [Run Cache](src/tablassert/runcache.py): content-addressed op digests, worth-caching guards, the shared-prefix planner, and the ephemeral run cache.
- [Localhost Web UI Server](src/tablassert/server.py): FastAPI/Uvicorn localhost-only web UI (`tablassert serve`) for YAML-free config authoring, validation, and build streaming.
- [Graph Study](src/tablassert/study.py): assertions over final KGX NDJSON.
- [Utilities](src/tablassert/utils.py): hashing, storage path constants, and filesystem helpers.
- [Package Init](src/tablassert/__init__.py): public package import surface and compiled-extension bootstrap.
Expand Down
4 changes: 4 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,10 @@ qc = [
log = [
"loguru>=0.7.3",
]
server = [
"fastapi>=0.115.0",
"uvicorn>=0.34.0",
]

[dependency-groups]
test = [
Expand Down
4 changes: 4 additions & 0 deletions scripts/vulture_whitelist.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@

from typing import Any

from tablassert.cli import serve
from tablassert.enums import ContentCategories, DataFormats, FillMethods, IngestCategories, ModelingCategories, ProvisionMechanisms, SourceStatuses
from tablassert.models import (
RIGFilteredContent,
Expand Down Expand Up @@ -105,3 +106,6 @@
RIGFutureModelingConsideration.consideration
RIGSupportingDataSourceInfo.terms_of_use_info
RIGProvenanceInfo.artifacts

# cli.py: Cyclopts subcommand registered by @APP.command decorator.
serve
26 changes: 26 additions & 0 deletions src/tablassert/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -987,6 +987,32 @@ def validate(
run(3, validate_pipeline, configuration_file)


@APP.command(name="serve")
def serve(
host: Annotated[str, cyclopts.Parameter(name=["--host"])] = "127.0.0.1",
port: Annotated[int, cyclopts.Parameter(name=["--port", "-p"])] = 8765,
output_dir: Annotated[Path, cyclopts.Parameter(name=["--output-dir", "-o"])] = Path("."),
) -> None:
"""Start the localhost-only Web UI for YAML-free config authoring and builds.

Requires the ``[server]`` extra (``fastapi`` and ``uvicorn``). Binds to localhost
only (``127.0.0.1``, ``localhost``, or ``::1``) with no remote exposure.

Args:
host: Loopback interface to bind (default: ``127.0.0.1``).
port: TCP port to listen on (default: ``8765``).
output_dir: Directory where generated ``graph.yaml`` and ``tables/table.yaml`` are written.
"""
extras.require("server", required_by="tablassert serve")
if host not in {"127.0.0.1", "localhost", "::1"}:
print(f"tablassert serve: refusing non-localhost host {host!r}; bind to 127.0.0.1, localhost, or ::1.", file=sys.stderr)
raise SystemExit(1)

from tablassert.server import run_server

run_server(host=host, port=port, output_dir=output_dir)


def _print_kgx_section(label: str, section: dict[str, Any], path: Path) -> None:
"""Print one ``validate-kgx`` per-file section (shared by the plain and ``--prune`` paths)."""
if section["missing"]:
Expand Down
2 changes: 2 additions & 0 deletions src/tablassert/extras.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@
"aria2": {"aria2c": "aria2"},
"qc": {"sklearn": "scikit-learn", "sentence_transformers": "sentence-transformers"},
"log": {"loguru": "loguru"},
"server": {"fastapi": "fastapi", "uvicorn": "uvicorn"},
}

# Import name -> owning extra, derived so the two can never drift apart.
Expand All @@ -46,6 +47,7 @@
"aria2": "the bundled aria2c downloader (used automatically by build-fullmap when installed)",
"qc": "the QC audit",
"log": "file and progress logging",
"server": "the localhost web UI (`tablassert serve`)",
}


Expand Down
Loading