Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ concurrency:
env:
# The Python-syncing jobs install this EXACT set, so they share one warm uv cache instead
# of racing to save mutually-useless ones. See the `ci` group in pyproject.toml.
SYNC_ARGS: --no-default-groups --group ci --extra cli --extra qc --extra log
SYNC_ARGS: --no-default-groups --group ci --extra cli --extra qc --extra log --extra excel

jobs:
# Release gate: a pyproject.toml push that does not change project.version (a dependency
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/security.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ concurrency:

env:
# Keep in sync with SYNC_ARGS in ci.yml so this workflow restores the same warm uv cache.
SYNC_ARGS: --no-default-groups --group ci --extra cli --extra qc --extra log
SYNC_ARGS: --no-default-groups --group ci --extra cli --extra qc --extra log --extra excel

jobs:
secrets:
Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Tablassert turns biomedical spreadsheets into KGX knowledge graphs for NCATS Tra

## Build / Test / Lint

- One-time setup: `make setup` (`uv sync --group dev --extra cli --extra qc --extra log`, then debug `maturin develop`). [verified]
- One-time setup: `make setup` (`uv sync --group dev --extra cli --extra qc --extra log --extra excel`, then debug `maturin develop`). [verified]
- After any Rust change, or when the extension may be stale: `make dev` (debug build; `make build` for release-mode performance checks only). [verified]
- Full local gate before a PR: `make check` (ruff lint + format check + pyright + vulture + deptry dead-code scans + pytest + cargo test + clippy -D warnings). [verified]
- Release version bump: `make patch` / `make minor` / `make major` bumps `pyproject.toml`, `uv.lock`, `CITATION.cff` and rolls the `CHANGELOG.md` Unreleased section (changelog prose stays manual). [verified]
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ All notable changes to this project are documented in this file.

### Changed

- **BREAKING: `fastexcel` moved from core dependencies to the optional `[excel]` extra.** `lib.excel()` preflights `extras.require("excel", required_by="Excel input")` before `pl.read_excel()` when `engine == "calamine"`, raising `MissingExtraError` with install commands instead of failing inside Polars; explicit non-calamine engines such as `openpyxl` remain usable without `[excel]`. ([#300](https://github.com/SkyeAv/Tablassert/issues/300))
- **CI runs only when a release is cut or on manual dispatch.** `.github/workflows/ci.yml` no longer triggers on pull requests; it runs on a push to main that changes `project.version` in `pyproject.toml` (the `chore(release):` commit, detected by the same `detect-pyproject-version-change` action the tag and PyPI workflows use) or via `workflow_dispatch`. The PR-only `changes` path filter, the push-back `autofix` job, the `python-smoke` subset, and the CI `changelog` job are removed; formatting is now a check-only `format` job, and the changelog requirement stays enforced by the pre-push hook. Run `make check` locally before opening a pull request.
- **The vulture allowlist lives in `scripts/` next to the other developer tooling.** `vulture_whitelist.py` moves to `scripts/vulture_whitelist.py` unchanged; the `[tool.vulture]` `paths` list drops the now-redundant root entry because `scripts` is already scanned, and the CI `python` path filter, the pre-commit hook comment, and the CONTRIBUTING dead-code section follow the new location. The gate is behaviorally identical: same vulture pin, same config, same allowlist content. ([#287](https://github.com/SkyeAv/Tablassert/pull/287))

Expand Down
8 changes: 4 additions & 4 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,12 @@ Thank you for helping make Tablassert reliable. Reliability here includes the de
```bash
git clone https://github.com/SkyeAv/Tablassert.git
cd Tablassert
uv sync --group dev --extra cli --extra qc --extra log
uv sync --group dev --extra cli --extra qc --extra log --extra excel
uv run maturin develop --manifest-path rust/Cargo.toml
uv run tablassert --help
```

`uv sync --group dev --extra cli --extra qc --extra log` installs the development tools plus the CLI runtime, optional QC runtime, and loguru logging. `maturin develop` builds the PyO3 extension from `rust/` and installs it into the uv-managed environment.
`uv sync --group dev --extra cli --extra qc --extra log --extra excel` installs the development tools plus the CLI runtime, optional QC runtime, and loguru logging. `maturin develop` builds the PyO3 extension from `rust/` and installs it into the uv-managed environment.

Shortcut:

Expand Down Expand Up @@ -49,7 +49,7 @@ The `Makefile` is intentionally small and mirrors the underlying commands:

| Target | Runs |
| --- | --- |
| `make setup` | `uv sync --group dev --extra cli --extra qc --extra log` and debug `maturin develop` |
| `make setup` | `uv sync --group dev --extra cli --extra qc --extra log --extra excel` and debug `maturin develop` |
| `make dev` | Debug editable extension build |
| `make build` | Release editable extension build |
| `make test` | Python tests with coverage |
Expand Down Expand Up @@ -118,7 +118,7 @@ What the gates cover:

- **Ruff linting and formatting.** The current tree enforces core pycodestyle/pyflakes safety checks plus stale-suppression detection. It also enforces an expanded rule set covering common bug patterns (bugbear), simplifications, Python-version upgrades, pytest style, import order, and comprehensions. Treat `uv run ruff check .` and `uv run ruff format --check .` as the stable interface rather than relying on individual rule codes.
- **Pyright.** Type checking runs through `uv run pyright`; the project is tightening this as a strict-inference ratchet over time.
- **Python tests.** The suite is offline and runs in parallel by default via [pytest-xdist](https://pypi.org/project/pytest-xdist/) (`-n auto` in `pyproject.toml`'s addopts, which also enable `--cov` so coverage is reported inline); Disable parallelism for a single serial run with `pytest -n 0`. CI installs the `ci` dependency group with the `cli`, `qc`, and `log` extras (`--no-default-groups --group ci --extra cli --extra qc --extra log`) and runs the suite as a single job rather than sharding it across a matrix of runners: most of a run is fixed overhead (interpreter start, imports, xdist worker spin-up, coverage init) rather than test execution, so splitting the suite would cost more in per-runner setup than it recovers.
- **Python tests.** The suite is offline and runs in parallel by default via [pytest-xdist](https://pypi.org/project/pytest-xdist/) (`-n auto` in `pyproject.toml`'s addopts, which also enable `--cov` so coverage is reported inline); Disable parallelism for a single serial run with `pytest -n 0`. CI installs the `ci` dependency group with the `cli`, `qc`, `log`, and `excel` extras (`--no-default-groups --group ci --extra cli --extra qc --extra log --extra excel`) and runs the suite as a single job rather than sharding it across a matrix of runners: most of a run is fixed overhead (interpreter start, imports, xdist worker spin-up, coverage init) rather than test execution, so splitting the suite would cost more in per-runner setup than it recovers.
- **Rust tests.** `cargo test --manifest-path rust/Cargo.toml` runs the extension's Rust unit tests.
- **Rust style and lints.** `cargo fmt --check` enforces formatting; clippy runs all targets with warnings denied.
- **Dead code.** `uv run vulture` finds unused Python functions, methods, classes, and attributes that ruff and pyright cannot see; its scope and suppressions live in `[tool.vulture]` and `scripts/vulture_whitelist.py` (names used dynamically, like config-vocabulary enum members and pydantic fields). `uv run deptry .` finds unused declared dependencies and imports that only work through a transitive dependency; its ignores live in `[tool.deptry]`. On the Rust side, clippy already denies `dead_code`, and cargo-machete (CI `rust` job and pre-push hook, no local install required) rejects unused crate dependencies.
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

# Install Python dev dependencies, CLI/QC/log extras, and the editable Rust extension.
setup:
uv sync --group dev --extra cli --extra qc --extra log
uv sync --group dev --extra cli --extra qc --extra log --extra excel
uv run maturin develop --manifest-path rust/Cargo.toml

# Build and install the Rust extension in release mode for local performance checks.
Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,7 @@ Python API to your project. Other extras are opt-in:
| `aria2` | bundled aria2c downloader, used automatically by `build-fullmap` when installed (Linux/Windows wheels only) | `pip install "tablassert[aria2]"` |
| `qc` | four-stage QC audit (exact match, fuzzy match, abbreviation expansion, and SapBERT embeddings) | `pip install "tablassert[qc]"` |
| `log` | loguru-backed file/progress logging (rotation, enqueue) | `pip install "tablassert[log]"` |
| `excel` | Excel (`.xlsx`) input via Polars' `calamine` engine | `pip install "tablassert[excel]"` |

Reaching a feature whose extra is missing never produces a bare `ModuleNotFoundError`: the failure
names the missing package and the install command that fixes it. QC is opt-in at build time
Expand Down Expand Up @@ -191,7 +192,7 @@ no LazyFrame setup or NLP preprocessing required. See the
## Developing

```bash
uv sync --group dev --extra cli --extra qc --extra log
uv sync --group dev --extra cli --extra qc --extra log --extra excel
uv run maturin develop --manifest-path rust/Cargo.toml
make check
```
Expand Down
5 changes: 5 additions & 0 deletions docs/configuration/table.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,11 @@ Defines the data file location and format.

#### Excel Source

> **Requires `[excel]` extra.** Reading `kind: excel` workbooks with the default `calamine`
> engine requires `fastexcel` (`pip install "tablassert[excel]"` or
> `uv tool install "tablassert[excel]"`). See [Optional Extras](../installation.md#optional-extras)
> for details.

| Field | Type | Required | Description |
| ------- | ------ | ---------- | ------------- |
| `kind` | String | No | Source kind. Model default is `"excel"`, but specify it explicitly in configs. |
Expand Down
2 changes: 1 addition & 1 deletion docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ run the test and docs gates locally, and lint/format. The canonical contributor
```bash
git clone https://github.com/SkyeAv/Tablassert.git
cd Tablassert
uv sync --group dev --extra cli --extra qc --extra log
uv sync --group dev --extra cli --extra qc --extra log --extra excel
uv run maturin develop --manifest-path rust/Cargo.toml
uv run tablassert --help
```
Expand Down
12 changes: 7 additions & 5 deletions docs/installation.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Installation

Install Tablassert's Python API, then add the `[cli]` extra to use the `tablassert` command and any of the `rt` / `aria2` / `qc` / `log` extras that match how you will use it (runtime compatibility, accelerated fullmap downloads, auditing mappings, or loguru-backed logging).
Install Tablassert's Python API, then add the `[cli]` extra to use the `tablassert` command and any of the `rt` / `aria2` / `qc` / `log` / `excel` extras that match how you will use it (runtime compatibility, accelerated fullmap downloads, auditing mappings, loguru-backed logging, or Excel input).

## Prerequisites

Expand Down Expand Up @@ -38,6 +38,7 @@ uv tool install "tablassert[cli]" # or: pip install "tablassert[cli]"
| `aria2` | Bundled aria2c downloader, used automatically by `build-fullmap` when installed (Linux/Windows wheels only) | `aria2==0.0.1b0` (imports as `aria2c`, bundles aria2c) |
| `qc` | QC runtime (exact match, fuzzy match, abbreviation expansion, then a SapBERT embedding audit) | `scikit-learn`, `sentence-transformers` (`torch` + `numpy` arrive transitively; `rapidfuzz` is a core dependency) |
| `log` | loguru-backed file/progress logging (rotation, enqueue) | `loguru` |
| `excel` | Excel (`.xlsx`) input via Polars' `calamine` engine | `fastexcel` |

Install any extra the same way: `uv tool install "tablassert[<extra>]"` or
`pip install "tablassert[<extra>]"`, and combine them as `"tablassert[cli,qc]"`. Wheels ship for
Expand All @@ -48,9 +49,9 @@ Linux and macOS; elsewhere pip builds from the sdist, which needs a Rust toolcha

The bundled aria2c dependency is GPL-2.0. Tablassert remains Apache-2.0 and does not vendor aria2c, but redistributors who ship the optional extra should review GPL-2.0 obligations.

Excel (`.xlsx`) input is read through Polars' `calamine` engine, which ships with the base install
(`fastexcel`). Workbooks that calamine cannot parse are rejected with its error; there is no fallback
engine.
Excel (`.xlsx`) input defaults to Polars' `calamine` engine, which requires the `[excel]` extra
(`fastexcel`). When `fastexcel` is absent, `lib.excel()` raises `MissingExtraError` before calling
`pl.read_excel()`. Passing an explicit non-calamine engine such as `openpyxl` does not require `[excel]`.

#### When an extra is missing

Expand All @@ -72,6 +73,7 @@ fails, not a failure report):
| `tablassert` | `[cli]` | Before importing the command application, so a base Python-API install reports the exact install command instead of a bare module error |
| `build-kg --qc` | `[qc]` | Before the build starts: the QC audit runs at the very end of the build, so a late failure would cost the entire entity-resolution pass |
| `build-fullmap` | `[aria2]` | Before the first download, to pick the downloader: bundled aria2c when the extra is installed, Python downloader otherwise (announced on stderr and logged either way; never a missing-extra failure) |
| `lib.excel()` | `[excel]` | Before `pl.read_excel()` when `engine == "calamine"` (explicit non-calamine engines such as `openpyxl` remain independent) |

A partially installed extra names every package it is still missing, so installing them is one step
rather than a retry loop. Library calls that reach an optional import directly (for example
Expand Down Expand Up @@ -101,7 +103,7 @@ Editable install plus the development tools; needs `uv` and a Rust toolchain.
```bash
git clone https://github.com/SkyeAv/Tablassert.git
cd Tablassert
uv sync --group dev --extra cli --extra qc --extra log
uv sync --group dev --extra cli --extra qc --extra log --extra excel
uv run maturin develop --manifest-path rust/Cargo.toml
uv run tablassert --help
```
Expand Down
4 changes: 2 additions & 2 deletions llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Current CLI behavior queries an embedded redb database whose primary file is `fu
## Quickstart
- [README](README.md): high-level overview, install snippets, and one-command graph build.
- [Documentation Site Home](docs/index.md): docs landing page and navigation context.
- [Installation Guide](docs/installation.md): three install paths (PyPI, GitHub main, development from source), platform and Rust-toolchain notes, and the missing-extra behavior. Optional extras: `cli`, `rt`, `aria2`, `qc`, and `log`.
- [Installation Guide](docs/installation.md): three install paths (PyPI, GitHub main, development from source), platform and Rust-toolchain notes, and the missing-extra behavior. Optional extras: `cli`, `rt`, `aria2`, `qc`, `log`, and `excel`.
- [CLI Reference](docs/cli.md): the five commands (`build-fullmap`, `build-kg`, `quick-map`, `validate`, and `validate-kgx`) plus the app-level `--version` flag.
- [Tutorial](docs/tutorial.md): first end-to-end run from CSV input to KGX NDJSON output.

Expand All @@ -35,7 +35,7 @@ Current CLI behavior queries an embedded redb database whose primary file is `fu
## Contributor Development
- [Contributing Guide](CONTRIBUTING.md): setup, style conventions, checks, and PR expectations.
- [Development](docs/development.md): docs-site contributor setup and the local `uv run mkdocs build --strict` docs gate.
- [Project Metadata](pyproject.toml): dependencies, tooling config, and CLI entrypoint. Optional extras: `cli`, `rt`, `aria2`, `qc`, and `log`.
- [Project Metadata](pyproject.toml): dependencies, tooling config, and CLI entrypoint. Optional extras: `cli`, `rt`, `aria2`, `qc`, `log`, and `excel`.
- [Tests](tests/): pytest coverage and fixtures.
- [MkDocs Configuration](mkdocs.yml): docs navigation and published pages.
- [GitHub Workflows](.github/workflows/): `ci.yml` runs lint, format, type, dead-code (vulture, deptry), test, rust (clippy, machete), and mkdocs checks when a release changes the project version or on manual dispatch; `docs.yml` builds and deploys MkDocs; `pipy.yml` checks the project version, builds sdist and wheel artifacts, and publishes to PyPI; `tag-version.yml` tags changed project versions; `paper.yml` tests the JOSS paper compile and uploads the PDF artifact; `security.yml` scans commits for leaked secrets with gitleaks and audits dependencies against advisory databases with pip-audit and cargo-audit for the Rust lockfile.
Expand Down
4 changes: 3 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,6 @@ dependencies = [
"rapidfuzz>=3.14.3",
"pydantic>=2.12.5",
"pyyaml>=6.0.3",
"fastexcel>=0.21.0",
]

[project.urls]
Expand Down Expand Up @@ -98,6 +97,9 @@ qc = [
log = [
"loguru>=0.7.3",
]
excel = [
"fastexcel>=0.21.0",
]

[dependency-groups]
test = [
Expand Down
2 changes: 2 additions & 0 deletions src/tablassert/extras.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@
"aria2": {"aria2c": "aria2"},
"qc": {"sklearn": "scikit-learn", "sentence_transformers": "sentence-transformers"},
"log": {"loguru": "loguru"},
"excel": {"fastexcel": "fastexcel"},
}

# Import name -> owning extra, derived so the two can never drift apart.
Expand All @@ -46,6 +47,7 @@
"aria2": "the bundled aria2c downloader (used automatically by build-fullmap when installed)",
"qc": "the QC audit",
"log": "file and progress logging",
"excel": "Excel input",
}


Expand Down
8 changes: 8 additions & 0 deletions src/tablassert/lib.py
Original file line number Diff line number Diff line change
Expand Up @@ -949,11 +949,19 @@ def excel(p: Path, sheet: str, engine: str = "calamine") -> pl.LazyFrame:
Returns:
LazyFrame over the sheet contents (no header inference).

Raises:
MissingExtraError: When ``engine == "calamine"`` and the ``[excel]`` extra
(``fastexcel``) is not installed.

Notes:
Headerless auto column names are remapped to the 1-based ``column_1``..
convention so existing table configs keep referencing the same names as
under polars 1.x (output parity).
"""
if engine == "calamine":
from tablassert.extras import require

require("excel", required_by="Excel input")
df: pl.DataFrame = pl.read_excel(
source=p,
sheet_name=sheet,
Expand Down
38 changes: 38 additions & 0 deletions tests/test_cover_lib.py
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,44 @@ def test_excel_reads_sheet_without_header(tmp_path: Path) -> None:
assert df.rows() == [("a", "b"), ("c", "d")]


def test_excel_missing_extra_raises_before_read_excel(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Default ``calamine`` Excel reads preflight ``[excel]`` before ``pl.read_excel``."""
from tablassert import extras
from tablassert.errors import MissingExtraError

real_find_spec = extras.find_spec

def _no_fastexcel(name: str, package: str | None = None) -> Any:
if name == "fastexcel":
return None
return real_find_spec(name, package)

def _fail_read_excel(*_args: Any, **_kwargs: Any) -> pl.DataFrame:
raise AssertionError("pl.read_excel must not be called when fastexcel is missing")

monkeypatch.setattr(extras, "find_spec", _no_fastexcel)
monkeypatch.setattr(pl, "read_excel", _fail_read_excel)

with pytest.raises(MissingExtraError) as exc_info:
lib.excel(tmp_path / "book.xlsx", "Sheet1")
message: str = str(exc_info.value)
assert "fastexcel" in message
assert 'pip install "tablassert[excel]"' in message
assert 'uv tool install "tablassert[excel]"' in message

wb: openpyxl.Workbook = openpyxl.Workbook()
ws = wb.active
assert ws is not None
ws.title = "Sheet1"
ws.append(["x", "y"])
xp: Path = tmp_path / "openpyxl.xlsx"
wb.save(xp)
monkeypatch.undo()
monkeypatch.setattr(extras, "find_spec", _no_fastexcel)
df: pl.DataFrame = lib.excel(xp, "Sheet1", engine="openpyxl").collect()
assert df.rows() == [("x", "y")]


def test_parquet_reads_schema_and_renames_columns_positionally(tmp_path: Path) -> None:
"""``parquet`` scans lazily and maps the file's real names to positional ``column_<n>``.

Expand Down
Loading