Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ All notable changes to this project are documented in this file.

### Changed

- **Profiled `build_graph_pipeline` and ported `p_value` scientific formatting to Rust (`rs.format_scientific_4e`).** Published the 50k-row release-build profile in `paper/benchmarks/pipeline_profile/README.md`, moved `lib.format_numeric()`'s per-row `"{:.4e}"` loop into `rust/src/lib.rs` with `rayon` parallelism (4.23x speedup on 100k floats: 22.80 ms to 5.39 ms median), and pinned byte-for-byte `{:.4e}` parity in `tests/test_format_numeric_golden.py`. ([#267](https://github.com/SkyeAv/Tablassert/issues/267))
- **CI runs only when a release is cut or on manual dispatch.** `.github/workflows/ci.yml` no longer triggers on pull requests; it runs on a push to main that changes `project.version` in `pyproject.toml` (the `chore(release):` commit, detected by the same `detect-pyproject-version-change` action the tag and PyPI workflows use) or via `workflow_dispatch`. The PR-only `changes` path filter, the push-back `autofix` job, the `python-smoke` subset, and the CI `changelog` job are removed; formatting is now a check-only `format` job, and the changelog requirement stays enforced by the pre-push hook. Run `make check` locally before opening a pull request.
- **The vulture allowlist lives in `scripts/` next to the other developer tooling.** `vulture_whitelist.py` moves to `scripts/vulture_whitelist.py` unchanged; the `[tool.vulture]` `paths` list drops the now-redundant root entry because `scripts` is already scanned, and the CI `python` path filter, the pre-commit hook comment, and the CONTRIBUTING dead-code section follow the new location. The gate is behaviorally identical: same vulture pin, same config, same allowlist content. ([#287](https://github.com/SkyeAv/Tablassert/pull/287))

Expand Down
30 changes: 30 additions & 0 deletions paper/benchmarks/pipeline_profile/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Pipeline Profiling and Rust Hot-Path Port (#267)

## Environment and Workload

- **Build**: release PyO3 extension (`make build`, `tablassert-22.3.3-cp313-cp313-macosx_11_0_arm64`)
- **Platform**: macOS arm64, Python 3.13.12
- **Workload**: 50,000-row gene-disease TSV with `p_value` + `study_size` coercion, 500-entity real `fullmap.redb`, and `uuid_fields: [subject, predicate, object]` with `uuid_on_collision: merge`

## Top Cost Centers (`cProfile`, `build_graph_pipeline` warm process)

Total wall time: **9.897s** (50,000 rows).

| Phase / Function | Time | Share | Notes |
| --- | ---: | ---: | --- |
| Stage 1 + Stage 2 (`multiprocessing.Pool` spawn + worker imports) | 7.92s | 80.0% | Fixed process-spawn overhead for 10 workers |
| `tablassert.rs.dedup_ndjson` | 0.980s | 9.9% | Already in Rust (`rust/src/ndjson.rs`) |
| `lib.compile_subgraph` (Polars + fullmap + formatting) | 0.783s | 7.9% | Includes per-row `lib.format_numeric` (`0.066s` at 50k) |
| `rig.rig_edge_type_info` / `runcache.canonical` / `biolink.class_fields` | 0.042s | 0.4% | Metadata and cache keying |

## Ported Hot Path: `format_scientific_4e`

`lib.format_numeric()` formatted `p_value` columns via a per-row Python comprehension (`[None if v is None else f"{v:.4e}" for v in df[c].to_list()]`).
Moving this formatting loop into `tablassert.rs.format_scientific_4e` (single pre-sized buffer per element + `rayon` parallelization via `py.detach` for batches >= 10,000 rows) preserves byte-identical Python `{:.4e}` output (0 diffs across 10,014 edge and random values).

### Before / After Benchmark (100,000 `p_value` floats, 9 runs)

| Implementation | Median | Stdev | Speedup |
| --- | ---: | ---: | ---: |
| Python `[f"{v:.4e}" ...]` | 22.80 ms | 0.41 ms | 1.00x |
| Rust `rs.format_scientific_4e` | 5.39 ms | 0.38 ms | **4.23x** |
52 changes: 52 additions & 0 deletions rust/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,43 @@ fn xxh64_file(path: PathBuf) -> PyResult<String> {
Ok(format!("{:016x}", hasher.digest()))
}

fn format_py_4e(value: f64) -> String {
use std::fmt::Write as _;

if value.is_nan() {
return "nan".to_string();
}
if value.is_infinite() {
return if value.is_sign_negative() {
"-inf".to_string()
} else {
"inf".to_string()
};
}
let mut raw = String::with_capacity(16);
let _ = write!(&mut raw, "{value:.4e}");
if let Some(e_pos) = raw.find('e') {
let exp: i32 = raw[e_pos + 1..].parse().unwrap_or(0);
raw.truncate(e_pos + 1);
let _ = write!(&mut raw, "{exp:+03}");
}
raw
}

/// Format a batch of optional floats using Python-compatible ``{:.4e}`` notation.
#[pyfunction]
fn format_scientific_4e(py: Python<'_>, values: Vec<Option<f64>>) -> Vec<Option<String>> {
use rayon::prelude::*;

py.detach(|| {
if values.len() >= 10_000 {
values.into_par_iter().map(|opt| opt.map(format_py_4e)).collect()
} else {
values.into_iter().map(|opt| opt.map(format_py_4e)).collect()
}
})
}

// Public re-exports of the fullmap `#[pyfunction]`s so Rust integration tests
// (`rust/tests/`) and other non-Python embedders can drive the exact production
// build/read path. The `fullmap` module itself stays private; only these
Expand Down Expand Up @@ -89,6 +126,7 @@ fn rs(module: &Bound<'_, PyModule>) -> PyResult<()> {
module.add_function(wrap_pyfunction!(uuid::namespace_uuid, module)?)?;
module.add_function(wrap_pyfunction!(xxh64, module)?)?;
module.add_function(wrap_pyfunction!(xxh64_file, module)?)?;
module.add_function(wrap_pyfunction!(format_scientific_4e, module)?)?;
Ok(())
}

Expand Down Expand Up @@ -119,6 +157,20 @@ mod tests {
assert_eq!(xxh64_file(empty).unwrap(), "ef46db3751d8e999");
}

#[test]
fn format_py_4e_matches_python_contract() {
use super::format_py_4e;

assert_eq!(format_py_4e(0.0), "0.0000e+00");
assert_eq!(format_py_4e(-0.0), "-0.0000e+00");
assert_eq!(format_py_4e(1e-3), "1.0000e-03");
assert_eq!(format_py_4e(0.05), "5.0000e-02");
assert_eq!(format_py_4e(1e100), "1.0000e+100");
assert_eq!(format_py_4e(f64::NAN), "nan");
assert_eq!(format_py_4e(f64::INFINITY), "inf");
assert_eq!(format_py_4e(f64::NEG_INFINITY), "-inf");
}

#[test]
fn xxh64_file_reports_io_failures_as_os_error() {
// WHY: callers must distinguish I/O failures from valid digests and see
Expand Down
5 changes: 4 additions & 1 deletion src/tablassert/lib.py
Original file line number Diff line number Diff line change
Expand Up @@ -796,7 +796,10 @@ def format_numeric(lf: pl.LazyFrame) -> pl.LazyFrame:
df = df.with_columns(pl.when(valid_count).then(number.cast(pl.Int64, strict=False)).otherwise(None).alias(c))
continue
fmt = "{:.4g}"
formatted: list[str | None] = [None if v is None else fmt.format(v) for v in df[c].to_list()]
if fmt == "{:.4e}":
formatted: list[str | None] = rs.format_scientific_4e(df[c].to_list())
else:
formatted = [None if v is None else fmt.format(v) for v in df[c].to_list()]
df = df.with_columns(pl.Series(c, formatted))
return df.lazy()

Expand Down
1 change: 1 addition & 0 deletions src/tablassert/rs.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,4 @@ def normalize_terms_l2(terms: list[str]) -> list[str]: ...
def taxon_allowlist_identity(taxon_allowlist: list[int]) -> str: ...
def xxh64(data: str) -> str: ...
def xxh64_file(path: str) -> str: ...
def format_scientific_4e(values: list[float | None]) -> list[str | None]: ...
25 changes: 25 additions & 0 deletions tests/test_format_numeric_golden.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
from __future__ import annotations

import random

import polars as pl

from tablassert import lib, rs


def test_format_scientific_4e_golden_parity() -> None:
"""Rust ``format_scientific_4e`` matches Python ``{:.4e}`` across edge and random floats."""
rng = random.Random(42)
values: list[float | None] = [None, 0.0, -0.0, 1.0, -1.0, 1e-3, 0.05, 1e-12, 1e12, 1e100, 1e-100, float("nan"), float("inf"), float("-inf")]
values.extend(rng.uniform(-1e6, 1e6) for _ in range(1000))
values.extend(10.0 ** rng.uniform(-15, 5) for _ in range(1000))

expected: list[str | None] = [None if v is None else f"{v:.4e}" for v in values]
assert rs.format_scientific_4e(values) == expected


def test_format_numeric_end_to_end_p_value_parity() -> None:
"""``lib.format_numeric`` preserves scientific notation and null values via the Rust path."""
lf: pl.LazyFrame = pl.DataFrame({"p_value": [0.001, 0.05, None, 1e-12], "gene": ["a", "b", "c", "d"]}).lazy()
out: pl.DataFrame = lib.format_numeric(lf).collect()
assert out["p_value"].to_list() == ["1.0000e-03", "5.0000e-02", None, "1.0000e-12"]