Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/release-benchmarks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ jobs:
echo "::error::Dispatch with --ref $RELEASE_TAG and the matching tag input"
exit 1
fi
target="$(uvx --python 3.15 --from research-repo-tools==0.1.9 research-repo-tools \
target="$(uvx --python 3.15 --from research-repo-tools==0.2.0 research-repo-tools \
--root "$RUNNER_TEMP" performance release-draft "$GH_REPO" "$RELEASE_TAG" \
--commit "$GITHUB_SHA" --expected-title "$RELEASE_TAG" | jq -c .)"
echo "target=$target" >> "$GITHUB_OUTPUT"
Expand Down Expand Up @@ -194,7 +194,7 @@ jobs:

[[ "$RELEASE_ASSET" == "la-stack-${RELEASE_TAG}-criterion-baseline.tar.gz" ]] || exit 1
printf '%s\n' "$RELEASE_TARGET" > "$RUNNER_TEMP/release-target.json"
uvx --python 3.15 --from research-repo-tools==0.1.9 research-repo-tools \
uvx --python 3.15 --from research-repo-tools==0.2.0 research-repo-tools \
--root "$RUNNER_TEMP" performance release-upload "$GH_REPO" "$RELEASE_TAG" \
"$GITHUB_WORKSPACE/$RELEASE_ASSET" --target "$RUNNER_TEMP/release-target.json"

Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ change files. Run bare `just` for the complete generated command list, including
arguments and descriptions. The [command policy](docs/dev/docs.md#command-discovery-and-placement)
owns recipe naming and documentation placement.

Setup, tool checks, updates, changelog, and opt-in CodeRabbit review use the published `research-repo-tools==0.1.9` package,
Setup, tool checks, updates, changelog, and opt-in CodeRabbit review use the published `research-repo-tools==0.2.0` package,
locked in the `tooling` dependency group and included by `dev`. Normal setup
and CI install it from PyPI through `uv sync --locked --group dev`; a sibling
checkout is unnecessary. To upgrade it deliberately, review the exact
Expand Down
13 changes: 8 additions & 5 deletions docs/BENCHMARKING.md
Original file line number Diff line number Diff line change
Expand Up @@ -336,7 +336,10 @@ records the run identity and the actual current/baseline origins of each series.
Historical CSV/JSON evidence remains frozen with its original provenance.
The README table shows median times in nanoseconds, with lower and upper 95%
confidence bounds below each timing. Its compact comparison columns show
speedup ratios; the confidence level is stated once in the table legend.
speedup ratios; the recorded confidence level is stated once in the table caption.
Shared `RunTableStyle` options produce this layout directly. Timing headings retain
measured versions, and each speedup names its peer-time numerator and la-stack-time
denominator. Ratios remain point estimates without ratio intervals or significance claims.
Missing, stale, or inconsistent input aborts publication atomically.
An existing local release tag must contain the exact selected asset bytes.
A future release tag may be absent during release preparation.
Expand Down Expand Up @@ -399,10 +402,10 @@ Serialization, validation, rendering, and promotion failures preserve existing
published reports and selection. Review and commit the whole new run and index.
Legacy snapshots under `docs/performance/` retain their original bytes,
hashes and framing; see the [legacy index](performance/README.md).
Their custom writers and report renderer are retired. Verified comparison-only
conversion uses `tooling/performance-legacy.toml` with the shared `performance convert`
command, retaining the original hashes and opaque metadata. It cannot reconstruct
raw samples or create complete-run provenance.
Their custom writers, report renderer and CSV conversion declaration are retired.
Read their original CSV/JSON and archived reports directly. They cannot reconstruct
raw samples or supply complete-run provenance. Publication accepts the declared
Just phase format; replay of the retired Python phase driver is unsupported.
The last published legacy report is preserved in the
[Markdown archive](archive/performance/README.md). The checked-in current report
keeps its historical format until a complete run is promoted; old summaries do
Expand Down
2 changes: 1 addition & 1 deletion docs/RELEASING.md
Original file line number Diff line number Diff line change
Expand Up @@ -321,7 +321,7 @@ release ID and tag commit before attaching
the archive. Do not edit the draft, move the tag,
or publish manually while the workflow is running.

The privileged jobs install the exact published `research-repo-tools==0.1.9`
The privileged jobs install the exact published `research-repo-tools==0.2.0`
package on Python 3.15 without repository checkout. Preflight emits versioned
target JSON with the captured ID, commit and title assertion. Its trusted job
output passes directly to the publisher, separately from the producer's inert
Expand Down
11 changes: 7 additions & 4 deletions docs/code_organization.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,8 @@ and review policy belong in [Security Checks](../SECURITY.md#security-checks).
The shared package also owns managed tool installation, verification, and update
implementation. `.python-version`, `rust-toolchain.toml`, and `pyproject.toml`
own consumer declarations; `scripts/tests/test_toolchain_integration.py` checks
actual recipe sequencing and native managed execution. The
the exact published pin, registry lock, installed version and distribution hashes,
alongside actual recipe sequencing and native managed execution. The
`.github/actions/setup-tools/action.yml` composite synchronizes the locked PyPI
package and exports verified paths for CI. Release callers disable its caches.

Expand All @@ -166,10 +167,12 @@ Performance consumers use shared Criterion parsing and estimate/comparison
validation, digest verification, archive extraction, byte-preserving document
sections, and multi-file transactions. Shared rendering produces complete-run
reports, coordinate SVG, timing CSV and document tables. Historical artifact
bytes and fingerprint framing remain frozen; declarative shared conversion reads
the original CSV/metadata without a legacy writer. Published v0.1.9
bytes and fingerprint framing remain frozen and readable directly. Published v0.2.0
owns common-harness orchestration, completeness, worktrees, complete run retention,
validated latest selection, and transactional report publication.
Its `RunTableStyle` API owns compact README table formatting; the consumer selects
dimension headings, intervals below timings, a common confidence legend and
ratio-only speedups without rewriting Markdown cells or prose.
Generic parsing, staging, and rollback tests belong upstream; local
tests verify the scientific and retained-artifact integration boundaries.

Expand All @@ -196,7 +199,7 @@ shared APIs for tracked-only measurements, metadata-free historical raw archives
and la-stack README publication.
`tooling/performance-plot.toml` declares dimensions, units, series and axes.
`tooling/performance-report.toml` owns the canonical report path, title, and shared history path.
`tooling/performance-legacy.toml` declares shared historical comparison conversion.
The obsolete CSV conversion declaration and pre-Just Python phase replay are removed.
The old comparison renderer, CSV/provenance writer, summary reader, plotter,
release-inventory wrapper and Python packaging are removed. Only the two
consumer adapters remain; `scripts/tests/test_performance_workflow.py` verifies
Expand Down
45 changes: 39 additions & 6 deletions docs/dev/performance-tooling-adoption.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,37 @@ runtime code comes from the installed package.

## Capability disposition

Follow-up [#285](https://github.com/acgetchell/la-stack/issues/285) adopts the
published `research-repo-tools==0.2.0` wheel and sdist, including
[upstream #110](https://github.com/acgetchell/research-repo-tools/issues/110).
The exact PyPI pin and lock replace v0.1.9; the package also advances its required
Just dependency to v1.59.0. `RunTableStyle` now supplies the six-column README
layout directly, so `_readme_table()` and its Markdown string/column rewrites
are removed. The shared renderer retains recorded intervals, the selected
statistic, confidence level and explicit ratio direction. Scientific selection,
original reference phases, README markers, evidence freshness and transactional
asset pointers remain consumer responsibilities.

Offline consumer checks exercise the actual `performance-readme` recipe with
mean and median estimates, original peer phases, retained history and the README
figure. No new measurements or regenerated immutable reports are required.
Offline rendering of the retained published run preserves its SVG and CSV bytes.
Current README republication still rejects the stale source digest and leaves
the figure and measured-asset pointers untouched.
The obsolete CSV conversion declaration and pre-Just Python phase replay are
removed. Retained complete runs use the declared Just driver; original legacy
CSV/JSON and Markdown records remain available without regeneration or conversion.
Closure of #285 requires native Linux/macOS/Windows CI on the adoption commit;
local validation alone does not establish those results.

| Upstream capability | Consumer adoption | Local responsibility |
| --- | --- | --- |
| #94 captured targets | Installed release-target CLI passes versioned JSON from trusted preflight to writer | Dispatch/ref equality, current attempt, title, privilege separation, asset filename and explicit publication |
| #95 coordinate plots | Shared coordinate SVG, table and CSV renderers | Scientific metric mappings, dimensions, nanoseconds and original phases |
| #96 provenance reports | Shared renderer 2 and publication-relative evidence links | Scientific eligibility and frozen historical records |
| #97 complete-run sections | Shared presentation and publication transaction | README markers, measured-asset pointers and current source/harness checks |
| #98 Criterion verification | Declarative shared discovery, versioned inventory and saved-sample APIs | Full Cargo suites/features, minimum scientific coverage and correctness gates |
| #110 compact table styles | Installed `RunTableStyle` selects headings, intervals, confidence caption and ratio-only columns | Scientific selections, ratio pairs and README publication policy |

The full-report standardization from #277 remains. New reports expose recorded
provenance and evidence links. Shared mixed-renderer history preserves immutable
Expand All @@ -31,7 +55,7 @@ The audit compared the dependency-only MCMC environment and declarative
| `benchmark_contract.py` | Deleted; metric/group/peer/API declarations moved to `tooling/performance-policy.toml`; historical digests stay frozen |
| `benchmark_summaries.py` | Deleted; frozen historical summaries are records rather than regeneration inputs |
| `criterion_dim_plot.py` | Deleted; shared complete-run SVG/CSV/table presentation replaces raw-Criterion/gnuplot rendering |
| `performance_artifacts.py` | Deleted; shared evidence/publication and declarative `performance-legacy.toml` replace CSV/provenance writers and readers |
| `performance_artifacts.py` | Deleted; shared evidence/publication replace writers and readers; historical CSV conversion retired in #285 |
| `release_baseline.py` | Deleted; native discovery uses the shared CLI and saved-copy verification uses the shared configured API |
| Local wheel/sdist and console-entry-point packaging | Removed; `uv` manages a dependency-only environment |
| Duplicate parser, renderer, legacy-writer and packaging tests | Removed; actual installed-package/configuration and scientific integration tests remain |
Expand All @@ -40,11 +64,9 @@ Retirement conditions are satisfied by installed replacement checks, preservatio
of original bytes/hashes and a documented supported workflow. The old report
layout need not be regenerated indefinitely. New reports use the shared format;
historical reports, CSV/JSON, indexes, plots and archives remain untouched.
Shared historical conversion verifies the original hash and retains opaque
metadata and original digest names. It produces comparison evidence, never
fabricated complete-run samples or newly captured host provenance. Retire its
conversion declaration only after verified companions exist for every original
and those original bytes remain independently available.
Historical conversion was verified during #276 and retired in #285. Original
summaries remain independently readable with their original hashes and opaque
metadata. They cannot provide complete-run samples or newly captured host provenance.

Only two production modules remain:

Expand Down Expand Up @@ -85,6 +107,17 @@ current published wheels support Python 3.15 and native gates pass.

## Size and validation evidence

The #285 consumer-test cleanup was checked against tests bundled in the published
v0.2.0 sdist. Shared implementation matrices are removed only where that release
already covers them; consumer policy and installed-package integration remain.

| Coverage | Published upstream owner | Retained consumer check |
| --- | --- | --- |
| Changelog rendering and repeatability | `tests/changelog/test_cliff_template.py` and `test_workflow.py` | Actual la-stack formatter/preservation policy, one authored-note generation and historical notes |
| Measurement completeness and failed gates | `tests/performance/public_complete_consumer.py` | Actual sampling policy, required scientific inventory, native gate commands and original peer phases |
| Package identity | Consumer-owned | One test in `test_toolchain_integration.py` checks pin, registry, installed version and wheel/sdist hashes; recipe tests verify preservation and forwarding |
| Release event identity and check-status matrices | `tests/releases/test_publishing.py` | Every configured native check remains required; actual workflow approval, immutability, notes and upload barriers remain tested |

Counts use physical lines from `wc -l scripts/*.py scripts/tests/*.py` and
`uv run --locked pytest --collect-only -q`. Baseline is `905aba7`; test counts
describe consumer checks, not measurements.
Expand Down
28 changes: 10 additions & 18 deletions docs/performance/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ the [Benchmarking guide](../BENCHMARKING.md) owns the current workflow.
## Contents

- [Saved runs](#saved-runs)
- [Regenerating reports](#regenerating-reports)
- [Reading retained reports](#reading-retained-reports)
- [Comparing measurements](#comparing-measurements)
- [Historical availability](#historical-availability)

Expand Down Expand Up @@ -38,27 +38,19 @@ measurements; missing current measurements are never filled from an older run.
`latest.json` identifies the last saved legacy snapshot. New complete-run
publication maintains its own shared index and latest pointer instead.

## Regenerating reports
## Reading retained reports

While no shared complete run has been retained, these commands can use the
committed legacy snapshot when scratch report inputs are absent:
Read these original CSV/JSON records and the
[archived Markdown reports](../archive/performance/README.md) directly.
Their writers, custom renderer and conversion declaration are retired.
`performance-doc` and `performance-readme` require shared complete-run evidence;
they do not consume these summaries. Unsupported legacy scratch inputs block
fallback to older history, so archive or remove those scratch files first.

```bash
just performance-doc
just performance-readme
```

Neither command reruns benchmarks. Existing scratch inputs take precedence;
partial or corrupt inputs fail validation rather than silently selecting an
older snapshot. README publication still checks the measured source and
benchmark contract against the current checkout, along with the lockfile and
release version. Committing only the saved artifacts does not invalidate a
snapshot: its original measurement commit remains in provenance. Legacy inputs
without a benchmark-contract digest also require the original commit.

Once a shared complete run is promoted, the commands select that history.
Legacy summaries cannot be converted losslessly into complete-run evidence:
they retain aggregate statistics and sample counts, but not the raw samples.
The [Benchmarking guide](../BENCHMARKING.md) owns current report and README
publication, including source freshness and measured-asset checks.

## Comparing measurements

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ required-version = "==0.13.1"
# Semgrep has its own locked Python 3.14 environment under tooling/semgrep.

[dependency-groups]
tooling = [ "research-repo-tools==0.1.9" ]
tooling = [ "research-repo-tools==0.2.0" ]
dev = [
{ include-group = "tooling" },
"actionlint-py==1.7.12.25",
Expand Down
24 changes: 14 additions & 10 deletions scripts/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# scripts/

Only two production Python adapters remain. Generic tools run from the installed
`research-repo-tools==0.1.9` base package; their declarations belong in
`research-repo-tools==0.2.0` base package; their declarations belong in
`tooling/` and `pyproject.toml`. The project is a dependency-only environment,
with no local Python distribution, build backend or console entry points.

Expand Down Expand Up @@ -60,9 +60,11 @@ table renderers. The former preserves current source/harness membership and
digests and advances the three existing measured-artifact pointers transactionally.
README candidates pass the shared Markdown formatter with repository policy
before any publication files are replaced.
The README adapter labels dimensions, keeps each confidence interval below its
timing, states the confidence level once, and selects the shared speedup columns.
Timing estimates and comparison arithmetic remain owned by the shared renderer.
README publication configures the shared `RunTableStyle` API to label dimensions,
keep both interval bounds below each timing, state the recorded confidence level
once, and select ratio-only speedup columns with explicit numerator/denominator
semantics. Series labels supply the timing headings and measured versions.
The shared renderer owns formatting, timing estimates and comparison arithmetic.

```bash
just performance-release
Expand All @@ -74,17 +76,19 @@ The raw-Criterion/gnuplot publisher, historical summary writer and custom
report renderer are retired after installed-package replacement checks.
Original CSV/JSON, reports, fingerprints and archives remain unchanged.
They are read directly as frozen records; no regeneration requirement keeps
their writers alive. For verified historical conversion into shared comparison
evidence, use `research-repo-tools performance convert` with
`tooling/performance-legacy.toml`; this never reconstructs complete runs from
summaries. Do not promote converted historical summaries as new measurements.
The conversion declaration remains until every retained original has a verified
shared comparison companion and its original bytes remain independently available.
their writers alive. The obsolete CSV conversion declaration and pre-Just Python
phase format are retired. Report and README publication require complete-run
evidence with the declared Just driver. Historical summaries lack raw samples
and cannot supply that evidence; their original bytes remain available directly.

Generic parser, renderer, transaction and provider failure matrices belong
upstream. Consumer tests retain scientific selection/phase/gate checks, actual
native recipe wiring, complete release archive round trips, captured-target
handoff, offline publication and original artifact hashes.
`scripts/tests/test_toolchain_integration.py` owns the single published-package
contract: manifest pin, registry lock, installed version and wheel/sdist hashes.
Dependency-update and release-workflow tests derive that requirement from the
manifest and verify it survives their actual recipe execution.

## Changelog and release tooling

Expand Down
Loading
Loading