Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,22 @@ jobs:
- name: Capture successful native validation identities
env:
GH_TOKEN: ${{ github.token }}
run: uv run --locked python -m tooling.release_evidence seal --output target/release-evidence/validation.json
run: |
uvx --python 3.15 --from research-repo-tools==0.2.4 python -I - <<'PY'
import os
import sys
from pathlib import Path
from research_repo_tools.cli import main
from research_repo_tools.release_metadata import read_package_info
root = Path.cwd()
sys.exit(main([
"release", "evidence", "create", "tooling/release-evidence.toml",
"--checkout", str(root), "--commit", os.environ["GITHUB_SHA"],
"--version", read_package_info(root).version,
"--run-id", os.environ["GITHUB_RUN_ID"], "--attempt", os.environ["GITHUB_RUN_ATTEMPT"],
"--output", "target/release-evidence/validation.json",
]))
PY
- name: Sign original main validation evidence
id: validation
uses: actions/attest@1e69f48acb82d1966a394da916b4c1698aa569d6 # v4.2.2
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/publish-crates.yml
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,8 @@ jobs:
set -euo pipefail
just release-gate "$RELEASE_TAG"
test -n "$CI_EVIDENCE"
just release-evidence --identity "$CI_EVIDENCE"
printf '%s\n' "$CI_EVIDENCE" > "$RUNNER_TEMP/approved-evidence.json"
just release-evidence --expected "$RUNNER_TEMP/approved-evidence.json"
uv run --locked --no-sync research-repo-tools release registry "$RELEASE_TAG" --expect absent

- name: Obtain a short-lived crates.io token
Expand Down
26 changes: 15 additions & 11 deletions docs/BENCHMARKING.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,15 @@ the commands measure and where their outputs go.

## Start Here

Run `just performance-prepare` before measurements to fetch the release tags.
This explicit Git operation and the measurement recipes' temporary worktrees
require authorization. Both labels of new measurements must be v0.5.0 or newer.
Run `just performance-prepare` before measurements using reviewed local tags.
Add `--fetch-tags --allow-git-mutations` to fetch tags explicitly. Fetching and
measurement recipes' temporary worktrees require authorization. Both labels of
new measurements must be v0.5.0 or newer.
Preparation also saves independent release tags and commits for a later
`just performance-readme` invocation in ignored
`target/bench-reports/release-inputs.json`. For an explicit release pair, use
`just performance-prepare "$CURRENT_TAG" "$PREVIOUS_TAG"`.
`target/bench-reports/release-inputs.json`. For an explicit prepared pair, set
`mode = "explicit"` and both `current` and `baseline` in
`tooling/performance-release.toml`. The default is publication chronology.

| Goal | Recipe |
| --- | --- |
Expand Down Expand Up @@ -437,19 +439,21 @@ still apply. README publication additionally requires live source freshness.

### Create The Release Performance Report

After updating the package version and fetching tags:
After updating the package version and reviewing local tags:

```bash
just performance-prepare
just performance-release
```

With no labels, the installed CLI infers the current package's release pair
using publication chronology. A prospective version compares the working tree
with the latest release; a published version compares existing tags. Explicit
labels select existing tags:
Preparation uses publication chronology. A prospective version compares the
working tree with the latest release; a published version compares existing
tags. The release command consumes that saved selection. For ad hoc measurement
of two existing tags followed by separate promotion:

```bash
just performance-release "$CURRENT_TAG" "$PREVIOUS_TAG"
just performance-measure "$CURRENT_TAG" "$PREVIOUS_TAG"
just performance-promote
```

`performance-prepare` captures both source commits before measurement and prints
Expand Down
49 changes: 31 additions & 18 deletions docs/RELEASING.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,9 @@ including when a rerun happens after UTC midnight.
## Reusable CI evidence

The publishing process uses the installed `research-repo-tools==0.2.4`
validation helpers. `tooling/release_evidence.py` supplies the Cargo-specific
policy; it does not use the shared package's wheel/sdist release commands.
release evidence commands. `tooling/release-evidence.toml` declares the repository,
package, workflow, native jobs/steps and signer policy. The guarantee is
`signed-source`; it does not promise promotion of an identical Cargo archive.
Every successful protected `main` push produces signed evidence after all three
native `just ci` jobs and the default/exact Cargo packaging job succeed.
The `release-evidence` job signs their exact commit, run, attempt, job IDs and
Expand All @@ -50,27 +51,32 @@ nonexpired artifact retrieved by immutable ID. The shared downloader checks its
authoritative ZIP size and SHA-256 digest and exact file inventory. GitHub CLI
then verifies the signed manifest's workflow, source commit, `main` ref, hosted
runner, push event, run and attempt. The manifest must match the live job and
package identities. Pending matching runs receive a bounded wait; missing,
package identities. Pending,
failed, ambiguous or expired evidence stops publication. PR, manual dispatch,
fork, local and partial-rerun results cannot replace it.

Local verification prints the validated identity as deterministic single-line
JSON. In Actions, `just release-evidence --github-output "$GITHUB_OUTPUT"`
uses the installed `research_repo_tools.ci.export_environment` helper to append
the `identity` output only after verification succeeds. The destination must
Local verification prints a success message; pass `--output <path>` to retain
the full approval selection. In Actions,
`just release-evidence --github-output "$GITHUB_OUTPUT"` uses the installed
transactional output writer to append `identity` and `commit` only after
verification succeeds. The destination must
already exist; export errors fail the command. Existing output records survive,
and normal or quiet Just execution transports the same JSON without capturing
stdout. Non-finite JSON values are rejected. The shared helper owns command-file
transport; this adapter retains the Cargo-specific evidence policy.
The Just recipe launches uv and Python directly so JSON arguments never pass
through the native Windows-to-MSYS shell boundary. Omitting `--identity` selects
fresh evidence; supplying an empty identity is an error.
stdout. The shared command owns JSON validation and command-file transport.
The Just recipe launches the exact published package with `uvx` outside the
observed checkout, using native Python argument transport on Windows.
Omitting `--expected` selects fresh evidence; a supplied empty or malformed
selection file is an error. Existing selection files can only be reused with
identical bytes. The installed downloader bounds archives at 32 MiB and the
signed manifest at 1 MiB; repository, digest and signature checks still apply.

The publishing workflow reuses these completed checks and passes the run/attempt
and artifact ID/digest through `steps.evidence.outputs.identity` and
`needs.verify.outputs.ci-evidence` to `CI_EVIDENCE`. After deployment approval,
it requires a nonempty value and revalidates that exact identity with
`just release-evidence --identity "$CI_EVIDENCE"` before token acquisition.
it requires a nonempty value, writes it to `RUNNER_TEMP/approved-evidence.json`,
and revalidates it with `just release-evidence --expected <path>` before token
acquisition. A new run, attempt, artifact, manifest or policy cannot replace
the approved selection.
It also retains the independent release metadata, exact-commit SARIF,
durable Criterion asset, immutable GitHub release and registry-absence gates.
Release benchmarks remain a separate scientific measurement workflow; CI
Expand All @@ -82,7 +88,7 @@ dry-run `.crate` archive. The shared tooling's Python distribution promotion
process has a different artifact contract.

Use a current GitHub CLI with `gh attestation verify` and the signer/source
constraints used by the adapter. See GitHub's
constraints used by the installed verifier. See GitHub's
[attestation verification options](https://cli.github.com/manual/gh_attestation_verify).

## Prepare the environment
Expand Down Expand Up @@ -213,7 +219,7 @@ After updating the package version, fetch the tags and capture the independent
source assertions before measurement:

```bash
just performance-prepare
just performance-prepare --fetch-tags --allow-git-mutations
```

Preparation selects the release pair using GitHub's publication history,
Expand All @@ -224,6 +230,13 @@ This Git-ignored file carries the tags and commits forward to
Review the printed tags and commits. A failed release lookup or missing tag
stops preparation with a diagnostic.

The plain `performance-prepare` command uses local tags without changing refs.
The shared preparation record also binds the package version and measurement
declaration. If you have an old adapter-format `release-inputs.json`, remove
that ignored scratch file once and prepare again. It is deliberately not
converted from measurement evidence. Moved refs or changed package/declaration
inputs require a new preparation before measurement or README publication.

No shell variables or exports are needed. `performance-prepare` does not set
`PREVIOUS_TAG`, `CURRENT_COMMIT`, or `PREVIOUS_COMMIT` in your shell; those
variables may remain unset. Do not run the old manual `git rev-parse`
Expand All @@ -235,8 +248,8 @@ Then run the long measurement command:
just performance-release
```

The no-argument recipe compares the prospective package version with the
latest stable published release. It measures and promotes the release report;
The no-argument recipe measures the independently prepared pair and promotes
the release report;
README publication remains a separate, retryable step below.

Review `docs/performance.md` and the complete
Expand Down
19 changes: 11 additions & 8 deletions docs/code_organization.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,9 +171,9 @@ and tag preparation through its CLI. Consumer identity and dependency-body polic
stay in `[tool.research-repo-tools.changelog]` in `pyproject.toml`, with
formatting in `changelog-rumdl.toml`; no common template is copied locally.
Focused integration checks live in `tests/tooling/test_changelog_integration.py`.
`tooling/release_changelog.py` temporarily resolves omitted prospective release
dates from the prepared citation metadata before delegating generation and
preview to the installed CLI. It does not update citation metadata or use a clock.
The installed CLI resolves omitted prospective dates from `CITATION.cff`, as
declared by `changelog.date-source`. Generation and preview do not update
citation metadata or use a clock.
The same dependency owns CodeRabbit review orchestration through thin Just
wrappers. `tests/tooling/test_review_integration.py` owns consumer wiring checks
with local stubs; the [contributor review workflow](../CONTRIBUTING.md#coderabbit-review)
Expand Down Expand Up @@ -262,9 +262,9 @@ freshness checks and tagged evidence links. Static named configurations under
linear/log choices. `tooling/plots/non-exact/` retains eight diagnostic metrics.
All views require explicit input ownership.

`tooling/release_performance.py` temporarily captures independent release tags and
commits during `performance-prepare` and supplies them to the separate
`performance-readme` publication step, pending equivalent published upstream commands.
`tooling/performance-release.toml` declares independent preparation storage and
the measurement, report and README policies. Installed `performance prepare`,
`release` and `readme` commands own those separately retriable phases.
The former measurement/publication implementations and root `scripts/` directory are retired.
The installed CLI owns orchestration, validation and publication transactions.
Working-tree measurements now include nonignored untracked regular files.
Expand All @@ -291,9 +291,12 @@ outputs remain outside production scans.
input validation, and inventory under the release workflow's shared setup timeout.

`.github/workflows/ci.yml` owns native checks, default/exact Cargo dry-runs and
signed main-push validation evidence. `tooling/release_evidence.py` supplies the
Cargo-specific policy around shared run/job validation and artifact retrieval;
signed main-push validation evidence. `tooling/release-evidence.toml` supplies
native/Cargo and signer policy to the installed verifier;
`tests/tooling/test_release_evidence.py` owns its provider-boundary regressions.
`tooling/release-assets.toml` declares the draft-target identity for the installed
`release-assets` recipe. Privileged benchmark jobs continue to use installed
asset commands from `RUNNER_TEMP`, without executing checkout code.
`.github/workflows/publish-crates.yml` owns the reviewed-release event, reuse of
that exact CI attempt and artifact, deployment approval and OIDC upload. The publishing declaration
in `pyproject.toml` selects `la-stack`, native checks and the Criterion asset;
Expand Down
28 changes: 26 additions & 2 deletions docs/dev/performance-tooling-adoption.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,17 +57,41 @@ or shell implementations of the shared workflow.
| `performance-local` | `performance measure`, `measurements/full.toml`, current versus latest published release |
| `performance-local-non-exact` | `performance measure`, `measurements/non-exact.toml`, current versus latest published release |
| `performance-measure` | Explicit current/baseline labels and a named full/non-exact declaration |
| `performance-release` | `performance measure`, followed by explicit `performance promote` |
| `performance-prepare` | `performance prepare` with `performance-release.toml`; independent saved assertions |
| `performance-release` | `performance release` with `performance-release.toml`; measure the prepared pair and promote |
| `performance-promote` | Explicit scratch payload and envelope, checked against `performance-report.toml` |
| `performance-doc` | Offline retained history through `performance promote` and the independent full policy |
| `performance-readme` | `performance publish` with `performance-readme.toml`, independent identities and transactional formatting |
| `performance-readme` | `performance readme` with `performance-release.toml`, prepared or explicit independent identities and transactional formatting |
| `plot-vs-linalg` | `performance publish --artifacts-only` with named full scratch, retained, or non-exact scratch views |
| `bench-release-check` | `performance verify-saved` with saved name, Criterion directory and `criterion-saved.toml` |
| `bench-release-check-suite` | `performance verify-saved --suite` with the complete independent policy and selected feature suite |
| Hosted suite packaging and assembly | `performance host`, `pack-suite` and `assemble-suites`; la-stack retains the native commands, matrix, artifact selection and budgets |
| `bench-compare` | Existing installed saved-Criterion comparison CLI |

Paths in the table are relative to `tooling/`.

### Release adapter migration

Published `research-repo-tools==0.2.4` now owns the former
`release_changelog.py`, `release_evidence.py` and `release_performance.py`
adapters. Citation dates are declarative in `pyproject.toml`; evidence, draft
assets and performance phases have dedicated TOML policies. The native Just
launchers observe checkout identity and forward arguments to the installed
commands. Evidence and asset verification use an isolated runtime outside the
checkout. Privileged benchmark jobs retain their existing installed asset
commands and captured target without checking out or executing repository code.

The upstream installed-consumer suites
[`public_evidence_consumer.py`](https://github.com/acgetchell/research-repo-tools/blob/v0.2.4/tests/releases/public_evidence_consumer.py)
and
[`public_dates_consumer.py`](https://github.com/acgetchell/research-repo-tools/blob/v0.2.4/tests/changelog/public_dates_consumer.py)
own generic output rollback, immutable handoffs, malformed metadata, signature
and date parsing regressions. Local tests retain the actual native/Cargo policy,
workflow approval barriers, real Just output transport, installed runtime
identity, stale preparation rejection and scientific publication fixtures.
The shared downloader's archive bound is 32 MiB; signed manifests remain
bounded at 1 MiB. Recovery uses the shared selection schema; legacy preparation
files must be replaced through a new independent preparation.
[Benchmarking](../BENCHMARKING.md) owns arguments and examples.

## Consumer contracts
Expand Down
Loading
Loading