Repository navigation
refactor(release): export verified CI evidence through shared tooling - #312
Conversation
- Append deterministic identity JSON with research-repo-tools v0.2.3 so normal and quiet Just execution preserve the same release evidence. - Reject missing output destinations and export failures while retaining signed evidence checks and exact identity revalidation after approval. - Forward verifier options through Just; pinned verification now uses `just release-evidence --identity JSON` instead of a positional identity. - Validate with `just ci` on macOS: 269 Python tests, 1,031 Rust tests, doctests, examples, benchmark compilation, and repository checks pass. Closes #303
|
No actionable comments were generated in the recent review.
|
| Layer / File(s) | Summary |
|---|---|
Verifier output and export tooling/release_evidence.py, justfile, tests/tooling/test_release_evidence.py, tooling/README.md |
The verifier accepts an optional identity pin and emits compact JSON locally or exports verified identity through the shared helper. The Just recipe forwards arguments. Tests cover successful exports, preserved records, serialization failures, and export errors. |
Publishing workflow handoff .github/workflows/publish-crates.yml, tests/tooling/test_release_publishing.py, docs/RELEASING.md |
The workflow exports evidence to GITHUB_OUTPUT, passes it through CI_EVIDENCE, and revalidates it after approval. Tests cover output handling and verify that only valid evidence proceeds to registry and token steps. The release guide describes the transport and approval flow. |
Priority:
Estimated code review effort: 3 (Moderate) | ~20 minutes
Change: Refactor
Sequence Diagram(s)
sequenceDiagram
participant GitHub Actions workflow
participant release-evidence verify
participant ci.export_environment
participant GITHUB_OUTPUT
participant publish step
GitHub Actions workflow->>release-evidence verify: verify with --github-output
release-evidence verify->>ci.export_environment: export verified identity
ci.export_environment->>GITHUB_OUTPUT: append identity output
GitHub Actions workflow->>publish step: pass CI_EVIDENCE after approval
publish step->>release-evidence verify: revalidate identity before token acquisition
Possibly related issues
- Write release evidence directly to GITHUB_OUTPUT using shared tooling #303 — Directly addresses the verifier’s
GITHUB_OUTPUTexport and shared helper. - Adopt shared release tooling and standardize recipes with la-stack delaunay#653 — Requires Delaunay to preserve the direct output-file evidence contract established here.
- Adopt shared release tooling and standardize recipes with Delaunay #304 — Requires preservation of the direct
GITHUB_OUTPUTtransport in the la-stack adoption work. - Extract shared release evidence and assets; standardize consumer naming research-repo-tools#140 — Specifies the shared
ci.export_environmenttransport used here.
Possibly related PRs
- acgetchell/la-stack#300 — Introduced the verifier and evidence identity contract changed by this PR.
- acgetchell/la-stack#299 — Added the dependency pin that provides the shared export helper.
Merge Risk: 
Low · up to 1edf7
-
Exporting to an existing output file without a final newline can merge identity into the preceding record. Ensure records are newline-terminated when using that path.
Pre-merge checks: 
4
-
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled. Title check The title clearly and concisely describes the main change: exporting verified CI evidence through shared tooling during release processing. Linked Issues check Issue #303is the only active direct target. Its coding requirements are assessed at reviewed head 1edf753.
1. Shared helper: tooling/release_evidence.py importsexport_environmentfrom `research_r…Out of Scope Changes check Each changed area maps to an objective of #303:
-.github/workflows/publish-crates.yml: direct verifier invocation with--github-output, and revalidation with--identity. Required by the issue.
…Full details: Linked Issues check
-
Explanation
Issue
#303is the only active direct target. Its coding requirements are assessed at reviewed head 1edf753.- Shared helper: tooling/release_evidence.py imports
export_environmentfromresearch_repo_tools.ciand callsexport_environment(args.github_output, ("identity",), environment={"identity": identity}). No local command-file writer exists. - Explicit output option:
verifyaccepts--github-output PATH. The justfilerelease-evidence *argsrecipe forwards arguments toverifyby insertingverifyintosys.argv. - Deterministic JSON: the identity is serialized with
sort_keys=True,separators=(",", ":"), andallow_nan=False. Without the option, the same JSON is printed to stdout. - Export after verification only:
verify()runs before serialization and export. A verification exception skips export and returns status 1. - Missing or unwritable destinations: a destination that is not an existing regular file raises
ValueError. Write failures raiseOSError.main()catches both and returns status 1. - Pinned verification: an explicitly empty identity fails at
json.loads, so it is rejected instead of selecting fresh evidence. The pin is checked for exact key shape, current run and attempt, artifact by ID, exact pin match, Sigstore attestation, and signed manifest. - Workflow: the PR summary and the previously reviewed workflow (038d16a) show the verify job invokes
just release-evidence --github-output "$GITHUB_OUTPUT"directly. Command substitution andprintfare removed. The chainsteps.evidence.outputs.identitytoneeds.verify.outputs.ci-evidencetoCI_EVIDENCEis kept. The publish job runstest -n "$CI_EVIDENCE"and then--identity "$CI_EVIDENCE"before token acquisition. - Tests: the PR test summary lists coverage for export, preserved existing records, no export on verification failure, non-finite rejection, export permission failure, normal and quiet execution, and empty or malformed identities. Test file contents were not read.
- Documentation: docs/RELEASING.md and tooling/README.md identify the shared helper and the consumer-owned Cargo evidence policy.
Unresolved uncertainty: the workflow diff at 1edf753 could not be read because changed blobs were unavailable. Workflow findings rely on the PR summary and the prior assessment. The test files, justfile recipe behavior under real Just execution, and the external helper source were not read. Hosted native CI for 1edf753 is pending. It is not a coding requirement.
- Shared helper: tooling/release_evidence.py imports
Full details: Out of Scope Changes check
-
Explanation
Each changed area maps to an objective of
#303:.github/workflows/publish-crates.yml: direct verifier invocation with--github-output, and revalidation with--identity. Required by the issue.tooling/release_evidence.py:--github-outputexport, deterministic JSON,verify(identity: str | None = None)with explicit empty identities rejected. Required by the issue.justfile:release-evidence *argsforwards arguments to the verifier. Required so the Just boundary carries--github-outputand--identity. The recipe launches uv and Python directly, which avoids Bash positional-argument expansion of JSON. This supports the identity handoff and is not an unrelated change.docs/RELEASING.mdandtooling/README.md: document the shared transport helper, the consumer-owned Cargo evidence policy, and subprocess-fixture guidance for the rewritten tests. Required by the issue acceptance and supporting.tests/tooling/test_release_evidence.pyandtests/tooling/test_release_publishing.py: cover export, pin handling, failure, and quiet-mode behavior. Supporting tests for the issue acceptance.
No change adds publication, approval, account, OIDC, or Cargo publish behavior. The PR summary states that the
crates-ioenvironment, OIDC permissions, and Cargo publish path are preserved. The workflow diff was not readable at 1edf753, so this rests on the PR summary and the prior assessment.
-
Finishing Touches
-
📝 Generate docstrings
-
- Commit to this branch
- Create a new PR
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #312 +/- ##
=======================================
Coverage 98.26% 98.26%
=======================================
Files 14 14
Lines 7215 7215
=======================================
Hits 7090 7090
Misses 125 125
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. |
- Pass the fixture import path through the environment instead of embedding Windows backslashes in Python source across Bash and native argv parsing. - Cover backslashes, apostrophes, and CRLF diagnostics through the real Just boundary while retaining exact LF command-file assertions. - Require the verifier's actual result and rejection reason, including in quiet mode, so bootstrap failures cannot satisfy negative test cases. - Document portable subprocess fixtures and validate with 271 Python tests, strict types, Markdown, spelling, and documentation checks on macOS. Refs #303
- Execute release fixture scripts from files instead of nested inline commands. - Preserve empty child environment values and verify the exact evidence arguments received before asserting release barriers. - Document the Windows transport boundaries and quiet-mode fixture diagnostics. - Validate with 271 Python tests, strict lint/types, Markdown, spelling, and documentation builds.
- Launch the evidence verifier through native uv and Python instead of passing JSON through MSYS positional argument parsing. - Reject explicitly empty identities instead of selecting fresh evidence. - Exercise the real native launch, exact received arguments, and output preservation in workflow regressions. - Validate with just ci on macOS, including 273 Python tests and the full Rust, documentation, and tooling checks.
The crates.io verification job captures recipe stdout, which loses evidence under quiet Just execution and can mix diagnostics into the JSON. This change appends the verified identity directly to
GITHUB_OUTPUTusing the installedresearch_repo_tools.ci.export_environmenthelper from v0.2.3.The verifier adds
--github-output PATH, emits deterministic single-line JSON with non-finite values rejected, preserves existing output records, and fails on missing destinations or export errors. Local verification still prints JSON. Pinned verification usesjust release-evidence --identity JSON; omitting the option selects fresh evidence, while an explicitly empty identity is rejected.The Just recipe launches uv and Python directly. Native Windows CI exposed MSYS argument expansion corrupting JSON passed through Bash positional arguments, including a valid pin becoming an empty string. The integration tests retain the real Just/uv/Python launch, assert the exact received identity, distinguish empty from unset environment values, and check the verifier's rejection reason even in quiet mode. Windows paths, CRLF diagnostics, and exact output-file bytes remain covered.
The existing job-output chain and nonempty evidence check remain intact. Signed evidence and the exact approved identity are revalidated before registry absence checks and token acquisition; Cargo gates, environment approval, OIDC permissions, and publication behavior are preserved. Shared command-file transport stays in research-repo-tools; this repository owns the Cargo evidence adapter and recipe.
Validation:
just cipasses on macOS for the final source: 273 Python tests, 1,031 Rust tests, doctests, examples, benchmark compilation, and the full tooling/documentation checks.1edf753is running; the prior failing run provides regression evidence for corrupted identity arguments.Closes #303
Summary by CodeRabbit