Skip to content

refactor(release): export verified CI evidence through shared tooling - #312

Merged
acgetchell merged 4 commits into
mainfrom
refactor/303-evidence-output
Oct 27, 2026
Merged

acgetchell merged 4 commits into
mainfrom
refactor/303-evidence-output

Conversation

@acgetchell

@acgetchell acgetchell commented Oct 27, 2026 •

Copy link
Copy Markdown
Owner

The crates.io verification job captures recipe stdout, which loses evidence under quiet Just execution and can mix diagnostics into the JSON. This change appends the verified identity directly to GITHUB_OUTPUT using the installed research_repo_tools.ci.export_environment helper from v0.2.3.

The verifier adds --github-output PATH, emits deterministic single-line JSON with non-finite values rejected, preserves existing output records, and fails on missing destinations or export errors. Local verification still prints JSON. Pinned verification uses just release-evidence --identity JSON; omitting the option selects fresh evidence, while an explicitly empty identity is rejected.

The Just recipe launches uv and Python directly. Native Windows CI exposed MSYS argument expansion corrupting JSON passed through Bash positional arguments, including a valid pin becoming an empty string. The integration tests retain the real Just/uv/Python launch, assert the exact received identity, distinguish empty from unset environment values, and check the verifier's rejection reason even in quiet mode. Windows paths, CRLF diagnostics, and exact output-file bytes remain covered.

The existing job-output chain and nonempty evidence check remain intact. Signed evidence and the exact approved identity are revalidated before registry absence checks and token acquisition; Cargo gates, environment approval, OIDC permissions, and publication behavior are preserved. Shared command-file transport stays in research-repo-tools; this repository owns the Cargo evidence adapter and recipe.

Validation:

  • just ci passes on macOS for the final source: 273 Python tests, 1,031 Rust tests, doctests, examples, benchmark compilation, and the full tooling/documentation checks.
  • Real workflow/Just regressions cover normal and quiet execution, malformed/stale/empty identity rejection, output preservation, and registry/token barriers.
  • Native Windows CI for commit 1edf753 is running; the prior failing run provides regression evidence for corrupted identity arguments.

Closes #303

Summary by CodeRabbit

  • Release Process
    • Automated publishing exports the verified release identity to GitHub Actions and rechecks that exact identity after approval, before proceeding. Invalid evidence, an empty identity, or output-file errors stop the process.
    • Local verification prints the identity as compact, consistent JSON. GitHub Actions records the verified identity without replacing existing output records; export occurs only after verification succeeds.
  • Documentation
    • Updated release guidance to explain how verification results are reported locally and in GitHub Actions.
    • Clarified identity validation and the checks performed before publishing continues.

- Append deterministic identity JSON with research-repo-tools v0.2.3 so
  normal and quiet Just execution preserve the same release evidence.
- Reject missing output destinations and export failures while retaining
  signed evidence checks and exact identity revalidation after approval.
- Forward verifier options through Just; pinned verification now uses
  `just release-evidence --identity JSON` instead of a positional identity.
- Validate with `just ci` on macOS: 269 Python tests, 1,031 Rust tests,
  doctests, examples, benchmark compilation, and repository checks pass.

Closes #303
@acgetchell
acgetchell enabled auto-merge (squash) October 27, 2026 00:06
@coderabbitai

coderabbitai Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 038d16a and 1edf753.

Files selected for processing: 6
  • docs/RELEASING.md
  • justfile
  • tests/tooling/test_release_evidence.py
  • tests/tooling/test_release_publishing.py
  • tooling/README.md
  • tooling/release_evidence.py
Files skipped from review as they are similar to previous changes: 1
  • tooling/README.md

Included review availability: This review used your included allowance. 5 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 8 reviews per hour.

Walkthrough

The verifier can export verified identity to GitHub Actions output without stdout capture. The publishing workflow passes that identity to the publish step for revalidation after approval and before token acquisition.

Changes: Release Evidence Transport

Layer / File(s) Summary
Verifier output and export
tooling/release_evidence.py, justfile, tests/tooling/test_release_evidence.py, tooling/README.md
The verifier accepts an optional identity pin and emits compact JSON locally or exports verified identity through the shared helper. The Just recipe forwards arguments. Tests cover successful exports, preserved records, serialization failures, and export errors.
Publishing workflow handoff
.github/workflows/publish-crates.yml, tests/tooling/test_release_publishing.py, docs/RELEASING.md
The workflow exports evidence to GITHUB_OUTPUT, passes it through CI_EVIDENCE, and revalidates it after approval. Tests cover output handling and verify that only valid evidence proceeds to registry and token steps. The release guide describes the transport and approval flow.

Priority: Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Refactor

Sequence Diagram(s)

sequenceDiagram
  participant GitHub Actions workflow
  participant release-evidence verify
  participant ci.export_environment
  participant GITHUB_OUTPUT
  participant publish step
  GitHub Actions workflow->>release-evidence verify: verify with --github-output
  release-evidence verify->>ci.export_environment: export verified identity
  ci.export_environment->>GITHUB_OUTPUT: append identity output
  GitHub Actions workflow->>publish step: pass CI_EVIDENCE after approval
  publish step->>release-evidence verify: revalidate identity before token acquisition
Loading

Possibly related issues

Possibly related PRs

Merge Risk: Low · up to 1edf7

Exporting to an existing output file without a final newline can merge identity into the preceding record. Ensure records are newline-terminated when using that path.

Pre-merge checks: Passed 4
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main change: exporting verified CI evidence through shared tooling during release processing.
Linked Issues check Passed Issue #303 is the only active direct target. Its coding requirements are assessed at reviewed head 1edf753.

1. Shared helper: tooling/release_evidence.py imports export_environment from `research_r…
Out of Scope Changes check Passed Each changed area maps to an objective of #303:
- .github/workflows/publish-crates.yml: direct verifier invocation with --github-output, and revalidation with --identity. Required by the issue.
…
Full details: Linked Issues check

Explanation

Issue #303 is the only active direct target. Its coding requirements are assessed at reviewed head 1edf753.

  1. Shared helper: tooling/release_evidence.py imports export_environment from research_repo_tools.ci and calls export_environment(args.github_output, ("identity",), environment={"identity": identity}). No local command-file writer exists.
  2. Explicit output option: verify accepts --github-output PATH. The justfile release-evidence *args recipe forwards arguments to verify by inserting verify into sys.argv.
  3. Deterministic JSON: the identity is serialized with sort_keys=True, separators=(",", ":"), and allow_nan=False. Without the option, the same JSON is printed to stdout.
  4. Export after verification only: verify() runs before serialization and export. A verification exception skips export and returns status 1.
  5. Missing or unwritable destinations: a destination that is not an existing regular file raises ValueError. Write failures raise OSError. main() catches both and returns status 1.
  6. Pinned verification: an explicitly empty identity fails at json.loads, so it is rejected instead of selecting fresh evidence. The pin is checked for exact key shape, current run and attempt, artifact by ID, exact pin match, Sigstore attestation, and signed manifest.
  7. Workflow: the PR summary and the previously reviewed workflow (038d16a) show the verify job invokes just release-evidence --github-output "$GITHUB_OUTPUT" directly. Command substitution and printf are removed. The chain steps.evidence.outputs.identity to needs.verify.outputs.ci-evidence to CI_EVIDENCE is kept. The publish job runs test -n "$CI_EVIDENCE" and then --identity "$CI_EVIDENCE" before token acquisition.
  8. Tests: the PR test summary lists coverage for export, preserved existing records, no export on verification failure, non-finite rejection, export permission failure, normal and quiet execution, and empty or malformed identities. Test file contents were not read.
  9. Documentation: docs/RELEASING.md and tooling/README.md identify the shared helper and the consumer-owned Cargo evidence policy.

Unresolved uncertainty: the workflow diff at 1edf753 could not be read because changed blobs were unavailable. Workflow findings rely on the PR summary and the prior assessment. The test files, justfile recipe behavior under real Just execution, and the external helper source were not read. Hosted native CI for 1edf753 is pending. It is not a coding requirement.

Full details: Out of Scope Changes check

Explanation

Each changed area maps to an objective of #303:

  • .github/workflows/publish-crates.yml: direct verifier invocation with --github-output, and revalidation with --identity. Required by the issue.
  • tooling/release_evidence.py: --github-output export, deterministic JSON, verify(identity: str | None = None) with explicit empty identities rejected. Required by the issue.
  • justfile: release-evidence *args forwards arguments to the verifier. Required so the Just boundary carries --github-output and --identity. The recipe launches uv and Python directly, which avoids Bash positional-argument expansion of JSON. This supports the identity handoff and is not an unrelated change.
  • docs/RELEASING.md and tooling/README.md: document the shared transport helper, the consumer-owned Cargo evidence policy, and subprocess-fixture guidance for the rewritten tests. Required by the issue acceptance and supporting.
  • tests/tooling/test_release_evidence.py and tests/tooling/test_release_publishing.py: cover export, pin handling, failure, and quiet-mode behavior. Supporting tests for the issue acceptance.

No change adds publication, approval, account, OIDC, or Cargo publish behavior. The PR summary states that the crates-io environment, OIDC permissions, and Cargo publish path are preserved. The workflow diff was not readable at 1edf753, so this rests on the PR summary and the prior assessment.

Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.26%. Comparing base (8769cc4) to head (1edf753).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #312   +/-   ##
=======================================
  Coverage   98.26%   98.26%           
=======================================
  Files          14       14           
  Lines        7215     7215           
=======================================
  Hits         7090     7090           
  Misses        125      125           
Flag Coverage Δ
unittests 98.26% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

coderabbitai[bot]
coderabbitai Bot previously approved these changes Oct 27, 2026
- Pass the fixture import path through the environment instead of embedding
  Windows backslashes in Python source across Bash and native argv parsing.
- Cover backslashes, apostrophes, and CRLF diagnostics through the real Just
  boundary while retaining exact LF command-file assertions.
- Require the verifier's actual result and rejection reason, including in
  quiet mode, so bootstrap failures cannot satisfy negative test cases.
- Document portable subprocess fixtures and validate with 271 Python tests,
  strict types, Markdown, spelling, and documentation checks on macOS.

Refs #303
coderabbitai[bot]
coderabbitai Bot previously approved these changes Oct 27, 2026
- Execute release fixture scripts from files instead of nested inline commands.
- Preserve empty child environment values and verify the exact evidence arguments received before asserting release barriers.
- Document the Windows transport boundaries and quiet-mode fixture diagnostics.
- Validate with 271 Python tests, strict lint/types, Markdown, spelling, and documentation builds.
coderabbitai[bot]
coderabbitai Bot previously approved these changes Oct 27, 2026
- Launch the evidence verifier through native uv and Python instead of passing JSON through MSYS positional argument parsing.
- Reject explicitly empty identities instead of selecting fresh evidence.
- Exercise the real native launch, exact received arguments, and output preservation in workflow regressions.
- Validate with just ci on macOS, including 273 Python tests and the full Rust, documentation, and tooling checks.
@acgetchell
acgetchell merged commit 7320d6c into main Oct 27, 2026
20 checks passed
@acgetchell
acgetchell deleted the refactor/303-evidence-output branch October 27, 2026 01:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Write release evidence directly to GITHUB_OUTPUT using shared tooling

1 participant