Repository navigation
Conversation
…piler
The MC/DC gate, the released mcdc-evidence tarball, the Pages dashboard
and the FEAT-072 release delta all measure scry_mcdc.wasm — a wasip1 core
module built by Cargo — while the thing we ship and sign is scry.wasm, a
wasip2 component built by Bazel. The wasip1/DWARF choice is deliberate
and documented. Two other skews were not.
BOTH MEASURED ON SHIPPED ARTIFACTS, because a build config is a
hypothesis about an artifact and not evidence about it.
(1) COMPILER. v3.3.0's signed scry-3.3.0-wasm32-wasip2.wasm records
$ wasm-tools metadata show scry-3.3.0-wasm32-wasip2.wasm
name scry_analyzer_component_wasm_lib_release_wasm_base.wasm
processed-by rustc [1.91.1 (ed61e7d7e 2025-11-07)]
confirming MODULE.bazel. The MC/DC subject had NO rust-toolchain.toml
anywhere in the repo and was built by whatever @stable resolved to that
day, so the skew widened on its own and nothing recorded which compiler
produced any given number.
(2) TOOL. v3.3.0's evidence manifest records witness_version "0.30.0",
while the CI gate installs rev 6ff96e75 — which git ls-remote resolves to
v0.29.0^{}, NOT v0.30.0. The floors were proved by one tool version and
the released evidence produced by another. Upstream is at v0.32.0.
#215'S PROPOSED FIX IS NOT POSSIBLE AS WRITTEN, and the measurement is
why: wasmtime 48.0.8 declares rust-version = 1.95.0, so the workspace
cannot be pinned to 1.91.1. It IS possible for the MC/DC subject alone,
because crates/scry-mcdc is its OWN workspace, depends only on
scry-sai-core, has zero wasmtime in its graph, and mcdc-gate.sh does
`cd "$(dirname "$0")"` before building — so a toolchain file there
governs exactly that build and nothing else.
THE SKEW WAS WORTH 13 CONDITIONS. Measured with witness held constant at
the gate's own pinned rev, changing ONLY the compiler:
rustc 1.96.1 -> 1.91.1
decisions_total 1626 -> 1677 (+51)
conditions_proved 395 -> 382 (-13)
decisions_full_mcdc 18 -> 16 (-2)
Both clear the floors (155 / 3) with margin, so the gate stays green. But
the evidence was measurably compiler-dependent while one side floated.
THE FILE IS NOT THE COMPILER. Comparing rust-toolchain.toml against
MODULE.bazel is a claim about what WILL compile — the same
config-as-evidence error this change exists to fix, one level up. rustup
precedence lets RUSTUP_TOOLCHAIN or a directory override beat a toolchain
file, and a green gate under the wrong rustc looks identical to a green
one. So the guard step also asserts the ACTIVE rustc where the build
happens and prints it, which puts the evidence's substrate in the CI log.
Negative-controlled, and the hazard is real rather than theoretical:
RUSTUP_TOOLCHAIN=1.98.1 DOES beat the file (measured: rustc 1.98.1) and
the assertion catches it.
A CONSEQUENCE THE MEASUREMENT EXPOSED: the FEAT-072 release delta builds
the previous tag in a git worktree, which predates the toolchain file —
so the old side would compile under @stable and the new side pinned, and
a ~51-decision compiler artifact would be reported as source change,
which is the one thing that delta exists to isolate. The delta step now
copies the toolchain file into the worktree first, and says so in its
status line.
TWO MORE REFERENCES WERE FOUND BY WRITING THE CHECK. The witness rev also
appears bare in two actions/cache keys:
key: witness-bins-${{ runner.os }}-6ff96e75a1b495aa165f1ee91abda3049fffe628
Bump an install rev and leave a cache key stale, and the cache serves
binaries the install step no longer names — the gate would report
coverage from a witness it does not pin. 4 references found by the first
version of the check, 6 by this one.
WIRED INTO A REQUIRED JOB, which needed a design decision. A check that
is intentionally red cannot be required, and one that is not wired is
gate-potency's defined-but-not-enforced hole. So EXPECTED_SKEWS reports a
known-open skew instead of failing it — and the allowlist is
SELF-CLEANING: an entry whose skew is no longer present FAILS and names
itself, so it cannot outlive the defect. Losing the subject entirely is
never allowlistable, even by its own key. The step sits in
`MC/DC (witness)`, no continue-on-error, no job-level if.
MUTATION-CHECKED ON THE FINAL FILE: 9 mutations (M1-M9), 9 killed,
unmutated control green. An earlier run of 7 did not cover the bare-SHA
scan, which was added afterwards; this is the full suite re-run on what
is committed.
THREE survived a first pass, and every one was a gap in the TABLE rather
than a defect in the code — the same error three times:
* the SHA-format regex was never load-bearing, because every tag case
also carried two DIFFERENT refs, so len(distinct) > 1 blocked them
regardless. Fixed with a case where every ref is an IDENTICAL tag.
* the lost-subject mutation was never tested by its own allowlist key;
the case allowlisted `witness-version`, not `no-refs`.
* dropping the bare-SHA scan's `witness`-line guard survived because no
unrelated 40-hex string exists in the workflows TODAY. scry#232 is
pinning every GitHub Action to a commit SHA, so an unguarded scan
would have started failing CI for no reason the day it merged. Fixed
by running the PARSER against synthetic workflow text, which none of
the other cases did — they all feed `verdict` pre-built tuples.
Each case exercised the OVERLAP of two defences rather than the gap
between them, which is also what survived the first attempt at scry#208.
CORRECTION CARRIED IN THE ARTIFACT: the first draft of this change called
the mcdc-evidence tarball "signed". IT IS NOT. Measured on v3.3.0 — of 20
primary assets, 18 are listed in a cosign-signed SHA256SUMS.txt and 18
also carry individual .sig/.pem pairs; scry-v3.3.0-mcdc-evidence.tar.gz
is in NEITHER and is the ONLY primary asset with no integrity protection.
The error came from filtering the asset list on `mcdc|evidence`, seeing
compliance-evidence.tar.gz return with its .pem and .sig, and carrying
the signing across to the neighbouring bundle. scry#215's own body says
"the evidence bundle we sign and ship", so it was already in circulation;
it also reached witness#360 and has been corrected there.
That makes it a REQUIREMENT VIOLATION, not a gap: REQ-010 — the
requirement this traces to — asks for the evidence "surfaced as a SIGNED
truth-table report". It is surfaced and it is not signed. Filed
separately rather than fixed here; adding an asset to the signing set is
a release-path change and this branch already touches release.yml once.
THE WITNESS SKEW IS LEFT OPEN DELIBERATELY. The v0.30.0 side is
internally coherent — the cargo install --tag and the compliance@v0.30.0
action match each other — so converging either changes the tool that
proved the floors or alters an unsimulated release path. Tracked by the
allowlist, which cannot rot.
VERIFIED (all local; no CI has run on this SHA):
rustc --version in crates/scry-mcdc -> 1.91.1 (ed61e7d7e 2025-11-07),
byte-identical to the shipped component's processed-by string
mcdc-gate.sh with witness 0.29.0 -> exit 0, 382 >= 155, 16 >= 3
check-evidence-substrate.py --self-test -> OK (13 cases + a parser
probe, both outcomes exercised)
check-evidence-substrate.py -> exit 0, 1 known-open skew, nothing stale
check-gate-coverage.py -> OK, 13 crates in both gates (ci.yml edited)
ci.yml and release.yml both still parse as YAML
rivet validate -> PASS
Refs: FEAT-102
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KkNzkNYzPh7366DkNijeNc
avrabe
force-pushed
the
feat215-evidence-substrate
branch
from
November 8, 2026 12:49
9a92931 to
de09aa9
Compare
📐 rivet artifact deltaPR: #236 Base SHA: Validationhead — `rivet validate` resultbase — `rivet validate` result (for comparison)Artifact stats
full stats — headDiff (base → head)AADL model — headPosted by the |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #215 (partially, deliberately — see What is left open). Also files #235 and witness#360, both found by doing this.
Both skews measured on shipped artifacts, not read off a config
A build config is a hypothesis about an artifact, not evidence about it. So both numbers came off the release.
(1) Compiler. v3.3.0's signed component:
which confirms
MODULE.bazel'srust.toolchain(versions = ["1.91.1"]). The MC/DC subject had norust-toolchain.tomlanywhere in the repo and was built by whateverdtolnay/rust-toolchain@stableresolved to that day.(2) Tool. v3.3.0's evidence manifest records
witness_version: "0.30.0", while the CI gate installs rev6ff96e75— whichgit ls-remoteresolves tov0.29.0^{}, not v0.30.0. The floors were proved by one tool version and the shipped evidence produced by another. Upstream is at v0.32.0.#215's proposed fix is not possible as written
"Pin the MC/DC path to the same rustc as MODULE.bazel" cannot apply to the workspace: wasmtime 48.0.8 declares
rust-version = 1.95.0and will not build under 1.91.1.It is possible for the MC/DC subject alone, and the reason is structural:
crates/scry-mcdcis its own workspace (a[workspace]table in itsCargo.toml)scry-sai-core—cargo tree --target wasm32-wasip1shows zero wasmtimemcdc-gate.shdoescd "$(dirname "$0")"before buildingSo a toolchain file there governs exactly that build and nothing else.
The skew was worth 13 conditions
Measured with witness held constant at the gate's own pinned rev, changing only the compiler:
decisions_totalconditions_proveddecisions_full_mcdcBoth clear the floors (155 / 3) with margin, so the gate stays green —
exit 0. But the evidence was measurably compiler-dependent while one side floated, which is #215's claim turned into a number.A consequence that measurement exposed
The FEAT-072 release delta builds the previous tag in a
git worktree, which predates the toolchain file. Without intervention the old side compiles under@stableand the new side pinned — and a ~51-decision compiler artifact would be reported as source change, which is the one thing that delta exists to isolate. The delta step now copies the toolchain file into the worktree first, and says so in its status line.Two more references found by writing the check
The witness rev also appears bare in two
actions/cachekeys:Bump an install rev and leave a cache key stale, and the cache serves binaries the install step no longer names — the gate would report coverage from a witness it does not pin. The check scans bare 40-hex SHAs on any line mentioning witness, the only way those sites are reachable. 4 references found by the first version of the check, 6 by this one.
Wiring needed a design decision
A check that is intentionally red cannot be required; one that is not wired is gate-potency's defined-but-not-enforced hole. So
EXPECTED_SKEWSreports a known-open skew instead of failing it — and the allowlist is self-cleaning: an entry whose skew is no longer present fails and names itself, so it cannot outlive the defect. Losing the subject entirely is never allowlistable, even by its own key.The step sits in
MC/DC (witness)— a required context, nocontinue-on-error, no job-levelif:.Mutation-checked: 9 mutations, 9 killed, control green
Two survived the first round, and both were gaps in the table, not the code — the same error twice:
len(distinct) > 1blocked them regardless. Fixed with a case where every ref is an identical tag.witness-version, notno-refs.Both exercised the overlap of two defences rather than the gap between them — which is also what survived the first attempt at #208.
What is left open
The witness skew, by decision. The v0.30.0 side is internally coherent — the
cargo install --tagand thecompliance@v0.30.0action match each other — so converging them either changes the tool that proved the floors (floors must be re-measured) or alters an unsimulated release path. Tracked by the allowlist, which cannot rot.What the evidence does support, stated so a dossier cannot widen it later: a source-level decision-coverage claim over
crates/scry-analyze-core, measured on a proxy binary now built with the same compiler as the shipped component and by the same witness as the gate. It is not coverage of the shippedscry.wasm— the two differ by target (wasip1 core module vs wasip2 component) and profile (opt-level 1 / debug 2 vs release), deliberately and irreducibly, because witness needs DWARF. The measured 13-condition compiler sensitivity is why that gap is not a formality.Also unchanged: the release's
mcdc-evidencejob runs with floors disabled (its own step name says so); floors are enforced only in CI.Verified
rustc --versionincrates/scry-mcdc1.91.1 (ed61e7d7e 2025-11-07)— byte-identical to the shippedprocessed-bymcdc-gate.shwith witness 0.29.0check-evidence-substrate.py --self-testcheck-evidence-substrate.pycheck-gate-coverage.pyci.ymlwas edited)release.ymlrivet validate🤖 Generated with Claude Code
https://claude.ai/code/session_01KkNzkNYzPh7366DkNijeNc