Repository navigation
scry#208: the gate-coverage guard stops counting text that merely mentions a gate - #234
Merged
Merged
Conversation
…tions a gate
tools/check-gate-coverage.py exists so scry#141 cannot recur — CI once ran
`cargo test` for 4 of 12 crates and 222 of 273 tests never executed. It
matched with `if tool in line`, a bare substring test, so text that
MENTIONS a gate counted as RUNNING one.
VERIFIED BY EXECUTION BEFORE THE FIX. Three shapes each credited {a, b}
as fully covered:
commented-out credited=True test=['a','b'] clippy=['a','b']
continue-on-error credited=True test=['a','b'] clippy=['a','b']
echoed in a string credited=True test=['a','b'] clippy=[]
The third is not in scry#208; it surfaced while writing the red case for
the other two. A job-level continue-on-error was equally uncounted-for.
THE SCRIPT'S OWN DOCSTRING already listed "a predicate that counted a
comment as a gate" among the failures its YAML rewrite escaped. The
rewrite fixed the awk-range bug (scry#157) and reintroduced that one.
Matching now (a) strips shell comments, (b) requires the tool at the
START of a command segment — splitting on &&, ||, ; and |, skipping
leading VAR=value assignments and a `cargo +toolchain` selector — and
(c) ignores any step or job with continue-on-error: true.
TWO MEASUREMENTS DURING THE MUTATION CHECK CHANGED THE TESTS, and both
are the reason this took more than one pass.
(1) Deleting the comment-stripping line SURVIVED the first four cases.
Requiring the tool at the start of a segment already rejects a line
beginning with `#`, so the two defences overlap and nothing exercised the
gap: a comment carrying a shell SEPARATOR, which splits into a segment
that DOES start with the tool.
(2) The replacement case still did not kill it. In a PLAIN YAML scalar,
`run: cmd # note`, YAML eats the comment before this script ever sees
it — measured directly:
plain scalar run = 'cargo build --workspace'
block scalar run = 'cargo build --workspace # or: cargo test -p a ...'
A shell comment only reaches the matcher inside `run: |`. Both new cases
were rewritten as block scalars, and only then did the mutation die. A
test that cannot reach the code it covers is not a weaker test; it is
not a test.
MUTATION-CHECKED: 7 mutations, 7 killed, unmutated control green.
M1 stop stripping comments KILLED (2 cases)
M2 invokes() back to substring KILLED (2 cases)
M3 credit continue-on-error STEPS KILLED (1 case)
M4 credit continue-on-error JOBS KILLED (1 case)
M5 over-strict: invokes() always False KILLED (10 cases)
M6 drop the +toolchain skip KILLED (1 case)
M7 drop the env-assignment skip KILLED (1 case)
M5 is the one that keeps the fix honest in the other direction: a
matcher that rejects everything would satisfy every negative case, so
six real invocation forms are asserted to still COUNT — plain,
`VAR=1 cargo test`, `cargo +1.99.0 test`, chained after &&, piped into
tee, and a trailing comment that must not leak its -p into the package
set.
POSITIVE CONTROL: against the real ci.yml the guard still reports
`OK — 13 publishable crates in BOTH gates`, exit 0. The fix is not
over-strict.
Refs: FEAT-101
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KkNzkNYzPh7366DkNijeNc
avrabe
force-pushed
the
feat208-gate-coverage-comments
branch
from
November 8, 2026 12:50
eac78b8 to
c39a7ee
Compare
📐 rivet artifact deltaPR: #234 Base SHA: Validationhead — `rivet validate` resultbase — `rivet validate` result (for comparison)Artifact stats
full stats — headDiff (base → head)AADL model — headPosted by the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #208.
tools/check-gate-coverage.pyexists so #141 cannot recur — CI once rancargo testfor 4 of 12 crates and 222 of 273 tests never executed. It matched withif tool in line, a bare substring test, so text that mentions a gate counted as running one.Verified by execution before the fix
Three shapes each credited
{a, b}as fully covered:The third is not in the issue — it surfaced while writing the red case for the other two. A job-level
continue-on-errorwas equally uncounted-for.The script's own docstring already listed "a predicate that counted a comment as a gate" among the failures its YAML rewrite escaped. The rewrite fixed the awk-range bug (#157) and reintroduced that one. Its
--self-testnever fed it a comment.The fix
Matching now:
&&,||,;and|, skipping leadingVAR=valueassignments and acargo +toolchainselectorcontinue-on-error: true— a step that cannot fail is not a gate, which is The Rocq and Verus proof jobs cannot fail anything — the v1.0 'flips to required' never happened #207 one level upTwo measurements during the mutation check changed the tests
This is the part worth reading, because both are ways a test can look right and check nothing.
(1) Deleting the comment-stripping line SURVIVED the first four cases. Requiring the tool at the start of a segment already rejects a line beginning with
#, so the two defences overlap and nothing exercised the gap: a comment carrying a shell separator, which splits into a segment that does start with the tool.(2) The replacement case still did not kill it. In a plain YAML scalar,
run: cmd # note, YAML eats the comment before this script ever sees it — measured directly:A shell comment only reaches the matcher inside
run: |. Both new cases were rewritten as block scalars, and only then did the mutation die. A test that cannot reach the code it covers is not a weaker test; it is not a test.Mutation-checked: 7 mutations, 7 killed, unmutated control green
invokes()back to substringcontinue-on-errorSTEPScontinue-on-errorJOBSinvokes()always False+toolchainskipM5 is the one that keeps the fix honest in the other direction — a matcher that rejects everything would satisfy every negative case. So six real invocation forms are asserted to still count: plain,
VAR=1 cargo test,cargo +1.99.0 test, chained after&&, piped intotee, and a trailing comment that must not leak its-pinto the package set.Positive control: against the real
ci.ymlthe guard still reportsOK — 13 publishable crates in BOTH gates, exit 0. The fix is not over-strict.Re-run on the rebased SHA (
eac78b8): 7/7 still killed, control green, realci.ymlstill accepted. Neithertools/check-gate-coverage.pynorci.ymlchanged on main in the interim (git log a2567b9..main --on both paths is empty).Residual
The guard credits an invocation in any job that can fail, not only in a job whose name is a required status context. A
cargo test -p xin a non-required job still counts as coverage for x. Closing that needs the required-check list, which lives intools/check-required-checks.py— itself #217. Not merged here.Comment stripping is textual: a literal
#inside a quoted shell string would truncate the line. No current workflow line has one, and the alternative is a shell parser.This repairs the guard, not the gate it guards. What ran green while the guard was blind? Nothing — the guard was correct in practice on the real
ci.ymlthroughout; all 13 crates were genuinely in both gates.🤖 Generated with Claude Code
https://claude.ai/code/session_01KkNzkNYzPh7366DkNijeNc