Repository navigation
Block on a pinned reviewer's own FAILED checklist item under a PASS verdict (sc-8592) - #281
Conversation
…erdict (sc-8592) ## Problem correctness-reviewer runs as a pinned single-pass judge, split into lens parts. It can write a checklist item with `status: 'fail'` and still return `VERDICT: PASS`. Today `enforceChecklistContract` voids that PASS and, under ship, re-runs the judge once on its own. If the retry returns a bare PASS that omits the item, the PASS wins and the defect ships as a non-blocking advisory. That happened on PR #227, where `GIT_ATTR_SOURCE` is ignored on git 2.38–2.39. On a plain `git commit`, the same contradiction went inconclusive, which lets the commit through at exit 2. Telemetry could not count how often a retry flipped FAILED to PASS: a parked attempt emits nothing, and the retried row carried only `retried: true`. ## Change - **A pinned reviewer's own FAILED item is the verdict** (`gate-engine/review/contracts/checklist.mts`). - The outcome becomes FAIL with no re-sample. The existing `reportFailedItems`⚠️ line names the item. - `applyOverrideValve` then runs as for any pinned FAIL. `domainExclusivityDrop` removes out-of-charter security/performance items, a per-lens waiver turns the block into a PASS, and the findings block prints the fix-or-waive command. - Plain `git commit`: exit 2 → 1. - **Unfinished checklists still recover** (`settleUnfinished`, which shares `runCascade`'s `recover` callback). Pending items left after every failed lens was waived or dropped take the ordinary solo recovery: - an exhausted retry fails a ship closed (exit 3) and is a REVIEW ERROR in review mode; - plain commit stays fail-open. - It never reads as a PASS on the first sample. - **Telemetry.** - `failed_under_pass: true` appears on `review_result`, on each `lens_parts` entry and on the merged lens row. - For an unpinned reviewer it is carried through `ParkedRecovery` to the retried or budget-skipped row. - A retry that flipped FAILED to PASS is a row with `retried && status === 'pass' && failed_under_pass`. - **Decision notes.** - `correctness-reviewer-precision`: this settles one sample's own contradiction. No second sample runs, so it is not the sc-2754 union, not Rejected (a) and not Rejected (d). - `review-gate-in-chain`: a cross-reference note. ## Scope assumption Unpinned cascade reviewers (api-security, backend-performance, frontend-*, commit-guard) keep the solo retry and gain only the telemetry flag. Their sc-5614 escalation restores first-pass failed rows, so an opus overturn leaves this same shape. `applyOverrideValve` gives them no waiver path (`overrides.mts:356`). Flipping them directly would turn verified overturns into blocks nobody can waive. Routing their non-escalated FAILED-under-PASS into the verify-only escalation is the open owner decision below. ## Tests `gate-engine/review/__tests__/checklist-recovery.test.mts`, new sc-8592 block: - **Incident replay** (four-way split, ship): exit 1, one judge call for the lens, `failed_under_pass` on the merged row and its lens part. - **Plain commit:** exit 1. - **Waiver:** a waived lens passes. - **Out-of-charter item:** dropped, and the result passes. - **Pending plus waived FAILED item:** the retry passes (exit 0); an exhausted retry gives exit 3 on ship and a REVIEW ERROR at exit 1 in review mode; plain commit stays at exit 2. - **Unpinned reviewer:** still retries, and the retried row carries `retried` and `failed_under_pass`. - **Budget-skipped retry:** keeps the flag and is not marked `retried`. Tests that pinned the old behaviour, rewritten in place: - `run-review.test.mts` (ratcheted, no net lines): exit 2→1. The byte-bound spill fixture now writes pass items for pinned reviewers, because a pinned FAILED item now blocks. So that test no longer checks FAIL rows for the byte bound. - `items-full.test.mts`: the capture test's item is now pass, so the test stays a pure pending-item miss. - The refused-lens test in `checklist-recovery.test.mts`: the FAILED lens now blocks. Evidence: - 919/919 across 23 related test files. - Revert proofs: disabling the flip fails 4 tests; forcing the old no-retry pending branch fails 2; dropping the skip-row flag fails 1. - Diff coverage: 28/29 added executable lines. The one miss is the existing unreachable `throw` in the moved recovery callback. ## Follow-ups (not in this PR) - A FAIL's `review_result.reason` has no byte bound. A fixture with 60 failed lenses wrote a 26 KB event line, beyond the 4 KB atomic window. Real correctness has 4 lenses, but any wide genuine FAIL has the same shape. (chore) - Telemetry does not record whether a flipped PASS also carried ADVISORY lines, which would size the accepted false-block cost below. (feature) - A voided attempt's failed items are missing from the ship digest's "Gate findings this run" block. This predates the change and now applies only to the unpinned path. (bug, sev 3) Accepted cost: a judge that downgraded a defect to ADVISORY but left its item marked `fail` now blocks until it is waived. Story: https://app.shortcut.com/benordlabs/story/8592 🤖 Generated with [Claude Code](https://claude.com/claude-code) Complete against acceptance criteria: NO — unpinned reviewers' FAILED-under-PASS retry: routing it into the opus escalation changes escalation semantics and cost, so it needs the owner's decision
|
No actionable comments were generated in the recent review. Walkthrough
Merge Risk:
|
Problem
correctness-reviewer runs as a pinned single-pass judge, split into lens parts. It can write a checklist item with
status: 'fail'and still returnVERDICT: PASS. TodayenforceChecklistContractvoids that PASS and, under ship, re-runs the judge once on its own. If the retry returns a bare PASS that omits the item, the PASS wins and the defect ships as a non-blocking advisory. That happened on PR #227, whereGIT_ATTR_SOURCEis ignored on git 2.38–2.39. On a plaingit commit, the same contradiction went inconclusive, which lets the commit through at exit 2. Telemetry could not count how often a retry flipped FAILED to PASS: a parked attempt emits nothing, and the retried row carried onlyretried: true.Change
gate-engine/review/contracts/checklist.mts).reportFailedItemsapplyOverrideValvethen runs as for any pinned FAIL.domainExclusivityDropremoves out-of-charter security/performance items, a per-lens waiver turns the block into a PASS, and the findings block prints the fix-or-waive command.git commit: exit 2 → 1.settleUnfinished, which sharesrunCascade'srecovercallback). Pending items left after every failed lens was waived or dropped take the ordinary solo recovery:failed_under_pass: trueappears onreview_result, on eachlens_partsentry and on the merged lens row.ParkedRecoveryto the retried or budget-skipped row.retried && status === 'pass' && failed_under_pass.correctness-reviewer-precision: this settles one sample's own contradiction. No second sample runs, so it is not the sc-2754 union, not Rejected (a) and not Rejected (d).review-gate-in-chain: a cross-reference note.Scope assumption
Unpinned cascade reviewers (api-security, backend-performance, frontend-*, commit-guard) keep the solo retry and gain only the telemetry flag. Their sc-5614 escalation restores first-pass failed rows, so an opus overturn leaves this same shape.
applyOverrideValvegives them no waiver path (overrides.mts:356). Flipping them directly would turn verified overturns into blocks nobody can waive. Routing their non-escalated FAILED-under-PASS into the verify-only escalation is the open owner decision below (sc-8953).Tests
gate-engine/review/__tests__/checklist-recovery.test.mts, new sc-8592 block:failed_under_passon the merged row and its lens part.retriedandfailed_under_pass.retried.Tests that pinned the old behaviour, rewritten in place:
run-review.test.mts(ratcheted, no net lines): exit 2→1. The byte-bound spill fixture now writes pass items for pinned reviewers, because a pinned FAILED item now blocks. So that test no longer checks FAIL rows for the byte bound.items-full.test.mts: the capture test's item is now pass, so the test stays a pure pending-item miss.checklist-recovery.test.mts: the FAILED lens now blocks.Evidence:
throwin the moved recovery callback.Follow-ups (not in this PR)
review_result.reasonhas no byte bound. A fixture with 60 failed lenses wrote a 26 KB event line, beyond the 4 KB atomic window. Real correctness has 4 lenses, but any wide genuine FAIL has the same shape. (chore; autonomous 8a811732)Accepted cost: a judge that downgraded a defect to ADVISORY but left its item marked
failnow blocks until it is waived.Story: https://app.shortcut.com/benordlabs/story/8592
🤖 Generated with Claude Code
Reviewer notes declined:
gate-engine/review/lens/split.mts:397, a cached lens part not carryingfailedUnderPass: a cache replay judges nothing this run, and the flip metric requiresretried, which a replay never sets, so no flip is undercounted. Stamping replays would recount the original attempt's row on every cache hit; cache hits already have their owncache_hitevent (gate-telemetry-self-describing).Not filed: none.
Complete against acceptance criteria: NO — unpinned reviewers' FAILED-under-PASS retry (sc-8953): routing it into the opus escalation changes escalation semantics and cost, so it needs the owner's decision
Automated evidence
Merge verdict: 🟡 human-review · advisory, at
47723daf55ce✅ Regression evidence · captured: 12 of 12 red-failing tests passed on green (setup 7s, red 95s, green 89s, cleanup 3s)
Head
47723daf55ce· basee7bfe421f0fbFailed on red (12 tests)
runReviewGate — checklist recovery on the strict ship path (sc-2088) a FAILED-item lens beside a refused lens blocks on its finding:AssertionError: expected 3 to be 1 // Object.is equalityrunReviewGate — a pinned reviewer's FAILED item under a PASS verdict blocks (sc-8592) the incident replay blocks on the first sample: no retry, flagged on the merged row:AssertionError: expected 3 to be 1 // Object.is equalityrunReviewGate — a pinned reviewer's FAILED item under a PASS verdict blocks (sc-8592) plain git commit blocks at exit 1 instead of failing open at exit 2:AssertionError: expected 2 to be 1 // Object.is equalityrunReviewGate — a pinned reviewer's FAILED item under a PASS verdict blocks (sc-8592) a waiver on the flipped lens turns it into a PASS:AssertionError: expected +0 to be 1 // Object.is equalityrunReviewGate — a pinned reviewer's FAILED item under a PASS verdict blocks (sc-8592) an out-of-charter FAILED item is dropped, not blocked on:AssertionError: expected { type: 'review_result', …(19) } to match object { status: 'pass', …(1) } (20 matching properties omitted from actual)runReviewGate — a pinned reviewer's FAILED item under a PASS verdict blocks (sc-8592) a waived FAILED item beside a pending one retries the pending hole solo:AssertionError: expected +0 to be 1 // Object.is equalityrunReviewGate — a pinned reviewer's FAILED item under a PASS verdict blocks (sc-8592) a pending hole the retry leaves open fails the ship closed, never a PASS:AssertionError: expected +0 to be 1 // Object.is equalityrunReviewGate — a pinned reviewer's FAILED item under a PASS verdict blocks (sc-8592) plain git commit keeps a pending hole beside a waived FAILED item fail-open, never a PASS:AssertionError: expected 2 to be 1 // Object.is equalityrunReviewGate — a pinned reviewer's FAILED item under a PASS verdict blocks (sc-8592) review mode reports an exhausted pending hole as a REVIEW ERROR, not inconclusive:AssertionError: expected +0 to be 1 // Object.is equalityrunReviewGate — a pinned reviewer's FAILED item under a PASS verdict blocks (sc-8592) an unpinned reviewer still retries solo, and its row counts the FAILED→PASS flip:AssertionError: expected { type: 'review_result', …(19) } to match object { status: 'pass', retried: true, …(1) } (19 matching properties omitted from actual)Test command
Captured runs, not proof of causality. Dependencies are the shipping checkout's installed packages. Stale unless the PR head is
47723daf55ce6f7c5f48305d7880432c413e948f.💬 Reviewer notes · 1 note at
47723daf55cegate-engine/lens/split.mtsstatus: 'pass'but withoutfailedUnderPass, so a replayed PASS that held a FAILED item dropsfailed_under_passfrom its merged row, undercounting flips on cache hits. · correctness