From 47723daf55ce6f7c5f48305d7880432c413e948f Mon Sep 17 00:00:00 2001 From: norvalbv Date: Fri, 20 Nov 2026 02:10:21 +0000 Subject: [PATCH] Block on a pinned reviewer's own FAILED checklist item under a PASS verdict (sc-8592) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Problem correctness-reviewer runs as a pinned single-pass judge, split into lens parts. It can write a checklist item with `status: 'fail'` and still return `VERDICT: PASS`. Today `enforceChecklistContract` voids that PASS and, under ship, re-runs the judge once on its own. If the retry returns a bare PASS that omits the item, the PASS wins and the defect ships as a non-blocking advisory. That happened on PR #227, where `GIT_ATTR_SOURCE` is ignored on git 2.38–2.39. On a plain `git commit`, the same contradiction went inconclusive, which lets the commit through at exit 2. Telemetry could not count how often a retry flipped FAILED to PASS: a parked attempt emits nothing, and the retried row carried only `retried: true`. ## Change - **A pinned reviewer's own FAILED item is the verdict** (`gate-engine/review/contracts/checklist.mts`). - The outcome becomes FAIL with no re-sample. The existing `reportFailedItems` ⚠️ line names the item. - `applyOverrideValve` then runs as for any pinned FAIL. `domainExclusivityDrop` removes out-of-charter security/performance items, a per-lens waiver turns the block into a PASS, and the findings block prints the fix-or-waive command. - Plain `git commit`: exit 2 → 1. - **Unfinished checklists still recover** (`settleUnfinished`, which shares `runCascade`'s `recover` callback). Pending items left after every failed lens was waived or dropped take the ordinary solo recovery: - an exhausted retry fails a ship closed (exit 3) and is a REVIEW ERROR in review mode; - plain commit stays fail-open. - It never reads as a PASS on the first sample. - **Telemetry.** - `failed_under_pass: true` appears on `review_result`, on each `lens_parts` entry and on the merged lens row. - For an unpinned reviewer it is carried through `ParkedRecovery` to the retried or budget-skipped row. - A retry that flipped FAILED to PASS is a row with `retried && status === 'pass' && failed_under_pass`. - **Decision notes.** - `correctness-reviewer-precision`: this settles one sample's own contradiction. No second sample runs, so it is not the sc-2754 union, not Rejected (a) and not Rejected (d). - `review-gate-in-chain`: a cross-reference note. ## Scope assumption Unpinned cascade reviewers (api-security, backend-performance, frontend-*, commit-guard) keep the solo retry and gain only the telemetry flag. Their sc-5614 escalation restores first-pass failed rows, so an opus overturn leaves this same shape. `applyOverrideValve` gives them no waiver path (`overrides.mts:356`). Flipping them directly would turn verified overturns into blocks nobody can waive. Routing their non-escalated FAILED-under-PASS into the verify-only escalation is the open owner decision below. ## Tests `gate-engine/review/__tests__/checklist-recovery.test.mts`, new sc-8592 block: - **Incident replay** (four-way split, ship): exit 1, one judge call for the lens, `failed_under_pass` on the merged row and its lens part. - **Plain commit:** exit 1. - **Waiver:** a waived lens passes. - **Out-of-charter item:** dropped, and the result passes. - **Pending plus waived FAILED item:** the retry passes (exit 0); an exhausted retry gives exit 3 on ship and a REVIEW ERROR at exit 1 in review mode; plain commit stays at exit 2. - **Unpinned reviewer:** still retries, and the retried row carries `retried` and `failed_under_pass`. - **Budget-skipped retry:** keeps the flag and is not marked `retried`. Tests that pinned the old behaviour, rewritten in place: - `run-review.test.mts` (ratcheted, no net lines): exit 2→1. The byte-bound spill fixture now writes pass items for pinned reviewers, because a pinned FAILED item now blocks. So that test no longer checks FAIL rows for the byte bound. - `items-full.test.mts`: the capture test's item is now pass, so the test stays a pure pending-item miss. - The refused-lens test in `checklist-recovery.test.mts`: the FAILED lens now blocks. Evidence: - 919/919 across 23 related test files. - Revert proofs: disabling the flip fails 4 tests; forcing the old no-retry pending branch fails 2; dropping the skip-row flag fails 1. - Diff coverage: 28/29 added executable lines. The one miss is the existing unreachable `throw` in the moved recovery callback. ## Follow-ups (not in this PR) - A FAIL's `review_result.reason` has no byte bound. A fixture with 60 failed lenses wrote a 26 KB event line, beyond the 4 KB atomic window. Real correctness has 4 lenses, but any wide genuine FAIL has the same shape. (chore) - Telemetry does not record whether a flipped PASS also carried ADVISORY lines, which would size the accepted false-block cost below. (feature) - A voided attempt's failed items are missing from the ship digest's "Gate findings this run" block. This predates the change and now applies only to the unpinned path. (bug, sev 3) Accepted cost: a judge that downgraded a defect to ADVISORY but left its item marked `fail` now blocks until it is waived. Story: https://app.shortcut.com/benordlabs/story/8592 🤖 Generated with [Claude Code](https://claude.com/claude-code) Complete against acceptance criteria: NO — unpinned reviewers' FAILED-under-PASS retry: routing it into the opus escalation changes escalation semantics and cost, so it needs the owner's decision --- .../correctness-reviewer-precision.md | 1 + docs/decisions/review-gate-in-chain.md | 1 + .../__tests__/checklist-recovery.test.mts | 207 +++++++++++++++++- .../review/__tests__/items-full.test.mts | 2 +- .../review/__tests__/run-review.test.mts | 10 +- gate-engine/review/cascade/reviewer.mts | 43 ++-- gate-engine/review/contracts/checklist.mts | 30 +++ gate-engine/review/lens/merge-results.mts | 4 + gate-engine/review/lens/split.mts | 1 + gate-engine/review/recovery/settle.mts | 5 + gate-engine/review/run-review.mts | 2 +- gate-engine/review/runtime.mts | 2 + 12 files changed, 271 insertions(+), 37 deletions(-) diff --git a/docs/decisions/correctness-reviewer-precision.md b/docs/decisions/correctness-reviewer-precision.md index 1540f1eaa..f45dd9ee8 100644 --- a/docs/decisions/correctness-reviewer-precision.md +++ b/docs/decisions/correctness-reviewer-precision.md @@ -38,3 +38,4 @@ created: 2026-07-10 - 2026-10-15 — Moved code is supplied as evidence, not arbitrated: reviewer stdin carries a MOVED LINES note from git --color-moved=blocks (sentinel colours) naming source paths and destination ranges, and the reviewer decides whether a defect there predates the move (ADVISORY) or is reached by another change (FAIL). Only cross-file arrivals are listed (a rename's new path counts as the same file): a file that also lost moved lines may be reshuffled in place (a dropped try/finally re-indents its body), which put 12 of 106 correctness FAIL rows in scope; cross-file-only puts 0 of 159 FAIL rows across the five reviewer corpora in scope. Rejected (d) stands: no gate-side downgrade. Evidence built outside cascadeVerdict (completeness, history replay, census) does not carry the note yet. - 2026-10-16 — sc-2328 (autonomous report ad9dceec): a completeness CRITICAL quoted compiled vitest source under node_modules to claim devkit's --retry.count=1 overrides a consumer's config retry, and a fixture refuted it. The story asked each finding to self-label VERIFIED or INFERRED, to downgrade INFERRED claims about third-party runtime behaviour, and to log the label. All three stay closed. Gate judges never execute (judge-mcp-profiles, sc-4709), so VERIFIED is unreachable and the label would be a constant INFERRED; a model self-classification in a typed field is Rejected (c) regardless. A downgrade keyed on the claim's topic is a deterministic arbiter, Rejected (d), and costs recall. A telemetry field would log the same constant. A display-only node_modules/dist citation tag was also declined: completeness output is one prose reason with no per-finding paths, so the tag would need a prose classifier, the approach the sc-2181 note abandoned. Shipped: commit-gates skill guidance only. A judge's runtime claim is an inference to check with a fixture before complying, and since completeness has no waive path, a refuted completeness finding goes to the user with the fixture output and a one-commit GUARD_COMPLETENESS_HARD=0 request. - 2026-10-08 — The class-fix hint now also renders when a security reviewer blocks on its input-validation lens (api-security, frontend-security), not only on error-and-edge-classification: a validator finding from either reviewer names one input of a class. Author guidance only; asking the judge to name the class in one finding stays on the benchmark path. +- 2026-11-20 — sc-8592 (autonomous report 65bae098): a pinned checklist reviewer (correctness and its lens parts) that returns PASS while its own checklist records a FAILED item now blocks on that item instead of being re-run solo. On PR #227 the solo retry returned a bare PASS that omitted the item, and a real defect shipped. This settles one sample's contradiction toward the structured claim the judge itself wrote. No second sample runs, so the sc-2754 deferral (blocking on a union of independent re-runs) and the unanimous-to-block rule are untouched. No judge is re-asked, so it is not Rejected (a). No matcher judges the code, so it is not Rejected (d): domainExclusivityDrop and per-lens waivers apply to the flipped FAIL as to any pinned FAIL. If the artifact still holds a pending item after every failed lens is waived or dropped, the pending hole takes the ordinary solo recovery; it never reads as a PASS on the first sample. Unpinned cascade reviewers keep the solo retry: their sc-5614 escalation restores first-pass failed rows, so an opus overturn leaves this same shape, and they have no waiver path. review_result, lens_parts and the merged row carry failed_under_pass, so retried && pass && failed_under_pass counts a retry that flipped FAILED to PASS. Accepted cost: an issue the judge downgraded to ADVISORY but left marked fail now blocks until it is waived. diff --git a/docs/decisions/review-gate-in-chain.md b/docs/decisions/review-gate-in-chain.md index 3b21f4145..dac60796d 100644 --- a/docs/decisions/review-gate-in-chain.md +++ b/docs/decisions/review-gate-in-chain.md @@ -150,3 +150,4 @@ created: 2026-07-02 - 2026-11-17 — When status, check-item or finalize finds no correctness checklist, the script names the path it read and lists the non-empty lens checklists present as --lens values (a lensed call sees only its own group, so the hint cannot steer one parallel judge into another's file), telling the judge to repeat the command with the flags its prompt shows. It never auto-selects a group, since parallel lens judges share one worktree, and never lists chunk indices or the un-lensed file: chunks share item names, so a chunk menu could steer one judge into another's artifact, and in split mode the gate never verifies the un-lensed file. The hint lives in the correctness script, not the shared checklist store, because the store is hashed into every reviewer's bench and review-mode identity. - 2026-11-19 — sc-8539 — A correctness lens judge's checklist scope is pinned by the gate, not typed by the judge. withStagedFiles exports DEVKIT_CORRECTNESS_LENS (the sorted group) and DEVKIT_CORRECTNESS_CHUNK into each derived lens judge's spawn env, cleared first for every other checklist reviewer. checklist.mjs takes its scope from that env when set: a bare generate/check-item/finalize works the lens-scoped state file the gate verifies, a flag repeating the scope runs, and a flag naming another group or chunk exits 1 leaving every checklist untouched (also sc-6705). Cause: since gateBrief rewrote the brief's bare commands into the granted literal, haiku lens judges ran them without --lens, wrote the un-scoped file, and 7 of 8 'checklist artifact missing' ships on 2026-11-17..19 failed closed on PASS verdicts. This qualifies the 2026-11-17 'never auto-select a group' clause: that held because the script could not tell which judge called it; the env now tells it. Interactive and monolith runs carry no env and behave as before. A judge that never calls the script stays a contract miss. - 2026-11-19 — The zero-scope notice now leads with SKIPPED (`guard-review: SKIPPED — no domain reviewer ran; …`), the token the ship's end-of-run banner tells a reader to scan for, and says 'domain' so it does not contradict completeness's own PASS line. Each spawned deterministic gate closes its `▶ deterministic:` header with one runner-printed verdict (passed / failed / could not run / opted out / BYPASSED / DEGRADED), derived from the same facts as its gate_step row, where a gate that verified nothing reads SKIPPED, BYPASSED or DEGRADED. The reviewer gate names the reviewers it selected on one roster line, per-reviewer skips print SKIPPED, and a replayed completeness PASS says whether this ship run or an earlier one judged it (ship_id stored with the PASS); the ship digest stays silent on not_selected under gate-verdict-attribution's silence clause. +- 2026-11-20 — sc-8592: checklist contract recovery (sc-1476/sc-2088) no longer covers a pinned reviewer's FAILED item under a PASS verdict. enforceChecklistContract returns FAIL through the override valve and schedules no retry. Pending and missing artifacts still recover solo. The ruling is recorded on correctness-reviewer-precision. diff --git a/gate-engine/review/__tests__/checklist-recovery.test.mts b/gate-engine/review/__tests__/checklist-recovery.test.mts index 56290578e..a19dd36ad 100644 --- a/gate-engine/review/__tests__/checklist-recovery.test.mts +++ b/gate-engine/review/__tests__/checklist-recovery.test.mts @@ -541,8 +541,8 @@ describe('runReviewGate — checklist recovery on the strict ship path (sc-2088) expect(remedies[0]).toContain('re-run devkit ship, and if it repeats move that judge'); }); - // A FAILED item is a candidate violation, so a refused sibling lens must not reword its remedy. - it('a refused lens beside a FAILED-item lens keeps both remedies', async () => { + // A pinned lens's FAILED item blocks on its own; a refused sibling must not soften that block. + it('a FAILED-item lens beside a refused lens blocks on its finding', async () => { delete process.env.GUARD_CORRECTNESS_SPLIT; const repo = consumerRepo({ backend: true }); shipEnv(repo); @@ -562,11 +562,10 @@ describe('runReviewGate — checklist recovery on the strict ship path (sc-2088) } return 'VERDICT: PASS'; }); - expect(await runReviewGate(repo, { exec })).toBe(3); - const remedies = err.mock.calls.flat().filter((l) => String(l).startsWith(' Remedy:')); - expect(remedies).toHaveLength(2); - expect(remedies.join('\n')).toContain('if it repeats move that judge'); - expect(remedies.join('\n')).toContain('fix a real violation if present'); + expect(await runReviewGate(repo, { exec })).toBe(1); + const out = err.mock.calls.flat().join('\n'); + expect(out).toContain(`[${lensGroupId(failing)}] — FAILED checklist item under a PASS verdict`); + expect(out).not.toContain('fix a real violation if present'); }); it('a lens refused on the wave that skips again with no refusal fails closed on the same model', async () => { @@ -1255,3 +1254,197 @@ describe('runReviewGate — a voided PASS prints its FAILED items (sc-5720)', () expect(err).toHaveBeenCalledWith('⚠️ guard-review: api-security-reviewer — (+2 more)'); }); }); + +describe("runReviewGate — a pinned reviewer's FAILED item under a PASS verdict blocks (sc-8592)", () => { + const CORRECTNESS = 'review:correctness-reviewer'; + const ITEM_LINE = 'FAILED checklist item under a PASS verdict: error-and-edge-classification'; + const shipEnv = (repo: string) => { + process.env.GUARD_AI_STRICT = '1'; + process.env.DEVKIT_GATE_EVENTS = join(repo, 'events.jsonl'); + process.env.DEVKIT_SHIP_ID = 'ship-sc-8592'; + }; + const reviewResult = (repo: string, reviewer: string) => + readFileSync(join(repo, 'events.jsonl'), 'utf8') + .trim() + .split('\n') + .map((l) => JSON.parse(l)) + .findLast((e) => e.type === 'review_result' && e.reviewer === reviewer); + // correctness says PASS over `items` on its first attempt and a clean artifact on any retry. + const contradicting = (repo: string, items: object[], target = CORRECTNESS) => { + let attempt = 0; + return mkExec(async ({ label }) => { + writeArtifact(repo, label); + if (label === target && ++attempt === 1) + writeFileSync(join(repo, reviewerFromLabel(label).stateFile), JSON.stringify({ items })); + return 'VERDICT: PASS'; + }); + }; + const edgeFail = { + name: 'error-and-edge-classification', + status: 'fail', + issues: ['0 read as absent'], + }; + + it('the incident replay blocks on the first sample: no retry, flagged on the merged row', async () => { + delete process.env.GUARD_CORRECTNESS_SPLIT; // the shipped four-way default + const repo = consumerRepo({ backend: true }); + shipEnv(repo); + const err = vi.spyOn(console, 'error').mockImplementation(() => {}); + const group = FOUR_WAY_LENS_GROUPS.find((g) => g.includes('error-and-edge-classification'))!; + const groupCalls: string[] = []; + const exec = mkExec(async ({ label, args }) => { + writeArtifact(repo, label); + if (label !== CORRECTNESS || !args[1].includes(`--lens ${[...group].sort().join(',')}`)) + return 'VERDICT: PASS'; + groupCalls.push(args[1]); + writeFileSync( + join(repo, `.claude/.correctness-review-${lensGroupId(group)}.json`), + JSON.stringify({ items: groupCalls.length === 1 ? [edgeFail] : [] }), + ); + return 'VERDICT: PASS'; + }); + expect(await runReviewGate(repo, { exec })).toBe(1); + expect(groupCalls).toHaveLength(1); + const out = err.mock.calls.flat().join('\n'); + expect(out).toContain(`[${lensGroupId(group)}] — ${ITEM_LINE}`); + expect(out).toContain('• error-and-edge-classification ['); // the findings block names it + const row = reviewResult(repo, 'correctness-reviewer'); + expect(row).toMatchObject({ status: 'fail', failed_under_pass: true }); + const part = row.lens_parts.find((p) => p.lens === lensGroupId(group)); + expect(part).toMatchObject({ status: 'fail', failed_under_pass: true }); + expect(row.lens_parts.filter((p) => p.failed_under_pass)).toHaveLength(1); + }); + + it('plain git commit blocks at exit 1 instead of failing open at exit 2', async () => { + const repo = consumerRepo({ backend: true }); + vi.spyOn(console, 'error').mockImplementation(() => {}); + expect(await runReviewGate(repo, { exec: contradicting(repo, [edgeFail]) })).toBe(1); + }); + + it('a waiver on the flipped lens turns it into a PASS', async () => { + const repo = consumerRepo({ backend: true }); + shipEnv(repo); + vi.spyOn(console, 'error').mockImplementation(() => {}); + expect(await runReviewGate(repo, { exec: contradicting(repo, [edgeFail]) })).toBe(1); + const fp = reviewResult(repo, 'correctness-reviewer').blocking[0].fp; + process.env[`OVERRIDE_${fp}_RATIONALE`] = 'absent and zero are the same state here'; + try { + expect(await runReviewGate(repo, { exec: contradicting(repo, [edgeFail]) })).toBe(0); + } finally { + delete process.env[`OVERRIDE_${fp}_RATIONALE`]; + } + }); + + it('an out-of-charter FAILED item is dropped, not blocked on', async () => { + const repo = consumerRepo({ backend: true }); + shipEnv(repo); + vi.spyOn(console, 'error').mockImplementation(() => {}); + const sqli = { + name: 'writer-reader-contracts', + status: 'fail', + issues: ['sql injection via the query string'], + }; + expect(await runReviewGate(repo, { exec: contradicting(repo, [sqli]) })).toBe(0); + expect(reviewResult(repo, 'correctness-reviewer')).toMatchObject({ + status: 'pass', + failed_under_pass: true, + }); + }); + + // Waiving the FAILED item leaves only the pending hole, which takes the ordinary solo recovery. + const pendingWithWaivedFail = async (repo: string, retries: number) => { + const items = [edgeFail, { name: 'state-transitions', status: 'pending', issues: [] }]; + const err = vi.spyOn(console, 'error').mockImplementation(() => {}); + expect(await runReviewGate(repo, { exec: contradicting(repo, items) })).toBe(1); + const fp = err.mock.calls + .flat() + .join('\n') + .match(/• error-and-edge-classification \[(\w+)\]/)?.[1]; + process.env[`OVERRIDE_${fp}_RATIONALE`] = 'absent and zero are the same state here'; + let attempt = 0; + const exec = mkExec(async ({ label }) => { + writeArtifact(repo, label); + if (label === CORRECTNESS && ++attempt <= 1 + retries) + writeFileSync(join(repo, reviewerFromLabel(label).stateFile), JSON.stringify({ items })); + return 'VERDICT: PASS'; + }); + try { + return { code: await runReviewGate(repo, { exec }), calls: () => attempt }; + } finally { + delete process.env[`OVERRIDE_${fp}_RATIONALE`]; + } + }; + + it('a waived FAILED item beside a pending one retries the pending hole solo', async () => { + const repo = consumerRepo({ backend: true }); + shipEnv(repo); + vi.spyOn(console, 'error').mockImplementation(() => {}); + const { code, calls } = await pendingWithWaivedFail(repo, 0); + expect(code).toBe(0); + expect(calls()).toBe(2); + const row = reviewResult(repo, 'correctness-reviewer'); + expect(row).toMatchObject({ status: 'pass', retried: true }); + expect(row.failed_under_pass).toBeUndefined(); // waived, not erased by the retry + }); + + it('a pending hole the retry leaves open fails the ship closed, never a PASS', async () => { + const repo = consumerRepo({ backend: true }); + shipEnv(repo); + vi.spyOn(console, 'error').mockImplementation(() => {}); + expect((await pendingWithWaivedFail(repo, 1)).code).toBe(3); + const row = reviewResult(repo, 'correctness-reviewer'); + expect(row).toMatchObject({ status: 'inconclusive', inconclusive_cause: 'response-contract' }); + expect(row.reason).toContain('1 item(s) never resolved: state-transitions'); + }); + + it('plain git commit keeps a pending hole beside a waived FAILED item fail-open, never a PASS', async () => { + const repo = consumerRepo({ backend: true }); + const { code, calls } = await pendingWithWaivedFail(repo, 0); + expect(code).toBe(2); + expect(calls()).toBe(1); + }); + + it('review mode reports an exhausted pending hole as a REVIEW ERROR, not inconclusive', async () => { + const repo = unprojected(consumerRepo({ backend: true })); + process.env.DEVKIT_RUN_MODE = 'review'; + process.env.DEVKIT_REVIEW_ASSET_ROOT = reviewAssets(); + process.env.GUARD_AI_STRICT = '1'; // review runs under the same strict shell as ship + const err = vi.spyOn(console, 'error').mockImplementation(() => {}); + const { code, calls } = await pendingWithWaivedFail(repo, 1); + expect(code).toBe(1); + expect(calls()).toBe(2); + expect(err.mock.calls.flat().join('\n')).toContain('checklist contract failed after one retry'); + }); + + it('an unpinned reviewer still retries solo, and its row counts the FAILED→PASS flip', async () => { + const repo = consumerRepo({ backend: true }); + shipEnv(repo); + vi.spyOn(console, 'error').mockImplementation(() => {}); + const target = 'review:api-security-reviewer'; + const exec = contradicting( + repo, + [{ name: 'input-validation', status: 'fail', issues: [] }], + target, + ); + expect(await runReviewGate(repo, { exec })).toBe(0); + expect(exec.mock.calls.filter(([o]) => o.label === target)).toHaveLength(2); + expect(reviewResult(repo, 'api-security-reviewer')).toMatchObject({ + status: 'pass', + retried: true, + failed_under_pass: true, + }); + }); + + it('a budget-skipped retry still records that the voided attempt held a FAILED item', async () => { + const repo = consumerRepo({ backend: true }); + shipEnv(repo); + vi.spyOn(console, 'error').mockImplementation(() => {}); + process.env.DEVKIT_GATE_DEADLINE_MS = String(Date.now() + 5_000); + const target = 'review:api-security-reviewer'; + const exec = contradicting(repo, [{ name: 'input-validation', status: 'fail' }], target); + expect(await runReviewGate(repo, { exec })).toBe(3); + const row = reviewResult(repo, 'api-security-reviewer'); + expect(row).toMatchObject({ status: 'inconclusive', failed_under_pass: true }); + expect(row.retried).toBeUndefined(); + }); +}); diff --git a/gate-engine/review/__tests__/items-full.test.mts b/gate-engine/review/__tests__/items-full.test.mts index 794d119ce..30d5534df 100644 --- a/gate-engine/review/__tests__/items-full.test.mts +++ b/gate-engine/review/__tests__/items-full.test.mts @@ -87,7 +87,7 @@ describe('attachItems itemsFull', () => { stateFile, JSON.stringify({ items: [ - { name: 'state-transitions', status: 'fail', issues: [text] }, + { name: 'state-transitions', status: 'pass', issues: [text] }, { name: 'concurrency-races', status: 'pending' }, ], }), diff --git a/gate-engine/review/__tests__/run-review.test.mts b/gate-engine/review/__tests__/run-review.test.mts index 1705804e3..215d93c31 100644 --- a/gate-engine/review/__tests__/run-review.test.mts +++ b/gate-engine/review/__tests__/run-review.test.mts @@ -767,7 +767,7 @@ describe('runReviewGate — cascade + exit contract', () => { const rows = Array.from({ length: 60 }, (_, i) => ({ [key === 'files' ? 'path' : 'name']: `src/main/deeply/nested/module-${i}/implementation.ts`, - status: 'fail', + status: reviewer.model ? 'pass' : 'fail', // a pinned reviewer's FAILED item blocks issues: [`issue A ${'x'.repeat(240)}`, `issue B ${'y'.repeat(240)}`], })); writeFileSync(join(repo, reviewer.stateFile), JSON.stringify({ [key]: rows })); @@ -792,7 +792,7 @@ describe('runReviewGate — cascade + exit contract', () => { // The counts and the per-status tally survive the spill — otherwise "did these lenses fire" // would be unanswerable exactly on the widest commits. expect(r.item_count).toBe(60); - expect(r.item_tally.fail).toBe(40); // the 40-item cap, all non-passing + expect(r.item_tally.fail ?? r.item_tally.pass).toBe(40); // the 40-item cap, one status } }); @@ -1117,16 +1117,16 @@ describe('runReviewGate — cascade + exit contract', () => { expect(err.mock.calls.flat().join('\n')).toContain('2 item(s) never resolved'); }); - it('PASS verdict but the checklist recorded FAILED items → inconclusive (mismatch), not cached', async () => { + it('PASS verdict over FAILED items: the pinned reviewer blocks, the rest void; none cached', async () => { const repo = consumerRepo({ backend: true }); vi.spyOn(console, 'error').mockImplementation(() => {}); const exec = mkExec(async ({ label }) => { writeArtifact(repo, label, { failed: 1 }); return 'VERDICT: PASS'; }); - expect(await runReviewGate(repo, { exec })).toBe(2); + expect(await runReviewGate(repo, { exec })).toBe(1); // conventions-reviewer has no checklist artifact to mismatch against — its PASS caches (1); - // the 4 checklist-driven reviewers void to inconclusive on the FAILED-item mismatch. + // correctness blocks on its own FAILED item; the cascade reviewers void to inconclusive. expect(Object.keys(loadCache(repo)).length).toBe(1); }); diff --git a/gate-engine/review/cascade/reviewer.mts b/gate-engine/review/cascade/reviewer.mts index c70de83fb..202c0dc10 100644 --- a/gate-engine/review/cascade/reviewer.mts +++ b/gate-engine/review/cascade/reviewer.mts @@ -32,6 +32,7 @@ import { enforceChecklistContract, reportDeniedCalls, retrievalDegradation, + settleUnfinished, type SpawnDenials, } from '../contracts/checklist.mts'; import { @@ -155,34 +156,30 @@ export async function runCascade( // Recovery below only schedules/classifies; it deletes this attempt's artifact without // running another judge. Keep its exact private evidence for the resulting inconclusive row. const captureState = opts.fullItems ? readChecklistState(cwd, sel.reviewer) : null; - res = await enforceChecklistContract( - sel, - res, - cwd, - opts.recovery !== undefined, - async (reason) => { - if (opts.recovery === 'defer') - return { ...res, status: 'inconclusive', reason, retryable: reason } as CascadeResult; - // Exhausted recovery: review blocks at exit 1 (REVIEW ERROR); ship stays inconclusive at - // exit 3, its strict retry/fail-closed contract. `assetRoot` is set only by review mode. - if (opts.recovery === 'final') { - const exhausted = `reviewer checklist contract failed after one retry — ${reason}`; - const terminal: CascadeResult = opts.assetRoot - ? { ...res, status: 'error', reason: exhausted } - : { ...res, status: 'inconclusive', inconclusiveCause: undefined, reason: exhausted }; - return terminal; - } - // Unreachable from the three direct callers (the reviewer/conventions/scale benches): they - // schedule no recovery, so enforceChecklistContract never invokes this callback at all. - throw new Error(`checklist recovery has no scheduling mode — ${reason}`); - }, - spawn.denied, - ); + const recover = async (reason: string): Promise => { + if (opts.recovery === 'defer') + return { ...res, status: 'inconclusive', reason, retryable: reason } as CascadeResult; + // Exhausted recovery: review blocks at exit 1 (REVIEW ERROR); ship stays inconclusive at + // exit 3, its strict retry/fail-closed contract. `assetRoot` is set only by review mode. + if (opts.recovery === 'final') { + const exhausted = `reviewer checklist contract failed after one retry — ${reason}`; + const terminal: CascadeResult = opts.assetRoot + ? { ...res, status: 'error', reason: exhausted } + : { ...res, status: 'inconclusive', inconclusiveCause: undefined, reason: exhausted }; + return terminal; + } + // Unreachable from the three direct callers (the reviewer/conventions/scale benches): they + // schedule no recovery, so enforceChecklistContract never invokes this callback at all. + throw new Error(`checklist recovery has no scheduling mode — ${reason}`); + }; + const scheduled = opts.recovery !== undefined; + res = await enforceChecklistContract(sel, res, cwd, scheduled, recover, spawn.denied); const disposition = applyOverrideValve(sel, res, cwd, { readState: () => readChecklistState(cwd, sel.reviewer), stagedDiff: () => gitCached(cwd, [], sel.files), }); const finalState = readChecklistState(cwd, sel.reviewer) ?? captureState; + res = await settleUnfinished(res, finalState, scheduled, recover); if (!retryableReason(res)) reportDeniedCalls(sel.reviewer, res, spawn.denied, finalState); // A FAIL keeps the uncut issue text so the findings block can print what the author must fix. attachItems(res, finalState, disposition, { full: opts.fullItems || res.status === 'fail' }); diff --git a/gate-engine/review/contracts/checklist.mts b/gate-engine/review/contracts/checklist.mts index 6f0675fc7..28df1e203 100644 --- a/gate-engine/review/contracts/checklist.mts +++ b/gate-engine/review/contracts/checklist.mts @@ -44,6 +44,14 @@ export async function enforceChecklistContract( const initialState = readChecklistState(cwd, selection.reviewer); const hole = verifyChecklist(initialState, 'PASS', denied); if (hole) reportFailedItems(selection.reviewer, initialState); + const failed = failedItems(initialState); + if (failed.length > 0) result.failedUnderPass = true; + // A pinned judge's own FAILED item is the verdict (a re-sample could erase it); the override + // valve then drops out-of-charter lenses, honours waivers and writes the reason. + if (failed.length > 0 && selection.reviewer.model) { + result.status = 'fail'; + return result; + } const refused = refusedCalls(initialState, denied); if (hole && recoveryScheduled) { // No "retrying" here: the callback runs no judge, and the solo phase announces itself. @@ -69,6 +77,28 @@ export async function enforceChecklistContract( return result; } +/** A waived or dropped FAILED item leaves the judge's pending items: they recover as any pending + * hole does, so an unfinished artifact never reads as a PASS. */ +export async function settleUnfinished( + res: ReviewOutcome, + state: ChecklistState | null, + recoveryScheduled: boolean, + retry: (reason: string) => Promise, +): Promise { + const items = state?.items ?? state?.files; + if (!res.failedUnderPass || res.status !== 'pass' || !Array.isArray(items)) return res; + if (!items.some((i) => i.status === 'pending')) return res; + const hole = verifyChecklist(state, 'PASS') ?? 'checklist incomplete'; + // The FAILED item was waived or dropped, not erased: a retry PASS here is no FAILED→PASS flip. + delete res.failedUnderPass; + const out = recoveryScheduled + ? await retry(hole) + : { ...res, status: 'inconclusive' as const, reason: hole }; + if (out.status === 'inconclusive' && out.inconclusiveCause === undefined) + markContractMiss(out, undefined); + return out; +} + function markContractMiss(result: ReviewOutcome, refused: number | undefined): void { result.inconclusiveCause = 'response-contract'; if (refused) result.refusedCalls = refused; diff --git a/gate-engine/review/lens/merge-results.mts b/gate-engine/review/lens/merge-results.mts index 4a27d9798..310f6a73f 100644 --- a/gate-engine/review/lens/merge-results.mts +++ b/gate-engine/review/lens/merge-results.mts @@ -32,6 +32,7 @@ type LensPartRow = { chunk_files_sha: string | null; model?: string; retried?: true; + failed_under_pass?: true; cached?: true; } & CoverageFields; @@ -49,6 +50,7 @@ type MergedLensRow = { lens_parts: LensPartRow[]; retried?: true; retry_phase?: 'deferred'; + failed_under_pass?: true; waivers?: unknown[]; transcript_ref?: string; }; @@ -97,6 +99,7 @@ export function emitMergedLensResults( }; if (p.res.model) row.model = p.res.model; if (p.retried) row.retried = true; + if (p.res.failedUnderPass) row.failed_under_pass = true; if (p.cached) row.cached = true; return row; }); @@ -118,6 +121,7 @@ export function emitMergedLensResults( event.retried = true; event.retry_phase = 'deferred'; } + if (parts.some((p) => p.res.failedUnderPass)) event.failed_under_pass = true; if (merged.waivers?.length) event.waivers = merged.waivers; // SAFETY: `merged` carries the ReviewOutcome item fields mergeItemVectors just rebuilt. const items = itemFields(merged as never); diff --git a/gate-engine/review/lens/split.mts b/gate-engine/review/lens/split.mts index 4fac5503f..881a52dcc 100644 --- a/gate-engine/review/lens/split.mts +++ b/gate-engine/review/lens/split.mts @@ -167,6 +167,7 @@ export type LensPart = { model?: string; items?: unknown[]; mcpDegraded?: { cause: string; cached?: true }; // cached: replayed, not judged this run + failedUnderPass?: true; }; secs: number; /** The task this part ran. Carries the diff AND the derived reviewer, whose `lens` is the only diff --git a/gate-engine/review/recovery/settle.mts b/gate-engine/review/recovery/settle.mts index 75eb7532c..e663bf8f5 100644 --- a/gate-engine/review/recovery/settle.mts +++ b/gate-engine/review/recovery/settle.mts @@ -165,6 +165,7 @@ export function settleReviewOutcome( // the fix would erase the field rate of the very failure mode it schedules around. NEVER in // any cache key or salt — a recovered PASS must serve the next run's hit. ...(retried ? { retried: true, retry_phase: 'deferred' } : {}), + failed_under_pass: res.failedUnderPass, // dropped by JSON when unset ...(res.waivers?.length ? { waivers: res.waivers } : {}), // The per-lens vector, passes included; empty when the judge left no artifact (see // evidence/items.mts for the shape and the spill rule). @@ -226,6 +227,8 @@ export interface ParkedRecovery { task: ReviewTask; reason: string; index: number; + /** The voided attempt held a FAILED item, so its retry's row can show a FAILED→PASS flip. */ + failedUnderPass?: true; } /** @@ -251,6 +254,7 @@ export async function runDeferredRecoveries( reason: `recovery deferred, budget exhausted — re-run to converge (${p.reason})`, inconclusiveCause: 'timeout', escalated: false, + failedUnderPass: p.failedUnderPass, }; // NOT marked retried: no deferred judge ran. The flag measures attempts that happened, so // counting a skip would inflate the very recovery rate it exists to report — the reason @@ -263,6 +267,7 @@ export async function runDeferredRecoveries( ); const t0 = Date.now(); const res = await runOne(p.task, p.reason, remaining - RECOVERY_SETTLE_MARGIN_MS); + if (p.failedUnderPass) res.failedUnderPass = true; results[p.index] = settleReviewOutcome(ctx, p.task, res, Date.now() - t0, true); } } diff --git a/gate-engine/review/run-review.mts b/gate-engine/review/run-review.mts index 62be68ef9..a5f02c213 100644 --- a/gate-engine/review/run-review.mts +++ b/gate-engine/review/run-review.mts @@ -429,7 +429,7 @@ export async function runReviewGate( .then((outcome) => { const res = settleReviewOutcome(sctx, t, outcome, Date.now() - t0); const reason = retryableReason(res); - if (reason) parked.push({ task: t, reason, index }); + if (reason) parked.push({ task: t, reason, index, failedUnderPass: res.failedUnderPass }); return res; }); }); diff --git a/gate-engine/review/runtime.mts b/gate-engine/review/runtime.mts index 726966e95..d36920f01 100644 --- a/gate-engine/review/runtime.mts +++ b/gate-engine/review/runtime.mts @@ -202,6 +202,8 @@ export interface ReviewOutcome { /** Refused tool calls of the spawn behind a checklist contract miss; unset when unmeasured or * the artifact holds a FAILED item, which is a candidate violation rather than a judge-side miss. */ refusedCalls?: number; + /** The judge said PASS while its own checklist held a FAILED item (this attempt or a voided one). */ + failedUnderPass?: true; transcript?: string; /** Both passes of an escalated run, each labelled with its model; archived instead of `transcript`. */ fullTranscript?: string;