Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/decisions/correctness-reviewer-precision.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,4 @@ created: 2026-07-10
- 2026-10-15 — Moved code is supplied as evidence, not arbitrated: reviewer stdin carries a MOVED LINES note from git --color-moved=blocks (sentinel colours) naming source paths and destination ranges, and the reviewer decides whether a defect there predates the move (ADVISORY) or is reached by another change (FAIL). Only cross-file arrivals are listed (a rename's new path counts as the same file): a file that also lost moved lines may be reshuffled in place (a dropped try/finally re-indents its body), which put 12 of 106 correctness FAIL rows in scope; cross-file-only puts 0 of 159 FAIL rows across the five reviewer corpora in scope. Rejected (d) stands: no gate-side downgrade. Evidence built outside cascadeVerdict (completeness, history replay, census) does not carry the note yet.
- 2026-10-16 — sc-2328 (autonomous report ad9dceec): a completeness CRITICAL quoted compiled vitest source under node_modules to claim devkit's --retry.count=1 overrides a consumer's config retry, and a fixture refuted it. The story asked each finding to self-label VERIFIED or INFERRED, to downgrade INFERRED claims about third-party runtime behaviour, and to log the label. All three stay closed. Gate judges never execute (judge-mcp-profiles, sc-4709), so VERIFIED is unreachable and the label would be a constant INFERRED; a model self-classification in a typed field is Rejected (c) regardless. A downgrade keyed on the claim's topic is a deterministic arbiter, Rejected (d), and costs recall. A telemetry field would log the same constant. A display-only node_modules/dist citation tag was also declined: completeness output is one prose reason with no per-finding paths, so the tag would need a prose classifier, the approach the sc-2181 note abandoned. Shipped: commit-gates skill guidance only. A judge's runtime claim is an inference to check with a fixture before complying, and since completeness has no waive path, a refuted completeness finding goes to the user with the fixture output and a one-commit GUARD_COMPLETENESS_HARD=0 request.
- 2026-10-08 — The class-fix hint now also renders when a security reviewer blocks on its input-validation lens (api-security, frontend-security), not only on error-and-edge-classification: a validator finding from either reviewer names one input of a class. Author guidance only; asking the judge to name the class in one finding stays on the benchmark path.
- 2026-11-20 — sc-8592 (autonomous report 65bae098): a pinned checklist reviewer (correctness and its lens parts) that returns PASS while its own checklist records a FAILED item now blocks on that item instead of being re-run solo. On PR #227 the solo retry returned a bare PASS that omitted the item, and a real defect shipped. This settles one sample's contradiction toward the structured claim the judge itself wrote. No second sample runs, so the sc-2754 deferral (blocking on a union of independent re-runs) and the unanimous-to-block rule are untouched. No judge is re-asked, so it is not Rejected (a). No matcher judges the code, so it is not Rejected (d): domainExclusivityDrop and per-lens waivers apply to the flipped FAIL as to any pinned FAIL. If the artifact still holds a pending item after every failed lens is waived or dropped, the pending hole takes the ordinary solo recovery; it never reads as a PASS on the first sample. Unpinned cascade reviewers keep the solo retry: their sc-5614 escalation restores first-pass failed rows, so an opus overturn leaves this same shape, and they have no waiver path. review_result, lens_parts and the merged row carry failed_under_pass, so retried && pass && failed_under_pass counts a retry that flipped FAILED to PASS. Accepted cost: an issue the judge downgraded to ADVISORY but left marked fail now blocks until it is waived.
1 change: 1 addition & 0 deletions docs/decisions/review-gate-in-chain.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,3 +150,4 @@ created: 2026-07-02
- 2026-11-17 — When status, check-item or finalize finds no correctness checklist, the script names the path it read and lists the non-empty lens checklists present as --lens values (a lensed call sees only its own group, so the hint cannot steer one parallel judge into another's file), telling the judge to repeat the command with the flags its prompt shows. It never auto-selects a group, since parallel lens judges share one worktree, and never lists chunk indices or the un-lensed file: chunks share item names, so a chunk menu could steer one judge into another's artifact, and in split mode the gate never verifies the un-lensed file. The hint lives in the correctness script, not the shared checklist store, because the store is hashed into every reviewer's bench and review-mode identity.
- 2026-11-19 — sc-8539 — A correctness lens judge's checklist scope is pinned by the gate, not typed by the judge. withStagedFiles exports DEVKIT_CORRECTNESS_LENS (the sorted group) and DEVKIT_CORRECTNESS_CHUNK into each derived lens judge's spawn env, cleared first for every other checklist reviewer. checklist.mjs takes its scope from that env when set: a bare generate/check-item/finalize works the lens-scoped state file the gate verifies, a flag repeating the scope runs, and a flag naming another group or chunk exits 1 leaving every checklist untouched (also sc-6705). Cause: since gateBrief rewrote the brief's bare commands into the granted literal, haiku lens judges ran them without --lens, wrote the un-scoped file, and 7 of 8 'checklist artifact missing' ships on 2026-11-17..19 failed closed on PASS verdicts. This qualifies the 2026-11-17 'never auto-select a group' clause: that held because the script could not tell which judge called it; the env now tells it. Interactive and monolith runs carry no env and behave as before. A judge that never calls the script stays a contract miss.
- 2026-11-19 — The zero-scope notice now leads with SKIPPED (`guard-review: SKIPPED — no domain reviewer ran; …`), the token the ship's end-of-run banner tells a reader to scan for, and says 'domain' so it does not contradict completeness's own PASS line. Each spawned deterministic gate closes its `▶ deterministic:` header with one runner-printed verdict (passed / failed / could not run / opted out / BYPASSED / DEGRADED), derived from the same facts as its gate_step row, where a gate that verified nothing reads SKIPPED, BYPASSED or DEGRADED. The reviewer gate names the reviewers it selected on one roster line, per-reviewer skips print SKIPPED, and a replayed completeness PASS says whether this ship run or an earlier one judged it (ship_id stored with the PASS); the ship digest stays silent on not_selected under gate-verdict-attribution's silence clause.
- 2026-11-20 — sc-8592: checklist contract recovery (sc-1476/sc-2088) no longer covers a pinned reviewer's FAILED item under a PASS verdict. enforceChecklistContract returns FAIL through the override valve and schedules no retry. Pending and missing artifacts still recover solo. The ruling is recorded on correctness-reviewer-precision.
207 changes: 200 additions & 7 deletions gate-engine/review/__tests__/checklist-recovery.test.mts
Original file line number Diff line number Diff line change
Expand Up @@ -541,8 +541,8 @@ describe('runReviewGate — checklist recovery on the strict ship path (sc-2088)
expect(remedies[0]).toContain('re-run devkit ship, and if it repeats move that judge');
});

// A FAILED item is a candidate violation, so a refused sibling lens must not reword its remedy.
it('a refused lens beside a FAILED-item lens keeps both remedies', async () => {
// A pinned lens's FAILED item blocks on its own; a refused sibling must not soften that block.
it('a FAILED-item lens beside a refused lens blocks on its finding', async () => {
delete process.env.GUARD_CORRECTNESS_SPLIT;
const repo = consumerRepo({ backend: true });
shipEnv(repo);
Expand All @@ -562,11 +562,10 @@ describe('runReviewGate — checklist recovery on the strict ship path (sc-2088)
}
return 'VERDICT: PASS';
});
expect(await runReviewGate(repo, { exec })).toBe(3);
const remedies = err.mock.calls.flat().filter((l) => String(l).startsWith(' Remedy:'));
expect(remedies).toHaveLength(2);
expect(remedies.join('\n')).toContain('if it repeats move that judge');
expect(remedies.join('\n')).toContain('fix a real violation if present');
expect(await runReviewGate(repo, { exec })).toBe(1);
const out = err.mock.calls.flat().join('\n');
expect(out).toContain(`[${lensGroupId(failing)}] — FAILED checklist item under a PASS verdict`);
expect(out).not.toContain('fix a real violation if present');
});

it('a lens refused on the wave that skips again with no refusal fails closed on the same model', async () => {
Expand Down Expand Up @@ -1255,3 +1254,197 @@ describe('runReviewGate — a voided PASS prints its FAILED items (sc-5720)', ()
expect(err).toHaveBeenCalledWith('⚠️ guard-review: api-security-reviewer — (+2 more)');
});
});

describe("runReviewGate — a pinned reviewer's FAILED item under a PASS verdict blocks (sc-8592)", () => {
const CORRECTNESS = 'review:correctness-reviewer';
const ITEM_LINE = 'FAILED checklist item under a PASS verdict: error-and-edge-classification';
const shipEnv = (repo: string) => {
process.env.GUARD_AI_STRICT = '1';
process.env.DEVKIT_GATE_EVENTS = join(repo, 'events.jsonl');
process.env.DEVKIT_SHIP_ID = 'ship-sc-8592';
};
const reviewResult = (repo: string, reviewer: string) =>
readFileSync(join(repo, 'events.jsonl'), 'utf8')
.trim()
.split('\n')
.map((l) => JSON.parse(l))
.findLast((e) => e.type === 'review_result' && e.reviewer === reviewer);
// correctness says PASS over `items` on its first attempt and a clean artifact on any retry.
const contradicting = (repo: string, items: object[], target = CORRECTNESS) => {
let attempt = 0;
return mkExec(async ({ label }) => {
writeArtifact(repo, label);
if (label === target && ++attempt === 1)
writeFileSync(join(repo, reviewerFromLabel(label).stateFile), JSON.stringify({ items }));
return 'VERDICT: PASS';
});
};
const edgeFail = {
name: 'error-and-edge-classification',
status: 'fail',
issues: ['0 read as absent'],
};

it('the incident replay blocks on the first sample: no retry, flagged on the merged row', async () => {
delete process.env.GUARD_CORRECTNESS_SPLIT; // the shipped four-way default
const repo = consumerRepo({ backend: true });
shipEnv(repo);
const err = vi.spyOn(console, 'error').mockImplementation(() => {});
const group = FOUR_WAY_LENS_GROUPS.find((g) => g.includes('error-and-edge-classification'))!;
const groupCalls: string[] = [];
const exec = mkExec(async ({ label, args }) => {
writeArtifact(repo, label);
if (label !== CORRECTNESS || !args[1].includes(`--lens ${[...group].sort().join(',')}`))
return 'VERDICT: PASS';
groupCalls.push(args[1]);
writeFileSync(
join(repo, `.claude/.correctness-review-${lensGroupId(group)}.json`),
JSON.stringify({ items: groupCalls.length === 1 ? [edgeFail] : [] }),
);
return 'VERDICT: PASS';
});
expect(await runReviewGate(repo, { exec })).toBe(1);
expect(groupCalls).toHaveLength(1);
const out = err.mock.calls.flat().join('\n');
expect(out).toContain(`[${lensGroupId(group)}] — ${ITEM_LINE}`);
expect(out).toContain('• error-and-edge-classification ['); // the findings block names it
const row = reviewResult(repo, 'correctness-reviewer');
expect(row).toMatchObject({ status: 'fail', failed_under_pass: true });
const part = row.lens_parts.find((p) => p.lens === lensGroupId(group));
expect(part).toMatchObject({ status: 'fail', failed_under_pass: true });
expect(row.lens_parts.filter((p) => p.failed_under_pass)).toHaveLength(1);
});

it('plain git commit blocks at exit 1 instead of failing open at exit 2', async () => {
const repo = consumerRepo({ backend: true });
vi.spyOn(console, 'error').mockImplementation(() => {});
expect(await runReviewGate(repo, { exec: contradicting(repo, [edgeFail]) })).toBe(1);
});

it('a waiver on the flipped lens turns it into a PASS', async () => {
const repo = consumerRepo({ backend: true });
shipEnv(repo);
vi.spyOn(console, 'error').mockImplementation(() => {});
expect(await runReviewGate(repo, { exec: contradicting(repo, [edgeFail]) })).toBe(1);
const fp = reviewResult(repo, 'correctness-reviewer').blocking[0].fp;
process.env[`OVERRIDE_${fp}_RATIONALE`] = 'absent and zero are the same state here';
try {
expect(await runReviewGate(repo, { exec: contradicting(repo, [edgeFail]) })).toBe(0);
} finally {
delete process.env[`OVERRIDE_${fp}_RATIONALE`];
}
});

it('an out-of-charter FAILED item is dropped, not blocked on', async () => {
const repo = consumerRepo({ backend: true });
shipEnv(repo);
vi.spyOn(console, 'error').mockImplementation(() => {});
const sqli = {
name: 'writer-reader-contracts',
status: 'fail',
issues: ['sql injection via the query string'],
};
expect(await runReviewGate(repo, { exec: contradicting(repo, [sqli]) })).toBe(0);
expect(reviewResult(repo, 'correctness-reviewer')).toMatchObject({
status: 'pass',
failed_under_pass: true,
});
});

// Waiving the FAILED item leaves only the pending hole, which takes the ordinary solo recovery.
const pendingWithWaivedFail = async (repo: string, retries: number) => {
const items = [edgeFail, { name: 'state-transitions', status: 'pending', issues: [] }];
const err = vi.spyOn(console, 'error').mockImplementation(() => {});
expect(await runReviewGate(repo, { exec: contradicting(repo, items) })).toBe(1);
const fp = err.mock.calls
.flat()
.join('\n')
.match(/• error-and-edge-classification \[(\w+)\]/)?.[1];
process.env[`OVERRIDE_${fp}_RATIONALE`] = 'absent and zero are the same state here';
let attempt = 0;
const exec = mkExec(async ({ label }) => {
writeArtifact(repo, label);
if (label === CORRECTNESS && ++attempt <= 1 + retries)
writeFileSync(join(repo, reviewerFromLabel(label).stateFile), JSON.stringify({ items }));
return 'VERDICT: PASS';
});
try {
return { code: await runReviewGate(repo, { exec }), calls: () => attempt };
} finally {
delete process.env[`OVERRIDE_${fp}_RATIONALE`];
}
};

it('a waived FAILED item beside a pending one retries the pending hole solo', async () => {
const repo = consumerRepo({ backend: true });
shipEnv(repo);
vi.spyOn(console, 'error').mockImplementation(() => {});
const { code, calls } = await pendingWithWaivedFail(repo, 0);
expect(code).toBe(0);
expect(calls()).toBe(2);
const row = reviewResult(repo, 'correctness-reviewer');
expect(row).toMatchObject({ status: 'pass', retried: true });
expect(row.failed_under_pass).toBeUndefined(); // waived, not erased by the retry
});

it('a pending hole the retry leaves open fails the ship closed, never a PASS', async () => {
const repo = consumerRepo({ backend: true });
shipEnv(repo);
vi.spyOn(console, 'error').mockImplementation(() => {});
expect((await pendingWithWaivedFail(repo, 1)).code).toBe(3);
const row = reviewResult(repo, 'correctness-reviewer');
expect(row).toMatchObject({ status: 'inconclusive', inconclusive_cause: 'response-contract' });
expect(row.reason).toContain('1 item(s) never resolved: state-transitions');
});

it('plain git commit keeps a pending hole beside a waived FAILED item fail-open, never a PASS', async () => {
const repo = consumerRepo({ backend: true });
const { code, calls } = await pendingWithWaivedFail(repo, 0);
expect(code).toBe(2);
expect(calls()).toBe(1);
});

it('review mode reports an exhausted pending hole as a REVIEW ERROR, not inconclusive', async () => {
const repo = unprojected(consumerRepo({ backend: true }));
process.env.DEVKIT_RUN_MODE = 'review';
process.env.DEVKIT_REVIEW_ASSET_ROOT = reviewAssets();
process.env.GUARD_AI_STRICT = '1'; // review runs under the same strict shell as ship
const err = vi.spyOn(console, 'error').mockImplementation(() => {});
const { code, calls } = await pendingWithWaivedFail(repo, 1);
expect(code).toBe(1);
expect(calls()).toBe(2);
expect(err.mock.calls.flat().join('\n')).toContain('checklist contract failed after one retry');
});

it('an unpinned reviewer still retries solo, and its row counts the FAILED→PASS flip', async () => {
const repo = consumerRepo({ backend: true });
shipEnv(repo);
vi.spyOn(console, 'error').mockImplementation(() => {});
const target = 'review:api-security-reviewer';
const exec = contradicting(
repo,
[{ name: 'input-validation', status: 'fail', issues: [] }],
target,
);
expect(await runReviewGate(repo, { exec })).toBe(0);
expect(exec.mock.calls.filter(([o]) => o.label === target)).toHaveLength(2);
expect(reviewResult(repo, 'api-security-reviewer')).toMatchObject({
status: 'pass',
retried: true,
failed_under_pass: true,
});
});

it('a budget-skipped retry still records that the voided attempt held a FAILED item', async () => {
const repo = consumerRepo({ backend: true });
shipEnv(repo);
vi.spyOn(console, 'error').mockImplementation(() => {});
process.env.DEVKIT_GATE_DEADLINE_MS = String(Date.now() + 5_000);
const target = 'review:api-security-reviewer';
const exec = contradicting(repo, [{ name: 'input-validation', status: 'fail' }], target);
expect(await runReviewGate(repo, { exec })).toBe(3);
const row = reviewResult(repo, 'api-security-reviewer');
expect(row).toMatchObject({ status: 'inconclusive', failed_under_pass: true });
expect(row.retried).toBeUndefined();
});
});
2 changes: 1 addition & 1 deletion gate-engine/review/__tests__/items-full.test.mts
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ describe('attachItems itemsFull', () => {
stateFile,
JSON.stringify({
items: [
{ name: 'state-transitions', status: 'fail', issues: [text] },
{ name: 'state-transitions', status: 'pass', issues: [text] },
{ name: 'concurrency-races', status: 'pending' },
],
}),
Expand Down
Loading