Skip to content

Block on a pinned reviewer's own FAILED checklist item under a PASS verdict (sc-8592) - #281

Merged
norvalbv merged 1 commit into
mainfrom
sc-8592-checklist-retry-failed-item-flips-to-pass
Nov 20, 2026
Merged

norvalbv merged 1 commit into
mainfrom
sc-8592-checklist-retry-failed-item-flips-to-pass

Conversation

@norvalbv

@norvalbv norvalbv commented Nov 20, 2026 •

Copy link
Copy Markdown
Owner

Problem

correctness-reviewer runs as a pinned single-pass judge, split into lens parts. It can write a checklist item with status: 'fail' and still return VERDICT: PASS. Today enforceChecklistContract voids that PASS and, under ship, re-runs the judge once on its own. If the retry returns a bare PASS that omits the item, the PASS wins and the defect ships as a non-blocking advisory. That happened on PR #227, where GIT_ATTR_SOURCE is ignored on git 2.38–2.39. On a plain git commit, the same contradiction went inconclusive, which lets the commit through at exit 2. Telemetry could not count how often a retry flipped FAILED to PASS: a parked attempt emits nothing, and the retried row carried only retried: true.

Change

  • A pinned reviewer's own FAILED item is the verdict (gate-engine/review/contracts/checklist.mts).
    • The outcome becomes FAIL with no re-sample. The existing reportFailedItems ⚠️ line names the item.
    • applyOverrideValve then runs as for any pinned FAIL. domainExclusivityDrop removes out-of-charter security/performance items, a per-lens waiver turns the block into a PASS, and the findings block prints the fix-or-waive command.
    • Plain git commit: exit 2 → 1.
  • Unfinished checklists still recover (settleUnfinished, which shares runCascade's recover callback). Pending items left after every failed lens was waived or dropped take the ordinary solo recovery:
    • an exhausted retry fails a ship closed (exit 3) and is a REVIEW ERROR in review mode;
    • plain commit stays fail-open.
    • It never reads as a PASS on the first sample.
  • Telemetry.
    • failed_under_pass: true appears on review_result, on each lens_parts entry and on the merged lens row.
    • For an unpinned reviewer it is carried through ParkedRecovery to the retried or budget-skipped row.
    • A retry that flipped FAILED to PASS is a row with retried && status === 'pass' && failed_under_pass.
  • Decision notes.
    • correctness-reviewer-precision: this settles one sample's own contradiction. No second sample runs, so it is not the sc-2754 union, not Rejected (a) and not Rejected (d).
    • review-gate-in-chain: a cross-reference note.

Scope assumption

Unpinned cascade reviewers (api-security, backend-performance, frontend-*, commit-guard) keep the solo retry and gain only the telemetry flag. Their sc-5614 escalation restores first-pass failed rows, so an opus overturn leaves this same shape. applyOverrideValve gives them no waiver path (overrides.mts:356). Flipping them directly would turn verified overturns into blocks nobody can waive. Routing their non-escalated FAILED-under-PASS into the verify-only escalation is the open owner decision below (sc-8953).

Tests

gate-engine/review/__tests__/checklist-recovery.test.mts, new sc-8592 block:

  • Incident replay (four-way split, ship): exit 1, one judge call for the lens, failed_under_pass on the merged row and its lens part.
  • Plain commit: exit 1.
  • Waiver: a waived lens passes.
  • Out-of-charter item: dropped, and the result passes.
  • Pending plus waived FAILED item: the retry passes (exit 0); an exhausted retry gives exit 3 on ship and a REVIEW ERROR at exit 1 in review mode; plain commit stays at exit 2.
  • Unpinned reviewer: still retries, and the retried row carries retried and failed_under_pass.
  • Budget-skipped retry: keeps the flag and is not marked retried.

Tests that pinned the old behaviour, rewritten in place:

  • run-review.test.mts (ratcheted, no net lines): exit 2→1. The byte-bound spill fixture now writes pass items for pinned reviewers, because a pinned FAILED item now blocks. So that test no longer checks FAIL rows for the byte bound.
  • items-full.test.mts: the capture test's item is now pass, so the test stays a pure pending-item miss.
  • The refused-lens test in checklist-recovery.test.mts: the FAILED lens now blocks.

Evidence:

  • 919/919 across 23 related test files.
  • Revert proofs: disabling the flip fails 4 tests; forcing the old no-retry pending branch fails 2; dropping the skip-row flag fails 1.
  • Diff coverage: 28/29 added executable lines. The one miss is the existing unreachable throw in the moved recovery callback.

Follow-ups (not in this PR)

  • A FAIL's review_result.reason has no byte bound. A fixture with 60 failed lenses wrote a 26 KB event line, beyond the 4 KB atomic window. Real correctness has 4 lenses, but any wide genuine FAIL has the same shape. (chore; autonomous 8a811732)
  • Telemetry does not record whether a flipped PASS also carried ADVISORY lines, which would size the accepted false-block cost below. (feature; sc-8954)
  • A voided attempt's failed items are missing from the ship digest's "Gate findings this run" block. This predates the change and now applies only to the unpinned path. (bug, sev 3; sc-5769)

Accepted cost: a judge that downgraded a defect to ADVISORY but left its item marked fail now blocks until it is waived.

Story: https://app.shortcut.com/benordlabs/story/8592

🤖 Generated with Claude Code

Reviewer notes declined:

  • correctness-reviewer, gate-engine/review/lens/split.mts:397, a cached lens part not carrying failedUnderPass: a cache replay judges nothing this run, and the flip metric requires retried, which a replay never sets, so no flip is undercounted. Stamping replays would recount the original attempt's row on every cache hit; cache hits already have their own cache_hit event (gate-telemetry-self-describing).

Not filed: none.

Complete against acceptance criteria: NO — unpinned reviewers' FAILED-under-PASS retry (sc-8953): routing it into the opus escalation changes escalation semantics and cost, so it needs the owner's decision

Automated evidence

Merge verdict: 🟡 human-review · advisory, at 47723daf55ce

  • decision-record
  • over-size: 87/50 lines, 7/4 files, default limit
✅ Regression evidence · captured: 12 of 12 red-failing tests passed on green (setup 7s, red 95s, green 89s, cleanup 3s)

Head 47723daf55ce · base e7bfe421f0fb

run what ran exit passed failed skipped
red PR test files on the PR base 1 193 12 0
green PR head 0 205 0 0
Failed on red (12 tests)
  • runReviewGate — checklist recovery on the strict ship path (sc-2088) a FAILED-item lens beside a refused lens blocks on its finding: AssertionError: expected 3 to be 1 // Object.is equality
  • runReviewGate — a pinned reviewer's FAILED item under a PASS verdict blocks (sc-8592) the incident replay blocks on the first sample: no retry, flagged on the merged row: AssertionError: expected 3 to be 1 // Object.is equality
  • runReviewGate — a pinned reviewer's FAILED item under a PASS verdict blocks (sc-8592) plain git commit blocks at exit 1 instead of failing open at exit 2: AssertionError: expected 2 to be 1 // Object.is equality
  • runReviewGate — a pinned reviewer's FAILED item under a PASS verdict blocks (sc-8592) a waiver on the flipped lens turns it into a PASS: AssertionError: expected +0 to be 1 // Object.is equality
  • runReviewGate — a pinned reviewer's FAILED item under a PASS verdict blocks (sc-8592) an out-of-charter FAILED item is dropped, not blocked on: AssertionError: expected { type: 'review_result', …(19) } to match object { status: 'pass', …(1) } (20 matching properties omitted from actual)
  • runReviewGate — a pinned reviewer's FAILED item under a PASS verdict blocks (sc-8592) a waived FAILED item beside a pending one retries the pending hole solo: AssertionError: expected +0 to be 1 // Object.is equality
  • runReviewGate — a pinned reviewer's FAILED item under a PASS verdict blocks (sc-8592) a pending hole the retry leaves open fails the ship closed, never a PASS: AssertionError: expected +0 to be 1 // Object.is equality
  • runReviewGate — a pinned reviewer's FAILED item under a PASS verdict blocks (sc-8592) plain git commit keeps a pending hole beside a waived FAILED item fail-open, never a PASS: AssertionError: expected 2 to be 1 // Object.is equality
  • runReviewGate — a pinned reviewer's FAILED item under a PASS verdict blocks (sc-8592) review mode reports an exhausted pending hole as a REVIEW ERROR, not inconclusive: AssertionError: expected +0 to be 1 // Object.is equality
  • runReviewGate — a pinned reviewer's FAILED item under a PASS verdict blocks (sc-8592) an unpinned reviewer still retries solo, and its row counts the FAILED→PASS flip: AssertionError: expected { type: 'review_result', …(19) } to match object { status: 'pass', retried: true, …(1) } (19 matching properties omitted from actual)
  • … and 2 more
Test command
[
  "node_modules/.bin/vitest",
  "run",
  "--reporter=json",
  "--outputFile.json=.devkit-evidence-report.json",
  "gate-engine/review/__tests__/checklist-recovery.test.mts",
  "gate-engine/review/__tests__/items-full.test.mts",
  "gate-engine/review/__tests__/run-review.test.mts"
]

Captured runs, not proof of causality. Dependencies are the shipping checkout's installed packages. Stale unless the PR head is 47723daf55ce6f7c5f48305d7880432c413e948f.

💬 Reviewer notes · 1 note at 47723daf55ce

gate-engine/lens/split.mts

  • L397 — a cached split lens part is rebuilt with status: 'pass' but without failedUnderPass, so a replayed PASS that held a FAILED item drops failed_under_pass from its merged row, undercounting flips on cache hits. · correctness

…erdict (sc-8592)

## Problem

correctness-reviewer runs as a pinned single-pass judge, split into lens parts. It can write a checklist item with `status: 'fail'` and still return `VERDICT: PASS`. Today `enforceChecklistContract` voids that PASS and, under ship, re-runs the judge once on its own. If the retry returns a bare PASS that omits the item, the PASS wins and the defect ships as a non-blocking advisory. That happened on PR #227, where `GIT_ATTR_SOURCE` is ignored on git 2.38–2.39. On a plain `git commit`, the same contradiction went inconclusive, which lets the commit through at exit 2. Telemetry could not count how often a retry flipped FAILED to PASS: a parked attempt emits nothing, and the retried row carried only `retried: true`.

## Change

- **A pinned reviewer's own FAILED item is the verdict** (`gate-engine/review/contracts/checklist.mts`).
  - The outcome becomes FAIL with no re-sample. The existing `reportFailedItems` ⚠️ line names the item.
  - `applyOverrideValve` then runs as for any pinned FAIL. `domainExclusivityDrop` removes out-of-charter security/performance items, a per-lens waiver turns the block into a PASS, and the findings block prints the fix-or-waive command.
  - Plain `git commit`: exit 2 → 1.
- **Unfinished checklists still recover** (`settleUnfinished`, which shares `runCascade`'s `recover` callback). Pending items left after every failed lens was waived or dropped take the ordinary solo recovery:
  - an exhausted retry fails a ship closed (exit 3) and is a REVIEW ERROR in review mode;
  - plain commit stays fail-open.
  - It never reads as a PASS on the first sample.
- **Telemetry.**
  - `failed_under_pass: true` appears on `review_result`, on each `lens_parts` entry and on the merged lens row.
  - For an unpinned reviewer it is carried through `ParkedRecovery` to the retried or budget-skipped row.
  - A retry that flipped FAILED to PASS is a row with `retried && status === 'pass' && failed_under_pass`.
- **Decision notes.**
  - `correctness-reviewer-precision`: this settles one sample's own contradiction. No second sample runs, so it is not the sc-2754 union, not Rejected (a) and not Rejected (d).
  - `review-gate-in-chain`: a cross-reference note.

## Scope assumption

Unpinned cascade reviewers (api-security, backend-performance, frontend-*, commit-guard) keep the solo retry and gain only the telemetry flag. Their sc-5614 escalation restores first-pass failed rows, so an opus overturn leaves this same shape. `applyOverrideValve` gives them no waiver path (`overrides.mts:356`). Flipping them directly would turn verified overturns into blocks nobody can waive. Routing their non-escalated FAILED-under-PASS into the verify-only escalation is the open owner decision below.

## Tests

`gate-engine/review/__tests__/checklist-recovery.test.mts`, new sc-8592 block:
- **Incident replay** (four-way split, ship): exit 1, one judge call for the lens, `failed_under_pass` on the merged row and its lens part.
- **Plain commit:** exit 1.
- **Waiver:** a waived lens passes.
- **Out-of-charter item:** dropped, and the result passes.
- **Pending plus waived FAILED item:** the retry passes (exit 0); an exhausted retry gives exit 3 on ship and a REVIEW ERROR at exit 1 in review mode; plain commit stays at exit 2.
- **Unpinned reviewer:** still retries, and the retried row carries `retried` and `failed_under_pass`.
- **Budget-skipped retry:** keeps the flag and is not marked `retried`.

Tests that pinned the old behaviour, rewritten in place:
- `run-review.test.mts` (ratcheted, no net lines): exit 2→1. The byte-bound spill fixture now writes pass items for pinned reviewers, because a pinned FAILED item now blocks. So that test no longer checks FAIL rows for the byte bound.
- `items-full.test.mts`: the capture test's item is now pass, so the test stays a pure pending-item miss.
- The refused-lens test in `checklist-recovery.test.mts`: the FAILED lens now blocks.

Evidence:
- 919/919 across 23 related test files.
- Revert proofs: disabling the flip fails 4 tests; forcing the old no-retry pending branch fails 2; dropping the skip-row flag fails 1.
- Diff coverage: 28/29 added executable lines. The one miss is the existing unreachable `throw` in the moved recovery callback.

## Follow-ups (not in this PR)

- A FAIL's `review_result.reason` has no byte bound. A fixture with 60 failed lenses wrote a 26 KB event line, beyond the 4 KB atomic window. Real correctness has 4 lenses, but any wide genuine FAIL has the same shape. (chore)
- Telemetry does not record whether a flipped PASS also carried ADVISORY lines, which would size the accepted false-block cost below. (feature)
- A voided attempt's failed items are missing from the ship digest's "Gate findings this run" block. This predates the change and now applies only to the unpinned path. (bug, sev 3)

Accepted cost: a judge that downgraded a defect to ADVISORY but left its item marked `fail` now blocks until it is waived.

Story: https://app.shortcut.com/benordlabs/story/8592

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Complete against acceptance criteria: NO — unpinned reviewers' FAILED-under-PASS retry: routing it into the opus escalation changes escalation semantics and cost, so it needs the owner's decision
@coderabbitai

coderabbitai Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

Pinned reviewers now block when a PASS verdict contains a failed checklist item. The change preserves ordinary recovery for pending items and propagates a marker for failed items through lens results and recovery outcomes.

Changes: Checklist gate behavior

Layer / File(s) Summary
Checklist outcome and recovery
gate-engine/review/runtime.mts, gate-engine/review/contracts/checklist.mts, gate-engine/review/cascade/reviewer.mts
PASS outcomes with failed checklist items receive a failedUnderPass marker. Pinned reviewers return FAIL. Cascade reviewers settle pending items through the recovery callback.
Failure marker propagation
gate-engine/review/lens/split.mts, gate-engine/review/lens/merge-results.mts, gate-engine/review/run-review.mts, gate-engine/review/recovery/settle.mts
Lens rows and review-result events expose failed_under_pass. Parked recovery and settled retry outcomes retain the marker.
Behavior tests and decision records
gate-engine/review/__tests__/*, docs/decisions/correctness-reviewer-precision.md, docs/decisions/review-gate-in-chain.md
Tests cover pinned and unpinned failures, waivers, pending-item recovery, and exhausted budgets. Decision records describe the updated handling and recovery rules.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant runCascade
  participant enforceChecklistContract
  participant settleUnfinished
  participant recoveryCallback
  runCascade->>enforceChecklistContract: result and recovery callback
  alt pinned reviewer has a failed item under PASS
    enforceChecklistContract-->>runCascade: FAIL with failedUnderPass
  else PASS has failedUnderPass and pending items
    enforceChecklistContract-->>runCascade: PASS with failedUnderPass
    runCascade->>settleUnfinished: result, checklist state, scheduling flag, callback
    settleUnfinished->>recoveryCallback: retry checklist hole when recovery is scheduled
  end
Loading
Merge Risk: Minimal · up to 47723

Pinned reviewers that return PASS with a failed checklist item now block instead of retrying, while pending-item recovery and waivers still work as described. No concrete merge-blocking defect remains after review.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title states the main change: a pinned reviewer's own FAILED checklist item under a PASS verdict now blocks. The summary, the objectives, and the changed files (checklist contract, settleUnfinishe…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Full details: Title check

Explanation

The title states the main change: a pinned reviewer's own FAILED checklist item under a PASS verdict now blocks. The summary, the objectives, and the changed files (checklist contract, settleUnfinished, failed_under_pass telemetry, and tests) all match this. The title includes the ticket reference sc-8592, which is acceptable and does not make it vague.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Warning

Some tools did not complete. Review the errors below.

LanguageTool

docs/decisions/review-gate-in-chain.md

LanguageTool could not check every text chunk; analysis is incomplete

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: d734f39d-6649-4056-8267-2c787319fdde
Commits

Reviewing files that changed from the base of the PR and between e7bfe42 and 47723da.

Files selected for processing: 12

  • docs/decisions/correctness-reviewer-precision.md
  • docs/decisions/review-gate-in-chain.md
  • gate-engine/review/__tests__/checklist-recovery.test.mts
  • gate-engine/review/__tests__/items-full.test.mts
  • gate-engine/review/__tests__/run-review.test.mts
  • gate-engine/review/cascade/reviewer.mts
  • gate-engine/review/contracts/checklist.mts
  • gate-engine/review/lens/merge-results.mts
  • gate-engine/review/lens/split.mts
  • gate-engine/review/recovery/settle.mts
  • gate-engine/review/run-review.mts
  • gate-engine/review/runtime.mts

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

@norvalbv
norvalbv merged commit cee3de4 into main Nov 20, 2026
8 checks passed
@norvalbv
norvalbv deleted the sc-8592-checklist-retry-failed-item-flips-to-pass branch November 20, 2026 08:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant