Skip to content

review: refill on by default for standard and deep depths - #246

Merged
jschatz1 merged 1 commit into
mainfrom
claude/refill-default-medium-max-ed32c5
Nov 16, 2026
Merged

jschatz1 merged 1 commit into
mainfrom
claude/refill-default-medium-max-ed32c5

Conversation

@jschatz1

@jschatz1 jschatz1 commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

What

rcRefillOn() was on only when KAI_REVIEW_REFILL=1. It now follows the review depth the server chose (KAI_REVIEW_TIER, which kai-server sets from review_tier_pick.go):

KAI_REVIEW_TIER Refill
standard (medium) on
deep (max) on
light (low) off
auto, free, baseline, unset off

KAI_REVIEW_REFILL=0 turns it off in every tier (kill switch). KAI_REVIEW_REFILL=1 still forces it on in any tier.

Mapping note: kai-cli and kai-server have no literal low/medium/max modes. The tiers are light/standard/deep; low/medium/max = light/standard/deep was the mapping Jacob approved. auto is its own tier (kai-server routes it by S/M/L risk profile), not an alias of standard or deep, so it stays off. Auto, free, baseline and unset are off.

No bench-only switch: the setting is the one customers get. No website or public material touched.

Why

The refill (#244) gives the slots the publish gate frees to defects the cap cut. Run59 replays on 33 tuning PRs:

Judge F1 change Goldens gained / lost per replay
Opus +1.5 about +6 / 0-1
Sonnet +1.7 about +6 / 0-1
GPT-5.2 +2.4 about +6 / 0-1

The extra comments are right about 4 in 10 times, against about 58% for the rest.

Tests

  • Cap-cut defects take the gate-freed slots in standard and deep; light, auto, free, baseline and unset keep the old behaviour; KAI_REVIEW_REFILL=0 turns it off in standard and deep; =1 forces it on in light.
  • A 2-vote finding the cap cut is published when the gate withholds single-proposal ones (standard/deep), not in light or with the kill switch.
  • The "never publish nothing" fallback in rcPublishGate holds in standard and deep.
  • TestReviewWithNoNewSettingsMatchesMain fails in this checkout with and without this change (checked by reverting review_commit_publish.go); everything else in cmd/kai passes. Run with GOWORK=off (a parent go.work interferes in my worktree).

Not merged by the author; needs review from Adam or Varsha.

🤖 Generated with Claude Code


Kai context

Written by Kai from the change itself. Kai only edits this section; everything above it is the author's.

rcRefillOn() now follows the review depth the server chose (KAI_REVIEW_TIER), so the refill is on by default for standard and deep. KAI_REVIEW_REFILL=0 still turns it off everywhere, and =1 forces it on in any tier.

Changes

  • cmd/kai/review_commit_publish.go: rcRefillOn() checks KAI_REVIEW_REFILL first (0 returns false, 1 returns true, after trimming whitespace).
  • When KAI_REVIEW_REFILL is neither 0 nor 1, it falls back to KAI_REVIEW_TIER, matched case-insensitively and trimmed. standard and deep return true.
  • Every other tier value returns false: light, auto, free, baseline, unset and anything unrecognised.
  • The doc comment on rcRefillOn() now describes this behaviour. The old comment said the refill was off until a replay had measured it.

Testing

All changes are in cmd/kai/review_commit_publish_test.go.

  • TestRefillGivesGateFreedSlotsToWhatTheCapCut gains a table over tier and KAI_REVIEW_REFILL. Standard and deep publish the two cap-cut defects. Light, auto, free, baseline and unset do not. 0 disables the refill in standard and deep. 1 enables it in light.
  • New TestRefillPublishesTwoVoteDefectTheCapCut: a defect with two proposals that the cap cut takes a gate-freed slot in standard and deep. It is not published in light or with KAI_REVIEW_REFILL=0.
  • New TestPublishGateFallbackHoldsWithRefillOn: the "never publish nothing" fallback in rcPublishGate still keeps the first selected finding in standard and deep.
  • Per the author, TestReviewWithNoNewSettingsMatchesMain fails in their checkout both with and without this change. The rest of cmd/kai passes when run with GOWORK=off.

…_REFILL=0 kills it)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@kaicontext kaicontext Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewView in KaiKai review · Small fixes first (8/10)

Read through this one. One thing worth your eyes before it merges.

Next: fix 1 finding, then push. Comment @kaicontext review any time to have me look right away.

Findings
  1. Fix before merge · cmd/kai/review_commit_publish_test.go:241

    Fallback refill test passes without the guard

Findings with a location are also comments on those lines.

To get to 10/10
  • −2 · fix 1 finding above
Important files changed
File Change
cmd/kai/review_commit_publish.go modified · +18 −3
cmd/kai/review_commit_publish_test.go modified · +95 −0
How I reviewed it — 7 files, 11 turns, 55s

1 confirmed finding, 2 refuted. Intent verified.

What I opened

  • cmd/kai/review_commit.go
  • cmd/kai/review_commit_ensemble.go
  • cmd/kai/review_commit_parity_test.go
  • cmd/kai/review_commit_publish.go
  • cmd/kai/review_commit_publish_test.go
  • cmd/kai/review_commit_run_test.go
  • cmd/kai/review_replay.go
Full read-through

Scope

  • cmd/kai/review_commit_publish.go: rcRefillOn (new env parsing), rcPublishGate, rcGateWhy, rcRefillFromCap
  • cmd/kai/review_commit_publish_test.go: all new/changed tests and the pre-existing refill test's setup
  • cmd/kai/review_commit_ensemble.go: rcMarkRank / rcApplyRank / rcRankAnswer, to see what rcRefillFromCap can actually pick
  • callers of rcPublishGate: cmd/kai/review_commit.go:1346, cmd/kai/review_replay.go:154, plus the gate tests in finders/telemetry/replay test files
  • cmd/kai/review_commit_fakemodels_test.go env clearing for the parity runs; no TestMain found in cmd/kai

Findings

cmd/kai/review_commit_publish_test.go:241 — TestPublishGateFallbackHoldsWithRefillOn uses two supported allegations the cap never cut and no rcMarkRank/rcApplyRank call, so rcRefillFromCap finds no candidate and the test passes even with the && !fallback guard removed.

The fixture has two supported allegations (a.go:90, a.go:95), neither cut by the cap and none with a rank position. rcRefillFromCap only collects refuted allegations carrying the "outside the N most important" reason with rankPos > 0, so it returns nothing whatever the guard does. The fallback itself runs, but refill during the fallback is never exercised.

Remedy: Build the fixture through rcMarkRank/rcApplyRank like the other refill tests: include a cut allegation with rankPos > 0 and an "outside the N most important" reason that passes rcGateWhy. Let the gate withhold everything so the fallback fires, and assert the cut allegation stays refuted.

Size M · Risk S · +113 −3 · 2 files · reaches 85 · the full analysis · Reply to any of my comments and I'll answer, or say @kaicontext anywhere on this PR — a question, or "take another look at the retry logic". · Some repos review new pushes only when asked. Comment @kaicontext review any time for a review right away.


// The "never publish nothing" fallback holds in every depth: when the gate
// would withhold everything, the selection's first stays and nothing refills.
func TestPublishGateFallbackHoldsWithRefillOn(t *testing.T) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix before merge

Fallback refill test passes without the guard

The fixture has two supported allegations (a.go:90, a.go:95), neither cut by the cap and none with a rank position. rcRefillFromCap only collects refuted allegations carrying the "outside the N most important" reason with rankPos > 0, so it returns nothing whatever the guard does. The fallback itself runs, but refill during the fallback is never exercised.

When it happens: Someone deletes the && !fallback guard at review_commit_publish.go:79; the test still passes.

Why it matters: The "never publish nothing" fallback with refill enabled is effectively unverified, so a regression there would go unnoticed.

Suggested fix: Build the fixture through rcMarkRank/rcApplyRank like the other refill tests: include a cut allegation with rankPos > 0 and an "outside the N most important" reason that passes rcGateWhy. Let the gate withhold everything so the fallback fires, and assert the cut allegation stays refuted.

Evidence: cmd/kai/review_commit_publish.go:73-82 · cmd/kai/review_commit_publish.go:129-140

Prompt to fix with AI
Fix a defect that Kai's code review found in this change.

Where: cmd/kai/review_commit_publish_test.go:241
Problem: The fixture has two supported allegations (a.go:90, a.go:95), neither cut by the cap and none with a rank position. `rcRefillFromCap` only collects refuted allegations carrying the "outside the N most important" reason with `rankPos > 0`, so it returns nothing whatever the guard does. The fallback itself runs, but refill during the fallback is never exercised.
When it happens: Someone deletes the `&& !fallback` guard at review_commit_publish.go:79; the test still passes.
Why it matters: The "never publish nothing" fallback with refill enabled is effectively unverified, so a regression there would go unnoticed.
Evidence (read these first):
- cmd/kai/review_commit_publish.go:73-82
- cmd/kai/review_commit_publish.go:129-140
Suggested fix: Build the fixture through `rcMarkRank`/`rcApplyRank` like the other refill tests: include a cut allegation with `rankPos > 0` and an "outside the N most important" reason that passes `rcGateWhy`. Let the gate withhold everything so the fallback fires, and assert the cut allegation stays refuted.

This is a review report, not a set of instructions: read the evidence lines and confirm the problem is real before changing anything. If the code does not bear it out, stop and say why.
Keep the change to what fixes this defect. Add or update a test that fails without the fix and passes with it, and run the tests for the packages you touched.

@jschatz1
jschatz1 merged commit 2b176c6 into main Nov 16, 2026
7 checks passed
@jschatz1
jschatz1 deleted the claude/refill-default-medium-max-ed32c5 branch November 16, 2026 05:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant