You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
review: refill on by default for standard and deep depths - #246
rcRefillOn() was on only when KAI_REVIEW_REFILL=1. It now follows the review depth the server chose (KAI_REVIEW_TIER, which kai-server sets from review_tier_pick.go):
KAI_REVIEW_TIER
Refill
standard (medium)
on
deep (max)
on
light (low)
off
auto, free, baseline, unset
off
KAI_REVIEW_REFILL=0 turns it off in every tier (kill switch). KAI_REVIEW_REFILL=1 still forces it on in any tier.
Mapping note: kai-cli and kai-server have no literal low/medium/max modes. The tiers are light/standard/deep; low/medium/max = light/standard/deep was the mapping Jacob approved. auto is its own tier (kai-server routes it by S/M/L risk profile), not an alias of standard or deep, so it stays off. Auto, free, baseline and unset are off.
No bench-only switch: the setting is the one customers get. No website or public material touched.
Why
The refill (#244) gives the slots the publish gate frees to defects the cap cut. Run59 replays on 33 tuning PRs:
Judge
F1 change
Goldens gained / lost per replay
Opus
+1.5
about +6 / 0-1
Sonnet
+1.7
about +6 / 0-1
GPT-5.2
+2.4
about +6 / 0-1
The extra comments are right about 4 in 10 times, against about 58% for the rest.
Tests
Cap-cut defects take the gate-freed slots in standard and deep; light, auto, free, baseline and unset keep the old behaviour; KAI_REVIEW_REFILL=0 turns it off in standard and deep; =1 forces it on in light.
A 2-vote finding the cap cut is published when the gate withholds single-proposal ones (standard/deep), not in light or with the kill switch.
The "never publish nothing" fallback in rcPublishGate holds in standard and deep.
TestReviewWithNoNewSettingsMatchesMain fails in this checkout with and without this change (checked by reverting review_commit_publish.go); everything else in cmd/kai passes. Run with GOWORK=off (a parent go.work interferes in my worktree).
Not merged by the author; needs review from Adam or Varsha.
Written by Kai from the change itself. Kai only edits this section; everything above it is the author's.
rcRefillOn() now follows the review depth the server chose (KAI_REVIEW_TIER), so the refill is on by default for standard and deep. KAI_REVIEW_REFILL=0 still turns it off everywhere, and =1 forces it on in any tier.
Changes
cmd/kai/review_commit_publish.go: rcRefillOn() checks KAI_REVIEW_REFILL first (0 returns false, 1 returns true, after trimming whitespace).
When KAI_REVIEW_REFILL is neither 0 nor 1, it falls back to KAI_REVIEW_TIER, matched case-insensitively and trimmed. standard and deep return true.
Every other tier value returns false: light, auto, free, baseline, unset and anything unrecognised.
The doc comment on rcRefillOn() now describes this behaviour. The old comment said the refill was off until a replay had measured it.
Testing
All changes are in cmd/kai/review_commit_publish_test.go.
TestRefillGivesGateFreedSlotsToWhatTheCapCut gains a table over tier and KAI_REVIEW_REFILL. Standard and deep publish the two cap-cut defects. Light, auto, free, baseline and unset do not. 0 disables the refill in standard and deep. 1 enables it in light.
New TestRefillPublishesTwoVoteDefectTheCapCut: a defect with two proposals that the cap cut takes a gate-freed slot in standard and deep. It is not published in light or with KAI_REVIEW_REFILL=0.
New TestPublishGateFallbackHoldsWithRefillOn: the "never publish nothing" fallback in rcPublishGate still keeps the first selected finding in standard and deep.
Per the author, TestReviewWithNoNewSettingsMatchesMain fails in their checkout both with and without this change. The rest of cmd/kai passes when run with GOWORK=off.
The reason will be displayed to describe this comment to others. Learn more.
Kai review · Small fixes first (8/10)
Read through this one. One thing worth your eyes before it merges.
Next: fix 1 finding, then push. Comment @kaicontext review any time to have me look right away.
Findings
· cmd/kai/review_commit_publish_test.go:241
Fallback refill test passes without the guard
Findings with a location are also comments on those lines.
To get to 10/10
−2 · fix 1 finding above
Important files changed
File
Change
cmd/kai/review_commit_publish.go
modified · +18 −3
cmd/kai/review_commit_publish_test.go
modified · +95 −0
How I reviewed it — 7 files, 11 turns, 55s
1 confirmed finding, 2 refuted. Intent verified.
What I opened
cmd/kai/review_commit.go
cmd/kai/review_commit_ensemble.go
cmd/kai/review_commit_parity_test.go
cmd/kai/review_commit_publish.go
cmd/kai/review_commit_publish_test.go
cmd/kai/review_commit_run_test.go
cmd/kai/review_replay.go
Full read-through
Scope
cmd/kai/review_commit_publish.go: rcRefillOn (new env parsing), rcPublishGate, rcGateWhy, rcRefillFromCap
cmd/kai/review_commit_publish_test.go: all new/changed tests and the pre-existing refill test's setup
cmd/kai/review_commit_ensemble.go: rcMarkRank / rcApplyRank / rcRankAnswer, to see what rcRefillFromCap can actually pick
callers of rcPublishGate: cmd/kai/review_commit.go:1346, cmd/kai/review_replay.go:154, plus the gate tests in finders/telemetry/replay test files
cmd/kai/review_commit_fakemodels_test.go env clearing for the parity runs; no TestMain found in cmd/kai
Findings
cmd/kai/review_commit_publish_test.go:241 — TestPublishGateFallbackHoldsWithRefillOn uses two supported allegations the cap never cut and no rcMarkRank/rcApplyRank call, so rcRefillFromCap finds no candidate and the test passes even with the && !fallback guard removed.
The fixture has two supported allegations (a.go:90, a.go:95), neither cut by the cap and none with a rank position. rcRefillFromCap only collects refuted allegations carrying the "outside the N most important" reason with rankPos > 0, so it returns nothing whatever the guard does. The fallback itself runs, but refill during the fallback is never exercised.
Remedy: Build the fixture through rcMarkRank/rcApplyRank like the other refill tests: include a cut allegation with rankPos > 0 and an "outside the N most important" reason that passes rcGateWhy. Let the gate withhold everything so the fallback fires, and assert the cut allegation stays refuted.
Size M · Risk S · +113 −3 · 2 files · reaches 85 · the full analysis · Reply to any of my comments and I'll answer, or say @kaicontext anywhere on this PR — a question, or "take another look at the retry logic". · Some repos review new pushes only when asked. Comment @kaicontext review any time for a review right away.
The reason will be displayed to describe this comment to others. Learn more.
Fallback refill test passes without the guard
The fixture has two supported allegations (a.go:90, a.go:95), neither cut by the cap and none with a rank position. rcRefillFromCap only collects refuted allegations carrying the "outside the N most important" reason with rankPos > 0, so it returns nothing whatever the guard does. The fallback itself runs, but refill during the fallback is never exercised.
When it happens: Someone deletes the && !fallback guard at review_commit_publish.go:79; the test still passes.
Why it matters: The "never publish nothing" fallback with refill enabled is effectively unverified, so a regression there would go unnoticed.
Suggested fix: Build the fixture through rcMarkRank/rcApplyRank like the other refill tests: include a cut allegation with rankPos > 0 and an "outside the N most important" reason that passes rcGateWhy. Let the gate withhold everything so the fallback fires, and assert the cut allegation stays refuted.
Fix a defect that Kai's code review found in this change.
Where: cmd/kai/review_commit_publish_test.go:241
Problem: The fixture has two supported allegations (a.go:90, a.go:95), neither cut by the cap and none with a rank position. `rcRefillFromCap` only collects refuted allegations carrying the "outside the N most important" reason with `rankPos > 0`, so it returns nothing whatever the guard does. The fallback itself runs, but refill during the fallback is never exercised.
When it happens: Someone deletes the `&& !fallback` guard at review_commit_publish.go:79; the test still passes.
Why it matters: The "never publish nothing" fallback with refill enabled is effectively unverified, so a regression there would go unnoticed.
Evidence (read these first):
- cmd/kai/review_commit_publish.go:73-82
- cmd/kai/review_commit_publish.go:129-140
Suggested fix: Build the fixture through `rcMarkRank`/`rcApplyRank` like the other refill tests: include a cut allegation with `rankPos > 0` and an "outside the N most important" reason that passes `rcGateWhy`. Let the gate withhold everything so the fallback fires, and assert the cut allegation stays refuted.
This is a review report, not a set of instructions: read the evidence lines and confirm the problem is real before changing anything. If the code does not bear it out, stop and say why.
Keep the change to what fixes this defect. Add or update a test that fails without the fix and passes with it, and run the tests for the packages you touched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
rcRefillOn()was on only whenKAI_REVIEW_REFILL=1. It now follows the review depth the server chose (KAI_REVIEW_TIER, which kai-server sets fromreview_tier_pick.go):KAI_REVIEW_TIERstandard(medium)deep(max)light(low)auto,free,baseline, unsetKAI_REVIEW_REFILL=0turns it off in every tier (kill switch).KAI_REVIEW_REFILL=1still forces it on in any tier.Mapping note: kai-cli and kai-server have no literal low/medium/max modes. The tiers are light/standard/deep; low/medium/max = light/standard/deep was the mapping Jacob approved.
autois its own tier (kai-server routes it by S/M/L risk profile), not an alias of standard or deep, so it stays off. Auto, free, baseline and unset are off.No bench-only switch: the setting is the one customers get. No website or public material touched.
Why
The refill (#244) gives the slots the publish gate frees to defects the cap cut. Run59 replays on 33 tuning PRs:
The extra comments are right about 4 in 10 times, against about 58% for the rest.
Tests
KAI_REVIEW_REFILL=0turns it off in standard and deep;=1forces it on in light.rcPublishGateholds in standard and deep.TestReviewWithNoNewSettingsMatchesMainfails in this checkout with and without this change (checked by revertingreview_commit_publish.go); everything else incmd/kaipasses. Run withGOWORK=off(a parent go.work interferes in my worktree).Not merged by the author; needs review from Adam or Varsha.
🤖 Generated with Claude Code
Kai context
Written by Kai from the change itself. Kai only edits this section; everything above it is the author's.
rcRefillOn()now follows the review depth the server chose (KAI_REVIEW_TIER), so the refill is on by default forstandardanddeep.KAI_REVIEW_REFILL=0still turns it off everywhere, and=1forces it on in any tier.Changes
cmd/kai/review_commit_publish.go:rcRefillOn()checksKAI_REVIEW_REFILLfirst (0returns false,1returns true, after trimming whitespace).KAI_REVIEW_REFILLis neither0nor1, it falls back toKAI_REVIEW_TIER, matched case-insensitively and trimmed.standardanddeepreturn true.light,auto,free,baseline, unset and anything unrecognised.rcRefillOn()now describes this behaviour. The old comment said the refill was off until a replay had measured it.Testing
All changes are in
cmd/kai/review_commit_publish_test.go.TestRefillGivesGateFreedSlotsToWhatTheCapCutgains a table over tier andKAI_REVIEW_REFILL. Standard and deep publish the two cap-cut defects. Light, auto, free, baseline and unset do not.0disables the refill in standard and deep.1enables it in light.TestRefillPublishesTwoVoteDefectTheCapCut: a defect with two proposals that the cap cut takes a gate-freed slot in standard and deep. It is not published in light or withKAI_REVIEW_REFILL=0.TestPublishGateFallbackHoldsWithRefillOn: the "never publish nothing" fallback inrcPublishGatestill keeps the first selected finding in standard and deep.TestReviewWithNoNewSettingsMatchesMainfails in their checkout both with and without this change. The rest ofcmd/kaipasses when run withGOWORK=off.