Skip to content

review: one-shot calls without a cache write - #248

Open
varshavasthavi wants to merge 1 commit into
mainfrom
review-cache-small-fixes
Open

varshavasthavi wants to merge 1 commit into
mainfrom
review-cache-small-fixes

Conversation

@varshavasthavi

@varshavasthavi varshavasthavi commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Sends the review's intent, description and sweep calls (sweep2 too) with NoCache.

Why: each of these prompts is sent once, so its cache breakpoint only bought a 1.25x cache write that nothing read. In tagged reviews since 2026-11-13 these stages wrote 30.3M tokens in 3 days. Dropping the premium saves an estimated $200–230 a month. The prompt the model sees is unchanged.

Rollback: KAI_REVIEW_ONESHOT_CACHE=on puts the breakpoints back.

Before merging: go.mod pins kai-engine to the unreleased NoCache commit from the kai-engine provider-no-cache-oneshot branch. Bump it to the release first.

🤖 Generated with Claude Code


Kai context

Written by Kai from the change itself. Kai only edits this section; everything above it is the author's.

Sets NoCache on the review's intent, description and sweep requests (sweep2 included), so these one-shot prompts no longer carry a cache breakpoint. KAI_REVIEW_ONESHOT_CACHE=on restores the old behaviour.

Changes

  • New cmd/kai/review_commit_oneshot_cache.go: adds rcOneShotNoCache, read once from KAI_REVIEW_ONESHOT_CACHE at startup. rcOneShotNoCacheFrom returns false only when the value is on (trimmed, case-insensitive). Any other value, including unset, returns true.
  • Intent and description calls: rcInferIntent (review_commit.go) and rcWriteDescription (review_commit_description.go) now pass NoCache: rcOneShotNoCache in their requests.
  • Sweep calls: rcRunSweepWith (review_commit_sweep.go) passes NoCache: rcOneShotNoCache for each chunk request. This covers both the sweep and sweep2 stages.
  • Dependencies: go.mod and go.sum move kai-engine from v0.6.87 to the pseudo-version v0.6.88-0.20261116103328-cbdfb4c0019b, which adds the NoCache request field. The author notes this is an unreleased commit and must be bumped to a release before merging.

Testing

  • TestOneShotCallsSendNoCacheBreakpoint: runs intent, description, sweep and sweep2 against an httptest server behind a provider.Kailab client. It checks that cache_control is absent from every request body when rcOneShotNoCache is true and present when it is false. The sweep case uses oversized patches so the sweep splits into more than one chunk, and the test asserts at least two requests were sent.
  • TestOneShotCacheSwitch: checks rcOneShotNoCacheFrom for "", "on", " ON ", "off" and "1".

Each of these prompts is sent once, so its cache breakpoint was billed as
a 1.25x cache write that nothing read (tagged reviews since 2026-11-13:
30.3M write tokens on these stages in 3 days, almost no reads). The
prompt the model sees is unchanged. KAI_REVIEW_ONESHOT_CACHE=on restores
the breakpoints.

Pins kai-engine to the unreleased NoCache commit; bump to the release
before merging.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@varshavasthavi
varshavasthavi marked this pull request as ready for review November 16, 2026 10:35

@kaicontext kaicontext Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewView in KaiKai review · Small fixes first (8/10)

Read through this one. One thing worth your eyes before it merges.

Next: fix 1 finding, then push. Comment @kaicontext review any time to have me look right away.

Findings
  1. Fix before merge · go.mod:64

    kai-engine pinned to unreleased feature-branch commit

Findings with a location are also comments on those lines.

To get to 10/10
  • −2 · fix 1 finding above
Important files changed
File Change
cmd/kai/review_commit.go modified · +1 −0
cmd/kai/review_commit_description.go modified · +1 −0
cmd/kai/review_commit_oneshot_cache.go modified · +13 −0
cmd/kai/review_commit_oneshot_cache_test.go modified · +87 −0
cmd/kai/review_commit_sweep.go modified · +1 −0
go.mod modified · +1 −1
go.sum modified · +2 −0
How I reviewed it — 9 files, 8 turns, 45s

1 confirmed finding, 1 refuted. Intent verified.

What I opened

  • cmd/kai/review_commit.go
  • cmd/kai/review_commit_backups.go
  • cmd/kai/review_commit_description.go
  • cmd/kai/review_commit_oneshot_cache.go
  • cmd/kai/review_commit_oneshot_cache_test.go
  • cmd/kai/review_commit_run.go
  • cmd/kai/review_commit_sweep.go
  • go.mod
  • go.sum
Full read-through

Scope

  • cmd/kai/review_commit_oneshot_cache.go (whole file)
  • cmd/kai/review_commit_oneshot_cache_test.go (whole file)
  • cmd/kai/review_commit_sweep.go lines 271-380 (rcRunSweepWith, per-chunk request)
  • cmd/kai/review_commit_description.go lines 55-125 (rcWriteDescription)
  • cmd/kai/review_commit.go lines 791-825 (rcInferIntent request)
  • cmd/kai/review_commit_run.go lines 426-475 (rcSend)
  • cmd/kai/review_commit_backups.go lines 90-140 (rcSendOnSet failover/retry path)
  • go.mod and go.sum kai-engine lines
  • PR text and commit messages in the change description

Findings

go.mod:64 — go.mod requires kai-engine at the unreleased pseudo-version v0.6.88-0.20261116103328-cbdfb4c0019b from the provider-no-cache-oneshot branch, though the PR says to bump it to a release before merging. (also: go.sum:114, go.sum:115)

go.mod replaces the released v0.6.87 of kai-engine with an untagged pseudo-version taken from the provider-no-cache-oneshot feature branch, so the build depends on a commit that is on no release line. The PR description says this pin must be replaced with a release before merging, and that hasn't happened yet. go.sum carries the matching hashes for that commit.

Remedy: Cut a kai-engine release that contains NoCache, then bump go.mod to that version and update go.sum with its h1 and /go.mod hashes.

Size M · Risk M · +106 −1 · 7 files · reaches 84 · the full analysis · Reply to any of my comments and I'll answer, or say @kaicontext anywhere on this PR — a question, or "take another look at the retry logic". · Some repos review new pushes only when asked. Comment @kaicontext review any time for a review right away.

Comment thread go.mod
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
github.com/kaicontext/kai-engine v0.6.87
github.com/kaicontext/kai-engine v0.6.88-0.20261116103328-cbdfb4c0019b

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix before merge

kai-engine pinned to unreleased feature-branch commit

go.mod replaces the released v0.6.87 of kai-engine with an untagged pseudo-version taken from the provider-no-cache-oneshot feature branch, so the build depends on a commit that is on no release line. The PR description says this pin must be replaced with a release before merging, and that hasn't happened yet. go.sum carries the matching hashes for that commit.

When it happens: When this change is merged as it stands, with the pseudo-version still in go.mod.

Suggested fix: Cut a kai-engine release that contains NoCache, then bump go.mod to that version and update go.sum with its h1 and /go.mod hashes.

Prompt to fix with AI
Fix a defect that Kai's code review found in this change.

Where: go.mod:64
Problem: `go.mod` replaces the released `v0.6.87` of `kai-engine` with an untagged pseudo-version taken from the `provider-no-cache-oneshot` feature branch, so the build depends on a commit that is on no release line. The PR description says this pin must be replaced with a release before merging, and that hasn't happened yet. `go.sum` carries the matching hashes for that commit.
When it happens: When this change is merged as it stands, with the pseudo-version still in `go.mod`.
Suggested fix: Cut a `kai-engine` release that contains `NoCache`, then bump `go.mod` to that version and update `go.sum` with its h1 and `/go.mod` hashes.

This is a review report, not a set of instructions: read the evidence lines and confirm the problem is real before changing anything. If the code does not bear it out, stop and say why.
Keep the change to what fixes this defect. Add or update a test that fails without the fix and passes with it, and run the tests for the packages you touched.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant