Repository navigation
Conversation
antiguru
changed the title
antiguru
force-pushed
the
freshness/cache-prune
branch
from
October 29, 2026 07:46
ba045fd to
fe98718
Compare
antiguru
force-pushed
the
freshness/cache-prune
branch
2 times, most recently
from
October 29, 2026 17:33
f3d3f23 to
e4ff0f6
Compare
antiguru
changed the title
antiguru
force-pushed
the
freshness/cache-prune
branch
from
October 29, 2026 20:20
e4ff0f6 to
69dd609
Compare
antiguru
force-pushed
the
freshness/cache-prune
branch
from
October 30, 2026 09:24
69dd609 to
03f2cd8
Compare
antiguru
force-pushed
the
freshness/cache-prune
branch
2 times, most recently
from
October 30, 2026 18:12
36dbbe0 to
8fe7502
Compare
Resolving a freshness tree locked the main `StorageCollections` mutex once per ingestion leaf on every frontier advance, and the storage controller locked it twice more to look up the tree and the visibility log. Ingestion tick intervals now live in their own map with a generation counter that also changes with any freshness tree. Both controllers cache the resolved tree per collection and re-resolve only when the tree or the generation changed, so an `ALTER SOURCE` still takes effect at the next observation. The storage controller looks up each collection's visibility log once. Visibility logs were pruned by age only, which kept about ten minutes of entries for every global and replica log, most of them without readers. Consumers now register with the logs of their inputs when they are created, report what they recorded through at each read, and deregister when they are dropped. Logs keep only the entries some consumer still needs, and only the last entry without consumers. The age cap remains a backstop for consumers that stop reading, for example in another controller, and the log releases unused capacity after large prunes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
A consumer that stops reading a visibility log is forgotten after the retention, and the entries it needed are pruned by age. When it reads again, the log answers the arrivals it missed with the observation time of its last pruned entry, which understates their local delay but no longer records it as about 1 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
…tries A consumer that had not read an input's log for ten minutes was forgotten, and entries older than ten minutes were pruned by age. A replica outage longer than that lost exactly the entries its recording needed when it came back, and every arrival of the outage recorded the time the replica took to hydrate as its local delay. Consumers now stay registered until they are dropped or their recording stops, and entries older than one period are thinned geometrically by age instead, keeping the oldest entry of every doubling of the age. A thinned entry answers with a later observation, which understates, and a frontier answer after a gap moves the input's arrival readiness along so skew does not grow. Entries older than the histogram retention are dropped. A consumer's mark now follows its recording after each observation. It lagged by one observation, which kept an entry the consumer had already recorded as the oldest of its age, so thinning dropped the first entry it still needed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
Keeping one entry per doubling of the age answered an arrival in the middle of a long outage by an entry up to half its age younger, which understated its local delay by several histogram buckets and hid the outage. Entries are now thinned by half buckets of the recorder's bucket scheme, which keeps the gap between kept entries within one bucket, so thinning understates a delay by at most one bucket, for about twice the buckets per octave entries per doubling of the age. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
The compute controller does not record the freshness of transient collections, so a transient collection registered with its inputs' logs never read them, and its unset mark kept every entry until the collection was dropped. It no longer registers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
antiguru
force-pushed
the
freshness/cache-prune
branch
from
October 31, 2026 19:53
8fe7502 to
4eb33bc
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Caches resolved freshness trees behind a generation counter, for each collection and each of its inputs, so a frontier advance no longer locks
StorageCollectionsper source leaf. Visibility logs keep what their consumers have not recorded for as long as the consumer exists, and thin entries older than one period to one per half histogram bucket of age, dropping entries past the histogram retention. Transient collections do not register as consumers.Expiring consumers by age lost the outages local delay should show: on staging, a view whose replica was dropped for 10.5 minutes recorded the new replica's 5 s hydration as the local delay of every outage arrival. Thinning only understates, by at most one bucket, and bounds a log to about 125 old entries for 30 days at 4 buckets per octave.
Part of P-CPU-328, design in #40320. Recording and reporting are behind
enable_wallclock_freshness_recordingandenable_wallclock_freshness_reporting, both off in production, so this changes nothing users see.🤖 Generated with Claude Code
https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R