Repository navigation
Conversation
antiguru
changed the title
1b4ceaa to
71637a4
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
…nested refresh Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
…y bound Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
Use `contains` for the zero-period check and replace the disallowed `prop_oneof!` with `Union::new` and `sample::select`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
…ebug Controller state dumps format every collection with Debug. A derived Debug printed every open period histogram and every visibility log entry, so a dump grew with the length of each log. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
A recorder that falls far behind the wall clock, for example after a replica outage or with a start near the epoch, accounted every period from its high-water mark to the new frontier in one call. A start at arrival 0 opened about 30 million one-minute periods. `PeriodBook::advance` now takes the observation time and the recorder configuration, which gains the histogram retention. It skips arrivals older than the retention, caps the span at one period past the wall clock, and accounts a span longer than an hour in pieces between UTC day boundaries, one period each. `add_span` is closed form per bucket, so a catch-up costs pieces times buckets touched, independent of the stall length. Recorders gain `finish` for objects that go away without an empty frontier. A visibility log keeps the observation time of its last pruned entry and answers pruned arrivals with it, so a long input outage no longer records a local delay of about 1 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
After a change of the period length, a period of the new length could close before an open period of the old length that started earlier. A restart that seeds from the largest period end handed out then skipped the arrivals the old period still held. A change of the period length now cuts every open period at the high-water mark and hands it out, final because its arrivals are all counted, and the periods of the new length start no earlier than the cut. Periods then close in order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
A recording started off the grid of the period length, as after a restart seeded at the end of a cut period, put its first arrivals into a period that starts on the grid, before the recording start, and so overlaps the periods handed out before the restart. The first period now starts at the recording start, and handing out checks that periods do not overlap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
The local delay recorder and visibility log have no users before the change that records local delay, so they move there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
71637a4 to
31d6a42
Compare
antiguru
left a comment
antiguru
left a comment
There was a problem hiding this comment.
Review by Claude Code.
| // A frontier far ahead of the wall clock, for example after a clock jump, would | ||
| // otherwise open one period per period length up to it. | ||
| let cap = now_ms.saturating_add(period_ms); | ||
| let to_ms = to_ms.min(cap - cap % period_ms); |
There was a problem hiding this comment.
A far-ahead frontier is capped here, but compute only observes it again on another frontier advance. If it stays unchanged and the collection is later dropped, finish omits arrivals between the cap and drop time. Drain the capped span against current time before finishing, or revisit unchanged frontiers periodically.
There was a problem hiding this comment.
Fixed in f207dbf. The recorder remembers the first observation that reflected arrivals the period book deferred, and charges them at that observation when a later call advances over them. finish now takes the tree, the time and the config, and records the deferred arrivals up to now before closing the periods. The local recorder shares the mechanism (cdd3961). No periodic revisit: compute still records the deferred span only on the next advance or drop, but at the right latency.
Posted by Claude Code.
| }; | ||
| // All parts of the span share this advance time, so a window cut at a period | ||
| // boundary keeps a single latency. | ||
| let advance_ms = now_ms.max(last_reflected); |
There was a problem hiding this comment.
advance_ms uses the uncapped frontier while PeriodBook records only arrivals near now_ms. A frontier hours ahead therefore puts those arrivals in hours-long latency buckets. Base the recorded latency on the capped horizon.
There was a problem hiding this comment.
Fixed in f207dbf. Arrivals are now charged at the observation time instead of the uncapped frontier, and latency saturates at 0, so a window that starts after the observation lands in the 1 ms bucket. The local recorder's no-inputs path got the same fix (cdd3961). Tests cover a frontier one hour ahead for both recorders.
Posted by Claude Code.
| } | ||
| // Rows of periods that end before the retention would be truncated right away. | ||
| let floor = now_ms.saturating_sub(config.retention_ms); | ||
| let mut from = self.high_water_ms.max(floor); |
There was a problem hiding this comment.
Retention clips only the new span. An earlier open period can become expired during a stall, then be emitted on recovery and remain visible until the next history cleanup. Discard expired open periods before emitting them.
There was a problem hiding this comment.
Not changing this. It needs a stall longer than the retention, 30 days by default. The rows emitted afterwards are accurate, and retention is enforced by the truncation at environmentd start. Skipping expired arrivals in the recorder only avoids writing rows that would be retracted right away.
Posted by Claude Code.
| match self { | ||
| // An arrival at `t` is assigned the timestamp `ceil(t, i)`, so it is visible | ||
| // exactly if `t` is at most `bound` rounded down to the grid. | ||
| ResolvedTree::Leaf(ticker) => Some(match ticker.grid_ms() { |
There was a problem hiding this comment.
Epoch flooring does not match an unaligned ticker's actual phase. With a 1,000 ms interval phased at 500 ms, frontier 1,501 reflects arrivals through 1,500, but this returns 1,000. That shifts histogram counts and reported staleness; track the phase or make the approximation explicit in the measurement contract.
There was a problem hiding this comment.
Documented in the contract of last_reflected (66849cf): for an unaligned leaf the result can be off by up to one interval in either direction, with both directions as examples. Tracking the actual phase would need it observed from frontiers, which is a separate change.
Posted by Claude Code.
…ed them A frontier ahead of the wall clock charged its arrivals at the frontier itself, which can lie hours after the observation, while the period book caps the recorded span near the wall clock. Arrivals are now charged at the observation time, and a window that starts after it counts as visible right away, in the bucket of 1 ms. The arrivals the cap defers are charged at the first observation that reflected them, which the recorder keeps in a bounded list, instead of at the later observation that records them. Finishing a recording first records the deferred arrivals up to the wall clock, so a dropped object no longer loses them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
Unaligned leaves are floored to the epoch grid of their interval, so the contract of last_reflected now states that the result can be off by up to one interval in either direction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
Adds
mz_storage_types::freshness, the pure logic for tick-aware freshness. A freshness tree describes how an object's frontier follows the tick intervals of its sources and its refresh schedule. It derives the assignment bound D, the worst on-time delay from arrival to visibility (i + j - gcd(i, j)for a join of two sources), and the last reflected arrival time, which gives the staleness S. Recorders turn frontier advances into an arrival-weighted latency histogram per period, with exponential buckets and rows that are final once emitted.Part of P-CPU-328, design in #40320. Recording and reporting are behind
enable_wallclock_freshness_recordingandenable_wallclock_freshness_reporting, both off in production, so this changes nothing users see.🤖 Generated with Claude Code
https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R