controllers: record local delay and skew for freshness attribution by antiguru · Pull Request #40335 · MaterializeInc/materialize

antiguru · 2026-10-29T06:37:05Z

Records where an object's freshness latency comes from, so the slow hop in a chain of sources, views and indexes is visible. Each arrival's end-to-end latency splits into upstream (until the slowest direct input showed it), skew (until every input's frontier passed the timestamp the object needs, capped at the wait the input grids cause when every input is on time), and local (what the hop itself adds). Local is recorded as measure = 'local' and skew as measure = 'skew', the latter only for objects with two or more direct inputs. Upstream is the slowest input's own end-to-end latency and is not stored. Storage-owned collections have no inputs, so their local delay equals their end-to-end latency and they record no local rows.

Skew separates grid alignment from processing. A join of a 7 s and an 11 s input waits up to 10 s for the faster input to pass the slower one's timestamp, which staging recorded as the join's local delay before this split. Both measures may understate and never overstate.

Part of P-CPU-328, design in #40320. Recording and reporting are behind enable_wallclock_freshness_recording and enable_wallclock_freshness_reporting, both off in production, so this changes nothing users see.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R

Adds the local delay recorder and the visibility log that it reads its inputs from. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R

Both controllers now record a second measure next to end_to_end in the global and per-replica freshness histograms. The local delay of an arrival is the time the object made it visible minus the time its slowest direct input did, so the object in a chain that adds the lag stands out. Every recorded collection keeps a visibility log of when it made arrivals visible. Storage-owned collections and storage sinks keep theirs in StorageCollections, so the compute controller reads them with the observation times of the controller that writes the collection. A compute collection's inputs are the imports its freshness tree was built from, and a replica collection reads the logs of its compute inputs on the same replica. Storage collections have no inputs, which makes their local delay equal to their end-to-end latency. Logs are pruned by age to ten periods, at least ten minutes. A consumer that lags an input by more than that records a local delay of 0 for those arrivals. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R

The inputs of a compute collection's local delay are now the imports that contributed a freshness tree, decided when the dataflow is created. An input without a tree never limits readiness. An input with a tree whose log cannot answer an arrival, because it has not recorded yet, was pruned, or is not installed on the replica, counts as ready at the observation. This records a local delay of 0, which understates and never overstates. Before, such an input was dropped and the delay was measured against the remaining inputs. A collection whose write frontier becomes empty records that every arrival is visible from then on, so its consumers' local delay counts from its completion instead of dropping to 0. The compute instance looks up its inputs' logs in StorageCollections once, when the collection is added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R

Rehydrating a replica put the kept recordings back into the new replica collections, also where none was kept. That replaced the recording the new collection starts with by none, and the next observation started a recording at the first observed frontier instead of by the creation rule. Only kept recordings replace new ones now. A visibility log answers pruned arrivals with the observation of its last pruned entry, so the comments and tests of the age pruning say so. The test recorder configurations gain the histogram retention. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R

Local delay measured from the slowest input showing an arrival charged the wait for the inputs' frontiers to align, up to 10 s for a join of a 7 s and an 11 s source, to the join itself, and leaked an input stall into the join for the arrivals just before it. Visibility logs now record the frontier timestamp next to the reflected arrival, and the recorder splits each arrival into skew, the wait until every input's frontier passed the timestamp the object needs, and local, the time from then until the object shows it. Local uses the raw observation time, so a frontier ahead of the wall clock no longer overstates it. Objects with two or more inputs record the new `skew` measure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R

Arrivals that the period book defers are charged at the first observation that reflected them, but input logs are read at the later call. An input entry observed in between answered after the charged observation and inflated skew, so both ready times are now clamped at the observation being charged. The list of held-back observations is capped by merging the neighbours with the closest observation times, which only understates. The property test now generates frontiers ahead of the wall clock and refresh inputs, clamps its oracle the same way, and checks that the split covers the end-to-end charge per arrival. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R

A frontier ahead of the wall clock lets an input show an arrival before the arrival's window starts, which made local delay and skew together exceed the end-to-end charge. Both ready times are now clamped up to the window start. The clamped latencies vary by window, so a group is split at the windows that start after each ready time, and the varying parts are bucketed by window start like the end-to-end span, which keeps the work bounded by the buckets touched. The property test checks the per-arrival split against the end-to-end charge without exceptions, generates longer leads with smaller leaves, and a new test drives the list of held-back observations past its bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R

Skew measured until every input's frontier passed the needed timestamp also caught an input that showed an arrival and then stalled before its frontier passed that timestamp. A join with a daily refresh view would charge a source stall around the refresh as skew to a whole day of arrivals. Skew is now capped at the wait the inputs' grids cause when every tick and refresh is on time, and the excess stays in upstream, which is not stored. Local delay is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R

The skew cap was the wait from the needed timestamp until every input's frontier passed it on time. An input that is itself a join or a refresh view shows an arrival only after its own inputs align, later than the needed timestamp, so the cap admitted up to that input's own wait of lateness into skew. The cap is now the on-time skew itself: from the last input showing the arrival, each from its own tree, to every input passing the needed timestamp. The compute controller keeps each input's tree next to its log for this. Unaligned leaves, whose phase is unknown, contribute no wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R

An unaligned leaf ticks at a phase the controller does not know. Taking its wait as 0 underestimated both when an input passes the needed timestamp and when it shows the arrival, and the latter raised the cap for an input that nests an unaligned leaf below a join or a refresh. The cap now takes the earliest pass time and the latest show time over all phases, an unaligned leaf showing an arrival at the latest one interval after it, so it never exceeds the on-time skew for any phase. An input whose tree no longer resolves now counts as ready at the observation instead of skipping the collection's recording, including its own visibility log. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R

The skew property test now generates inputs whose trees nest joins and refresh nodes in any order, such as a join that reads a refresh view or another join, and checks them against its own on-time model at the real phase of unaligned leaves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R

Storage-owned collections have no inputs inside Materialize, so their local delay equals their end-to-end latency, and their local rows repeated their end-to-end rows at the cost of a second recorder. A recording can now be limited to the end-to-end latency, and the storage controller records its collections that way. Their visibility logs, which compute consumers read, stay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R

antiguru added this pull request to stack #40339 October 29, 2026 06:37

antiguru changed the title ~~freshness/local delay~~ controllers: record local delay for freshness attribution Oct 29, 2026

antiguru force-pushed the freshness/local-delay branch from a3bdae7 to f7d1d60 Compare October 29, 2026 07:46

antiguru force-pushed the freshness/local-delay branch from f7d1d60 to 8987e20 Compare October 29, 2026 11:51

antiguru changed the title ~~controllers: record local delay for freshness attribution~~ controllers: record local delay and skew for freshness attribution Oct 29, 2026

antiguru force-pushed the freshness/local-delay branch from 4e1d0e4 to 63282ac Compare October 29, 2026 20:20

antiguru force-pushed the freshness/local-delay branch from 63282ac to 4f164e1 Compare October 30, 2026 09:24

antiguru force-pushed the freshness/local-delay branch from 4f164e1 to 34094e6 Compare October 30, 2026 09:52

antiguru and others added 12 commits October 31, 2026 20:51

storage-types: model local delay with input visibility logs

c33a502

Adds the local delay recorder and the visibility log that it reads its inputs from. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R

antiguru force-pushed the freshness/local-delay branch from 34094e6 to f2a7346 Compare October 31, 2026 19:53

Navigation Menu

Repository navigation

controllers: record local delay and skew for freshness attribution - #40335

controllers: record local delay and skew for freshness attribution#40335
antiguru wants to merge 12 commits into
freshness/storage-recordingfrom
freshness/local-delay

antiguru commented Oct 29, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

antiguru commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

antiguru commented Oct 29, 2026 •

edited

Loading