Skip to content

design: tick-aware freshness reporting and recording - #40320

Draft
antiguru wants to merge 19 commits into
MaterializeInc:mainfrom
antiguru:tick-aware-freshness-design
Draft

antiguru wants to merge 19 commits into
MaterializeInc:mainfrom
antiguru:tick-aware-freshness-design

Conversation

@antiguru

@antiguru antiguru commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Design notes for P-CPU-328 (Granular Freshness Monitoring): report freshness in milliseconds as max(D, S), where D is the assignment bound derived from a controller-side freshness tree and S the measured staleness, and record arrival-weighted latency histograms per object and period, end to end and as local delay per hop, for the Console to query. The implementation plan is next to the design.

The document is a working record for review comments and is not meant to merge alongside the implementation, which is the stack #40330 to #40338. misc/lean/Mz/Freshness.lean proves the two-input bound i + j - gcd(i, j) and its tightness.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R

antiguru and others added 6 commits October 28, 2026 22:16
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
antiguru and others added 13 commits October 29, 2026 07:38
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
Staging showed that joins of inputs with different tick grids recorded the
wait for the coarser grid as local delay, and that an outage longer than the
10 minute consumer expiry recorded the new replica's hydration time instead
of the outage. Each arrival now splits into upstream, skew and local, and
input logs keep entries for their consumers until they are dropped, thinned
geometrically by age.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
…oles

Record the overnight staging run: start recordings after a restart at the
controller's start, treat introspection collections of a cluster without
replicas as a hole, explain why the per-minute history reads one minute
later than the histogram, add measured row cost per object, and note that
local delay picked up a region-wide consensus latency rise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R
…or storage

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LD5gyKhqBoA2DEzJk8jR1R

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant