Skip to content

compute: Render regions inline without timely subgraphs - #40387

Draft
antiguru wants to merge 4 commits into
MaterializeInc:mainfrom
antiguru:compute-noop-regions-v2
Draft

antiguru wants to merge 4 commits into
MaterializeInc:mainfrom
antiguru:compute-noop-regions-v2

Conversation

@antiguru

@antiguru antiguru commented Nov 1, 2026

Copy link
Copy Markdown
Member

Compute rendering wraps imports, objects, let bindings, the main body, and joins, reduces, top-k and sinks in timely regions. Each region is a full subgraph with its own progress tracker and all-worker progress exchange, so a frontier advance is propagated and broadcast once per region it crosses, on every worker. For replicas with many idle dataflows this per-tick bookkeeping dominates CPU, and it grows with the worker count rather than with data.

This change renders those regions inline: the closure runs in the enclosing scope and entering or leaving is the identity, through crate-local inline_named, enter_inline and leave_inline in render/inline_region.rs. Operators are unchanged. BuildRegion stays a real region, because index imports enter it from the dataflow scope.

Measured locally on 16 workers with static tables ticking once per second, upstream/main against this branch: 48 dataflows go from 1.38 to 1.00 cores (-28%), 426 dataflows from 12.5 to 8.4 cores (-32%), with scopes dropping from 7,968 to 2,252, reachability log records by 82%, and hydration from about 65 s to 50 s. Lag is unchanged.

Introspection gets shallower: mz_dataflow_addresses and mz_dataflow_operator_parents lose the region levels, the dataflow visualizer nests less, and mz_lir_mapping operator ranges cover fewer operators. introspection/relations.slt changes accordingly. Testdrive files that inspect operator addresses may need updates once CI runs them.

🤖 Generated with Claude Code

antiguru and others added 4 commits October 31, 2026 16:25
Each timely region is a full subgraph with its own progress tracker and
progress exchange, and the scheduler visits it several times per tick.
Compute rendering only uses regions to group operators, so run every
region body in the enclosing scope and make entering and leaving the
identity. Operators are unchanged, only the region scopes disappear.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant