Skip to content

cluster: Do not exert merge effort on spine layer 0 - #40385

Open
antiguru wants to merge 2 commits into
MaterializeInc:mainfrom
antiguru:compute-exert-skip-layer0
Open

antiguru wants to merge 2 commits into
MaterializeInc:mainfrom
antiguru:compute-exert-skip-layer0

Conversation

@antiguru

@antiguru antiguru commented Oct 31, 2026 •

Copy link
Copy Markdown
Member

Every frontier advance appends an empty batch to each arrangement at spine layer 0. When a worker's smallest batch holds a single update, it sits at layer 0 too, so every tick starts a merge there. The exertion policy grants effort for any layer with two batches, which re-activates the arrange operator, and the merged batch moves back to layer 0, so the cycle repeats for as long as the arrangement is idle.

Measured on a 2-worker replica with a small join and reduce, the affected ArrangeAccumulable and ReduceAccumulable operators drop from 2.07 to 1.02 schedules per tick, their enclosing BuildRegion from 4.10 to 1.52, and merge grants from 54 to 15 per second. A plain differential reproduction shows the same pattern, and TimelyDataflow/differential-dataflow#1063 fixes the spine side. Turning exertion off entirely did not change replica CPU measurably on a 426-dataflow workload, so this removes wasted wakeups rather than a large CPU cost.

Layer 0 merges hold at most two updates and complete with the next insertion's fuel, so the policy now stops before layer 0. The new dyncfg compute_arrangement_exert_skip_layer0 (on by default) controls this. The policy is installed into each timely runtime before configuration arrives, so compute stores the flag in a process-wide atomic on each configuration update and the policy reads it per evaluation, which lets the flag be turned off without a replica restart. The policy moves into exert_effort with unit tests for both flag settings.

🤖 Generated with Claude Code

antiguru and others added 2 commits October 31, 2026 19:48
Every frontier advance appends an empty batch to each arrangement, and it
lands at layer 0. When a worker's smallest batch holds a single update, it
also sits at layer 0, so each tick starts a merge there. The exertion policy
grants effort for any layer with two batches, and the spine re-activates the
arrange operator to apply it. The merged batch is then moved back to layer 0,
so this repeats on every tick for as long as the arrangement is idle. On a
16-worker replica this roughly doubles the schedules of small arrangements
and their reduce operators, and of every scope above them.

Layer 0 merges hold at most two updates and complete with the fuel of the
next insertion, so the policy now stops before layer 0. The policy moves into
a function so it can be unit tested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`compute_arrangement_exert_skip_layer0`, on by default, controls whether the
exertion policy skips spine layer 0. The policy is installed into each timely
runtime before configuration arrives, so compute stores the flag in a
process-wide atomic on every configuration update, and the policy reads it on
each evaluation. Turning the flag off restores the previous policy without a
replica restart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@antiguru
antiguru marked this pull request as ready for review October 31, 2026 19:13
@antiguru
antiguru requested review from a team as code owners October 31, 2026 19:13

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant