feat(seinode): pace pod-template drift with a per-namespace roll budget (spec 012)

[bdchatham]

Summary

A pod-template drift (seid image, cell sidecar image, isolation) makes every affected SeiNode build its update plan in the same reconcile. A cell sidecar bump therefore restarts every node pod in the cell at once. On 2026-10-07 this paused arctic-1 for about 30 seconds in each cell, and every pacific-1 RPC node in prod restarted together. This PR adds a per-namespace drift-roll budget (spec 012, PLT-1399).

Design

• Config: rollout.driftUpdateBudgetPercent in the controller config file. 0 or unset keeps today's behavior. Values outside 0..100 fail startup validation.
• Slots: a namespace gets max(1, N*percent/100) slots, where N counts its Running, unpaused SeiNodes.
• Slot order: nodes with NodeUpdateInProgress=True (sorted by name), then drifted nodes waiting for a slot (sorted by name). A node holds a slot when its position is below the slot count. Every node computes the same order from the same cache, so a stale read delays a roll; it never overfills the slots.
• Plan gate (every node): in ResolvePlan, a drift update plan with no slot is dropped. The node reports NodeUpdateInProgress=False/UpdateDeferred with the slot holders, and it retries on the 30-second status poll.
• Template gate (nodeConfig nodes): their StatefulSet is RollingUpdate, so the reconciler also skips the StatefulSet apply while the drift waits. The gate applies only when no plan is active and no reset or hold change is pending. It mirrors the existing holdForWorkflow skip.
• Not paced: resets, holds, config updates, init plans, and resizes.

Known limit

A drift update stuck in progress keeps its slot, so the namespace's further drift waits until an operator clears it. The UpdateDeferred message names the holder.

Verification

• Planner tests cover: slot count and order, in-progress nodes first, the minimum of one slot, paused and non-Running exclusions, disabled and unwired cases, the plan gate, a reset ignoring the budget, and the template-gate truth table.
• A reconciler test shows that a waiting nodeConfig node keeps its StatefulSet template and reports UpdateDeferred. It fails if the controller gate is removed. The plan-gate test likewise fails without the gate.
• A platform test covers the config load and the 0..100 bound.
• gofmt, go vet, golangci-lint --new-from-merge-base, and go test ./... (20 packages) are clean. make manifests generate leaves no diff.
• Next: a harbor e2e with the budget at 25% during the next cell sidecar bump.

🤖 Generated with Claude Code

[bdchatham]

@seidroid review

[cursor]

PR Summary

Medium Risk
Changes rollout behavior for all pod-template drift in namespaces where the budget is enabled; incorrect slot or pin logic could delay updates or cause unexpected StatefulSet rolls on nodeConfig nodes.

Overview
Adds a per-namespace drift-roll budget so cell-wide pod-template drift (seid image, sidecar image, isolation) no longer restarts every SeiNode at once.

Operators set rollout.driftUpdateBudgetPercent in controller config (0 or unset = unchanged behavior; values outside 0–100 fail startup). Each namespace gets max(1, N × percent / 100) slots among Running, unpaused nodes. Slot order is in-progress updates first (by name), then drifted waiters (by name). The planner lists namespace SeiNodes via an uncached reader wired from main.go.

Plan gate: ResolvePlan drops pod-template drift update plans when no slot is free and sets NodeUpdateInProgress=False with reason UpdateDeferred and a message naming slot holders. Resets, holds, config updates, init, and resize are not gated.

Template gate: For nodeConfig nodes whose StatefulSet uses RollingUpdate, the node reconciler calls DriftRenderNode before applying the StatefulSet so waiters render from a copy pinned to running seid/sidecar/isolation while other spec fields (pause, resize, config refs) still apply.

Spec 012 documents the feature; planner, reconciler, and platform load tests cover slots, deferral, pinning, and config validation.

^{Reviewed by Cursor Bugbot for commit 372b45c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Stale Bugbot comment from a previous run.

[seidroid]

Adds a per-namespace budget for pod-template drift updates, enforced in two places: a plan gate in ResolvePlan, and a StatefulSet-apply gate for nodeConfig nodes. The template gate blocks the replicas=0 apply that spec.paused depends on, so pausing a drifted nodeConfig node does nothing while the budget is full; that breaks an existing contract and decides the verdict. Of codex's two findings I kept both, as suggestions: the fallback that skips waiting nodes (line 272) and the bypass while another plan is active (line 292). No other reader contributed.

Blocking

1 finding on the changed lines, as inline comments.

Non-blocking

3 findings on the changed lines, as inline comments.

_{seidroid review · decision request_changes · session 564dbb6bb94f48c89d8fc91df71f21be · turn resp_claude_c6c026935ebc1b656624ed4bc0245d52 · item 99749fc78fa25116b5e551facda80019}

Findings: 1 blocking | 3 non-blocking | 4 posted inline

[bdchatham]

@seidroid review

[seidroid]

Since my last review, the template gate no longer skips the StatefulSet apply. It renders from a copy pinned to the running images, so spec.paused and other fields still apply, and paused nodes are never pinned. The fallback now counts waiting nodes, the budget list reads uncached while the node controller reconciles one node at a time, and the plan-active leak is documented in Known limits. I checked all four threads against the code and each is addressed, with nothing new that blocks; codex found nothing, which agrees with my reading.

_{seidroid review · decision approve · session 564dbb6bb94f48c89d8fc91df71f21be · turn resp_claude_63605e0b752ca3d6d24e5a0926dd917d · item 3fd5a80aae435e44816fcba76fd4842a}

Findings: 0 blocking | 0 non-blocking | 0 posted inline

Superseded: the latest review found nothing blocking in this change.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit af57277. Configure here.}

[bdchatham]

@seidroid review

[seidroid]

Since my last review, the only change is in pinnedToRunning: when status has not yet recorded a running image, it now takes that image from the live StatefulSet's template (via the new noderesource.TemplateImages), and it leaves the node unpinned when no running seid image is known. That is correct, because the unpinned case has no pod to roll and the plan gate still defers the plan; my four earlier threads stay addressed, and codex again found nothing, which matches my reading.

_{seidroid review · decision approve · session 564dbb6bb94f48c89d8fc91df71f21be · turn resp_claude_ca4aa634be81f796c62b2a1439f30ef2 · item dd7b616b8afe5a5499e36a4ae26342ae}

Findings: 0 blocking | 0 non-blocking | 0 posted inline