Repository navigation
kubernetes: add opt-in backoff after a failed poll - #222
Open
ceolinrenato wants to merge 1 commit into
Open
ceolinrenato wants to merge 1 commit into
ceolinrenato wants to merge 1 commit into
Conversation
Polling a degraded Kubernetes API server at a fixed interval keeps adding load exactly when it can least afford it, which slows recovery. Add an opt-in kubernetes_poll_backoff option that applies capped exponential backoff with jitter after a failed poll, keeping the last known membership in the meantime. Without the option the strategy keeps the fixed polling interval. The backoff curve lives in Cluster.Strategy.Kubernetes.Backoff so it can be tested without sleeping.
ceolinrenato
marked this pull request as draft
November 4, 2026 20:35
ceolinrenato
marked this pull request as ready for review
November 4, 2026 20:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of changes
The "why": when the Kubernetes API server is degraded, polling it at the fixed
:polling_intervalkeeps adding load exactly when it can least afford it. We hit this in production: our pods run different sets of OTP applications on their Erlang nodes, so we had one topology per application and N pollers on the same interval. A failing control plane got worse because every one of them kept retrying at the fixed interval, and recovery took longer than the outage itself.We now run a custom strategy that does a single poll for all of them and backs off after a failure, among other things. The backoff is generic and does not depend on the rest of that strategy, so I thought it was worth contributing back upstream. It does not change the cost of a single poll, it just gives the control plane room to recover between them.
The change is opt-in so nothing changes for existing configurations:
kubernetes_poll_backoffunset (default): fixed interval, identical to today.kubernetes_poll_backoff: [base_ms: 100, max_ms: 30_000, jitter: 0.2]: capped exponential backoff with jitter added after each failed poll.trueuses the defaults (50/60_000/0.2).A failed poll already preserves the last known membership; this only changes when the next poll happens. The attempt counter travels on the scheduled message, so
State.metakeeps its current shape and no public struct changes.The curve lives in
Cluster.Strategy.Kubernetes.Backoff(same internal-module pattern asTLSandEndpointSlice) so it can be unit tested without sleeping.Checked locally against CONTRIBUTING.md:
mix format --check-formatted,mix deps.unlock --check-unused,mix hex.audit, test and prod compiles with--warnings-as-errors,mix test --warnings-as-errors(171 passing),mix docs --warnings-as-errorsandmix hex.build.Separate idea: selectors for heterogeneous node basenames
Not part of this PR. It comes out of the same incident as the backoff, but it is a separate idea and a larger change.
kubernetes_node_basenameis fixed per topology, so a deployment where pods run different sets of OTP applications on their Erlang nodes needs one topology per application. That is what turned one poller into N during the incident above. An alternative would be to let a selector drive a heterogeneous topology: select every application at once and derive each node's basename from a label on the pod.One topology and one poll would then cover several basenames, instead of one topology and one poller per application. This needs its own design discussion, so I'm only raising the idea here. #65 proposed a similar thing at the config level and was closed unmerged; linking it as prior art.
Checklist