fix: keep IDLE and CONNECTING SubConns on the hashring - #8
Merged
Merged
Conversation
Ring membership followed READY state exactly (#4). That policy causes constant churn on healthy clusters. Servers that limit connection age send GOAWAY on an interval, the client SubConn moves READY to IDLE, and the member leaves the ring until it reconnects. Every exit remaps the member's keys to its neighbors, and every re-entry maps them back. Callers that depend on stable key placement, such as caches keyed by ring owner, lose locality on each cycle. SpiceDB hit this class of churn before and fixed it with state-insensitive ring membership (authzed/spicedb#1310). Its dispatch servers send GOAWAY every 30 seconds by default. Ring membership now changes on failure, not on every state change: - A SubConn joins the ring when the resolver provides it. - IDLE and CONNECTING SubConns keep their membership. These states normally resolve in milliseconds, and the balancer reconnects IDLE SubConns at once. - TRANSIENT_FAILURE and Shutdown remove the member. Keys move to the next closest backend. This keeps the fast failover from #4 for backends that refuse connections. The trade-off is the black-hole case: an address that accepts no connection holds its keys in CONNECTING until the dial times out, once, and then its keys move. The dial timeout is MinConnectTimeout (20 seconds by default) and the dialer can lower it to bound the window. The e2e test sets it to 300ms and shows prompt failover.
tstirrat15
approved these changes
Sep 25, 2026
tstirrat15
left a comment
Contributor
There was a problem hiding this comment.
This makes sense to me. Let's get it in and test it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Since #4, ring membership follows READY state exactly. That policy causes constant churn on healthy clusters. Servers that limit connection age send GOAWAY on an interval, the client SubConn moves READY to IDLE, and the member leaves the ring until it reconnects. Every exit remaps the member's keys to its neighbors, and every re-entry maps them back. Callers that depend on stable key placement, such as caches keyed by ring owner, lose locality on each cycle.
There is history here: SpiceDB hit this class of churn and fixed it with state-insensitive ring membership in authzed/spicedb#1310 ("the previous implementation was recomputing the hashring any time a subconnection moved from ready->idle or back, which happened frequently"). SpiceDB's dispatch servers send GOAWAY every 30 seconds by default (
dispatch-cluster-max-conn-age), so under READY-gated membership every node's view of every peer leaves and rejoins the ring on that interval, forever, in a healthy cluster.The problem #4 fixed is real: before it, a key hashed to a dead-but-still-resolved backend hung until the resolver caught up. This PR keeps that fix for the common case and removes the churn.
Change
Ring membership now changes on failure, not on every state change:
The trade-off is the black-hole case: an address that accepts TCP but never completes a handshake holds its keys in CONNECTING until the dial times out, once, and then its keys move. The dial timeout is
MinConnectTimeout(20 seconds by default) and the dialer can lower it to bound the window.Tests
Four new unit tests pin the policy: a resolver-provided backend owns keys before READY, IDLE keeps membership and reconnects, CONNECTING keeps membership, and a ReplicationFactor rebuild keeps a CONNECTING member while it excludes a failed one. All four failed before the change.
The #4 end-to-end black-hole test is rewritten, not removed: with
MinConnectTimeoutset to 300ms it shows that an RPC hashed to a black-holed backend fails over promptly after the dial times out.Note
This textually overlaps #6 in
UpdateClientConnStateandUpdateSubConnState. Whichever merges second needs a small rebase.