Skip to content

fix: keep IDLE and CONNECTING SubConns on the hashring - #8

Merged
tstirrat15 merged 1 commit into
mainfrom
stable-ring-membership
Sep 25, 2026
Merged

tstirrat15 merged 1 commit into
mainfrom
stable-ring-membership

Conversation

@ecordell

Copy link
Copy Markdown
Contributor

Problem

Since #4, ring membership follows READY state exactly. That policy causes constant churn on healthy clusters. Servers that limit connection age send GOAWAY on an interval, the client SubConn moves READY to IDLE, and the member leaves the ring until it reconnects. Every exit remaps the member's keys to its neighbors, and every re-entry maps them back. Callers that depend on stable key placement, such as caches keyed by ring owner, lose locality on each cycle.

There is history here: SpiceDB hit this class of churn and fixed it with state-insensitive ring membership in authzed/spicedb#1310 ("the previous implementation was recomputing the hashring any time a subconnection moved from ready->idle or back, which happened frequently"). SpiceDB's dispatch servers send GOAWAY every 30 seconds by default (dispatch-cluster-max-conn-age), so under READY-gated membership every node's view of every peer leaves and rejoins the ring on that interval, forever, in a healthy cluster.

The problem #4 fixed is real: before it, a key hashed to a dead-but-still-resolved backend hung until the resolver caught up. This PR keeps that fix for the common case and removes the churn.

Change

Ring membership now changes on failure, not on every state change:

The trade-off is the black-hole case: an address that accepts TCP but never completes a handshake holds its keys in CONNECTING until the dial times out, once, and then its keys move. The dial timeout is MinConnectTimeout (20 seconds by default) and the dialer can lower it to bound the window.

Tests

Four new unit tests pin the policy: a resolver-provided backend owns keys before READY, IDLE keeps membership and reconnects, CONNECTING keeps membership, and a ReplicationFactor rebuild keeps a CONNECTING member while it excludes a failed one. All four failed before the change.

The #4 end-to-end black-hole test is rewritten, not removed: with MinConnectTimeout set to 300ms it shows that an RPC hashed to a black-holed backend fails over promptly after the dial times out.

Note

This textually overlaps #6 in UpdateClientConnState and UpdateSubConnState. Whichever merges second needs a small rebase.

Ring membership followed READY state exactly (#4). That policy causes
constant churn on healthy clusters. Servers that limit connection age
send GOAWAY on an interval, the client SubConn moves READY to IDLE,
and the member leaves the ring until it reconnects. Every exit remaps
the member's keys to its neighbors, and every re-entry maps them back.
Callers that depend on stable key placement, such as caches keyed by
ring owner, lose locality on each cycle. SpiceDB hit this class of
churn before and fixed it with state-insensitive ring membership
(authzed/spicedb#1310). Its dispatch servers send GOAWAY every 30
seconds by default.

Ring membership now changes on failure, not on every state change:

- A SubConn joins the ring when the resolver provides it.
- IDLE and CONNECTING SubConns keep their membership. These states
  normally resolve in milliseconds, and the balancer reconnects IDLE
  SubConns at once.
- TRANSIENT_FAILURE and Shutdown remove the member. Keys move to the
  next closest backend. This keeps the fast failover from #4 for
  backends that refuse connections.

The trade-off is the black-hole case: an address that accepts no
connection holds its keys in CONNECTING until the dial times out,
once, and then its keys move. The dial timeout is MinConnectTimeout
(20 seconds by default) and the dialer can lower it to bound the
window. The e2e test sets it to 300ms and shows prompt failover.

@tstirrat15 tstirrat15 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense to me. Let's get it in and test it.

@tstirrat15
tstirrat15 merged commit d4e3d0e into main Sep 25, 2026
6 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 25, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants