[He-Pin]

Motivation

Losing the coordinator singleton lease currently sends the same termination message as a normal singleton hand-over. The coordinator drains and permanently stops the cached local shard region even though the node can continue hosting shards. Inferring the termination reason from the cluster read view also races with normal hand-over because that view and the singleton manager process membership updates independently.

Modification

• Add an internal, local construction option that selects separate singleton termination messages for normal hand-over and lease loss. Existing singleton callers retain their original termination message.
• Configure sharding to use its existing Terminate protocol for hand-over and PoisonPill for lease loss, stopping the coordinator supervisor and its child without draining the local region.
• Preserve the existing coordinator drain behavior without a membership-status gate.
• Cover lease loss while reacquisition is pending and hand-over while the membership read view is still Up. Document the behavior for classic and typed sharding.

The construction option is local to the singleton manager; no wire message, serializer, or public API is changed.

Result

A coordinator lease loss preserves the local shard region and its running entities. Normal singleton hand-over still waits for local entities and the region to terminate before completing.

Tests

• The lease-loss regression watches the old singleton and region and verifies a different singleton instance after reacquisition; both lifecycle regression tests passed.
• sbt "cluster-sharding / Test / testOnly org.apache.pekko.cluster.sharding.ClusterShardingCoordinatorLeaseLossSpec": 2 passed.
• sbt "cluster-tools / Test / testOnly org.apache.pekko.cluster.singleton.ClusterSingletonLeaseSpec": 8 passed.
• Directional checks: the lease-loss test fails with the origin/main implementation; the hand-over test fails with the previous membership-status-gated implementation.
• sbt "+cluster-tools/mimaReportBinaryIssues" "+cluster-sharding/mimaReportBinaryIssues": passed on Scala 2.13.18 and 3.3.8.
• sbt headerCreateAll "+cluster-sharding/headerCheckAll" "+cluster-tools/headerCheckAll": passed.
• scalafmt --list --mode diff-ref=origin/main and git diff --check: passed.
• Scoped sbt "project cluster-sharding" headerCreateAll sortImports "+headerCheckAll": passed after the final test changes.
• sbt sortImports: environment failure in unrelated stream-tests and multi-node-testkit tasks, NoSuchMethodError: scala.meta.package$.Input(). Generated unrelated import changes were discarded.
• Shared integration validation with the other reviewed PRs, on JDK 17: sbt headerCreateAll +mimaReportBinaryIssues +headerCheckAll checkCodeStyle docs/paradox — passed. Cross-version checks passed on Scala 2.13.18 and 3.3.8.
• Shared integration sbt -Dpekko.test.tags.exclude=performance,timing,long-running -Dpekko.test.multi-in-test=false validatePullRequest — running with the PR-test exclusions documented in CONTRIBUTING.md.

References

Fixes #3697

[pjfanning]

Thanks, the lease-loss case clearly needs fixing. My main concern is that the leaving/not-leaving decision is inferred from membership state at the time Terminate arrives, which can race with a real hand-over; details inline.

[pjfanning]

cluster.selfMember comes from ClusterReadView, which is updated by its own subscriber actor, separately from ClusterSingletonManager's subscription. In a real leave or self-down, the singleton manager can act on Leaving/Exiting/MemberDowned (or a HandOverToMe from the new oldest) and send Terminate before the read view has applied that status. If CoordinatedShutdown hasn't started yet, selfIsLeavingCluster is false and the coordinator stops without draining the local region. The hand-off optimisation is then silently skipped, and shards on the leaving node can be reallocated while their entities are still alive there.

More generally, the root issue is that the singleton manager sends the same terminationMessage for lease loss and for hand-over, so the coordinator has to guess why from membership. Telling it the actual reason would avoid the race, e.g. a distinct termination message or flag for lease-loss stops from ClusterSingletonManager. That touches cluster-tools, so it could be a follow-up, but it seems worth considering before relying on this check.

[pjfanning]

Removed, PreparingForShutdown and ReadyForShutdown aren't treated as leaving. During a full cluster shutdown, or after self is Removed but before the Cluster extension is terminated (when the system wasn't stopped via CoordinatedShutdown), a Terminate takes the not-leaving path and skips the drain. The coordinator already tracks full-cluster shutdown in its preparingForShutdown flag, which could be included here. (nit: a Set of leaving statuses would read more easily than the == chain.)

[pjfanning]

This covers the lease-loss branch, but nothing checks that the leaving/shutdown branch still drains the local region before the coordinator stops, and that branch now has a new gate in front of it. A regression that made selfIsLeavingCluster always false would pass here. Could we add a directional case, e.g. cluster.leave(selfAddress) (or start CoordinatedShutdown), wait for the status, then send Terminate and assert the local region gets GracefulShutdown before the coordinator stops?

[pjfanning]

nit: this re-sends 2 on every awaitAssert retry, so on slow CI several replies can pile up in testActor. The region buffers until the coordinator is back, so a single region ! 2 followed by expectMsg(20.seconds, 2) should be enough and is deterministic.

[pjfanning]

nit: this applies equally to classic sharding (same ShardCoordinator), but the classic docs/src/main/paradox/cluster-sharding.md doesn't mention it. Worth adding the same note there, or linking to it.