[He-Pin]

Motivation

A Replicator started after another member has left Up can first observe that member in Leaving, Exiting, Down, PreparingForShutdown, or ReadyForShutdown. Building sender recognition from separate quorum and lifecycle sets misses these initial states and drops replication traffic from members that have not been removed.

Modification

• Maintain one private HashSet[UniqueAddress] for incoming Read, Write, and DeltaPropagation membership checks. Update it through one MemberEvent entry point and remove identities on MemberRemoved, leaving quorum and gossip-target sets independent.
• Preserve role filtering, self recognition, and UID boundaries, including a different incarnation at the local address. Retain the existing member-handler methods.
• Intentionally recognize Down members until removal, matching the behavior of members that were already known before becoming Down.
• Give every test its own Replicator and probe. Cover all five late states with independent read/write and delta tests, rejection after removal, ordinary membership states, self, unknown senders, roles, and old incarnations.
• Update both documentation pages to describe membership eligibility and acknowledge that key-specific pruning can reject deltas before MemberRemoved.

Result

The incoming sender check performs one hash-set membership lookup and has one place that records newly observed member events. Recognizing a sender does not expand consistency quorums or gossip targets and does not bypass the delta pruning checks.

Tests

• JDK 17: sbt "distributed-data / Test / testOnly org.apache.pekko.cluster.ddata.ReplicatorKnownNodeSpec" — 18/18 passed.

• Directional check using origin/main's Replicator.scala through temporary Compile / unmanagedSources, with -- -z "first observed" — all 10 regressions failed at the expected missing WriteAck, including the five delta tests.

• Removal mutation check using a temporary source with only knownSenderNodes -= m.uniqueAddress removed, with -- -z "until removed" — all five cases failed because the removed sender still received ReadResult.

• scalafmt --mode diff-ref=origin/main, scalafmt --list --mode diff-ref=origin/main, and git diff --check — passed.

• The Down delta case also passed alone with -- -z "accept deltas from a member first observed in Down".

• sbt -Dpekko.test.tags.exclude=performance,timing,long-running -Dpekko.test.multi-in-test=false "distributed-data / Test / test" — 216 tests passed across 22 suites; one existing cache-wraparound test remains explicitly ignored. Multi-JVM tests were excluded.

• sbt "project distributed-data" sortImports — completed. Unrelated generated import movement and a duplicated existing comment were discarded; the reviewed file tree is unchanged.

• sbt "distributed-data / headerCreateAll" "+distributed-data / headerCheckAll" "distributed-data / scalafmtCheckAll" — passed.

• sbt "+distributed-data / mimaReportBinaryIssues" — passed on Scala 2.13.18 and 3.3.8.

• Earlier shared integration (before this follow-up) sbt -Dpekko.test.tags.exclude=performance,timing,long-running -Dpekko.test.multi-in-test=false validatePullRequest — completed, but did not pass on macOS/arm64. LevelDB JNI lacks a compatible arm64 library (four persistence test forks failed); NettyTransportSpec could not bind the unavailable local address 127.0.1.1; two existing TCP readiness tests timed out. Docker-backed DNS tests were skipped because Docker was unavailable.

• The two TCP failures passed when rerun in isolation against the earlier compiled integration code, without changing assertions or timeouts. The main-production baseline also exhibited a neighboring readiness timeout; those two exact cases did not fail in the repeated baseline runs. The full-run failures are retained in the validation record.

• No further local full validatePullRequest run was performed for this follow-up, as requested; verification is limited to the focused and affected-module checks above.

• Follow-up shared integration: ReplicatorKnownNodeSpec 18/18 and sbt docs/paradox passed with this revision.

References

Fixes #3689

[pjfanning]

The fix looks right for the reported case: senders first seen after Up stay known until MemberRemoved, and role matching is applied consistently on add and remove. A few design points and test gaps inline.

[pjfanning]

This now makes members first seen as Down (via MemberDowned) known senders, so a downed node's Read/Write/DeltaPropagation gets merged until MemberRemoved arrives. In a split-brain scenario, a Replicator that only learns of node X through MemberDowned previously ignored X's writes; now it applies writes from a node the cluster has already decided to remove. Older Replicators already accept these since nodes isn't cleared on Down, so this extends existing behaviour rather than introducing it — but is including Down intentional? It would be good to state that explicitly (or exclude it).

[pjfanning]

The known-sender rule is now spread over six sets (nodes, weaklyUpNodes, joiningNodes, exitingNodes, otherStatusNodes, self). That fragmentation is the root cause of #3689: if a handler forgets to update a set, or a new MemberStatus is added, senders are silently dropped again. Consider a single "members not yet removed" set, populated from every role-matching MemberEvent and cleared on MemberRemoved, used only by isKnownNode.

[pjfanning]

isKnownNode runs on every Read, Write and DeltaPropagation and now does up to five SortedSet (O(log n)) lookups before the self check. otherStatusNodes is only used for membership tests so it doesn't need to be sorted; a single HashSet of known nodes (see the comment above) would make this one O(1) lookup.

[pjfanning]

These cases cover only Read and Write, but the PR description and both docs pages say DeltaPropagation from these members is also accepted. Please add a DeltaPropagation case so a regression in that path is caught.

[pjfanning]

All cases share one lazy replicator and the ImplicitSender testActor, so members added in earlier cases stay known, and a late WriteAck/ReadResult from one case could be consumed by the next. Running a single case with -z or reordering changes the replicator's state (e.g. the role and incarnation cases run with five other members already known). A fresh replicator (and probe) per case would make these independent.

[pjfanning]

The removal test only covers MemberLeft and MemberExited. It would be worth also asserting that Down, PreparingForShutdown and ReadyForShutdown members are rejected after MemberRemoved, since those go through otherStatusNodes; otherwise a regression in clearing that set would leave downed members known forever with no failing test.

[pjfanning]

"accepted until those members are removed" overstates it for deltas: receiveDeltaPropagation also drops deltas when isNodeRemoved is true, which includes keys whose pruning state already contains the sender, even before MemberRemoved. Maybe soften the wording. The same text is duplicated in typed/distributed-data.md, so both copies need the change.