Document base-copy liveness as available from v5.3.6, with refused from v5.4.0

learn/administration/operating-replication.mdx

@@ -102,7 +102,7 @@ There is one socket per database per peer, so the sockets present in the respons
 | Node registry (`hdb_nodes`) | Lives in `system`, so it propagates with it. Each node rewrites its own self-record from its `harper-config.yaml` routes on restart or component reload | Scoping applied through `add_node` alone does not survive a restart. Put durable constraints in config routes, and see below if you need the registry kept per cluster |
 | Destructive schema changes  | `drop_database` and `drop_table` replicate by default; `drop_attribute` does not                                                                        | A drop is a cluster-wide event unless you pass `"replicated": false`. Verify the result on every node                                                                  |
 
-Measure incremental catch-up and full synchronization separately. Peers retain transaction history for [`logging.auditRetention`](/reference/v5/logging/configuration#loggingauditretention) (default `3d`), but purge it sooner under disk pressure. When a peer has purged history that the saved position would replay, it logs `forcing a bounded base-copy resync` and sends its current rows as writes. <VersionBadge type="changed" version="v5.4.0" /> A database that the peer has not written to since then no longer forces a copy; see [when a peer forces a base copy](/reference/v5/replication/overview#when-a-peer-forces-a-base-copy) for the cases that still do. Incremental replay resumes from the copy's start time. The copy never clears tables or deletes rows: purged tombstones are absent, and retained tombstones arrive as writes without values that the receiver skips. It also runs in reverse. The returning node sends its rows to peers, including writes only it took. A copied row wins where it is newer; where a peer has purged a deletion's tombstone, the deleted row can reappear.
+Measure incremental catch-up and full synchronization separately. Peers retain transaction history for [`logging.auditRetention`](/reference/v5/logging/configuration#loggingauditretention) (default `3d`), but purge it sooner under disk pressure. When a peer has purged history that the saved position would replay, it logs `forcing a bounded base-copy resync` and sends its current rows as writes. <VersionBadge type="changed" version="v5.3.6" /> A database that the peer has not written to since then no longer forces a copy; see [when a peer forces a base copy](/reference/v5/replication/overview#when-a-peer-forces-a-base-copy) for the cases that still do. Incremental replay resumes from the copy's start time. The copy never clears tables or deletes rows: purged tombstones are absent, and retained tombstones arrive as writes without values that the receiver skips. It also runs in reverse. The returning node sends its rows to peers, including writes only it took. A copied row wins where it is newer; where a peer has purged a deletion's tombstone, the deleted row can reappear.
 
 :::danger
 **A long-absent node must not reconnect with its existing data.** If it was disconnected longer than its peers retain history, or you cannot show otherwise, keep it out of rotation and isolate replication before Harper starts. Then rebuild whichever side does not hold the authoritative data:
@@ -166,7 +166,7 @@ What each field is telling you:
 - **`lastCommitConfirmed`** is the last time this peer acknowledged receiving one of your commits. If it stops advancing while you are still writing, your writes are not landing on that peer.
 - **`lastReceivedRemoteTime`** is the source node's timestamp on the newest transaction you have received.
 - **`lastReceivedLocalTime`** is your own clock when you received it.
-- **`stuckCopy`** <VersionBadge version="v5.4.0" /> appears only while a base copy of that database from that peer is not finishing: `retrying`, `held` (copies of this database from that peer are on the hold schedule) or `refused` (this node's sharding rules refused the copy). When it disappears, that does not prove the copy finished or that the data matches, so verify as described below. Its `since` and `retryAt` are ISO 8601 strings, not GMT strings like the fields above. See [Stuck Copies](/reference/v5/replication/clustering#stuck-copies) for its fields and states.
+- **`stuckCopy`** <VersionBadge version="v5.3.6" /> appears only while a base copy of that database from that peer is not finishing: `retrying`, `held` (copies of this database from that peer are on the hold schedule) or, on v5.4.0 and later, `refused` (this node's sharding rules refused the copy). When it disappears, that does not prove the copy finished or that the data matches, so verify as described below. Its `since` and `retryAt` are ISO 8601 strings, not GMT strings like the fields above. See [Stuck Copies](/reference/v5/replication/clustering#stuck-copies) for its fields and states.
 
 Check connection state and sustained latency alongside commit progress during writes. Compare `lastReceivedRemoteTime` with the source's expected write timeline, then use a sentinel or journey test to verify freshness. `sendingMessage` appears only while a transaction is being sent; its absence on an idle socket is normal.
 

reference/replication/clustering.md

@@ -202,10 +202,12 @@ While a database is taking its initial full copy, these timing fields hold the l
 
 ### Stuck Copies
 
-<VersionBadge version="v5.4.0" />
+<VersionBadge version="v5.3.6" />
 
 A `database_sockets` entry has a `stuckCopy` object while a [base copy](./overview.md#initial-synchronization-and-resynchronization) of that database from that peer is not finishing. Otherwise the field is absent. The node receiving the copy reports it, on its socket to the source. Copies also run in reverse, so that can be a peer of the node that came back rather than the node itself. Run `cluster_status` on every node.
 
+On the 5.3 line, `state` is only `retrying` or `held`. The `refused` state and everything below about refusals need [schema-defined sharding](./residency.md), which starts in v5.4.0.
+
 ```json
 {
 	"database": "data",
@@ -222,7 +224,7 @@ A `database_sockets` entry has a `stuckCopy` object while a [base copy](./overvi
 
 | Field     | Description                                                                                                                                                  |
 | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| `state`   | `retrying`, `held` or `refused` (below)                                                                                                                      |
+| `state`   | `retrying`, `held` or, from v5.4.0, `refused` (below)                                                                                                        |
 | `reason`  | `the copy made no durable progress from its start position` for `retrying` and `held`. For `refused`, the sharding rule that refused the copy                |
 | `count`   | For `retrying` and `held`, how many times a copy has started from the same position. For `refused`, how many times in a row the copy was refused             |
 | `since`   | When the socket entered its current state. Another attempt or hold in the same state does not change it                                                      |
@@ -234,7 +236,7 @@ The states:
 
 - **`retrying`**: a copy from this peer has started at least twice from the same position, with no durable progress in between. Reconnects follow the normal schedule.
 - **`held`**: the session that made the fifth start from one position has closed. This node stops reconnecting that database to that peer for a while. Nothing in that database replicates from the peer during the hold, and the peer's copy stops too. The first hold lasts 60 seconds. After that, each reconnect makes one attempt. If it fails from the same position, the next hold is a random time between 60 seconds and a limit that doubles each time (2, 4, 8 minutes), up to 15 minutes.
-- **`refused`**: this node refused the copy under its [sharding](./residency.md#recovery-sources) rules. With the reason `Owned sharding slots require a complete co-owner or backup restore`, this node owns slots and the peer cannot supply them. Typically the peer is not a complete co-owner, so make one reachable or restore a complete backup. Reconnects are held on the same schedule as `held`. With the reason `a sharded origin requires its origin or a complete co-owner`, the copy carried rows from a sharded origin that only that origin or a complete co-owner may send. Reconnects follow the normal schedule, because this usually follows a sharding map change.
+- **`refused`** <VersionBadge version="v5.4.0" />: this node refused the copy under its [sharding](./residency.md#recovery-sources) rules. With the reason `Owned sharding slots require a complete co-owner or backup restore`, this node owns slots and the peer cannot supply them. Typically the peer is not a complete co-owner, so make one reachable or restore a complete backup. Reconnects are held on the same schedule as `held`. With the reason `a sharded origin requires its origin or a complete co-owner`, the copy carried rows from a sharded origin that only that origin or a complete co-owner may send. Reconnects follow the normal schedule, because this usually follows a sharding map change.
 
 The attempt limit, the hold bounds and the 60-second quiet period below are fixed. No setting changes them.
 

reference/replication/overview.md

@@ -378,26 +378,26 @@ You can also specify a `start_time` in the `add_node` operation to limit the ini
 
 ### When a Peer Forces a Base Copy
 
-<VersionBadge type="changed" version="v5.4.0" />
+<VersionBadge type="changed" version="v5.3.6" />
 
-The peer that would send the catch-up decides, per database, when the returning node subscribes. It logs `forcing a bounded base-copy resync` when it forces a copy. Before v5.4, a peer forced one whenever the returning node's position was older than `logging.auditRetention` or than the peer's oldest retained entry. A peer that had written nothing to a database since then therefore forced a full copy even though nothing was missed.
+The peer that would send the catch-up decides, per database, when the returning node subscribes. It logs `forcing a bounded base-copy resync` when it forces a copy. Before v5.3.6, a peer forced one whenever the returning node's position was older than `logging.auditRetention` or than the peer's oldest retained entry. A peer that had written nothing to a database since then therefore forced a full copy even though nothing was missed.
 
-From v5.4, a peer records how far each prune of its transaction logs reached. It forces a copy when the returning node's position, for a log it would replay, is below that point. A position that is old only because the peer wrote nothing to that database since, such as an idle or never-written transaction log, no longer forces a copy. A copy can still be forced although nothing was missed, in these cases:
+From v5.3.6, a peer records how far each prune of its transaction logs reached. It forces a copy when the returning node's position, for a log it would replay, is below that point. A position that is old only because the peer wrote nothing to that database since, such as an idle or never-written transaction log, no longer forces a copy. A copy can still be forced although nothing was missed, in these cases:
 
-- The peer has no trustworthy record of what it pruned. It then uses the pre-v5.4 rule.
+- The peer has no trustworthy record of what it pruned. It then uses the pre-v5.3.6 rule.
 - After [`delete_transaction_logs_before`](../database/transaction.md#delete_transaction_logs_before), a node whose position is below the point that purge reached (at most the timestamp it was given) gets a base copy, even if the peer wrote nothing to that database after that position.
-- The peer has no record of what was removed from a transaction log that was already purged empty before it ran v5.4. After the upgrade, such a log can force one more base copy of that database for each node returning with a position older than that purge.
+- The peer has no record of what was removed from a transaction log that was already purged empty before it first ran v5.3.6 or later. After that upgrade, such a log can force one more base copy of that database for each node returning with a position older than that purge.
 - On RocksDB, the storage engine also purges transaction logs on its own after three days. If the peer itself was stopped for longer than that, a node whose position is older than three days gets one base copy of the database from it.
 
 A copy that keeps failing is reported as [`stuckCopy`](./clustering.md#stuck-copies) in `cluster_status`, and its reconnects are held back instead of retried indefinitely.
 
 ### Rows From Unnamed Origins
 
-<VersionBadge version="v5.4.0" />
+<VersionBadge version="v5.3.6" />
 
-Each replicated row records its origin, the node that wrote it, as a numeric id that each node maps to a node name. A node can hold rows whose origin id has no name on that node, for example data migrated from v4, or rows from a node whose name mapping no longer exists. Before v5.4, a peer receiving such a row could not translate its origin, so a base copy that included it failed and restarted from the same position indefinitely.
+Each replicated row records its origin, the node that wrote it, as a numeric id that each node maps to a node name. A node can hold rows whose origin id has no name on that node, for example data migrated from v4, or rows from a node whose name mapping no longer exists. Before v5.3.6, a peer receiving such a row could not translate its origin, so a base copy that included it failed and restarted from the same position indefinitely.
 
-From v5.4, the sending node names that origin `unnamed-origin-<id>@<node name>`, where `<id>` is its origin id on the sending node and `<node name>` is the sending node's own name. Receivers treat it like any other origin. A v5.4 node receiving from an older node derives the same name itself and logs a warning: `<node name> sent <database> records from origin id <id>, which has no name there; applying them as origin unnamed-origin-<id>@<node name>`. It derives at most 64 new names per connection; past that, the connection closes and reconnects.
+From v5.3.6, the sending node names that origin `unnamed-origin-<id>@<node name>`, where `<id>` is its origin id on the sending node and `<node name>` is the sending node's own name. Receivers treat it like any other origin. A node on v5.3.6 or later receiving from an older node derives the same name itself and logs a warning: `<node name> sent <database> records from origin id <id>, which has no name there; applying them as origin unnamed-origin-<id>@<node name>`. It derives at most 64 new names per connection; past that, the connection closes and reconnects.
 
 In `<node name>`, each character other than ASCII letters, digits, `.` and `-` becomes `_` followed by its four-digit lowercase hexadecimal code, so `:` becomes `_003a` and `_` becomes `_005f`. These names identify where rows came from; no node by that name exists. They can appear in log lines on any node that receives or relays those rows.
 

release-notes/v5-lincoln/5.4.md

@@ -41,13 +41,13 @@ Snapshot-only recovery of a sharded node requires a fresh root and a new node na
 
 ### Base Copies That Could Not Finish
 
-[Stop forcing base copies that never finish](https://github.com/HarperFast/harper-pro/pull/1249) fixes three ways a base copy could start without need or restart forever:
+[Stop forcing base copies that never finish](https://github.com/HarperFast/harper-pro/pull/1249) fixes three ways a base copy could start without need or restart forever. The same fixes are also in 5.3.6 ([harper-pro#1262](https://github.com/HarperFast/harper-pro/pull/1262)), except refusal as a sharded owner: `stuckCopy` there has only `retrying` and `held`.
 
 - **Idle databases no longer force copies.** A peer used to force a base copy whenever a returning node's position was older than `logging.auditRetention` or its own oldest retained entry, even for a database it had not written to since. It now forces one only when a prune may have removed history the node would replay. See [When a Peer Forces a Base Copy](/reference/v5/replication/overview#when-a-peer-forces-a-base-copy).
 - **Rows with no origin name now copy.** Rows whose origin has no node name on the sending node, such as data migrated from v4, made the receiver fail and restart the copy. The sender now names that origin `unnamed-origin-<id>@<node name>`, and those names can appear in logs. See [Rows From Unnamed Origins](/reference/v5/replication/overview#rows-from-unnamed-origins).
 - **Stuck copies are held and reported.** When a copy starts five times from the same position, or the receiving node refuses it as a sharded owner, that node now holds its reconnects to the peer for that database: 60 seconds at first, then longer each time, up to 15 minutes. Before, it retried every few seconds. `cluster_status` reports the state as `stuckCopy` on the database socket, and each change of state logs one line. See [Stuck Copies](/reference/v5/replication/clustering#stuck-copies).
 
-**Upgrade impact:** after the upgrade, a transaction log that was already purged empty can force one more base copy of its database for each node returning with a position older than that purge.
+**Upgrade impact:** after upgrading from a version before 5.3.6, a transaction log that was already purged empty can force one more base copy of its database for each node returning with a position older than that purge.
 
 ### LMDB Replication Confirmation