kriszyp · Nov 3, 2026 · Nov 3, 2026 · Nov 3, 2026 · Nov 3, 2026 · Nov 3, 2026
@@ -168,7 +168,7 @@ Also include these constraints in the restore procedure:
 
 ## Restoring in a replicated cluster
 
-A restored node may be behind peers that have continued accepting writes. Peers can replay newer writes, including the destructive mutations you intended to undo. If its restored data and saved replication position predate the peers' retained history, copies run in both directions and can bring deleted rows back onto peers. Follow the [long-absent node policy](./operating-replication.mdx#what-moves-automatically-and-what-does-not) before reconnecting. Restoring one node does not make its backup authoritative; choose which data to preserve or repair and rehearse the re-entry policy before reconnecting it.
+A restored node may be behind peers that have continued accepting writes. Peers can replay newer writes, including the destructive mutations you intended to undo. If peers have purged history that its saved replication position would replay, [copies run](/reference/v5/replication/overview#when-a-peer-forces-a-base-copy) in both directions and can bring deleted rows back onto peers. Follow the [long-absent node policy](./operating-replication.mdx#what-moves-automatically-and-what-does-not) before reconnecting. Restoring one node does not make its backup authoritative; choose which data to preserve or repair and rehearse the re-entry policy before reconnecting it.
 
 ```mermaid
 flowchart TD

@@ -325,7 +325,7 @@ Record the third number as node re-entry time, including convergence and applica
 
 ## Operational notes
 
-- **A returning node is not the same as a new node.** A node whose databases have never synchronized downloads them in full. A briefly offline node catches up from retained history; a long-absent node needs the rebuild or restore policy above. Both need to finish before traffic arrives, but they take very different amounts of time, so do not budget for the first when you are planning a routine restart.
+- **A returning node is not the same as a new node.** A node whose databases have never synchronized downloads them in full. A briefly offline node usually catches up from retained history ([when it gets a full copy instead](/reference/v5/replication/overview#when-a-peer-forces-a-base-copy)); a long-absent node needs the rebuild or restore policy above. Both need to finish before traffic arrives, but they take very different amounts of time, so do not budget for the first when you are planning a routine restart.
 - **Set unavailable before recovery work, not after.** Any operation that touches data on a node, including a restore, should happen with the node out of rotation.
 - **Configuration changes need a restart to take effect**, so a node that has been reconfigured but not restarted is running the old configuration while reporting the new one. Sequence the restart into the same maintenance window.
 - **Fabric routes by its status component.** It does not automatically consume your readiness route. For application-driven availability, see [Taking an Unhealthy Node Out of Rotation](../developers/taking-a-node-out-of-rotation.mdx).

@@ -95,14 +95,14 @@ There is one socket per database per peer, so the sockets present in the respons
 | Data mutations              | Insert, update, upsert, delete, and bulk load replicate for tables in scope                                                                             | Monitor connections, latency, and convergence for the databases that serve critical journeys                                                                           |
 | Transactions                | Replicated atomically, and a transaction may span multiple tables                                                                                       | Tables that need to commit together must live in the same database                                                                                                     |
 | A brand new node            | Requests a full copy of each in-scope database from every peer it has no resume cursor for, then enters incremental replication                         | Budget N transfers, not one. See the note below before sizing a join                                                                                                   |
-| A returning node            | Resynchronizes automatically to catch up on the transactions it missed                                                                                  | Full copy when peers no longer hold the needed history (`logging.auditRetention`, sooner under disk pressure). See below                                               |
+| A returning node            | Resynchronizes automatically to catch up on the transactions it missed                                                                                  | Full copy when peers no longer hold the needed history ([when](/reference/v5/replication/overview#when-a-peer-forces-a-base-copy)). See below                          |
 | Component deployment        | `deploy_component` replicates by default; pass `"replicated": false` to hold it to one node                                                             | Cluster-wide is the default, so single-node validation is the case you must ask for                                                                                    |
 | Configuration               | `set_configuration` supports `"replicated": true` <VersionBadge version="v5.2.0" />                                                                     | Never replicate node-local parameters. See the warning below                                                                                                           |
 | Users and roles             | Propagate when the `system` database is in replication scope, which is the default                                                                      | If you narrow scope to exclude `system`, identity must be provisioned on every node by your own automation                                                             |
 | Node registry (`hdb_nodes`) | Lives in `system`, so it propagates with it. Each node rewrites its own self-record from its `harper-config.yaml` routes on restart or component reload | Scoping applied through `add_node` alone does not survive a restart. Put durable constraints in config routes, and see below if you need the registry kept per cluster |
 | Destructive schema changes  | `drop_database` and `drop_table` replicate by default; `drop_attribute` does not                                                                        | A drop is a cluster-wide event unless you pass `"replicated": false`. Verify the result on every node                                                                  |
 
-Measure incremental catch-up and full synchronization separately. Peers retain transaction history for [`logging.auditRetention`](/reference/v5/logging/configuration#loggingauditretention) (default `3d`), but purge it sooner under disk pressure. When a saved position predates a peer's oldest retained history, the peer logs `forcing a bounded base-copy resync` and sends its current rows as writes. Incremental replay resumes from the copy's start time. The copy never clears tables or deletes rows: purged tombstones are absent, and retained tombstones arrive as writes without values that the receiver skips. It also runs in reverse. The returning node sends its rows to peers, including writes only it took. A copied row wins where it is newer; where a peer has purged a deletion's tombstone, the deleted row can reappear.
+Measure incremental catch-up and full synchronization separately. Peers retain transaction history for [`logging.auditRetention`](/reference/v5/logging/configuration#loggingauditretention) (default `3d`), but purge it sooner under disk pressure. When a peer has purged history that the saved position would replay, it logs `forcing a bounded base-copy resync` and sends its current rows as writes. <VersionBadge type="changed" version="v5.4.0" /> A database that the peer has not written to since then no longer forces a copy; see [when a peer forces a base copy](/reference/v5/replication/overview#when-a-peer-forces-a-base-copy) for the cases that still do. Incremental replay resumes from the copy's start time. The copy never clears tables or deletes rows: purged tombstones are absent, and retained tombstones arrive as writes without values that the receiver skips. It also runs in reverse. The returning node sends its rows to peers, including writes only it took. A copied row wins where it is newer; where a peer has purged a deletion's tombstone, the deleted row can reappear.
 
 :::danger
 **A long-absent node must not reconnect with its existing data.** If it was disconnected longer than its peers retain history, or you cannot show otherwise, keep it out of rotation and isolate replication before Harper starts. Then rebuild whichever side does not hold the authoritative data:
@@ -166,6 +166,7 @@ What each field is telling you:
 - **`lastCommitConfirmed`** is the last time this peer acknowledged receiving one of your commits. If it stops advancing while you are still writing, your writes are not landing on that peer.
 - **`lastReceivedRemoteTime`** is the source node's timestamp on the newest transaction you have received.
 - **`lastReceivedLocalTime`** is your own clock when you received it.
+- **`stuckCopy`** <VersionBadge version="v5.4.0" /> appears only while a base copy of that database from that peer is not finishing: `retrying`, `held` (copies of this database from that peer are on the hold schedule) or `refused` (this node's sharding rules refused the copy). When it disappears, that does not prove the copy finished or that the data matches, so verify as described below. Its `since` and `retryAt` are ISO 8601 strings, not GMT strings like the fields above. See [Stuck Copies](/reference/v5/replication/clustering#stuck-copies) for its fields and states.
 
 Check connection state and sustained latency alongside commit progress during writes. Compare `lastReceivedRemoteTime` with the source's expected write timeline, then use a sentinel or journey test to verify freshness. `sendingMessage` appears only while a transaction is being sent; its absence on an idle socket is normal.
 

@@ -136,7 +136,7 @@ Deletes transaction log entries older than the specified timestamp. Deprecated i
 Deletes transaction log entries older than the specified timestamp.
 
 :::warning Database-wide and irreversible
-On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This purges whole log files whose entries predate the timestamp, removing [`read_audit_log`](#read_audit_log) history for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender detects that the requested start predates its retained history and forces a full base copy instead of incremental catch-up — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. The same history can be required to bridge a [full-text index rebuild](../full-text-search/overview.md#startup-and-recovery) from its table scan to current writes. Cleanup that removes the bridge during a rebuild can fail that attempt; stop aggressive cleanup before retrying so the next attempt can scan from a fresh committed-log boundary. Size automatic retention for the workload.
+On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This purges whole log files whose entries predate the timestamp, removing [`read_audit_log`](#read_audit_log) history for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica: the sender forces a full base copy instead of incremental catch-up. That happens for any replica whose position is below the point the purge reached (at most the purge timestamp), even if the database had no writes after that position (see [when a peer forces a base copy](../replication/overview.md#when-a-peer-forces-a-base-copy)). A full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. The same history can be required to bridge a [full-text index rebuild](../full-text-search/overview.md#startup-and-recovery) from its table scan to current writes. Cleanup that removes the bridge during a rebuild can fail that attempt; stop aggressive cleanup before retrying so the next attempt can scan from a fresh committed-log boundary. Size automatic retention for the workload.
 :::
 
 Parameters:

@@ -1822,7 +1822,7 @@ POST; requires `super_user`. Takes `database`, `table` and a proposed key-derive
 
 ### `cluster_status`
 
-Returns connection state for all cluster nodes, including per-database socket status and replication timing statistics (`lastCommitConfirmed`, `lastReceivedRemoteTime`, `lastReceivedLocalTime`). Its top-level `sharding` object reports per-database map identity and placement blockers; see [Sharding Status](../replication/clustering.md#sharding-status).
+Returns connection state for all cluster nodes, including per-database socket status and replication timing statistics (`lastCommitConfirmed`, `lastReceivedRemoteTime`, `lastReceivedLocalTime`). A socket whose base copy is not finishing also carries `stuckCopy`; see [Stuck Copies](../replication/clustering.md#stuck-copies). Its top-level `sharding` object reports per-database map identity and placement blockers; see [Sharding Status](../replication/clustering.md#sharding-status).
 
 ```json
 { "operation": "cluster_status" }

@@ -200,6 +200,54 @@ Returns an array of status objects from the cluster, including active WebSocket
 
 While a database is taking its initial full copy, these timing fields hold the literal string `"Copying"` rather than a timestamp. Anything that parses them as dates needs to handle that value, since a node taking a full copy is exactly the node you are most likely to be watching.
 
+### Stuck Copies
+
+<VersionBadge version="v5.4.0" />
+
+A `database_sockets` entry has a `stuckCopy` object while a [base copy](./overview.md#initial-synchronization-and-resynchronization) of that database from that peer is not finishing. Otherwise the field is absent. The node receiving the copy reports it, on its socket to the source. Copies also run in reverse, so that can be a peer of the node that came back rather than the node itself. Run `cluster_status` on every node.
+
+```json
+{
+	"database": "data",
+	"connected": false,
+	"stuckCopy": {
+		"state": "held",
+		"reason": "the copy made no durable progress from its start position",
+		"count": 5,
+		"since": "2026-11-03T14:20:11.204Z",
+		"retryAt": "2026-11-03T14:21:11.204Z"
+	}
+}
+```
+
+| Field     | Description                                                                                                                                                  |
+| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `state`   | `retrying`, `held` or `refused` (below)                                                                                                                      |
+| `reason`  | `the copy made no durable progress from its start position` for `retrying` and `held`. For `refused`, the sharding rule that refused the copy                |
+| `count`   | For `retrying` and `held`, how many times a copy has started from the same position. For `refused`, how many times in a row the copy was refused             |
+| `since`   | When the socket entered its current state. Another attempt or hold in the same state does not change it                                                      |
+| `retryAt` | When the socket reconnects after a hold. Present only while it waits out the hold, so it is absent once the reconnect opens, even if `state` is still `held` |
+
+`since` and `retryAt` are ISO 8601 strings, unlike the GMT strings in the timing fields above.
+
+The states:
+
+- **`retrying`**: a copy from this peer has started at least twice from the same position, with no durable progress in between. Reconnects follow the normal schedule.
+- **`held`**: the session that made the fifth start from one position has closed. This node stops reconnecting that database to that peer for a while. Nothing in that database replicates from the peer during the hold, and the peer's copy stops too. The first hold lasts 60 seconds. After that, each reconnect makes one attempt. If it fails from the same position, the next hold is a random time between 60 seconds and a limit that doubles each time (2, 4, 8 minutes), up to 15 minutes.
+- **`refused`**: this node refused the copy under its [sharding](./residency.md#recovery-sources) rules. With the reason `Owned sharding slots require a complete co-owner or backup restore`, this node owns slots and the peer cannot supply them. Typically the peer is not a complete co-owner, so make one reachable or restore a complete backup. Reconnects are held on the same schedule as `held`. With the reason `a sharded origin requires its origin or a complete co-owner`, the copy carried rows from a sharded origin that only that origin or a complete co-owner may send. Reconnects follow the normal schedule, because this usually follows a sharding map change.
+
+The attempt limit, the hold bounds and the 60-second quiet period below are fixed. No setting changes them.
+
+`stuckCopy` disappears in these cases:
+
+- The copy makes durable progress, continues from a different position, or finishes. This also starts the hold schedule over.
+- A later connection to the peer stays open for 60 seconds without starting or refusing a copy. This clears only the report. The start count and the hold limit stay, so if a copy later fails from the same position again, the socket shows `retrying` with the count carried on, and the next hold follows the schedule from where it left off.
+- A `refused` socket settles when the session of a later, accepted copy closes. It clears, or becomes `retrying` if that copy also started twice from one position.
+
+A missing `stuckCopy` does not show that a copy finished or that the two nodes hold the same data.
+
+Each change of state, or of the refusal reason, logs one line beginning `Replication of <database> from <node>:`. The level is `error` when the socket is held, `warn` when it starts retrying or is refused, and `info` when the report clears, such as `copy is progressing again`. A held socket logs, for example, `copy stopped after 5 attempts from one position; retrying in 60s: the copy made no durable progress from its start position`. Further attempts or holds in the same state log nothing more.
+
 ---
 
 ### Sharding Status