From e9420d7ca931fbb2ba815f9d36d85b6fa0cdba32 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 3 Nov 2026 07:38:49 -0700 Subject: [PATCH 1/6] Document stuck base copies, purge-bound retention copies and unnamed origins for 5.4 Companion to HarperFast/harper-pro#1249: the cluster_status stuckCopy field (retrying, held, refused), the per-log purge-bound rule that stops idle databases from forcing base copies, and unnamed-origin-@ naming. Co-Authored-By: Claude Opus 5.5 Dispatch-Task: docs-harper-pro-1249-base-copy-liveness --- .../administration/operating-replication.mdx | 3 +- reference/database/transaction.md | 2 +- reference/operations-api/operations.md | 2 +- reference/replication/clustering.md | 48 +++++++++++++++++++ reference/replication/overview.md | 26 +++++++++- release-notes/v5-lincoln/5.4.md | 10 ++++ 6 files changed, 87 insertions(+), 4 deletions(-) diff --git a/learn/administration/operating-replication.mdx b/learn/administration/operating-replication.mdx index c36bf3310..77c4c6688 100644 --- a/learn/administration/operating-replication.mdx +++ b/learn/administration/operating-replication.mdx @@ -102,7 +102,7 @@ There is one socket per database per peer, so the sockets present in the respons | Node registry (`hdb_nodes`) | Lives in `system`, so it propagates with it. Each node rewrites its own self-record from its `harper-config.yaml` routes on restart or component reload | Scoping applied through `add_node` alone does not survive a restart. Put durable constraints in config routes, and see below if you need the registry kept per cluster | | Destructive schema changes | `drop_database` and `drop_table` replicate by default; `drop_attribute` does not | A drop is a cluster-wide event unless you pass `"replicated": false`. Verify the result on every node | -Measure incremental catch-up and full synchronization separately. Peers retain transaction history for [`logging.auditRetention`](/reference/v5/logging/configuration#loggingauditretention) (default `3d`), but purge it sooner under disk pressure. When a saved position predates a peer's oldest retained history, the peer logs `forcing a bounded base-copy resync` and sends its current rows as writes. Incremental replay resumes from the copy's start time. The copy never clears tables or deletes rows: purged tombstones are absent, and retained tombstones arrive as writes without values that the receiver skips. It also runs in reverse. The returning node sends its rows to peers, including writes only it took. A copied row wins where it is newer; where a peer has purged a deletion's tombstone, the deleted row can reappear. +Measure incremental catch-up and full synchronization separately. Peers retain transaction history for [`logging.auditRetention`](/reference/v5/logging/configuration#loggingauditretention) (default `3d`), but purge it sooner under disk pressure. When a peer has purged history the saved position would replay, it logs `forcing a bounded base-copy resync` and sends its current rows as writes. A database the peer has not written to since no longer forces a copy; see [when a peer forces a base copy](/reference/v5/replication/overview#when-a-peer-forces-a-base-copy) for the cases that still do. Incremental replay resumes from the copy's start time. The copy never clears tables or deletes rows: purged tombstones are absent, and retained tombstones arrive as writes without values that the receiver skips. It also runs in reverse. The returning node sends its rows to peers, including writes only it took. A copied row wins where it is newer; where a peer has purged a deletion's tombstone, the deleted row can reappear. :::danger **A long-absent node must not reconnect with its existing data.** If it was disconnected longer than its peers retain history, or you cannot show otherwise, keep it out of rotation and isolate replication before Harper starts. Then rebuild whichever side does not hold the authoritative data: @@ -166,6 +166,7 @@ What each field is telling you: - **`lastCommitConfirmed`** is the last time this peer acknowledged receiving one of your commits. If it stops advancing while you are still writing, your writes are not landing on that peer. - **`lastReceivedRemoteTime`** is the source node's timestamp on the newest transaction you have received. - **`lastReceivedLocalTime`** is your own clock when you received it. +- **`stuckCopy`** appears only while a base copy of that database from that peer is not finishing: `retrying`, `held` (this node has stopped reconnecting to that peer for a while) or `refused` (this node's sharding ownership refused the copy). When it disappears, that does not prove the copy finished or that the data matches, so verify as described below. See [Stuck Copies](/reference/v5/replication/clustering#stuck-copies) for its fields and states. Check connection state and sustained latency alongside commit progress during writes. Compare `lastReceivedRemoteTime` with the source's expected write timeline, then use a sentinel or journey test to verify freshness. `sendingMessage` appears only while a transaction is being sent; its absence on an idle socket is normal. diff --git a/reference/database/transaction.md b/reference/database/transaction.md index 8d934d229..04bb52d0f 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -136,7 +136,7 @@ Deletes transaction log entries older than the specified timestamp. Deprecated i Deletes transaction log entries older than the specified timestamp. :::warning Database-wide and irreversible -On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This purges whole log files whose entries predate the timestamp, removing [`read_audit_log`](#read_audit_log) history for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender detects that the requested start predates its retained history and forces a full base copy instead of incremental catch-up — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. The same history can be required to bridge a [full-text index rebuild](../full-text-search/overview.md#startup-and-recovery) from its table scan to current writes. Cleanup that removes the bridge during a rebuild can fail that attempt; stop aggressive cleanup before retrying so the next attempt can scan from a fresh committed-log boundary. Size automatic retention for the workload. +On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This purges whole log files whose entries predate the timestamp, removing [`read_audit_log`](#read_audit_log) history for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender detects that the purge removed history the replica would replay and forces a full base copy instead of incremental catch-up (see [when a peer forces a base copy](../replication/overview.md#when-a-peer-forces-a-base-copy)) — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. The same history can be required to bridge a [full-text index rebuild](../full-text-search/overview.md#startup-and-recovery) from its table scan to current writes. Cleanup that removes the bridge during a rebuild can fail that attempt; stop aggressive cleanup before retrying so the next attempt can scan from a fresh committed-log boundary. Size automatic retention for the workload. ::: Parameters: diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 657ab1251..d4d711688 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -1822,7 +1822,7 @@ POST; requires `super_user`. Takes `database`, `table` and a proposed key-derive ### `cluster_status` -Returns connection state for all cluster nodes, including per-database socket status and replication timing statistics (`lastCommitConfirmed`, `lastReceivedRemoteTime`, `lastReceivedLocalTime`). Its top-level `sharding` object reports per-database map identity and placement blockers; see [Sharding Status](../replication/clustering.md#sharding-status). +Returns connection state for all cluster nodes, including per-database socket status and replication timing statistics (`lastCommitConfirmed`, `lastReceivedRemoteTime`, `lastReceivedLocalTime`). A socket whose base copy is not finishing also carries `stuckCopy`; see [Stuck Copies](../replication/clustering.md#stuck-copies). Its top-level `sharding` object reports per-database map identity and placement blockers; see [Sharding Status](../replication/clustering.md#sharding-status). ```json { "operation": "cluster_status" } diff --git a/reference/replication/clustering.md b/reference/replication/clustering.md index f35142dfd..60940b3c9 100644 --- a/reference/replication/clustering.md +++ b/reference/replication/clustering.md @@ -200,6 +200,54 @@ Returns an array of status objects from the cluster, including active WebSocket While a database is taking its initial full copy, these timing fields hold the literal string `"Copying"` rather than a timestamp. Anything that parses them as dates needs to handle that value, since a node taking a full copy is exactly the node you are most likely to be watching. +### Stuck Copies + + + +A `database_sockets` entry has a `stuckCopy` object while a [base copy](./overview.md#initial-synchronization-and-resynchronization) of that database from that peer is not finishing. Otherwise the field is absent. The node receiving the copy reports it, on its socket to the source, so look on the node that is catching up. + +```json +{ + "database": "data", + "connected": false, + "stuckCopy": { + "state": "held", + "reason": "the copy made no durable progress from its start position", + "count": 5, + "since": "2026-11-03T14:20:11.204Z", + "retryAt": "2026-11-03T14:21:11.204Z" + } +} +``` + +| Field | Description | +| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `state` | `retrying`, `held` or `refused` (below) | +| `reason` | `the copy made no durable progress from its start position` for `retrying` and `held`. For `refused`, the sharding rule that refused the copy | +| `count` | For `retrying` and `held`, how many times a copy has started from the same position. For `refused`, how many times in a row the copy was refused | +| `since` | When the socket entered its current state. Another attempt or hold in the same state does not change it | +| `retryAt` | When the socket reconnects after a hold. Present only while it waits out the hold, so it is absent once the reconnect opens, even if `state` is still `held` | + +`since` and `retryAt` are ISO 8601 strings, unlike the GMT strings in the timing fields above. + +The states: + +- **`retrying`**: a copy from this peer has started at least twice from the same position, with no durable progress in between. Reconnects follow the normal schedule. +- **`held`**: the session that made the fifth start from one position has closed. This node stops reconnecting that database to that peer for a while. Nothing in that database replicates from the peer during the hold, and the peer's copy stops too. The first hold lasts 60 seconds. After that, each reconnect makes one attempt. If it fails from the same position, the next hold is a random time between 60 seconds and a limit that doubles each time (2, 4, 8 minutes), up to 15 minutes. +- **`refused`**: this node refused the copy because of its [sharding](./residency.md#recovery-sources) ownership. With the reason `Owned sharding slots require a complete co-owner or backup restore`, this node owns slots and the peer cannot supply them. Typically the peer is not a complete co-owner, so make one reachable or restore a complete backup. Reconnects are held on the same schedule as `held`. With the reason `a sharded origin requires its origin or a complete co-owner`, the copy carried rows from a sharded origin that only that origin or a complete co-owner may send. Reconnects follow the normal schedule, because this usually follows a sharding map change. + +The attempt limit, the hold bounds and the 60-second quiet period below are fixed. No setting changes them. + +`stuckCopy` disappears in these cases: + +- The copy makes durable progress, continues from a different position, or finishes. This also starts the hold schedule over. +- A later connection to the peer stays open for 60 seconds without starting or refusing a copy. This clears only the report. The start count and the hold limit stay, so if a copy later fails from the same position again, the socket shows `retrying` with the count carried on, and the next hold follows the schedule from where it left off. +- A `refused` socket settles when the session of a later, accepted copy closes. It clears, or becomes `retrying` if that copy also started twice from one position. + +A missing `stuckCopy` does not show that a copy finished or that the two nodes hold the same data. + +Each change of state logs one line beginning `Replication of from :`. The level is `error` when the socket is held, `warn` when it starts retrying or is refused, and `info` when the report clears, such as `copy is progressing again`. A held socket logs, for example, `copy stopped after 5 attempts from one position; retrying in 60s: the copy made no durable progress from its start position`. Further attempts or holds in the same state log nothing more. + --- ### Sharding Status diff --git a/reference/replication/overview.md b/reference/replication/overview.md index 33d258f14..cf07bc128 100644 --- a/reference/replication/overview.md +++ b/reference/replication/overview.md @@ -364,7 +364,7 @@ See [Clustering Operations](./clustering.md#cluster-status) for the full respons Initial synchronization is decided per peer and per database. A new node can request a full copy from each in-scope peer for which it has no resume cursor, so a join can load several peers at once. [`add_node` with `isLeader: true`](./clustering.md#add-node) identifies a bootstrap source; it does not prevent full-copy requests to other cursorless peers. After synchronization, replication continues incrementally. -If a node goes offline and comes back, it resynchronizes automatically to catch up on missed transactions. If its saved position predates a peer's oldest retained history, it receives a base copy instead; [`logging.auditRetention`](../logging/configuration.md#loggingauditretention) defaults to `3d`, but disk pressure can purge history sooner. A base copy sends current rows as writes without clearing tables or deleting rows, so rows deleted elsewhere remain on the receiver. Copies also run in reverse: rows from the returning node can reappear on peers whose deletion tombstones have been purged. There is no resync or reconcile operation, and neither copy completion nor subsequent replay proves that the data matches. Keep a long-absent node isolated from replication before startup, then [remove it](./clustering.md#remove-node), rebuild it with empty databases, and [add it as a new node](./clustering.md#add-node) when peers are authoritative. When it is authoritative, keep it isolated, stop every peer before rebuilding any, rebuild each peer the same way, and only then lift the isolation one peer at a time, so each copies from it. [Operating Replication](/learn/administration/operating-replication#what-moves-automatically-and-what-does-not) gives the full order. +If a node goes offline and comes back, it resynchronizes automatically to catch up on missed transactions. If a peer has purged history the node would need to replay, it receives a base copy instead ([when a peer forces one](#when-a-peer-forces-a-base-copy)); [`logging.auditRetention`](../logging/configuration.md#loggingauditretention) defaults to `3d`, but disk pressure can purge history sooner. A base copy sends current rows as writes without clearing tables or deleting rows, so rows deleted elsewhere remain on the receiver. Copies also run in reverse: rows from the returning node can reappear on peers whose deletion tombstones have been purged. There is no resync or reconcile operation, and neither copy completion nor subsequent replay proves that the data matches. Keep a long-absent node isolated from replication before startup, then [remove it](./clustering.md#remove-node), rebuild it with empty databases, and [add it as a new node](./clustering.md#add-node) when peers are authoritative. When it is authoritative, keep it isolated, stop every peer before rebuilding any, rebuild each peer the same way, and only then lift the isolation one peer at a time, so each copies from it. [Operating Replication](/learn/administration/operating-replication#what-moves-automatically-and-what-does-not) gives the full order. You can also specify a `start_time` in the `add_node` operation to limit the initial download to data since a given point in time: @@ -376,6 +376,30 @@ You can also specify a `start_time` in the `add_node` operation to limit the ini } ``` +### When a Peer Forces a Base Copy + + + +The peer that would send the catch-up decides, per database, when the returning node subscribes. It logs `forcing a bounded base-copy resync` when it forces a copy. Before v5.4, a peer forced one whenever the returning node's position was older than `logging.auditRetention` or than the peer's oldest retained entry. A peer that had written nothing to a database since then therefore forced a full copy even though nothing was missed. + +From v5.4, a peer records how far each prune of its transaction logs reached. It forces a copy when the returning node's position, for a log it would replay, is below that point. A position that is old only because the peer wrote nothing to that database since, such as an idle or never-written transaction log, no longer forces a copy. A copy can still be forced although nothing was missed, in these cases: + +- The peer has no trustworthy record of what it pruned. It then uses the pre-v5.4 rule. +- The peer has no record of what was removed from a transaction log that was already purged empty before it ran v5.4. After the upgrade, such a log can force one more base copy of that database for each node returning with a position older than that purge. +- On RocksDB, the storage engine also purges transaction logs on its own after three days. If the peer itself was stopped for longer than that, a node whose position is older than three days gets one base copy of the database from it. + +A copy that keeps failing is reported as [`stuckCopy`](./clustering.md#stuck-copies) in `cluster_status`, and its reconnects are held back instead of retried indefinitely. + +### Rows From Unnamed Origins + + + +Each replicated row records its origin, the node that wrote it, as a numeric id that each node maps to a node name. A node can hold rows whose origin id has no name on that node, for example data migrated from v4, or rows from a node whose name mapping no longer exists. Before v5.4, a peer receiving such a row could not translate its origin, so a base copy that included it failed and restarted from the same position indefinitely. + +From v5.4, the sending node names that origin `unnamed-origin-@`, where `` is its origin id on the sending node and `` is the sending node's own name. Receivers treat it like any other origin. A v5.4 node receiving from an older node derives the same name itself and logs a warning: ` sent records from origin id , which has no name there; applying them as origin unnamed-origin-@`. It derives at most 64 new names per connection; past that, the connection closes and reconnects. + +In ``, each character other than ASCII letters, digits, `.` and `-` becomes `_` followed by its four-digit lowercase hexadecimal code, so `:` becomes `_003a` and `_` becomes `_005f`. These names identify where rows came from; no node by that name exists. They can appear in log lines on any node that receives or relays those rows. + ## Replicated Transactions The following data operations are replicated across the cluster: diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 31f4bdf00..b64b22c0f 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -34,6 +34,16 @@ Co-owner snapshot recovery does not restore the replacement's former local audit Snapshot-only recovery of a sharded node requires a fresh root and a new node name. Reusing the original name requires a complete stopped-node backup preserving audit through the last accepted write. A reused name missing its origin history refuses sharded writes and recovery-source eligibility, with both remedies in cluster status. See [sharding recovery](/reference/v5/replication/residency#recovering-a-lost-node). +### Base Copies That Could Not Finish + +[Stop forcing base copies that never finish](https://github.com/HarperFast/harper-pro/pull/1249) fixes three ways a base copy could start without need or restart forever: + +- **Idle databases no longer force copies.** A peer used to force a base copy whenever a returning node's position was older than `logging.auditRetention` or its own oldest retained entry, even for a database it had not written to since. It now forces one only when a prune may have removed history the node would replay. See [When a Peer Forces a Base Copy](/reference/v5/replication/overview#when-a-peer-forces-a-base-copy). +- **Rows with no origin name now copy.** Rows whose origin has no node name on the sending node, such as data migrated from v4, made the receiver fail and restart the copy. The sender now names that origin `unnamed-origin-@`, and those names can appear in logs. See [Rows From Unnamed Origins](/reference/v5/replication/overview#rows-from-unnamed-origins). +- **Stuck copies are held and reported.** When a copy starts five times from the same position, or the receiving node refuses it as a sharded owner, that node now holds its reconnects to the peer for that database: 60 seconds at first, then longer each time, up to 15 minutes. Before, it retried every few seconds. `cluster_status` reports the state as `stuckCopy` on the database socket, and each change of state logs one line. See [Stuck Copies](/reference/v5/replication/clustering#stuck-copies). + +**Upgrade impact:** after the upgrade, a transaction log that was already purged empty can force one more base copy of its database for each node returning with a position older than that purge. + ### LMDB Replication Confirmation LMDB writes that request `replicatedConfirmation` now wait for peer acknowledgments. Applications that previously received immediate success can now observe replication latency or an uncertain committed outcome if confirmation fails. See [Confirmation Failure After a Local Commit](/reference/v5/replication/sharding#confirmation-failure-after-a-local-commit) before retrying a failed write. From a1b2dff7acaff911486ffcacd29071644cbfbe31 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 3 Nov 2026 07:52:44 -0700 Subject: [PATCH 2/6] Tighten stuck-copy and purge-copy wording from review Name the reporting node for reverse copies, count refusal-reason changes as log transitions, scope `held` to one database, and list manual delete_transaction_logs_before purges among the conservative copy cases. Co-Authored-By: Claude Opus 5.5 Dispatch-Task: docs-harper-pro-1249-base-copy-liveness --- learn/administration/operating-replication.mdx | 2 +- reference/database/transaction.md | 2 +- reference/replication/clustering.md | 6 +++--- reference/replication/overview.md | 1 + 4 files changed, 6 insertions(+), 5 deletions(-) diff --git a/learn/administration/operating-replication.mdx b/learn/administration/operating-replication.mdx index 77c4c6688..cfa707a62 100644 --- a/learn/administration/operating-replication.mdx +++ b/learn/administration/operating-replication.mdx @@ -166,7 +166,7 @@ What each field is telling you: - **`lastCommitConfirmed`** is the last time this peer acknowledged receiving one of your commits. If it stops advancing while you are still writing, your writes are not landing on that peer. - **`lastReceivedRemoteTime`** is the source node's timestamp on the newest transaction you have received. - **`lastReceivedLocalTime`** is your own clock when you received it. -- **`stuckCopy`** appears only while a base copy of that database from that peer is not finishing: `retrying`, `held` (this node has stopped reconnecting to that peer for a while) or `refused` (this node's sharding ownership refused the copy). When it disappears, that does not prove the copy finished or that the data matches, so verify as described below. See [Stuck Copies](/reference/v5/replication/clustering#stuck-copies) for its fields and states. +- **`stuckCopy`** appears only while a base copy of that database from that peer is not finishing: `retrying`, `held` (copies of this database from that peer are on the hold schedule) or `refused` (this node's sharding rules refused the copy). When it disappears, that does not prove the copy finished or that the data matches, so verify as described below. See [Stuck Copies](/reference/v5/replication/clustering#stuck-copies) for its fields and states. Check connection state and sustained latency alongside commit progress during writes. Compare `lastReceivedRemoteTime` with the source's expected write timeline, then use a sentinel or journey test to verify freshness. `sendingMessage` appears only while a transaction is being sent; its absence on an idle socket is normal. diff --git a/reference/database/transaction.md b/reference/database/transaction.md index 04bb52d0f..c08e7be56 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -136,7 +136,7 @@ Deletes transaction log entries older than the specified timestamp. Deprecated i Deletes transaction log entries older than the specified timestamp. :::warning Database-wide and irreversible -On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This purges whole log files whose entries predate the timestamp, removing [`read_audit_log`](#read_audit_log) history for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender detects that the purge removed history the replica would replay and forces a full base copy instead of incremental catch-up (see [when a peer forces a base copy](../replication/overview.md#when-a-peer-forces-a-base-copy)) — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. The same history can be required to bridge a [full-text index rebuild](../full-text-search/overview.md#startup-and-recovery) from its table scan to current writes. Cleanup that removes the bridge during a rebuild can fail that attempt; stop aggressive cleanup before retrying so the next attempt can scan from a fresh committed-log boundary. Size automatic retention for the workload. +On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This purges whole log files whose entries predate the timestamp, removing [`read_audit_log`](#read_audit_log) history for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender forces a full base copy instead of incremental catch-up for a replica whose position is older than the purge timestamp, even if the database had no writes after that position (see [when a peer forces a base copy](../replication/overview.md#when-a-peer-forces-a-base-copy)) — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. The same history can be required to bridge a [full-text index rebuild](../full-text-search/overview.md#startup-and-recovery) from its table scan to current writes. Cleanup that removes the bridge during a rebuild can fail that attempt; stop aggressive cleanup before retrying so the next attempt can scan from a fresh committed-log boundary. Size automatic retention for the workload. ::: Parameters: diff --git a/reference/replication/clustering.md b/reference/replication/clustering.md index 60940b3c9..9562a35e9 100644 --- a/reference/replication/clustering.md +++ b/reference/replication/clustering.md @@ -204,7 +204,7 @@ While a database is taking its initial full copy, these timing fields hold the l -A `database_sockets` entry has a `stuckCopy` object while a [base copy](./overview.md#initial-synchronization-and-resynchronization) of that database from that peer is not finishing. Otherwise the field is absent. The node receiving the copy reports it, on its socket to the source, so look on the node that is catching up. +A `database_sockets` entry has a `stuckCopy` object while a [base copy](./overview.md#initial-synchronization-and-resynchronization) of that database from that peer is not finishing. Otherwise the field is absent. The node receiving the copy reports it, on its socket to the source. Copies also run in reverse, so that can be a peer of the node that came back rather than the node itself. Run `cluster_status` on every node. ```json { @@ -234,7 +234,7 @@ The states: - **`retrying`**: a copy from this peer has started at least twice from the same position, with no durable progress in between. Reconnects follow the normal schedule. - **`held`**: the session that made the fifth start from one position has closed. This node stops reconnecting that database to that peer for a while. Nothing in that database replicates from the peer during the hold, and the peer's copy stops too. The first hold lasts 60 seconds. After that, each reconnect makes one attempt. If it fails from the same position, the next hold is a random time between 60 seconds and a limit that doubles each time (2, 4, 8 minutes), up to 15 minutes. -- **`refused`**: this node refused the copy because of its [sharding](./residency.md#recovery-sources) ownership. With the reason `Owned sharding slots require a complete co-owner or backup restore`, this node owns slots and the peer cannot supply them. Typically the peer is not a complete co-owner, so make one reachable or restore a complete backup. Reconnects are held on the same schedule as `held`. With the reason `a sharded origin requires its origin or a complete co-owner`, the copy carried rows from a sharded origin that only that origin or a complete co-owner may send. Reconnects follow the normal schedule, because this usually follows a sharding map change. +- **`refused`**: this node refused the copy under its [sharding](./residency.md#recovery-sources) rules. With the reason `Owned sharding slots require a complete co-owner or backup restore`, this node owns slots and the peer cannot supply them. Typically the peer is not a complete co-owner, so make one reachable or restore a complete backup. Reconnects are held on the same schedule as `held`. With the reason `a sharded origin requires its origin or a complete co-owner`, the copy carried rows from a sharded origin that only that origin or a complete co-owner may send. Reconnects follow the normal schedule, because this usually follows a sharding map change. The attempt limit, the hold bounds and the 60-second quiet period below are fixed. No setting changes them. @@ -246,7 +246,7 @@ The attempt limit, the hold bounds and the 60-second quiet period below are fixe A missing `stuckCopy` does not show that a copy finished or that the two nodes hold the same data. -Each change of state logs one line beginning `Replication of from :`. The level is `error` when the socket is held, `warn` when it starts retrying or is refused, and `info` when the report clears, such as `copy is progressing again`. A held socket logs, for example, `copy stopped after 5 attempts from one position; retrying in 60s: the copy made no durable progress from its start position`. Further attempts or holds in the same state log nothing more. +Each change of state, or of the refusal reason, logs one line beginning `Replication of from :`. The level is `error` when the socket is held, `warn` when it starts retrying or is refused, and `info` when the report clears, such as `copy is progressing again`. A held socket logs, for example, `copy stopped after 5 attempts from one position; retrying in 60s: the copy made no durable progress from its start position`. Further attempts or holds in the same state log nothing more. --- diff --git a/reference/replication/overview.md b/reference/replication/overview.md index cf07bc128..93779ed73 100644 --- a/reference/replication/overview.md +++ b/reference/replication/overview.md @@ -385,6 +385,7 @@ The peer that would send the catch-up decides, per database, when the returning From v5.4, a peer records how far each prune of its transaction logs reached. It forces a copy when the returning node's position, for a log it would replay, is below that point. A position that is old only because the peer wrote nothing to that database since, such as an idle or never-written transaction log, no longer forces a copy. A copy can still be forced although nothing was missed, in these cases: - The peer has no trustworthy record of what it pruned. It then uses the pre-v5.4 rule. +- After [`delete_transaction_logs_before`](../database/transaction.md#delete_transaction_logs_before), a node whose position is older than the timestamp it was given gets a base copy, even if the peer wrote nothing to that database after that position. - The peer has no record of what was removed from a transaction log that was already purged empty before it ran v5.4. After the upgrade, such a log can force one more base copy of that database for each node returning with a position older than that purge. - On RocksDB, the storage engine also purges transaction logs on its own after three days. If the peer itself was stopped for longer than that, a node whose position is older than three days gets one base copy of the database from it. From 84b2299a14180a579f51bf2d2fc6af1a0772e27b Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 3 Nov 2026 08:00:42 -0700 Subject: [PATCH 3/6] Describe the manual-purge copy point as at most the given timestamp Co-Authored-By: Claude Opus 5.5 Dispatch-Task: docs-harper-pro-1249-base-copy-liveness --- reference/database/transaction.md | 2 +- reference/replication/overview.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/database/transaction.md b/reference/database/transaction.md index c08e7be56..96f761662 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -136,7 +136,7 @@ Deletes transaction log entries older than the specified timestamp. Deprecated i Deletes transaction log entries older than the specified timestamp. :::warning Database-wide and irreversible -On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This purges whole log files whose entries predate the timestamp, removing [`read_audit_log`](#read_audit_log) history for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender forces a full base copy instead of incremental catch-up for a replica whose position is older than the purge timestamp, even if the database had no writes after that position (see [when a peer forces a base copy](../replication/overview.md#when-a-peer-forces-a-base-copy)) — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. The same history can be required to bridge a [full-text index rebuild](../full-text-search/overview.md#startup-and-recovery) from its table scan to current writes. Cleanup that removes the bridge during a rebuild can fail that attempt; stop aggressive cleanup before retrying so the next attempt can scan from a fresh committed-log boundary. Size automatic retention for the workload. +On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This purges whole log files whose entries predate the timestamp, removing [`read_audit_log`](#read_audit_log) history for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender forces a full base copy instead of incremental catch-up for a replica whose position is below the point the purge reached (at most the purge timestamp), even if the database had no writes after that position (see [when a peer forces a base copy](../replication/overview.md#when-a-peer-forces-a-base-copy)) — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. The same history can be required to bridge a [full-text index rebuild](../full-text-search/overview.md#startup-and-recovery) from its table scan to current writes. Cleanup that removes the bridge during a rebuild can fail that attempt; stop aggressive cleanup before retrying so the next attempt can scan from a fresh committed-log boundary. Size automatic retention for the workload. ::: Parameters: diff --git a/reference/replication/overview.md b/reference/replication/overview.md index 93779ed73..d6869d462 100644 --- a/reference/replication/overview.md +++ b/reference/replication/overview.md @@ -385,7 +385,7 @@ The peer that would send the catch-up decides, per database, when the returning From v5.4, a peer records how far each prune of its transaction logs reached. It forces a copy when the returning node's position, for a log it would replay, is below that point. A position that is old only because the peer wrote nothing to that database since, such as an idle or never-written transaction log, no longer forces a copy. A copy can still be forced although nothing was missed, in these cases: - The peer has no trustworthy record of what it pruned. It then uses the pre-v5.4 rule. -- After [`delete_transaction_logs_before`](../database/transaction.md#delete_transaction_logs_before), a node whose position is older than the timestamp it was given gets a base copy, even if the peer wrote nothing to that database after that position. +- After [`delete_transaction_logs_before`](../database/transaction.md#delete_transaction_logs_before), a node whose position is below the point that purge reached (at most the timestamp it was given) gets a base copy, even if the peer wrote nothing to that database after that position. - The peer has no record of what was removed from a transaction log that was already purged empty before it ran v5.4. After the upgrade, such a log can force one more base copy of that database for each node returning with a position older than that purge. - On RocksDB, the storage engine also purges transaction logs on its own after three days. If the peer itself was stopped for longer than that, a node whose position is older than three days gets one base copy of the database from it. From 23f6a49cfdd59abed73d9f49a3b5f1c7752f2990 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 3 Nov 2026 08:03:27 -0700 Subject: [PATCH 4/6] Point the Learn copy statements at the 5.4 base-copy rule Co-Authored-By: Claude Opus 5.5 Dispatch-Task: docs-harper-pro-1249-base-copy-liveness --- learn/administration/backup-and-recovery.mdx | 2 +- learn/administration/operating-replication.mdx | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/learn/administration/backup-and-recovery.mdx b/learn/administration/backup-and-recovery.mdx index 2ab17f47f..3ec8fad44 100644 --- a/learn/administration/backup-and-recovery.mdx +++ b/learn/administration/backup-and-recovery.mdx @@ -168,7 +168,7 @@ Also include these constraints in the restore procedure: ## Restoring in a replicated cluster -A restored node may be behind peers that have continued accepting writes. Peers can replay newer writes, including the destructive mutations you intended to undo. If its restored data and saved replication position predate the peers' retained history, copies run in both directions and can bring deleted rows back onto peers. Follow the [long-absent node policy](./operating-replication.mdx#what-moves-automatically-and-what-does-not) before reconnecting. Restoring one node does not make its backup authoritative; choose which data to preserve or repair and rehearse the re-entry policy before reconnecting it. +A restored node may be behind peers that have continued accepting writes. Peers can replay newer writes, including the destructive mutations you intended to undo. If peers have purged history its saved replication position would replay, [copies run](/reference/v5/replication/overview#when-a-peer-forces-a-base-copy) in both directions and can bring deleted rows back onto peers. Follow the [long-absent node policy](./operating-replication.mdx#what-moves-automatically-and-what-does-not) before reconnecting. Restoring one node does not make its backup authoritative; choose which data to preserve or repair and rehearse the re-entry policy before reconnecting it. ```mermaid flowchart TD diff --git a/learn/administration/operating-replication.mdx b/learn/administration/operating-replication.mdx index cfa707a62..d61815ada 100644 --- a/learn/administration/operating-replication.mdx +++ b/learn/administration/operating-replication.mdx @@ -95,7 +95,7 @@ There is one socket per database per peer, so the sockets present in the respons | Data mutations | Insert, update, upsert, delete, and bulk load replicate for tables in scope | Monitor connections, latency, and convergence for the databases that serve critical journeys | | Transactions | Replicated atomically, and a transaction may span multiple tables | Tables that need to commit together must live in the same database | | A brand new node | Requests a full copy of each in-scope database from every peer it has no resume cursor for, then enters incremental replication | Budget N transfers, not one. See the note below before sizing a join | -| A returning node | Resynchronizes automatically to catch up on the transactions it missed | Full copy when peers no longer hold the needed history (`logging.auditRetention`, sooner under disk pressure). See below | +| A returning node | Resynchronizes automatically to catch up on the transactions it missed | Full copy when peers no longer hold the needed history ([when](/reference/v5/replication/overview#when-a-peer-forces-a-base-copy)). See below | | Component deployment | `deploy_component` replicates by default; pass `"replicated": false` to hold it to one node | Cluster-wide is the default, so single-node validation is the case you must ask for | | Configuration | `set_configuration` supports `"replicated": true` | Never replicate node-local parameters. See the warning below | | Users and roles | Propagate when the `system` database is in replication scope, which is the default | If you narrow scope to exclude `system`, identity must be provisioned on every node by your own automation | @@ -166,7 +166,7 @@ What each field is telling you: - **`lastCommitConfirmed`** is the last time this peer acknowledged receiving one of your commits. If it stops advancing while you are still writing, your writes are not landing on that peer. - **`lastReceivedRemoteTime`** is the source node's timestamp on the newest transaction you have received. - **`lastReceivedLocalTime`** is your own clock when you received it. -- **`stuckCopy`** appears only while a base copy of that database from that peer is not finishing: `retrying`, `held` (copies of this database from that peer are on the hold schedule) or `refused` (this node's sharding rules refused the copy). When it disappears, that does not prove the copy finished or that the data matches, so verify as described below. See [Stuck Copies](/reference/v5/replication/clustering#stuck-copies) for its fields and states. +- **`stuckCopy`** appears only while a base copy of that database from that peer is not finishing: `retrying`, `held` (copies of this database from that peer are on the hold schedule) or `refused` (this node's sharding rules refused the copy). When it disappears, that does not prove the copy finished or that the data matches, so verify as described below. Its `since` and `retryAt` are ISO 8601 strings, not GMT strings like the fields above. See [Stuck Copies](/reference/v5/replication/clustering#stuck-copies) for its fields and states. Check connection state and sustained latency alongside commit progress during writes. Compare `lastReceivedRemoteTime` with the source's expected write timeline, then use a sentinel or journey test to verify freshness. `sendingMessage` appears only while a transaction is being sent; its absence on an idle socket is normal. From f4b84d422f82fea1e5e3bf0533fd358174a8e5d8 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 3 Nov 2026 08:16:21 -0700 Subject: [PATCH 5/6] Smooth the purged-history wording and split the purge warning Co-Authored-By: Claude Opus 5.5 Dispatch-Task: docs-harper-pro-1249-base-copy-liveness --- learn/administration/backup-and-recovery.mdx | 2 +- learn/administration/operating-replication.mdx | 2 +- reference/database/transaction.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/learn/administration/backup-and-recovery.mdx b/learn/administration/backup-and-recovery.mdx index 3ec8fad44..150782185 100644 --- a/learn/administration/backup-and-recovery.mdx +++ b/learn/administration/backup-and-recovery.mdx @@ -168,7 +168,7 @@ Also include these constraints in the restore procedure: ## Restoring in a replicated cluster -A restored node may be behind peers that have continued accepting writes. Peers can replay newer writes, including the destructive mutations you intended to undo. If peers have purged history its saved replication position would replay, [copies run](/reference/v5/replication/overview#when-a-peer-forces-a-base-copy) in both directions and can bring deleted rows back onto peers. Follow the [long-absent node policy](./operating-replication.mdx#what-moves-automatically-and-what-does-not) before reconnecting. Restoring one node does not make its backup authoritative; choose which data to preserve or repair and rehearse the re-entry policy before reconnecting it. +A restored node may be behind peers that have continued accepting writes. Peers can replay newer writes, including the destructive mutations you intended to undo. If peers have purged history that its saved replication position would replay, [copies run](/reference/v5/replication/overview#when-a-peer-forces-a-base-copy) in both directions and can bring deleted rows back onto peers. Follow the [long-absent node policy](./operating-replication.mdx#what-moves-automatically-and-what-does-not) before reconnecting. Restoring one node does not make its backup authoritative; choose which data to preserve or repair and rehearse the re-entry policy before reconnecting it. ```mermaid flowchart TD diff --git a/learn/administration/operating-replication.mdx b/learn/administration/operating-replication.mdx index d61815ada..506ff405e 100644 --- a/learn/administration/operating-replication.mdx +++ b/learn/administration/operating-replication.mdx @@ -102,7 +102,7 @@ There is one socket per database per peer, so the sockets present in the respons | Node registry (`hdb_nodes`) | Lives in `system`, so it propagates with it. Each node rewrites its own self-record from its `harper-config.yaml` routes on restart or component reload | Scoping applied through `add_node` alone does not survive a restart. Put durable constraints in config routes, and see below if you need the registry kept per cluster | | Destructive schema changes | `drop_database` and `drop_table` replicate by default; `drop_attribute` does not | A drop is a cluster-wide event unless you pass `"replicated": false`. Verify the result on every node | -Measure incremental catch-up and full synchronization separately. Peers retain transaction history for [`logging.auditRetention`](/reference/v5/logging/configuration#loggingauditretention) (default `3d`), but purge it sooner under disk pressure. When a peer has purged history the saved position would replay, it logs `forcing a bounded base-copy resync` and sends its current rows as writes. A database the peer has not written to since no longer forces a copy; see [when a peer forces a base copy](/reference/v5/replication/overview#when-a-peer-forces-a-base-copy) for the cases that still do. Incremental replay resumes from the copy's start time. The copy never clears tables or deletes rows: purged tombstones are absent, and retained tombstones arrive as writes without values that the receiver skips. It also runs in reverse. The returning node sends its rows to peers, including writes only it took. A copied row wins where it is newer; where a peer has purged a deletion's tombstone, the deleted row can reappear. +Measure incremental catch-up and full synchronization separately. Peers retain transaction history for [`logging.auditRetention`](/reference/v5/logging/configuration#loggingauditretention) (default `3d`), but purge it sooner under disk pressure. When a peer has purged history that the saved position would replay, it logs `forcing a bounded base-copy resync` and sends its current rows as writes. A database that the peer has not written to since then no longer forces a copy; see [when a peer forces a base copy](/reference/v5/replication/overview#when-a-peer-forces-a-base-copy) for the cases that still do. Incremental replay resumes from the copy's start time. The copy never clears tables or deletes rows: purged tombstones are absent, and retained tombstones arrive as writes without values that the receiver skips. It also runs in reverse. The returning node sends its rows to peers, including writes only it took. A copied row wins where it is newer; where a peer has purged a deletion's tombstone, the deleted row can reappear. :::danger **A long-absent node must not reconnect with its existing data.** If it was disconnected longer than its peers retain history, or you cannot show otherwise, keep it out of rotation and isolate replication before Harper starts. Then rebuild whichever side does not hold the authoritative data: diff --git a/reference/database/transaction.md b/reference/database/transaction.md index 96f761662..55cd8d3f1 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -136,7 +136,7 @@ Deletes transaction log entries older than the specified timestamp. Deprecated i Deletes transaction log entries older than the specified timestamp. :::warning Database-wide and irreversible -On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This purges whole log files whose entries predate the timestamp, removing [`read_audit_log`](#read_audit_log) history for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender forces a full base copy instead of incremental catch-up for a replica whose position is below the point the purge reached (at most the purge timestamp), even if the database had no writes after that position (see [when a peer forces a base copy](../replication/overview.md#when-a-peer-forces-a-base-copy)) — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. The same history can be required to bridge a [full-text index rebuild](../full-text-search/overview.md#startup-and-recovery) from its table scan to current writes. Cleanup that removes the bridge during a rebuild can fail that attempt; stop aggressive cleanup before retrying so the next attempt can scan from a fresh committed-log boundary. Size automatic retention for the workload. +On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This purges whole log files whose entries predate the timestamp, removing [`read_audit_log`](#read_audit_log) history for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica: the sender forces a full base copy instead of incremental catch-up. That happens for any replica whose position is below the point the purge reached (at most the purge timestamp), even if the database had no writes after that position (see [when a peer forces a base copy](../replication/overview.md#when-a-peer-forces-a-base-copy)). A full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. The same history can be required to bridge a [full-text index rebuild](../full-text-search/overview.md#startup-and-recovery) from its table scan to current writes. Cleanup that removes the bridge during a rebuild can fail that attempt; stop aggressive cleanup before retrying so the next attempt can scan from a fresh committed-log boundary. Size automatic retention for the workload. ::: Parameters: From 33a9412650ec5ccaca17f92f5f48b52b7cba1e2f Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 3 Nov 2026 08:24:52 -0700 Subject: [PATCH 6/6] Link the health-checks returning-node note to the 5.4 base-copy rule Co-Authored-By: Claude Opus 5.5 Dispatch-Task: docs-harper-pro-1249-base-copy-liveness --- learn/administration/health-checks-and-traffic-admission.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/learn/administration/health-checks-and-traffic-admission.mdx b/learn/administration/health-checks-and-traffic-admission.mdx index d523cd5ff..366601d8e 100644 --- a/learn/administration/health-checks-and-traffic-admission.mdx +++ b/learn/administration/health-checks-and-traffic-admission.mdx @@ -325,7 +325,7 @@ Record the third number as node re-entry time, including convergence and applica ## Operational notes -- **A returning node is not the same as a new node.** A node whose databases have never synchronized downloads them in full. A briefly offline node catches up from retained history; a long-absent node needs the rebuild or restore policy above. Both need to finish before traffic arrives, but they take very different amounts of time, so do not budget for the first when you are planning a routine restart. +- **A returning node is not the same as a new node.** A node whose databases have never synchronized downloads them in full. A briefly offline node usually catches up from retained history ([when it gets a full copy instead](/reference/v5/replication/overview#when-a-peer-forces-a-base-copy)); a long-absent node needs the rebuild or restore policy above. Both need to finish before traffic arrives, but they take very different amounts of time, so do not budget for the first when you are planning a routine restart. - **Set unavailable before recovery work, not after.** Any operation that touches data on a node, including a restore, should happen with the node out of rotation. - **Configuration changes need a restart to take effect**, so a node that has been reconfigured but not restarted is running the old configuration while reporting the new one. Sequence the restart into the same maintenance window. - **Fabric routes by its status component.** It does not automatically consume your readiness route. For application-driven availability, see [Taking an Unhealthy Node Out of Rotation](../developers/taking-a-node-out-of-rotation.mdx).