diff --git a/reference/configuration/options.md b/reference/configuration/options.md index ee62df608..c219c1d53 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -286,6 +286,9 @@ replication: - `securePort` — Secure replication port; _Default_: `9933` (changed from `9925` in v4.5.0) - `threads` — Worker threads dedicated to replication, which then own the replication port and every subscription and load no application code; requires `port` or `securePort`; see [Dedicated Replication Threads](../replication/overview.md#dedicated-replication-threads); _Default_: `0` (replication runs on the HTTP worker threads) (Added in: v5.4.0) - `enableRootCAs` — Verify against Node.js Mozilla CA store; _Default_: `true` +- `pingInterval` — Interval between replication keep-alive pings (ms); _Default_: `30000` +- `pingTimeout` — Time without received socket activity before a replication connection is considered dead (ms); _Default_: twice `pingInterval`. +- `confirmationTimeout` — Timeout in milliseconds for explicit `replicatedConfirmation`, read when a waiter starts. _Default_: `900000`, or `blobTimeout` when greater. An explicitly configured value overrides that fallback, including a smaller value. Range: `1`–`2147483647`. (Added in: v5.4.0) - `blobTimeout` — Blob transfer timeout (ms); _Default_: `120000` - `blobGapReconnectMs` — Interval (ms) for the blob-gap watchdog: when a transient blob save failure pins a replication resume cursor, the connection is forced to reconnect on this cadence so the gapped blob is re-streamed and, during a bulk copy, the copy resumes from the last banked cursor. Lower values heal gaps faster at the cost of more reconnects on a link whose faults never heal; _Default_: the `blobTimeout` value - `copyCursorFlushBytes` — Bytes of applied bulk-copy data between durable flushes of the copy resume cursor (RocksDB); _Default_: `67108864` @@ -293,6 +296,7 @@ replication: - `blobSendDrainTimeout` — Maximum time (ms) a worker waits for in-flight replication blob **sends** to finish before shutting down during a restart, so a rolling restart (e.g., a component deploy reload) doesn't interrupt a transfer in progress. Only sends that are still making progress are waited on; `0` disables draining; _Default_: `600000` - `failOver` — Failover to alternate node if peer unreachable; _Default_: `true` - `shard` — Shard ID for traffic routing; see [Sharding](../replication/sharding.md) +- `shards` — Complete array of unique shard IDs, integers `0`–`65535`; order does not matter. Required for a declarative database with no installed map; custom-map conversion can omit it. All peers must agree, and the local `shard` must be included when configured. Changing it never recomputes an installed map; see [Complete shard-id set](../replication/sharding.md#complete-shard-id-set). (Added in: v5.4.0) - `mtls.certificateVerification` — Certificate revocation checking (CRL/OCSP) for replication connections; see [Certificate Verification](../security/certificate-verification.md) - `logging` — Replication event logging; sub-options: `path`, `level`, `tag`, `stdStreams`. See [Logging Configuration](../logging/configuration.md) diff --git a/reference/database/api.md b/reference/database/api.md index 6197ec4a6..e48b907ae 100644 --- a/reference/database/api.md +++ b/reference/database/api.md @@ -134,6 +134,12 @@ Two details matter outside a request context: - **`await` the `transaction()` call, and catch it.** An unawaited call lets the timer callback finish before the commit resolves, so a failed write is never observed; awaiting without a `catch` still leaves the rejection unhandled, because nothing awaits the callback itself. - **Guard against overlap when a job can outlast its interval.** A timer fires on schedule regardless of whether the previous run has committed, so two runs would open two transactions over the same rows. The `running` flag above skips a tick instead. +### Replication confirmation + +Set `replicatedConfirmation` on a fresh context before calling `transaction()` to request confirmation for its writes. Pass the callback's `txn` argument to table methods to bind them explicitly. Reusing a context with an active transaction does not change that transaction's confirmation settings. See the [independent-transaction example](../replication/sharding.md#programmatic-replication-control), [uncertain-outcome behavior](../replication/sharding.md#confirmation-failure-after-a-local-commit), and [LMDB confirmation behavior](../replication/sharding.md#confirmation-outcomes). + +For `@sharded` tables, confirmation counts other owners in each affected owner set of the first written database; a temporary replica cannot satisfy it. `replicateTo` does not change those owners. See [declarative ownership](../replication/residency.md#ownership-reads-and-writes). + ### Nesting If `transaction()` is called with a context that already has an active transaction, it reuses that transaction, executes the callback immediately, and returns. This makes `transaction()` safe to call defensively to ensure a transaction is always active. diff --git a/reference/database/schema.md b/reference/database/schema.md index 7035be5bf..d56f1850c 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -73,19 +73,19 @@ type MyTable @table { Optional arguments: -| Argument | Type | Default | Description | -| ------------------------ | --------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | -| `table` | `String` | type name | Override the table name | -| `database` | `String` | `"data"` | Database to place the table in | -| `expiration` | `Int` | — | Seconds until a record goes stale (useful for caching tables) | -| `eviction` | `Int` | `0` | Additional seconds after `expiration` before a record is physically removed | -| `scanInterval` | `Int` | `(expiration + eviction) / 4` | Seconds between eviction scans | -| `replicate` | `Boolean` | true | Enable replication of this table | -| `replicaExpiration` | `Float` | `3600` | Seconds a record fetched from another node is kept as a [local replica](#replicaexpiration-and-replicarefreshfraction); `0` stops new admissions | -| `replicaRefreshFraction` | `Float` | `0.5` | Fraction of `replicaExpiration` after which a read of a replica renews its lifetime; `1` never renews | -| `audit` | `Boolean` | `logging.auditLog` for ordinary tables | Enable the table's transaction log; new tables with `@fullText` require explicit `true` | -| `cacheControl` | `String` | — | `Cache-Control` header value emitted on anonymous GET/HEAD 200/304 responses for this table | -| `randomAccessFields` | `Boolean` | `storage.randomAccessFields` | [Pin this table's record encoding](#randomaccessfields) | +| Argument | Type | Default | Description | +| ------------------------ | --------- | -------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | +| `table` | `String` | type name | Override the table name | +| `database` | `String` | `"data"` | Database to place the table in | +| `expiration` | `Int` | — | Seconds until a record goes stale (useful for caching tables) | +| `eviction` | `Int` | `0` | Additional seconds after `expiration` before a record is physically removed | +| `scanInterval` | `Int` | `(expiration + eviction) / 4` | Seconds between eviction scans | +| `replicate` | `Boolean` | true | Enable replication of this table | +| `replicaExpiration` | `Float` | `3600` | Seconds to keep a [local residency replica](#replicaexpiration-and-replicarefreshfraction); `0` stops new admissions | +| `replicaRefreshFraction` | `Float` | `0.5` | Fraction of `replicaExpiration` after which a read of a replica renews its lifetime; `1` never renews | +| `audit` | `Boolean` | `logging.auditLog` for ordinary tables | Enable the table's transaction log; new tables with `@fullText` require explicit `true` | +| `cacheControl` | `String` | — | `Cache-Control` header value emitted on anonymous GET/HEAD 200/304 responses for this table | +| `randomAccessFields` | `Boolean` | `storage.randomAccessFields` | [Pin this table's record encoding](#randomaccessfields) | **`expiration`, `eviction`, and `scanInterval`** @@ -141,11 +141,13 @@ Local replicas are RocksDB-only. On the deprecated LMDB engine, remote reads rem On RocksDB, schema rescans apply changes to already-loaded tables on other workers. The new lifetime applies to subsequent admissions and renewals; existing deadlines are not rewritten. Setting `replicaExpiration: 0` stops new admissions and the sweep; existing replicas remain until overwritten or evicted. -On a table whose records live on specific nodes (see [sharding](../replication/sharding.md#by-record-content-setresidency)), a node that does not hold a record fetches it from a node that does. The fetched record is kept locally as a _replica_: a complete copy that reads and searches on that node use without contacting another node again. A replica does not move the record and is never replicated onward. A write by an owner reaches the node as an invalidation, which turns the replica back into a stub, so the next read fetches the new version. Tables sharded by primary key only (`setResidencyById`) keep no replicas. +On a table whose records live on specific nodes (see [sharding](../replication/sharding.md#by-record-content-setresidency)), a node that does not hold a record fetches it from a node that does. The fetched record is kept locally as a _replica_: a complete copy that reads and searches on that node use without contacting another node again. A replica does not move the record and is never replicated onward. A write by an owner reaches the node as an invalidation, which turns the replica back into a stub, so the next read fetches the new version. Tables using the older [`setResidencyById`](../replication/sharding.md#by-primary-key-only-setresidencybyid) mechanism keep no replicas. Schema-defined `@sharded` tables follow the rules below. - **`replicaExpiration`** — Seconds a replica is kept. A sweep runs every quarter of this time, with a minimum interval of one second, and reduces replicas past their lifetime back to their indexed stubs. A replica can still be served until the sweep reaches it; the next read after demotion fetches the record again. `0` disables new replica admissions and the sweep. A replica never outlives the record's own `expiration`, if it has one. - **`replicaRefreshFraction`** — A read of a replica after this fraction of `replicaExpiration` has passed renews the lifetime, so records that stay in use stay local. Must be greater than `0` and at most `1`; `1` gives every replica a fixed lifetime. +[Schema-defined sharding](../replication/residency.md#ownership-reads-and-writes) tables with secondary indexes can retain replicas for key-derived slots. With positive `replicaExpiration`, a non-owner writer also keeps its complete write as a temporary replica under the same expiry and replacement rules. This local copy never counts as owner confirmation. Setting the lifetime to `0` does not drain existing replicas; see the [rollback procedure](../replication/residency.md#rollback). + ```graphql # Keep fetched records for a day; a read after six hours renews them type Order @table(replicaExpiration: 86400, replicaRefreshFraction: 0.25) { @@ -238,6 +240,12 @@ export class Product extends tables.Product { See [REST Headers / Cache-Control](../rest/headers.md#cache-control) for the full caching behavior and how `Cache-Control` interacts with authentication headers. +### `@sharded` + + + +Defines immutable primary-key placement on a schema-defined RocksDB table. LMDB declarations are refused; legacy residency functions are unchanged. `slots` is the database-wide count (`1` through `65536`), adopting the installed map's count or defaulting to `1024`. The key hash selects a slot and the installed database map selects its shard. `by` and `slotFrom` are not supported yet; see [concurrent-move design (#1178)](https://github.com/HarperFast/harper-pro/issues/1178) and [Schema-defined sharding](../replication/residency.md). + ### `@export` Exposes the table as an externally accessible resource endpoint, available via REST, MQTT, and other interfaces. @@ -767,7 +775,9 @@ export class Reports extends tables.Reports { For vector queries, `rowFilter` participates in HNSW traversal. A caller therefore receives the k nearest _matching_ records rather than "nearest k, minus filtered records." `rowFilter` can apply to every candidate visited; prefer indexed query conditions for predicates that can be expressed declaratively. -`rowFilter` applies only when a non-raw `put` or `invalidate` event carries an authoritative row value. Delete tombstones, value-less invalidations, raw events, and published messages may not contain the complete current record. A subscription that needs such events can also provide an `eventFilter`: +On a non-owner for a sharded table, invalidation values contain only the local index projection. A `rowFilter` on an unindexed field therefore sees it as missing, so a negated predicate can pass; do not use one to decide whether a subscriber may learn a record's id or indexed fields. See the [subscription limitation](../replication/residency.md#subscription-filters). + +`rowFilter` applies only when a non-raw `put` or `invalidate` event carries a row value. Delete tombstones, value-less invalidations, raw events, and published messages may not contain the complete current record. A subscription that needs such events can also provide an `eventFilter`: ```javascript subscribe(request) { diff --git a/reference/database/system-tables.md b/reference/database/system-tables.md index a86d47a2e..c8c8d3c39 100644 --- a/reference/database/system-tables.md +++ b/reference/database/system-tables.md @@ -133,6 +133,12 @@ This table is managed automatically by the Data Loader. No direct interaction is ## Replication Tables +### `hdb_residency_slots` + + + +Stores one authoritative [shard map](../replication/residency.md#installed-maps) snapshot per database, keyed by `database`, with `version`, `digest`, `slots`, and complete nonoverlapping `ranges`. Install custom maps through the super-user `initialize_shard_map` operation. Active assignments cannot move, and deleting an installed map is refused. A lost or corrupted snapshot must be recovered; it is never regenerated from configuration. + ### `hdb_nodes` Stores the configuration and state of known nodes in a cluster, including connection details, replication settings, and revoked certificate serial numbers. diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 8b7439933..0897cde81 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -1780,16 +1780,18 @@ Operations for configuring and managing Harper cluster replication. Detailed documentation: [Replication & Clustering](../replication/clustering.md) -| Operation | Description | Role Required | -| ----------------------- | --------------------------------------------------------------- | ------------- | -| `add_node` | Adds a Harper instance to the cluster | super_user | -| `update_node` | Modifies an existing node's subscriptions | super_user | -| `remove_node` | Removes a node from the cluster | super_user | -| `cluster_status` | Returns current cluster connection status | super_user | -| `configure_cluster` | Bulk-creates/resets cluster subscriptions across multiple nodes | super_user | -| `cluster_set_routes` | Adds routes to the replication routes config (PATCH/upsert) | super_user | -| `cluster_get_routes` | Returns the current replication routes config | super_user | -| `cluster_delete_routes` | Removes routes from the replication routes config | super_user | +| Operation | Description | Role Required | +| ----------------------- | ---------------------------------------------------------------- | ------------- | +| `add_node` | Adds a Harper instance to the cluster | super_user | +| `update_node` | Modifies an existing node's subscriptions | super_user | +| `remove_node` | Removes a node from the cluster | super_user | +| `initialize_shard_map` | Installs a custom database slot map without moving records | super_user | +| `verify_shard_map` | Compares a legacy placement function with a proposed slot policy | super_user | +| `cluster_status` | Returns current cluster connection status | super_user | +| `configure_cluster` | Bulk-creates/resets cluster subscriptions across multiple nodes | super_user | +| `cluster_set_routes` | Adds routes to the replication routes config (PATCH/upsert) | super_user | +| `cluster_get_routes` | Returns the current replication routes config | super_user | +| `cluster_delete_routes` | Removes routes from the replication routes config | super_user | ### `add_node` @@ -1804,9 +1806,23 @@ Adds a remote Harper node to the cluster. If `subscriptions` are not provided, a } ``` +### `initialize_shard_map` + + + +POST; requires `super_user` and a RocksDB database. Takes `database`, `slots` and complete nonoverlapping `ranges` (`start`, exclusive `end`, `shard`). Returns the installed versioned snapshot. Installs metadata only and refuses changes to active slot assignments. See [Installed maps](../replication/residency.md#installed-maps). + +For a legacy conversion, call with `database`, `existing_owner: true` and `expected_digest` instead. This records the operator's assertion that this local owning node has complete bodies for its slots; Harper does not verify completeness. It requires an already-installed map with exactly that digest, rejects `slots` or `ranges` in the same call, and does not declare a sharded table. The receipt is local to the captured root generations and tables. Assert on each owning node, with legacy writers paused until activation is ready throughout the cluster. See [Converting an existing table](../replication/residency.md#convert-an-existing-by-id-table-without-moving-records). + +### `verify_shard_map` + + + +POST; requires `super_user`. Takes `database`, `table` and a proposed key-derived `policy` (`slots`). The existing application worker must still run its legacy function, and a custom map must be installed. Returns `checked`, `mismatches`, bounded samples, observed slots, departed node names (stored legacy owners absent from the current known-node list) and map identity. It neither rewrites records nor activates the policy. See [Compare placement](../replication/residency.md#3-compare-placement-on-every-node). + ### `cluster_status` -Returns connection state for all cluster nodes, including per-database socket status and replication timing statistics (`lastCommitConfirmed`, `lastReceivedRemoteTime`, `lastReceivedLocalTime`). +Returns connection state for all cluster nodes, including per-database socket status and replication timing statistics (`lastCommitConfirmed`, `lastReceivedRemoteTime`, `lastReceivedLocalTime`). Its top-level `sharding` object reports per-database map identity and placement blockers; see [Sharding Status](../replication/clustering.md#sharding-status). ```json { "operation": "cluster_status" } diff --git a/reference/replication/clustering.md b/reference/replication/clustering.md index 29c8955bd..f35142dfd 100644 --- a/reference/replication/clustering.md +++ b/reference/replication/clustering.md @@ -184,7 +184,8 @@ Returns an array of status objects from the cluster, including active WebSocket } ], "node_name": "server-1.domain.com", - "is_enabled": true + "is_enabled": true, + "sharding": {} } ``` @@ -201,6 +202,31 @@ While a database is taking its initial full copy, these timing fields hold the l --- +### Sharding Status + + + +The top-level `cluster_status.sharding` field is keyed by database and describes active or attempted declarative tables. It is empty in the legacy-table example above. Each entry has: + +| Field | Meaning | +| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `version`, `digest` | Installed map identity; absent if no map is installed. | +| `shardsDigest` | Digest of the configured shard set, or `null` when no list is configured. | +| `blocked` | Whether a configuration/map error or a recovery phase other than `ready` blocks placement. | +| `state` | `quarantined` for a startup quarantine, `held` for a live configuration/map error, the current `recovery.phase` while recovery is incomplete, or `available`. | +| `restartRequired` | `true` for startup quarantine: repair the configuration and restart the full process. `false` does not mean recovery is complete. | +| `recovery` | Bootstrap/recovery phase and reason, including a missing existing-owner completeness assertion in `recovery.reason`. `origin-history-lost` refuses a reused name without its audit; restore a complete backup or rejoin with a fresh root under a new node name. | +| `waitingOrigins` | Origins waiting for a direct source or certified same-shard relay, with the missing source described. | +| `errors` | Reasons, such as differing peer lists, a local shard outside its list, a missing required list/map, or a refused peer map. | + +A clear status is not proof that an owner is connected; write admission also checks the required replication route. See [configuration agreement](./sharding.md#complete-shard-id-set) and [map recovery](./residency.md#installed-maps). + +### Shard map operations + +The POST operations [`initialize_shard_map`](./residency.md#installed-maps) and [`verify_shard_map`](./residency.md#3-compare-placement-on-every-node) require `super_user`. The former installs a snapshot without moving records; the latter compares the actual legacy function against a proposed key policy without modifying data. Their request examples, parameters and result fields are in the linked procedures. + +--- + ### Configure Cluster Bulk creates or resets subscriptions for any number of remote nodes. **Resets and replaces any existing clustering setup.** diff --git a/reference/replication/overview.md b/reference/replication/overview.md index 7c75aab2c..33d258f14 100644 --- a/reference/replication/overview.md +++ b/reference/replication/overview.md @@ -99,6 +99,8 @@ With dedicated replication threads: - On macOS and Windows, which lack `SO_REUSEPORT`, only one replication thread listens on the replication port; the others carry outbound subscriptions. - Changing the number of threads takes effect only on a full `restart` of Harper; `restart_service` does not start or stop replication threads. An unscoped `restart_service` does restart the existing replication threads, and deploying or dropping a component does not. +Schema-defined [sharding](./residency.md) runs on application, operations and dedicated replication threads; it is an alternative to the thread-local residency setters. + ## Securing Connections Harper supports PKI-based security and authorization for replication connections. Two authentication methods are supported: diff --git a/reference/replication/residency.md b/reference/replication/residency.md new file mode 100644 index 000000000..9616814d8 --- /dev/null +++ b/reference/replication/residency.md @@ -0,0 +1,240 @@ +--- +title: Schema-defined sharding +--- + +# Schema-defined sharding + +Schema-defined sharding requires RocksDB. `@sharded` and `initialize_shard_map` refuse LMDB databases; schema refusal fences only the affected database and keeps loading healthy databases in the component. The older per-record residency functions remain available on LMDB. An LMDB database created with an earlier prerelease sharding policy must be exported using that prerelease and recreated on RocksDB before reapplying the schema; removing the directive does not erase the persisted policy. + + + +`@sharded` fixes a table's placement policy in its schema. Every application, operations, and replication thread applies the same policy. The primary key selects a slot, the database's installed slot map selects a shard, and `replication.shard` identifies that shard's current nodes. + +Upgrade every schema and data writer before activation. Wire compatibility with earlier peers allows delivery and reads; it does not make an earlier writer enforce the policy. With an installed custom map, recovery-capable peers may omit `replication.shards` only when neither side configures a list. This configuration fallback does not waive the sharding-recovery capability requirement: an older peer can receive compatible records and serve reads, but its announcement holds sharded writes until it is upgraded or removed. + +## Configuration + +For a new declarative table with no installed map, configure the complete shard-id set on every node. A custom-map conversion does not require this list. Leave it unset consistently if it was not configured; differing shard-list digests still hold placement. All members must run the sharding initialization protocol before converted tables can resume writes. See [`replication.shards`](./sharding.md#complete-shard-id-set) for initialization and mismatch handling. + +```yaml +replication: + shard: 0 + shards: [0, 1] + databases: '*' +``` + +Give shard `1`'s nodes `shard: 1` and the same `shards` list. The list belongs to deployment configuration; application schemas contain slot policies, not cluster shard IDs. + +## Schema directive + +```graphql +type Item @table @sharded { + id: ID @primaryKey + category: String @indexed + body: String +} +``` + +`slots` is the optional database-wide count, an integer from `1` through `65536`. It adopts the installed map's count, or defaults to `1024`. An explicit count must equal the installed map's count. + +The slot is the unsigned FNV-1a 32-bit hash of the primary key's ordered-binary encoding, modulo that count. The offset basis is `2166136261` and multiplier `16777619`. Numeric `-0` and `0` use the same encoding; numeric `5` and string `"5"` are distinct inputs. Updating other fields cannot change the slot. + +Only primary-key hashing is supported. Schema declarations using `by` or `slotFrom` are refused. Attribute-derived placement and custom expressions await [concurrent-move design (#1178)](https://github.com/HarperFast/harper-pro/issues/1178). + +Declarative tables require [per-table audit logging](../database/transaction.md#enabling-the-transaction-log-per-table) and table replication. Harper refuses activation with `@table(audit: false)`, effective `logging.auditLog: false`, or `@table(replicate: false)`, and refuses disabling these table settings later. The replication route must reach an owner. + +The policy is immutable once activated; changing its count or removing the directive is refused. Redeploying the same policy is idempotent. `setResidency` and `setResidencyById` are refused on declarative tables. Existing tables can adopt primary-key hashing only with a map that preserves their placement. + +## Ownership, reads, and writes + +Owners store full records. With secondary indexes, other nodes keep index projections and receive invalidations. Reads fetch the full body from an owner. A non-owner can cache a temporary replica. Replicas retain the canonical record identity and do not become owners or count toward owner confirmation. + +A table without secondary indexes stores nothing on non-owners; reads resolve the owner from the key. Such tables do not cache replicas. + +`replicateTo` is ignored for placement on a declarative table; the installed slot map determines where full records are retained. A non-owner accepts a local write and sends the complete body through replication to the owners. On an indexed RocksDB table with positive `replicaExpiration`, it also retains the complete written body as a temporary local replica, allowing an immediate local read while delivery is pending. A newer invalidation replaces it, and the replica sweep expires it. With expiration zero, it retains only the index projection; an index-free key policy retains nothing. A replica is a cache, so crash recovery may restore a projection instead. Ingest-heavy non-owners should budget space for approximately their serialized bytes written per second multiplied by `replicaExpiration`, or set expiration to zero. File-backed audit bodies remain until audit retention permits reclamation, even after a replica expires; include that delivery history in the disk budget. Refetched cache files have their own replica lifetime. This requires a configured replication route to an owner. Cross-shard query fan-out and write forwarding are not provided by this directive. Application database branches do not support sharding because branches do not replicate. + +Local-only writes succeed without replication confirmation. They do not deliver a local audit body to owners. The node retains a writer replica, index projection, or no row according to its replica and index settings. + +Explicit `replicatedConfirmation` uses `replication.confirmationTimeout` (when the configuration key is unset, the greater of `900000` ms and `replication.blobTimeout`). A failure after the local commit does not roll the write back; see [Confirmation Failure After a Local Commit](./sharding.md#confirmation-failure-after-a-local-commit). + +### Subscription filters + +Non-owner invalidation events contain only indexed fields. A subscription `rowFilter` evaluates that event payload even if a complete replica exists locally. Predicates on unindexed fields can miss updates; a negated predicate can also pass because the field is missing. Use indexed fields for these predicates. Do not use an unindexed event predicate to decide whether a subscriber may learn a record's id or indexed fields. + +### Configuration failures and peer identity + +Invalid sharding configuration at startup quarantines the affected database before replay. Its reads and writes return `DATABASE_SHARDING_BLOCKED` with the configuration reason and `retryable: false`; replication does not subscribe or advance its cursor. New tables declared in that database are deferred until a repaired restart; healthy tables in the same application schema still load. Other databases and the operations API remain available. `cluster_status.sharding` reports the blocked database. Repair the configuration and restart the node to replay its retained tail. Repairing a system map or restarting an individual worker does not clear the quarantine. + +A disagreement detected after startup places the affected database in a live hold. Reads and writes, including those on its unsharded tables, return `DATABASE_SHARDING_HELD` with `retryable: true`. Other databases in the same schema continue serving. Repairing the configuration clears this hold on every worker without a worker restart. Each process observes the root file; after the 150 ms settling interval, affected replication connections reconnect automatically to refresh peer agreement. Wait for the held state to clear before retrying writes. In `cluster_status.sharding`, `state: held` identifies this recoverable condition; `state: quarantined` and `restartRequired: true` identify the startup quarantine described above. A rolling shard-list configuration change can hold every sharded database until all peers agree. If an environment variable such as `HARPER_SET_CONFIG` supplies the shard list, update that provisioning value too: a full process restart re-applies it and can undo a file-only repair. + +A peer's announced node name must match its authenticated identity or an alias explicitly bound by its configured route or node record. For a canonical hostname reached through an IP address, configure both identities: + +```yaml +replication: + routes: + - hostname: node-b + url: wss://192.0.2.20:9933 +``` + +The canonical name is used for topology and confirmations. An announcement cannot create an alias or claim another known node's identity. + +## Recovery sources + +An owning node receives a live stream for origin X directly from X or through a co-owner in its own shard. Failover selects a peer in the receiver's shard. If neither source is reachable, that origin waits without advancing its cursor; `cluster_status.sharding` identifies the waiting origin and the missing source. Non-owners may relay projections to other non-owners. + +Bootstrapping or making a full copy of an owning node requires a same-shard peer or a complete backup for its owned slots. A copy from any peer remains allowed for slots the receiver does not own. A fast clone owning zero slots can bootstrap from any peer. + +On first initialization, sharded writes return a retryable 503 until the configured members have connected and acknowledged the initial history state. Requests do not keep transactions open while initialization waits. A missing member is reported as waiting rather than treated as proof of an empty shard. Once initialized, ordinary operation does not require every member to remain connected. Replacement nodes learn the existing history state and recover a complete owned baseline before they can act as a complete recovery source. + +A fresh node may start its owned slots empty only when the installed map is the initial version and those slots have no write history. An empty current table is not sufficient: deleted records and lost storage do not erase history. Otherwise, if the entire shard is lost, restore a complete backup. Status reports the backup requirement instead of treating an incomplete copy as recovery. Maps are installed before declarative writes begin; conversion of existing data uses the paused operator-assertion procedure below. Later ownership changes require the [slot-move workflow](https://github.com/HarperFast/harper-pro/issues/1053), which copies from the old owners. + +### Recovering a lost node + +Choose one recovery route before restarting a node whose local data or audit history was lost: + +- **Keep the original node name:** restore a complete backup taken while the node was stopped after its last accepted write, preserving the database roots and their audit history. An earlier backup that omits accepted writes does not qualify for same-name recovery. Restore the matching catalog metadata with the audit; copying table rows alone is insufficient. Restart the node with its original configuration and confirm that `cluster_status.sharding..recovery.phase` becomes `ready` before admitting writes. +- **Recover from co-owner snapshots:** stop the failed node, provision a fresh data root with a **new `node.hostname`**, and configure the same `replication.shard` and shard list. Join it through the normal [`add_node` procedure](./clustering.md#add-node). Keep a complete same-shard peer available for the copy. Remove the failed name through [`remove_node`](./clustering.md#remove-node), including on any peers that were offline during removal, and check the new node's recovery status before admitting writes. + +A node reusing a name whose earlier origin history is known to peers, without the corresponding local history, refuses sharded writes and cannot serve as an owned-slot recovery source. Its recovery status reports `origin-history-lost` and names both remedies: restore a complete backup, or rejoin under a new node name. Waiting for its clock to advance does not repair the missing history. + +While this node lacks its frozen origin start, outgoing replication for the affected database is held, including ordinary tables sharing its stream. Ordinary local writes can still succeed; that response does not confirm delivery to another node. Stop all application writes to that database before recovery, preserve any unique ordinary-table writes accepted after the failure, and export or restore them before replacing the root. Other databases remain independent. + +Co-owners continue relaying the old name's retained audit entries after membership removal. Keep those co-owners and their audit history available until lagging peers have received the old writes; membership removal does not replace normal audit retention requirements. The new name's writes use a separate origin, so receiving them does not certify receipt of the old origin's pending writes. + +If an entire shard was lost, a new name alone cannot recover its bodies: restore a complete backup. Each name restored without its original audit needs the snapshot-only route above. A new storage root in an initialized database cannot borrow another root's origin history; recover missing history before enabling writes. + +Snapshot recovery also marks the database's record-lock lineage as recloned: it cannot prove continuity of its earlier local history, including for ordinary tables in the same database. A complete backup preserving the original history avoids that limitation. Future [origin incarnations](https://github.com/HarperFast/harper-pro/issues/1123#issuecomment-6414845915) will allow a wiped node to keep its hostname while originating under a new root generation; that is not supported in this release. + +Every member must support sharding recovery admission. An authenticated announcement from an older binary holds sharded writes, even after initialization; ordinary tables and system metadata remain available. This observation survives restart and matching map/schema messages cannot clear it. Upgrade the peer or remove it from the configured membership. An offline peer's unannounced downgrade cannot be detected until it reconnects. + +Arbitrary cross-shard relay and clone layouts for owners are unsupported in this release. [Complete-body recovery from origin audit logs](https://github.com/HarperFast/harper-pro/issues/1123) tracks broader topology support, including handling origins whose full audit history is no longer retained. + +## Installed maps + +Each database has one versioned snapshot in `system.hdb_residency_slots`. The snapshot contains `database`, `version`, `digest`, `slots`, and complete, nonoverlapping `ranges` with exclusive `end` values. Every sharded table in that database uses the same slot count. + +First activation requires a writable node; it may need to persist both the map and activation metadata. Activate the policy before switching a node to read-only mode. + +For a new declarative table with no map, first activation installs version `1` from the explicit count (or `1024` when omitted) and sorted `replication.shards`: slot `k` maps to `shards[k % shards.length]`. Equal inputs produce the same snapshot and digest on simultaneous installers. The installed snapshot is then authoritative. Changes to the configured shard list never regenerate it. + +`slots` is a database-wide value. An omitted count adopts the installed map's count. An explicit different count is refused with an error naming both counts. Once a sharded table is active, changing the count is refused because it would change hash placement. Separate maps per slot count are deferred: custom-map administration and future [slot moves (#1053)](https://github.com/HarperFast/harper-pro/issues/1053) would need to apply to every map. Do not run incompatible first schema deployments concurrently. + +An existing table requires a custom map before activation, even if this node currently holds no rows. A custom map can be installed without `replication.shards`: + +```json +{ + "operation": "initialize_shard_map", + "database": "data", + "slots": 10, + "ranges": [ + { "start": 0, "end": 5, "shard": 0 }, + { "start": 5, "end": 10, "shard": 1 } + ] +} +``` + +This POST operation requires `super_user` and returns the installed snapshot. It changes metadata only. An exact retry is a no-op. Once a declarative table is active, neither existing assignments nor the database slot count can change. Same-version divergent snapshots are refused rather than resolved by last-write-wins. Deleting an installed map is refused. If its stored snapshot is lost or corrupted, Harper does not reconstruct it from configuration. + +Setup exchanges carry the map version/digest and a separate shard-list digest. A missing or conflicting map blocks dependent placement and replication progress and appears in `cluster_status.sharding`. A conflict on the system stream can also hold later system updates. Repair the conflicting configuration/map and reconnect; do not assume that skipping the offending record is safe. Resolving independently initialized, differently placed databases may require restoring or exporting the losing data. Reassignments that move data are a separate feature. + +There is no operation to delete or reset an active map. If an accidentally initialized joining node has no unique data, stop its traffic and reprovision it with a fresh data directory; receive or install the cluster's exact snapshot before deploying the declarative application. Do not delete individual system-table files. If it has unique data, preserve a backup and restore/export that data before reprovisioning. In custom-map deployments, leaving `replication.shards` unset on every participant prevents premature default installation; lists must still agree across peers. + +## Limitations + +Attribute-derived sharding and custom slot expressions are unavailable pending [concurrent-move design (#1178)](https://github.com/HarperFast/harper-pro/issues/1178). Slot migration is separate work in [#1053](https://github.com/HarperFast/harper-pro/issues/1053). + +Owner recovery requires the [sources described above](#recovery-sources). Arbitrary cross-shard owner recovery is deferred to [#1123](https://github.com/HarperFast/harper-pro/issues/1123). + +## Older per-record residency APIs + +`setResidency`, `setResidencyById`, `getResidency`, and residency lists retain their names and remain the older per-record mechanism. Use the conversion procedure below when adopting schema-defined sharding for an existing by-id table. + +## Convert an existing by-id table without moving records + +Check eligibility before installing a map: the legacy function must agree with the primary-key hash and proposed ranges for every record. Last-digit or modulo recipes are not generally equivalent to this hash. Compare the functions offline first; custom `slotFrom` policies remain unsupported pending [#1178](https://github.com/HarperFast/harper-pro/issues/1178). An installed map cannot be deleted. + +### 1. Prepare the existing table + +Back up the database and upgrade all writers. Keep the current `setResidencyById` function running during the comparison. + +### 2. Install the equivalent map + +The existing function must agree with the supported primary-key hash and the proposed map for every stored record. Arbitrary legacy expressions cannot be adopted yet. Install the equivalent map from one designated node. Wait for the same snapshot on every participant before enabling the directive. On a newly joining node in a custom-map deployment, install or receive that snapshot before deploying its declarative application; otherwise first activation would initialize a default map from the local list. + +### 3. Compare placement on every node + +Run `verify_shard_map` on each node while its application worker still runs the old function: + +```json +{ + "operation": "verify_shard_map", + "database": "data", + "table": "Item", + "policy": { "slots": 10 } +} +``` + +This POST operation requires `super_user`. It scans local records and compares the actual old function's numeric shard with the proposed policy and installed map. It reports `checked`, `mismatches`, bounded mismatch samples, observed slots, departed node names, and map identity. It does not rewrite records or activate the policy. Hostname-array functions are not converted: the first nonnumeric return rejects the operation with HTTP 400, rather than counting as a mismatch or inferring a shard. `departedNodeNames` lists names found in stored legacy residency lists that are absent from the node's current known-node list (including itself). A sample of present keys cannot prove equivalence for every future key; the operator must confirm the functions are equivalent. + +### 4. Assert each existing owner's completeness + +Require zero mismatches and matching maps on every participant. Placement comparison examines the records present on a node; it does not prove that the node holds every record it should own. The operator must establish that completeness independently. + +Pause legacy writes across the cluster before asserting completeness, and keep them paused through activation. On each existing owner, submit the digest returned by map installation: + +```json +{ + "operation": "initialize_shard_map", + "database": "data", + "existing_owner": true, + "expected_digest": "" +} +``` + +This is an **operator assertion of completeness**, not something Harper verifies. It is bound to every captured local database root generation, its existing tables and the expected map digest. A partial set of receipts is uncertified, and a later legacy table conversion needs a new assertion. A digest mismatch is refused without storing an assertion. The operation does not declare a sharded table, and `verify_shard_map` remains read-only. + +Without the assertion, a converted legacy root stays uncertified. It may serve and copy nonowned data, but it cannot supply recovery for owned slots; `cluster_status.sharding` names the missing assertion. A replaced root does not inherit an assertion from replicated schema metadata. An uncertified node may still return a locally present body; completeness gating cannot detect that such a body is stale. It must not report an owned miss as authoritative absence or act as an owned recovery source. + +### 5. Activate the directive + +Deploy the directive and remove the setter. Wait for sharding initialization to complete on the owners before resuming writes. A member still running the previous binary cannot acknowledge initialization: writes remain refused and status names the missing member until it is upgraded. Writes can continue during the earlier placement comparison; the completeness assertion and activation form the paused cutover. + +Existing residency IDs remain readable but no longer decide placement. Key-derived records receive no new slot metadata. A stale component that calls a forbidden setter after activation fails visibly and must be updated. + +### Convert an existing unsharded table + +An existing table without a residency function can also gain `@sharded`. Pause writes, install the desired custom map, and establish that every future owner already has all the records assigned to it. For a previously fully replicated table, check that every future owner has caught up before proceeding. Use the same `initialize_shard_map` request with `existing_owner: true` and the exact `expected_digest` on each owner, then deploy the key-derived directive and wait for initialization before resuming writes. + +This is the same operator assertion of completeness as the by-id procedure; Harper does not verify that every required record is present. `verify_shard_map` compares a legacy by-id function and does not replace this assertion for an unsharded table. A mismatched digest stores no assertion. Without the assertion, status names the missing completeness assertion and the node cannot supply owned-slot recovery. + +Key-derived conversion retains existing non-owner full copies locally; it does not relocate, purge, rewrite, or change the versions of existing rows. Those copies do not confer ownership or qualify the node as an owned-slot recovery source. Index-free reads load from the owner rather than trusting a retained non-owner body; indexed tables receive ordinary invalidations. Bootstrap may add missing non-owner index projections. Subsequent writes follow the new replica/projection placement rules. Conversion alone therefore does not remove previously replicated bodies from a region; account for those copies when planning a data-residency cutover. + +### Rollback + +The in-place procedure below applies to key-derived tables converted from an equivalent `setResidencyById` policy. It does not restore full replication for a table that had no residency function: removing `@sharded` cannot backfill full bodies on former non-owners. For that conversion, restore a complete preactivation backup instead. Preserve and reconcile writes accepted after the backup before restoring it, or those writes can be lost. + +For a key-derived policy, drain cached RocksDB replicas before changing binaries: an earlier binary can interpret one as a full owned record. Key policies themselves need no slot-format conversion. + +1. Back up the database. Pause application requests and background jobs on **every node**, including reads that could admit another replica. Keep them paused through the downgrade. +2. Set `@table(replicaExpiration: 0)` in each affected table's schema and restart every node on the **current** binary, preserving its sharding directive. This table argument is in seconds; zero disables new replica admissions. It does not rewrite existing replica deadlines, so waiting for a new, shorter lifetime does not establish a drain. +3. Run the following trusted maintenance code for each affected table on every node. Run it inside the application worker with access to `tables`; do not expose it as a public endpoint. This procedure is for the 5.4 implementation and uses internal primary metadata (`REPLICA = 4`); verify the matching implementation before using it with another release. It evicts only cached replicas, preserving owned rows and legacy-compatible invalidated index projections. The runtime TTL call below repeats step 2 defensively on this table handle and preserves its other TTL settings. These APIs are trusted worker internals, not Operations API commands. + +```js +async function drainResidencyReplicas(Table) { + const REPLICA = 4; + Table.setTTLExpiration({ replicaExpiration: 0 }); + for (const entry of Table.primaryStore.getRange({ start: true })) { + if (entry.metadataFlags & REPLICA) await Table.evict(entry.key, entry.value, entry.version); + } + let remaining = 0; + for (const entry of Table.primaryStore.getRange({ start: true })) if (entry.metadataFlags & REPLICA) remaining++; + if (remaining) throw new Error(`${remaining} replicas remain; do not downgrade`); + return { remaining }; +} +await drainResidencyReplicas(tables.Item); +``` + +4. Require `{ remaining: 0 }` for every affected table on every node. Eviction is version-checked; a busy replica can survive a pass, so a nonzero result refuses the downgrade. Do not use ordinary record counts as proof: projections and replicas share keys. +5. Remove the maintenance code, restore the previous binary with the equivalent legacy schema and `setResidencyById` function, then resume traffic. Keep the installed slot map for a later upgrade. Local assertion receipts also survive restart and downgrade: their trust remains the operator's responsibility while the equivalent legacy placement is in use, not a new verification by Harper on re-upgrade. + +There is no built-in replica-drain operation. If you cannot run and verify this maintenance procedure, restore a preactivation backup instead of attempting an in-place downgrade. diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index b1e382fa7..3c97ada26 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -12,15 +12,19 @@ title: Sharding — expanded sharding functionality: Harper now honors write requests with residency information that will not be stored on the local node, and nodes can be declaratively configured as part of a shard. -Harper's replication system supports sharding — storing different data across different subsets of nodes — while still allowing data to be accessed from any node in the cluster. This enables horizontal scalability for storage and write performance, while maintaining optimal data locality and consistency. +Harper's replication system supports sharding — storing different data across different subsets of nodes. Single-key reads can fetch records from their owners across shards. Sharding distributes storage and write work across nodes. -When sharding is configured, requests for records that don't reside on the handling node are automatically forwarded to the appropriate node transparently. Clients do not need to know where data is stored. +A key read for a record not held locally can fetch its body from an owner. This does not provide cross-shard query fan-out or write forwarding. With [schema-defined sharding](./residency.md#ownership-reads-and-writes), a non-owner write commits locally and replicates the complete body to its owners through the configured replication routes. By default (without sharding), Harper replicates all data to all nodes. -## Approaches to Sharding +## Schema-defined sharding -There are two main approaches: +In v5.4, [`@sharded`](./residency.md) hashes primary keys to fixed slots on every worker. Slot counts and the installed slot map belong to the database. Attribute-derived policies and custom expressions are refused pending [concurrent-move design (#1178)](https://github.com/HarperFast/harper-pro/issues/1178). + +## Older per-record mechanisms + +The `setResidency`, `setResidencyById`, `getResidency` and residency-list APIs below retain their names. They are the older mechanism, with two approaches: **Dynamic sharding** — the location (residency) of records is determined dynamically based on where the record was written, the record's data, or a custom function. Records can be relocated dynamically based on where they are accessed. Residency information is specific to each record. @@ -28,6 +32,8 @@ There are two main approaches: ## Dynamic Sharding +The configuration, REST, Operations API and programmatic `replicateTo` examples below control legacy residency. On `@sharded` tables, `replicateTo` does not override the installed map: only the map determines owners. See [ownership, reads, and writes](./residency.md#ownership-reads-and-writes). + ### Replication Count The simplest way to limit replication is to configure a replication count. Set `replicateTo` in the `replication` section of `harper-config.yaml` to specify how many additional nodes data should be replicated to: @@ -111,8 +117,8 @@ export class Order extends tables.Order { ```javascript import { tables, transaction } from 'harper'; -await transaction({ replicatedConfirmation: 1, replicateTo: 2 }, async () => { - await tables.Order.put(order); +await transaction({ replicatedConfirmation: 1, replicateTo: 2 }, async (txn) => { + await tables.Order.put(order, txn); }); ``` @@ -120,6 +126,14 @@ That transaction commits on its own, separately from any request transaction tha Values your code sets on a context apply for every caller, including users without the `super_user` permission. Derive them from application logic, not from request input such as headers, query parameters, or the record body: copying a client-supplied value into `replicateTo` or `replicatedConfirmation` gives that client the control the `super_user` check removes. +On `@sharded` tables, `replicateTo` does not change placement; the database map determines owners. Explicit `replicatedConfirmation` counts acknowledgments from other owners for each affected owner set in the first written database. A local temporary replica cannot satisfy it. See [ownership, reads, and writes](./residency.md#ownership-reads-and-writes). + +### Confirmation outcomes + + + +LMDB now waits for requested peer confirmation, as RocksDB does. This can add peer latency to existing LMDB applications that use `confirm` or `replicatedConfirmation`. Explicit confirmation still applies only to the first database written by the transaction. + ### Confirmation Failure After a Local Commit @@ -140,7 +154,7 @@ For example, a REST error for a confirmation timeout can contain: } ``` -The existing error status, code, and message remain available. An operations error has its usual `error` field plus `code`, `replicationUnconfirmed: true`, and `retryable: false`. Programmatic errors wrap the original rejection in `cause`; text-only protocol errors use the new `ReplicationConfirmationError` class name. Check this flag before compensating or retrying a non-idempotent write, because replaying the request could repeat an insert or apply an update twice. The flag does not return auto-generated record keys; use client-selected identifiers when you need to inspect the written records after a failed response. +The existing error status, code, and message remain available. An operations error has its usual `error` field plus `code`, `replicationUnconfirmed: true`, and `retryable: false`. Programmatic errors wrap the original rejection in `cause`; text-only protocol errors use the new `ReplicationConfirmationError` class name. Check this flag before compensating or retrying a non-idempotent write, because replaying the request could repeat an insert or apply an update twice. The flag does not return auto-generated record keys; use client-selected identifiers and apply the application's concurrency controls when you need to inspect the written records after a failed response. The flag covers writes issued to the reported commit, not later application steps that were never reached after its rejection. If an enclosing scope rolls back its pending writes, its error omits the flag while preserving `retryable: false` because an independent inner write may already have committed. @@ -159,7 +173,29 @@ replication: shard: 1 ``` -Or assign shards per route: +### Complete shard-id set + + + +For the first default map of a new [schema-defined sharding](./residency.md) table, configure `replication.shards` next to `replication.shard`: + +```yaml +replication: + shard: 0 + shards: [0, 1] +``` + +`replication.shards` is the complete set of shard IDs available to the deployment: unique integers from `0` through `65535`. Every node uses the same set; order does not matter. Its own `replication.shard` must be in the list. + +The list is required only for a database that has declarative tables and no installed slot map. First activation computes the default map solely from the slot count and the sorted list and installs it as a versioned snapshot. Concurrent installers with equal inputs produce the same snapshot and digest. A database converted with a custom map does not need the list. + +Once installed, the snapshot is authoritative. Changing the list never recomputes placement. Adding an ID only makes that shard available for future reassignment; it moves no records. + +Setup exchanges include a digest of the list. Nodes with different lists refuse declarative writes on both sides, even after a map is installed. A configured list on one peer and no list on another also differs. A node whose own shard is outside its list, or which needs an initial map but has no list, also refuses placement. The condition appears in `cluster_status.sharding` and is logged once. Plan for this write refusal during a rolling list change. + +### Other shard configuration + +Shards can also be assigned per route: ```yaml replication: @@ -180,7 +216,7 @@ Or dynamically via the operations API by including `shard` in an `add_node` or ` } ``` -Once shards are configured, use `setResidency` or `setResidencyById` (described below) to assign records to specific shards. +Once shards are configured, use [schema-defined sharding](./residency.md) for a schema policy applied on every thread, or the function APIs below for legacy application-controlled placement. ## Custom Sharding @@ -234,7 +270,7 @@ MyTable.setResidencyById((id) => { ## Disabling Cross-Node Access -By default, sharding allows data stored on specific nodes to be accessed from any node — requests are forwarded transparently. To disable this and only return data stored on the local node (including a replica it currently holds), set `replicateFrom` to `false`. +By default, a key read for a record not stored locally fetches it from a node that holds it. To disable this and only return data stored on the local node (including a replica it currently holds), set `replicateFrom` to `false`. Via the operations API: @@ -256,5 +292,6 @@ X-Replicate-From: none ## See Also +- [Schema-defined sharding](./residency.md) — Fixed slot policies and authoritative database maps - [Replication Overview](./overview.md) — How Harper's replication system works - [Clustering Operations](./clustering.md) — Operations API for managing cluster nodes diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 7bc962206..9e115f41d 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -247,9 +247,11 @@ All properties are optional: | `includeSuperseded` | Include record mutations superseded by a later version. Defaults to `false`, or `true` when `rawEvents` is enabled. An explicit value overrides that default. | | `rawEvents` | Return raw audit events rather than reconstructing full records. Defaults to `false`. | | `omitCurrent` | Do not send the current/retained record as the first update. | -| `rowFilter` | Synchronous JavaScript predicate applied to authoritative row values. | +| `rowFilter` | Synchronous JavaScript predicate applied to row payloads; sharded non-owner invalidations contain only indexed projections. | | `eventFilter` | Synchronous JavaScript predicate for events that may not carry an authoritative row. | +On a sharded non-owner, `rowFilter` sees indexed fields from the invalidation payload even when a full replica is cached locally. Use indexed fields in these predicates: missing unindexed fields can suppress matching updates or satisfy a negated predicate. Do not use such a predicate to decide whether a subscriber may learn a record's id or indexed fields. See [subscription filters](../replication/residency.md#subscription-filters). + #### Superseded record updates diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 8216b8f45..99de04070 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -22,6 +22,20 @@ These corrections are not released yet. ## Replication +### Sharding + +`@sharded` hashes primary keys to fixed slots on every thread, including dedicated replication workers. Per-database slot maps remain authoritative after initialization; `replication.shards` supplies the deployment shard set for a first default map. Conversion checks support equivalent legacy by-id policies without rewriting records. Attribute-derived policies and custom expressions are refused pending [concurrent-move design (#1178)](https://github.com/HarperFast/harper-pro/issues/1178). See [Schema-defined sharding](/reference/v5/replication/residency). + +Sharding configuration failures are isolated to the affected database. A live disagreement holds that database until the configuration is repaired on all peers; healthy databases in the same application keep serving, including during worker replacement. Invalid configuration at process startup still requires a repaired full restart because its audit replay was deferred. First-time sharding initialization waits for the configured members; owner recovery requires a certified same-shard source or a complete backup. See [Configuration failures and recovery sources](/reference/v5/replication/residency#configuration-failures-and-peer-identity). + +Co-owner snapshot recovery does not restore the replacement's former local audit lineage. Cluster record-lock freshness treats the recovered database as recloned, including its ordinary tables; use a complete backup to preserve that history. See [Recovery sources](/reference/v5/replication/residency#recovery-sources). + +Snapshot-only recovery of a sharded node requires a fresh root and a new node name. Reusing the original name requires a complete stopped-node backup preserving audit through the last accepted write. A reused name missing its origin history refuses sharded writes and recovery-source eligibility, with both remedies in cluster status. See [sharding recovery](/reference/v5/replication/residency#recovering-a-lost-node). + +### LMDB Replication Confirmation + +LMDB writes that request `replicatedConfirmation` now wait for peer acknowledgments. Applications that previously received immediate success can now observe replication latency or an uncertain committed outcome if confirmation fails. See [Confirmation Failure After a Local Commit](/reference/v5/replication/sharding#confirmation-failure-after-a-local-commit) before retrying a failed write. + ### Dedicated Replication Threads `replication.threads` (default `0`) starts worker threads that run replication only. They own the replication port and every subscription, so a peer catching up no longer competes with application requests. These threads load no application code, which limits applications that use `setResidencyById` or `setComputedAttribute` on indexed attributes. See [Dedicated Replication Threads](/reference/v5/replication/overview#dedicated-replication-threads). diff --git a/sidebarsReference.ts b/sidebarsReference.ts index 8ce7f8104..da02d4efa 100644 --- a/sidebarsReference.ts +++ b/sidebarsReference.ts @@ -637,7 +637,12 @@ const sidebars: SidebarsConfig = { { type: 'doc', id: 'replication/sharding', - label: 'Sharding', + label: 'Sharding and legacy APIs', + }, + { + type: 'doc', + id: 'replication/residency', + label: 'Schema-defined sharding', }, ], },