From 63bd54ab39fb1f0558cfba24724513b214216195 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 23 Oct 2026 11:30:54 -0600 Subject: [PATCH 01/32] Document declarative slot residency and conversion Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/replication/residency.md | 129 +++++++++++++++++++++++++++++ reference/replication/sharding.md | 24 +++++- sidebarsReference.ts | 5 ++ 3 files changed, 157 insertions(+), 1 deletion(-) create mode 100644 reference/replication/residency.md diff --git a/reference/replication/residency.md b/reference/replication/residency.md new file mode 100644 index 000000000..bb2f4135c --- /dev/null +++ b/reference/replication/residency.md @@ -0,0 +1,129 @@ +--- +title: Declarative residency +--- + +# Declarative residency + + + +`@residency` fixes a table's placement policy in its schema. Every application, operations, and replication thread applies the same policy. A key or record selects a slot, the database's installed slot map selects a shard, and `replication.shard` identifies that shard's current nodes. + +Upgrade every schema and data writer before activation. Wire compatibility with earlier peers allows delivery and reads; it does not make an earlier writer enforce the policy. + +## Configuration + +Configure the complete shard-id set on every node. See [`replication.shards`](./sharding.md#complete-shard-id-set) for initialization and mismatch handling. + +```yaml +replication: + shard: 0 + shards: [0, 1] + databases: '*' +``` + +Give shard `1`'s nodes `shard: 1` and the same `shards` list. The list belongs to deployment configuration; application schemas contain slot policies, not cluster shard IDs. + +## Schema directive + +```graphql +type Item @table @residency(slots: 1024) { + id: ID @primaryKey + category: String @indexed + body: String +} +``` + +| Argument | Meaning | +| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `slots` | Required integer from `1` through `65536`. | +| `slotFrom` | Optional JavaScript expression returning an integer from `0` through `slots - 1`. Key policies expose `id`. | +| `byAttribute` | Default `false`. When `true`, the expression also exposes `record` and its attribute names, and the slot is stored with the record. Requires a secondary index. | + +Without `slotFrom`, the slot is the unsigned FNV-1a 32-bit hash of the key's ordered-binary encoding, modulo `slots`. Its offset basis is `2166136261` and its multiplier is `16777619`. + +An explicit key policy can preserve an existing partition function: + +```graphql +type Item @table @residency(slots: 10, slotFrom: "Number(id.slice(-1))") { + id: ID @primaryKey + category: String @indexed + body: String +} +``` + +Expressions must be self-contained, deterministic, and synchronous. They compile on every thread without application helpers. A non-integer or out-of-range result refuses the write. Schema expressions are trusted code, as with computed expressions; they are not a sandbox. + +The policy is immutable once activated, including removal of the directive. Redeploying the same policy is idempotent. Calling `setResidency` or `setResidencyById` on a declarative table fails; remove those calls when converting. + +## Ownership, reads, and writes + +Owners store full records. With secondary indexes, other nodes keep index projections and receive invalidations. Reads can fetch the full body from an owner and cache a temporary replica. Replicas retain the canonical record identity and do not become owners or count toward owner confirmation. + +A key-policy table without secondary indexes stores nothing on non-owners; reads resolve the owner from the key. Such tables do not cache replicas. Attribute policies require a secondary index so a non-owner can retain the stored slot. + +`replicateTo` selects delivery targets; it cannot change declarative ownership. A non-owner write requires a configured replication route to an owner. Cross-shard query fan-out and write forwarding are not provided by this directive. + +### Attribute-slot changes + +```graphql +type RegionalItem @table @residency(slots: 10, slotFrom: "region", byAttribute: true) { + id: ID @primaryKey + region: Int @indexed + body: String +} +``` + +Changing `region` sends the complete canonical body to the new owners and invalidates old owners at the same version. When the writer is outside the new owner set, success requires at least one new owner to confirm durable receipt. If no new owner is connected, the change is refused before commit. + +A connection can fail after admission. A confirmation failure after local commit means the outcome is uncertain; it does not mean the transaction rolled back. Inspect the record and retry with the application's concurrency controls. Audit retention bounds replay for disconnected peers; it is not an indefinite handoff queue. + +## Installed maps + +Each database has one versioned snapshot in `system.hdb_residency_slots`. The snapshot contains `database`, `version`, `digest`, `slots`, and complete, nonoverlapping `ranges` with exclusive `end` values. Tables may use different covered prefixes of the map. + +For a new declarative table with no map, first activation installs version `1` from only the declared slot count and sorted `replication.shards`: slot `k` maps to `shards[k % shards.length]`. Equal inputs produce the same snapshot and digest on simultaneous installers. The installed snapshot is then authoritative. Changes to the configured shard list never regenerate it. + +An existing table requires a custom map before activation, even if this node currently holds no rows. A custom map can be installed without `replication.shards`: + +```json +{ + "operation": "initialize_residency_slots", + "database": "data", + "slots": 10, + "ranges": [ + { "start": 0, "end": 5, "shard": 0 }, + { "start": 5, "end": 10, "shard": 1 } + ] +} +``` + +This POST operation requires `super_user`. It changes metadata only. An exact retry is a no-op. Once a declarative table is active, existing assignments cannot change; coverage can only extend without changing active slots. Same-version divergent snapshots are refused rather than resolved by last-write-wins. Deleting the installed map does not trigger default reconstruction. + +Setup exchanges carry the map version/digest and a separate shard-list digest. A missing or conflicting map blocks dependent placement and replication progress and appears in `cluster_status.residency`. A conflict on the system stream can also hold later system updates. Repair the conflicting configuration/map and reconnect; do not assume that skipping the offending record is safe. Resolving independently initialized, differently placed databases may require restoring or exporting the losing data. Reassignments that move data are a separate feature. + +## Convert an existing by-id table without moving records + +1. Back up the database and upgrade all writers. Keep the current `setResidencyById` function running during the comparison. +2. Install a custom map that reproduces the function's numeric shard result exactly, from one designated node. Wait for the same snapshot on every participant. +3. Run `verify_residency_slots` on each node while its application worker still runs the old function: + + ```json + { + "operation": "verify_residency_slots", + "database": "data", + "table": "Item", + "policy": { "slots": 10, "slotFrom": "Number(id.slice(-1))" } + } + ``` + + This POST operation requires `super_user`. It scans local records and compares the actual old function's numeric shard with the proposed policy and installed map. It reports `checked`, `mismatches`, bounded mismatch samples, observed slots, departed node names, and map identity. It does not rewrite records or activate the policy. A sample of present keys cannot prove equivalence for every future key; the operator must confirm the functions are equivalent. + +4. Require zero mismatches and matching maps on every participant. Deploy the directive and remove the setter. The existing deployment can keep writing during the comparison; this is not a quiescing migration operation. + +Existing residency IDs remain readable but no longer decide placement. Key-derived records receive no new slot metadata. A stale component that calls a forbidden setter after activation fails visibly and must be updated. + +### Rollback + +For a key-derived policy, restore the previous binary together with the equivalent legacy schema and `setResidencyById` function. No record conversion is needed. Keep the installed map for a later upgrade. + +An attribute policy writes a new local primary/audit format. In-place downgrade to an earlier binary is unsupported. Restore a backup from before activation; writes since that backup may be lost. Protocol fallback does not make an earlier binary understand the new local storage format. diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index b1e382fa7..6d4bc1ee1 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -159,6 +159,28 @@ replication: shard: 1 ``` +### Complete shard-id set + + + +For [declarative residency](./residency.md), configure `replication.shards` next to `replication.shard`: + +```yaml +replication: + shard: 0 + shards: [0, 1] +``` + +`replication.shards` is the complete set of shard IDs available to the deployment: unique integers from `0` through `65535`. Every node uses the same set; order does not matter. Its own `replication.shard` must be in the list. + +The list is required only for a database that has declarative tables and no installed slot map. First activation computes the default map solely from the slot count and the sorted list and installs it as a versioned snapshot. Concurrent installers with equal inputs produce the same snapshot and digest. A database converted with a custom map does not need the list. + +Once installed, the snapshot is authoritative. Changing the list never recomputes placement. Adding an ID only makes that shard available for future reassignment; it moves no records. + +Setup exchanges include a digest of the list. Nodes with different lists refuse declarative writes on both sides, even after a map is installed. A configured list on one peer and no list on another also differs. A node whose own shard is outside its list, or which needs an initial map but has no list, also refuses placement. The condition appears in `cluster_status.residency` and is logged once. Plan for this write refusal during a rolling list change. + +### Other shard configuration + Or assign shards per route: ```yaml @@ -180,7 +202,7 @@ Or dynamically via the operations API by including `shard` in an `add_node` or ` } ``` -Once shards are configured, use `setResidency` or `setResidencyById` (described below) to assign records to specific shards. +Once shards are configured, use [declarative residency](./residency.md) for a schema policy applied on every thread, or the function APIs below for legacy application-controlled placement. ## Custom Sharding diff --git a/sidebarsReference.ts b/sidebarsReference.ts index 8ce7f8104..88ab6cbd1 100644 --- a/sidebarsReference.ts +++ b/sidebarsReference.ts @@ -639,6 +639,11 @@ const sidebars: SidebarsConfig = { id: 'replication/sharding', label: 'Sharding', }, + { + type: 'doc', + id: 'replication/residency', + label: 'Declarative residency', + }, ], }, { From bc09e2519ff83e408e4688d49b12f281e276f8a3 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 23 Oct 2026 11:46:21 -0600 Subject: [PATCH 02/32] docs: state application branch residency limitation Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/replication/residency.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/replication/residency.md b/reference/replication/residency.md index bb2f4135c..e8f34d8fa 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -61,7 +61,7 @@ Owners store full records. With secondary indexes, other nodes keep index projec A key-policy table without secondary indexes stores nothing on non-owners; reads resolve the owner from the key. Such tables do not cache replicas. Attribute policies require a secondary index so a non-owner can retain the stored slot. -`replicateTo` selects delivery targets; it cannot change declarative ownership. A non-owner write requires a configured replication route to an owner. Cross-shard query fan-out and write forwarding are not provided by this directive. +`replicateTo` selects delivery targets; it cannot change declarative ownership. A non-owner write requires a configured replication route to an owner. Cross-shard query fan-out and write forwarding are not provided by this directive. Application database branches do not support declarative residency because branches do not replicate. ### Attribute-slot changes From 6285d39fb0594f10f2167cdaa96e479c5cca4a1f Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 23 Oct 2026 13:13:00 -0600 Subject: [PATCH 03/32] docs: make conversion steps and example independently readable Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/replication/residency.md | 42 ++++++++++++++++++------------ 1 file changed, 26 insertions(+), 16 deletions(-) diff --git a/reference/replication/residency.md b/reference/replication/residency.md index e8f34d8fa..897531218 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -103,22 +103,32 @@ Setup exchanges carry the map version/digest and a separate shard-list digest. A ## Convert an existing by-id table without moving records -1. Back up the database and upgrade all writers. Keep the current `setResidencyById` function running during the comparison. -2. Install a custom map that reproduces the function's numeric shard result exactly, from one designated node. Wait for the same snapshot on every participant. -3. Run `verify_residency_slots` on each node while its application worker still runs the old function: - - ```json - { - "operation": "verify_residency_slots", - "database": "data", - "table": "Item", - "policy": { "slots": 10, "slotFrom": "Number(id.slice(-1))" } - } - ``` - - This POST operation requires `super_user`. It scans local records and compares the actual old function's numeric shard with the proposed policy and installed map. It reports `checked`, `mismatches`, bounded mismatch samples, observed slots, departed node names, and map identity. It does not rewrite records or activate the policy. A sample of present keys cannot prove equivalence for every future key; the operator must confirm the functions are equivalent. - -4. Require zero mismatches and matching maps on every participant. Deploy the directive and remove the setter. The existing deployment can keep writing during the comparison; this is not a quiescing migration operation. +### 1. Prepare the existing table + +Back up the database and upgrade all writers. Keep the current `setResidencyById` function running during the comparison. + +### 2. Install the equivalent map + +Install a custom map that reproduces the function's numeric shard result exactly, from one designated node. Wait for the same snapshot on every participant. + +### 3. Compare placement on every node + +Run `verify_residency_slots` on each node while its application worker still runs the old function: + +```json +{ + "operation": "verify_residency_slots", + "database": "data", + "table": "Item", + "policy": { "slots": 10, "slotFrom": "Number(id.slice(-1))" } +} +``` + +This POST operation requires `super_user`. It scans local records and compares the actual old function's numeric shard with the proposed policy and installed map. It reports `checked`, `mismatches`, bounded mismatch samples, observed slots, departed node names, and map identity. It does not rewrite records or activate the policy. A sample of present keys cannot prove equivalence for every future key; the operator must confirm the functions are equivalent. + +### 4. Activate the directive + +Require zero mismatches and matching maps on every participant. Deploy the directive and remove the setter. The existing deployment can keep writing during the comparison; this is not a quiescing migration operation. Existing residency IDs remain readable but no longer decide placement. Key-derived records receive no new slot metadata. A stale component that calls a forbidden setter after activation fails visibly and must be updated. From b9326e417afc3a9c8f10e25c8cd6eec55abaf421 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 23 Oct 2026 13:21:59 -0600 Subject: [PATCH 04/32] docs: limit online residency adoption to key policies Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/replication/residency.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/replication/residency.md b/reference/replication/residency.md index 897531218..b35bfed7b 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -53,7 +53,7 @@ type Item @table @residency(slots: 10, slotFrom: "Number(id.slice(-1))") { Expressions must be self-contained, deterministic, and synchronous. They compile on every thread without application helpers. A non-integer or out-of-range result refuses the write. Schema expressions are trusted code, as with computed expressions; they are not a sandbox. -The policy is immutable once activated, including removal of the directive. Redeploying the same policy is idempotent. Calling `setResidency` or `setResidencyById` on a declarative table fails; remove those calls when converting. +The policy is immutable once activated, including removal of the directive. Redeploying the same policy is idempotent. Calling `setResidency` or `setResidencyById` on a declarative table fails; remove those calls when converting. Existing tables can adopt a key-derived policy with an equivalent custom map. Attribute policies must be declared on a new table because existing rows have no stored slots. ## Ownership, reads, and writes From 6eef5137fe92a8d9a1705cd83b6394f35e20fb22 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 23 Oct 2026 14:55:41 -0600 Subject: [PATCH 05/32] docs: clarify shard agreement for legacy wire fallback Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/replication/residency.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/replication/residency.md b/reference/replication/residency.md index b35bfed7b..5a05283db 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -8,7 +8,7 @@ title: Declarative residency `@residency` fixes a table's placement policy in its schema. Every application, operations, and replication thread applies the same policy. A key or record selects a slot, the database's installed slot map selects a shard, and `replication.shard` identifies that shard's current nodes. -Upgrade every schema and data writer before activation. Wire compatibility with earlier peers allows delivery and reads; it does not make an earlier writer enforce the policy. +Upgrade every schema and data writer before activation. Wire compatibility with earlier peers allows delivery and reads; it does not make an earlier writer enforce the policy. Fallback to peers that cannot advertise a shard-list digest requires an installed custom map and no configured list on either side; otherwise setup disagrees and placement is blocked. ## Configuration From a3b021bbe099ecd3901ef1a5530f99b2da81d1eb Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 23 Oct 2026 17:52:44 -0600 Subject: [PATCH 06/32] docs: require replica drain before residency downgrade Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/replication/residency.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/reference/replication/residency.md b/reference/replication/residency.md index 5a05283db..0fe778e2f 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -57,7 +57,7 @@ The policy is immutable once activated, including removal of the directive. Rede ## Ownership, reads, and writes -Owners store full records. With secondary indexes, other nodes keep index projections and receive invalidations. Reads can fetch the full body from an owner and cache a temporary replica. Replicas retain the canonical record identity and do not become owners or count toward owner confirmation. +Owners store full records. With secondary indexes, other nodes keep index projections and receive invalidations. Reads fetch the full body from an owner. RocksDB can cache a temporary replica; LMDB proxies the read without caching it. Replicas retain the canonical record identity and do not become owners or count toward owner confirmation. A key-policy table without secondary indexes stores nothing on non-owners; reads resolve the owner from the key. Such tables do not cache replicas. Attribute policies require a secondary index so a non-owner can retain the stored slot. @@ -134,6 +134,8 @@ Existing residency IDs remain readable but no longer decide placement. Key-deriv ### Rollback -For a key-derived policy, restore the previous binary together with the equivalent legacy schema and `setResidencyById` function. No record conversion is needed. Keep the installed map for a later upgrade. +For a key-derived policy, first stop new replica admissions on every node, demote or evict existing RocksDB replicas, and verify that none remain. An earlier binary can interpret a replica as a full owned record. Lowering `replicaExpiration` does not rewrite existing deadlines or prove that the table is drained. + +After verifying the drain, restore the previous binary together with the equivalent legacy schema and `setResidencyById` function. Key policies need no slot-format conversion. Keep the installed map for a later upgrade. An attribute policy writes a new local primary/audit format. In-place downgrade to an earlier binary is unsupported. Restore a backup from before activation; writes since that backup may be lost. Protocol fallback does not make an earlier binary understand the new local storage format. From eb216650e63e65c9944f0c141759820e6dd2a148 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 23 Oct 2026 18:08:42 -0600 Subject: [PATCH 07/32] docs: state the slot-map precedence over replicateTo Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/replication/residency.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/replication/residency.md b/reference/replication/residency.md index 0fe778e2f..11f9c6f9d 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -61,7 +61,7 @@ Owners store full records. With secondary indexes, other nodes keep index projec A key-policy table without secondary indexes stores nothing on non-owners; reads resolve the owner from the key. Such tables do not cache replicas. Attribute policies require a secondary index so a non-owner can retain the stored slot. -`replicateTo` selects delivery targets; it cannot change declarative ownership. A non-owner write requires a configured replication route to an owner. Cross-shard query fan-out and write forwarding are not provided by this directive. Application database branches do not support declarative residency because branches do not replicate. +`replicateTo` cannot override a declarative policy; the installed slot map determines where full records are retained. A non-owner write requires a configured replication route to an owner. Cross-shard query fan-out and write forwarding are not provided by this directive. Application database branches do not support declarative residency because branches do not replicate. ### Attribute-slot changes From af6170effe01050a4772bc85777f0c27e83f4a7c Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 23 Oct 2026 18:58:17 -0600 Subject: [PATCH 08/32] Complete residency references and verified rollback procedure Document conditional shard configuration, initialization and verification operations, status diagnostics and schema metadata. Give operators an executable replica-drain procedure and require custom-map installation before declarative application deployment on joining nodes. Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/configuration/options.md | 1 + reference/database/schema.md | 6 ++++ reference/database/system-tables.md | 6 ++++ reference/operations-api/operations.md | 34 ++++++++++++++++------- reference/replication/clustering.md | 21 ++++++++++++++ reference/replication/overview.md | 2 ++ reference/replication/residency.md | 38 ++++++++++++++++++++------ reference/replication/sharding.md | 8 ++++-- release-notes/v5-lincoln/5.4.md | 4 +++ 9 files changed, 99 insertions(+), 21 deletions(-) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index ee62df608..9d0b25738 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -293,6 +293,7 @@ replication: - `blobSendDrainTimeout` — Maximum time (ms) a worker waits for in-flight replication blob **sends** to finish before shutting down during a restart, so a rolling restart (e.g., a component deploy reload) doesn't interrupt a transfer in progress. Only sends that are still making progress are waited on; `0` disables draining; _Default_: `600000` - `failOver` — Failover to alternate node if peer unreachable; _Default_: `true` - `shard` — Shard ID for traffic routing; see [Sharding](../replication/sharding.md) +- `shards` — Complete array of unique shard IDs, integers `0`–`65535`; order does not matter. Required for a declarative database with no installed map; custom-map conversion can omit it. All peers must agree, and the local `shard` must be included when configured. Changing it never recomputes an installed map; see [Complete shard-id set](../replication/sharding.md#complete-shard-id-set). (Added in: v5.4.0) - `mtls.certificateVerification` — Certificate revocation checking (CRL/OCSP) for replication connections; see [Certificate Verification](../security/certificate-verification.md) - `logging` — Replication event logging; sub-options: `path`, `level`, `tag`, `stdStreams`. See [Logging Configuration](../logging/configuration.md) diff --git a/reference/database/schema.md b/reference/database/schema.md index 7035be5bf..be9e3f613 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -238,6 +238,12 @@ export class Product extends tables.Product { See [REST Headers / Cache-Control](../rest/headers.md#cache-control) for the full caching behavior and how `Cache-Control` interacts with authentication headers. +### `@residency` + + + +Defines an immutable placement policy on a schema-defined table. `slots` is required (`1`–`65536`); `slotFrom` optionally supplies a deterministic JavaScript expression; `byAttribute: true` evaluates the record and requires a secondary index. The database's installed map resolves each slot to a deployment shard. See [Declarative Residency](../replication/residency.md) for examples, configuration, conversion and rollback. + ### `@export` Exposes the table as an externally accessible resource endpoint, available via REST, MQTT, and other interfaces. diff --git a/reference/database/system-tables.md b/reference/database/system-tables.md index a86d47a2e..3d9c91dbe 100644 --- a/reference/database/system-tables.md +++ b/reference/database/system-tables.md @@ -133,6 +133,12 @@ This table is managed automatically by the Data Loader. No direct interaction is ## Replication Tables +### `hdb_residency_slots` + + + +Stores one authoritative [residency map](../replication/residency.md#installed-maps) snapshot per database, keyed by `database`, with `version`, `digest`, `slots`, and complete nonoverlapping `ranges`. Install custom maps through the super-user `initialize_residency_slots` operation. Active assignments cannot move, and deleting a map does not regenerate it from configuration. + ### `hdb_nodes` Stores the configuration and state of known nodes in a cluster, including connection details, replication settings, and revoked certificate serial numbers. diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 8b7439933..b28d0f9e3 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -1780,16 +1780,18 @@ Operations for configuring and managing Harper cluster replication. Detailed documentation: [Replication & Clustering](../replication/clustering.md) -| Operation | Description | Role Required | -| ----------------------- | --------------------------------------------------------------- | ------------- | -| `add_node` | Adds a Harper instance to the cluster | super_user | -| `update_node` | Modifies an existing node's subscriptions | super_user | -| `remove_node` | Removes a node from the cluster | super_user | -| `cluster_status` | Returns current cluster connection status | super_user | -| `configure_cluster` | Bulk-creates/resets cluster subscriptions across multiple nodes | super_user | -| `cluster_set_routes` | Adds routes to the replication routes config (PATCH/upsert) | super_user | -| `cluster_get_routes` | Returns the current replication routes config | super_user | -| `cluster_delete_routes` | Removes routes from the replication routes config | super_user | +| Operation | Description | Role Required | +| ---------------------------- | ---------------------------------------------------------------- | ------------- | +| `add_node` | Adds a Harper instance to the cluster | super_user | +| `update_node` | Modifies an existing node's subscriptions | super_user | +| `remove_node` | Removes a node from the cluster | super_user | +| `initialize_residency_slots` | Installs a custom database slot map without moving records | super_user | +| `verify_residency_slots` | Compares a legacy placement function with a proposed slot policy | super_user | +| `cluster_status` | Returns current cluster connection status | super_user | +| `configure_cluster` | Bulk-creates/resets cluster subscriptions across multiple nodes | super_user | +| `cluster_set_routes` | Adds routes to the replication routes config (PATCH/upsert) | super_user | +| `cluster_get_routes` | Returns the current replication routes config | super_user | +| `cluster_delete_routes` | Removes routes from the replication routes config | super_user | ### `add_node` @@ -1804,6 +1806,18 @@ Adds a remote Harper node to the cluster. If `subscriptions` are not provided, a } ``` +### `initialize_residency_slots` + + + +POST; requires `super_user`. Takes `database`, `slots` and complete nonoverlapping `ranges` (`start`, exclusive `end`, `shard`). Returns the installed versioned snapshot. Installs metadata only and refuses changes to active slot assignments. See [Installed maps](../replication/residency.md#installed-maps). + +### `verify_residency_slots` + + + +POST; requires `super_user`. Takes `database`, `table` and a proposed key-derived `policy` (`slots`, optional `slotFrom`). The existing application worker must still run its legacy function, and a custom map must be installed. Returns `checked`, `mismatches`, bounded samples, observed slots, departed node names and map identity. It neither rewrites records nor activates the policy. See [Compare placement](../replication/residency.md#3-compare-placement-on-every-node). + ### `cluster_status` Returns connection state for all cluster nodes, including per-database socket status and replication timing statistics (`lastCommitConfirmed`, `lastReceivedRemoteTime`, `lastReceivedLocalTime`). diff --git a/reference/replication/clustering.md b/reference/replication/clustering.md index 29c8955bd..e46fd6a16 100644 --- a/reference/replication/clustering.md +++ b/reference/replication/clustering.md @@ -201,6 +201,27 @@ While a database is taking its initial full copy, these timing fields hold the l --- +### Declarative Residency Status + + + +A `cluster_status` object also includes `residency`, keyed by database, for active or attempted declarative declarations. Each entry has: + +| Field | Meaning | +| ------------------- | -------------------------------------------------------------------------------------------------------------------------- | +| `version`, `digest` | Installed map identity; absent if no map is installed. | +| `shardsDigest` | Digest of the configured shard set, or `null` when no list is configured. | +| `blocked` | Whether a reported configuration or map-agreement condition blocks placement. | +| `errors` | Reasons, such as differing peer lists, a local shard outside its list, a missing required list/map, or a refused peer map. | + +A clear status is not proof that an owner is connected; write admission also checks the required replication route and, for a non-owner attribute-slot transition, a connected new owner. See [configuration agreement](./sharding.md#complete-shard-id-set) and [map recovery](./residency.md#installed-maps). + +### Residency Map Operations + +The POST operations [`initialize_residency_slots`](./residency.md#installed-maps) and [`verify_residency_slots`](./residency.md#3-compare-placement-on-every-node) require `super_user`. The former installs a snapshot without moving records; the latter compares the actual legacy function against a proposed key policy without modifying data. Their request examples, parameters and result fields are in the linked procedures. + +--- + ### Configure Cluster Bulk creates or resets subscriptions for any number of remote nodes. **Resets and replaces any existing clustering setup.** diff --git a/reference/replication/overview.md b/reference/replication/overview.md index 7c75aab2c..5ed2acb32 100644 --- a/reference/replication/overview.md +++ b/reference/replication/overview.md @@ -99,6 +99,8 @@ With dedicated replication threads: - On macOS and Windows, which lack `SO_REUSEPORT`, only one replication thread listens on the replication port; the others carry outbound subscriptions. - Changing the number of threads takes effect only on a full `restart` of Harper; `restart_service` does not start or stop replication threads. An unscoped `restart_service` does restart the existing replication threads, and deploying or dropping a component does not. +Schema-defined [declarative residency](./residency.md) runs on application, operations and dedicated replication threads; it is an alternative to the thread-local residency setters. + ## Securing Connections Harper supports PKI-based security and authorization for replication connections. Two authentication methods are supported: diff --git a/reference/replication/residency.md b/reference/replication/residency.md index 11f9c6f9d..6cec549cf 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -12,7 +12,7 @@ Upgrade every schema and data writer before activation. Wire compatibility with ## Configuration -Configure the complete shard-id set on every node. See [`replication.shards`](./sharding.md#complete-shard-id-set) for initialization and mismatch handling. +For a new declarative table with no installed map, configure the complete shard-id set on every node. A custom-map conversion does not require this list; leave it unset on both sides when using legacy-peer fallback. See [`replication.shards`](./sharding.md#complete-shard-id-set) for initialization and mismatch handling. ```yaml replication: @@ -59,9 +59,9 @@ The policy is immutable once activated, including removal of the directive. Rede Owners store full records. With secondary indexes, other nodes keep index projections and receive invalidations. Reads fetch the full body from an owner. RocksDB can cache a temporary replica; LMDB proxies the read without caching it. Replicas retain the canonical record identity and do not become owners or count toward owner confirmation. -A key-policy table without secondary indexes stores nothing on non-owners; reads resolve the owner from the key. Such tables do not cache replicas. Attribute policies require a secondary index so a non-owner can retain the stored slot. +A key-policy table without secondary indexes stores nothing on non-owners; reads resolve the owner from the key. Such tables do not cache replicas. Attribute policies require a secondary index so a non-owner can retain the stored slot. If an attribute record's slot has not arrived locally, DELETE is refused until placement is known; local absence cannot prove that the record is absent on its owners. -`replicateTo` cannot override a declarative policy; the installed slot map determines where full records are retained. A non-owner write requires a configured replication route to an owner. Cross-shard query fan-out and write forwarding are not provided by this directive. Application database branches do not support declarative residency because branches do not replicate. +`replicateTo` cannot override a declarative policy; the installed slot map determines where full records are retained. A non-owner accepts a local write, retains only its index projection (or nothing for an index-free key policy), and sends the complete body through replication to the owners. This requires a configured replication route to an owner. Cross-shard query fan-out and write forwarding are not provided by this directive. Application database branches do not support declarative residency because branches do not replicate. ### Attribute-slot changes @@ -75,7 +75,7 @@ type RegionalItem @table @residency(slots: 10, slotFrom: "region", byAttribute: Changing `region` sends the complete canonical body to the new owners and invalidates old owners at the same version. When the writer is outside the new owner set, success requires at least one new owner to confirm durable receipt. If no new owner is connected, the change is refused before commit. -A connection can fail after admission. A confirmation failure after local commit means the outcome is uncertain; it does not mean the transaction rolled back. Inspect the record and retry with the application's concurrency controls. Audit retention bounds replay for disconnected peers; it is not an indefinite handoff queue. +A connection can fail after admission. An error with `replicationUnconfirmed: true` after local commit means the outcome is uncertain; it does not mean the transaction rolled back. Inspect the record and retry with the application's concurrency controls. Audit retention bounds replay for disconnected peers; it is not an indefinite handoff queue. ## Installed maps @@ -97,7 +97,7 @@ An existing table requires a custom map before activation, even if this node cur } ``` -This POST operation requires `super_user`. It changes metadata only. An exact retry is a no-op. Once a declarative table is active, existing assignments cannot change; coverage can only extend without changing active slots. Same-version divergent snapshots are refused rather than resolved by last-write-wins. Deleting the installed map does not trigger default reconstruction. +This POST operation requires `super_user` and returns the installed snapshot. It changes metadata only. An exact retry is a no-op. Once a declarative table is active, existing assignments cannot change; coverage can only extend without changing active slots. Same-version divergent snapshots are refused rather than resolved by last-write-wins. Deleting the installed map does not trigger default reconstruction. Setup exchanges carry the map version/digest and a separate shard-list digest. A missing or conflicting map blocks dependent placement and replication progress and appears in `cluster_status.residency`. A conflict on the system stream can also hold later system updates. Repair the conflicting configuration/map and reconnect; do not assume that skipping the offending record is safe. Resolving independently initialized, differently placed databases may require restoring or exporting the losing data. Reassignments that move data are a separate feature. @@ -109,7 +109,7 @@ Back up the database and upgrade all writers. Keep the current `setResidencyById ### 2. Install the equivalent map -Install a custom map that reproduces the function's numeric shard result exactly, from one designated node. Wait for the same snapshot on every participant. +Install a custom map that reproduces the function's numeric shard result exactly, from one designated node. Wait for the same snapshot on every participant before enabling the directive. On a newly joining node in a custom-map deployment, install or receive that snapshot before deploying its declarative application; otherwise first activation would initialize a default map from the local list. ### 3. Compare placement on every node @@ -134,8 +134,30 @@ Existing residency IDs remain readable but no longer decide placement. Key-deriv ### Rollback -For a key-derived policy, first stop new replica admissions on every node, demote or evict existing RocksDB replicas, and verify that none remain. An earlier binary can interpret a replica as a full owned record. Lowering `replicaExpiration` does not rewrite existing deadlines or prove that the table is drained. +For a key-derived policy, drain cached RocksDB replicas before changing binaries: an earlier binary can interpret one as a full owned record. LMDB does not cache these replicas. Key policies themselves need no slot-format conversion. + +1. Back up the database. Pause application requests and background jobs on **every node**, including reads that could admit another replica. Keep them paused through the downgrade. +2. Set `@table(replicaExpiration: 0)` in each affected table's schema and restart every node on the **current** binary, preserving its residency directive. This table argument is in seconds; zero disables new replica admissions. It does not rewrite existing replica deadlines, so waiting for a new, shorter lifetime does not establish a drain. +3. Run the following trusted maintenance code for each affected table on every node. Run it inside the application worker with access to `tables`; do not expose it as a public endpoint. It uses this version's primary metadata (`REPLICA = 4`) to evict only cached replicas, preserving owned rows and index projections. + +```js +async function drainResidencyReplicas(Table) { + const REPLICA = 4; + Table.setTTLExpiration({ replicaExpiration: 0 }); + for (const entry of Table.primaryStore.getRange({ start: true })) { + if (entry.metadataFlags & REPLICA) await Table.evict(entry.key, entry.value, entry.version); + } + let remaining = 0; + for (const entry of Table.primaryStore.getRange({ start: true })) if (entry.metadataFlags & REPLICA) remaining++; + if (remaining) throw new Error(`${remaining} replicas remain; do not downgrade`); + return { remaining }; +} +await drainResidencyReplicas(tables.Item); +``` + +4. Require `{ remaining: 0 }` for every affected table on every node. Eviction is version-checked; a busy replica can survive a pass, so a nonzero result refuses the downgrade. Do not use ordinary record counts as proof: projections and replicas share keys. +5. Remove the maintenance code, restore the previous binary with the equivalent legacy schema and `setResidencyById` function, then resume traffic. Keep the installed slot map for a later upgrade. -After verifying the drain, restore the previous binary together with the equivalent legacy schema and `setResidencyById` function. Key policies need no slot-format conversion. Keep the installed map for a later upgrade. +There is no built-in replica-drain operation. If you cannot run and verify this maintenance procedure, restore a preactivation backup instead of attempting an in-place downgrade. An attribute policy writes a new local primary/audit format. In-place downgrade to an earlier binary is unsupported. Restore a backup from before activation; writes since that backup may be lost. Protocol fallback does not make an earlier binary understand the new local storage format. diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index 6d4bc1ee1..bc17922bc 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -14,7 +14,7 @@ title: Sharding Harper's replication system supports sharding — storing different data across different subsets of nodes — while still allowing data to be accessed from any node in the cluster. This enables horizontal scalability for storage and write performance, while maintaining optimal data locality and consistency. -When sharding is configured, requests for records that don't reside on the handling node are automatically forwarded to the appropriate node transparently. Clients do not need to know where data is stored. +A key read for a record not held locally can fetch its body from an owner. This does not provide cross-shard query fan-out or write forwarding. With [declarative residency](./residency.md#ownership-reads-and-writes), a non-owner write commits locally and replicates the complete body to its owners through the configured replication routes. By default (without sharding), Harper replicates all data to all nodes. @@ -163,7 +163,7 @@ replication: -For [declarative residency](./residency.md), configure `replication.shards` next to `replication.shard`: +For the first default map of a new [declarative residency](./residency.md) table, configure `replication.shards` next to `replication.shard`: ```yaml replication: @@ -181,7 +181,7 @@ Setup exchanges include a digest of the list. Nodes with different lists refuse ### Other shard configuration -Or assign shards per route: +Shards can also be assigned per route: ```yaml replication: @@ -278,5 +278,7 @@ X-Replicate-From: none ## See Also +- [Declarative Residency](./residency.md) + - [Replication Overview](./overview.md) — How Harper's replication system works - [Clustering Operations](./clustering.md) — Operations API for managing cluster nodes diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 8216b8f45..0edfd5b55 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -22,6 +22,10 @@ These corrections are not released yet. ## Replication +### Declarative Residency + +`@residency` applies a fixed key- or attribute-based slot policy on every thread, including dedicated replication workers. Per-database slot maps remain authoritative after initialization; `replication.shards` supplies the complete deployment shard set for a first default map. Super-user map initialization and read-only conversion checks support adopting an equivalent legacy by-id policy without rewriting records. Attribute-slot changes from a non-owner require a new owner's confirmation. See [Declarative Residency](/reference/v5/replication/residency), including its storage-format and rollback requirements. + ### Dedicated Replication Threads `replication.threads` (default `0`) starts worker threads that run replication only. They own the replication port and every subscription, so a peer catching up no longer competes with application requests. These threads load no application code, which limits applications that use `setResidencyById` or `setComputedAttribute` on indexed attributes. See [Dedicated Replication Threads](/reference/v5/replication/overview#dedicated-replication-threads). From e7aac2f6c4e8f7af2bcbb1d14520e00f07512ae2 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 23 Oct 2026 20:09:10 -0600 Subject: [PATCH 09/32] docs: clarify initialization and conversion boundaries Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/operations-api/operations.md | 4 ++-- reference/replication/clustering.md | 5 +++-- reference/replication/residency.md | 10 +++++++--- reference/replication/sharding.md | 2 +- 4 files changed, 13 insertions(+), 8 deletions(-) diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index b28d0f9e3..b905097f4 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -1816,11 +1816,11 @@ POST; requires `super_user`. Takes `database`, `slots` and complete nonoverlappi -POST; requires `super_user`. Takes `database`, `table` and a proposed key-derived `policy` (`slots`, optional `slotFrom`). The existing application worker must still run its legacy function, and a custom map must be installed. Returns `checked`, `mismatches`, bounded samples, observed slots, departed node names and map identity. It neither rewrites records nor activates the policy. See [Compare placement](../replication/residency.md#3-compare-placement-on-every-node). +POST; requires `super_user`. Takes `database`, `table` and a proposed key-derived `policy` (`slots`, optional `slotFrom`). The existing application worker must still run its legacy function, and a custom map must be installed. Returns `checked`, `mismatches`, bounded samples, observed slots, departed node names (stored legacy owners absent from the current known-node list) and map identity. It neither rewrites records nor activates the policy. See [Compare placement](../replication/residency.md#3-compare-placement-on-every-node). ### `cluster_status` -Returns connection state for all cluster nodes, including per-database socket status and replication timing statistics (`lastCommitConfirmed`, `lastReceivedRemoteTime`, `lastReceivedLocalTime`). +Returns connection state for all cluster nodes, including per-database socket status and replication timing statistics (`lastCommitConfirmed`, `lastReceivedRemoteTime`, `lastReceivedLocalTime`). Its top-level `residency` object reports per-database map identity and placement blockers; see [Declarative Residency Status](../replication/clustering.md#declarative-residency-status). ```json { "operation": "cluster_status" } diff --git a/reference/replication/clustering.md b/reference/replication/clustering.md index e46fd6a16..8573ecc2f 100644 --- a/reference/replication/clustering.md +++ b/reference/replication/clustering.md @@ -184,7 +184,8 @@ Returns an array of status objects from the cluster, including active WebSocket } ], "node_name": "server-1.domain.com", - "is_enabled": true + "is_enabled": true, + "residency": {} } ``` @@ -205,7 +206,7 @@ While a database is taking its initial full copy, these timing fields hold the l -A `cluster_status` object also includes `residency`, keyed by database, for active or attempted declarative declarations. Each entry has: +The top-level `cluster_status.residency` field is keyed by database and describes active or attempted declarative tables. It is empty in the legacy-table example above. Each entry has: | Field | Meaning | | ------------------- | -------------------------------------------------------------------------------------------------------------------------- | diff --git a/reference/replication/residency.md b/reference/replication/residency.md index 6cec549cf..acf86954d 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -51,7 +51,7 @@ type Item @table @residency(slots: 10, slotFrom: "Number(id.slice(-1))") { } ``` -Expressions must be self-contained, deterministic, and synchronous. They compile on every thread without application helpers. A non-integer or out-of-range result refuses the write. Schema expressions are trusted code, as with computed expressions; they are not a sandbox. +Expressions must be self-contained, deterministic, and synchronous. They compile on every thread without application helpers. An expression that throws, returns a Promise, or produces a non-integer or out-of-range result refuses the write. The last-digit example assumes string keys, matching the existing function being converted. Schema expressions are trusted code, as with computed expressions; they are not a sandbox. The policy is immutable once activated, including removal of the directive. Redeploying the same policy is idempotent. Calling `setResidency` or `setResidencyById` on a declarative table fails; remove those calls when converting. Existing tables can adopt a key-derived policy with an equivalent custom map. Attribute policies must be declared on a new table because existing rows have no stored slots. @@ -83,6 +83,8 @@ Each database has one versioned snapshot in `system.hdb_residency_slots`. The sn For a new declarative table with no map, first activation installs version `1` from only the declared slot count and sorted `replication.shards`: slot `k` maps to `shards[k % shards.length]`. Equal inputs produce the same snapshot and digest on simultaneous installers. The installed snapshot is then authoritative. Changes to the configured shard list never regenerate it. +The first table activated supplies the default slot count; Harper does not choose the maximum across a schema. For tables with different slot counts, install one custom map covering the largest count and wait for the same snapshot on every node before deploying the tables. A later table needing more slots requires an explicit `initialize_residency_slots` extension that preserves every active assignment. It never extends the default automatically. Do not run different initial schema deployments concurrently. + An existing table requires a custom map before activation, even if this node currently holds no rows. A custom map can be installed without `replication.shards`: ```json @@ -101,6 +103,8 @@ This POST operation requires `super_user` and returns the installed snapshot. It Setup exchanges carry the map version/digest and a separate shard-list digest. A missing or conflicting map blocks dependent placement and replication progress and appears in `cluster_status.residency`. A conflict on the system stream can also hold later system updates. Repair the conflicting configuration/map and reconnect; do not assume that skipping the offending record is safe. Resolving independently initialized, differently placed databases may require restoring or exporting the losing data. Reassignments that move data are a separate feature. +There is no operation to delete or reset an active map. If an accidentally initialized joining node has no unique data, stop its traffic and reprovision it with a fresh data directory; receive or install the cluster's exact snapshot before deploying the declarative application. Do not delete individual system-table files. If it has unique data, preserve a backup and restore/export that data before reprovisioning. In custom-map deployments, leaving `replication.shards` unset on every participant prevents premature default installation; lists must still agree across peers. + ## Convert an existing by-id table without moving records ### 1. Prepare the existing table @@ -124,7 +128,7 @@ Run `verify_residency_slots` on each node while its application worker still run } ``` -This POST operation requires `super_user`. It scans local records and compares the actual old function's numeric shard with the proposed policy and installed map. It reports `checked`, `mismatches`, bounded mismatch samples, observed slots, departed node names, and map identity. It does not rewrite records or activate the policy. A sample of present keys cannot prove equivalence for every future key; the operator must confirm the functions are equivalent. +This POST operation requires `super_user`. It scans local records and compares the actual old function's numeric shard with the proposed policy and installed map. It reports `checked`, `mismatches`, bounded mismatch samples, observed slots, departed node names, and map identity. It does not rewrite records or activate the policy. Hostname-array functions are not converted: the first nonnumeric return rejects the operation with HTTP 400, rather than counting as a mismatch or inferring a shard. `departedNodeNames` lists names found in stored legacy residency lists that are absent from the node's current known-node list (including itself). A sample of present keys cannot prove equivalence for every future key; the operator must confirm the functions are equivalent. ### 4. Activate the directive @@ -138,7 +142,7 @@ For a key-derived policy, drain cached RocksDB replicas before changing binaries 1. Back up the database. Pause application requests and background jobs on **every node**, including reads that could admit another replica. Keep them paused through the downgrade. 2. Set `@table(replicaExpiration: 0)` in each affected table's schema and restart every node on the **current** binary, preserving its residency directive. This table argument is in seconds; zero disables new replica admissions. It does not rewrite existing replica deadlines, so waiting for a new, shorter lifetime does not establish a drain. -3. Run the following trusted maintenance code for each affected table on every node. Run it inside the application worker with access to `tables`; do not expose it as a public endpoint. It uses this version's primary metadata (`REPLICA = 4`) to evict only cached replicas, preserving owned rows and index projections. +3. Run the following trusted maintenance code for each affected table on every node. Run it inside the application worker with access to `tables`; do not expose it as a public endpoint. This procedure is for the 5.4 implementation and uses internal primary metadata (`REPLICA = 4`); verify the matching implementation before using it with another release. It evicts only cached replicas, preserving owned rows and legacy-compatible invalidated index projections. The runtime TTL call below repeats step 2 defensively on this table handle and preserves its other TTL settings. These APIs are trusted worker internals, not Operations API commands. ```js async function drainResidencyReplicas(Table) { diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index bc17922bc..8b22d7a03 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -256,7 +256,7 @@ MyTable.setResidencyById((id) => { ## Disabling Cross-Node Access -By default, sharding allows data stored on specific nodes to be accessed from any node — requests are forwarded transparently. To disable this and only return data stored on the local node (including a replica it currently holds), set `replicateFrom` to `false`. +By default, a key read for a record not stored locally fetches it from a node that holds it. To disable this and only return data stored on the local node (including a replica it currently holds), set `replicateFrom` to `false`. Via the operations API: From fe2ab73238aad33685c60fc20491177f81249424 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 23 Oct 2026 20:56:50 -0600 Subject: [PATCH 10/32] docs: clarify residency prerequisites and confirmation behavior Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/replication/residency.md | 6 ++++-- release-notes/v5-lincoln/5.4.md | 4 ++++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/reference/replication/residency.md b/reference/replication/residency.md index acf86954d..e0201d266 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -39,6 +39,8 @@ type Item @table @residency(slots: 1024) { | `slotFrom` | Optional JavaScript expression returning an integer from `0` through `slots - 1`. Key policies expose `id`. | | `byAttribute` | Default `false`. When `true`, the expression also exposes `record` and its attribute names, and the slot is stored with the record. Requires a secondary index. | +Declarative tables require audit logging and table replication. Harper refuses activation with `@table(audit: false)`, effective `logging.auditLog: false`, or `@table(replicate: false)`, and refuses disabling these table settings later. The replication route must also reach an owner. + Without `slotFrom`, the slot is the unsigned FNV-1a 32-bit hash of the key's ordered-binary encoding, modulo `slots`. Its offset basis is `2166136261` and its multiplier is `16777619`. An explicit key policy can preserve an existing partition function: @@ -59,7 +61,7 @@ The policy is immutable once activated, including removal of the directive. Rede Owners store full records. With secondary indexes, other nodes keep index projections and receive invalidations. Reads fetch the full body from an owner. RocksDB can cache a temporary replica; LMDB proxies the read without caching it. Replicas retain the canonical record identity and do not become owners or count toward owner confirmation. -A key-policy table without secondary indexes stores nothing on non-owners; reads resolve the owner from the key. Such tables do not cache replicas. Attribute policies require a secondary index so a non-owner can retain the stored slot. If an attribute record's slot has not arrived locally, DELETE is refused until placement is known; local absence cannot prove that the record is absent on its owners. +A key-policy table without secondary indexes stores nothing on non-owners; reads resolve the owner from the key. Such tables do not cache replicas. Attribute policies require a secondary index so a non-owner can retain the stored slot. If an attribute record's slot is unknown locally, DELETE returns 503; local absence cannot prove that the record is absent on its owners. This also applies to never-created keys and purged tombstones. Retrying alone cannot resolve permanent unknown placement; send the deletion to a node that still has the slot metadata. `replicateTo` cannot override a declarative policy; the installed slot map determines where full records are retained. A non-owner accepts a local write, retains only its index projection (or nothing for an index-free key policy), and sends the complete body through replication to the owners. This requires a configured replication route to an owner. Cross-shard query fan-out and write forwarding are not provided by this directive. Application database branches do not support declarative residency because branches do not replicate. @@ -73,7 +75,7 @@ type RegionalItem @table @residency(slots: 10, slotFrom: "region", byAttribute: } ``` -Changing `region` sends the complete canonical body to the new owners and invalidates old owners at the same version. When the writer is outside the new owner set, success requires at least one new owner to confirm durable receipt. If no new owner is connected, the change is refused before commit. +Changing `region` sends the complete canonical body to the new owners and invalidates old owners at the same version. This handoff uses full-record replacement ordering: an older concurrent patch does not merge into the replacement. Use application concurrency controls when writers can update a record while its slot changes. When the writer is outside the new owner set, success requires at least one new owner to confirm durable receipt. If no new owner is connected, the change is refused before commit. A connection can fail after admission. An error with `replicationUnconfirmed: true` after local commit means the outcome is uncertain; it does not mean the transaction rolled back. Inspect the record and retry with the application's concurrency controls. Audit retention bounds replay for disconnected peers; it is not an indefinite handoff queue. diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 0edfd5b55..8ac90cfc1 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -26,6 +26,10 @@ These corrections are not released yet. `@residency` applies a fixed key- or attribute-based slot policy on every thread, including dedicated replication workers. Per-database slot maps remain authoritative after initialization; `replication.shards` supplies the complete deployment shard set for a first default map. Super-user map initialization and read-only conversion checks support adopting an equivalent legacy by-id policy without rewriting records. Attribute-slot changes from a non-owner require a new owner's confirmation. See [Declarative Residency](/reference/v5/replication/residency), including its storage-format and rollback requirements. +### LMDB Replication Confirmation + +LMDB writes that request `replicatedConfirmation` now wait for peer acknowledgments through the shared replication callback. Confirmed batches use abortable child transactions. Applications that previously received immediate success can now observe replication latency or an uncertain committed outcome if confirmation fails. + ### Dedicated Replication Threads `replication.threads` (default `0`) starts worker threads that run replication only. They own the replication port and every subscription, so a peer catching up no longer competes with application requests. These threads load no application code, which limits applications that use `setResidencyById` or `setComputedAttribute` on indexed attributes. See [Dedicated Replication Threads](/reference/v5/replication/overview#dedicated-replication-threads). From 5a3c6fd05d5cec8ed092df78b505100d96b1d3a6 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 23 Oct 2026 21:24:30 -0600 Subject: [PATCH 11/32] docs: clarify confirmation failures and unknown slot deletion Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/database/system-tables.md | 2 +- reference/replication/residency.md | 6 +++--- reference/replication/sharding.md | 10 +++++++++- release-notes/v5-lincoln/5.4.md | 2 +- 4 files changed, 14 insertions(+), 6 deletions(-) diff --git a/reference/database/system-tables.md b/reference/database/system-tables.md index 3d9c91dbe..ce4b0937e 100644 --- a/reference/database/system-tables.md +++ b/reference/database/system-tables.md @@ -137,7 +137,7 @@ This table is managed automatically by the Data Loader. No direct interaction is -Stores one authoritative [residency map](../replication/residency.md#installed-maps) snapshot per database, keyed by `database`, with `version`, `digest`, `slots`, and complete nonoverlapping `ranges`. Install custom maps through the super-user `initialize_residency_slots` operation. Active assignments cannot move, and deleting a map does not regenerate it from configuration. +Stores one authoritative [residency map](../replication/residency.md#installed-maps) snapshot per database, keyed by `database`, with `version`, `digest`, `slots`, and complete nonoverlapping `ranges`. Install custom maps through the super-user `initialize_residency_slots` operation. Active assignments cannot move, and deleting an installed map is refused. A lost or corrupted snapshot must be recovered; it is never regenerated from configuration. ### `hdb_nodes` diff --git a/reference/replication/residency.md b/reference/replication/residency.md index e0201d266..930848bd1 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -39,7 +39,7 @@ type Item @table @residency(slots: 1024) { | `slotFrom` | Optional JavaScript expression returning an integer from `0` through `slots - 1`. Key policies expose `id`. | | `byAttribute` | Default `false`. When `true`, the expression also exposes `record` and its attribute names, and the slot is stored with the record. Requires a secondary index. | -Declarative tables require audit logging and table replication. Harper refuses activation with `@table(audit: false)`, effective `logging.auditLog: false`, or `@table(replicate: false)`, and refuses disabling these table settings later. The replication route must also reach an owner. +Declarative tables require [per-table audit logging](../database/transaction.md#enabling-the-transaction-log-per-table) and table replication. Harper refuses activation with `@table(audit: false)`, effective `logging.auditLog: false`, or `@table(replicate: false)`, and refuses disabling these table settings later. The replication route must also reach an owner. Without `slotFrom`, the slot is the unsigned FNV-1a 32-bit hash of the key's ordered-binary encoding, modulo `slots`. Its offset basis is `2166136261` and its multiplier is `16777619`. @@ -61,7 +61,7 @@ The policy is immutable once activated, including removal of the directive. Rede Owners store full records. With secondary indexes, other nodes keep index projections and receive invalidations. Reads fetch the full body from an owner. RocksDB can cache a temporary replica; LMDB proxies the read without caching it. Replicas retain the canonical record identity and do not become owners or count toward owner confirmation. -A key-policy table without secondary indexes stores nothing on non-owners; reads resolve the owner from the key. Such tables do not cache replicas. Attribute policies require a secondary index so a non-owner can retain the stored slot. If an attribute record's slot is unknown locally, DELETE returns 503; local absence cannot prove that the record is absent on its owners. This also applies to never-created keys and purged tombstones. Retrying alone cannot resolve permanent unknown placement; send the deletion to a node that still has the slot metadata. +A key-policy table without secondary indexes stores nothing on non-owners; reads resolve the owner from the key. Such tables do not cache replicas. Attribute policies require a secondary index so a non-owner can retain the stored slot. If an attribute record's slot is unknown locally, DELETE returns 503; local absence cannot prove that the record is absent on its owners. The message begins `Attribute residency slot is unknown`. This response does not distinguish a delayed projection from a never-created key or a purged tombstone. If another node still has the slot metadata, send the deletion there. If no node has it, DELETE cannot succeed anywhere; retrying alone cannot fix that condition. Stop automatic retries and use application-level knowledge to establish whether the key ever existed. Local absence is not proof that it was deleted. `replicateTo` cannot override a declarative policy; the installed slot map determines where full records are retained. A non-owner accepts a local write, retains only its index projection (or nothing for an index-free key policy), and sends the complete body through replication to the owners. This requires a configured replication route to an owner. Cross-shard query fan-out and write forwarding are not provided by this directive. Application database branches do not support declarative residency because branches do not replicate. @@ -101,7 +101,7 @@ An existing table requires a custom map before activation, even if this node cur } ``` -This POST operation requires `super_user` and returns the installed snapshot. It changes metadata only. An exact retry is a no-op. Once a declarative table is active, existing assignments cannot change; coverage can only extend without changing active slots. Same-version divergent snapshots are refused rather than resolved by last-write-wins. Deleting the installed map does not trigger default reconstruction. +This POST operation requires `super_user` and returns the installed snapshot. It changes metadata only. An exact retry is a no-op. Once a declarative table is active, existing assignments cannot change; coverage can only extend without changing active slots. Same-version divergent snapshots are refused rather than resolved by last-write-wins. Deleting an installed map is refused. If its stored snapshot is lost or corrupted, Harper does not reconstruct it from configuration. Setup exchanges carry the map version/digest and a separate shard-list digest. A missing or conflicting map blocks dependent placement and replication progress and appears in `cluster_status.residency`. A conflict on the system stream can also hold later system updates. Repair the conflicting configuration/map and reconnect; do not assume that skipping the offending record is safe. Resolving independently initialized, differently placed databases may require restoring or exporting the losing data. Reassignments that move data are a separate feature. diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index 8b22d7a03..62adc8f2b 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -120,6 +120,12 @@ That transaction commits on its own, separately from any request transaction tha Values your code sets on a context apply for every caller, including users without the `super_user` permission. Derive them from application logic, not from request input such as headers, query parameters, or the record body: copying a client-supplied value into `replicateTo` or `replicatedConfirmation` gives that client the control the `super_user` check removes. +### Confirmation outcomes + + + +LMDB now waits for requested peer confirmation, as RocksDB does. This can add peer latency to existing LMDB applications that use `confirm` or `replicatedConfirmation`. Explicit confirmation still applies only to the first database written by the transaction. + ### Confirmation Failure After a Local Commit @@ -140,7 +146,7 @@ For example, a REST error for a confirmation timeout can contain: } ``` -The existing error status, code, and message remain available. An operations error has its usual `error` field plus `code`, `replicationUnconfirmed: true`, and `retryable: false`. Programmatic errors wrap the original rejection in `cause`; text-only protocol errors use the new `ReplicationConfirmationError` class name. Check this flag before compensating or retrying a non-idempotent write, because replaying the request could repeat an insert or apply an update twice. The flag does not return auto-generated record keys; use client-selected identifiers when you need to inspect the written records after a failed response. +The existing error status, code, and message remain available. An operations error has its usual `error` field plus `code`, `replicationUnconfirmed: true`, and `retryable: false`. Programmatic errors wrap the original rejection in `cause`; text-only protocol errors use the new `ReplicationConfirmationError` class name. Check this flag before compensating or retrying a non-idempotent write, because replaying the request could repeat an insert or apply an update twice. The flag does not return auto-generated record keys; use client-selected identifiers and apply the application's concurrency controls when you need to inspect the written records after a failed response. The flag covers writes issued to the reported commit, not later application steps that were never reached after its rejection. If an enclosing scope rolls back its pending writes, its error omits the flag while preserving `retryable: false` because an independent inner write may already have committed. @@ -148,6 +154,8 @@ Pre-commit refusals, including a confirmation count that cannot be met, do not c Confirmation currently applies to the first database a transaction writes to. This outcome flag covers the linked local commits; it does not enable confirmation for later databases. +[Attribute-slot transitions](./residency.md#attribute-slot-changes) can require new-owner confirmation automatically even when the caller did not request an explicit confirmation count. + ## Static Sharding ### Basic Static Shard Configuration diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 8ac90cfc1..b7a79fcdf 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -28,7 +28,7 @@ These corrections are not released yet. ### LMDB Replication Confirmation -LMDB writes that request `replicatedConfirmation` now wait for peer acknowledgments through the shared replication callback. Confirmed batches use abortable child transactions. Applications that previously received immediate success can now observe replication latency or an uncertain committed outcome if confirmation fails. +LMDB writes that request `replicatedConfirmation` now wait for peer acknowledgments. Applications that previously received immediate success can now observe replication latency or an uncertain committed outcome if confirmation fails. See [Confirmation outcomes](/reference/v5/replication/sharding#confirmation-outcomes) before retrying a failed write. ### Dedicated Replication Threads From 474e8d7ec30ba3edb0cf414a0bcb5ee23ba8144f Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 23 Oct 2026 21:52:03 -0600 Subject: [PATCH 12/32] Clarify single-key reads and unknown-slot deletion guidance Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/replication/residency.md | 2 +- reference/replication/sharding.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/replication/residency.md b/reference/replication/residency.md index 930848bd1..10f3eaf40 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -61,7 +61,7 @@ The policy is immutable once activated, including removal of the directive. Rede Owners store full records. With secondary indexes, other nodes keep index projections and receive invalidations. Reads fetch the full body from an owner. RocksDB can cache a temporary replica; LMDB proxies the read without caching it. Replicas retain the canonical record identity and do not become owners or count toward owner confirmation. -A key-policy table without secondary indexes stores nothing on non-owners; reads resolve the owner from the key. Such tables do not cache replicas. Attribute policies require a secondary index so a non-owner can retain the stored slot. If an attribute record's slot is unknown locally, DELETE returns 503; local absence cannot prove that the record is absent on its owners. The message begins `Attribute residency slot is unknown`. This response does not distinguish a delayed projection from a never-created key or a purged tombstone. If another node still has the slot metadata, send the deletion there. If no node has it, DELETE cannot succeed anywhere; retrying alone cannot fix that condition. Stop automatic retries and use application-level knowledge to establish whether the key ever existed. Local absence is not proof that it was deleted. +A key-policy table without secondary indexes stores nothing on non-owners; reads resolve the owner from the key. Such tables do not cache replicas. Attribute policies require a secondary index so a non-owner can retain the stored slot. If an attribute record's slot is unknown locally, DELETE returns 503 with a message beginning `Attribute residency slot is unknown`. This response does not distinguish a delayed projection from a never-created key or a purged tombstone; local absence cannot prove that the record is absent on its owners. If another node still has the slot metadata, send the deletion there. If no node has it, DELETE cannot succeed anywhere; retrying alone cannot fix that condition. Stop automatic retries and use application-level knowledge to establish whether the key ever existed. Local absence is not proof that it was deleted. `replicateTo` cannot override a declarative policy; the installed slot map determines where full records are retained. A non-owner accepts a local write, retains only its index projection (or nothing for an index-free key policy), and sends the complete body through replication to the owners. This requires a configured replication route to an owner. Cross-shard query fan-out and write forwarding are not provided by this directive. Application database branches do not support declarative residency because branches do not replicate. diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index 62adc8f2b..4d2f58629 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -12,7 +12,7 @@ title: Sharding — expanded sharding functionality: Harper now honors write requests with residency information that will not be stored on the local node, and nodes can be declaratively configured as part of a shard. -Harper's replication system supports sharding — storing different data across different subsets of nodes — while still allowing data to be accessed from any node in the cluster. This enables horizontal scalability for storage and write performance, while maintaining optimal data locality and consistency. +Harper's replication system supports sharding — storing different data across different subsets of nodes. Single-key reads can fetch records from their owners across shards. Sharding distributes storage and write work across nodes. A key read for a record not held locally can fetch its body from an owner. This does not provide cross-shard query fan-out or write forwarding. With [declarative residency](./residency.md#ownership-reads-and-writes), a non-owner write commits locally and replicates the complete body to its owners through the configured replication routes. From afd13548f3e7415575d944f5ac9e8409d3a41a0f Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Sat, 24 Oct 2026 19:54:01 -0600 Subject: [PATCH 13/32] Document residency quarantine, writer replicas and confirmation timeouts Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/configuration/options.md | 2 ++ reference/replication/residency.md | 25 +++++++++++++++++++++++-- 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 9d0b25738..8efc2e67c 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -286,6 +286,8 @@ replication: - `securePort` — Secure replication port; _Default_: `9933` (changed from `9925` in v4.5.0) - `threads` — Worker threads dedicated to replication, which then own the replication port and every subscription and load no application code; requires `port` or `securePort`; see [Dedicated Replication Threads](../replication/overview.md#dedicated-replication-threads); _Default_: `0` (replication runs on the HTTP worker threads) (Added in: v5.4.0) - `enableRootCAs` — Verify against Node.js Mozilla CA store; _Default_: `true` +- `confirmationTimeout` — Timeout in milliseconds for explicit `replicatedConfirmation`, read when a waiter starts. Default: `900000`, or `blobTimeout` when greater. An explicitly configured value overrides that fallback, including a smaller value. Range: `1`–`2147483647`. (Added in: v5.4.0) +- `residencyConfirmationTimeout` — Timeout in milliseconds for automatic new-owner confirmation on an attribute-slot change. Default: `30000`. Independent of blob-transfer timeout. A timeout returns an uncertain-outcome 504 after local commit; it does not roll back the write. Range: `1`–`2147483647`. (Added in: v5.4.0) - `blobTimeout` — Blob transfer timeout (ms); _Default_: `120000` - `blobGapReconnectMs` — Interval (ms) for the blob-gap watchdog: when a transient blob save failure pins a replication resume cursor, the connection is forced to reconnect on this cadence so the gapped blob is re-streamed and, during a bulk copy, the copy resumes from the last banked cursor. Lower values heal gaps faster at the cost of more reconnects on a link whose faults never heal; _Default_: the `blobTimeout` value - `copyCursorFlushBytes` — Bytes of applied bulk-copy data between durable flushes of the copy resume cursor (RocksDB); _Default_: `67108864` diff --git a/reference/replication/residency.md b/reference/replication/residency.md index 10f3eaf40..234d2ae69 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -63,7 +63,7 @@ Owners store full records. With secondary indexes, other nodes keep index projec A key-policy table without secondary indexes stores nothing on non-owners; reads resolve the owner from the key. Such tables do not cache replicas. Attribute policies require a secondary index so a non-owner can retain the stored slot. If an attribute record's slot is unknown locally, DELETE returns 503 with a message beginning `Attribute residency slot is unknown`. This response does not distinguish a delayed projection from a never-created key or a purged tombstone; local absence cannot prove that the record is absent on its owners. If another node still has the slot metadata, send the deletion there. If no node has it, DELETE cannot succeed anywhere; retrying alone cannot fix that condition. Stop automatic retries and use application-level knowledge to establish whether the key ever existed. Local absence is not proof that it was deleted. -`replicateTo` cannot override a declarative policy; the installed slot map determines where full records are retained. A non-owner accepts a local write, retains only its index projection (or nothing for an index-free key policy), and sends the complete body through replication to the owners. This requires a configured replication route to an owner. Cross-shard query fan-out and write forwarding are not provided by this directive. Application database branches do not support declarative residency because branches do not replicate. +`replicateTo` cannot override a declarative policy; the installed slot map determines where full records are retained. A non-owner accepts a local write and sends the complete body through replication to the owners. On an indexed RocksDB table with positive `replicaExpiration`, it also retains the complete written body as a temporary local replica, allowing an immediate local read while delivery is pending. A newer invalidation replaces it, and the replica sweep expires it. With expiration zero or LMDB, it retains only the index projection; an index-free key policy retains nothing. A replica is a cache, so crash recovery may restore a projection instead. Ingest-heavy non-owners should budget space for approximately their serialized bytes written per second multiplied by `replicaExpiration`, or set expiration to zero. This requires a configured replication route to an owner. Cross-shard query fan-out and write forwarding are not provided by this directive. Application database branches do not support declarative residency because branches do not replicate. ### Attribute-slot changes @@ -75,10 +75,31 @@ type RegionalItem @table @residency(slots: 10, slotFrom: "region", byAttribute: } ``` -Changing `region` sends the complete canonical body to the new owners and invalidates old owners at the same version. This handoff uses full-record replacement ordering: an older concurrent patch does not merge into the replacement. Use application concurrency controls when writers can update a record while its slot changes. When the writer is outside the new owner set, success requires at least one new owner to confirm durable receipt. If no new owner is connected, the change is refused before commit. +Changing `region` sends the complete canonical body to the new owners and invalidates old owners at the same version. This handoff uses full-record replacement ordering: an older concurrent patch does not merge into the replacement. Use application concurrency controls when writers can update a record while its slot changes. When the writer is outside the new owner set, success requires at least one new owner to confirm durable receipt. If no new owner has an active subscription and recent received activity within `replication.pingTimeout`, the change is refused before commit. This can also refuse an owner whose base copy is not progressing. + +Automatic slot-move confirmation waits up to `replication.residencyConfirmationTimeout` (default `30000` ms). Explicit `replicatedConfirmation` uses `replication.confirmationTimeout` instead (default `900000` ms, raised to `blobTimeout` when the confirmation setting is omitted). Both settings are read when a confirmation starts. An explicit setting can lower that timeout independently of blob transfers. Slow transfers can therefore exceed the automatic timeout even on a healthy link. A connection can fail after admission. An error with `replicationUnconfirmed: true` after local commit means the outcome is uncertain; it does not mean the transaction rolled back. Inspect the record and retry with the application's concurrency controls. Audit retention bounds replay for disconnected peers; it is not an indefinite handoff queue. +### Subscription filters + +Non-owner invalidation events contain only indexed fields. A subscription `rowFilter` evaluates that event payload even if a complete replica exists locally. Predicates on unindexed fields can miss updates; a negated predicate can also pass because the field is missing. Use indexed fields for these predicates. Do not use an unindexed event predicate to decide whether a subscriber may learn a record's id or indexed fields. + +### Configuration failures and peer identity + +Invalid residency configuration at startup quarantines the affected database before replay. Its reads and writes return `DATABASE_RESIDENCY_BLOCKED` with the configuration reason and `retryable: false`; replication does not subscribe or advance its cursor. Other databases and the operations API remain available. `cluster_status.residency` reports the blocked database. Repair the configuration and restart the node to replay its retained tail. Repairing a system map or restarting an individual worker does not clear the quarantine. + +A peer's announced node name must match its authenticated identity or an alias explicitly bound by its configured route or node record. For a canonical hostname reached through an IP address, configure both identities: + +```yaml +replication: + routes: + - hostname: node-b + url: wss://192.0.2.20:9933 +``` + +The canonical name is used for topology and confirmations. An announcement cannot create an alias or claim another known node's identity. + ## Installed maps Each database has one versioned snapshot in `system.hdb_residency_slots`. The snapshot contains `database`, `version`, `digest`, `slots`, and complete, nonoverlapping `ranges` with exclusive `end` values. Tables may use different covered prefixes of the map. From 94dd0353d791dd17a9d5d599a9aad52c5738f90a Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Sat, 24 Oct 2026 21:58:40 -0600 Subject: [PATCH 14/32] Connect replica lifetime settings to declarative writer caches Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/database/schema.md | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/reference/database/schema.md b/reference/database/schema.md index be9e3f613..3a5b517e9 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -73,19 +73,19 @@ type MyTable @table { Optional arguments: -| Argument | Type | Default | Description | -| ------------------------ | --------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | -| `table` | `String` | type name | Override the table name | -| `database` | `String` | `"data"` | Database to place the table in | -| `expiration` | `Int` | — | Seconds until a record goes stale (useful for caching tables) | -| `eviction` | `Int` | `0` | Additional seconds after `expiration` before a record is physically removed | -| `scanInterval` | `Int` | `(expiration + eviction) / 4` | Seconds between eviction scans | -| `replicate` | `Boolean` | true | Enable replication of this table | -| `replicaExpiration` | `Float` | `3600` | Seconds a record fetched from another node is kept as a [local replica](#replicaexpiration-and-replicarefreshfraction); `0` stops new admissions | -| `replicaRefreshFraction` | `Float` | `0.5` | Fraction of `replicaExpiration` after which a read of a replica renews its lifetime; `1` never renews | -| `audit` | `Boolean` | `logging.auditLog` for ordinary tables | Enable the table's transaction log; new tables with `@fullText` require explicit `true` | -| `cacheControl` | `String` | — | `Cache-Control` header value emitted on anonymous GET/HEAD 200/304 responses for this table | -| `randomAccessFields` | `Boolean` | `storage.randomAccessFields` | [Pin this table's record encoding](#randomaccessfields) | +| Argument | Type | Default | Description | +| ------------------------ | --------- | -------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | +| `table` | `String` | type name | Override the table name | +| `database` | `String` | `"data"` | Database to place the table in | +| `expiration` | `Int` | — | Seconds until a record goes stale (useful for caching tables) | +| `eviction` | `Int` | `0` | Additional seconds after `expiration` before a record is physically removed | +| `scanInterval` | `Int` | `(expiration + eviction) / 4` | Seconds between eviction scans | +| `replicate` | `Boolean` | true | Enable replication of this table | +| `replicaExpiration` | `Float` | `3600` | Seconds to keep a [local residency replica](#replicaexpiration-and-replicarefreshfraction); `0` stops new admissions | +| `replicaRefreshFraction` | `Float` | `0.5` | Fraction of `replicaExpiration` after which a read of a replica renews its lifetime; `1` never renews | +| `audit` | `Boolean` | `logging.auditLog` for ordinary tables | Enable the table's transaction log; new tables with `@fullText` require explicit `true` | +| `cacheControl` | `String` | — | `Cache-Control` header value emitted on anonymous GET/HEAD 200/304 responses for this table | +| `randomAccessFields` | `Boolean` | `storage.randomAccessFields` | [Pin this table's record encoding](#randomaccessfields) | **`expiration`, `eviction`, and `scanInterval`** @@ -143,6 +143,8 @@ On RocksDB, schema rescans apply changes to already-loaded tables on other worke On a table whose records live on specific nodes (see [sharding](../replication/sharding.md#by-record-content-setresidency)), a node that does not hold a record fetches it from a node that does. The fetched record is kept locally as a _replica_: a complete copy that reads and searches on that node use without contacting another node again. A replica does not move the record and is never replicated onward. A write by an owner reaches the node as an invalidation, which turns the replica back into a stub, so the next read fetches the new version. Tables sharded by primary key only (`setResidencyById`) keep no replicas. +[Declarative residency](../replication/residency.md#ownership-reads-and-writes) tables with secondary indexes can retain replicas for either key- or attribute-derived slots. With positive `replicaExpiration`, a non-owner writer also keeps its complete write as a temporary replica under the same expiry and replacement rules. This local copy never counts as owner confirmation. Setting the lifetime to `0` does not drain existing replicas; see the [rollback procedure](../replication/residency.md#rollback). + - **`replicaExpiration`** — Seconds a replica is kept. A sweep runs every quarter of this time, with a minimum interval of one second, and reduces replicas past their lifetime back to their indexed stubs. A replica can still be served until the sweep reaches it; the next read after demotion fetches the record again. `0` disables new replica admissions and the sweep. A replica never outlives the record's own `expiration`, if it has one. - **`replicaRefreshFraction`** — A read of a replica after this fraction of `replicaExpiration` has passed renews the lifetime, so records that stay in use stay local. Must be greater than `0` and at most `1`; `1` gives every replica a fixed lifetime. From c568f5f697ab5f9abc59a8d2e3a14899e4657dbc Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Sat, 24 Oct 2026 23:32:14 -0600 Subject: [PATCH 15/32] Clarify quarantined declarations and audit blob retention costs Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/replication/residency.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/replication/residency.md b/reference/replication/residency.md index 234d2ae69..886dbd5f5 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -63,7 +63,7 @@ Owners store full records. With secondary indexes, other nodes keep index projec A key-policy table without secondary indexes stores nothing on non-owners; reads resolve the owner from the key. Such tables do not cache replicas. Attribute policies require a secondary index so a non-owner can retain the stored slot. If an attribute record's slot is unknown locally, DELETE returns 503 with a message beginning `Attribute residency slot is unknown`. This response does not distinguish a delayed projection from a never-created key or a purged tombstone; local absence cannot prove that the record is absent on its owners. If another node still has the slot metadata, send the deletion there. If no node has it, DELETE cannot succeed anywhere; retrying alone cannot fix that condition. Stop automatic retries and use application-level knowledge to establish whether the key ever existed. Local absence is not proof that it was deleted. -`replicateTo` cannot override a declarative policy; the installed slot map determines where full records are retained. A non-owner accepts a local write and sends the complete body through replication to the owners. On an indexed RocksDB table with positive `replicaExpiration`, it also retains the complete written body as a temporary local replica, allowing an immediate local read while delivery is pending. A newer invalidation replaces it, and the replica sweep expires it. With expiration zero or LMDB, it retains only the index projection; an index-free key policy retains nothing. A replica is a cache, so crash recovery may restore a projection instead. Ingest-heavy non-owners should budget space for approximately their serialized bytes written per second multiplied by `replicaExpiration`, or set expiration to zero. This requires a configured replication route to an owner. Cross-shard query fan-out and write forwarding are not provided by this directive. Application database branches do not support declarative residency because branches do not replicate. +`replicateTo` cannot override a declarative policy; the installed slot map determines where full records are retained. A non-owner accepts a local write and sends the complete body through replication to the owners. On an indexed RocksDB table with positive `replicaExpiration`, it also retains the complete written body as a temporary local replica, allowing an immediate local read while delivery is pending. A newer invalidation replaces it, and the replica sweep expires it. With expiration zero or LMDB, it retains only the index projection; an index-free key policy retains nothing. A replica is a cache, so crash recovery may restore a projection instead. Ingest-heavy non-owners should budget space for approximately their serialized bytes written per second multiplied by `replicaExpiration`, or set expiration to zero. File-backed audit bodies remain until audit retention permits reclamation, even after a replica expires; include that delivery history in the disk budget. Refetched cache files have their own replica lifetime. This requires a configured replication route to an owner. Cross-shard query fan-out and write forwarding are not provided by this directive. Application database branches do not support declarative residency because branches do not replicate. ### Attribute-slot changes @@ -87,7 +87,7 @@ Non-owner invalidation events contain only indexed fields. A subscription `rowFi ### Configuration failures and peer identity -Invalid residency configuration at startup quarantines the affected database before replay. Its reads and writes return `DATABASE_RESIDENCY_BLOCKED` with the configuration reason and `retryable: false`; replication does not subscribe or advance its cursor. Other databases and the operations API remain available. `cluster_status.residency` reports the blocked database. Repair the configuration and restart the node to replay its retained tail. Repairing a system map or restarting an individual worker does not clear the quarantine. +Invalid residency configuration at startup quarantines the affected database before replay. Its reads and writes return `DATABASE_RESIDENCY_BLOCKED` with the configuration reason and `retryable: false`; replication does not subscribe or advance its cursor. New tables declared in that database are deferred until a repaired restart; healthy tables in the same application schema still load. Other databases and the operations API remain available. `cluster_status.residency` reports the blocked database. Repair the configuration and restart the node to replay its retained tail. Repairing a system map or restarting an individual worker does not clear the quarantine. A peer's announced node name must match its authenticated identity or an alias explicitly bound by its configured route or node record. For a canonical hostname reached through an IP address, configure both identities: From 6fbcc0c8bb46bea5ef5e2b5298b4e3b4644435b8 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Sat, 24 Oct 2026 23:59:15 -0600 Subject: [PATCH 16/32] Clarify residency activation and transaction confirmation examples Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/configuration/options.md | 6 ++++-- reference/database/api.md | 4 ++++ reference/database/schema.md | 8 +++++--- reference/replication/residency.md | 6 ++++-- reference/replication/sharding.md | 7 +++---- 5 files changed, 20 insertions(+), 11 deletions(-) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 8efc2e67c..abfa1c83f 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -286,8 +286,10 @@ replication: - `securePort` — Secure replication port; _Default_: `9933` (changed from `9925` in v4.5.0) - `threads` — Worker threads dedicated to replication, which then own the replication port and every subscription and load no application code; requires `port` or `securePort`; see [Dedicated Replication Threads](../replication/overview.md#dedicated-replication-threads); _Default_: `0` (replication runs on the HTTP worker threads) (Added in: v5.4.0) - `enableRootCAs` — Verify against Node.js Mozilla CA store; _Default_: `true` -- `confirmationTimeout` — Timeout in milliseconds for explicit `replicatedConfirmation`, read when a waiter starts. Default: `900000`, or `blobTimeout` when greater. An explicitly configured value overrides that fallback, including a smaller value. Range: `1`–`2147483647`. (Added in: v5.4.0) -- `residencyConfirmationTimeout` — Timeout in milliseconds for automatic new-owner confirmation on an attribute-slot change. Default: `30000`. Independent of blob-transfer timeout. A timeout returns an uncertain-outcome 504 after local commit; it does not roll back the write. Range: `1`–`2147483647`. (Added in: v5.4.0) +- `pingInterval` — Interval between replication keep-alive pings (ms); _Default_: `30000` +- `pingTimeout` — Time without received socket activity before a replication connection is considered dead (ms); _Default_: twice `pingInterval`. Automatic slot-move admission also requires a new owner with received activity inside this window. +- `confirmationTimeout` — Timeout in milliseconds for explicit `replicatedConfirmation`, read when a waiter starts. _Default_: `900000`, or `blobTimeout` when greater. An explicitly configured value overrides that fallback, including a smaller value. Range: `1`–`2147483647`. (Added in: v5.4.0) +- `residencyConfirmationTimeout` — Timeout in milliseconds for automatic new-owner confirmation on an attribute-slot change. _Default_: `30000`. Independent of blob-transfer timeout. A timeout returns an uncertain-outcome 504 after local commit; it does not roll back the write. Range: `1`–`2147483647`. (Added in: v5.4.0) - `blobTimeout` — Blob transfer timeout (ms); _Default_: `120000` - `blobGapReconnectMs` — Interval (ms) for the blob-gap watchdog: when a transient blob save failure pins a replication resume cursor, the connection is forced to reconnect on this cadence so the gapped blob is re-streamed and, during a bulk copy, the copy resumes from the last banked cursor. Lower values heal gaps faster at the cost of more reconnects on a link whose faults never heal; _Default_: the `blobTimeout` value - `copyCursorFlushBytes` — Bytes of applied bulk-copy data between durable flushes of the copy resume cursor (RocksDB); _Default_: `67108864` diff --git a/reference/database/api.md b/reference/database/api.md index 6197ec4a6..646c7a229 100644 --- a/reference/database/api.md +++ b/reference/database/api.md @@ -134,6 +134,10 @@ Two details matter outside a request context: - **`await` the `transaction()` call, and catch it.** An unawaited call lets the timer callback finish before the commit resolves, so a failed write is never observed; awaiting without a `catch` still leaves the rejection unhandled, because nothing awaits the callback itself. - **Guard against overlap when a job can outlast its interval.** A timer fires on schedule regardless of whether the previous run has committed, so two runs would open two transactions over the same rows. The `running` flag above skips a tick instead. +### Replication confirmation + +Set `replicatedConfirmation` on a fresh context before calling `transaction()` to request confirmation for its writes. Pass the callback's `txn` argument to table methods to bind them explicitly. Reusing a context with an active transaction does not change that transaction's confirmation settings. See [replication confirmation](../replication/sharding.md#programmatic-replication-control) for the independent-transaction example and uncertain-outcome behavior. + ### Nesting If `transaction()` is called with a context that already has an active transaction, it reuses that transaction, executes the callback immediately, and returns. This makes `transaction()` safe to call defensively to ensure a transaction is always active. diff --git a/reference/database/schema.md b/reference/database/schema.md index 3a5b517e9..ef24ade53 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -143,11 +143,11 @@ On RocksDB, schema rescans apply changes to already-loaded tables on other worke On a table whose records live on specific nodes (see [sharding](../replication/sharding.md#by-record-content-setresidency)), a node that does not hold a record fetches it from a node that does. The fetched record is kept locally as a _replica_: a complete copy that reads and searches on that node use without contacting another node again. A replica does not move the record and is never replicated onward. A write by an owner reaches the node as an invalidation, which turns the replica back into a stub, so the next read fetches the new version. Tables sharded by primary key only (`setResidencyById`) keep no replicas. -[Declarative residency](../replication/residency.md#ownership-reads-and-writes) tables with secondary indexes can retain replicas for either key- or attribute-derived slots. With positive `replicaExpiration`, a non-owner writer also keeps its complete write as a temporary replica under the same expiry and replacement rules. This local copy never counts as owner confirmation. Setting the lifetime to `0` does not drain existing replicas; see the [rollback procedure](../replication/residency.md#rollback). - - **`replicaExpiration`** — Seconds a replica is kept. A sweep runs every quarter of this time, with a minimum interval of one second, and reduces replicas past their lifetime back to their indexed stubs. A replica can still be served until the sweep reaches it; the next read after demotion fetches the record again. `0` disables new replica admissions and the sweep. A replica never outlives the record's own `expiration`, if it has one. - **`replicaRefreshFraction`** — A read of a replica after this fraction of `replicaExpiration` has passed renews the lifetime, so records that stay in use stay local. Must be greater than `0` and at most `1`; `1` gives every replica a fixed lifetime. +[Declarative residency](../replication/residency.md#ownership-reads-and-writes) tables with secondary indexes can retain replicas for either key- or attribute-derived slots. With positive `replicaExpiration`, a non-owner writer also keeps its complete write as a temporary replica under the same expiry and replacement rules. This local copy never counts as owner confirmation. Setting the lifetime to `0` does not drain existing replicas; see the [rollback procedure](../replication/residency.md#rollback). + ```graphql # Keep fetched records for a day; a read after six hours renews them type Order @table(replicaExpiration: 86400, replicaRefreshFraction: 0.25) { @@ -775,7 +775,9 @@ export class Reports extends tables.Reports { For vector queries, `rowFilter` participates in HNSW traversal. A caller therefore receives the k nearest _matching_ records rather than "nearest k, minus filtered records." `rowFilter` can apply to every candidate visited; prefer indexed query conditions for predicates that can be expressed declaratively. -`rowFilter` applies only when a non-raw `put` or `invalidate` event carries an authoritative row value. Delete tombstones, value-less invalidations, raw events, and published messages may not contain the complete current record. A subscription that needs such events can also provide an `eventFilter`: +On a declarative residency non-owner, invalidation values contain only the local index projection. A `rowFilter` on an unindexed field therefore sees it as missing; see the [subscription limitation](../replication/residency.md#ownership-reads-and-writes). + +`rowFilter` applies only when a non-raw `put` or `invalidate` event carries a row value. Delete tombstones, value-less invalidations, raw events, and published messages may not contain the complete current record. A subscription that needs such events can also provide an `eventFilter`: ```javascript subscribe(request) { diff --git a/reference/replication/residency.md b/reference/replication/residency.md index 886dbd5f5..47e107d88 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -63,7 +63,7 @@ Owners store full records. With secondary indexes, other nodes keep index projec A key-policy table without secondary indexes stores nothing on non-owners; reads resolve the owner from the key. Such tables do not cache replicas. Attribute policies require a secondary index so a non-owner can retain the stored slot. If an attribute record's slot is unknown locally, DELETE returns 503 with a message beginning `Attribute residency slot is unknown`. This response does not distinguish a delayed projection from a never-created key or a purged tombstone; local absence cannot prove that the record is absent on its owners. If another node still has the slot metadata, send the deletion there. If no node has it, DELETE cannot succeed anywhere; retrying alone cannot fix that condition. Stop automatic retries and use application-level knowledge to establish whether the key ever existed. Local absence is not proof that it was deleted. -`replicateTo` cannot override a declarative policy; the installed slot map determines where full records are retained. A non-owner accepts a local write and sends the complete body through replication to the owners. On an indexed RocksDB table with positive `replicaExpiration`, it also retains the complete written body as a temporary local replica, allowing an immediate local read while delivery is pending. A newer invalidation replaces it, and the replica sweep expires it. With expiration zero or LMDB, it retains only the index projection; an index-free key policy retains nothing. A replica is a cache, so crash recovery may restore a projection instead. Ingest-heavy non-owners should budget space for approximately their serialized bytes written per second multiplied by `replicaExpiration`, or set expiration to zero. File-backed audit bodies remain until audit retention permits reclamation, even after a replica expires; include that delivery history in the disk budget. Refetched cache files have their own replica lifetime. This requires a configured replication route to an owner. Cross-shard query fan-out and write forwarding are not provided by this directive. Application database branches do not support declarative residency because branches do not replicate. +`replicateTo` is ignored for placement on a declarative table; the installed slot map determines where full records are retained. A non-owner accepts a local write and sends the complete body through replication to the owners. On an indexed RocksDB table with positive `replicaExpiration`, it also retains the complete written body as a temporary local replica, allowing an immediate local read while delivery is pending. A newer invalidation replaces it, and the replica sweep expires it. With expiration zero or LMDB, it retains only the index projection; an index-free key policy retains nothing. A replica is a cache, so crash recovery may restore a projection instead. Ingest-heavy non-owners should budget space for approximately their serialized bytes written per second multiplied by `replicaExpiration`, or set expiration to zero. File-backed audit bodies remain until audit retention permits reclamation, even after a replica expires; include that delivery history in the disk budget. Refetched cache files have their own replica lifetime. This requires a configured replication route to an owner. Cross-shard query fan-out and write forwarding are not provided by this directive. Application database branches do not support declarative residency because branches do not replicate. ### Attribute-slot changes @@ -75,7 +75,7 @@ type RegionalItem @table @residency(slots: 10, slotFrom: "region", byAttribute: } ``` -Changing `region` sends the complete canonical body to the new owners and invalidates old owners at the same version. This handoff uses full-record replacement ordering: an older concurrent patch does not merge into the replacement. Use application concurrency controls when writers can update a record while its slot changes. When the writer is outside the new owner set, success requires at least one new owner to confirm durable receipt. If no new owner has an active subscription and recent received activity within `replication.pingTimeout`, the change is refused before commit. This can also refuse an owner whose base copy is not progressing. +Changing `region` sends the complete canonical body to the new owners and invalidates old owners at the same version. This handoff uses full-record replacement ordering: an older concurrent patch does not merge into the replacement. Use application concurrency controls when writers can update a record while its slot changes. When the writer is outside the new owner set, success requires at least one new owner to confirm durable receipt. If no new owner has an active subscription and recent received activity within [`replication.pingTimeout`](../configuration/options.md#replication), the change is refused before commit. This can also refuse an owner whose base copy is not progressing. Automatic slot-move confirmation waits up to `replication.residencyConfirmationTimeout` (default `30000` ms). Explicit `replicatedConfirmation` uses `replication.confirmationTimeout` instead (default `900000` ms, raised to `blobTimeout` when the confirmation setting is omitted). Both settings are read when a confirmation starts. An explicit setting can lower that timeout independently of blob transfers. Slow transfers can therefore exceed the automatic timeout even on a healthy link. @@ -104,6 +104,8 @@ The canonical name is used for topology and confirmations. An announcement canno Each database has one versioned snapshot in `system.hdb_residency_slots`. The snapshot contains `database`, `version`, `digest`, `slots`, and complete, nonoverlapping `ranges` with exclusive `end` values. Tables may use different covered prefixes of the map. +First activation requires a writable node; it may need to persist both the map and activation metadata. Activate the policy before switching a node to read-only mode. + For a new declarative table with no map, first activation installs version `1` from only the declared slot count and sorted `replication.shards`: slot `k` maps to `shards[k % shards.length]`. Equal inputs produce the same snapshot and digest on simultaneous installers. The installed snapshot is then authoritative. Changes to the configured shard list never regenerate it. The first table activated supplies the default slot count; Harper does not choose the maximum across a schema. For tables with different slot counts, install one custom map covering the largest count and wait for the same snapshot on every node before deploying the tables. A later table needing more slots requires an explicit `initialize_residency_slots` extension that preserves every active assignment. It never extends the default automatically. Do not run different initial schema deployments concurrently. diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index 4d2f58629..250e5b15e 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -111,8 +111,8 @@ export class Order extends tables.Order { ```javascript import { tables, transaction } from 'harper'; -await transaction({ replicatedConfirmation: 1, replicateTo: 2 }, async () => { - await tables.Order.put(order); +await transaction({ replicatedConfirmation: 1, replicateTo: 2 }, async (txn) => { + await tables.Order.put(order, txn); }); ``` @@ -286,7 +286,6 @@ X-Replicate-From: none ## See Also -- [Declarative Residency](./residency.md) - +- [Declarative Residency](./residency.md) — Fixed slot policies and authoritative database maps - [Replication Overview](./overview.md) — How Harper's replication system works - [Clustering Operations](./clustering.md) — Operations API for managing cluster nodes From 99202a8f4f56c5bfe06456bc1b1f3a077ce75c80 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Sun, 25 Oct 2026 00:12:40 -0600 Subject: [PATCH 17/32] Link the subscription projection limitation directly Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/database/schema.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/database/schema.md b/reference/database/schema.md index ef24ade53..256079ddc 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -775,7 +775,7 @@ export class Reports extends tables.Reports { For vector queries, `rowFilter` participates in HNSW traversal. A caller therefore receives the k nearest _matching_ records rather than "nearest k, minus filtered records." `rowFilter` can apply to every candidate visited; prefer indexed query conditions for predicates that can be expressed declaratively. -On a declarative residency non-owner, invalidation values contain only the local index projection. A `rowFilter` on an unindexed field therefore sees it as missing; see the [subscription limitation](../replication/residency.md#ownership-reads-and-writes). +On a declarative residency non-owner, invalidation values contain only the local index projection. A `rowFilter` on an unindexed field therefore sees it as missing; see the [subscription limitation](../replication/residency.md#subscription-filters). `rowFilter` applies only when a non-raw `put` or `invalidate` event carries a row value. Delete tombstones, value-less invalidations, raw events, and published messages may not contain the complete current record. A subscription that needs such events can also provide an `eventFilter`: From d196d2cdb461791fe25d33b9c603097c4c873722 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Sun, 25 Oct 2026 02:48:33 -0600 Subject: [PATCH 18/32] docs: link confirmation outcomes directly Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/database/api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/database/api.md b/reference/database/api.md index 646c7a229..ea01768e0 100644 --- a/reference/database/api.md +++ b/reference/database/api.md @@ -136,7 +136,7 @@ Two details matter outside a request context: ### Replication confirmation -Set `replicatedConfirmation` on a fresh context before calling `transaction()` to request confirmation for its writes. Pass the callback's `txn` argument to table methods to bind them explicitly. Reusing a context with an active transaction does not change that transaction's confirmation settings. See [replication confirmation](../replication/sharding.md#programmatic-replication-control) for the independent-transaction example and uncertain-outcome behavior. +Set `replicatedConfirmation` on a fresh context before calling `transaction()` to request confirmation for its writes. Pass the callback's `txn` argument to table methods to bind them explicitly. Reusing a context with an active transaction does not change that transaction's confirmation settings. See the [independent-transaction example](../replication/sharding.md#programmatic-replication-control) and [uncertain-outcome behavior](../replication/sharding.md#confirmation-outcomes). ### Nesting From 8edb11abb2082acf68ee45f2727b1194e606cc06 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Sun, 25 Oct 2026 02:54:40 -0600 Subject: [PATCH 19/32] docs: clarify new-owner confirmation scope Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/replication/clustering.md | 2 +- release-notes/v5-lincoln/5.4.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/replication/clustering.md b/reference/replication/clustering.md index 8573ecc2f..a7dc747ac 100644 --- a/reference/replication/clustering.md +++ b/reference/replication/clustering.md @@ -215,7 +215,7 @@ The top-level `cluster_status.residency` field is keyed by database and describe | `blocked` | Whether a reported configuration or map-agreement condition blocks placement. | | `errors` | Reasons, such as differing peer lists, a local shard outside its list, a missing required list/map, or a refused peer map. | -A clear status is not proof that an owner is connected; write admission also checks the required replication route and, for a non-owner attribute-slot transition, a connected new owner. See [configuration agreement](./sharding.md#complete-shard-id-set) and [map recovery](./residency.md#installed-maps). +A clear status is not proof that an owner is connected; write admission also checks the required replication route and, for an attribute-slot transition from a writer outside the new owner set, a connected new owner. See [configuration agreement](./sharding.md#complete-shard-id-set) and [map recovery](./residency.md#installed-maps). ### Residency Map Operations diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index b7a79fcdf..b803155f1 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -24,7 +24,7 @@ These corrections are not released yet. ### Declarative Residency -`@residency` applies a fixed key- or attribute-based slot policy on every thread, including dedicated replication workers. Per-database slot maps remain authoritative after initialization; `replication.shards` supplies the complete deployment shard set for a first default map. Super-user map initialization and read-only conversion checks support adopting an equivalent legacy by-id policy without rewriting records. Attribute-slot changes from a non-owner require a new owner's confirmation. See [Declarative Residency](/reference/v5/replication/residency), including its storage-format and rollback requirements. +`@residency` applies a fixed key- or attribute-based slot policy on every thread, including dedicated replication workers. Per-database slot maps remain authoritative after initialization; `replication.shards` supplies the complete deployment shard set for a first default map. Super-user map initialization and read-only conversion checks support adopting an equivalent legacy by-id policy without rewriting records. Attribute-slot changes from a writer outside the new owner set require a new owner's confirmation. See [Declarative Residency](/reference/v5/replication/residency), including its storage-format and rollback requirements. ### LMDB Replication Confirmation From 7a2a361373877a078b058c20ee9d35cf616f4600 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Sun, 25 Oct 2026 08:42:52 -0600 Subject: [PATCH 20/32] Document database-wide sharding and regional slot expressions Explain hashed tenant affinity, logical regional slots, immutable policy, older residency APIs, and the spread follow-up in #1122. Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/configuration/options.md | 2 +- reference/database/schema.md | 8 +-- reference/database/system-tables.md | 2 +- reference/operations-api/operations.md | 30 ++++---- reference/replication/clustering.md | 8 +-- reference/replication/overview.md | 2 +- reference/replication/residency.md | 96 ++++++++++++++++++-------- reference/replication/sharding.md | 20 +++--- release-notes/v5-lincoln/5.4.md | 4 +- sidebarsReference.ts | 4 +- 10 files changed, 109 insertions(+), 67 deletions(-) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index abfa1c83f..d62764e27 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -289,7 +289,7 @@ replication: - `pingInterval` — Interval between replication keep-alive pings (ms); _Default_: `30000` - `pingTimeout` — Time without received socket activity before a replication connection is considered dead (ms); _Default_: twice `pingInterval`. Automatic slot-move admission also requires a new owner with received activity inside this window. - `confirmationTimeout` — Timeout in milliseconds for explicit `replicatedConfirmation`, read when a waiter starts. _Default_: `900000`, or `blobTimeout` when greater. An explicitly configured value overrides that fallback, including a smaller value. Range: `1`–`2147483647`. (Added in: v5.4.0) -- `residencyConfirmationTimeout` — Timeout in milliseconds for automatic new-owner confirmation on an attribute-slot change. _Default_: `30000`. Independent of blob-transfer timeout. A timeout returns an uncertain-outcome 504 after local commit; it does not roll back the write. Range: `1`–`2147483647`. (Added in: v5.4.0) +- `shardingConfirmationTimeout` — Timeout in milliseconds for automatic new-owner confirmation on an attribute-slot change. _Default_: `30000`. Independent of blob-transfer timeout. A timeout returns an uncertain-outcome 504 after local commit; it does not roll back the write. Range: `1`–`2147483647`. (Added in: v5.4.0) - `blobTimeout` — Blob transfer timeout (ms); _Default_: `120000` - `blobGapReconnectMs` — Interval (ms) for the blob-gap watchdog: when a transient blob save failure pins a replication resume cursor, the connection is forced to reconnect on this cadence so the gapped blob is re-streamed and, during a bulk copy, the copy resumes from the last banked cursor. Lower values heal gaps faster at the cost of more reconnects on a link whose faults never heal; _Default_: the `blobTimeout` value - `copyCursorFlushBytes` — Bytes of applied bulk-copy data between durable flushes of the copy resume cursor (RocksDB); _Default_: `67108864` diff --git a/reference/database/schema.md b/reference/database/schema.md index 256079ddc..736fd07f4 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -146,7 +146,7 @@ On a table whose records live on specific nodes (see [sharding](../replication/s - **`replicaExpiration`** — Seconds a replica is kept. A sweep runs every quarter of this time, with a minimum interval of one second, and reduces replicas past their lifetime back to their indexed stubs. A replica can still be served until the sweep reaches it; the next read after demotion fetches the record again. `0` disables new replica admissions and the sweep. A replica never outlives the record's own `expiration`, if it has one. - **`replicaRefreshFraction`** — A read of a replica after this fraction of `replicaExpiration` has passed renews the lifetime, so records that stay in use stay local. Must be greater than `0` and at most `1`; `1` gives every replica a fixed lifetime. -[Declarative residency](../replication/residency.md#ownership-reads-and-writes) tables with secondary indexes can retain replicas for either key- or attribute-derived slots. With positive `replicaExpiration`, a non-owner writer also keeps its complete write as a temporary replica under the same expiry and replacement rules. This local copy never counts as owner confirmation. Setting the lifetime to `0` does not drain existing replicas; see the [rollback procedure](../replication/residency.md#rollback). +[Sharding](../replication/residency.md#ownership-reads-and-writes) tables with secondary indexes can retain replicas for either key- or attribute-derived slots. With positive `replicaExpiration`, a non-owner writer also keeps its complete write as a temporary replica under the same expiry and replacement rules. This local copy never counts as owner confirmation. Setting the lifetime to `0` does not drain existing replicas; see the [rollback procedure](../replication/residency.md#rollback). ```graphql # Keep fetched records for a day; a read after six hours renews them @@ -240,11 +240,11 @@ export class Product extends tables.Product { See [REST Headers / Cache-Control](../rest/headers.md#cache-control) for the full caching behavior and how `Cache-Control` interacts with authentication headers. -### `@residency` +### `@sharded` -Defines an immutable placement policy on a schema-defined table. `slots` is required (`1`–`65536`); `slotFrom` optionally supplies a deterministic JavaScript expression; `byAttribute: true` evaluates the record and requires a secondary index. The database's installed map resolves each slot to a deployment shard. See [Declarative Residency](../replication/residency.md) for examples, configuration, conversion and rollback. +Defines an immutable placement policy on a schema-defined table. `@sharded` hashes the primary key; `by: "tenantId"` hashes that attribute's non-null value with the same hash. `slotFrom` optionally chooses the integer logical slot with a compiled deterministic expression. `slots` is a database-wide value (`1` through `65536`): omit it to adopt the installed map's count, or `1024` if no map exists; an explicit different count is refused. Attribute policies require a secondary index. The database map resolves each slot to a deployment shard. See [Sharding](../replication/residency.md) for tenant affinity, regional placement, configuration, conversion, limitations and rollback. ### `@export` @@ -775,7 +775,7 @@ export class Reports extends tables.Reports { For vector queries, `rowFilter` participates in HNSW traversal. A caller therefore receives the k nearest _matching_ records rather than "nearest k, minus filtered records." `rowFilter` can apply to every candidate visited; prefer indexed query conditions for predicates that can be expressed declaratively. -On a declarative residency non-owner, invalidation values contain only the local index projection. A `rowFilter` on an unindexed field therefore sees it as missing; see the [subscription limitation](../replication/residency.md#subscription-filters). +On a non-owner for a sharded table, invalidation values contain only the local index projection. A `rowFilter` on an unindexed field therefore sees it as missing; see the [subscription limitation](../replication/residency.md#subscription-filters). `rowFilter` applies only when a non-raw `put` or `invalidate` event carries a row value. Delete tombstones, value-less invalidations, raw events, and published messages may not contain the complete current record. A subscription that needs such events can also provide an `eventFilter`: diff --git a/reference/database/system-tables.md b/reference/database/system-tables.md index ce4b0937e..c8c8d3c39 100644 --- a/reference/database/system-tables.md +++ b/reference/database/system-tables.md @@ -137,7 +137,7 @@ This table is managed automatically by the Data Loader. No direct interaction is -Stores one authoritative [residency map](../replication/residency.md#installed-maps) snapshot per database, keyed by `database`, with `version`, `digest`, `slots`, and complete nonoverlapping `ranges`. Install custom maps through the super-user `initialize_residency_slots` operation. Active assignments cannot move, and deleting an installed map is refused. A lost or corrupted snapshot must be recovered; it is never regenerated from configuration. +Stores one authoritative [shard map](../replication/residency.md#installed-maps) snapshot per database, keyed by `database`, with `version`, `digest`, `slots`, and complete nonoverlapping `ranges`. Install custom maps through the super-user `initialize_shard_map` operation. Active assignments cannot move, and deleting an installed map is refused. A lost or corrupted snapshot must be recovered; it is never regenerated from configuration. ### `hdb_nodes` diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index b905097f4..541063728 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -1780,18 +1780,18 @@ Operations for configuring and managing Harper cluster replication. Detailed documentation: [Replication & Clustering](../replication/clustering.md) -| Operation | Description | Role Required | -| ---------------------------- | ---------------------------------------------------------------- | ------------- | -| `add_node` | Adds a Harper instance to the cluster | super_user | -| `update_node` | Modifies an existing node's subscriptions | super_user | -| `remove_node` | Removes a node from the cluster | super_user | -| `initialize_residency_slots` | Installs a custom database slot map without moving records | super_user | -| `verify_residency_slots` | Compares a legacy placement function with a proposed slot policy | super_user | -| `cluster_status` | Returns current cluster connection status | super_user | -| `configure_cluster` | Bulk-creates/resets cluster subscriptions across multiple nodes | super_user | -| `cluster_set_routes` | Adds routes to the replication routes config (PATCH/upsert) | super_user | -| `cluster_get_routes` | Returns the current replication routes config | super_user | -| `cluster_delete_routes` | Removes routes from the replication routes config | super_user | +| Operation | Description | Role Required | +| ----------------------- | ---------------------------------------------------------------- | ------------- | +| `add_node` | Adds a Harper instance to the cluster | super_user | +| `update_node` | Modifies an existing node's subscriptions | super_user | +| `remove_node` | Removes a node from the cluster | super_user | +| `initialize_shard_map` | Installs a custom database slot map without moving records | super_user | +| `verify_shard_map` | Compares a legacy placement function with a proposed slot policy | super_user | +| `cluster_status` | Returns current cluster connection status | super_user | +| `configure_cluster` | Bulk-creates/resets cluster subscriptions across multiple nodes | super_user | +| `cluster_set_routes` | Adds routes to the replication routes config (PATCH/upsert) | super_user | +| `cluster_get_routes` | Returns the current replication routes config | super_user | +| `cluster_delete_routes` | Removes routes from the replication routes config | super_user | ### `add_node` @@ -1806,13 +1806,13 @@ Adds a remote Harper node to the cluster. If `subscriptions` are not provided, a } ``` -### `initialize_residency_slots` +### `initialize_shard_map` POST; requires `super_user`. Takes `database`, `slots` and complete nonoverlapping `ranges` (`start`, exclusive `end`, `shard`). Returns the installed versioned snapshot. Installs metadata only and refuses changes to active slot assignments. See [Installed maps](../replication/residency.md#installed-maps). -### `verify_residency_slots` +### `verify_shard_map` @@ -1820,7 +1820,7 @@ POST; requires `super_user`. Takes `database`, `table` and a proposed key-derive ### `cluster_status` -Returns connection state for all cluster nodes, including per-database socket status and replication timing statistics (`lastCommitConfirmed`, `lastReceivedRemoteTime`, `lastReceivedLocalTime`). Its top-level `residency` object reports per-database map identity and placement blockers; see [Declarative Residency Status](../replication/clustering.md#declarative-residency-status). +Returns connection state for all cluster nodes, including per-database socket status and replication timing statistics (`lastCommitConfirmed`, `lastReceivedRemoteTime`, `lastReceivedLocalTime`). Its top-level `sharding` object reports per-database map identity and placement blockers; see [Sharding Status](../replication/clustering.md#sharding-status). ```json { "operation": "cluster_status" } diff --git a/reference/replication/clustering.md b/reference/replication/clustering.md index a7dc747ac..ad88bdd4f 100644 --- a/reference/replication/clustering.md +++ b/reference/replication/clustering.md @@ -185,7 +185,7 @@ Returns an array of status objects from the cluster, including active WebSocket ], "node_name": "server-1.domain.com", "is_enabled": true, - "residency": {} + "sharding": {} } ``` @@ -202,11 +202,11 @@ While a database is taking its initial full copy, these timing fields hold the l --- -### Declarative Residency Status +### Sharding Status -The top-level `cluster_status.residency` field is keyed by database and describes active or attempted declarative tables. It is empty in the legacy-table example above. Each entry has: +The top-level `cluster_status.sharding` field is keyed by database and describes active or attempted declarative tables. It is empty in the legacy-table example above. Each entry has: | Field | Meaning | | ------------------- | -------------------------------------------------------------------------------------------------------------------------- | @@ -219,7 +219,7 @@ A clear status is not proof that an owner is connected; write admission also che ### Residency Map Operations -The POST operations [`initialize_residency_slots`](./residency.md#installed-maps) and [`verify_residency_slots`](./residency.md#3-compare-placement-on-every-node) require `super_user`. The former installs a snapshot without moving records; the latter compares the actual legacy function against a proposed key policy without modifying data. Their request examples, parameters and result fields are in the linked procedures. +The POST operations [`initialize_shard_map`](./residency.md#installed-maps) and [`verify_shard_map`](./residency.md#3-compare-placement-on-every-node) require `super_user`. The former installs a snapshot without moving records; the latter compares the actual legacy function against a proposed key policy without modifying data. Their request examples, parameters and result fields are in the linked procedures. --- diff --git a/reference/replication/overview.md b/reference/replication/overview.md index 5ed2acb32..33d258f14 100644 --- a/reference/replication/overview.md +++ b/reference/replication/overview.md @@ -99,7 +99,7 @@ With dedicated replication threads: - On macOS and Windows, which lack `SO_REUSEPORT`, only one replication thread listens on the replication port; the others carry outbound subscriptions. - Changing the number of threads takes effect only on a full `restart` of Harper; `restart_service` does not start or stop replication threads. An unscoped `restart_service` does restart the existing replication threads, and deploying or dropping a component does not. -Schema-defined [declarative residency](./residency.md) runs on application, operations and dedicated replication threads; it is an alternative to the thread-local residency setters. +Schema-defined [sharding](./residency.md) runs on application, operations and dedicated replication threads; it is an alternative to the thread-local residency setters. ## Securing Connections diff --git a/reference/replication/residency.md b/reference/replication/residency.md index 47e107d88..3746ab19d 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -1,12 +1,12 @@ --- -title: Declarative residency +title: Schema-defined sharding --- -# Declarative residency +# Schema-defined sharding -`@residency` fixes a table's placement policy in its schema. Every application, operations, and replication thread applies the same policy. A key or record selects a slot, the database's installed slot map selects a shard, and `replication.shard` identifies that shard's current nodes. +`@sharded` fixes a table's placement policy in its schema. Every application, operations, and replication thread applies the same policy. A key or record selects a slot, the database's installed slot map selects a shard, and `replication.shard` identifies that shard's current nodes. Upgrade every schema and data writer before activation. Wire compatibility with earlier peers allows delivery and reads; it does not make an earlier writer enforce the policy. Fallback to peers that cannot advertise a shard-list digest requires an installed custom map and no configured list on either side; otherwise setup disagrees and placement is blocked. @@ -26,58 +26,88 @@ Give shard `1`'s nodes `shard: 1` and the same `shards` list. The list belongs t ## Schema directive ```graphql -type Item @table @residency(slots: 1024) { +type Item @table @sharded { id: ID @primaryKey category: String @indexed body: String } ``` -| Argument | Meaning | -| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `slots` | Required integer from `1` through `65536`. | -| `slotFrom` | Optional JavaScript expression returning an integer from `0` through `slots - 1`. Key policies expose `id`. | -| `byAttribute` | Default `false`. When `true`, the expression also exposes `record` and its attribute names, and the slot is stored with the record. Requires a secondary index. | +| Argument | Meaning | +| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `slots` | Optional database-wide integer from `1` through `65536`. Uses the installed map's count, or `1024` when no map exists. An explicit count must equal the database map's count. | +| `by` | Optional attribute name. Its non-null value is hashed instead of the primary key. Requires a secondary index. | +| `slotFrom` | Optional deterministic JavaScript expression returning an integer in `[0, slots)`. Without `by`, it exposes `id`; with `by`, it also exposes `record` and its attribute names. | Declarative tables require [per-table audit logging](../database/transaction.md#enabling-the-transaction-log-per-table) and table replication. Harper refuses activation with `@table(audit: false)`, effective `logging.auditLog: false`, or `@table(replicate: false)`, and refuses disabling these table settings later. The replication route must also reach an owner. -Without `slotFrom`, the slot is the unsigned FNV-1a 32-bit hash of the key's ordered-binary encoding, modulo `slots`. Its offset basis is `2166136261` and its multiplier is `16777619`. +Without `slotFrom`, the slot is the unsigned FNV-1a 32-bit hash of the primary key's ordered-binary encoding, modulo the database slot count. With `by`, the same hash uses that attribute's value instead. Strings and numbers are supported; a null or missing value is refused. The hash's offset basis is `2166136261` and its multiplier is `16777619`. + +`@sharded(by: "tenantId")` keeps each tenant's records together: + +```graphql +type TenantItem @table(database: "tenant_data") @sharded(by: "tenantId") { + id: ID @primaryKey + tenantId: String @indexed + body: String +} +``` + +Different record keys with the same `tenantId` select the same slot on every node. The database map then resolves that slot to its current shard. An explicit key policy can preserve an existing partition function: ```graphql -type Item @table @residency(slots: 10, slotFrom: "Number(id.slice(-1))") { +type Item @table @sharded(slots: 10, slotFrom: "Number(id.slice(-1))") { id: ID @primaryKey category: String @indexed body: String } ``` -Expressions must be self-contained, deterministic, and synchronous. They compile on every thread without application helpers. An expression that throws, returns a Promise, or produces a non-integer or out-of-range result refuses the write. The last-digit example assumes string keys, matching the existing function being converted. Schema expressions are trusted code, as with computed expressions; they are not a sandbox. +Expressions must be self-contained, deterministic, and synchronous. They compile on every thread without application helpers. An expression that throws, returns a Promise, or produces a non-integer or out-of-range result refuses the write before commit. Invalid-result errors name the result and database slot count. The `by` value must still be non-null when an expression is supplied. The last-digit example assumes string keys, matching the existing function being converted. Schema expressions are trusted code, as with computed expressions; they are not a sandbox. -The policy is immutable once activated, including removal of the directive. Redeploying the same policy is idempotent. Calling `setResidency` or `setResidencyById` on a declarative table fails; remove those calls when converting. Existing tables can adopt a key-derived policy with an equivalent custom map. Attribute policies must be declared on a new table because existing rows have no stored slots. +The policy is immutable once activated, including changing `by`, changing or adding/removing `slotFrom`, and removing the directive. These changes could move existing records and are not a slot-migration operation. Redeploying the same policy is idempotent. Calling `setResidency` or `setResidencyById` on a declarative table fails; remove those calls when converting. Existing tables can adopt a key-derived policy with an equivalent custom map. Attribute policies must be declared on a new table because existing rows have no stored slots. ## Ownership, reads, and writes Owners store full records. With secondary indexes, other nodes keep index projections and receive invalidations. Reads fetch the full body from an owner. RocksDB can cache a temporary replica; LMDB proxies the read without caching it. Replicas retain the canonical record identity and do not become owners or count toward owner confirmation. -A key-policy table without secondary indexes stores nothing on non-owners; reads resolve the owner from the key. Such tables do not cache replicas. Attribute policies require a secondary index so a non-owner can retain the stored slot. If an attribute record's slot is unknown locally, DELETE returns 503 with a message beginning `Attribute residency slot is unknown`. This response does not distinguish a delayed projection from a never-created key or a purged tombstone; local absence cannot prove that the record is absent on its owners. If another node still has the slot metadata, send the deletion there. If no node has it, DELETE cannot succeed anywhere; retrying alone cannot fix that condition. Stop automatic retries and use application-level knowledge to establish whether the key ever existed. Local absence is not proof that it was deleted. +A key-policy table without secondary indexes stores nothing on non-owners; reads resolve the owner from the key. Such tables do not cache replicas. Attribute policies require a secondary index so a non-owner can retain the stored slot. If an attribute record's slot is unknown locally, DELETE returns 503 with a message beginning `Attribute sharding slot is unknown`. This response does not distinguish a delayed projection from a never-created key or a purged tombstone; local absence cannot prove that the record is absent on its owners. If another node still has the slot metadata, send the deletion there. If no node has it, DELETE cannot succeed anywhere; retrying alone cannot fix that condition. Stop automatic retries and use application-level knowledge to establish whether the key ever existed. Local absence is not proof that it was deleted. + +`replicateTo` is ignored for placement on a declarative table; the installed slot map determines where full records are retained. A non-owner accepts a local write and sends the complete body through replication to the owners. On an indexed RocksDB table with positive `replicaExpiration`, it also retains the complete written body as a temporary local replica, allowing an immediate local read while delivery is pending. A newer invalidation replaces it, and the replica sweep expires it. With expiration zero or LMDB, it retains only the index projection; an index-free key policy retains nothing. A replica is a cache, so crash recovery may restore a projection instead. Ingest-heavy non-owners should budget space for approximately their serialized bytes written per second multiplied by `replicaExpiration`, or set expiration to zero. File-backed audit bodies remain until audit retention permits reclamation, even after a replica expires; include that delivery history in the disk budget. Refetched cache files have their own replica lifetime. This requires a configured replication route to an owner. Cross-shard query fan-out and write forwarding are not provided by this directive. Application database branches do not support sharding because branches do not replicate. + +### Regional placement and attribute-slot changes + +`@sharded(by: "region", slotFrom: ...)` with `initialize_shard_map` is how to meet data-residency requirements. The expression selects logical slots; the map assigns those slots to regional shards. The schema does not encode shard IDs or node names. -`replicateTo` is ignored for placement on a declarative table; the installed slot map determines where full records are retained. A non-owner accepts a local write and sends the complete body through replication to the owners. On an indexed RocksDB table with positive `replicaExpiration`, it also retains the complete written body as a temporary local replica, allowing an immediate local read while delivery is pending. A newer invalidation replaces it, and the replica sweep expires it. With expiration zero or LMDB, it retains only the index projection; an index-free key policy retains nothing. A replica is a cache, so crash recovery may restore a projection instead. Ingest-heavy non-owners should budget space for approximately their serialized bytes written per second multiplied by `replicaExpiration`, or set expiration to zero. File-backed audit bodies remain until audit retention permits reclamation, even after a replica expires; include that delivery history in the disk budget. Refetched cache files have their own replica lifetime. This requires a configured replication route to an owner. Cross-shard query fan-out and write forwarding are not provided by this directive. Application database branches do not support declarative residency because branches do not replicate. +Install this map before declaring the table. Configure Asia's nodes with `replication.shard: 0` and Europe's nodes with `replication.shard: 1`: -### Attribute-slot changes +```json +{ + "operation": "initialize_shard_map", + "database": "regional", + "slots": 3, + "ranges": [ + { "start": 0, "end": 2, "shard": 0 }, + { "start": 2, "end": 3, "shard": 1 } + ] +} +``` ```graphql -type RegionalItem @table @residency(slots: 10, slotFrom: "region", byAttribute: true) { +type RegionalItem @table(database: "regional") @sharded(by: "region", slotFrom: "region == 'asia' ? 1 : 2") { id: ID @primaryKey - region: Int @indexed + region: String @indexed body: String } ``` +The omitted count adopts the regional database's three slots. `asia` selects slot 1 on shard 0; other non-null region values select slot 2 on shard 1. Slot 0 remains available for other policies. The map contains slot ranges and shard IDs, never pinned attribute values. + Changing `region` sends the complete canonical body to the new owners and invalidates old owners at the same version. This handoff uses full-record replacement ordering: an older concurrent patch does not merge into the replacement. Use application concurrency controls when writers can update a record while its slot changes. When the writer is outside the new owner set, success requires at least one new owner to confirm durable receipt. If no new owner has an active subscription and recent received activity within [`replication.pingTimeout`](../configuration/options.md#replication), the change is refused before commit. This can also refuse an owner whose base copy is not progressing. -Automatic slot-move confirmation waits up to `replication.residencyConfirmationTimeout` (default `30000` ms). Explicit `replicatedConfirmation` uses `replication.confirmationTimeout` instead (default `900000` ms, raised to `blobTimeout` when the confirmation setting is omitted). Both settings are read when a confirmation starts. An explicit setting can lower that timeout independently of blob transfers. Slow transfers can therefore exceed the automatic timeout even on a healthy link. +Automatic slot-move confirmation waits up to `replication.shardingConfirmationTimeout` (default `30000` ms). Explicit `replicatedConfirmation` uses `replication.confirmationTimeout` instead (default `900000` ms, raised to `blobTimeout` when the confirmation setting is omitted). Both settings are read when a confirmation starts. An explicit setting can lower that timeout independently of blob transfers. Slow transfers can therefore exceed the automatic timeout even on a healthy link. A connection can fail after admission. An error with `replicationUnconfirmed: true` after local commit means the outcome is uncertain; it does not mean the transaction rolled back. Inspect the record and retry with the application's concurrency controls. Audit retention bounds replay for disconnected peers; it is not an indefinite handoff queue. @@ -87,7 +117,7 @@ Non-owner invalidation events contain only indexed fields. A subscription `rowFi ### Configuration failures and peer identity -Invalid residency configuration at startup quarantines the affected database before replay. Its reads and writes return `DATABASE_RESIDENCY_BLOCKED` with the configuration reason and `retryable: false`; replication does not subscribe or advance its cursor. New tables declared in that database are deferred until a repaired restart; healthy tables in the same application schema still load. Other databases and the operations API remain available. `cluster_status.residency` reports the blocked database. Repair the configuration and restart the node to replay its retained tail. Repairing a system map or restarting an individual worker does not clear the quarantine. +Invalid sharding configuration at startup quarantines the affected database before replay. Its reads and writes return `DATABASE_SHARDING_BLOCKED` with the configuration reason and `retryable: false`; replication does not subscribe or advance its cursor. New tables declared in that database are deferred until a repaired restart; healthy tables in the same application schema still load. Other databases and the operations API remain available. `cluster_status.sharding` reports the blocked database. Repair the configuration and restart the node to replay its retained tail. Repairing a system map or restarting an individual worker does not clear the quarantine. A peer's announced node name must match its authenticated identity or an alias explicitly bound by its configured route or node record. For a canonical hostname reached through an IP address, configure both identities: @@ -102,19 +132,19 @@ The canonical name is used for topology and confirmations. An announcement canno ## Installed maps -Each database has one versioned snapshot in `system.hdb_residency_slots`. The snapshot contains `database`, `version`, `digest`, `slots`, and complete, nonoverlapping `ranges` with exclusive `end` values. Tables may use different covered prefixes of the map. +Each database has one versioned snapshot in `system.hdb_residency_slots`. The snapshot contains `database`, `version`, `digest`, `slots`, and complete, nonoverlapping `ranges` with exclusive `end` values. Every sharded table in that database uses the same slot count. First activation requires a writable node; it may need to persist both the map and activation metadata. Activate the policy before switching a node to read-only mode. -For a new declarative table with no map, first activation installs version `1` from only the declared slot count and sorted `replication.shards`: slot `k` maps to `shards[k % shards.length]`. Equal inputs produce the same snapshot and digest on simultaneous installers. The installed snapshot is then authoritative. Changes to the configured shard list never regenerate it. +For a new declarative table with no map, first activation installs version `1` from the explicit count (or `1024` when omitted) and sorted `replication.shards`: slot `k` maps to `shards[k % shards.length]`. Equal inputs produce the same snapshot and digest on simultaneous installers. The installed snapshot is then authoritative. Changes to the configured shard list never regenerate it. -The first table activated supplies the default slot count; Harper does not choose the maximum across a schema. For tables with different slot counts, install one custom map covering the largest count and wait for the same snapshot on every node before deploying the tables. A later table needing more slots requires an explicit `initialize_residency_slots` extension that preserves every active assignment. It never extends the default automatically. Do not run different initial schema deployments concurrently. +`slots` is a database-wide value. An omitted count adopts the installed map's count. An explicit different count is refused with an error naming both counts. Once a sharded table is active, changing the count is refused because it would change hash placement. Separate maps per slot count are deferred: custom-map administration and future [slot moves (#1053)](https://github.com/HarperFast/harper-pro/issues/1053) would need to apply to every map. Do not run incompatible first schema deployments concurrently. An existing table requires a custom map before activation, even if this node currently holds no rows. A custom map can be installed without `replication.shards`: ```json { - "operation": "initialize_residency_slots", + "operation": "initialize_shard_map", "database": "data", "slots": 10, "ranges": [ @@ -124,12 +154,20 @@ An existing table requires a custom map before activation, even if this node cur } ``` -This POST operation requires `super_user` and returns the installed snapshot. It changes metadata only. An exact retry is a no-op. Once a declarative table is active, existing assignments cannot change; coverage can only extend without changing active slots. Same-version divergent snapshots are refused rather than resolved by last-write-wins. Deleting an installed map is refused. If its stored snapshot is lost or corrupted, Harper does not reconstruct it from configuration. +This POST operation requires `super_user` and returns the installed snapshot. It changes metadata only. An exact retry is a no-op. Once a declarative table is active, neither existing assignments nor the database slot count can change. Same-version divergent snapshots are refused rather than resolved by last-write-wins. Deleting an installed map is refused. If its stored snapshot is lost or corrupted, Harper does not reconstruct it from configuration. -Setup exchanges carry the map version/digest and a separate shard-list digest. A missing or conflicting map blocks dependent placement and replication progress and appears in `cluster_status.residency`. A conflict on the system stream can also hold later system updates. Repair the conflicting configuration/map and reconnect; do not assume that skipping the offending record is safe. Resolving independently initialized, differently placed databases may require restoring or exporting the losing data. Reassignments that move data are a separate feature. +Setup exchanges carry the map version/digest and a separate shard-list digest. A missing or conflicting map blocks dependent placement and replication progress and appears in `cluster_status.sharding`. A conflict on the system stream can also hold later system updates. Repair the conflicting configuration/map and reconnect; do not assume that skipping the offending record is safe. Resolving independently initialized, differently placed databases may require restoring or exporting the losing data. Reassignments that move data are a separate feature. There is no operation to delete or reset an active map. If an accidentally initialized joining node has no unique data, stop its traffic and reprovision it with a fresh data directory; receive or install the cluster's exact snapshot before deploying the declarative application. Do not delete individual system-table files. If it has unique data, preserve a backup and restore/export that data before reprovisioning. In custom-map deployments, leaving `replication.shards` unset on every participant prevents premature default installation; lists must still agree across peers. +## Limitations + +Every record with the same `by` value lands in one slot. One region or large tenant therefore cannot be split across shards merely by moving slots. A future `spread` argument (name open) is tracked in [#1122](https://github.com/HarperFast/harper-pro/issues/1122). It would distribute one value over a range using the primary-key hash, with `spread: 1` preserving today's behavior; moving those slots would be handled by #1053. This argument is not available in this release. + +## Older per-record residency APIs + +`setResidency`, `setResidencyById`, `getResidency`, and residency lists retain their names and remain the older per-record mechanism. Use the conversion procedure below when adopting schema-defined sharding for an existing by-id table. + ## Convert an existing by-id table without moving records ### 1. Prepare the existing table @@ -142,11 +180,11 @@ Install a custom map that reproduces the function's numeric shard result exactly ### 3. Compare placement on every node -Run `verify_residency_slots` on each node while its application worker still runs the old function: +Run `verify_shard_map` on each node while its application worker still runs the old function: ```json { - "operation": "verify_residency_slots", + "operation": "verify_shard_map", "database": "data", "table": "Item", "policy": { "slots": 10, "slotFrom": "Number(id.slice(-1))" } @@ -166,7 +204,7 @@ Existing residency IDs remain readable but no longer decide placement. Key-deriv For a key-derived policy, drain cached RocksDB replicas before changing binaries: an earlier binary can interpret one as a full owned record. LMDB does not cache these replicas. Key policies themselves need no slot-format conversion. 1. Back up the database. Pause application requests and background jobs on **every node**, including reads that could admit another replica. Keep them paused through the downgrade. -2. Set `@table(replicaExpiration: 0)` in each affected table's schema and restart every node on the **current** binary, preserving its residency directive. This table argument is in seconds; zero disables new replica admissions. It does not rewrite existing replica deadlines, so waiting for a new, shorter lifetime does not establish a drain. +2. Set `@table(replicaExpiration: 0)` in each affected table's schema and restart every node on the **current** binary, preserving its sharding directive. This table argument is in seconds; zero disables new replica admissions. It does not rewrite existing replica deadlines, so waiting for a new, shorter lifetime does not establish a drain. 3. Run the following trusted maintenance code for each affected table on every node. Run it inside the application worker with access to `tables`; do not expose it as a public endpoint. This procedure is for the 5.4 implementation and uses internal primary metadata (`REPLICA = 4`); verify the matching implementation before using it with another release. It evicts only cached replicas, preserving owned rows and legacy-compatible invalidated index projections. The runtime TTL call below repeats step 2 defensively on this table handle and preserves its other TTL settings. These APIs are trusted worker internals, not Operations API commands. ```js diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index 250e5b15e..d04671519 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -14,13 +14,17 @@ title: Sharding Harper's replication system supports sharding — storing different data across different subsets of nodes. Single-key reads can fetch records from their owners across shards. Sharding distributes storage and write work across nodes. -A key read for a record not held locally can fetch its body from an owner. This does not provide cross-shard query fan-out or write forwarding. With [declarative residency](./residency.md#ownership-reads-and-writes), a non-owner write commits locally and replicates the complete body to its owners through the configured replication routes. +A key read for a record not held locally can fetch its body from an owner. This does not provide cross-shard query fan-out or write forwarding. With [sharding](./residency.md#ownership-reads-and-writes), a non-owner write commits locally and replicates the complete body to its owners through the configured replication routes. By default (without sharding), Harper replicates all data to all nodes. -## Approaches to Sharding +## Schema-defined sharding -There are two main approaches: +In v5.4, use [`@sharded`](./residency.md) for a shared schema policy across all workers. `@sharded` hashes primary keys, `@sharded(by: "tenantId")` keeps a tenant together, and a compiled `slotFrom` expression plus a custom map supports regional data-residency requirements. Slot counts belong to the database. See the [one-slot-per-value limitation](./residency.md#limitations). + +## Older per-record mechanisms + +The `setResidency`, `setResidencyById`, `getResidency` and residency-list APIs below retain their names. They are the older mechanism, with two approaches: **Dynamic sharding** — the location (residency) of records is determined dynamically based on where the record was written, the record's data, or a custom function. Records can be relocated dynamically based on where they are accessed. Residency information is specific to each record. @@ -154,7 +158,7 @@ Pre-commit refusals, including a confirmation count that cannot be met, do not c Confirmation currently applies to the first database a transaction writes to. This outcome flag covers the linked local commits; it does not enable confirmation for later databases. -[Attribute-slot transitions](./residency.md#attribute-slot-changes) can require new-owner confirmation automatically even when the caller did not request an explicit confirmation count. +[Attribute-slot transitions](./residency.md#regional-placement-and-attribute-slot-changes) can require new-owner confirmation automatically even when the caller did not request an explicit confirmation count. ## Static Sharding @@ -171,7 +175,7 @@ replication: -For the first default map of a new [declarative residency](./residency.md) table, configure `replication.shards` next to `replication.shard`: +For the first default map of a new [sharding](./residency.md) table, configure `replication.shards` next to `replication.shard`: ```yaml replication: @@ -185,7 +189,7 @@ The list is required only for a database that has declarative tables and no inst Once installed, the snapshot is authoritative. Changing the list never recomputes placement. Adding an ID only makes that shard available for future reassignment; it moves no records. -Setup exchanges include a digest of the list. Nodes with different lists refuse declarative writes on both sides, even after a map is installed. A configured list on one peer and no list on another also differs. A node whose own shard is outside its list, or which needs an initial map but has no list, also refuses placement. The condition appears in `cluster_status.residency` and is logged once. Plan for this write refusal during a rolling list change. +Setup exchanges include a digest of the list. Nodes with different lists refuse declarative writes on both sides, even after a map is installed. A configured list on one peer and no list on another also differs. A node whose own shard is outside its list, or which needs an initial map but has no list, also refuses placement. The condition appears in `cluster_status.sharding` and is logged once. Plan for this write refusal during a rolling list change. ### Other shard configuration @@ -210,7 +214,7 @@ Or dynamically via the operations API by including `shard` in an `add_node` or ` } ``` -Once shards are configured, use [declarative residency](./residency.md) for a schema policy applied on every thread, or the function APIs below for legacy application-controlled placement. +Once shards are configured, use [sharding](./residency.md) for a schema policy applied on every thread, or the function APIs below for legacy application-controlled placement. ## Custom Sharding @@ -286,6 +290,6 @@ X-Replicate-From: none ## See Also -- [Declarative Residency](./residency.md) — Fixed slot policies and authoritative database maps +- [Sharding](./residency.md) — Fixed slot policies and authoritative database maps - [Replication Overview](./overview.md) — How Harper's replication system works - [Clustering Operations](./clustering.md) — Operations API for managing cluster nodes diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index b803155f1..9dc28aff1 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -22,9 +22,9 @@ These corrections are not released yet. ## Replication -### Declarative Residency +### Sharding -`@residency` applies a fixed key- or attribute-based slot policy on every thread, including dedicated replication workers. Per-database slot maps remain authoritative after initialization; `replication.shards` supplies the complete deployment shard set for a first default map. Super-user map initialization and read-only conversion checks support adopting an equivalent legacy by-id policy without rewriting records. Attribute-slot changes from a writer outside the new owner set require a new owner's confirmation. See [Declarative Residency](/reference/v5/replication/residency), including its storage-format and rollback requirements. +`@sharded` applies a fixed key- or attribute-based slot policy on every thread, including dedicated replication workers. Per-database slot maps remain authoritative after initialization; `replication.shards` supplies the complete deployment shard set for a first default map. Super-user map initialization and read-only conversion checks support adopting an equivalent legacy by-id policy without rewriting records. Attribute-slot changes from a writer outside the new owner set require a new owner's confirmation. See [Sharding](/reference/v5/replication/residency), including its storage-format and rollback requirements. ### LMDB Replication Confirmation diff --git a/sidebarsReference.ts b/sidebarsReference.ts index 88ab6cbd1..da02d4efa 100644 --- a/sidebarsReference.ts +++ b/sidebarsReference.ts @@ -637,12 +637,12 @@ const sidebars: SidebarsConfig = { { type: 'doc', id: 'replication/sharding', - label: 'Sharding', + label: 'Sharding and legacy APIs', }, { type: 'doc', id: 'replication/residency', - label: 'Declarative residency', + label: 'Schema-defined sharding', }, ], }, From d287cce0ec9b82df6f2cbc10680064d492d142e8 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 26 Oct 2026 06:07:19 -0600 Subject: [PATCH 21/32] docs: define sharded recovery and existing-owner admission Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/database/schema.md | 6 +- reference/operations-api/operations.md | 4 +- reference/replication/clustering.md | 16 +++-- reference/replication/residency.md | 88 ++++++++++++++++++++++---- reference/replication/sharding.md | 10 +-- release-notes/v5-lincoln/5.4.md | 8 ++- 6 files changed, 103 insertions(+), 29 deletions(-) diff --git a/reference/database/schema.md b/reference/database/schema.md index 736fd07f4..ccc86eaf5 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -141,12 +141,12 @@ Local replicas are RocksDB-only. On the deprecated LMDB engine, remote reads rem On RocksDB, schema rescans apply changes to already-loaded tables on other workers. The new lifetime applies to subsequent admissions and renewals; existing deadlines are not rewritten. Setting `replicaExpiration: 0` stops new admissions and the sweep; existing replicas remain until overwritten or evicted. -On a table whose records live on specific nodes (see [sharding](../replication/sharding.md#by-record-content-setresidency)), a node that does not hold a record fetches it from a node that does. The fetched record is kept locally as a _replica_: a complete copy that reads and searches on that node use without contacting another node again. A replica does not move the record and is never replicated onward. A write by an owner reaches the node as an invalidation, which turns the replica back into a stub, so the next read fetches the new version. Tables sharded by primary key only (`setResidencyById`) keep no replicas. +On a table whose records live on specific nodes (see [sharding](../replication/sharding.md#by-record-content-setresidency)), a node that does not hold a record fetches it from a node that does. The fetched record is kept locally as a _replica_: a complete copy that reads and searches on that node use without contacting another node again. A replica does not move the record and is never replicated onward. A write by an owner reaches the node as an invalidation, which turns the replica back into a stub, so the next read fetches the new version. Tables using the older [`setResidencyById`](../replication/sharding.md#by-primary-key-only-setresidencybyid) mechanism keep no replicas. Schema-defined `@sharded` tables follow the rules below. - **`replicaExpiration`** — Seconds a replica is kept. A sweep runs every quarter of this time, with a minimum interval of one second, and reduces replicas past their lifetime back to their indexed stubs. A replica can still be served until the sweep reaches it; the next read after demotion fetches the record again. `0` disables new replica admissions and the sweep. A replica never outlives the record's own `expiration`, if it has one. - **`replicaRefreshFraction`** — A read of a replica after this fraction of `replicaExpiration` has passed renews the lifetime, so records that stay in use stay local. Must be greater than `0` and at most `1`; `1` gives every replica a fixed lifetime. -[Sharding](../replication/residency.md#ownership-reads-and-writes) tables with secondary indexes can retain replicas for either key- or attribute-derived slots. With positive `replicaExpiration`, a non-owner writer also keeps its complete write as a temporary replica under the same expiry and replacement rules. This local copy never counts as owner confirmation. Setting the lifetime to `0` does not drain existing replicas; see the [rollback procedure](../replication/residency.md#rollback). +[Schema-defined sharding](../replication/residency.md#ownership-reads-and-writes) tables with secondary indexes can retain replicas for either key- or attribute-derived slots. With positive `replicaExpiration`, a non-owner writer also keeps its complete write as a temporary replica under the same expiry and replacement rules. This local copy never counts as owner confirmation. Setting the lifetime to `0` does not drain existing replicas; see the [rollback procedure](../replication/residency.md#rollback). ```graphql # Keep fetched records for a day; a read after six hours renews them @@ -244,7 +244,7 @@ See [REST Headers / Cache-Control](../rest/headers.md#cache-control) for the ful -Defines an immutable placement policy on a schema-defined table. `@sharded` hashes the primary key; `by: "tenantId"` hashes that attribute's non-null value with the same hash. `slotFrom` optionally chooses the integer logical slot with a compiled deterministic expression. `slots` is a database-wide value (`1` through `65536`): omit it to adopt the installed map's count, or `1024` if no map exists; an explicit different count is refused. Attribute policies require a secondary index. The database map resolves each slot to a deployment shard. See [Sharding](../replication/residency.md) for tenant affinity, regional placement, configuration, conversion, limitations and rollback. +Defines an immutable placement policy on a schema-defined RocksDB table. An LMDB declaration is refused within its database; legacy residency functions on LMDB are unchanged. `@sharded` hashes the primary key; `by: "tenantId"` hashes that attribute's non-null value with the same hash. `slotFrom` optionally chooses the integer logical slot with a compiled deterministic expression. `slots` is a database-wide value (`1` through `65536`): omit it to adopt the installed map's count, or `1024` if no map exists; an explicit different count is refused. Attribute policies require a secondary index. The database map resolves each slot to a deployment shard. See [Schema-defined sharding](../replication/residency.md) for tenant affinity, regional placement, configuration, conversion, limitations and rollback. ### `@export` diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 541063728..1073d1441 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -1810,7 +1810,9 @@ Adds a remote Harper node to the cluster. If `subscriptions` are not provided, a -POST; requires `super_user`. Takes `database`, `slots` and complete nonoverlapping `ranges` (`start`, exclusive `end`, `shard`). Returns the installed versioned snapshot. Installs metadata only and refuses changes to active slot assignments. See [Installed maps](../replication/residency.md#installed-maps). +POST; requires `super_user` and a RocksDB database. Takes `database`, `slots` and complete nonoverlapping `ranges` (`start`, exclusive `end`, `shard`). Returns the installed versioned snapshot. Installs metadata only and refuses changes to active slot assignments. See [Installed maps](../replication/residency.md#installed-maps). + +For a legacy conversion, call with `database`, `existing_owner: true` and `expected_digest` instead. This records the operator's assertion that this local owning node has complete bodies for its slots; Harper does not verify completeness. It requires an already-installed map with exactly that digest, rejects `slots` or `ranges` in the same call, and does not declare a sharded table. The receipt is local to the captured root generations and tables. Assert on each owning node, with legacy writers paused until activation is ready throughout the cluster. See [Converting an existing table](../replication/residency.md#convert-an-existing-by-id-table-without-moving-records). ### `verify_shard_map` diff --git a/reference/replication/clustering.md b/reference/replication/clustering.md index ad88bdd4f..336afae4f 100644 --- a/reference/replication/clustering.md +++ b/reference/replication/clustering.md @@ -208,16 +208,18 @@ While a database is taking its initial full copy, these timing fields hold the l The top-level `cluster_status.sharding` field is keyed by database and describes active or attempted declarative tables. It is empty in the legacy-table example above. Each entry has: -| Field | Meaning | -| ------------------- | -------------------------------------------------------------------------------------------------------------------------- | -| `version`, `digest` | Installed map identity; absent if no map is installed. | -| `shardsDigest` | Digest of the configured shard set, or `null` when no list is configured. | -| `blocked` | Whether a reported configuration or map-agreement condition blocks placement. | -| `errors` | Reasons, such as differing peer lists, a local shard outside its list, a missing required list/map, or a refused peer map. | +| Field | Meaning | +| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `version`, `digest` | Installed map identity; absent if no map is installed. | +| `shardsDigest` | Digest of the configured shard set, or `null` when no list is configured. | +| `blocked` | Whether a reported configuration or map-agreement condition blocks placement. | +| `recovery` | Bootstrap/recovery phase and reason. `origin-history-lost` refuses a reused name without its audit; restore a complete backup or rejoin with a fresh root under a new node name. | +| `waitingOrigins` | Origins waiting for a direct source or certified same-shard relay, with the missing source described. | +| `errors` | Reasons, such as differing peer lists, a local shard outside its list, a missing required list/map, or a refused peer map. | A clear status is not proof that an owner is connected; write admission also checks the required replication route and, for an attribute-slot transition from a writer outside the new owner set, a connected new owner. See [configuration agreement](./sharding.md#complete-shard-id-set) and [map recovery](./residency.md#installed-maps). -### Residency Map Operations +### Shard map operations The POST operations [`initialize_shard_map`](./residency.md#installed-maps) and [`verify_shard_map`](./residency.md#3-compare-placement-on-every-node) require `super_user`. The former installs a snapshot without moving records; the latter compares the actual legacy function against a proposed key policy without modifying data. Their request examples, parameters and result fields are in the linked procedures. diff --git a/reference/replication/residency.md b/reference/replication/residency.md index 3746ab19d..7a427ca2d 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -4,6 +4,8 @@ title: Schema-defined sharding # Schema-defined sharding +Schema-defined sharding requires RocksDB. `@sharded` and `initialize_shard_map` refuse LMDB databases; schema refusal fences only the affected database and keeps loading healthy databases in the component. The older per-record residency functions remain available on LMDB. An LMDB database created with an earlier prerelease sharding policy must be exported using that prerelease and recreated on RocksDB before reapplying the schema; removing the directive does not erase the persisted policy. + `@sharded` fixes a table's placement policy in its schema. Every application, operations, and replication thread applies the same policy. A key or record selects a slot, the database's installed slot map selects a shard, and `replication.shard` identifies that shard's current nodes. @@ -12,7 +14,7 @@ Upgrade every schema and data writer before activation. Wire compatibility with ## Configuration -For a new declarative table with no installed map, configure the complete shard-id set on every node. A custom-map conversion does not require this list; leave it unset on both sides when using legacy-peer fallback. See [`replication.shards`](./sharding.md#complete-shard-id-set) for initialization and mismatch handling. +For a new declarative table with no installed map, configure the complete shard-id set on every node. A custom-map conversion does not require this list. Leave it unset consistently if it was not configured; differing shard-list digests still hold placement. All members must run the sharding initialization protocol before converted tables can resume writes. See [`replication.shards`](./sharding.md#complete-shard-id-set) for initialization and mismatch handling. ```yaml replication: @@ -33,15 +35,15 @@ type Item @table @sharded { } ``` -| Argument | Meaning | -| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `slots` | Optional database-wide integer from `1` through `65536`. Uses the installed map's count, or `1024` when no map exists. An explicit count must equal the database map's count. | -| `by` | Optional attribute name. Its non-null value is hashed instead of the primary key. Requires a secondary index. | -| `slotFrom` | Optional deterministic JavaScript expression returning an integer in `[0, slots)`. Without `by`, it exposes `id`; with `by`, it also exposes `record` and its attribute names. | +| Argument | Meaning | +| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `slots` | Optional database-wide integer from `1` through `65536`. Uses the installed map's count, or `1024` when no map exists. An explicit count must equal the database map's count. | +| `by` | Optional stored, declared attribute name; may be the primary key, but cannot be computed. Default placement hashes its non-null string or finite-number value. Requires a secondary index, including when naming the primary key. | +| `slotFrom` | Optional deterministic JavaScript expression returning an integer in `[0, slots)`. Without `by`, it exposes `id`; with `by`, it also exposes `record` and its attribute names. | Declarative tables require [per-table audit logging](../database/transaction.md#enabling-the-transaction-log-per-table) and table replication. Harper refuses activation with `@table(audit: false)`, effective `logging.auditLog: false`, or `@table(replicate: false)`, and refuses disabling these table settings later. The replication route must also reach an owner. -Without `slotFrom`, the slot is the unsigned FNV-1a 32-bit hash of the primary key's ordered-binary encoding, modulo the database slot count. With `by`, the same hash uses that attribute's value instead. Strings and numbers are supported; a null or missing value is refused. The hash's offset basis is `2166136261` and its multiplier is `16777619`. +Without `slotFrom`, the slot is the unsigned FNV-1a 32-bit hash of the primary key's ordered-binary encoding, modulo the database slot count. With `by`, the same hash uses that attribute's value instead. Default hashing supports strings and finite numbers; a null or missing value, `NaN`, or infinity is refused. Numeric `-0` and `0` use the same encoding for placement. Numeric `5` and string `"5"` remain distinct typed hash inputs. Default `by` declarations must name a stored `ID`, `String`, `Int`, `Float`, `Long`, or `Any` field; other stored types require `slotFrom`, and relationship or computed fields cannot be used. The hash's offset basis is `2166136261` and its multiplier is `16777619`. `@sharded(by: "tenantId")` keeps each tenant's records together: @@ -71,11 +73,11 @@ The policy is immutable once activated, including changing `by`, changing or add ## Ownership, reads, and writes -Owners store full records. With secondary indexes, other nodes keep index projections and receive invalidations. Reads fetch the full body from an owner. RocksDB can cache a temporary replica; LMDB proxies the read without caching it. Replicas retain the canonical record identity and do not become owners or count toward owner confirmation. +Owners store full records. With secondary indexes, other nodes keep index projections and receive invalidations. Reads fetch the full body from an owner. A non-owner can cache a temporary replica. Replicas retain the canonical record identity and do not become owners or count toward owner confirmation. A key-policy table without secondary indexes stores nothing on non-owners; reads resolve the owner from the key. Such tables do not cache replicas. Attribute policies require a secondary index so a non-owner can retain the stored slot. If an attribute record's slot is unknown locally, DELETE returns 503 with a message beginning `Attribute sharding slot is unknown`. This response does not distinguish a delayed projection from a never-created key or a purged tombstone; local absence cannot prove that the record is absent on its owners. If another node still has the slot metadata, send the deletion there. If no node has it, DELETE cannot succeed anywhere; retrying alone cannot fix that condition. Stop automatic retries and use application-level knowledge to establish whether the key ever existed. Local absence is not proof that it was deleted. -`replicateTo` is ignored for placement on a declarative table; the installed slot map determines where full records are retained. A non-owner accepts a local write and sends the complete body through replication to the owners. On an indexed RocksDB table with positive `replicaExpiration`, it also retains the complete written body as a temporary local replica, allowing an immediate local read while delivery is pending. A newer invalidation replaces it, and the replica sweep expires it. With expiration zero or LMDB, it retains only the index projection; an index-free key policy retains nothing. A replica is a cache, so crash recovery may restore a projection instead. Ingest-heavy non-owners should budget space for approximately their serialized bytes written per second multiplied by `replicaExpiration`, or set expiration to zero. File-backed audit bodies remain until audit retention permits reclamation, even after a replica expires; include that delivery history in the disk budget. Refetched cache files have their own replica lifetime. This requires a configured replication route to an owner. Cross-shard query fan-out and write forwarding are not provided by this directive. Application database branches do not support sharding because branches do not replicate. +`replicateTo` is ignored for placement on a declarative table; the installed slot map determines where full records are retained. A non-owner accepts a local write and sends the complete body through replication to the owners. On an indexed RocksDB table with positive `replicaExpiration`, it also retains the complete written body as a temporary local replica, allowing an immediate local read while delivery is pending. A newer invalidation replaces it, and the replica sweep expires it. With expiration zero, it retains only the index projection; an index-free key policy retains nothing. A replica is a cache, so crash recovery may restore a projection instead. Ingest-heavy non-owners should budget space for approximately their serialized bytes written per second multiplied by `replicaExpiration`, or set expiration to zero. File-backed audit bodies remain until audit retention permits reclamation, even after a replica expires; include that delivery history in the disk budget. Refetched cache files have their own replica lifetime. This requires a configured replication route to an owner. Cross-shard query fan-out and write forwarding are not provided by this directive. Application database branches do not support sharding because branches do not replicate. ### Regional placement and attribute-slot changes @@ -119,6 +121,8 @@ Non-owner invalidation events contain only indexed fields. A subscription `rowFi Invalid sharding configuration at startup quarantines the affected database before replay. Its reads and writes return `DATABASE_SHARDING_BLOCKED` with the configuration reason and `retryable: false`; replication does not subscribe or advance its cursor. New tables declared in that database are deferred until a repaired restart; healthy tables in the same application schema still load. Other databases and the operations API remain available. `cluster_status.sharding` reports the blocked database. Repair the configuration and restart the node to replay its retained tail. Repairing a system map or restarting an individual worker does not clear the quarantine. +A disagreement detected after startup places the affected database in a live hold. Reads and writes, including those on its unsharded tables, return `DATABASE_SHARDING_HELD` with `retryable: true`. Other databases in the same schema continue serving. Repairing the configuration clears this hold on every worker without a worker restart. Each process observes the root file; after the 150 ms settling interval, affected replication connections reconnect automatically to refresh peer agreement. Wait for the held state to clear before retrying writes. In `cluster_status.sharding`, `state: held` identifies this recoverable condition; `state: quarantined` and `restartRequired: true` identify the startup quarantine described above. A rolling shard-list configuration change can hold every sharded database until all peers agree. If an environment variable such as `HARPER_SET_CONFIG` supplies the shard list, update that provisioning value too: a full process restart re-applies it and can undo a file-only repair. + A peer's announced node name must match its authenticated identity or an alias explicitly bound by its configured route or node record. For a canonical hostname reached through an IP address, configure both identities: ```yaml @@ -130,6 +134,35 @@ replication: The canonical name is used for topology and confirmations. An announcement cannot create an alias or claim another known node's identity. +## Recovery sources + +An owning node receives a live stream for origin X directly from X or through a co-owner in its own shard. Failover selects a peer in the receiver's shard. If neither source is reachable, that origin waits without advancing its cursor; `cluster_status.sharding` identifies the waiting origin and the missing source. Non-owners may relay projections to other non-owners. + +Bootstrapping or making a full copy of an owning node requires a same-shard peer or a complete backup for its owned slots. A copy from any peer remains allowed for slots the receiver does not own. A fast clone owning zero slots can bootstrap from any peer. + +On first initialization, sharded writes return a retryable 503 until the configured members have connected and acknowledged the initial history state. Requests do not keep transactions open while initialization waits. A missing member is reported as waiting rather than treated as proof of an empty shard. Once initialized, ordinary operation does not require every member to remain connected. Replacement nodes learn the existing history state and recover a complete owned baseline before they can act as a complete recovery source. + +A fresh node may start its owned slots empty only when the installed map is the initial version and those slots have no write history. An empty current table is not sufficient: deleted records and lost storage do not erase history. Otherwise, if the entire shard is lost, restore a complete backup. Status reports the backup requirement instead of treating an incomplete copy as recovery. Maps are installed before declarative writes begin; conversion of existing data uses the paused operator-assertion procedure below. Later ownership changes require the [slot-move workflow](https://github.com/HarperFast/harper-pro/issues/1053), which copies from the old owners. + +### Recovering a lost node + +Choose one recovery route before restarting a node whose local data or audit history was lost: + +- **Keep the original node name:** restore a complete backup taken while the node was stopped after its last accepted write, preserving the database roots and their audit history. An earlier backup that omits accepted writes does not qualify for same-name recovery. Restore the matching catalog metadata with the audit; copying table rows alone is insufficient. Restart the node with its original configuration and confirm that `cluster_status.sharding..recovery.phase` becomes `ready` before admitting writes. +- **Recover from co-owner snapshots:** stop the failed node, provision a fresh data root with a **new `node.hostname`**, and configure the same `replication.shard` and shard list. Join it through the normal [`add_node` procedure](./clustering.md#add-node). Keep a complete same-shard peer available for the copy. Remove the failed name through [`remove_node`](./clustering.md#remove-node), including on any peers that were offline during removal, and check the new node's recovery status before admitting writes. + +A node reusing a name whose earlier origin history is known to peers, without the corresponding local history, refuses sharded writes and cannot serve as an owned-slot recovery source. Its recovery status reports `origin-history-lost` and names both remedies: restore a complete backup, or rejoin under a new node name. Waiting for its clock to advance does not repair the missing history. + +Co-owners continue relaying the old name's retained audit entries after membership removal. Keep those co-owners and their audit history available until lagging peers have received the old writes; membership removal does not replace normal audit retention requirements. The new name's writes use a separate origin, so receiving them does not certify receipt of the old origin's pending writes. + +If an entire shard was lost, a new name alone cannot recover its bodies: restore a complete backup. Each name restored without its original audit needs the snapshot-only route above. A new storage root in an initialized database cannot borrow another root's origin history; recover missing history before enabling writes. + +Snapshot recovery also marks the database's record-lock lineage as recloned: it cannot prove continuity of its earlier local history, including for ordinary tables in the same database. A complete backup preserving the original history avoids that limitation. Future [origin incarnations](https://github.com/HarperFast/harper-pro/issues/1123#issuecomment-6414845915) will allow a wiped node to keep its hostname while originating under a new root generation; that is not supported in this release. + +Every member must support sharding recovery admission. An authenticated announcement from an older binary holds sharded writes, even after initialization; ordinary tables and system metadata remain available. This observation survives restart and matching map/schema messages cannot clear it. Upgrade the peer or remove it from the configured membership. An offline peer's unannounced downgrade cannot be detected until it reconnects. + +Arbitrary cross-shard relay and clone layouts for owners are unsupported in this release. [Complete-body recovery from origin audit logs](https://github.com/HarperFast/harper-pro/issues/1123) tracks broader topology support, including handling origins whose full audit history is no longer retained. + ## Installed maps Each database has one versioned snapshot in `system.hdb_residency_slots`. The snapshot contains `database`, `version`, `digest`, `slots`, and complete, nonoverlapping `ranges` with exclusive `end` values. Every sharded table in that database uses the same slot count. @@ -162,6 +195,8 @@ There is no operation to delete or reset an active map. If an accidentally initi ## Limitations +Owner recovery requires the [sources described above](#recovery-sources). Arbitrary cross-shard owner recovery is deferred to [#1123](https://github.com/HarperFast/harper-pro/issues/1123). + Every record with the same `by` value lands in one slot. One region or large tenant therefore cannot be split across shards merely by moving slots. A future `spread` argument (name open) is tracked in [#1122](https://github.com/HarperFast/harper-pro/issues/1122). It would distribute one value over a range using the primary-key hash, with `spread: 1` preserving today's behavior; moving those slots would be handled by #1053. This argument is not available in this release. ## Older per-record residency APIs @@ -193,15 +228,42 @@ Run `verify_shard_map` on each node while its application worker still runs the This POST operation requires `super_user`. It scans local records and compares the actual old function's numeric shard with the proposed policy and installed map. It reports `checked`, `mismatches`, bounded mismatch samples, observed slots, departed node names, and map identity. It does not rewrite records or activate the policy. Hostname-array functions are not converted: the first nonnumeric return rejects the operation with HTTP 400, rather than counting as a mismatch or inferring a shard. `departedNodeNames` lists names found in stored legacy residency lists that are absent from the node's current known-node list (including itself). A sample of present keys cannot prove equivalence for every future key; the operator must confirm the functions are equivalent. -### 4. Activate the directive +### 4. Assert each existing owner's completeness -Require zero mismatches and matching maps on every participant. Deploy the directive and remove the setter. The existing deployment can keep writing during the comparison; this is not a quiescing migration operation. +Require zero mismatches and matching maps on every participant. Placement comparison examines the records present on a node; it does not prove that the node holds every record it should own. The operator must establish that completeness independently. + +Pause legacy writes across the cluster before asserting completeness, and keep them paused through activation. On each existing owner, submit the digest returned by map installation: + +```json +{ + "operation": "initialize_shard_map", + "database": "data", + "existing_owner": true, + "expected_digest": "" +} +``` + +This is an **operator assertion of completeness**, not something Harper verifies. It is bound to every captured local database root generation, its existing tables and the expected map digest. A partial set of receipts is uncertified, and a later legacy table conversion needs a new assertion. A digest mismatch is refused without storing an assertion. The operation does not declare a sharded table, and `verify_shard_map` remains read-only. + +Without the assertion, a converted legacy root stays uncertified. It may serve and copy nonowned data, but it cannot supply recovery for owned slots; `cluster_status.sharding` names the missing assertion. A replaced root does not inherit an assertion from replicated schema metadata. An uncertified node may still return a locally present body; completeness gating cannot detect that such a body is stale. It must not report an owned miss as authoritative absence or act as an owned recovery source. + +### 5. Activate the directive + +Deploy the directive and remove the setter. Wait for sharding initialization to complete on the owners before resuming writes. A member still running the previous binary cannot acknowledge initialization: writes remain refused and status names the missing member until it is upgraded. Writes can continue during the earlier placement comparison; the completeness assertion and activation form the paused cutover. Existing residency IDs remain readable but no longer decide placement. Key-derived records receive no new slot metadata. A stale component that calls a forbidden setter after activation fails visibly and must be updated. +### Convert an existing unsharded table + +An existing table without a residency function can also gain `@sharded`. Pause writes, install the desired custom map, and establish that every future owner already has all the records assigned to it. For a previously fully replicated table, check that every future owner has caught up before proceeding. Use the same `initialize_shard_map` request with `existing_owner: true` and the exact `expected_digest` on each owner, then deploy the key-derived directive and wait for initialization before resuming writes. + +This is the same operator assertion of completeness as the by-id procedure; Harper does not verify that every required record is present. `verify_shard_map` compares a legacy by-id function and does not replace this assertion for an unsharded table. A mismatched digest stores no assertion. Without the assertion, status names the missing completeness assertion and the node cannot supply owned-slot recovery. + +Key-derived conversion retains existing non-owner full copies locally; it does not relocate, purge, rewrite, or change the versions of existing rows. Those copies do not confer ownership or qualify the node as an owned-slot recovery source. Index-free reads load from the owner rather than trusting a retained non-owner body; indexed tables receive ordinary invalidations. Bootstrap may add missing non-owner index projections. Subsequent writes follow the new replica/projection placement rules. Conversion alone therefore does not remove previously replicated bodies from a region; account for those copies when planning a data-residency cutover. + ### Rollback -For a key-derived policy, drain cached RocksDB replicas before changing binaries: an earlier binary can interpret one as a full owned record. LMDB does not cache these replicas. Key policies themselves need no slot-format conversion. +For a key-derived policy, drain cached RocksDB replicas before changing binaries: an earlier binary can interpret one as a full owned record. Key policies themselves need no slot-format conversion. 1. Back up the database. Pause application requests and background jobs on **every node**, including reads that could admit another replica. Keep them paused through the downgrade. 2. Set `@table(replicaExpiration: 0)` in each affected table's schema and restart every node on the **current** binary, preserving its sharding directive. This table argument is in seconds; zero disables new replica admissions. It does not rewrite existing replica deadlines, so waiting for a new, shorter lifetime does not establish a drain. @@ -223,7 +285,7 @@ await drainResidencyReplicas(tables.Item); ``` 4. Require `{ remaining: 0 }` for every affected table on every node. Eviction is version-checked; a busy replica can survive a pass, so a nonzero result refuses the downgrade. Do not use ordinary record counts as proof: projections and replicas share keys. -5. Remove the maintenance code, restore the previous binary with the equivalent legacy schema and `setResidencyById` function, then resume traffic. Keep the installed slot map for a later upgrade. +5. Remove the maintenance code, restore the previous binary with the equivalent legacy schema and `setResidencyById` function, then resume traffic. Keep the installed slot map for a later upgrade. Local assertion receipts also survive restart and downgrade: their trust remains the operator's responsibility while the equivalent legacy placement is in use, not a new verification by Harper on re-upgrade. There is no built-in replica-drain operation. If you cannot run and verify this maintenance procedure, restore a preactivation backup instead of attempting an in-place downgrade. diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index d04671519..175ef54c3 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -14,7 +14,7 @@ title: Sharding Harper's replication system supports sharding — storing different data across different subsets of nodes. Single-key reads can fetch records from their owners across shards. Sharding distributes storage and write work across nodes. -A key read for a record not held locally can fetch its body from an owner. This does not provide cross-shard query fan-out or write forwarding. With [sharding](./residency.md#ownership-reads-and-writes), a non-owner write commits locally and replicates the complete body to its owners through the configured replication routes. +A key read for a record not held locally can fetch its body from an owner. This does not provide cross-shard query fan-out or write forwarding. With [schema-defined sharding](./residency.md#ownership-reads-and-writes), a non-owner write commits locally and replicates the complete body to its owners through the configured replication routes. By default (without sharding), Harper replicates all data to all nodes. @@ -124,6 +124,8 @@ That transaction commits on its own, separately from any request transaction tha Values your code sets on a context apply for every caller, including users without the `super_user` permission. Derive them from application logic, not from request input such as headers, query parameters, or the record body: copying a client-supplied value into `replicateTo` or `replicatedConfirmation` gives that client the control the `super_user` check removes. +On `@sharded` tables, `replicateTo` does not change placement; the database map determines owners. Explicit `replicatedConfirmation` counts acknowledgments from other owners for each affected owner set in the first written database. A local temporary replica cannot satisfy it. See [ownership, reads, and writes](./residency.md#ownership-reads-and-writes). + ### Confirmation outcomes @@ -175,7 +177,7 @@ replication: -For the first default map of a new [sharding](./residency.md) table, configure `replication.shards` next to `replication.shard`: +For the first default map of a new [schema-defined sharding](./residency.md) table, configure `replication.shards` next to `replication.shard`: ```yaml replication: @@ -214,7 +216,7 @@ Or dynamically via the operations API by including `shard` in an `add_node` or ` } ``` -Once shards are configured, use [sharding](./residency.md) for a schema policy applied on every thread, or the function APIs below for legacy application-controlled placement. +Once shards are configured, use [schema-defined sharding](./residency.md) for a schema policy applied on every thread, or the function APIs below for legacy application-controlled placement. ## Custom Sharding @@ -290,6 +292,6 @@ X-Replicate-From: none ## See Also -- [Sharding](./residency.md) — Fixed slot policies and authoritative database maps +- [Schema-defined sharding](./residency.md) — Fixed slot policies and authoritative database maps - [Replication Overview](./overview.md) — How Harper's replication system works - [Clustering Operations](./clustering.md) — Operations API for managing cluster nodes diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 9dc28aff1..62c4c87be 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -24,7 +24,13 @@ These corrections are not released yet. ### Sharding -`@sharded` applies a fixed key- or attribute-based slot policy on every thread, including dedicated replication workers. Per-database slot maps remain authoritative after initialization; `replication.shards` supplies the complete deployment shard set for a first default map. Super-user map initialization and read-only conversion checks support adopting an equivalent legacy by-id policy without rewriting records. Attribute-slot changes from a writer outside the new owner set require a new owner's confirmation. See [Sharding](/reference/v5/replication/residency), including its storage-format and rollback requirements. +`@sharded` applies a fixed key- or attribute-based slot policy on every thread, including dedicated replication workers. Per-database slot maps remain authoritative after initialization; `replication.shards` supplies the complete deployment shard set for a first default map. Super-user map initialization and read-only conversion checks support adopting an equivalent legacy by-id policy without rewriting records. Attribute-slot changes from a writer outside the new owner set require a new owner's confirmation. See [Schema-defined sharding](/reference/v5/replication/residency), including its storage-format and rollback requirements. + +Sharding configuration failures are isolated to the affected database. A live disagreement holds that database until the configuration is repaired on all peers; healthy databases in the same application keep serving, including during worker replacement. Invalid configuration at process startup still requires a repaired full restart because its audit replay was deferred. First-time sharding initialization waits for the configured members; owner recovery requires a certified same-shard source or a complete backup. See [Configuration failures and recovery sources](/reference/v5/replication/residency#configuration-failures-and-peer-identity). + +Co-owner snapshot recovery does not restore the replacement’s former local audit lineage. Cluster record-lock freshness treats the recovered database as recloned, including its ordinary tables; use a complete backup to preserve that history. See [Recovery sources](/reference/v5/replication/residency#recovery-sources). + +Snapshot-only recovery of a sharded node requires a fresh root and a new node name. Reusing the original name requires a complete stopped-node backup preserving audit through the last accepted write. A reused name missing its origin history refuses sharded writes and recovery-source eligibility, with both remedies in cluster status. See [sharding recovery](/reference/v5/replication/residency#recovering-a-lost-node). ### LMDB Replication Confirmation From cb94b25a2ec8ac5b387056fa4c2f7c7db94208c0 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 26 Oct 2026 12:08:56 -0600 Subject: [PATCH 22/32] docs: preserve ordinary writes during sharding root recovery Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/replication/residency.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/reference/replication/residency.md b/reference/replication/residency.md index 7a427ca2d..11dfe0e29 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -153,6 +153,8 @@ Choose one recovery route before restarting a node whose local data or audit his A node reusing a name whose earlier origin history is known to peers, without the corresponding local history, refuses sharded writes and cannot serve as an owned-slot recovery source. Its recovery status reports `origin-history-lost` and names both remedies: restore a complete backup, or rejoin under a new node name. Waiting for its clock to advance does not repair the missing history. +While this node lacks its frozen origin start, outgoing replication for the affected database is held, including ordinary tables sharing its stream. Ordinary local writes can still succeed; that response does not confirm delivery to another node. Stop all application writes to that database before recovery, preserve any unique ordinary-table writes accepted after the failure, and export or restore them before replacing the root. Other databases remain independent. + Co-owners continue relaying the old name's retained audit entries after membership removal. Keep those co-owners and their audit history available until lagging peers have received the old writes; membership removal does not replace normal audit retention requirements. The new name's writes use a separate origin, so receiving them does not certify receipt of the old origin's pending writes. If an entire shard was lost, a new name alone cannot recover its bodies: restore a complete backup. Each name restored without its original audit needs the snapshot-only route above. A new storage root in an initialized database cannot borrow another root's origin history; recover missing history before enabling writes. From 13df85f9366b652b608dad1ce2e181398f3a458b Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 26 Oct 2026 15:30:20 -0600 Subject: [PATCH 23/32] docs: clarify sharding placement and recovery boundaries Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/database/api.md | 2 ++ reference/replication/clustering.md | 18 ++++++++++-------- reference/replication/residency.md | 10 +++++++--- reference/replication/sharding.md | 4 +++- reference/resources/resource-api.md | 4 +++- release-notes/v5-lincoln/5.4.md | 2 +- 6 files changed, 26 insertions(+), 14 deletions(-) diff --git a/reference/database/api.md b/reference/database/api.md index ea01768e0..905b4f47b 100644 --- a/reference/database/api.md +++ b/reference/database/api.md @@ -138,6 +138,8 @@ Two details matter outside a request context: Set `replicatedConfirmation` on a fresh context before calling `transaction()` to request confirmation for its writes. Pass the callback's `txn` argument to table methods to bind them explicitly. Reusing a context with an active transaction does not change that transaction's confirmation settings. See the [independent-transaction example](../replication/sharding.md#programmatic-replication-control) and [uncertain-outcome behavior](../replication/sharding.md#confirmation-outcomes). +For `@sharded` tables, confirmation counts other owners in each affected owner set of the first written database; a temporary replica cannot satisfy it. `replicateTo` does not change those owners. See [declarative ownership](../replication/residency.md#ownership-reads-and-writes). + ### Nesting If `transaction()` is called with a context that already has an active transaction, it reuses that transaction, executes the callback immediately, and returns. This makes `transaction()` safe to call defensively to ensure a transaction is always active. diff --git a/reference/replication/clustering.md b/reference/replication/clustering.md index 336afae4f..15ea7ab8d 100644 --- a/reference/replication/clustering.md +++ b/reference/replication/clustering.md @@ -208,14 +208,16 @@ While a database is taking its initial full copy, these timing fields hold the l The top-level `cluster_status.sharding` field is keyed by database and describes active or attempted declarative tables. It is empty in the legacy-table example above. Each entry has: -| Field | Meaning | -| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `version`, `digest` | Installed map identity; absent if no map is installed. | -| `shardsDigest` | Digest of the configured shard set, or `null` when no list is configured. | -| `blocked` | Whether a reported configuration or map-agreement condition blocks placement. | -| `recovery` | Bootstrap/recovery phase and reason. `origin-history-lost` refuses a reused name without its audit; restore a complete backup or rejoin with a fresh root under a new node name. | -| `waitingOrigins` | Origins waiting for a direct source or certified same-shard relay, with the missing source described. | -| `errors` | Reasons, such as differing peer lists, a local shard outside its list, a missing required list/map, or a refused peer map. | +| Field | Meaning | +| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `version`, `digest` | Installed map identity; absent if no map is installed. | +| `shardsDigest` | Digest of the configured shard set, or `null` when no list is configured. | +| `blocked` | Whether a configuration/map error or a recovery phase other than `ready` blocks placement. | +| `state` | `quarantined` for a startup quarantine, `held` for a live configuration/map error, the current `recovery.phase` while recovery is incomplete, or `available`. | +| `restartRequired` | `true` for startup quarantine: repair the configuration and restart the full process. `false` does not mean recovery is complete. | +| `recovery` | Bootstrap/recovery phase and reason, including a missing existing-owner completeness assertion in `recovery.reason`. `origin-history-lost` refuses a reused name without its audit; restore a complete backup or rejoin with a fresh root under a new node name. | +| `waitingOrigins` | Origins waiting for a direct source or certified same-shard relay, with the missing source described. | +| `errors` | Reasons, such as differing peer lists, a local shard outside its list, a missing required list/map, or a refused peer map. | A clear status is not proof that an owner is connected; write admission also checks the required replication route and, for an attribute-slot transition from a writer outside the new owner set, a connected new owner. See [configuration agreement](./sharding.md#complete-shard-id-set) and [map recovery](./residency.md#installed-maps). diff --git a/reference/replication/residency.md b/reference/replication/residency.md index 11dfe0e29..6f95da7e0 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -10,7 +10,7 @@ Schema-defined sharding requires RocksDB. `@sharded` and `initialize_shard_map` `@sharded` fixes a table's placement policy in its schema. Every application, operations, and replication thread applies the same policy. A key or record selects a slot, the database's installed slot map selects a shard, and `replication.shard` identifies that shard's current nodes. -Upgrade every schema and data writer before activation. Wire compatibility with earlier peers allows delivery and reads; it does not make an earlier writer enforce the policy. Fallback to peers that cannot advertise a shard-list digest requires an installed custom map and no configured list on either side; otherwise setup disagrees and placement is blocked. +Upgrade every schema and data writer before activation. Wire compatibility with earlier peers allows delivery and reads; it does not make an earlier writer enforce the policy. With an installed custom map, recovery-capable peers may omit `replication.shards` only when neither side configures a list. This configuration fallback does not waive the sharding-recovery capability requirement: an older peer can receive compatible records and serve reads, but its announcement holds sharded writes until it is upgraded or removed. ## Configuration @@ -81,7 +81,9 @@ A key-policy table without secondary indexes stores nothing on non-owners; reads ### Regional placement and attribute-slot changes -`@sharded(by: "region", slotFrom: ...)` with `initialize_shard_map` is how to meet data-residency requirements. The expression selects logical slots; the map assigns those slots to regional shards. The schema does not encode shard IDs or node names. +`@sharded(by: "region", slotFrom: ...)` with `initialize_shard_map` places full owned records on regional shards. The expression selects logical slots; the map assigns those slots to shards. The schema does not encode shard IDs or node names. + +Owner placement is not a boundary on every copy or transfer. Non-owners retain indexed projections, can cache full read/write replicas, and retain audit bodies for writes they accept. Reads can transfer the complete body across regions. Setting `@table(replicaExpiration: 0)` prevents new temporary replicas; it does not purge existing replicas or writer audit history. Account for indexed fields, request routing, caches and audit retention when planning regional data placement. Install this map before declaring the table. Configure Asia's nodes with `replication.shard: 0` and Europe's nodes with `replication.shard: 1`: @@ -109,7 +111,7 @@ The omitted count adopts the regional database's three slots. `asia` selects slo Changing `region` sends the complete canonical body to the new owners and invalidates old owners at the same version. This handoff uses full-record replacement ordering: an older concurrent patch does not merge into the replacement. Use application concurrency controls when writers can update a record while its slot changes. When the writer is outside the new owner set, success requires at least one new owner to confirm durable receipt. If no new owner has an active subscription and recent received activity within [`replication.pingTimeout`](../configuration/options.md#replication), the change is refused before commit. This can also refuse an owner whose base copy is not progressing. -Automatic slot-move confirmation waits up to `replication.shardingConfirmationTimeout` (default `30000` ms). Explicit `replicatedConfirmation` uses `replication.confirmationTimeout` instead (default `900000` ms, raised to `blobTimeout` when the confirmation setting is omitted). Both settings are read when a confirmation starts. An explicit setting can lower that timeout independently of blob transfers. Slow transfers can therefore exceed the automatic timeout even on a healthy link. +Automatic slot-move confirmation waits up to `replication.shardingConfirmationTimeout` (default `30000` ms). Explicit `replicatedConfirmation` uses `replication.confirmationTimeout` instead (default `900000` ms, raised to `replication.blobTimeout` when the configuration key `replication.confirmationTimeout` is unset). Both settings are read when a confirmation starts. An explicit setting can lower that timeout independently of blob transfers. Slow transfers can therefore exceed the automatic timeout even on a healthy link. A connection can fail after admission. An error with `replicationUnconfirmed: true` after local commit means the outcome is uncertain; it does not mean the transaction rolled back. Inspect the record and retry with the application's concurrency controls. Audit retention bounds replay for disconnected peers; it is not an indefinite handoff queue. @@ -265,6 +267,8 @@ Key-derived conversion retains existing non-owner full copies locally; it does n ### Rollback +The in-place procedure below applies to key-derived tables converted from an equivalent `setResidencyById` policy. It does not restore full replication for a table that had no residency function: removing `@sharded` cannot backfill full bodies on former non-owners. For that conversion, restore a complete preactivation backup instead. Preserve and reconcile writes accepted after the backup before restoring it, or those writes can be lost. + For a key-derived policy, drain cached RocksDB replicas before changing binaries: an earlier binary can interpret one as a full owned record. Key policies themselves need no slot-format conversion. 1. Back up the database. Pause application requests and background jobs on **every node**, including reads that could admit another replica. Keep them paused through the downgrade. diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index 175ef54c3..90a334d0c 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -20,7 +20,7 @@ By default (without sharding), Harper replicates all data to all nodes. ## Schema-defined sharding -In v5.4, use [`@sharded`](./residency.md) for a shared schema policy across all workers. `@sharded` hashes primary keys, `@sharded(by: "tenantId")` keeps a tenant together, and a compiled `slotFrom` expression plus a custom map supports regional data-residency requirements. Slot counts belong to the database. See the [one-slot-per-value limitation](./residency.md#limitations). +In v5.4, use [`@sharded`](./residency.md) for a shared schema policy across all workers. `@sharded` hashes primary keys, `@sharded(by: "tenantId")` keeps a tenant together, and a compiled `slotFrom` expression plus a custom map places owners on regional shards. [Regional placement](./residency.md#regional-placement-and-attribute-slot-changes) also explains the projections, replicas, audit bodies and read transfers that can cross those regions. Slot counts belong to the database. See the [one-slot-per-value limitation](./residency.md#limitations). ## Older per-record mechanisms @@ -32,6 +32,8 @@ The `setResidency`, `setResidencyById`, `getResidency` and residency-list APIs b ## Dynamic Sharding +The configuration, REST, Operations API and programmatic `replicateTo` examples below control legacy residency. On `@sharded` tables, `replicateTo` does not override the installed map: only the map determines owners. See [ownership, reads, and writes](./residency.md#ownership-reads-and-writes). + ### Replication Count The simplest way to limit replication is to configure a replication count. Set `replicateTo` in the `replication` section of `harper-config.yaml` to specify how many additional nodes data should be replicated to: diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 7bc962206..9368cf0d7 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -247,9 +247,11 @@ All properties are optional: | `includeSuperseded` | Include record mutations superseded by a later version. Defaults to `false`, or `true` when `rawEvents` is enabled. An explicit value overrides that default. | | `rawEvents` | Return raw audit events rather than reconstructing full records. Defaults to `false`. | | `omitCurrent` | Do not send the current/retained record as the first update. | -| `rowFilter` | Synchronous JavaScript predicate applied to authoritative row values. | +| `rowFilter` | Synchronous JavaScript predicate applied to row payloads; sharded non-owner invalidations contain only indexed projections. | | `eventFilter` | Synchronous JavaScript predicate for events that may not carry an authoritative row. | +On a sharded non-owner, `rowFilter` sees indexed fields from the invalidation payload even when a full replica is cached locally. Use indexed fields in these predicates: missing unindexed fields can suppress matching updates or satisfy a negated predicate. Do not use such a predicate to decide whether a subscriber may learn an unindexed value. See [subscription filters](../replication/residency.md#subscription-filters). + #### Superseded record updates diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 62c4c87be..55dc9b4b6 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -28,7 +28,7 @@ These corrections are not released yet. Sharding configuration failures are isolated to the affected database. A live disagreement holds that database until the configuration is repaired on all peers; healthy databases in the same application keep serving, including during worker replacement. Invalid configuration at process startup still requires a repaired full restart because its audit replay was deferred. First-time sharding initialization waits for the configured members; owner recovery requires a certified same-shard source or a complete backup. See [Configuration failures and recovery sources](/reference/v5/replication/residency#configuration-failures-and-peer-identity). -Co-owner snapshot recovery does not restore the replacement’s former local audit lineage. Cluster record-lock freshness treats the recovered database as recloned, including its ordinary tables; use a complete backup to preserve that history. See [Recovery sources](/reference/v5/replication/residency#recovery-sources). +Co-owner snapshot recovery does not restore the replacement's former local audit lineage. Cluster record-lock freshness treats the recovered database as recloned, including its ordinary tables; use a complete backup to preserve that history. See [Recovery sources](/reference/v5/replication/residency#recovery-sources). Snapshot-only recovery of a sharded node requires a fresh root and a new node name. Reusing the original name requires a complete stopped-node backup preserving audit through the last accepted write. A reused name missing its origin history refuses sharded writes and recovery-source eligibility, with both remedies in cluster status. See [sharding recovery](/reference/v5/replication/residency#recovering-a-lost-node). From afdbff699936b051673257720b3553d140f39657 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 26 Oct 2026 23:46:52 -0600 Subject: [PATCH 24/32] docs: explain local-only slot transition refusal Describe the application-only refusal before native submission, internal apply exemptions, ordinary local-only behavior and affected-database rollback boundary. Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/replication/residency.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/reference/replication/residency.md b/reference/replication/residency.md index 6f95da7e0..e48249d8b 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -111,6 +111,10 @@ The omitted count adopts the regional database's three slots. `asia` selects slo Changing `region` sends the complete canonical body to the new owners and invalidates old owners at the same version. This handoff uses full-record replacement ordering: an older concurrent patch does not merge into the replacement. Use application concurrency controls when writers can update a record while its slot changes. When the writer is outside the new owner set, success requires at least one new owner to confirm durable receipt. If no new owner has an active subscription and recent received activity within [`replication.pingTimeout`](../configuration/options.md#replication), the change is refused before commit. This can also refuse an owner whose base copy is not progressing. +An application or operation write marked `localOnly` is refused before commit if its attribute change moves the record to a slot whose owners exclude the writer. The error names the table and record and explains that a local-only write cannot deliver the moved record to its new owners. Nothing is committed. Local-only updates that leave the slot unchanged, or move it to owners that still include the writer, succeed without replication confirmation. This refusal does not apply to replication receipt, snapshot copy-apply, or other internal apply paths. Replicable slot moves still require the new-owner acknowledgment described above. + +Ordinary local-only creates and same-slot updates keep the existing local placement: a non-owner can retain only a projection or omit the row, and its local-only audit body is not delivered to owners. The new refusal applies only to an attribute-slot transition away from the writer. Rollback covers the affected database's pending transaction; it does not undo earlier commits in another database. + Automatic slot-move confirmation waits up to `replication.shardingConfirmationTimeout` (default `30000` ms). Explicit `replicatedConfirmation` uses `replication.confirmationTimeout` instead (default `900000` ms, raised to `replication.blobTimeout` when the configuration key `replication.confirmationTimeout` is unset). Both settings are read when a confirmation starts. An explicit setting can lower that timeout independently of blob transfers. Slow transfers can therefore exceed the automatic timeout even on a healthy link. A connection can fail after admission. An error with `replicationUnconfirmed: true` after local commit means the outcome is uncertain; it does not mean the transaction rolled back. Inspect the record and retry with the application's concurrency controls. Audit retention bounds replay for disconnected peers; it is not an indefinite handoff queue. From a266db2e0ac0a5a3fc64320aba05d3014ceffaab Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 26 Oct 2026 23:57:39 -0600 Subject: [PATCH 25/32] Clarify local-only writer replica retention Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/replication/residency.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/replication/residency.md b/reference/replication/residency.md index e48249d8b..1c14e1f95 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -113,7 +113,7 @@ Changing `region` sends the complete canonical body to the new owners and invali An application or operation write marked `localOnly` is refused before commit if its attribute change moves the record to a slot whose owners exclude the writer. The error names the table and record and explains that a local-only write cannot deliver the moved record to its new owners. Nothing is committed. Local-only updates that leave the slot unchanged, or move it to owners that still include the writer, succeed without replication confirmation. This refusal does not apply to replication receipt, snapshot copy-apply, or other internal apply paths. Replicable slot moves still require the new-owner acknowledgment described above. -Ordinary local-only creates and same-slot updates keep the existing local placement: a non-owner can retain only a projection or omit the row, and its local-only audit body is not delivered to owners. The new refusal applies only to an attribute-slot transition away from the writer. Rollback covers the affected database's pending transaction; it does not undo earlier commits in another database. +Ordinary local-only creates and same-slot updates keep the existing local placement: a non-owner can retain a writer replica, keep an indexed projection, or omit the row, depending on the table's replica and index settings. Its local-only audit body is not delivered to owners. The new refusal applies only to an attribute-slot transition away from the writer. Rollback covers the affected database's pending transaction; it does not undo earlier commits in another database. Automatic slot-move confirmation waits up to `replication.shardingConfirmationTimeout` (default `30000` ms). Explicit `replicatedConfirmation` uses `replication.confirmationTimeout` instead (default `900000` ms, raised to `replication.blobTimeout` when the configuration key `replication.confirmationTimeout` is unset). Both settings are read when a confirmation starts. An explicit setting can lower that timeout independently of blob transfers. Slow transfers can therefore exceed the automatic timeout even on a healthy link. From dfdaa20548c60814d879343f93bc432b285f0892 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 27 Oct 2026 10:04:00 -0600 Subject: [PATCH 26/32] Document the attribute-slot retry gap and marker-gated refusal Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/replication/residency.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/reference/replication/residency.md b/reference/replication/residency.md index 1c14e1f95..6383e5fcb 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -203,6 +203,10 @@ There is no operation to delete or reset an active map. If an accidentally initi ## Limitations +A known concurrent-write gap affects attribute-derived slots. If a PATCH encounters a native storage retry while a concurrent PUT changes the slot-deriving attribute, the retried PATCH can land in a new slot without delivering the complete record to its new owners. Key-derived slots are unaffected. This gap remains when transaction-log abort markers are unavailable or disabled; it closes once the storage binding supplies them and Harper enables them. + +With markers active for the logged transaction, Harper instead refuses a retry whose merged slot changed, returning a retryable 409 that names the table and record. The refused write commits nothing. Re-execute the application operation as a fresh attempt so normal placement, local-only refusal and new-owner confirmation rules apply. Harper does not automatically re-run the application callback or serialize attribute-slot writes. + Owner recovery requires the [sources described above](#recovery-sources). Arbitrary cross-shard owner recovery is deferred to [#1123](https://github.com/HarperFast/harper-pro/issues/1123). Every record with the same `by` value lands in one slot. One region or large tenant therefore cannot be split across shards merely by moving slots. A future `spread` argument (name open) is tracked in [#1122](https://github.com/HarperFast/harper-pro/issues/1122). It would distribute one value over a range using the primary-key hash, with `spread: 1` preserving today's behavior; moving those slots would be handled by #1053. This argument is not available in this release. From 4094d27873b6ac4f7787fd57505536711ec589fa Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 27 Oct 2026 16:10:30 -0600 Subject: [PATCH 27/32] Clarify the binding affected by the attribute-slot retry gap Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/replication/residency.md | 2 ++ release-notes/v5-lincoln/5.4.md | 2 ++ 2 files changed, 4 insertions(+) diff --git a/reference/replication/residency.md b/reference/replication/residency.md index 6383e5fcb..dd16a2243 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -205,6 +205,8 @@ There is no operation to delete or reset an active map. If an accidentally initi A known concurrent-write gap affects attribute-derived slots. If a PATCH encounters a native storage retry while a concurrent PUT changes the slot-deriving attribute, the retried PATCH can land in a new slot without delivering the complete record to its new owners. Key-derived slots are unaffected. This gap remains when transaction-log abort markers are unavailable or disabled; it closes once the storage binding supplies them and Harper enables them. +The RocksDB 2.12.0 binding does not provide this protection. The fixes are tracked in [rocksdb-js#970](https://github.com/HarperFast/rocksdb-js/pull/970) and [#973](https://github.com/HarperFast/rocksdb-js/pull/973); Harper must also enable the markers. This sharding feature does not enable them or add an operator setting for them. Until that rollout, avoid overlapping PATCHes with slot-changing PUTs on the same record. A successful response alone does not prove that this race delivered the complete record to its new owners. + With markers active for the logged transaction, Harper instead refuses a retry whose merged slot changed, returning a retryable 409 that names the table and record. The refused write commits nothing. Re-execute the application operation as a fresh attempt so normal placement, local-only refusal and new-owner confirmation rules apply. Harper does not automatically re-run the application callback or serialize attribute-slot writes. Owner recovery requires the [sources described above](#recovery-sources). Arbitrary cross-shard owner recovery is deferred to [#1123](https://github.com/HarperFast/harper-pro/issues/1123). diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 55dc9b4b6..50b898667 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -26,6 +26,8 @@ These corrections are not released yet. `@sharded` applies a fixed key- or attribute-based slot policy on every thread, including dedicated replication workers. Per-database slot maps remain authoritative after initialization; `replication.shards` supplies the complete deployment shard set for a first default map. Super-user map initialization and read-only conversion checks support adopting an equivalent legacy by-id policy without rewriting records. Attribute-slot changes from a writer outside the new owner set require a new owner's confirmation. See [Schema-defined sharding](/reference/v5/replication/residency), including its storage-format and rollback requirements. +Attribute-derived slots have a known retry race when a PATCH overlaps a slot-changing PUT: the new owners can miss the complete record. RocksDB 2.12.0 lacks the transaction-log abort markers needed to refuse this retry safely. See [Limitations](/reference/v5/replication/residency#limitations) for the trigger and marker rollout requirements; key-derived slots are unaffected. + Sharding configuration failures are isolated to the affected database. A live disagreement holds that database until the configuration is repaired on all peers; healthy databases in the same application keep serving, including during worker replacement. Invalid configuration at process startup still requires a repaired full restart because its audit replay was deferred. First-time sharding initialization waits for the configured members; owner recovery requires a certified same-shard source or a complete backup. See [Configuration failures and recovery sources](/reference/v5/replication/residency#configuration-failures-and-peer-identity). Co-owner snapshot recovery does not restore the replacement's former local audit lineage. Cluster record-lock freshness treats the recovered database as recloned, including its ordinary tables; use a complete backup to preserve that history. See [Recovery sources](/reference/v5/replication/residency#recovery-sources). From 21c2f9983a56fa6c221097e71696d1e7f67f1864 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 27 Oct 2026 18:22:29 -0600 Subject: [PATCH 28/32] docs: describe the 5.4 abort-marker default and minimal retry fallback Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/replication/residency.md | 6 ++---- release-notes/v5-lincoln/5.4.md | 2 +- 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/reference/replication/residency.md b/reference/replication/residency.md index dd16a2243..44e0f1ac4 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -203,11 +203,9 @@ There is no operation to delete or reset an active map. If an accidentally initi ## Limitations -A known concurrent-write gap affects attribute-derived slots. If a PATCH encounters a native storage retry while a concurrent PUT changes the slot-deriving attribute, the retried PATCH can land in a new slot without delivering the complete record to its new owners. Key-derived slots are unaffected. This gap remains when transaction-log abort markers are unavailable or disabled; it closes once the storage binding supplies them and Harper enables them. +For attribute-derived slots, a PATCH racing a slot-changing PUT returns a retryable 409 naming the table and record when its logged native retry has abort markers. Re-execute the application operation as a fresh attempt; Harper does not automatically re-run the callback. An invalid slot expression then receives the normal pre-log 400. -The RocksDB 2.12.0 binding does not provide this protection. The fixes are tracked in [rocksdb-js#970](https://github.com/HarperFast/rocksdb-js/pull/970) and [#973](https://github.com/HarperFast/rocksdb-js/pull/973); Harper must also enable the markers. This sharding feature does not enable them or add an operator setting for them. Until that rollout, avoid overlapping PATCHes with slot-changing PUTs on the same record. A successful response alone does not prove that this race delivered the complete record to its new owners. - -With markers active for the logged transaction, Harper instead refuses a retry whose merged slot changed, returning a retryable 409 that names the table and record. The refused write commits nothing. Re-execute the application operation as a fresh attempt so normal placement, local-only refusal and new-owner confirmation rules apply. Harper does not automatically re-run the application callback or serialize attribute-slot writes. +This race is closed once abort markers are enabled (5.4 default, [rocksdb-js#970](https://github.com/HarperFast/rocksdb-js/pull/970)/[#973](https://github.com/HarperFast/rocksdb-js/pull/973); Harper enablement pending); without them, a logged retry commits at its admitted slot. Owner recovery requires the [sources described above](#recovery-sources). Arbitrary cross-shard owner recovery is deferred to [#1123](https://github.com/HarperFast/harper-pro/issues/1123). diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 50b898667..ab88b9ed2 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -26,7 +26,7 @@ These corrections are not released yet. `@sharded` applies a fixed key- or attribute-based slot policy on every thread, including dedicated replication workers. Per-database slot maps remain authoritative after initialization; `replication.shards` supplies the complete deployment shard set for a first default map. Super-user map initialization and read-only conversion checks support adopting an equivalent legacy by-id policy without rewriting records. Attribute-slot changes from a writer outside the new owner set require a new owner's confirmation. See [Schema-defined sharding](/reference/v5/replication/residency), including its storage-format and rollback requirements. -Attribute-derived slots have a known retry race when a PATCH overlaps a slot-changing PUT: the new owners can miss the complete record. RocksDB 2.12.0 lacks the transaction-log abort markers needed to refuse this retry safely. See [Limitations](/reference/v5/replication/residency#limitations) for the trigger and marker rollout requirements; key-derived slots are unaffected. +Attribute-derived-slot retries return a retryable 409 when placement changes after logging; abort markers are planned on by default in 5.4 (Harper enablement pending). See [Limitations](/reference/v5/replication/residency#limitations). Sharding configuration failures are isolated to the affected database. A live disagreement holds that database until the configuration is repaired on all peers; healthy databases in the same application keep serving, including during worker replacement. Invalid configuration at process startup still requires a repaired full restart because its audit replay was deferred. First-time sharding initialization waits for the configured members; owner recovery requires a certified same-shard source or a complete backup. See [Configuration failures and recovery sources](/reference/v5/replication/residency#configuration-failures-and-peer-identity). From 9c368713055b49e0860db5f3f5e2b264d61ce218 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 27 Oct 2026 22:39:51 -0600 Subject: [PATCH 29/32] Document primary-key sharding and defer concurrent attribute moves Dispatch-Task: hp-980-slot-residency Co-Authored-By: GPT-5 Codex --- reference/configuration/options.md | 1 - reference/database/schema.md | 4 +- reference/operations-api/operations.md | 2 +- reference/replication/residency.md | 92 ++++---------------------- reference/replication/sharding.md | 4 +- release-notes/v5-lincoln/5.4.md | 4 +- 6 files changed, 17 insertions(+), 90 deletions(-) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index d62764e27..0bb74a60c 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -289,7 +289,6 @@ replication: - `pingInterval` — Interval between replication keep-alive pings (ms); _Default_: `30000` - `pingTimeout` — Time without received socket activity before a replication connection is considered dead (ms); _Default_: twice `pingInterval`. Automatic slot-move admission also requires a new owner with received activity inside this window. - `confirmationTimeout` — Timeout in milliseconds for explicit `replicatedConfirmation`, read when a waiter starts. _Default_: `900000`, or `blobTimeout` when greater. An explicitly configured value overrides that fallback, including a smaller value. Range: `1`–`2147483647`. (Added in: v5.4.0) -- `shardingConfirmationTimeout` — Timeout in milliseconds for automatic new-owner confirmation on an attribute-slot change. _Default_: `30000`. Independent of blob-transfer timeout. A timeout returns an uncertain-outcome 504 after local commit; it does not roll back the write. Range: `1`–`2147483647`. (Added in: v5.4.0) - `blobTimeout` — Blob transfer timeout (ms); _Default_: `120000` - `blobGapReconnectMs` — Interval (ms) for the blob-gap watchdog: when a transient blob save failure pins a replication resume cursor, the connection is forced to reconnect on this cadence so the gapped blob is re-streamed and, during a bulk copy, the copy resumes from the last banked cursor. Lower values heal gaps faster at the cost of more reconnects on a link whose faults never heal; _Default_: the `blobTimeout` value - `copyCursorFlushBytes` — Bytes of applied bulk-copy data between durable flushes of the copy resume cursor (RocksDB); _Default_: `67108864` diff --git a/reference/database/schema.md b/reference/database/schema.md index ccc86eaf5..11061d121 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -146,7 +146,7 @@ On a table whose records live on specific nodes (see [sharding](../replication/s - **`replicaExpiration`** — Seconds a replica is kept. A sweep runs every quarter of this time, with a minimum interval of one second, and reduces replicas past their lifetime back to their indexed stubs. A replica can still be served until the sweep reaches it; the next read after demotion fetches the record again. `0` disables new replica admissions and the sweep. A replica never outlives the record's own `expiration`, if it has one. - **`replicaRefreshFraction`** — A read of a replica after this fraction of `replicaExpiration` has passed renews the lifetime, so records that stay in use stay local. Must be greater than `0` and at most `1`; `1` gives every replica a fixed lifetime. -[Schema-defined sharding](../replication/residency.md#ownership-reads-and-writes) tables with secondary indexes can retain replicas for either key- or attribute-derived slots. With positive `replicaExpiration`, a non-owner writer also keeps its complete write as a temporary replica under the same expiry and replacement rules. This local copy never counts as owner confirmation. Setting the lifetime to `0` does not drain existing replicas; see the [rollback procedure](../replication/residency.md#rollback). +[Schema-defined sharding](../replication/residency.md#ownership-reads-and-writes) tables with secondary indexes can retain replicas for key-derived slots. With positive `replicaExpiration`, a non-owner writer also keeps its complete write as a temporary replica under the same expiry and replacement rules. This local copy never counts as owner confirmation. Setting the lifetime to `0` does not drain existing replicas; see the [rollback procedure](../replication/residency.md#rollback). ```graphql # Keep fetched records for a day; a read after six hours renews them @@ -244,7 +244,7 @@ See [REST Headers / Cache-Control](../rest/headers.md#cache-control) for the ful -Defines an immutable placement policy on a schema-defined RocksDB table. An LMDB declaration is refused within its database; legacy residency functions on LMDB are unchanged. `@sharded` hashes the primary key; `by: "tenantId"` hashes that attribute's non-null value with the same hash. `slotFrom` optionally chooses the integer logical slot with a compiled deterministic expression. `slots` is a database-wide value (`1` through `65536`): omit it to adopt the installed map's count, or `1024` if no map exists; an explicit different count is refused. Attribute policies require a secondary index. The database map resolves each slot to a deployment shard. See [Schema-defined sharding](../replication/residency.md) for tenant affinity, regional placement, configuration, conversion, limitations and rollback. +Defines immutable primary-key placement on a schema-defined RocksDB table. LMDB declarations are refused; legacy residency functions are unchanged. `slots` is the database-wide count (`1` through `65536`), adopting the installed map's count or defaulting to `1024`. The key hash selects a slot and the installed database map selects its shard. `by` and `slotFrom` are not supported yet; see [concurrent-move design (#1178)](https://github.com/HarperFast/harper-pro/issues/1178) and [Schema-defined sharding](../replication/residency.md). ### `@export` diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 1073d1441..0897cde81 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -1818,7 +1818,7 @@ For a legacy conversion, call with `database`, `existing_owner: true` and `expec -POST; requires `super_user`. Takes `database`, `table` and a proposed key-derived `policy` (`slots`, optional `slotFrom`). The existing application worker must still run its legacy function, and a custom map must be installed. Returns `checked`, `mismatches`, bounded samples, observed slots, departed node names (stored legacy owners absent from the current known-node list) and map identity. It neither rewrites records nor activates the policy. See [Compare placement](../replication/residency.md#3-compare-placement-on-every-node). +POST; requires `super_user`. Takes `database`, `table` and a proposed key-derived `policy` (`slots`). The existing application worker must still run its legacy function, and a custom map must be installed. Returns `checked`, `mismatches`, bounded samples, observed slots, departed node names (stored legacy owners absent from the current known-node list) and map identity. It neither rewrites records nor activates the policy. See [Compare placement](../replication/residency.md#3-compare-placement-on-every-node). ### `cluster_status` diff --git a/reference/replication/residency.md b/reference/replication/residency.md index 44e0f1ac4..c06e88948 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -8,7 +8,7 @@ Schema-defined sharding requires RocksDB. `@sharded` and `initialize_shard_map` -`@sharded` fixes a table's placement policy in its schema. Every application, operations, and replication thread applies the same policy. A key or record selects a slot, the database's installed slot map selects a shard, and `replication.shard` identifies that shard's current nodes. +`@sharded` fixes a table's placement policy in its schema. Every application, operations, and replication thread applies the same policy. The primary key selects a slot, the database's installed slot map selects a shard, and `replication.shard` identifies that shard's current nodes. Upgrade every schema and data writer before activation. Wire compatibility with earlier peers allows delivery and reads; it does not make an earlier writer enforce the policy. With an installed custom map, recovery-capable peers may omit `replication.shards` only when neither side configures a list. This configuration fallback does not waive the sharding-recovery capability requirement: an older peer can receive compatible records and serve reads, but its announcement holds sharded writes until it is upgraded or removed. @@ -35,89 +35,27 @@ type Item @table @sharded { } ``` -| Argument | Meaning | -| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `slots` | Optional database-wide integer from `1` through `65536`. Uses the installed map's count, or `1024` when no map exists. An explicit count must equal the database map's count. | -| `by` | Optional stored, declared attribute name; may be the primary key, but cannot be computed. Default placement hashes its non-null string or finite-number value. Requires a secondary index, including when naming the primary key. | -| `slotFrom` | Optional deterministic JavaScript expression returning an integer in `[0, slots)`. Without `by`, it exposes `id`; with `by`, it also exposes `record` and its attribute names. | +`slots` is the optional database-wide count, an integer from `1` through `65536`. It adopts the installed map's count, or defaults to `1024`. An explicit count must equal the installed map's count. -Declarative tables require [per-table audit logging](../database/transaction.md#enabling-the-transaction-log-per-table) and table replication. Harper refuses activation with `@table(audit: false)`, effective `logging.auditLog: false`, or `@table(replicate: false)`, and refuses disabling these table settings later. The replication route must also reach an owner. +The slot is the unsigned FNV-1a 32-bit hash of the primary key's ordered-binary encoding, modulo that count. The offset basis is `2166136261` and multiplier `16777619`. Numeric `-0` and `0` use the same encoding; numeric `5` and string `"5"` are distinct inputs. Updating other fields cannot change the slot. -Without `slotFrom`, the slot is the unsigned FNV-1a 32-bit hash of the primary key's ordered-binary encoding, modulo the database slot count. With `by`, the same hash uses that attribute's value instead. Default hashing supports strings and finite numbers; a null or missing value, `NaN`, or infinity is refused. Numeric `-0` and `0` use the same encoding for placement. Numeric `5` and string `"5"` remain distinct typed hash inputs. Default `by` declarations must name a stored `ID`, `String`, `Int`, `Float`, `Long`, or `Any` field; other stored types require `slotFrom`, and relationship or computed fields cannot be used. The hash's offset basis is `2166136261` and its multiplier is `16777619`. +Only primary-key hashing is supported. Schema declarations using `by` or `slotFrom` are refused. Attribute-derived placement and custom expressions await [concurrent-move design (#1178)](https://github.com/HarperFast/harper-pro/issues/1178). -`@sharded(by: "tenantId")` keeps each tenant's records together: +Declarative tables require [per-table audit logging](../database/transaction.md#enabling-the-transaction-log-per-table) and table replication. Harper refuses activation with `@table(audit: false)`, effective `logging.auditLog: false`, or `@table(replicate: false)`, and refuses disabling these table settings later. The replication route must reach an owner. -```graphql -type TenantItem @table(database: "tenant_data") @sharded(by: "tenantId") { - id: ID @primaryKey - tenantId: String @indexed - body: String -} -``` - -Different record keys with the same `tenantId` select the same slot on every node. The database map then resolves that slot to its current shard. - -An explicit key policy can preserve an existing partition function: - -```graphql -type Item @table @sharded(slots: 10, slotFrom: "Number(id.slice(-1))") { - id: ID @primaryKey - category: String @indexed - body: String -} -``` - -Expressions must be self-contained, deterministic, and synchronous. They compile on every thread without application helpers. An expression that throws, returns a Promise, or produces a non-integer or out-of-range result refuses the write before commit. Invalid-result errors name the result and database slot count. The `by` value must still be non-null when an expression is supplied. The last-digit example assumes string keys, matching the existing function being converted. Schema expressions are trusted code, as with computed expressions; they are not a sandbox. - -The policy is immutable once activated, including changing `by`, changing or adding/removing `slotFrom`, and removing the directive. These changes could move existing records and are not a slot-migration operation. Redeploying the same policy is idempotent. Calling `setResidency` or `setResidencyById` on a declarative table fails; remove those calls when converting. Existing tables can adopt a key-derived policy with an equivalent custom map. Attribute policies must be declared on a new table because existing rows have no stored slots. +The policy is immutable once activated; changing its count or removing the directive is refused. Redeploying the same policy is idempotent. `setResidency` and `setResidencyById` are refused on declarative tables. Existing tables can adopt primary-key hashing only with a map that preserves their placement. ## Ownership, reads, and writes Owners store full records. With secondary indexes, other nodes keep index projections and receive invalidations. Reads fetch the full body from an owner. A non-owner can cache a temporary replica. Replicas retain the canonical record identity and do not become owners or count toward owner confirmation. -A key-policy table without secondary indexes stores nothing on non-owners; reads resolve the owner from the key. Such tables do not cache replicas. Attribute policies require a secondary index so a non-owner can retain the stored slot. If an attribute record's slot is unknown locally, DELETE returns 503 with a message beginning `Attribute sharding slot is unknown`. This response does not distinguish a delayed projection from a never-created key or a purged tombstone; local absence cannot prove that the record is absent on its owners. If another node still has the slot metadata, send the deletion there. If no node has it, DELETE cannot succeed anywhere; retrying alone cannot fix that condition. Stop automatic retries and use application-level knowledge to establish whether the key ever existed. Local absence is not proof that it was deleted. +A table without secondary indexes stores nothing on non-owners; reads resolve the owner from the key. Such tables do not cache replicas. `replicateTo` is ignored for placement on a declarative table; the installed slot map determines where full records are retained. A non-owner accepts a local write and sends the complete body through replication to the owners. On an indexed RocksDB table with positive `replicaExpiration`, it also retains the complete written body as a temporary local replica, allowing an immediate local read while delivery is pending. A newer invalidation replaces it, and the replica sweep expires it. With expiration zero, it retains only the index projection; an index-free key policy retains nothing. A replica is a cache, so crash recovery may restore a projection instead. Ingest-heavy non-owners should budget space for approximately their serialized bytes written per second multiplied by `replicaExpiration`, or set expiration to zero. File-backed audit bodies remain until audit retention permits reclamation, even after a replica expires; include that delivery history in the disk budget. Refetched cache files have their own replica lifetime. This requires a configured replication route to an owner. Cross-shard query fan-out and write forwarding are not provided by this directive. Application database branches do not support sharding because branches do not replicate. -### Regional placement and attribute-slot changes +Local-only writes succeed without replication confirmation. They do not deliver a local audit body to owners. The node retains a writer replica, index projection, or no row according to its replica and index settings. -`@sharded(by: "region", slotFrom: ...)` with `initialize_shard_map` places full owned records on regional shards. The expression selects logical slots; the map assigns those slots to shards. The schema does not encode shard IDs or node names. - -Owner placement is not a boundary on every copy or transfer. Non-owners retain indexed projections, can cache full read/write replicas, and retain audit bodies for writes they accept. Reads can transfer the complete body across regions. Setting `@table(replicaExpiration: 0)` prevents new temporary replicas; it does not purge existing replicas or writer audit history. Account for indexed fields, request routing, caches and audit retention when planning regional data placement. - -Install this map before declaring the table. Configure Asia's nodes with `replication.shard: 0` and Europe's nodes with `replication.shard: 1`: - -```json -{ - "operation": "initialize_shard_map", - "database": "regional", - "slots": 3, - "ranges": [ - { "start": 0, "end": 2, "shard": 0 }, - { "start": 2, "end": 3, "shard": 1 } - ] -} -``` - -```graphql -type RegionalItem @table(database: "regional") @sharded(by: "region", slotFrom: "region == 'asia' ? 1 : 2") { - id: ID @primaryKey - region: String @indexed - body: String -} -``` - -The omitted count adopts the regional database's three slots. `asia` selects slot 1 on shard 0; other non-null region values select slot 2 on shard 1. Slot 0 remains available for other policies. The map contains slot ranges and shard IDs, never pinned attribute values. - -Changing `region` sends the complete canonical body to the new owners and invalidates old owners at the same version. This handoff uses full-record replacement ordering: an older concurrent patch does not merge into the replacement. Use application concurrency controls when writers can update a record while its slot changes. When the writer is outside the new owner set, success requires at least one new owner to confirm durable receipt. If no new owner has an active subscription and recent received activity within [`replication.pingTimeout`](../configuration/options.md#replication), the change is refused before commit. This can also refuse an owner whose base copy is not progressing. - -An application or operation write marked `localOnly` is refused before commit if its attribute change moves the record to a slot whose owners exclude the writer. The error names the table and record and explains that a local-only write cannot deliver the moved record to its new owners. Nothing is committed. Local-only updates that leave the slot unchanged, or move it to owners that still include the writer, succeed without replication confirmation. This refusal does not apply to replication receipt, snapshot copy-apply, or other internal apply paths. Replicable slot moves still require the new-owner acknowledgment described above. - -Ordinary local-only creates and same-slot updates keep the existing local placement: a non-owner can retain a writer replica, keep an indexed projection, or omit the row, depending on the table's replica and index settings. Its local-only audit body is not delivered to owners. The new refusal applies only to an attribute-slot transition away from the writer. Rollback covers the affected database's pending transaction; it does not undo earlier commits in another database. - -Automatic slot-move confirmation waits up to `replication.shardingConfirmationTimeout` (default `30000` ms). Explicit `replicatedConfirmation` uses `replication.confirmationTimeout` instead (default `900000` ms, raised to `replication.blobTimeout` when the configuration key `replication.confirmationTimeout` is unset). Both settings are read when a confirmation starts. An explicit setting can lower that timeout independently of blob transfers. Slow transfers can therefore exceed the automatic timeout even on a healthy link. - -A connection can fail after admission. An error with `replicationUnconfirmed: true` after local commit means the outcome is uncertain; it does not mean the transaction rolled back. Inspect the record and retry with the application's concurrency controls. Audit retention bounds replay for disconnected peers; it is not an indefinite handoff queue. +Explicit `replicatedConfirmation` uses `replication.confirmationTimeout` (default `900000` ms, raised to `replication.blobTimeout` when unset). An error with `replicationUnconfirmed: true` after local commit means the requested confirmation was not obtained; it does not mean rollback. Inspect the record and apply the application's concurrency controls before retrying. ### Subscription filters @@ -203,14 +141,10 @@ There is no operation to delete or reset an active map. If an accidentally initi ## Limitations -For attribute-derived slots, a PATCH racing a slot-changing PUT returns a retryable 409 naming the table and record when its logged native retry has abort markers. Re-execute the application operation as a fresh attempt; Harper does not automatically re-run the callback. An invalid slot expression then receives the normal pre-log 400. - -This race is closed once abort markers are enabled (5.4 default, [rocksdb-js#970](https://github.com/HarperFast/rocksdb-js/pull/970)/[#973](https://github.com/HarperFast/rocksdb-js/pull/973); Harper enablement pending); without them, a logged retry commits at its admitted slot. +Attribute-derived sharding and custom slot expressions are unavailable pending [concurrent-move design (#1178)](https://github.com/HarperFast/harper-pro/issues/1178). Slot migration is separate work in [#1053](https://github.com/HarperFast/harper-pro/issues/1053). Owner recovery requires the [sources described above](#recovery-sources). Arbitrary cross-shard owner recovery is deferred to [#1123](https://github.com/HarperFast/harper-pro/issues/1123). -Every record with the same `by` value lands in one slot. One region or large tenant therefore cannot be split across shards merely by moving slots. A future `spread` argument (name open) is tracked in [#1122](https://github.com/HarperFast/harper-pro/issues/1122). It would distribute one value over a range using the primary-key hash, with `spread: 1` preserving today's behavior; moving those slots would be handled by #1053. This argument is not available in this release. - ## Older per-record residency APIs `setResidency`, `setResidencyById`, `getResidency`, and residency lists retain their names and remain the older per-record mechanism. Use the conversion procedure below when adopting schema-defined sharding for an existing by-id table. @@ -223,7 +157,7 @@ Back up the database and upgrade all writers. Keep the current `setResidencyById ### 2. Install the equivalent map -Install a custom map that reproduces the function's numeric shard result exactly, from one designated node. Wait for the same snapshot on every participant before enabling the directive. On a newly joining node in a custom-map deployment, install or receive that snapshot before deploying its declarative application; otherwise first activation would initialize a default map from the local list. +The existing function must agree with the supported primary-key hash and the proposed map for every stored record. Arbitrary legacy expressions cannot be adopted yet. Install the equivalent map from one designated node. Wait for the same snapshot on every participant before enabling the directive. On a newly joining node in a custom-map deployment, install or receive that snapshot before deploying its declarative application; otherwise first activation would initialize a default map from the local list. ### 3. Compare placement on every node @@ -234,7 +168,7 @@ Run `verify_shard_map` on each node while its application worker still runs the "operation": "verify_shard_map", "database": "data", "table": "Item", - "policy": { "slots": 10, "slotFrom": "Number(id.slice(-1))" } + "policy": { "slots": 10 } } ``` @@ -302,5 +236,3 @@ await drainResidencyReplicas(tables.Item); 5. Remove the maintenance code, restore the previous binary with the equivalent legacy schema and `setResidencyById` function, then resume traffic. Keep the installed slot map for a later upgrade. Local assertion receipts also survive restart and downgrade: their trust remains the operator's responsibility while the equivalent legacy placement is in use, not a new verification by Harper on re-upgrade. There is no built-in replica-drain operation. If you cannot run and verify this maintenance procedure, restore a preactivation backup instead of attempting an in-place downgrade. - -An attribute policy writes a new local primary/audit format. In-place downgrade to an earlier binary is unsupported. Restore a backup from before activation; writes since that backup may be lost. Protocol fallback does not make an earlier binary understand the new local storage format. diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index 90a334d0c..3c97ada26 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -20,7 +20,7 @@ By default (without sharding), Harper replicates all data to all nodes. ## Schema-defined sharding -In v5.4, use [`@sharded`](./residency.md) for a shared schema policy across all workers. `@sharded` hashes primary keys, `@sharded(by: "tenantId")` keeps a tenant together, and a compiled `slotFrom` expression plus a custom map places owners on regional shards. [Regional placement](./residency.md#regional-placement-and-attribute-slot-changes) also explains the projections, replicas, audit bodies and read transfers that can cross those regions. Slot counts belong to the database. See the [one-slot-per-value limitation](./residency.md#limitations). +In v5.4, [`@sharded`](./residency.md) hashes primary keys to fixed slots on every worker. Slot counts and the installed slot map belong to the database. Attribute-derived policies and custom expressions are refused pending [concurrent-move design (#1178)](https://github.com/HarperFast/harper-pro/issues/1178). ## Older per-record mechanisms @@ -162,8 +162,6 @@ Pre-commit refusals, including a confirmation count that cannot be met, do not c Confirmation currently applies to the first database a transaction writes to. This outcome flag covers the linked local commits; it does not enable confirmation for later databases. -[Attribute-slot transitions](./residency.md#regional-placement-and-attribute-slot-changes) can require new-owner confirmation automatically even when the caller did not request an explicit confirmation count. - ## Static Sharding ### Basic Static Shard Configuration diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index ab88b9ed2..fdf1b06a0 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -24,9 +24,7 @@ These corrections are not released yet. ### Sharding -`@sharded` applies a fixed key- or attribute-based slot policy on every thread, including dedicated replication workers. Per-database slot maps remain authoritative after initialization; `replication.shards` supplies the complete deployment shard set for a first default map. Super-user map initialization and read-only conversion checks support adopting an equivalent legacy by-id policy without rewriting records. Attribute-slot changes from a writer outside the new owner set require a new owner's confirmation. See [Schema-defined sharding](/reference/v5/replication/residency), including its storage-format and rollback requirements. - -Attribute-derived-slot retries return a retryable 409 when placement changes after logging; abort markers are planned on by default in 5.4 (Harper enablement pending). See [Limitations](/reference/v5/replication/residency#limitations). +`@sharded` hashes primary keys to fixed slots on every thread, including dedicated replication workers. Per-database slot maps remain authoritative after initialization; `replication.shards` supplies the deployment shard set for a first default map. Conversion checks support equivalent legacy by-id policies without rewriting records. Attribute-derived policies and custom expressions are refused pending [concurrent-move design (#1178)](https://github.com/HarperFast/harper-pro/issues/1178). See [Schema-defined sharding](/reference/v5/replication/residency). Sharding configuration failures are isolated to the affected database. A live disagreement holds that database until the configuration is repaired on all peers; healthy databases in the same application keep serving, including during worker replacement. Invalid configuration at process startup still requires a repaired full restart because its audit replay was deferred. First-time sharding initialization waits for the configured members; owner recovery requires a certified same-shard source or a complete backup. See [Configuration failures and recovery sources](/reference/v5/replication/residency#configuration-failures-and-peer-identity). From 6c4312ce7be566faf5cd948186c86902a2371276 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 27 Oct 2026 23:00:16 -0600 Subject: [PATCH 30/32] Remove leftover attribute admission prose and clarify key conversion eligibility Dispatch-Task: hp-980-slot-residency Dispatch-Task: docs-753-rebase-20261028 Dispatch-Task: pr-maint-2615cc297c5c7e582179e0c7f65fb1b3 Co-Authored-By: GPT-5 Codex Co-Authored-By: Gemini --- reference/configuration/options.md | 2 +- reference/replication/clustering.md | 2 +- reference/replication/residency.md | 4 +++- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 0bb74a60c..c219c1d53 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -287,7 +287,7 @@ replication: - `threads` — Worker threads dedicated to replication, which then own the replication port and every subscription and load no application code; requires `port` or `securePort`; see [Dedicated Replication Threads](../replication/overview.md#dedicated-replication-threads); _Default_: `0` (replication runs on the HTTP worker threads) (Added in: v5.4.0) - `enableRootCAs` — Verify against Node.js Mozilla CA store; _Default_: `true` - `pingInterval` — Interval between replication keep-alive pings (ms); _Default_: `30000` -- `pingTimeout` — Time without received socket activity before a replication connection is considered dead (ms); _Default_: twice `pingInterval`. Automatic slot-move admission also requires a new owner with received activity inside this window. +- `pingTimeout` — Time without received socket activity before a replication connection is considered dead (ms); _Default_: twice `pingInterval`. - `confirmationTimeout` — Timeout in milliseconds for explicit `replicatedConfirmation`, read when a waiter starts. _Default_: `900000`, or `blobTimeout` when greater. An explicitly configured value overrides that fallback, including a smaller value. Range: `1`–`2147483647`. (Added in: v5.4.0) - `blobTimeout` — Blob transfer timeout (ms); _Default_: `120000` - `blobGapReconnectMs` — Interval (ms) for the blob-gap watchdog: when a transient blob save failure pins a replication resume cursor, the connection is forced to reconnect on this cadence so the gapped blob is re-streamed and, during a bulk copy, the copy resumes from the last banked cursor. Lower values heal gaps faster at the cost of more reconnects on a link whose faults never heal; _Default_: the `blobTimeout` value diff --git a/reference/replication/clustering.md b/reference/replication/clustering.md index 15ea7ab8d..f35142dfd 100644 --- a/reference/replication/clustering.md +++ b/reference/replication/clustering.md @@ -219,7 +219,7 @@ The top-level `cluster_status.sharding` field is keyed by database and describes | `waitingOrigins` | Origins waiting for a direct source or certified same-shard relay, with the missing source described. | | `errors` | Reasons, such as differing peer lists, a local shard outside its list, a missing required list/map, or a refused peer map. | -A clear status is not proof that an owner is connected; write admission also checks the required replication route and, for an attribute-slot transition from a writer outside the new owner set, a connected new owner. See [configuration agreement](./sharding.md#complete-shard-id-set) and [map recovery](./residency.md#installed-maps). +A clear status is not proof that an owner is connected; write admission also checks the required replication route. See [configuration agreement](./sharding.md#complete-shard-id-set) and [map recovery](./residency.md#installed-maps). ### Shard map operations diff --git a/reference/replication/residency.md b/reference/replication/residency.md index c06e88948..906c4ebbb 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -55,7 +55,7 @@ A table without secondary indexes stores nothing on non-owners; reads resolve th Local-only writes succeed without replication confirmation. They do not deliver a local audit body to owners. The node retains a writer replica, index projection, or no row according to its replica and index settings. -Explicit `replicatedConfirmation` uses `replication.confirmationTimeout` (default `900000` ms, raised to `replication.blobTimeout` when unset). An error with `replicationUnconfirmed: true` after local commit means the requested confirmation was not obtained; it does not mean rollback. Inspect the record and apply the application's concurrency controls before retrying. +Explicit `replicatedConfirmation` uses `replication.confirmationTimeout` (when the configuration key is unset, the greater of `900000` ms and `replication.blobTimeout`). An error with `replicationUnconfirmed: true` after local commit means the requested confirmation was not obtained; it does not mean rollback. Inspect the record and apply the application's concurrency controls before retrying. ### Subscription filters @@ -151,6 +151,8 @@ Owner recovery requires the [sources described above](#recovery-sources). Arbitr ## Convert an existing by-id table without moving records +Check eligibility before installing a map: the legacy function must agree with the primary-key hash and proposed ranges for every record. Last-digit or modulo recipes are not generally equivalent to this hash. Compare the functions offline first; custom `slotFrom` policies remain unsupported pending [#1178](https://github.com/HarperFast/harper-pro/issues/1178). An installed map cannot be deleted. + ### 1. Prepare the existing table Back up the database and upgrade all writers. Keep the current `setResidencyById` function running during the comparison. From 0aba8cf72a6c27399f431dee3701ece123460374 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 2 Nov 2026 16:08:47 -0700 Subject: [PATCH 31/32] Point confirmation-failure guidance at the single explanation The Confirmation Failure After a Local Commit section in sharding.md (from #740) is the one place that explains replicationUnconfirmed and retry safety. Replace the restatement in residency.md with a short link, and point the API reference and 5.4 release note at that section rather than the LMDB-only Confirmation outcomes section. Dispatch-Task: docs-753-rebase-20261102 Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_017xoBAASx3a6cCDKgFGeZem --- reference/database/api.md | 2 +- reference/replication/residency.md | 2 +- release-notes/v5-lincoln/5.4.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/reference/database/api.md b/reference/database/api.md index 905b4f47b..e48b907ae 100644 --- a/reference/database/api.md +++ b/reference/database/api.md @@ -136,7 +136,7 @@ Two details matter outside a request context: ### Replication confirmation -Set `replicatedConfirmation` on a fresh context before calling `transaction()` to request confirmation for its writes. Pass the callback's `txn` argument to table methods to bind them explicitly. Reusing a context with an active transaction does not change that transaction's confirmation settings. See the [independent-transaction example](../replication/sharding.md#programmatic-replication-control) and [uncertain-outcome behavior](../replication/sharding.md#confirmation-outcomes). +Set `replicatedConfirmation` on a fresh context before calling `transaction()` to request confirmation for its writes. Pass the callback's `txn` argument to table methods to bind them explicitly. Reusing a context with an active transaction does not change that transaction's confirmation settings. See the [independent-transaction example](../replication/sharding.md#programmatic-replication-control), [uncertain-outcome behavior](../replication/sharding.md#confirmation-failure-after-a-local-commit), and [LMDB confirmation behavior](../replication/sharding.md#confirmation-outcomes). For `@sharded` tables, confirmation counts other owners in each affected owner set of the first written database; a temporary replica cannot satisfy it. `replicateTo` does not change those owners. See [declarative ownership](../replication/residency.md#ownership-reads-and-writes). diff --git a/reference/replication/residency.md b/reference/replication/residency.md index 906c4ebbb..9616814d8 100644 --- a/reference/replication/residency.md +++ b/reference/replication/residency.md @@ -55,7 +55,7 @@ A table without secondary indexes stores nothing on non-owners; reads resolve th Local-only writes succeed without replication confirmation. They do not deliver a local audit body to owners. The node retains a writer replica, index projection, or no row according to its replica and index settings. -Explicit `replicatedConfirmation` uses `replication.confirmationTimeout` (when the configuration key is unset, the greater of `900000` ms and `replication.blobTimeout`). An error with `replicationUnconfirmed: true` after local commit means the requested confirmation was not obtained; it does not mean rollback. Inspect the record and apply the application's concurrency controls before retrying. +Explicit `replicatedConfirmation` uses `replication.confirmationTimeout` (when the configuration key is unset, the greater of `900000` ms and `replication.blobTimeout`). A failure after the local commit does not roll the write back; see [Confirmation Failure After a Local Commit](./sharding.md#confirmation-failure-after-a-local-commit). ### Subscription filters diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index fdf1b06a0..99de04070 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -34,7 +34,7 @@ Snapshot-only recovery of a sharded node requires a fresh root and a new node na ### LMDB Replication Confirmation -LMDB writes that request `replicatedConfirmation` now wait for peer acknowledgments. Applications that previously received immediate success can now observe replication latency or an uncertain committed outcome if confirmation fails. See [Confirmation outcomes](/reference/v5/replication/sharding#confirmation-outcomes) before retrying a failed write. +LMDB writes that request `replicatedConfirmation` now wait for peer acknowledgments. Applications that previously received immediate success can now observe replication latency or an uncertain committed outcome if confirmation fails. See [Confirmation Failure After a Local Commit](/reference/v5/replication/sharding#confirmation-failure-after-a-local-commit) before retrying a failed write. ### Dedicated Replication Threads From cfdf7b2af6d91bf320cd03551192ac2ff0f64f55 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 2 Nov 2026 16:18:39 -0700 Subject: [PATCH 32/32] Correct what the rowFilter warnings say a subscriber can learn Non-owner invalidation payloads carry only indexed fields, so the exposure a negated predicate creates is a record's id and indexed fields, not an unindexed value. Match the wording in residency.md on the resource API and schema pages. Dispatch-Task: docs-753-rebase-20261102 Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_017xoBAASx3a6cCDKgFGeZem --- reference/database/schema.md | 2 +- reference/resources/resource-api.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/database/schema.md b/reference/database/schema.md index 11061d121..d56f1850c 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -775,7 +775,7 @@ export class Reports extends tables.Reports { For vector queries, `rowFilter` participates in HNSW traversal. A caller therefore receives the k nearest _matching_ records rather than "nearest k, minus filtered records." `rowFilter` can apply to every candidate visited; prefer indexed query conditions for predicates that can be expressed declaratively. -On a non-owner for a sharded table, invalidation values contain only the local index projection. A `rowFilter` on an unindexed field therefore sees it as missing; see the [subscription limitation](../replication/residency.md#subscription-filters). +On a non-owner for a sharded table, invalidation values contain only the local index projection. A `rowFilter` on an unindexed field therefore sees it as missing, so a negated predicate can pass; do not use one to decide whether a subscriber may learn a record's id or indexed fields. See the [subscription limitation](../replication/residency.md#subscription-filters). `rowFilter` applies only when a non-raw `put` or `invalidate` event carries a row value. Delete tombstones, value-less invalidations, raw events, and published messages may not contain the complete current record. A subscription that needs such events can also provide an `eventFilter`: diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 9368cf0d7..9e115f41d 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -250,7 +250,7 @@ All properties are optional: | `rowFilter` | Synchronous JavaScript predicate applied to row payloads; sharded non-owner invalidations contain only indexed projections. | | `eventFilter` | Synchronous JavaScript predicate for events that may not carry an authoritative row. | -On a sharded non-owner, `rowFilter` sees indexed fields from the invalidation payload even when a full replica is cached locally. Use indexed fields in these predicates: missing unindexed fields can suppress matching updates or satisfy a negated predicate. Do not use such a predicate to decide whether a subscriber may learn an unindexed value. See [subscription filters](../replication/residency.md#subscription-filters). +On a sharded non-owner, `rowFilter` sees indexed fields from the invalidation payload even when a full replica is cached locally. Use indexed fields in these predicates: missing unindexed fields can suppress matching updates or satisfy a negated predicate. Do not use such a predicate to decide whether a subscriber may learn a record's id or indexed fields. See [subscription filters](../replication/residency.md#subscription-filters). #### Superseded record updates