docs: clarify RocksDB no-fsync durability and LMDB-only writeAsync

reference/configuration/options.md

@@ -280,7 +280,7 @@ replication:
 
 - `hostname` <VersionBadge type="changed" version="v5.3.0" /> — This instance's hostname within the cluster. Subject to the same bare hostname or IP literal requirement as [`node.hostname`](#node) — a URL, `host:port`, or non-string value fails startup. When both are set, `node.hostname` wins.
 - `url` — WebSocket URL peers use to connect to this instance
-- `databases` — Databases to replicate; _Default_: `"*"` (all). Each entry supports `name` and `sharded`
+- `databases` — Databases to replicate; _Default_: `"*"` (all). Each entry supports `name`, `replicateTo`, and `sharded`
 - `routes` — Peer nodes; URL strings or `{hostname, port, startTime, revokedCertificates}` objects
 - `port` — Replication port
 - `securePort` — Secure replication port; _Default_: `9933` (changed from `9925` in v4.5.0)
@@ -325,7 +325,7 @@ storage:
 
   The runtime budgets apply together and are not hard bounds on total process memory. See [Set Configuration](./operations.md#set-configuration) for when request and native runtime changes take effect. Use the usual underscore-separated parameter names, such as `storage_fullText_maxRawHitsPerRequest`. See [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for costs; see [readiness and recovery](../resources/query-optimization.md#readiness-and-recovery) for failures.
 
-- `writeAsync` — Disable disk sync for higher throughput (**disables durability guarantees**); _Default_: `false`
+- `writeAsync` <EngineBadge engines="LMDB" /> — Disable disk sync on commit for the legacy LMDB storage engine (**disables local disk durability guarantees on LMDB**). Has no effect on RocksDB, which does not `fsync` commits; see [Storage Tuning — Durability vs. Throughput](../database/storage-tuning.md#durability-vs-throughput); _Default_: `false`
 - `caching` — In-memory record caching; _Default_: `true`
 - `compression` — LZ4 record compression; _Default_: `true` (enabled by default since v4.3.0). Sub-options: `dictionary`, `threshold`
 - `compactOnStart` — Compact all non-system databases on startup; _Default_: `false` (Added in: v4.3.0)

reference/database/overview.md

@@ -110,7 +110,7 @@ For deeper coverage of each database feature, see the dedicated pages in this se
 - **[Full-Text Search](../full-text-search/overview.md)** — BM25-ranked search over stored text fields on audited RocksDB tables
 - **[API](./api.md)** — The `tables`, `databases`, `transaction()`, and `createBlob()` globals for interacting with the database from code
 - **[Data Loader](./data-loader.md)** — Loading seed or initial data into tables as part of component deployment
-- **[Storage Algorithm](./storage-algorithm.md)** — How Harper stores data using LMDB with universal indexing and ACID compliance
+- **[Storage Algorithm](./storage-algorithm.md)** — How Harper stores data using RocksDB (or legacy LMDB) with universal indexing and ACID transactions
 - **[Storage Tuning](./storage-tuning.md)** — Tuning `storage.*` options for production: durability vs. throughput, compression, blob paths, reclamation
 - **[Jobs](./jobs.md)** — Asynchronous bulk data operations (CSV import/export, S3 import/export)
 - **[System Tables](./system-tables.md)** — Harper internal tables for analytics, data loader state, and other system features

reference/database/storage-algorithm.md

@@ -7,7 +7,7 @@ title: Storage Algorithm
 
 # Storage Algorithm
 
-Harper's storage algorithm is the foundation of all database functionality. It is built on top of [RocksDB](https://rocksdb.org/) (the default) or [LMDB](https://www.symas.com/lmdb) (legacy), both high-performance key-value stores, and extends them with automatic indexing, query-language-agnostic data access, and ACID compliance.
+Harper's storage algorithm is the foundation of all database functionality. It is built on top of [RocksDB](https://rocksdb.org/) (the default) or [LMDB](https://www.symas.com/lmdb) (legacy), both high-performance key-value stores, and extends them with automatic indexing, query-language-agnostic data access, and ACID transaction guarantees.
 
 RocksDB is the default storage engine for new installations. LMDB databases from prior versions are still supported and loaded automatically when detected.
 
@@ -17,12 +17,12 @@ Harper's storage layer is decoupled from any specific query language. Data inser
 
 ## ACID Compliance
 
-Harper provides full ACID compliance on each node:
+Harper provides ACID transaction guarantees:
 
 - **Atomicity**: All writes in a transaction either fully commit or fully roll back
 - **Consistency**: Each transaction moves data from one valid state to another
 - **Isolation**: Reads use snapshots and do not block writes; writes do not block reads
-- **Durability**: RocksDB commits are persisted via its Write-Ahead Log (WAL); LMDB uses memory-mapped file writes
+- **Durability**: On RocksDB, when the [transaction log](./transaction.md) is enabled (the default), committed writes are recorded in Harper's transaction log in the OS page cache (without `fsync`; RocksDB's own WAL is disabled) and replayed on startup, surviving Harper process crashes on the node (disabling the transaction log via `logging.auditLog: false` or `@table(audit: false)` also disables process-crash recovery for unflushed memtable writes). Durability across OS, power, or host failures is provided in a Harper Pro cluster by per-write [replication confirmation](../replication/sharding.md) across nodes in separate failure domains (see [Storage Tuning — Durability vs. Throughput](./storage-tuning.md#durability-vs-throughput)). LMDB uses memory-mapped file writes (`fsync` on commit by default unless [`storage.writeAsync`](./storage-tuning.md#storagewriteasync) is enabled)
 
 Harper uses application-level locking to serialize schema changes and table creation, ensuring write ordering without deadlocks.
 
@@ -69,7 +69,7 @@ Harper inherits strong performance properties from its storage engines:
 
 - **LSM-tree writes**: Optimized for write-heavy workloads via log-structured merge trees
 - **Block cache**: Configurable in-memory block cache (defaults to 25% of available system memory)
-- **WAL durability**: Write-Ahead Log provides crash recovery without sacrificing throughput
+- **Transaction log recovery**: When transaction logging is enabled (the default), commits are written to Harper's transaction log in the OS page cache without blocking on `fsync` and replayed on startup for process-crash recovery
 - **Compression**: Native support for multiple compression algorithms per level
 
 **LMDB (legacy)**:

reference/database/storage-tuning.md

@@ -10,21 +10,38 @@ For a quick reference of every option, see [Configuration Options — `storage`]
 
 ## Durability vs. Throughput
 
+On RocksDB (the default storage engine), Harper does not call `fsync` or `fdatasync` when committing a transaction, and RocksDB's own Write-Ahead Log (WAL) is disabled for table stores. When the [transaction log](./transaction.md) is enabled (the default via [`logging.auditLog: true`](../logging/configuration.md), inherited by tables unless [`@table(audit: false)`](./transaction.md#enabling-the-transaction-log-per-table)), each acknowledged commit is written to Harper's transaction log (`database/<db>/transaction_logs/`) using a standard write to the OS page cache. On startup, Harper replays the transaction log to recover any committed writes that had not yet been flushed from RocksDB's in-memory memtables to table files.
+
+What an acknowledged write survives on a single node:
+
+- **Harper process crash or restart**: Survived when transaction logging is enabled (the default). Committed bytes are already in the OS page cache, so the operating system preserves and flushes them to disk even if the Harper process terminates unexpectedly, and Harper replays the transaction log on restart. If transaction logging is disabled globally (`logging.auditLog: false`) or for a table (`@table(audit: false)`), no transaction-log entry is written while RocksDB's WAL remains off, so unflushed memtable writes are also lost on a Harper process crash.
+- **OS crash, kernel panic, power loss, or host failure**: Recent writes that the OS has not yet flushed from the page cache to disk can be lost. On a single-node or Harper core deployment (where replication is not available), durability against OS or power failure depends on periodic OS page-cache writeback. The database remains structurally consistent — only the most recent un-flushed commits are at risk.
+
+For durability against machine, OS, power, or storage failure in a multi-node (Harper Pro) cluster, Harper relies on **replication confirmation** rather than local disk `fsync`. In production failures such as host failure or storage hardware loss, a local disk fails or becomes unreachable together with the rest of the machine, whereas an acknowledgement from a peer node on an independent machine means the write has been committed off-host. Because confirming RocksDB peers also commit to their own OS page cache without `fsync` and confirmation counts any `<k>` responding peers, surviving a power outage or site-wide failure requires peers in separate power or failure domains and setting `<k>` greater than the number of peers that share the writer's failure domain.
+
+To require confirmation from `<k>` peer nodes before a write returns, use [replication confirmation](../replication/sharding.md):
+
+- **Operations API**: Set [`replicatedConfirmation: <k>`](../replication/sharding.md#replication-control-via-operations-api) in the operation payload. Omit `replicateTo` to use the database's configured replication (every node by default, unless [`replication.databases[].replicateTo`](../replication/sharding.md#replication-count) is set), or set `replicateTo` only when you also want to limit residency (dynamic sharding) to a fixed set of nodes.
+- **Resource API**: Set [`context.replicatedConfirmation = <k>`](../replication/sharding.md#programmatic-replication-control) in your resource method (leaving `context.replicateTo` unset to use the database's configured replication, unless you also want to restrict residency).
+- **REST**: Pass [`X-Replicate-To: *;confirm=<k>`](../replication/sharding.md#replication-control-via-rest-header) on the request (requires `super_user` permissions; for non-superuser write paths, set `context.replicatedConfirmation` in a Resource method instead) to wait for `<k>` confirmations while keeping the database's configured replication (`*` leaves per-request residency unset; passing a number `<n>` or node list in `X-Replicate-To: <n>;confirm=<k>` pins the record's residency to `<n>` additional nodes from those present at write time).
+
 ### `storage.writeAsync`
 
+<EngineBadge engines="LMDB" />
+
 Type: `boolean`
 
 Default: `false`
 
-Disables `fsync` on commit. Writes return as soon as data is queued to the page cache, dramatically increasing throughput on write-heavy workloads.
+Applies only to the legacy LMDB storage engine. This setting has no effect on RocksDB, which never calls `fsync` on commit.
 
-**This disables durability guarantees.** A power loss or OS crash between the application commit and the OS flushing pages to disk can lose recently committed transactions. The database itself remains structurally consistent — only the most recent writes are at risk.
+On LMDB, enabling `writeAsync` disables `fsync` on commit so writes return as soon as data is queued to the OS page cache, matching RocksDB's commit behavior and increasing throughput on write-heavy workloads. A power loss or OS crash before the OS flushes pages to disk can lose recently committed transactions, while the database remains structurally consistent.
 
-Enable only when:
+Enable on LMDB databases when:
 
+- Durability across machine failures is handled via [replication confirmation](../replication/sharding.md).
 - The data is reproducible from an upstream source (e.g., caches with `sourcedFrom`).
-- The workload is bulk ingest where some loss on crash is acceptable and the operation can be re-run.
-- Replication provides durability — peers acknowledge writes before they could be lost.
+- The workload is bulk ingest where some loss on OS crash or power loss is acceptable and the operation can be re-run.
 
 ```yaml
 storage:
@@ -367,9 +384,10 @@ Retains the pre-compaction backup files after `compactOnStart` runs. Recommended
 
 **Write-heavy ingest, durability via replication:**
 
+On RocksDB (the default), commits already avoid local disk `fsync`; pair a higher queue-time threshold in `harper-config.yaml` with per-write [replication confirmation](../replication/sharding.md) (`replicatedConfirmation` in the Operations or Resource API, or `X-Replicate-To: *;confirm=<k>` over REST — set on each write operation or request, not in `harper-config.yaml`) when writes must be confirmed by peer nodes before returning (on legacy LMDB databases, also add `writeAsync: true`):
+
 ```yaml
 storage:
-  writeAsync: true
   prefetchWrites: true
   maxTransactionQueueTime: 60s
 ```
@@ -409,6 +427,7 @@ The [storage option reference](../configuration/options.md#storage) lists the fu
 
 - [Configuration Options](../configuration/options.md) — full list of `storage` options
 - [Storage Algorithm](./storage-algorithm.md) — how Harper stores records and indexes on disk
+- [Sharding](../replication/sharding.md) — configuring `replicateTo` and replication confirmation for cross-node durability
 - [Schema — `@table(randomAccessFields:)`](./schema.md#randomaccessfields) — pinning one table's record encoding
 - [Compaction](./compaction.md) — reclaiming space inside existing database files
 - [Resource API — `sourcedFrom`](../resources/resource-api.md#sourcedfromresource-options) — caching tables that interact with reclamation

reference/replication/sharding.md

@@ -30,14 +30,17 @@ There are two main approaches:
 
 ### Replication Count
 
-The simplest way to limit replication is to configure a replication count. Set `replicateTo` in the `replication` section of `harper-config.yaml` to specify how many additional nodes data should be replicated to:
+The simplest way to limit replication is to configure a per-database replication count. Set `replicateTo` on a database entry under `replication.databases` in `harper-config.yaml` to specify how many additional nodes data should be replicated to:
 
 ```yaml
 replication:
-  replicateTo: 2
+  databases:
+    - system
+    - name: data
+      replicateTo: 2
 ```
 
-This ensures each record is stored on three nodes total (the node that first stored it, plus two others).
+This ensures each record in `data` is stored on three nodes total (the node that first stored it, plus two others). Because setting `replication.databases` to a list replaces the `'*'` default (all databases), include `system` and every other database the cluster should continue replicating (see [Data Selection](./overview.md#data-selection)).
 
 ### Replication Control via REST Header
 
@@ -58,7 +61,12 @@ PUT /MyTable/3
 X-Replicate-To: node1,node2
 ```
 
-The `confirm` parameter can be combined with explicit node lists.
+The `confirm` parameter can be combined with explicit node lists, or with `*` to require confirmation while keeping the database's configured replication (no per-request residency; passing `X-Replicate-To` requires `super_user` permissions, and non-superuser write paths can set [`context.replicatedConfirmation`](#programmatic-replication-control) in a Resource method):
+
+```http
+PUT /MyTable/3
+X-Replicate-To: *;confirm=1
+```
 
 ### Replication Control via Operations API