Document storage.rocks.transactionLogAbortMarkers and its 5.4 default

reference/analytics/overview.md

@@ -208,7 +208,9 @@ outstanding commit on a thread exceeds
 (default 45s), Harper rejects new record updates and publishes from new application requests on that
 thread with `Outstanding write transactions have too long of queue, please try again later` (HTTP
 503). Deletes and writes applied from a canonical source (e.g. replication or a caching source)
-bypass this check.
+bypass this check. The same duration also limits how long a write keeps retrying conflicts with other
+writes, which fails with a separate 503, code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT`; see
+[`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers).
 
 <VersionBadge type="changed" version="v5.2.1" />
 

reference/configuration/operations.md

@@ -16,7 +16,7 @@ For the full list of configurable options, see [Configuration Options](./options
 
 ## Set Configuration
 
-Modifies one or more Harper configuration parameters. **Most changes require a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`). A change to `replication.threads` takes effect only on a full `restart`.
+Modifies one or more Harper configuration parameters. **Most changes require a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`, [`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) and [`storage.rocks.transactionLogWriteIntents`](../database/storage-tuning.md#storagerockstransactionlogwriteintents)). A change to `replication.threads` takes effect only on a full `restart`.
 
 Full-text request work limits (`storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds`) take effect in HTTP workers after `restart_service` or a full restart. Changes to `storage.fullText.runtime` require a full Harper process restart; `restart_service` does not replace the native process budget. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging).
 
@@ -86,7 +86,7 @@ To restart the whole cluster afterward, follow with [restart_service](../operati
 }
 ```
 
-`restart_service` restarts only worker threads, which keep the process's V8 flags. A change to [`threads.v8Flags`](./options.md#applying-v8-flags) takes effect only after a [restart](../operations-api/operations.md#restart), sent to each node in turn.
+`restart_service` restarts only worker threads, which keep the process's V8 flags and its `storage.rocks.transactionLogAbortMarkers` and `storage.rocks.transactionLogWriteIntents` values. A change to [`threads.v8Flags`](./options.md#applying-v8-flags), [`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) or [`storage.rocks.transactionLogWriteIntents`](../database/storage-tuning.md#storagerockstransactionlogwriteintents) takes effect only after a [restart](../operations-api/operations.md#restart), sent to each node in turn.
 
 ---
 

reference/configuration/options.md

@@ -330,7 +330,7 @@ storage:
 - `compression` — LZ4 record compression; _Default_: `true` (enabled by default since v4.3.0). Sub-options: `dictionary`, `threshold`
 - `compactOnStart` — Compact all non-system databases on startup; _Default_: `false` (Added in: v4.3.0)
 - `compactOnStartKeepBackup` — Retain compaction backups; _Default_: `false`
-- `maxTransactionQueueTime` — Per-thread write-commit duration threshold for HTTP 503 backpressure; see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionqueuetime); _Default_: `45s`
+- `maxTransactionQueueTime` — Per-thread write-commit duration threshold for HTTP 503 backpressure, and the limit on how long a write keeps retrying conflicts with other writes (a separate 503, code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT`); see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionqueuetime); _Default_: `45s`
 - `noReadAhead` — Advise OS against read-ahead; _Default_: `false`
 - `prefetchWrites` — Prefetch before write transactions; _Default_: `true`
 - `randomAccessFields` <VersionBadge version="v5.1.0" /> — Encode records as typed random-access structures, so a single field can be read without decoding the whole record. Best for tables whose records have a stable set of fields with stable types; leave off for wide, sparse, or variably typed schemas. Read by each table when its store opens, so a change applies to existing tables on restart, and ignored entirely by tables declaring the [`@table(randomAccessFields:)`](../database/schema.md#randomaccessfields) directive; _Default_: `false`. See [Storage Tuning — Record Encoding](../database/storage-tuning.md#storagerandomaccessfields)
@@ -345,6 +345,8 @@ storage:
 - `rocks.writeBufferManagerSize` — Process-wide cap (bytes) on RocksDB memtable memory across all databases. `0` disables; _Default_: one third of `blockCacheSize` (enabled). See [Storage Tuning — RocksDB Memory](../database/storage-tuning.md#rocksdb-memory) (Added in: v5.1.0)
 - `rocks.writeBufferManagerCostToCache` — When enabled, memtable memory and the block cache share a single, unified memory pool. During heavy write bursts, the block cache dynamically shrinks to give memtables more room. Once those memtables flush to disk, the block cache automatically reclaims that memory space. _Default_: `true`. Has no effect when `writeBufferManagerSize` is `0`. (Added in: v5.1.0)
 - `rocks.writeBufferManagerAllowStall` — Stall writes when memtable memory exceeds `writeBufferManagerSize` (hard cap) instead of allowing brief overshoot with more aggressive flushing (soft cap); _Default_: `true`. (Added in: v5.1.0)
+- `rocks.transactionLogAbortMarkers` <VersionBadge version="v5.4.0" /> — Mark the transaction-log entry of a transaction that fails after logging as aborted, so replication, subscriptions, `read_audit_log` and crash replay skip it, and give up on a write that keeps conflicting instead of retrying it without limit. `false` keeps the 5.3 behavior. Read once per process: a change takes effect after a [`restart`](../operations-api/operations.md#restart), not `restart_service`. Harper 5.3.4 and earlier cannot read the markers. _Default_: `true`. See [Storage Tuning — RocksDB Transaction Log](../database/storage-tuning.md#storagerockstransactionlogabortmarkers)
+- `rocks.transactionLogWriteIntents` <VersionBadge version="v5.4.0" /> — Make a commit that would conflict with an older, still-open transaction wait for that transaction to finish, usually before writing its transaction-log entry. Shortens the longest waits under heavy write contention, at the cost of more retries and lower throughput when transactions are large or contention is low. No effect when `rocks.transactionLogAbortMarkers` is `false`. Read once per process: a change takes effect after a [`restart`](../operations-api/operations.md#restart), not `restart_service`. _Default_: `false`. See [Storage Tuning — RocksDB Transaction Log](../database/storage-tuning.md#storagerockstransactionlogwriteintents)
 
 ---
 

reference/database/storage-tuning.md

@@ -45,6 +45,8 @@ Beginning with v5.2.1, Harper tracks every outstanding commit attempt for this c
 
 The [`transaction-commit-time` metric](../analytics/overview.md#storage-metrics) provides the corresponding commit-latency distribution, but only records each attempt after it settles.
 
+The same duration also limits how long one write keeps retrying conflicts with other writes before it fails with a different HTTP 503, code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT`. See [`storage.rocks.transactionLogAbortMarkers`](#storagerockstransactionlogabortmarkers) for which writes this applies to.
+
 Lower this in latency-sensitive systems where it is better to shed load early than to let request queues grow. Raise it when occasional disk-write bursts are expected and the application can tolerate longer commit latency.
 
 ```yaml
@@ -307,6 +309,50 @@ The default (`true`) strictly bounds total memtable memory, applying write backp
 
 This option is the only `WriteBufferManager` setting that can be changed at runtime — `costToCache` is fixed at first creation.
 
+## RocksDB Transaction Log
+
+### `storage.rocks.transactionLogAbortMarkers`
+
+<VersionBadge version="v5.4.0" /> <EngineBadge engines="RocksDB" />
+
+Type: `boolean`
+
+Default: `true`
+
+Applies only when `storage.engine` is `rocksdb`. Harper writes a transaction's entry to the [transaction log](./transaction.md) before the transaction commits. With this setting on, when a transaction fails after that point, its entry is marked aborted, and every reader of the log skips it: replication, subscriptions, [`read_audit_log`](./transaction.md#read_audit_log), and replay after a crash. A transaction is still replayed on restart if the process crashes before it commits or is marked, including when a disk error delays its marker.
+
+It also bounds a write that keeps conflicting with other writes. The write fails with HTTP `500` after 40 commit conflicts, or with HTTP `503` and code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT` once it has been in conflict for longer than [`storage.maxTransactionQueueTime`](#storagemaxtransactionqueuetime), whichever comes first. A `503` that carries `retryable: false` may have been partly applied, so do not resend that request as is. Writes applied from replication or a caching source, and a transaction part of which has already committed, still retry until they commit.
+
+Set it to `false` to keep the 5.3 behavior, where a write whose log entry was already written retries without limit. Any value other than `false`, `0`, `no` or `off` leaves it on. Harper reads the setting once per process, so a change takes effect after a [`restart`](../operations-api/operations.md#restart) of each node, or a process or container restart. [`restart_service`](../operations-api/operations.md#restart_service) restarts only worker threads and does not apply it.
+
+**Downgrades:** Harper 5.3.4 and earlier cannot read abort markers and would treat a marked transaction as committed. Do not run them on a node's database files, or a backup of them, after the node has run 5.4 with this setting on. Harper 5.3.5 and later read the markers. Setting it to `false` later stops new markers but does not remove ones already written.
+
+```yaml
+storage:
+  rocks:
+    transactionLogAbortMarkers: false
+```
+
+### `storage.rocks.transactionLogWriteIntents`
+
+<VersionBadge version="v5.4.0" /> <EngineBadge engines="RocksDB" />
+
+Type: `boolean`
+
+Default: `false`
+
+When `true`, a commit that would conflict with a write from an older transaction that is still open is refused before it writes its transaction-log entry, or before it commits if an earlier attempt already wrote that entry. It then waits for that transaction to commit or abort, and retries. A refused attempt does not count toward the 40-conflict limit of [`transactionLogAbortMarkers`](#storagerockstransactionlogabortmarkers).
+
+This shortens the longest waits when many transactions write the same records at once. It adds retries and lowers throughput when transactions are large or contention is low, so leave it off unless overlapping writes to the same records see long commit delays.
+
+It has no effect when `transactionLogAbortMarkers` is `false`. Like that setting, it is read once per process: a change takes effect after a [`restart`](../operations-api/operations.md#restart) of each node, not [`restart_service`](../operations-api/operations.md#restart_service).
+
+```yaml
+storage:
+  rocks:
+    transactionLogWriteIntents: true
+```
+
 ## Storage Reclamation
 
 `storage.reclamation` controls how Harper evicts data from caching tables (tables with [`sourcedFrom`](../resources/resource-api.md#sourcedfromresource-options)) when disk usage runs high. Reclamation does **not** affect non-caching tables — those rely on explicit deletion, TTL expiration, or [compaction](./compaction.md).

reference/database/transaction.md

@@ -21,6 +21,8 @@ Available since: v4.1.0
 
 On RocksDB (the default storage engine) the transaction log is physically a single database-wide log shared by all tables: history is read per table, while deletion operates on the whole database's log.
 
+<VersionBadge type="changed" version="v5.4.0" /> By default on RocksDB, a transaction that fails after its entry was written to the log is marked aborted and left out of the log's history, including `read_audit_log` results. The [`storage.rocks.transactionLogAbortMarkers`](./storage-tuning.md#storagerockstransactionlogabortmarkers) setting controls this.
+
 The transaction log is **enabled by default**. To disable it, set [`logging.auditLog`](../logging/configuration.md) to `false` in `harper-config.yaml` and restart Harper.
 
 > The transaction log is required for real-time messaging (WebSocket and MQTT subscriptions), replication and full-text indexing. Do not disable it while any of these features are in use.

reference/operations-api/operations.md

@@ -1845,7 +1845,7 @@ Detailed documentation: [Configuration Overview](../configuration/overview.md)
 
 Updates configuration parameters in `harper-config.yaml`. Most changes require a restart (`restart` or `restart_service`) to take effect. See [Set Configuration](../configuration/operations.md#set-configuration) for full-text restart requirements.
 
-Supports `"replicated": true` <VersionBadge version="v5.2.0" /> to apply the same change to all cluster nodes in one call; per-node outcomes are returned in the response's `replicated` array. Only send cluster-appropriate parameters when replicating — node-local parameters (ports, `node.hostname`, file paths, TLS material, `replication.hostname`/`url`/`routes`) would overwrite every peer's local values. To apply the change cluster-wide, follow with `restart_service` using `"replicated": true` (which restarts nodes one at a time); `restart_service` restarts only worker threads, though, so changes to [`threads.v8Flags`](../configuration/options.md#applying-v8-flags) or `storage.fullText.runtime.*` take effect only after a `restart` sent to each node in turn (`restart` does not replicate). See [Configuration Operations](../configuration/operations.md#set-configuration) for details.
+Supports `"replicated": true` <VersionBadge version="v5.2.0" /> to apply the same change to all cluster nodes in one call; per-node outcomes are returned in the response's `replicated` array. Only send cluster-appropriate parameters when replicating — node-local parameters (ports, `node.hostname`, file paths, TLS material, `replication.hostname`/`url`/`routes`) would overwrite every peer's local values. To apply the change cluster-wide, follow with `restart_service` using `"replicated": true` (which restarts nodes one at a time); `restart_service` restarts only worker threads, though, so changes to [`threads.v8Flags`](../configuration/options.md#applying-v8-flags), `storage.fullText.runtime.*`, [`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) or [`storage.rocks.transactionLogWriteIntents`](../database/storage-tuning.md#storagerockstransactionlogwriteintents) take effect only after a `restart` sent to each node in turn (`restart` does not replicate). See [Configuration Operations](../configuration/operations.md#set-configuration) for details.
 
 ```json
 {