From d98a6191c220129c1a69449c757f173953997d83 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 2 Nov 2026 13:55:02 -0700 Subject: [PATCH 1/6] Document storage.rocks.transactionLogAbortMarkers and its 5.4 default Harper 5.4 (harper#3928) turns transaction-log abort markers on by default: a transaction that fails after logging is marked aborted and skipped by every log reader, and a conflicting logged write is bounded by the 500/503 limits. Adds the configuration reference and storage-tuning entry, the process-restart rule in the set_configuration procedures, a transaction-log cross-link, and the 5.4 release and downgrade notes. Co-Authored-By: Claude Opus 5.5 Dispatch-Task: docs-harper-3928 --- reference/configuration/operations.md | 4 ++-- reference/configuration/options.md | 1 + reference/database/storage-tuning.md | 26 ++++++++++++++++++++++++++ reference/database/transaction.md | 2 ++ reference/operations-api/operations.md | 2 +- release-notes/v5-lincoln/5.4.md | 8 ++++++++ 6 files changed, 40 insertions(+), 3 deletions(-) diff --git a/reference/configuration/operations.md b/reference/configuration/operations.md index faea62523..bc533bae1 100644 --- a/reference/configuration/operations.md +++ b/reference/configuration/operations.md @@ -16,7 +16,7 @@ For the full list of configurable options, see [Configuration Options](./options ## Set Configuration -Modifies one or more Harper configuration parameters. **Most changes require a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`). A change to `replication.threads` takes effect only on a full `restart`. +Modifies one or more Harper configuration parameters. **Most changes require a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags` and [`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers)). A change to `replication.threads` takes effect only on a full `restart`. Full-text request work limits (`storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds`) take effect in HTTP workers after `restart_service` or a full restart. Changes to `storage.fullText.runtime` require a full Harper process restart; `restart_service` does not replace the native process budget. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). @@ -86,7 +86,7 @@ To restart the whole cluster afterward, follow with [restart_service](../operati } ``` -`restart_service` restarts only worker threads, which keep the process's V8 flags. A change to [`threads.v8Flags`](./options.md#applying-v8-flags) takes effect only after a [restart](../operations-api/operations.md#restart), sent to each node in turn. +`restart_service` restarts only worker threads, which keep the process's V8 flags and its `storage.rocks.transactionLogAbortMarkers` value. A change to [`threads.v8Flags`](./options.md#applying-v8-flags) or [`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) takes effect only after a [restart](../operations-api/operations.md#restart), sent to each node in turn. --- diff --git a/reference/configuration/options.md b/reference/configuration/options.md index ee62df608..cb54ad5a7 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -345,6 +345,7 @@ storage: - `rocks.writeBufferManagerSize` — Process-wide cap (bytes) on RocksDB memtable memory across all databases. `0` disables; _Default_: one third of `blockCacheSize` (enabled). See [Storage Tuning — RocksDB Memory](../database/storage-tuning.md#rocksdb-memory) (Added in: v5.1.0) - `rocks.writeBufferManagerCostToCache` — When enabled, memtable memory and the block cache share a single, unified memory pool. During heavy write bursts, the block cache dynamically shrinks to give memtables more room. Once those memtables flush to disk, the block cache automatically reclaims that memory space. _Default_: `true`. Has no effect when `writeBufferManagerSize` is `0`. (Added in: v5.1.0) - `rocks.writeBufferManagerAllowStall` — Stall writes when memtable memory exceeds `writeBufferManagerSize` (hard cap) instead of allowing brief overshoot with more aggressive flushing (soft cap); _Default_: `true`. (Added in: v5.1.0) +- `rocks.transactionLogAbortMarkers` — Mark the transaction-log entry of a transaction that fails after logging as aborted, so replication, subscriptions, `read_audit_log` and crash replay skip it, and give up on a write that keeps conflicting instead of retrying it without limit. `false` keeps the 5.3 behavior. Read once per process: a change takes effect after a [`restart`](../operations-api/operations.md#restart), not `restart_service`. Harper 5.3.4 and earlier cannot read the markers; _Default_: `true`. See [Storage Tuning — RocksDB Transaction Log](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) --- diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index bf2a99806..618657232 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -45,6 +45,8 @@ Beginning with v5.2.1, Harper tracks every outstanding commit attempt for this c The [`transaction-commit-time` metric](../analytics/overview.md#storage-metrics) provides the corresponding commit-latency distribution, but only records each attempt after it settles. +The same duration also limits how long one write keeps retrying conflicts with other writes before it fails with a different HTTP 503, code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT`. See [`storage.rocks.transactionLogAbortMarkers`](#storagerockstransactionlogabortmarkers) for which writes this applies to. + Lower this in latency-sensitive systems where it is better to shed load early than to let request queues grow. Raise it when occasional disk-write bursts are expected and the application can tolerate longer commit latency. ```yaml @@ -307,6 +309,30 @@ The default (`true`) strictly bounds total memtable memory, applying write backp This option is the only `WriteBufferManager` setting that can be changed at runtime — `costToCache` is fixed at first creation. +## RocksDB Transaction Log + +### `storage.rocks.transactionLogAbortMarkers` + + + +Type: `boolean` + +Default: `true` + +Applies only when `storage.engine` is `rocksdb`. Harper writes a transaction's entry to the [transaction log](./transaction.md) before the transaction commits. With this setting on, when a transaction fails after that point, its entry is marked aborted, and every reader of the log skips it: replication, subscriptions, [`read_audit_log`](./transaction.md#read_audit_log), and replay after a crash. A transaction interrupted by a crash before it commits or is marked is still replayed on restart. + +It also bounds a write that keeps conflicting with other writes. The write fails with HTTP `500` after 40 retries, or with HTTP `503` and code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT` once it has been in conflict for longer than [`storage.maxTransactionQueueTime`](#storagemaxtransactionqueuetime). A `503` that carries `retryable: false` may have been partly applied, so do not resend that request as is. Writes applied from replication or a caching source, and a transaction part of which has already committed, still retry until they commit. With the setting off, a write whose log entry was already written retries without limit. + +Set it to `false` to keep the 5.3 behavior. Any value other than `false`, `0`, `no` or `off` leaves it on. Harper reads the setting once per process, so a change takes effect after a [`restart`](../operations-api/operations.md#restart) of each node, or a process or container restart. [`restart_service`](../operations-api/operations.md#restart_service) restarts only worker threads and does not apply it. + +**Downgrades:** Harper 5.3.4 and earlier cannot read abort markers and would treat a marked transaction as committed. Do not run them on a node's database files, or a backup of them, after the node has run 5.4 with this setting on. Harper 5.3.5 and later read the markers. Setting it to `false` later stops new markers but does not remove ones already written. + +```yaml +storage: + rocks: + transactionLogAbortMarkers: false +``` + ## Storage Reclamation `storage.reclamation` controls how Harper evicts data from caching tables (tables with [`sourcedFrom`](../resources/resource-api.md#sourcedfromresource-options)) when disk usage runs high. Reclamation does **not** affect non-caching tables — those rely on explicit deletion, TTL expiration, or [compaction](./compaction.md). diff --git a/reference/database/transaction.md b/reference/database/transaction.md index da616e19d..9ffb1a58b 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -21,6 +21,8 @@ Available since: v4.1.0 On RocksDB (the default storage engine) the transaction log is physically a single database-wide log shared by all tables: history is read per table, while deletion operates on the whole database's log. + On RocksDB, a transaction that fails after its entry was written to the log is marked aborted and left out of the log's history; see [`storage.rocks.transactionLogAbortMarkers`](./storage-tuning.md#storagerockstransactionlogabortmarkers). + The transaction log is **enabled by default**. To disable it, set [`logging.auditLog`](../logging/configuration.md) to `false` in `harper-config.yaml` and restart Harper. > The transaction log is required for real-time messaging (WebSocket and MQTT subscriptions), replication and full-text indexing. Do not disable it while any of these features are in use. diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 981103058..6c12c8004 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -1845,7 +1845,7 @@ Detailed documentation: [Configuration Overview](../configuration/overview.md) Updates configuration parameters in `harper-config.yaml`. Most changes require a restart (`restart` or `restart_service`) to take effect. See [Set Configuration](../configuration/operations.md#set-configuration) for full-text restart requirements. -Supports `"replicated": true` to apply the same change to all cluster nodes in one call; per-node outcomes are returned in the response's `replicated` array. Only send cluster-appropriate parameters when replicating — node-local parameters (ports, `node.hostname`, file paths, TLS material, `replication.hostname`/`url`/`routes`) would overwrite every peer's local values. To apply the change cluster-wide, follow with `restart_service` using `"replicated": true` (which restarts nodes one at a time); `restart_service` restarts only worker threads, though, so changes to [`threads.v8Flags`](../configuration/options.md#applying-v8-flags) or `storage.fullText.runtime.*` take effect only after a `restart` sent to each node in turn (`restart` does not replicate). See [Configuration Operations](../configuration/operations.md#set-configuration) for details. +Supports `"replicated": true` to apply the same change to all cluster nodes in one call; per-node outcomes are returned in the response's `replicated` array. Only send cluster-appropriate parameters when replicating — node-local parameters (ports, `node.hostname`, file paths, TLS material, `replication.hostname`/`url`/`routes`) would overwrite every peer's local values. To apply the change cluster-wide, follow with `restart_service` using `"replicated": true` (which restarts nodes one at a time); `restart_service` restarts only worker threads, though, so changes to [`threads.v8Flags`](../configuration/options.md#applying-v8-flags), `storage.fullText.runtime.*` or [`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) take effect only after a `restart` sent to each node in turn (`restart` does not replicate). See [Configuration Operations](../configuration/operations.md#set-configuration) for details. ```json { diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 916e1b5af..5334cd4c0 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -12,6 +12,8 @@ All patch release notes for 5.4.x are available on the [releases page](https://g Full-text indexes using fulltext 0.5.0 with current field-based declarations, or prerelease 0.6, rebuild locally after this upgrade. Plan for search unavailability and review the [Tantivy upgrade requirements](#tantivy-027-and-existing-index-rebuilds) before deploying; direct downgrade to fulltext 0.5.x is unsupported. +Harper 5.4 marks failed transactions as aborted in the RocksDB transaction log by default ([`storage.rocks.transactionLogAbortMarkers`](/reference/v5/database/storage-tuning#storagerockstransactionlogabortmarkers); see [Failed Transactions Are Marked Aborted](#failed-transactions-are-marked-aborted-in-the-transaction-log)). Harper 5.3.4 and earlier cannot read those markers and would treat the failed transactions as committed. Do not downgrade a node to them, or open its database files or backups with them, after it has run 5.4 with the setting on; Harper 5.3.5 and later read the markers. Setting it to `false` later stops new markers but does not remove ones already written. + ## Nested Relationship Constraints [Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279) and [Fix wrong rows, unfiltered selects and ignored sorts in relationship queries](https://github.com/HarperFast/harper/pull/3775) correct REST and programmatic relationship queries that accepted nested constraints, non-leading relationship conditions, or top-level relationship sorts without applying them completely. Filters, ordering, offset, and limit inside `select` apply per parent in collection and single-record reads, and parents remain present when the nested selection has no matches. Outer relationship filters — whether they lead the query or follow another condition — constrain the selected children as well as the parents, so an identical filter does not need to be repeated inside the nested selection. When a top-level sort across a to-many relationship (such as `sort(products.name)`) runs in memory — when the target attribute is not `@indexed`, a more selective condition on another attribute leads the query, or `enforceExecutionOrder: true` is set — Harper now orders matched parent records by the list of each parent's related values in relationship order, compared element by element, instead of ignoring the sort; when an `@indexed` target attribute leads the query, Harper continues to order via the related table's index join. @@ -134,6 +136,12 @@ MCP `initialize` is now limited per client identity, on both profiles, by the ne The new `threads.v8Flags` setting takes a V8 flag, or a list of them, and applies them to the Harper process before its first worker thread starts, so every worker thread runs with them. It is for flags such as `--optimize-for-size`, which Node refuses in `NODE_OPTIONS` and which cannot be passed to a worker thread. It can also be set through `HARPER_SET_CONFIG` or `HARPER_CONFIG`. An entry must start with `--`; `set_configuration` refuses one that does not, and Harper stops at startup with an error naming the setting. Unlike the command line, this does not apply a flag's implied flags, so list them yourself (for example `--max-semi-space-size=1` alongside `--optimize-for-size`). A changed value takes effect when the process restarts (the `restart` operation, on each node); `restart_service` restarts only worker threads and does not apply it. Not applied under Bun. See [Applying V8 flags](/reference/v5/configuration/options#applying-v8-flags). +## Transactions + +### Failed Transactions Are Marked Aborted in the Transaction Log + +The new `storage.rocks.transactionLogAbortMarkers` setting is on by default (RocksDB only). When a transaction fails after its entry was written to the transaction log, the entry is now marked aborted, and replication, subscriptions, `read_audit_log` and replay after a crash all skip it. A write that keeps conflicting with other writes after its log entry was written no longer retries without limit: like other conflicting writes, it fails with HTTP `500` after 40 retries, or with HTTP `503` and code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT` once it has been in conflict for longer than `storage.maxTransactionQueueTime`. Writes applied from replication or a caching source, and a transaction part of which has already committed, still retry until they commit. Set the setting to `false` to keep the 5.3 behavior. Harper reads it once per process, so a change needs a `restart` of each node; `restart_service` does not apply it. Harper 5.3.4 and earlier cannot read the markers; see the [Upgrade Notes](#upgrade-notes). See [`storage.rocks.transactionLogAbortMarkers`](/reference/v5/database/storage-tuning#storagerockstransactionlogabortmarkers) ([harper#3928](https://github.com/HarperFast/harper/pull/3928)). + ## Backups ### Restore Checks a Backup Before Deleting the Database From 091373e4bdd244b45c9aede229409106e9a5fd14 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 2 Nov 2026 14:03:38 -0700 Subject: [PATCH 2/6] Qualify abort-marker claims and cross-link the conflict-timeout 503 Note that an entry whose marker could not be written is still replayed, condition the transaction-log sentence on the default, and name the TRANSACTION_COMMIT_CONFLICT_TIMEOUT 503 in the maxTransactionQueueTime summaries in options.md and the analytics overview. Co-Authored-By: Claude Opus 5.5 Dispatch-Task: docs-harper-3928 --- reference/analytics/overview.md | 4 +++- reference/configuration/options.md | 2 +- reference/database/storage-tuning.md | 2 +- reference/database/transaction.md | 2 +- 4 files changed, 6 insertions(+), 4 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index c18f491da..f720368d4 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -208,7 +208,9 @@ outstanding commit on a thread exceeds (default 45s), Harper rejects new record updates and publishes from new application requests on that thread with `Outstanding write transactions have too long of queue, please try again later` (HTTP 503). Deletes and writes applied from a canonical source (e.g. replication or a caching source) -bypass this check. +bypass this check. The same duration also limits how long a write keeps retrying conflicts with other +writes, which fails with a separate 503, code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT`; see +[`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers). diff --git a/reference/configuration/options.md b/reference/configuration/options.md index cb54ad5a7..5765fc1a8 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -330,7 +330,7 @@ storage: - `compression` — LZ4 record compression; _Default_: `true` (enabled by default since v4.3.0). Sub-options: `dictionary`, `threshold` - `compactOnStart` — Compact all non-system databases on startup; _Default_: `false` (Added in: v4.3.0) - `compactOnStartKeepBackup` — Retain compaction backups; _Default_: `false` -- `maxTransactionQueueTime` — Per-thread write-commit duration threshold for HTTP 503 backpressure; see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionqueuetime); _Default_: `45s` +- `maxTransactionQueueTime` — Per-thread write-commit duration threshold for HTTP 503 backpressure, and the limit on how long a write keeps retrying conflicts with other writes (a separate 503, code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT`); see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionqueuetime); _Default_: `45s` - `noReadAhead` — Advise OS against read-ahead; _Default_: `false` - `prefetchWrites` — Prefetch before write transactions; _Default_: `true` - `randomAccessFields` — Encode records as typed random-access structures, so a single field can be read without decoding the whole record. Best for tables whose records have a stable set of fields with stable types; leave off for wide, sparse, or variably typed schemas. Read by each table when its store opens, so a change applies to existing tables on restart, and ignored entirely by tables declaring the [`@table(randomAccessFields:)`](../database/schema.md#randomaccessfields) directive; _Default_: `false`. See [Storage Tuning — Record Encoding](../database/storage-tuning.md#storagerandomaccessfields) diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index 618657232..887983ded 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -319,7 +319,7 @@ Type: `boolean` Default: `true` -Applies only when `storage.engine` is `rocksdb`. Harper writes a transaction's entry to the [transaction log](./transaction.md) before the transaction commits. With this setting on, when a transaction fails after that point, its entry is marked aborted, and every reader of the log skips it: replication, subscriptions, [`read_audit_log`](./transaction.md#read_audit_log), and replay after a crash. A transaction interrupted by a crash before it commits or is marked is still replayed on restart. +Applies only when `storage.engine` is `rocksdb`. Harper writes a transaction's entry to the [transaction log](./transaction.md) before the transaction commits. With this setting on, when a transaction fails after that point, its entry is marked aborted, and every reader of the log skips it: replication, subscriptions, [`read_audit_log`](./transaction.md#read_audit_log), and replay after a crash. A transaction interrupted by a crash before it commits or is marked, or one whose marker could not be written because of a disk error, is still replayed on restart. It also bounds a write that keeps conflicting with other writes. The write fails with HTTP `500` after 40 retries, or with HTTP `503` and code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT` once it has been in conflict for longer than [`storage.maxTransactionQueueTime`](#storagemaxtransactionqueuetime). A `503` that carries `retryable: false` may have been partly applied, so do not resend that request as is. Writes applied from replication or a caching source, and a transaction part of which has already committed, still retry until they commit. With the setting off, a write whose log entry was already written retries without limit. diff --git a/reference/database/transaction.md b/reference/database/transaction.md index 9ffb1a58b..2398e27ac 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -21,7 +21,7 @@ Available since: v4.1.0 On RocksDB (the default storage engine) the transaction log is physically a single database-wide log shared by all tables: history is read per table, while deletion operates on the whole database's log. - On RocksDB, a transaction that fails after its entry was written to the log is marked aborted and left out of the log's history; see [`storage.rocks.transactionLogAbortMarkers`](./storage-tuning.md#storagerockstransactionlogabortmarkers). + By default on RocksDB, a transaction that fails after its entry was written to the log is marked aborted and left out of the log's history, including `read_audit_log` results. The [`storage.rocks.transactionLogAbortMarkers`](./storage-tuning.md#storagerockstransactionlogabortmarkers) setting controls this. The transaction log is **enabled by default**. To disable it, set [`logging.auditLog`](../logging/configuration.md) to `false` in `harper-config.yaml` and restart Harper. From c57bae62914c31c5f2b87753102025d240bf5677 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 2 Nov 2026 14:09:24 -0700 Subject: [PATCH 3/6] Tighten the marker-failure replay caveat and the 40-conflict wording Co-Authored-By: Claude Opus 5.5 Dispatch-Task: docs-harper-3928 --- reference/database/storage-tuning.md | 4 ++-- release-notes/v5-lincoln/5.4.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index 887983ded..6bf72d98a 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -319,9 +319,9 @@ Type: `boolean` Default: `true` -Applies only when `storage.engine` is `rocksdb`. Harper writes a transaction's entry to the [transaction log](./transaction.md) before the transaction commits. With this setting on, when a transaction fails after that point, its entry is marked aborted, and every reader of the log skips it: replication, subscriptions, [`read_audit_log`](./transaction.md#read_audit_log), and replay after a crash. A transaction interrupted by a crash before it commits or is marked, or one whose marker could not be written because of a disk error, is still replayed on restart. +Applies only when `storage.engine` is `rocksdb`. Harper writes a transaction's entry to the [transaction log](./transaction.md) before the transaction commits. With this setting on, when a transaction fails after that point, its entry is marked aborted, and every reader of the log skips it: replication, subscriptions, [`read_audit_log`](./transaction.md#read_audit_log), and replay after a crash. A transaction is still replayed on restart if the process crashes before it commits or is marked, including when a disk error delays its marker. -It also bounds a write that keeps conflicting with other writes. The write fails with HTTP `500` after 40 retries, or with HTTP `503` and code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT` once it has been in conflict for longer than [`storage.maxTransactionQueueTime`](#storagemaxtransactionqueuetime). A `503` that carries `retryable: false` may have been partly applied, so do not resend that request as is. Writes applied from replication or a caching source, and a transaction part of which has already committed, still retry until they commit. With the setting off, a write whose log entry was already written retries without limit. +It also bounds a write that keeps conflicting with other writes. The write fails with HTTP `500` after 40 commit conflicts, or with HTTP `503` and code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT` once it has been in conflict for longer than [`storage.maxTransactionQueueTime`](#storagemaxtransactionqueuetime), whichever comes first. A `503` that carries `retryable: false` may have been partly applied, so do not resend that request as is. Writes applied from replication or a caching source, and a transaction part of which has already committed, still retry until they commit. With the setting off, a write whose log entry was already written retries without limit. Set it to `false` to keep the 5.3 behavior. Any value other than `false`, `0`, `no` or `off` leaves it on. Harper reads the setting once per process, so a change takes effect after a [`restart`](../operations-api/operations.md#restart) of each node, or a process or container restart. [`restart_service`](../operations-api/operations.md#restart_service) restarts only worker threads and does not apply it. diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 5334cd4c0..9111123db 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -140,7 +140,7 @@ The new `threads.v8Flags` setting takes a V8 flag, or a list of them, and applie ### Failed Transactions Are Marked Aborted in the Transaction Log -The new `storage.rocks.transactionLogAbortMarkers` setting is on by default (RocksDB only). When a transaction fails after its entry was written to the transaction log, the entry is now marked aborted, and replication, subscriptions, `read_audit_log` and replay after a crash all skip it. A write that keeps conflicting with other writes after its log entry was written no longer retries without limit: like other conflicting writes, it fails with HTTP `500` after 40 retries, or with HTTP `503` and code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT` once it has been in conflict for longer than `storage.maxTransactionQueueTime`. Writes applied from replication or a caching source, and a transaction part of which has already committed, still retry until they commit. Set the setting to `false` to keep the 5.3 behavior. Harper reads it once per process, so a change needs a `restart` of each node; `restart_service` does not apply it. Harper 5.3.4 and earlier cannot read the markers; see the [Upgrade Notes](#upgrade-notes). See [`storage.rocks.transactionLogAbortMarkers`](/reference/v5/database/storage-tuning#storagerockstransactionlogabortmarkers) ([harper#3928](https://github.com/HarperFast/harper/pull/3928)). +The new `storage.rocks.transactionLogAbortMarkers` setting is on by default (RocksDB only). When a transaction fails after its entry was written to the transaction log, the entry is now marked aborted, and replication, subscriptions, `read_audit_log` and replay after a crash all skip it. A write that keeps conflicting with other writes after its log entry was written no longer retries without limit: like other conflicting writes, it fails with HTTP `500` after 40 commit conflicts, or with HTTP `503` and code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT` once it has been in conflict for longer than `storage.maxTransactionQueueTime`, whichever comes first. Writes applied from replication or a caching source, and a transaction part of which has already committed, still retry until they commit. Set the setting to `false` to keep the 5.3 behavior. Harper reads it once per process, so a change needs a `restart` of each node; `restart_service` does not apply it. Harper 5.3.4 and earlier cannot read the markers; see the [Upgrade Notes](#upgrade-notes). See [`storage.rocks.transactionLogAbortMarkers`](/reference/v5/database/storage-tuning#storagerockstransactionlogabortmarkers) ([harper#3928](https://github.com/HarperFast/harper/pull/3928)). ## Backups From c60518ac6bfc453b1a9609135bc9da31728f1df9 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 2 Nov 2026 14:21:16 -0700 Subject: [PATCH 4/6] Split combined warning sentences and badge the RocksDB-only entry Co-Authored-By: Claude Opus 5.5 Dispatch-Task: docs-harper-3928 --- reference/configuration/options.md | 2 +- reference/database/storage-tuning.md | 6 +++--- reference/database/transaction.md | 2 +- release-notes/v5-lincoln/5.4.md | 4 ++-- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 5765fc1a8..e4763399d 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -345,7 +345,7 @@ storage: - `rocks.writeBufferManagerSize` — Process-wide cap (bytes) on RocksDB memtable memory across all databases. `0` disables; _Default_: one third of `blockCacheSize` (enabled). See [Storage Tuning — RocksDB Memory](../database/storage-tuning.md#rocksdb-memory) (Added in: v5.1.0) - `rocks.writeBufferManagerCostToCache` — When enabled, memtable memory and the block cache share a single, unified memory pool. During heavy write bursts, the block cache dynamically shrinks to give memtables more room. Once those memtables flush to disk, the block cache automatically reclaims that memory space. _Default_: `true`. Has no effect when `writeBufferManagerSize` is `0`. (Added in: v5.1.0) - `rocks.writeBufferManagerAllowStall` — Stall writes when memtable memory exceeds `writeBufferManagerSize` (hard cap) instead of allowing brief overshoot with more aggressive flushing (soft cap); _Default_: `true`. (Added in: v5.1.0) -- `rocks.transactionLogAbortMarkers` — Mark the transaction-log entry of a transaction that fails after logging as aborted, so replication, subscriptions, `read_audit_log` and crash replay skip it, and give up on a write that keeps conflicting instead of retrying it without limit. `false` keeps the 5.3 behavior. Read once per process: a change takes effect after a [`restart`](../operations-api/operations.md#restart), not `restart_service`. Harper 5.3.4 and earlier cannot read the markers; _Default_: `true`. See [Storage Tuning — RocksDB Transaction Log](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) +- `rocks.transactionLogAbortMarkers` — Mark the transaction-log entry of a transaction that fails after logging as aborted, so replication, subscriptions, `read_audit_log` and crash replay skip it, and give up on a write that keeps conflicting instead of retrying it without limit. `false` keeps the 5.3 behavior. Read once per process: a change takes effect after a [`restart`](../operations-api/operations.md#restart), not `restart_service`. Harper 5.3.4 and earlier cannot read the markers. _Default_: `true`. See [Storage Tuning — RocksDB Transaction Log](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) --- diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index 6bf72d98a..d9a516b4a 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -313,7 +313,7 @@ This option is the only `WriteBufferManager` setting that can be changed at runt ### `storage.rocks.transactionLogAbortMarkers` - + Type: `boolean` @@ -321,9 +321,9 @@ Default: `true` Applies only when `storage.engine` is `rocksdb`. Harper writes a transaction's entry to the [transaction log](./transaction.md) before the transaction commits. With this setting on, when a transaction fails after that point, its entry is marked aborted, and every reader of the log skips it: replication, subscriptions, [`read_audit_log`](./transaction.md#read_audit_log), and replay after a crash. A transaction is still replayed on restart if the process crashes before it commits or is marked, including when a disk error delays its marker. -It also bounds a write that keeps conflicting with other writes. The write fails with HTTP `500` after 40 commit conflicts, or with HTTP `503` and code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT` once it has been in conflict for longer than [`storage.maxTransactionQueueTime`](#storagemaxtransactionqueuetime), whichever comes first. A `503` that carries `retryable: false` may have been partly applied, so do not resend that request as is. Writes applied from replication or a caching source, and a transaction part of which has already committed, still retry until they commit. With the setting off, a write whose log entry was already written retries without limit. +It also bounds a write that keeps conflicting with other writes. The write fails with HTTP `500` after 40 commit conflicts, or with HTTP `503` and code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT` once it has been in conflict for longer than [`storage.maxTransactionQueueTime`](#storagemaxtransactionqueuetime), whichever comes first. A `503` that carries `retryable: false` may have been partly applied, so do not resend that request as is. Writes applied from replication or a caching source, and a transaction part of which has already committed, still retry until they commit. -Set it to `false` to keep the 5.3 behavior. Any value other than `false`, `0`, `no` or `off` leaves it on. Harper reads the setting once per process, so a change takes effect after a [`restart`](../operations-api/operations.md#restart) of each node, or a process or container restart. [`restart_service`](../operations-api/operations.md#restart_service) restarts only worker threads and does not apply it. +Set it to `false` to keep the 5.3 behavior, where a write whose log entry was already written retries without limit. Any value other than `false`, `0`, `no` or `off` leaves it on. Harper reads the setting once per process, so a change takes effect after a [`restart`](../operations-api/operations.md#restart) of each node, or a process or container restart. [`restart_service`](../operations-api/operations.md#restart_service) restarts only worker threads and does not apply it. **Downgrades:** Harper 5.3.4 and earlier cannot read abort markers and would treat a marked transaction as committed. Do not run them on a node's database files, or a backup of them, after the node has run 5.4 with this setting on. Harper 5.3.5 and later read the markers. Setting it to `false` later stops new markers but does not remove ones already written. diff --git a/reference/database/transaction.md b/reference/database/transaction.md index 2398e27ac..8d934d229 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -21,7 +21,7 @@ Available since: v4.1.0 On RocksDB (the default storage engine) the transaction log is physically a single database-wide log shared by all tables: history is read per table, while deletion operates on the whole database's log. - By default on RocksDB, a transaction that fails after its entry was written to the log is marked aborted and left out of the log's history, including `read_audit_log` results. The [`storage.rocks.transactionLogAbortMarkers`](./storage-tuning.md#storagerockstransactionlogabortmarkers) setting controls this. + By default on RocksDB, a transaction that fails after its entry was written to the log is marked aborted and left out of the log's history, including `read_audit_log` results. The [`storage.rocks.transactionLogAbortMarkers`](./storage-tuning.md#storagerockstransactionlogabortmarkers) setting controls this. The transaction log is **enabled by default**. To disable it, set [`logging.auditLog`](../logging/configuration.md) to `false` in `harper-config.yaml` and restart Harper. diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 9111123db..24539369b 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -12,7 +12,7 @@ All patch release notes for 5.4.x are available on the [releases page](https://g Full-text indexes using fulltext 0.5.0 with current field-based declarations, or prerelease 0.6, rebuild locally after this upgrade. Plan for search unavailability and review the [Tantivy upgrade requirements](#tantivy-027-and-existing-index-rebuilds) before deploying; direct downgrade to fulltext 0.5.x is unsupported. -Harper 5.4 marks failed transactions as aborted in the RocksDB transaction log by default ([`storage.rocks.transactionLogAbortMarkers`](/reference/v5/database/storage-tuning#storagerockstransactionlogabortmarkers); see [Failed Transactions Are Marked Aborted](#failed-transactions-are-marked-aborted-in-the-transaction-log)). Harper 5.3.4 and earlier cannot read those markers and would treat the failed transactions as committed. Do not downgrade a node to them, or open its database files or backups with them, after it has run 5.4 with the setting on; Harper 5.3.5 and later read the markers. Setting it to `false` later stops new markers but does not remove ones already written. +Harper 5.4 marks failed transactions as aborted in the RocksDB transaction log by default ([`storage.rocks.transactionLogAbortMarkers`](/reference/v5/database/storage-tuning#storagerockstransactionlogabortmarkers); see [Failed Transactions Are Marked Aborted](#failed-transactions-are-marked-aborted-in-the-transaction-log)). Harper 5.3.4 and earlier cannot read those markers and would treat the failed transactions as committed. Do not downgrade a node to them, or open its database files or backups with them, after it has run 5.4 with the setting on. Harper 5.3.5 and later read the markers. Setting it to `false` later stops new markers but does not remove ones already written. ## Nested Relationship Constraints @@ -140,7 +140,7 @@ The new `threads.v8Flags` setting takes a V8 flag, or a list of them, and applie ### Failed Transactions Are Marked Aborted in the Transaction Log -The new `storage.rocks.transactionLogAbortMarkers` setting is on by default (RocksDB only). When a transaction fails after its entry was written to the transaction log, the entry is now marked aborted, and replication, subscriptions, `read_audit_log` and replay after a crash all skip it. A write that keeps conflicting with other writes after its log entry was written no longer retries without limit: like other conflicting writes, it fails with HTTP `500` after 40 commit conflicts, or with HTTP `503` and code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT` once it has been in conflict for longer than `storage.maxTransactionQueueTime`, whichever comes first. Writes applied from replication or a caching source, and a transaction part of which has already committed, still retry until they commit. Set the setting to `false` to keep the 5.3 behavior. Harper reads it once per process, so a change needs a `restart` of each node; `restart_service` does not apply it. Harper 5.3.4 and earlier cannot read the markers; see the [Upgrade Notes](#upgrade-notes). See [`storage.rocks.transactionLogAbortMarkers`](/reference/v5/database/storage-tuning#storagerockstransactionlogabortmarkers) ([harper#3928](https://github.com/HarperFast/harper/pull/3928)). +The new `storage.rocks.transactionLogAbortMarkers` setting is on by default (RocksDB only). When a transaction fails after its entry was written to the transaction log, the entry is now marked aborted, and replication, subscriptions, `read_audit_log` and replay after a crash all skip it. A write that keeps conflicting with other writes after its log entry was written no longer retries without limit: like other conflicting writes, it fails with HTTP `500` after 40 commit conflicts, or with HTTP `503` and code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT` once it has been in conflict for longer than `storage.maxTransactionQueueTime`, whichever comes first. Writes applied from replication or a caching source, and a transaction part of which has already committed, still retry until they commit. Set the setting to `false` to keep the 5.3 behavior. Harper reads it once per process, so a change needs a `restart` of each node; `restart_service` does not apply it. Harper 5.3.4 and earlier cannot read the markers. See the [Upgrade Notes](#upgrade-notes) and [`storage.rocks.transactionLogAbortMarkers`](/reference/v5/database/storage-tuning#storagerockstransactionlogabortmarkers) ([harper#3928](https://github.com/HarperFast/harper/pull/3928)). ## Backups From 04f2a4921bb6a0d050c9cd770a00d18cea887b7d Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 2 Nov 2026 15:29:03 -0700 Subject: [PATCH 5/6] Document storage.rocks.transactionLogWriteIntents, the opt-in companion to abort markers Adds the setting next to transactionLogAbortMarkers in Storage Tuning and the options list, names it in the three passages that list full-restart settings, and mentions it in the 5.4 release note. Dispatch-Task: docs-harper-3928-intents Co-Authored-By: Claude Opus 5.5 --- reference/configuration/operations.md | 4 ++-- reference/configuration/options.md | 1 + reference/database/storage-tuning.md | 20 ++++++++++++++++++++ reference/operations-api/operations.md | 2 +- release-notes/v5-lincoln/5.4.md | 2 ++ 5 files changed, 26 insertions(+), 3 deletions(-) diff --git a/reference/configuration/operations.md b/reference/configuration/operations.md index bc533bae1..2e9b089b3 100644 --- a/reference/configuration/operations.md +++ b/reference/configuration/operations.md @@ -16,7 +16,7 @@ For the full list of configurable options, see [Configuration Options](./options ## Set Configuration -Modifies one or more Harper configuration parameters. **Most changes require a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags` and [`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers)). A change to `replication.threads` takes effect only on a full `restart`. +Modifies one or more Harper configuration parameters. **Most changes require a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`, [`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) and [`storage.rocks.transactionLogWriteIntents`](../database/storage-tuning.md#storagerockstransactionlogwriteintents)). A change to `replication.threads` takes effect only on a full `restart`. Full-text request work limits (`storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds`) take effect in HTTP workers after `restart_service` or a full restart. Changes to `storage.fullText.runtime` require a full Harper process restart; `restart_service` does not replace the native process budget. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). @@ -86,7 +86,7 @@ To restart the whole cluster afterward, follow with [restart_service](../operati } ``` -`restart_service` restarts only worker threads, which keep the process's V8 flags and its `storage.rocks.transactionLogAbortMarkers` value. A change to [`threads.v8Flags`](./options.md#applying-v8-flags) or [`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) takes effect only after a [restart](../operations-api/operations.md#restart), sent to each node in turn. +`restart_service` restarts only worker threads, which keep the process's V8 flags and its `storage.rocks.transactionLogAbortMarkers` and `storage.rocks.transactionLogWriteIntents` values. A change to [`threads.v8Flags`](./options.md#applying-v8-flags), [`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) or [`storage.rocks.transactionLogWriteIntents`](../database/storage-tuning.md#storagerockstransactionlogwriteintents) takes effect only after a [restart](../operations-api/operations.md#restart), sent to each node in turn. --- diff --git a/reference/configuration/options.md b/reference/configuration/options.md index e4763399d..fc8f33c10 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -346,6 +346,7 @@ storage: - `rocks.writeBufferManagerCostToCache` — When enabled, memtable memory and the block cache share a single, unified memory pool. During heavy write bursts, the block cache dynamically shrinks to give memtables more room. Once those memtables flush to disk, the block cache automatically reclaims that memory space. _Default_: `true`. Has no effect when `writeBufferManagerSize` is `0`. (Added in: v5.1.0) - `rocks.writeBufferManagerAllowStall` — Stall writes when memtable memory exceeds `writeBufferManagerSize` (hard cap) instead of allowing brief overshoot with more aggressive flushing (soft cap); _Default_: `true`. (Added in: v5.1.0) - `rocks.transactionLogAbortMarkers` — Mark the transaction-log entry of a transaction that fails after logging as aborted, so replication, subscriptions, `read_audit_log` and crash replay skip it, and give up on a write that keeps conflicting instead of retrying it without limit. `false` keeps the 5.3 behavior. Read once per process: a change takes effect after a [`restart`](../operations-api/operations.md#restart), not `restart_service`. Harper 5.3.4 and earlier cannot read the markers. _Default_: `true`. See [Storage Tuning — RocksDB Transaction Log](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) +- `rocks.transactionLogWriteIntents` — Refuse a commit that would conflict with an older, still-open transaction before it writes its transaction-log entry, and wait for that transaction to finish. Shortens the longest waits under heavy write contention, at the cost of more retries and lower throughput when transactions are large or contention is low. No effect when `rocks.transactionLogAbortMarkers` is `false`. Read once per process: a change takes effect after a [`restart`](../operations-api/operations.md#restart), not `restart_service`. _Default_: `false`. See [Storage Tuning — RocksDB Transaction Log](../database/storage-tuning.md#storagerockstransactionlogwriteintents) --- diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index d9a516b4a..87f816ab1 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -333,6 +333,26 @@ storage: transactionLogAbortMarkers: false ``` +### `storage.rocks.transactionLogWriteIntents` + + + +Type: `boolean` + +Default: `false` + +When `true`, a commit that would conflict with a write from an older transaction that is still open is refused before it writes its transaction-log entry. It then waits for that transaction to commit or abort, and retries. A refused attempt does not count toward the 40-conflict limit of [`transactionLogAbortMarkers`](#storagerockstransactionlogabortmarkers). + +This shortens the longest waits when many transactions write the same records at once. It adds retries and lowers throughput when transactions are large or contention is low, so leave it off unless overlapping writes to the same records see long commit delays. + +It has no effect when `transactionLogAbortMarkers` is `false`. Like that setting, it is read once per process: a change takes effect after a [`restart`](../operations-api/operations.md#restart) of each node, not [`restart_service`](../operations-api/operations.md#restart_service). + +```yaml +storage: + rocks: + transactionLogWriteIntents: true +``` + ## Storage Reclamation `storage.reclamation` controls how Harper evicts data from caching tables (tables with [`sourcedFrom`](../resources/resource-api.md#sourcedfromresource-options)) when disk usage runs high. Reclamation does **not** affect non-caching tables — those rely on explicit deletion, TTL expiration, or [compaction](./compaction.md). diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 6c12c8004..672d4b703 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -1845,7 +1845,7 @@ Detailed documentation: [Configuration Overview](../configuration/overview.md) Updates configuration parameters in `harper-config.yaml`. Most changes require a restart (`restart` or `restart_service`) to take effect. See [Set Configuration](../configuration/operations.md#set-configuration) for full-text restart requirements. -Supports `"replicated": true` to apply the same change to all cluster nodes in one call; per-node outcomes are returned in the response's `replicated` array. Only send cluster-appropriate parameters when replicating — node-local parameters (ports, `node.hostname`, file paths, TLS material, `replication.hostname`/`url`/`routes`) would overwrite every peer's local values. To apply the change cluster-wide, follow with `restart_service` using `"replicated": true` (which restarts nodes one at a time); `restart_service` restarts only worker threads, though, so changes to [`threads.v8Flags`](../configuration/options.md#applying-v8-flags), `storage.fullText.runtime.*` or [`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) take effect only after a `restart` sent to each node in turn (`restart` does not replicate). See [Configuration Operations](../configuration/operations.md#set-configuration) for details. +Supports `"replicated": true` to apply the same change to all cluster nodes in one call; per-node outcomes are returned in the response's `replicated` array. Only send cluster-appropriate parameters when replicating — node-local parameters (ports, `node.hostname`, file paths, TLS material, `replication.hostname`/`url`/`routes`) would overwrite every peer's local values. To apply the change cluster-wide, follow with `restart_service` using `"replicated": true` (which restarts nodes one at a time); `restart_service` restarts only worker threads, though, so changes to [`threads.v8Flags`](../configuration/options.md#applying-v8-flags), `storage.fullText.runtime.*`, [`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) or [`storage.rocks.transactionLogWriteIntents`](../database/storage-tuning.md#storagerockstransactionlogwriteintents) take effect only after a `restart` sent to each node in turn (`restart` does not replicate). See [Configuration Operations](../configuration/operations.md#set-configuration) for details. ```json { diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 24539369b..d78500fe4 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -142,6 +142,8 @@ The new `threads.v8Flags` setting takes a V8 flag, or a list of them, and applie The new `storage.rocks.transactionLogAbortMarkers` setting is on by default (RocksDB only). When a transaction fails after its entry was written to the transaction log, the entry is now marked aborted, and replication, subscriptions, `read_audit_log` and replay after a crash all skip it. A write that keeps conflicting with other writes after its log entry was written no longer retries without limit: like other conflicting writes, it fails with HTTP `500` after 40 commit conflicts, or with HTTP `503` and code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT` once it has been in conflict for longer than `storage.maxTransactionQueueTime`, whichever comes first. Writes applied from replication or a caching source, and a transaction part of which has already committed, still retry until they commit. Set the setting to `false` to keep the 5.3 behavior. Harper reads it once per process, so a change needs a `restart` of each node; `restart_service` does not apply it. Harper 5.3.4 and earlier cannot read the markers. See the [Upgrade Notes](#upgrade-notes) and [`storage.rocks.transactionLogAbortMarkers`](/reference/v5/database/storage-tuning#storagerockstransactionlogabortmarkers) ([harper#3928](https://github.com/HarperFast/harper/pull/3928)). +The new [`storage.rocks.transactionLogWriteIntents`](/reference/v5/database/storage-tuning#storagerockstransactionlogwriteintents) setting, off by default, makes a commit that would conflict with an older, still-open transaction wait for it before writing its transaction-log entry. This shortens the longest waits under heavy write contention, but adds retries and lowers throughput when transactions are large or contention is low. It needs abort markers on and a full `restart` to change. + ## Backups ### Restore Checks a Backup Before Deleting the Database From e6211c1a2b4ac80ba9f19d10ad84279e1fa2c8ee Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 2 Nov 2026 15:32:40 -0700 Subject: [PATCH 6/6] Say an already-logged retry is refused before commit, not before its log write Dispatch-Task: docs-harper-3928-intents Co-Authored-By: Claude Opus 5.5 --- reference/configuration/options.md | 2 +- reference/database/storage-tuning.md | 2 +- release-notes/v5-lincoln/5.4.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index fc8f33c10..3b990a98c 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -346,7 +346,7 @@ storage: - `rocks.writeBufferManagerCostToCache` — When enabled, memtable memory and the block cache share a single, unified memory pool. During heavy write bursts, the block cache dynamically shrinks to give memtables more room. Once those memtables flush to disk, the block cache automatically reclaims that memory space. _Default_: `true`. Has no effect when `writeBufferManagerSize` is `0`. (Added in: v5.1.0) - `rocks.writeBufferManagerAllowStall` — Stall writes when memtable memory exceeds `writeBufferManagerSize` (hard cap) instead of allowing brief overshoot with more aggressive flushing (soft cap); _Default_: `true`. (Added in: v5.1.0) - `rocks.transactionLogAbortMarkers` — Mark the transaction-log entry of a transaction that fails after logging as aborted, so replication, subscriptions, `read_audit_log` and crash replay skip it, and give up on a write that keeps conflicting instead of retrying it without limit. `false` keeps the 5.3 behavior. Read once per process: a change takes effect after a [`restart`](../operations-api/operations.md#restart), not `restart_service`. Harper 5.3.4 and earlier cannot read the markers. _Default_: `true`. See [Storage Tuning — RocksDB Transaction Log](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) -- `rocks.transactionLogWriteIntents` — Refuse a commit that would conflict with an older, still-open transaction before it writes its transaction-log entry, and wait for that transaction to finish. Shortens the longest waits under heavy write contention, at the cost of more retries and lower throughput when transactions are large or contention is low. No effect when `rocks.transactionLogAbortMarkers` is `false`. Read once per process: a change takes effect after a [`restart`](../operations-api/operations.md#restart), not `restart_service`. _Default_: `false`. See [Storage Tuning — RocksDB Transaction Log](../database/storage-tuning.md#storagerockstransactionlogwriteintents) +- `rocks.transactionLogWriteIntents` — Make a commit that would conflict with an older, still-open transaction wait for that transaction to finish, usually before writing its transaction-log entry. Shortens the longest waits under heavy write contention, at the cost of more retries and lower throughput when transactions are large or contention is low. No effect when `rocks.transactionLogAbortMarkers` is `false`. Read once per process: a change takes effect after a [`restart`](../operations-api/operations.md#restart), not `restart_service`. _Default_: `false`. See [Storage Tuning — RocksDB Transaction Log](../database/storage-tuning.md#storagerockstransactionlogwriteintents) --- diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index 87f816ab1..e169a4823 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -341,7 +341,7 @@ Type: `boolean` Default: `false` -When `true`, a commit that would conflict with a write from an older transaction that is still open is refused before it writes its transaction-log entry. It then waits for that transaction to commit or abort, and retries. A refused attempt does not count toward the 40-conflict limit of [`transactionLogAbortMarkers`](#storagerockstransactionlogabortmarkers). +When `true`, a commit that would conflict with a write from an older transaction that is still open is refused before it writes its transaction-log entry, or before it commits if an earlier attempt already wrote that entry. It then waits for that transaction to commit or abort, and retries. A refused attempt does not count toward the 40-conflict limit of [`transactionLogAbortMarkers`](#storagerockstransactionlogabortmarkers). This shortens the longest waits when many transactions write the same records at once. It adds retries and lowers throughput when transactions are large or contention is low, so leave it off unless overlapping writes to the same records see long commit delays. diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index d78500fe4..9cf24eb2a 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -142,7 +142,7 @@ The new `threads.v8Flags` setting takes a V8 flag, or a list of them, and applie The new `storage.rocks.transactionLogAbortMarkers` setting is on by default (RocksDB only). When a transaction fails after its entry was written to the transaction log, the entry is now marked aborted, and replication, subscriptions, `read_audit_log` and replay after a crash all skip it. A write that keeps conflicting with other writes after its log entry was written no longer retries without limit: like other conflicting writes, it fails with HTTP `500` after 40 commit conflicts, or with HTTP `503` and code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT` once it has been in conflict for longer than `storage.maxTransactionQueueTime`, whichever comes first. Writes applied from replication or a caching source, and a transaction part of which has already committed, still retry until they commit. Set the setting to `false` to keep the 5.3 behavior. Harper reads it once per process, so a change needs a `restart` of each node; `restart_service` does not apply it. Harper 5.3.4 and earlier cannot read the markers. See the [Upgrade Notes](#upgrade-notes) and [`storage.rocks.transactionLogAbortMarkers`](/reference/v5/database/storage-tuning#storagerockstransactionlogabortmarkers) ([harper#3928](https://github.com/HarperFast/harper/pull/3928)). -The new [`storage.rocks.transactionLogWriteIntents`](/reference/v5/database/storage-tuning#storagerockstransactionlogwriteintents) setting, off by default, makes a commit that would conflict with an older, still-open transaction wait for it before writing its transaction-log entry. This shortens the longest waits under heavy write contention, but adds retries and lowers throughput when transactions are large or contention is low. It needs abort markers on and a full `restart` to change. +The new [`storage.rocks.transactionLogWriteIntents`](/reference/v5/database/storage-tuning#storagerockstransactionlogwriteintents) setting, off by default, makes a commit that would conflict with an older, still-open transaction wait for that transaction to finish, usually before writing its transaction-log entry. This shortens the longest waits under heavy write contention, but adds retries and lowers throughput when transactions are large or contention is low. It needs abort markers on and a full `restart` to change. ## Backups