diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index c18f491da..f720368d4 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -208,7 +208,9 @@ outstanding commit on a thread exceeds (default 45s), Harper rejects new record updates and publishes from new application requests on that thread with `Outstanding write transactions have too long of queue, please try again later` (HTTP 503). Deletes and writes applied from a canonical source (e.g. replication or a caching source) -bypass this check. +bypass this check. The same duration also limits how long a write keeps retrying conflicts with other +writes, which fails with a separate 503, code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT`; see +[`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers). diff --git a/reference/configuration/operations.md b/reference/configuration/operations.md index faea62523..2e9b089b3 100644 --- a/reference/configuration/operations.md +++ b/reference/configuration/operations.md @@ -16,7 +16,7 @@ For the full list of configurable options, see [Configuration Options](./options ## Set Configuration -Modifies one or more Harper configuration parameters. **Most changes require a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`). A change to `replication.threads` takes effect only on a full `restart`. +Modifies one or more Harper configuration parameters. **Most changes require a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`, [`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) and [`storage.rocks.transactionLogWriteIntents`](../database/storage-tuning.md#storagerockstransactionlogwriteintents)). A change to `replication.threads` takes effect only on a full `restart`. Full-text request work limits (`storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds`) take effect in HTTP workers after `restart_service` or a full restart. Changes to `storage.fullText.runtime` require a full Harper process restart; `restart_service` does not replace the native process budget. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). @@ -86,7 +86,7 @@ To restart the whole cluster afterward, follow with [restart_service](../operati } ``` -`restart_service` restarts only worker threads, which keep the process's V8 flags. A change to [`threads.v8Flags`](./options.md#applying-v8-flags) takes effect only after a [restart](../operations-api/operations.md#restart), sent to each node in turn. +`restart_service` restarts only worker threads, which keep the process's V8 flags and its `storage.rocks.transactionLogAbortMarkers` and `storage.rocks.transactionLogWriteIntents` values. A change to [`threads.v8Flags`](./options.md#applying-v8-flags), [`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) or [`storage.rocks.transactionLogWriteIntents`](../database/storage-tuning.md#storagerockstransactionlogwriteintents) takes effect only after a [restart](../operations-api/operations.md#restart), sent to each node in turn. --- diff --git a/reference/configuration/options.md b/reference/configuration/options.md index ee62df608..3b990a98c 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -330,7 +330,7 @@ storage: - `compression` — LZ4 record compression; _Default_: `true` (enabled by default since v4.3.0). Sub-options: `dictionary`, `threshold` - `compactOnStart` — Compact all non-system databases on startup; _Default_: `false` (Added in: v4.3.0) - `compactOnStartKeepBackup` — Retain compaction backups; _Default_: `false` -- `maxTransactionQueueTime` — Per-thread write-commit duration threshold for HTTP 503 backpressure; see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionqueuetime); _Default_: `45s` +- `maxTransactionQueueTime` — Per-thread write-commit duration threshold for HTTP 503 backpressure, and the limit on how long a write keeps retrying conflicts with other writes (a separate 503, code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT`); see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionqueuetime); _Default_: `45s` - `noReadAhead` — Advise OS against read-ahead; _Default_: `false` - `prefetchWrites` — Prefetch before write transactions; _Default_: `true` - `randomAccessFields` — Encode records as typed random-access structures, so a single field can be read without decoding the whole record. Best for tables whose records have a stable set of fields with stable types; leave off for wide, sparse, or variably typed schemas. Read by each table when its store opens, so a change applies to existing tables on restart, and ignored entirely by tables declaring the [`@table(randomAccessFields:)`](../database/schema.md#randomaccessfields) directive; _Default_: `false`. See [Storage Tuning — Record Encoding](../database/storage-tuning.md#storagerandomaccessfields) @@ -345,6 +345,8 @@ storage: - `rocks.writeBufferManagerSize` — Process-wide cap (bytes) on RocksDB memtable memory across all databases. `0` disables; _Default_: one third of `blockCacheSize` (enabled). See [Storage Tuning — RocksDB Memory](../database/storage-tuning.md#rocksdb-memory) (Added in: v5.1.0) - `rocks.writeBufferManagerCostToCache` — When enabled, memtable memory and the block cache share a single, unified memory pool. During heavy write bursts, the block cache dynamically shrinks to give memtables more room. Once those memtables flush to disk, the block cache automatically reclaims that memory space. _Default_: `true`. Has no effect when `writeBufferManagerSize` is `0`. (Added in: v5.1.0) - `rocks.writeBufferManagerAllowStall` — Stall writes when memtable memory exceeds `writeBufferManagerSize` (hard cap) instead of allowing brief overshoot with more aggressive flushing (soft cap); _Default_: `true`. (Added in: v5.1.0) +- `rocks.transactionLogAbortMarkers` — Mark the transaction-log entry of a transaction that fails after logging as aborted, so replication, subscriptions, `read_audit_log` and crash replay skip it, and give up on a write that keeps conflicting instead of retrying it without limit. `false` keeps the 5.3 behavior. Read once per process: a change takes effect after a [`restart`](../operations-api/operations.md#restart), not `restart_service`. Harper 5.3.4 and earlier cannot read the markers. _Default_: `true`. See [Storage Tuning — RocksDB Transaction Log](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) +- `rocks.transactionLogWriteIntents` — Make a commit that would conflict with an older, still-open transaction wait for that transaction to finish, usually before writing its transaction-log entry. Shortens the longest waits under heavy write contention, at the cost of more retries and lower throughput when transactions are large or contention is low. No effect when `rocks.transactionLogAbortMarkers` is `false`. Read once per process: a change takes effect after a [`restart`](../operations-api/operations.md#restart), not `restart_service`. _Default_: `false`. See [Storage Tuning — RocksDB Transaction Log](../database/storage-tuning.md#storagerockstransactionlogwriteintents) --- diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index bf2a99806..e169a4823 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -45,6 +45,8 @@ Beginning with v5.2.1, Harper tracks every outstanding commit attempt for this c The [`transaction-commit-time` metric](../analytics/overview.md#storage-metrics) provides the corresponding commit-latency distribution, but only records each attempt after it settles. +The same duration also limits how long one write keeps retrying conflicts with other writes before it fails with a different HTTP 503, code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT`. See [`storage.rocks.transactionLogAbortMarkers`](#storagerockstransactionlogabortmarkers) for which writes this applies to. + Lower this in latency-sensitive systems where it is better to shed load early than to let request queues grow. Raise it when occasional disk-write bursts are expected and the application can tolerate longer commit latency. ```yaml @@ -307,6 +309,50 @@ The default (`true`) strictly bounds total memtable memory, applying write backp This option is the only `WriteBufferManager` setting that can be changed at runtime — `costToCache` is fixed at first creation. +## RocksDB Transaction Log + +### `storage.rocks.transactionLogAbortMarkers` + + + +Type: `boolean` + +Default: `true` + +Applies only when `storage.engine` is `rocksdb`. Harper writes a transaction's entry to the [transaction log](./transaction.md) before the transaction commits. With this setting on, when a transaction fails after that point, its entry is marked aborted, and every reader of the log skips it: replication, subscriptions, [`read_audit_log`](./transaction.md#read_audit_log), and replay after a crash. A transaction is still replayed on restart if the process crashes before it commits or is marked, including when a disk error delays its marker. + +It also bounds a write that keeps conflicting with other writes. The write fails with HTTP `500` after 40 commit conflicts, or with HTTP `503` and code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT` once it has been in conflict for longer than [`storage.maxTransactionQueueTime`](#storagemaxtransactionqueuetime), whichever comes first. A `503` that carries `retryable: false` may have been partly applied, so do not resend that request as is. Writes applied from replication or a caching source, and a transaction part of which has already committed, still retry until they commit. + +Set it to `false` to keep the 5.3 behavior, where a write whose log entry was already written retries without limit. Any value other than `false`, `0`, `no` or `off` leaves it on. Harper reads the setting once per process, so a change takes effect after a [`restart`](../operations-api/operations.md#restart) of each node, or a process or container restart. [`restart_service`](../operations-api/operations.md#restart_service) restarts only worker threads and does not apply it. + +**Downgrades:** Harper 5.3.4 and earlier cannot read abort markers and would treat a marked transaction as committed. Do not run them on a node's database files, or a backup of them, after the node has run 5.4 with this setting on. Harper 5.3.5 and later read the markers. Setting it to `false` later stops new markers but does not remove ones already written. + +```yaml +storage: + rocks: + transactionLogAbortMarkers: false +``` + +### `storage.rocks.transactionLogWriteIntents` + + + +Type: `boolean` + +Default: `false` + +When `true`, a commit that would conflict with a write from an older transaction that is still open is refused before it writes its transaction-log entry, or before it commits if an earlier attempt already wrote that entry. It then waits for that transaction to commit or abort, and retries. A refused attempt does not count toward the 40-conflict limit of [`transactionLogAbortMarkers`](#storagerockstransactionlogabortmarkers). + +This shortens the longest waits when many transactions write the same records at once. It adds retries and lowers throughput when transactions are large or contention is low, so leave it off unless overlapping writes to the same records see long commit delays. + +It has no effect when `transactionLogAbortMarkers` is `false`. Like that setting, it is read once per process: a change takes effect after a [`restart`](../operations-api/operations.md#restart) of each node, not [`restart_service`](../operations-api/operations.md#restart_service). + +```yaml +storage: + rocks: + transactionLogWriteIntents: true +``` + ## Storage Reclamation `storage.reclamation` controls how Harper evicts data from caching tables (tables with [`sourcedFrom`](../resources/resource-api.md#sourcedfromresource-options)) when disk usage runs high. Reclamation does **not** affect non-caching tables — those rely on explicit deletion, TTL expiration, or [compaction](./compaction.md). diff --git a/reference/database/transaction.md b/reference/database/transaction.md index da616e19d..8d934d229 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -21,6 +21,8 @@ Available since: v4.1.0 On RocksDB (the default storage engine) the transaction log is physically a single database-wide log shared by all tables: history is read per table, while deletion operates on the whole database's log. + By default on RocksDB, a transaction that fails after its entry was written to the log is marked aborted and left out of the log's history, including `read_audit_log` results. The [`storage.rocks.transactionLogAbortMarkers`](./storage-tuning.md#storagerockstransactionlogabortmarkers) setting controls this. + The transaction log is **enabled by default**. To disable it, set [`logging.auditLog`](../logging/configuration.md) to `false` in `harper-config.yaml` and restart Harper. > The transaction log is required for real-time messaging (WebSocket and MQTT subscriptions), replication and full-text indexing. Do not disable it while any of these features are in use. diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 981103058..672d4b703 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -1845,7 +1845,7 @@ Detailed documentation: [Configuration Overview](../configuration/overview.md) Updates configuration parameters in `harper-config.yaml`. Most changes require a restart (`restart` or `restart_service`) to take effect. See [Set Configuration](../configuration/operations.md#set-configuration) for full-text restart requirements. -Supports `"replicated": true` to apply the same change to all cluster nodes in one call; per-node outcomes are returned in the response's `replicated` array. Only send cluster-appropriate parameters when replicating — node-local parameters (ports, `node.hostname`, file paths, TLS material, `replication.hostname`/`url`/`routes`) would overwrite every peer's local values. To apply the change cluster-wide, follow with `restart_service` using `"replicated": true` (which restarts nodes one at a time); `restart_service` restarts only worker threads, though, so changes to [`threads.v8Flags`](../configuration/options.md#applying-v8-flags) or `storage.fullText.runtime.*` take effect only after a `restart` sent to each node in turn (`restart` does not replicate). See [Configuration Operations](../configuration/operations.md#set-configuration) for details. +Supports `"replicated": true` to apply the same change to all cluster nodes in one call; per-node outcomes are returned in the response's `replicated` array. Only send cluster-appropriate parameters when replicating — node-local parameters (ports, `node.hostname`, file paths, TLS material, `replication.hostname`/`url`/`routes`) would overwrite every peer's local values. To apply the change cluster-wide, follow with `restart_service` using `"replicated": true` (which restarts nodes one at a time); `restart_service` restarts only worker threads, though, so changes to [`threads.v8Flags`](../configuration/options.md#applying-v8-flags), `storage.fullText.runtime.*`, [`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) or [`storage.rocks.transactionLogWriteIntents`](../database/storage-tuning.md#storagerockstransactionlogwriteintents) take effect only after a `restart` sent to each node in turn (`restart` does not replicate). See [Configuration Operations](../configuration/operations.md#set-configuration) for details. ```json { diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 916e1b5af..9cf24eb2a 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -12,6 +12,8 @@ All patch release notes for 5.4.x are available on the [releases page](https://g Full-text indexes using fulltext 0.5.0 with current field-based declarations, or prerelease 0.6, rebuild locally after this upgrade. Plan for search unavailability and review the [Tantivy upgrade requirements](#tantivy-027-and-existing-index-rebuilds) before deploying; direct downgrade to fulltext 0.5.x is unsupported. +Harper 5.4 marks failed transactions as aborted in the RocksDB transaction log by default ([`storage.rocks.transactionLogAbortMarkers`](/reference/v5/database/storage-tuning#storagerockstransactionlogabortmarkers); see [Failed Transactions Are Marked Aborted](#failed-transactions-are-marked-aborted-in-the-transaction-log)). Harper 5.3.4 and earlier cannot read those markers and would treat the failed transactions as committed. Do not downgrade a node to them, or open its database files or backups with them, after it has run 5.4 with the setting on. Harper 5.3.5 and later read the markers. Setting it to `false` later stops new markers but does not remove ones already written. + ## Nested Relationship Constraints [Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279) and [Fix wrong rows, unfiltered selects and ignored sorts in relationship queries](https://github.com/HarperFast/harper/pull/3775) correct REST and programmatic relationship queries that accepted nested constraints, non-leading relationship conditions, or top-level relationship sorts without applying them completely. Filters, ordering, offset, and limit inside `select` apply per parent in collection and single-record reads, and parents remain present when the nested selection has no matches. Outer relationship filters — whether they lead the query or follow another condition — constrain the selected children as well as the parents, so an identical filter does not need to be repeated inside the nested selection. When a top-level sort across a to-many relationship (such as `sort(products.name)`) runs in memory — when the target attribute is not `@indexed`, a more selective condition on another attribute leads the query, or `enforceExecutionOrder: true` is set — Harper now orders matched parent records by the list of each parent's related values in relationship order, compared element by element, instead of ignoring the sort; when an `@indexed` target attribute leads the query, Harper continues to order via the related table's index join. @@ -134,6 +136,14 @@ MCP `initialize` is now limited per client identity, on both profiles, by the ne The new `threads.v8Flags` setting takes a V8 flag, or a list of them, and applies them to the Harper process before its first worker thread starts, so every worker thread runs with them. It is for flags such as `--optimize-for-size`, which Node refuses in `NODE_OPTIONS` and which cannot be passed to a worker thread. It can also be set through `HARPER_SET_CONFIG` or `HARPER_CONFIG`. An entry must start with `--`; `set_configuration` refuses one that does not, and Harper stops at startup with an error naming the setting. Unlike the command line, this does not apply a flag's implied flags, so list them yourself (for example `--max-semi-space-size=1` alongside `--optimize-for-size`). A changed value takes effect when the process restarts (the `restart` operation, on each node); `restart_service` restarts only worker threads and does not apply it. Not applied under Bun. See [Applying V8 flags](/reference/v5/configuration/options#applying-v8-flags). +## Transactions + +### Failed Transactions Are Marked Aborted in the Transaction Log + +The new `storage.rocks.transactionLogAbortMarkers` setting is on by default (RocksDB only). When a transaction fails after its entry was written to the transaction log, the entry is now marked aborted, and replication, subscriptions, `read_audit_log` and replay after a crash all skip it. A write that keeps conflicting with other writes after its log entry was written no longer retries without limit: like other conflicting writes, it fails with HTTP `500` after 40 commit conflicts, or with HTTP `503` and code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT` once it has been in conflict for longer than `storage.maxTransactionQueueTime`, whichever comes first. Writes applied from replication or a caching source, and a transaction part of which has already committed, still retry until they commit. Set the setting to `false` to keep the 5.3 behavior. Harper reads it once per process, so a change needs a `restart` of each node; `restart_service` does not apply it. Harper 5.3.4 and earlier cannot read the markers. See the [Upgrade Notes](#upgrade-notes) and [`storage.rocks.transactionLogAbortMarkers`](/reference/v5/database/storage-tuning#storagerockstransactionlogabortmarkers) ([harper#3928](https://github.com/HarperFast/harper/pull/3928)). + +The new [`storage.rocks.transactionLogWriteIntents`](/reference/v5/database/storage-tuning#storagerockstransactionlogwriteintents) setting, off by default, makes a commit that would conflict with an older, still-open transaction wait for that transaction to finish, usually before writing its transaction-log entry. This shortens the longest waits under heavy write contention, but adds retries and lowers throughput when transactions are large or contention is low. It needs abort markers on and a full `restart` to change. + ## Backups ### Restore Checks a Backup Before Deleting the Database