Document full-text paging and configurable search limits

reference/configuration/operations.md

@@ -16,14 +16,18 @@ For the full list of configurable options, see [Configuration Options](./options
 
 ## Set Configuration
 
-Modifies one or more Harper configuration parameters. **Requires a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`). A change to `replication.threads` takes effect only on a full `restart`.
+Modifies one or more Harper configuration parameters. **Most changes require a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`). A change to `replication.threads` takes effect only on a full `restart`.
+
+Full-text request work limits (`storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds`) take effect in HTTP workers after `restart_service` or a full restart. Changes to `storage.fullText.runtime` require a full Harper process restart; `restart_service` does not replace the native process budget. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging).
 
 `operation` _(required)_ — must be `set_configuration`
 
 `replicated` _(optional)_ <VersionBadge version="v5.2.0" /> — set to `true` to apply the same configuration change to all cluster nodes in a single call. The origin node applies the change first; if its local write fails, nothing is sent to peers. Requires replication to be configured.
 
 Additional properties correspond to configuration keys in underscore-separated format (e.g. `logging_level` for `logging.level`, `clustering_enabled` for `clustering.enabled`).
 
+For full-text settings, use individual parameter names such as `storage_fullText_maxRawHitsPerRequest` or `storage_fullText_runtime_maxSearchThreads`. `set_configuration` does not accept a whole `storage_fullText` object; use `storage.fullText` as the section name only in the YAML configuration file.
+
 :::warning Replicate cluster-appropriate parameters only
 A replicated call sends the exact same values to every node. Do not include node-local parameters — ports, `node.hostname`, `rootPath`, storage/log file paths, TLS certificate or key paths, or `replication.hostname`/`url`/`routes` — as they would overwrite each peer's local values.
 :::
@@ -42,7 +46,7 @@ A replicated call sends the exact same values to every node. Do not include node
 
 ```json
 {
-	"message": "Configuration successfully set. You must restart HarperDB for new config settings to take effect."
+	"message": "Configuration successfully set. You must restart Harper for new config settings to take effect."
 }
 ```
 

reference/configuration/options.md

@@ -232,7 +232,7 @@ logging:
   level: warn
   root: ~/hdb/log
   stdStreams: false
-  auditLog: false
+  auditLog: true
   rotation:
     interval: 1D
     maxSize: 100M
@@ -244,7 +244,7 @@ logging:
 - `path` — Explicit log file path (overrides `root`)
 - `stdStreams` — Write to stdout/stderr; _Default_: `false`
 - `console` — Include `console.*` output; _Default_: `true`
-- `auditLog` — Enable table transaction audit logging; _Default_: `false`
+- `auditLog` — Enable table transaction audit logging; _Default_: `true`
 - `auditRetention` — Audit log retention duration; _Default_: `3d`
 - `external` — Logging for components using the logger API; sub-options: `level`, `path`
 - `rotation.enabled` / `rotation.compress` / `rotation.interval` / `rotation.maxSize` / `rotation.path` — Log file rotation (activates when `interval` or `maxSize` is set)
@@ -312,6 +312,19 @@ storage:
 ```
 
 - `engine` — The database storage engine to use for new databases. Currently supported engines are `rocksdb` and `lmdb`. The default is `rocksdb`. Existing databases will use the engine that was used when they were created.
+- `fullText` <VersionBadge version="v5.4.0" /> — Full-text search work and native process limits. Sub-options:
+  - `maxRawHitsPerRequest` — Maximum fetched hits per request, including bounded prefix replay when traversal is needed; default `10000`.
+  - `maxResultsPerRequest` — Maximum retained offset plus limit. A request must also fit the separate raw-hit budget; default `10000`.
+  - `requestBudgetMilliseconds` — Execution budget across native batches, source checks and highlights, capped by the native execution limit; default `5000` ms. Reader acquisition and freshness waiting have separate bounds.
+  - `runtime.maxResidentIndexes` — Maximum resident native indexes; default `128`.
+  - `runtime.maxIndexingThreads` — Maximum aggregate indexing-thread reservations across resident native indexes; default `128`.
+  - `runtime.maxSearchThreads` — Maximum aggregate search-thread reservations across resident native indexes; default `256`.
+  - `runtime.maxWriterMemoryBytes` — Aggregate writer-memory reservations across resident native indexes; default `4294967296` (4 GiB).
+  - `runtime.maxQueuedBytes` — Aggregate queued-write reservations across resident native indexes; default `8589934592` (8 GiB).
+  - `runtime.maxExpensiveSearches` — Maximum concurrent expensive native searches; default `8`.
+
+  The runtime budgets apply together and are not hard bounds on total process memory. See [Set Configuration](./operations.md#set-configuration) for when request and native runtime changes take effect. Use the usual underscore-separated parameter names, such as `storage_fullText_maxRawHitsPerRequest`. See [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for costs; see [readiness and recovery](../resources/query-optimization.md#readiness-and-recovery) for failures.
+
 - `writeAsync` — Disable disk sync for higher throughput (**disables durability guarantees**); _Default_: `false`
 - `caching` — In-memory record caching; _Default_: `true`
 - `compression` — LZ4 record compression; _Default_: `true` (enabled by default since v4.3.0). Sub-options: `dictionary`, `threshold`

reference/configuration/overview.md

@@ -21,7 +21,7 @@ For a complete reference of all available configuration options, see [Configurat
 
 ## The Configuration File
 
-To change a configuration value, edit `harper-config.yaml` and save. **Harper must be restarted for changes to take effect.**
+To change a configuration value, edit `harper-config.yaml` and save. **Restart Harper to apply file edits.** See [Set Configuration](./operations.md#set-configuration) for settings that require a full process restart.
 
 Configuration keys use camelCase (e.g. `operationsApi`). Nested keys use dot notation conceptually (e.g. `operationsApi.network.port`).
 

reference/database/storage-tuning.md

@@ -202,7 +202,7 @@ Changing this setting is safe for existing data — records already written keep
 
 To pin one table's encoding regardless of the global setting, use the [`@table(randomAccessFields:)`](./schema.md#randomaccessfields) directive when creating the table; editing the directive later does not repin an existing table. Tables that carry the directive ignore this option entirely, which also means a fleet-wide change to this setting will not move them.
 
-The typed dictionary is bounded, and reaching the bound is not an error: records with novel shapes past that point still write and read correctly, they are simply stored without random-access field encoding. [`describe_table`](../operations-api/operations.md#describe_table) reports both the current size and the ceiling in force (`typed_structure_count` against `typed_structure_limit`, 256 by default), which is the reliable way to see whether a table's shapes are staying within it. The classic dictionary is bounded far lower — 32 shared named-record structures — so a table with more than 32 distinct field-name sets sits at that number as a matter of course; it is a normal reading, not a leak.
+The typed dictionary is bounded, and reaching the bound is not an error: records with novel shapes past that point still write and read correctly, they are simply stored without random-access field encoding. [`describe_table`](../operations-api/operations.md#record-structure-dictionaries) reports both the current size and the ceiling in force (`typed_structure_count` against `typed_structure_limit`, 256 by default), which is the reliable way to see whether a table's shapes are staying within it. The classic dictionary is bounded far lower — 32 shared named-record structures — so a table with more than 32 distinct field-name sets sits at that number as a matter of course; it is a normal reading, not a leak.
 
 ## RocksDB Memory
 
@@ -395,6 +395,16 @@ storage:
   compression: true
 ```
 
+## `storage.fullText`
+
+<VersionBadge version="v5.4.0" />
+
+Type: `object`
+
+Default: See the [storage option reference](../configuration/options.md#storage).
+
+The [storage option reference](../configuration/options.md#storage) lists the full-text request and native process defaults. Raise the fetched-hit budget to allow more filtering work; raising it does not raise the separate retained-result limit. Larger limits can increase CPU, source reads and memory use for every permitted query. Native process budgets also share capacity across indexes, so tune them with concurrent workloads rather than a single query. See [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for paging costs and restart requirements; see [readiness and recovery](../resources/query-optimization.md#readiness-and-recovery) for failures.
+
 ## Related
 
 - [Configuration Options](../configuration/options.md) — full list of `storage` options

reference/database/transaction.md

@@ -23,7 +23,7 @@ On RocksDB (the default storage engine) the transaction log is physically a sing
 
 The transaction log is **enabled by default**. To disable it, set [`logging.auditLog`](../logging/configuration.md) to `false` in `harper-config.yaml` and restart Harper.
 
-> The transaction log is required for real-time messaging (WebSocket and MQTT subscriptions) and replication. Do not disable it if real-time features or replication are in use.
+> The transaction log is required for real-time messaging (WebSocket and MQTT subscriptions), replication and full-text indexing. Do not disable it while any of these features are in use.
 
 ## Operations
 
@@ -134,7 +134,7 @@ Deletes transaction log entries older than the specified timestamp. Deprecated i
 Deletes transaction log entries older than the specified timestamp.
 
 :::warning Database-wide and irreversible
-On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This purges whole log files whose entries predate the timestamp, removing [`read_audit_log`](#read_audit_log) history for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender detects that the requested start predates its retained history and forces a full base copy instead of incremental catch-up — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position.
+On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This purges whole log files whose entries predate the timestamp, removing [`read_audit_log`](#read_audit_log) history for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender detects that the requested start predates its retained history and forces a full base copy instead of incremental catch-up — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. A full-text rebuild also needs retained history to replay writes committed during its scan; purging that history can force another full rebuild and extend the search outage. Pause manual cleanup during an upgrade rebuild and size automatic retention for the workload.
 :::
 
 Parameters:
@@ -204,7 +204,7 @@ type Dog @table(audit: true) {
 }
 ```
 
-This overrides the [`logging.auditLog`](../logging/configuration.md) global configuration for that specific table.
+This overrides the [`logging.auditLog`](../logging/configuration.md) global configuration for that specific table. New schema tables with `@fullText` must explicitly declare `@table(audit: true)`, even when global audit logging is enabled. Existing audited tables can add an index without restating `audit`. Keep auditing enabled while any full-text index is declared. Setting `@table(audit: false)` while a full-text index is declared is rejected during schema registration; this is a per-table prerequisite, independent of the global default.
 
 ## Related Documentation
 

reference/logging/configuration.md

@@ -102,11 +102,11 @@ Default: `true`
 
 Enables transaction logging. When enabled, Harper records every insert, update, and delete to the database's transaction log. This history is read with the [`read_audit_log`](../database/transaction.md#read_audit_log) operation — a legacy name; there is a single transaction log, not a separate audit table.
 
-See [Database / Transaction Logging](../database/transaction.md) for details on the transaction log.
+Full-text indexing requires transaction logging. New schema tables with `@fullText` must explicitly declare `@table(audit: true)`, even when this global setting is enabled. Existing audited tables can add an index without restating `audit`; disabling table auditing while an index is declared is rejected. See [Database / Transaction Logging](../database/transaction.md) for details on the transaction log.
 
 ```yaml
 logging:
-  auditLog: false
+  auditLog: true
 ```
 
 ### `logging.auditRetention`
@@ -339,7 +339,7 @@ logging:
   file: true
   stdStreams: false
   console: false
-  auditLog: false
+  auditLog: true
   auditRetention: 3d
   rotation:
     enabled: true