From 16b346b92f1b4d3d392d26278d07422b3e64b2dc Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 27 Oct 2026 13:07:15 -0600 Subject: [PATCH 1/5] Document cut-off JSON streams and storage.maxTransactionOpenTime A streamed JSON array that fails after its 200 head now omits the closing bracket and aborts the connection (harper#3664). Document that contract next to the NDJSON and SSE terminal records, the maxTransactionOpenTime limit that commonly triggers it on long reads, and export_local's new failure outcome. Dispatch-Task: docs-harper-3664 Co-Authored-By: Claude Opus 5.5 --- reference/configuration/options.md | 1 + reference/database/jobs.md | 3 +++ reference/database/storage-tuning.md | 18 ++++++++++++++ reference/operations-api/operations.md | 2 ++ reference/rest/content-types.md | 33 ++++++++++++++++++++++++++ release-notes/v5-lincoln/5.4.md | 8 +++++++ 6 files changed, 65 insertions(+) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 005e5ecac..dd81bf5b5 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -335,6 +335,7 @@ storage: - `compactOnStart` — Compact all non-system databases on startup; _Default_: `false` (Added in: v4.3.0) - `compactOnStartKeepBackup` — Retain compaction backups; _Default_: `false` - `maxTransactionQueueTime` — Per-thread write-commit duration threshold for HTTP 503 backpressure, and the limit on how long a write keeps retrying conflicts with other writes (a separate 503, code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT`); see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionqueuetime); _Default_: `45s` +- `maxTransactionOpenTime` — Milliseconds a transaction may stay open without activity before Harper aborts its writes or releases its read snapshot; see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionopentime); _Default_: `30000` - `noReadAhead` — Advise OS against read-ahead; _Default_: `false` - `prefetchWrites` — Prefetch before write transactions; _Default_: `true` - `randomAccessFields` — Encode records as typed random-access structures, so a single field can be read without decoding the whole record. Best for tables whose records have a stable set of fields with stable types; leave off for wide, sparse, or variably typed schemas. Read by each table when its store opens, so a change applies to existing tables on restart, and ignored entirely by tables declaring the [`@table(randomAccessFields:)`](../database/schema.md#randomaccessfields) directive; _Default_: `false`. See [Storage Tuning — Record Encoding](../database/storage-tuning.md#storagerandomaccessfields) diff --git a/reference/database/jobs.md b/reference/database/jobs.md index cb5a57af0..f8c28d3b7 100644 --- a/reference/database/jobs.md +++ b/reference/database/jobs.md @@ -14,6 +14,7 @@ Job status values: - `IN_PROGRESS` — the job is currently running - `COMPLETE` — the job finished successfully +- `ERROR` — the job failed; its `message` holds the error ## Bulk Operations @@ -159,6 +160,8 @@ Exports table data to a local file in JSON or CSV format. - `filename` _(optional)_ — filename without extension; auto-generated from epoch timestamp if omitted + If the query fails partway through a `json` export, the job ends with status `ERROR` and the partial file is removed. Before v5.4.0, the job completed and the file could hold a complete-looking array ending in an `error` element. + ```json { "operation": "export_local", diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index e169a4823..b1a8a30af 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -54,6 +54,24 @@ storage: maxTransactionQueueTime: 30s ``` +### `storage.maxTransactionOpenTime` + +Type: `number` (milliseconds) + +Default: `30000` + +The maximum time a transaction may stay open without activity. Harper checks open transactions once per this interval, so a stalled transaction is closed between one and two intervals after its last activity. + +- A transaction with uncommitted writes from an application request is aborted, and the request fails with status `422`. Split long-running work into smaller transactions. +- A read-only transaction has its read snapshot released. On RocksDB, a read that continues afterwards fails with `ReadSnapshotExpiredError` (status `503`). + +A streamed response, such as a REST collection query, holds its read snapshot while the client reads. On RocksDB, if the client stops reading for longer than this limit, the read fails when the client resumes. The `200` status is already sent by then, so a JSON response is cut off rather than returned as a `503`; see [When a Streamed Response Fails](../rest/content-types.md#when-a-streamed-response-fails). Clients that read large results slowly should read promptly, page with [`limit(start,end)`](../rest/querying.md#limitend-or-limitstartend), or raise this limit. + +```yaml +storage: + maxTransactionOpenTime: 60000 +``` + ## Compression ### `storage.compression` diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 657ab1251..66a8916d7 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -343,6 +343,8 @@ All bulk import/export operations are asynchronous and return a job ID. Use [`ge Exports query results to a local path on the server. Formats: `json` or `csv`. + If the query fails partway through a `json` export, the job ends with status `ERROR` and the partial file is removed. Before v5.4.0, the job completed and the file could hold a complete-looking array ending in an `error` element. + ```json { "operation": "export_local", diff --git a/reference/rest/content-types.md b/reference/rest/content-types.md index 236384725..61982bd82 100644 --- a/reference/rest/content-types.md +++ b/reference/rest/content-types.md @@ -48,6 +48,39 @@ Comma-separated values format, suitable for data export and spreadsheet import/e **When to use**: Ad-hoc data export, spreadsheet workflows, batch data processing. Not recommended for frequent or production API use. +## When a Streamed Response Fails + +A collection query, or any resource that returns an iterable, is streamed: Harper sends the `200` status and headers, then writes records as it reads them. If reading fails after that, the status can no longer change, so the failure is reported in the body. + +### JSON + + + +When a streamed JSON array fails partway through, Harper writes an `{"error": ": "}` element, leaves out the closing `]`, and aborts the response. On HTTP/1.1 the connection closes without the final chunk; on HTTP/2 the stream is reset. + +``` +[{"id":1},{"id":2},{"error":"ReadSnapshotExpiredError: Read scan snapshot expired; ..."} +``` + +Treat a cut-off body as a failed read, even though the status was `200`. A network error while reading the body, or `response.json()` rejecting, means the result is incomplete. The `error` element is a best-effort diagnostic: if the client is not reading when the failure happens, it can be lost along with unread records. The abort always happens. + +Before v5.4.0, Harper closed the array after the `error` element and ended the response normally, so a partial result parsed as a complete array. + +A request that writes and then streams a result, such as a `POST` whose handler returns query results, has already committed its writes when the stream fails. Retrying it can apply them again. The `error` element includes `replicationUnconfirmed: true` when the writes committed locally but their replication was not confirmed, and `retryable: false` when the request should not be retried. + +On RocksDB, a long streamed read can fail this way when the client stops reading for longer than [`storage.maxTransactionOpenTime`](../database/storage-tuning.md#storagemaxtransactionopentime). + +### NDJSON and Server-Sent Events + +These formats end a failed stream with a terminal record and are unchanged in v5.4.0: + +| Format | Terminal record | +| ------------------------------------------------------------------- | -------------------------------------------------------- | +| NDJSON (`application/x-ndjson`, `application/ndjson`) | A final line `{"$harperStreamError": { ... }}` | +| [Server-Sent Events](./server-sent-events.md) (`text/event-stream`) | An `event: harper-error` event whose `data` is `{ ... }` | + +The object has `error` (the error code or class name), `message`, and, when they apply, `status`, `replicationUnconfirmed: true`, and `retryable: false`. + ## Content Type via URL Extension As an alternative to the `Accept` header, responses can be requested in a specific format using file-style URL extensions: diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 31f4bdf00..b49755344 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -152,6 +152,14 @@ The job behind a replicated `restart_service` (and behind a `restart: "rolling"` MCP `initialize` is now limited per client identity, on both profiles, by the new `mcp..rateLimit.initializePerSecond` (default `10`) and `initializeBurst` (default `50`). Each session is a stored row kept for the idle timeout, and the application profile accepts anonymous `initialize`, so this limit is on by default; set `initializePerSecond: 0` to turn it off. A denied `initialize` gets HTTP `429` with a `Retry-After` header and a JSON-RPC error, and no session is created. Clients that open sessions in bursts, such as a test suite or a pool of agents behind one IP address, may now see that `429` after upgrading. `initialize` also stores only the names of the client capabilities Harper checks (`sampling`, `elicitation`, `roots`), not the capabilities object the client sent. See [`initializePerSecond`](/reference/v5/mcp/configuration#mcpprofileratelimitinitializepersecond) ([harper#3537](https://github.com/HarperFast/harper/pull/3537)). +## REST + +### A JSON Response That Fails Mid-Stream Is Cut Off + +[Cut off a JSON response that fails mid-stream](https://github.com/HarperFast/harper/pull/3664) changes how a streamed JSON array reports a failure after its `200` status is sent. Harper still writes an `{"error": ...}` element, but now leaves out the closing `]` and aborts the connection, instead of closing the array and ending the response normally. A partial result no longer parses as a complete one. Clients should treat a cut-off body, such as `response.json()` rejecting, as a failed read. NDJSON and Server-Sent Events responses are unchanged. A `json` `export_local` whose query fails now ends its job with `ERROR` and removes the partial file. + +A common trigger is a client that stops reading a large collection query for longer than `storage.maxTransactionOpenTime` (default 30 seconds), which on RocksDB fails the read with `ReadSnapshotExpiredError`. See [When a Streamed Response Fails](/reference/v5/rest/content-types#when-a-streamed-response-fails) and [`storage.maxTransactionOpenTime`](/reference/v5/database/storage-tuning#storagemaxtransactionopentime) ([harper#2392](https://github.com/HarperFast/harper/issues/2392)). + ## Configuration ### Applying V8 Flags From the Configuration From 701b893759f0bd0671b910a5a345324133c209b9 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 27 Oct 2026 13:13:58 -0600 Subject: [PATCH 2/5] Scope maxTransactionOpenTime to RocksDB and cover CSV/MessagePack failures LMDB keeps a fixed 30-second monitor and ignores the setting. Say so, note the option takes milliseconds rather than a duration string, say how CSV and MessagePack report a mid-stream failure, and list every job status. Dispatch-Task: docs-harper-3664 Co-Authored-By: Claude Opus 5.5 --- reference/configuration/options.md | 2 +- reference/database/jobs.md | 1 + reference/database/storage-tuning.md | 4 +++- reference/rest/content-types.md | 8 +++++--- 4 files changed, 10 insertions(+), 5 deletions(-) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index dd81bf5b5..2047fe821 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -335,7 +335,7 @@ storage: - `compactOnStart` — Compact all non-system databases on startup; _Default_: `false` (Added in: v4.3.0) - `compactOnStartKeepBackup` — Retain compaction backups; _Default_: `false` - `maxTransactionQueueTime` — Per-thread write-commit duration threshold for HTTP 503 backpressure, and the limit on how long a write keeps retrying conflicts with other writes (a separate 503, code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT`); see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionqueuetime); _Default_: `45s` -- `maxTransactionOpenTime` — Milliseconds a transaction may stay open without activity before Harper aborts its writes or releases its read snapshot; see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionopentime); _Default_: `30000` +- `maxTransactionOpenTime` — Milliseconds a transaction may stay open without activity before Harper aborts its writes or releases its read snapshot (RocksDB only; LMDB uses a fixed 30 seconds); see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionopentime); _Default_: `30000` - `noReadAhead` — Advise OS against read-ahead; _Default_: `false` - `prefetchWrites` — Prefetch before write transactions; _Default_: `true` - `randomAccessFields` — Encode records as typed random-access structures, so a single field can be read without decoding the whole record. Best for tables whose records have a stable set of fields with stable types; leave off for wide, sparse, or variably typed schemas. Read by each table when its store opens, so a change applies to existing tables on restart, and ignored entirely by tables declaring the [`@table(randomAccessFields:)`](../database/schema.md#randomaccessfields) directive; _Default_: `false`. See [Storage Tuning — Record Encoding](../database/storage-tuning.md#storagerandomaccessfields) diff --git a/reference/database/jobs.md b/reference/database/jobs.md index f8c28d3b7..97284cfcc 100644 --- a/reference/database/jobs.md +++ b/reference/database/jobs.md @@ -12,6 +12,7 @@ Harper uses an asynchronous job system for long-running data operations. When a Job status values: +- `CREATED` — the job is queued and has not started - `IN_PROGRESS` — the job is currently running - `COMPLETE` — the job finished successfully - `ERROR` — the job failed; its `message` holds the error diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index b1a8a30af..0e0eefb99 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -56,11 +56,13 @@ storage: ### `storage.maxTransactionOpenTime` + + Type: `number` (milliseconds) Default: `30000` -The maximum time a transaction may stay open without activity. Harper checks open transactions once per this interval, so a stalled transaction is closed between one and two intervals after its last activity. +The maximum time a transaction may stay open without activity. Give a number of milliseconds; unlike `maxTransactionQueueTime`, this option does not take a duration string such as `30s`. LMDB databases ignore it and use a fixed 30 seconds. Harper checks open transactions once per this interval, so a stalled transaction is closed between one and two intervals after its last activity. - A transaction with uncommitted writes from an application request is aborted, and the request fails with status `422`. Split long-running work into smaller transactions. - A read-only transaction has its read snapshot released. On RocksDB, a read that continues afterwards fails with `ReadSnapshotExpiredError` (status `503`). diff --git a/reference/rest/content-types.md b/reference/rest/content-types.md index 61982bd82..01406c90f 100644 --- a/reference/rest/content-types.md +++ b/reference/rest/content-types.md @@ -50,7 +50,7 @@ Comma-separated values format, suitable for data export and spreadsheet import/e ## When a Streamed Response Fails -A collection query, or any resource that returns an iterable, is streamed: Harper sends the `200` status and headers, then writes records as it reads them. If reading fails after that, the status can no longer change, so the failure is reported in the body. +A collection query, or any resource that returns an iterable, is streamed: Harper sends the `200` status and headers, then writes records as it reads them. If reading fails after that, the status can no longer change, so the failure shows up in the body. ### JSON @@ -70,9 +70,9 @@ A request that writes and then streams a result, such as a `POST` whose handler On RocksDB, a long streamed read can fail this way when the client stops reading for longer than [`storage.maxTransactionOpenTime`](../database/storage-tuning.md#storagemaxtransactionopentime). -### NDJSON and Server-Sent Events +### Other Formats -These formats end a failed stream with a terminal record and are unchanged in v5.4.0: +NDJSON and Server-Sent Events end a failed stream with a terminal record, unchanged in v5.4.0: | Format | Terminal record | | ------------------------------------------------------------------- | -------------------------------------------------------- | @@ -81,6 +81,8 @@ These formats end a failed stream with a terminal record and are unchanged in v5 The object has `error` (the error code or class name), `message`, and, when they apply, `status`, `replicationUnconfirmed: true`, and `retryable: false`. +CSV and MessagePack responses write no error record. Harper aborts the response, which can end at a record boundary and look like a shorter, valid result, so treat a network error while reading as a failed read. + ## Content Type via URL Extension As an alternative to the `Accept` header, responses can be requested in a specific format using file-style URL extensions: From 5c81aa2cbcbbf2fb63b849208a6b50a98e8e065b Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 27 Oct 2026 13:18:49 -0600 Subject: [PATCH 3/5] Limit the abort-without-record note to MessagePack A CSV stream does not forward its source error yet (harper#3658), so it does not abort. Dispatch-Task: docs-harper-3664 Co-Authored-By: Claude Opus 5.5 --- reference/rest/content-types.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/rest/content-types.md b/reference/rest/content-types.md index 01406c90f..e5629d53a 100644 --- a/reference/rest/content-types.md +++ b/reference/rest/content-types.md @@ -81,7 +81,7 @@ NDJSON and Server-Sent Events end a failed stream with a terminal record, unchan The object has `error` (the error code or class name), `message`, and, when they apply, `status`, `replicationUnconfirmed: true`, and `retryable: false`. -CSV and MessagePack responses write no error record. Harper aborts the response, which can end at a record boundary and look like a shorter, valid result, so treat a network error while reading as a failed read. +A MessagePack response writes no error record. Harper aborts it, and the body can end at a record boundary and look like a shorter, valid result, so treat a network error while reading as a failed read. ## Content Type via URL Extension From bf67aa49c565659699660b639a705d698fdf2971 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 28 Oct 2026 09:48:28 -0600 Subject: [PATCH 4/5] Say which streamed formats report a failure in the body; list CREATED in get_job Dispatch-Task: pr-maint-badb3af9cefd2ca3a3b418701a8045f1 Co-Authored-By: Claude Sonnet 5.5 --- reference/operations-api/operations.md | 2 +- reference/rest/content-types.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 66a8916d7..a947157b6 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -2321,7 +2321,7 @@ Detailed documentation: [Database Jobs](../database/jobs.md) ### `get_job` -Returns job status (`COMPLETE`, `IN_PROGRESS`, `ERROR`), timing, and result message for the specified job ID. Bulk import/export operations return a job ID on initiation, as does a `deploy_component` with `"restart": "rolling"`, as `restartJobId`. [`restart_service`](#restart_service) also returns one; with `"replicated": true`, its job's `status` and `result.replicated` report whether each node restarted, as [`replicated`](#restart_service-replicated) describes. +Returns job status (`CREATED`, `IN_PROGRESS`, `COMPLETE`, `ERROR`), timing, and result message for the specified job ID. Bulk import/export operations return a job ID on initiation, as does a `deploy_component` with `"restart": "rolling"`, as `restartJobId`. [`restart_service`](#restart_service) also returns one; with `"replicated": true`, its job's `status` and `result.replicated` report whether each node restarted, as [`replicated`](#restart_service-replicated) describes. ```json { "operation": "get_job", "id": "4a982782-929a-4507-8794-26dae1132def" } diff --git a/reference/rest/content-types.md b/reference/rest/content-types.md index e5629d53a..25ed64c4b 100644 --- a/reference/rest/content-types.md +++ b/reference/rest/content-types.md @@ -50,7 +50,7 @@ Comma-separated values format, suitable for data export and spreadsheet import/e ## When a Streamed Response Fails -A collection query, or any resource that returns an iterable, is streamed: Harper sends the `200` status and headers, then writes records as it reads them. If reading fails after that, the status can no longer change, so the failure shows up in the body. +A collection query, or any resource that returns an iterable, is streamed: Harper sends the `200` status and headers, then writes records as it reads them. If reading fails after that, the status can no longer change, so JSON, NDJSON and Server-Sent Events report the failure in the body. ### JSON From ae5cb3d3df59c1a1c22e075e848eb64f9b4cfdc6 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 30 Oct 2026 10:50:02 -0600 Subject: [PATCH 5/5] docs: clarify maxTransactionOpenTime renewal and update sibling JSON stream guidance Dispatch-Task: pr-maint-badb3af9cefd2ca3a3b418701a8045f1 Co-Authored-By: Gemini --- reference/configuration/options.md | 2 +- reference/database/storage-tuning.md | 8 ++++---- reference/full-text-search/overview.md | 2 +- reference/full-text-search/querying.md | 2 +- reference/resources/query-optimization.md | 2 +- reference/rest/querying.md | 2 +- 6 files changed, 9 insertions(+), 9 deletions(-) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 2047fe821..e0b221bc2 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -335,7 +335,7 @@ storage: - `compactOnStart` — Compact all non-system databases on startup; _Default_: `false` (Added in: v4.3.0) - `compactOnStartKeepBackup` — Retain compaction backups; _Default_: `false` - `maxTransactionQueueTime` — Per-thread write-commit duration threshold for HTTP 503 backpressure, and the limit on how long a write keeps retrying conflicts with other writes (a separate 503, code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT`); see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionqueuetime); _Default_: `45s` -- `maxTransactionOpenTime` — Milliseconds a transaction may stay open without activity before Harper aborts its writes or releases its read snapshot (RocksDB only; LMDB uses a fixed 30 seconds); see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionopentime); _Default_: `30000` +- `maxTransactionOpenTime` — Milliseconds a transaction may stay open without activity before Harper aborts its uncommitted writes (which only writes renew) or releases its read-only snapshot (which reads renew; RocksDB only, LMDB uses a fixed 30 seconds); see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionopentime); _Default_: `30000` - `noReadAhead` — Advise OS against read-ahead; _Default_: `false` - `prefetchWrites` — Prefetch before write transactions; _Default_: `true` - `randomAccessFields` — Encode records as typed random-access structures, so a single field can be read without decoding the whole record. Best for tables whose records have a stable set of fields with stable types; leave off for wide, sparse, or variably typed schemas. Read by each table when its store opens, so a change applies to existing tables on restart, and ignored entirely by tables declaring the [`@table(randomAccessFields:)`](../database/schema.md#randomaccessfields) directive; _Default_: `false`. See [Storage Tuning — Record Encoding](../database/storage-tuning.md#storagerandomaccessfields) diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index 0e0eefb99..10eee4666 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -56,16 +56,16 @@ storage: ### `storage.maxTransactionOpenTime` - + Type: `number` (milliseconds) Default: `30000` -The maximum time a transaction may stay open without activity. Give a number of milliseconds; unlike `maxTransactionQueueTime`, this option does not take a duration string such as `30s`. LMDB databases ignore it and use a fixed 30 seconds. Harper checks open transactions once per this interval, so a stalled transaction is closed between one and two intervals after its last activity. +The maximum time a transaction may stay open without activity. Give a number of milliseconds; unlike `maxTransactionQueueTime`, this option does not take a duration string such as `30s`. Available since v5.0.0; LMDB databases ignore it and use a fixed 30 seconds. Harper checks open transactions once per this interval, so a stalled transaction is closed between one and two intervals after its last qualifying activity: -- A transaction with uncommitted writes from an application request is aborted, and the request fails with status `422`. Split long-running work into smaller transactions. -- A read-only transaction has its read snapshot released. On RocksDB, a read that continues afterwards fails with `ReadSnapshotExpiredError` (status `503`). +- A transaction with uncommitted writes from an application request must keep writing; reads alone do not renew its timeout. When it expires, the writes are aborted and the request fails with status `422`. Split long-running work into smaller transactions. +- A read-only transaction is renewed by reads. When it expires, its read snapshot is released; on RocksDB, a read that continues afterwards fails with `ReadSnapshotExpiredError` (status `503`). A streamed response, such as a REST collection query, holds its read snapshot while the client reads. On RocksDB, if the client stops reading for longer than this limit, the read fails when the client resumes. The `200` status is already sent by then, so a JSON response is cut off rather than returned as a `503`; see [When a Streamed Response Fails](../rest/content-types.md#when-a-streamed-response-fails). Clients that read large results slowly should read promptly, page with [`limit(start,end)`](../rest/querying.md#limitend-or-limitstartend), or raise this limit. diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index 39d9df04b..1be46047b 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -70,7 +70,7 @@ Tantivy 0.27 changes English stemming even though the schema analyzer remains `e 1. Drain the node's full-text requests and stop all old Harper processes that share its storage, including background workers. Do not run old and new engine versions against one local index. 2. Confirm authoritative records and recovery backups are available. Reserve disk headroom and time for rebuilding every local full-text index, including temporarily retained retired files. Measure rebuild duration and disk growth with representative data before setting the rollout window. 3. Start the v5.4 Harper version. Use [`describe_table`](../operations-api/operations.md#describe_table) for each affected table, using an authorized operator account that already has read access to every indexed source field, and check every `full_text_indexes[].readiness.state`. Metadata is permission-filtered; a missing entry does not prove that no index exists. Keep full-text traffic drained until all required indexes are `ready`. -4. Run representative searches through the same REST or permission-checked application path and roles used in production, with `maxIndexLagMilliseconds: 0` and `waitForIndexMilliseconds: 30000` on each full-text condition. These wait for coverage of writes committed on the queried node before the search; they do not wait for replication delivery, and readiness alone does not establish that coverage. Check response bodies for terminal stream errors as well as HTTP status. A JSON-array response can start with HTTP 200 and end with an `error` element. +4. Run representative searches through the same REST or permission-checked application path and roles used in production, with `maxIndexLagMilliseconds: 0` and `waitForIndexMilliseconds: 30000` on each full-text condition. These wait for coverage of writes committed on the queried node before the search; they do not wait for replication delivery, and readiness alone does not establish that coverage. Check for stream or JSON-parse failures and terminal stream errors as well as HTTP status: from v5.4.0, a JSON-array response that fails after `200` leaves out the closing `]` and aborts the connection, and may end with a best-effort `error` element or with no terminal diagnostic if interrupted mid-record (see [When a Streamed Response Fails](../rest/content-types.md#when-a-streamed-response-fails)). 5. Retry write responses carrying `DERIVED_INDEX_LAGGING` with bounded backoff while replay catches up. Restore full-text traffic only after readiness and current-coverage searches succeed. A node-by-node upgrade can use other ready replicas only if the application's routing supports it; the rebuilding node remains unavailable for full-text search. During a mixed-version rollout, matching and ranking can differ between replicas. Pre-v5.4 replicas also reject prefix requests whose `offset + limit` exceeds 100 with `400`. Route related searches consistently and check the application's behavior before using this approach. See [freshness controls](./querying.md#freshness-controls) for the condition fields and streaming behavior. Qualify multi-node rollout, sustained-write capacity, relevance and performance against the application's workload before production rollout. A successful rebuild alone does not establish those properties. diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 4980eb120..acc945669 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -186,7 +186,7 @@ All full-text conditions combined into one query must use the same freshness val A non-waiting HTTP query can return `Harper-Index-Coverage` with the admitted state, lag upper bound, and requested tolerance. A zero-size page performs no native search and carries no coverage proof. A waiting `Table.search()` establishes coverage as its iterator is consumed. A `search_by_conditions` request completes only after its wait and search finish. Waiting queries do not emit a coverage header before completion. -If acceptable coverage is not reached before `waitForIndexMilliseconds` expires, Harper returns `503` with `code: "DERIVED_INDEX_LAGGING"` and `retryable: true`. For `Table.search()`, the error is raised while the async iterator is consumed. An HTTP response may already be streaming, so clients should also handle a terminal stream error rather than relying only on the initial status. +If acceptable coverage is not reached before `waitForIndexMilliseconds` expires, Harper returns `503` with `code: "DERIVED_INDEX_LAGGING"` and `retryable: true`. For `Table.search()`, the error is raised while the async iterator is consumed. An HTTP response may already be streaming, so clients should also handle a cut-off stream (which may end without a terminal diagnostic) or terminal stream error rather than relying only on the initial status; see [When a Streamed Response Fails](../rest/content-types.md#when-a-streamed-response-fails). The same code is also used when prolonged derived-index lag causes Harper to reject local writes. Query waits can be retried according to the caller's freshness needs; rejected writes should use backoff while the index catches up. See [Write backpressure](./overview.md#write-backpressure). diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index 3da6e188b..369de83e1 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -101,7 +101,7 @@ For example, with `offset: 20` and `limit: 10`, Harper skips the first 20 eligib Use a bounded `limit` for large result sets. An unbounded search that exceeds its native window, fetched-hit limit or retained-result limit throws 400; use a bounded query. Exhausting the execution-time budget instead throws 503. `offset + limit` must not exceed either `storage.fullText.maxResultsPerRequest` or `storage.fullText.maxRawHitsPerRequest`; replayed or rejected hits can exhaust the raw-work budget earlier. Raising only the raw-hit budget allows more filtering work without raising the retained-record target. Offset requests revisit earlier matches and can retain up to `offset + limit` accepted records. Raising work limits can increase CPU, reads and memory use. -The fetched-hit budget includes the initial batch replay when Harper switches to native traversal. It counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. A budget failure is a failed query, not a completed short page. If a REST response has already started streaming, a later query error ends that stream instead of changing its HTTP status. JSON-array clients must check for a terminal object shaped like `{ "error": "ErrorName: message" }` and treat the response as failed. For other formats, detect stream or decoding failures; status 200 and count headers alone do not prove completion. See [REST paging](../rest/querying.md#pagination-and-total-count). When there are no further native matches, Harper returns the eligible records found, which may be an empty result. This differs from reaching a budget while more matches remain. +The fetched-hit budget includes the initial batch replay when Harper switches to native traversal. It counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. A budget failure is a failed query, not a completed short page. If a REST response has already started streaming, a later query error cuts off a JSON-array stream and aborts the connection—with a best-effort `{ "error": "ErrorName: message" }` element or with no terminal diagnostic if interrupted mid-record—instead of changing its HTTP status, so JSON-array clients must treat a stream or JSON-parse failure, or a terminal `error` element, as a failed response; see [When a Streamed Response Fails](../rest/content-types.md#when-a-streamed-response-fails). For other formats, detect terminal error records or stream and decoding failures; status 200 and count headers alone do not prove completion. See [REST paging](../rest/querying.md#pagination-and-total-count). When there are no further native matches, Harper returns the eligible records found, which may be an empty result. This differs from reaching a budget while more matches remain. Configure the three request limits and six native process budgets under [`storage.fullText`](../configuration/options.md#storage). See [Set Configuration](../configuration/operations.md#set-configuration) for restart requirements. The execution budget is not a hard response-time guarantee: reader acquisition, query construction and some native operations can take additional time. Freshness waiting and reader acquisition use separate bounds. diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 72faec949..3904d04ac 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -214,7 +214,7 @@ Use `limit(start,end)` to page through a collection, and opt in to a total match Counting is opt-in: without the header, no count is computed and no count headers are returned. It applies only to `GET`/`HEAD` requests and requires a bounded page — the page size (`end - start`, or `end` in the one-argument form) must be a non-negative integer no larger than **10,000**, and the window end (`offset + page size`) no larger than **1,000,000**. A request with no `limit()`, one outside those bounds, or a non-numeric one is served normally with no count headers, since counting an unbounded page would defeat the point of paging. -Full-text and autocomplete queries use the same paging syntax. The page window counts eligible records after record filtering, rather than raw full-text hits. Harper fetches more ranked batches to fill that window when needed. A page can contain fewer records when no further matches remain; reaching a search work or time limit fails the query. A failure detected before response headers can set an error HTTP status. After streaming starts, the response can retain HTTP 200 and end with an error. For JSON-array responses, check for a terminal object shaped like `{ "error": "ErrorName: message" }` and treat the response as failed; do not render that element as a record. For other formats, detect stream or decoding failures as well. Neither status 200 nor count headers prove that a page completed successfully. Each page request runs a new query, so concurrent changes can cause repeats or omissions between pages. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for search work limits and recovery guidance. +Full-text and autocomplete queries use the same paging syntax. The page window counts eligible records after record filtering, rather than raw full-text hits. Harper fetches more ranked batches to fill that window when needed. A page can contain fewer records when no further matches remain; reaching a search work or time limit fails the query. A failure detected before response headers can set an error HTTP status. After streaming starts, the response retains HTTP 200 and a failed JSON-array stream is cut off and aborted—with a best-effort `{ "error": "ErrorName: message" }` element or with no terminal diagnostic if interrupted mid-record—so treat a stream or JSON-parse error, or a terminal `error` element, as a failed page rather than a record; see [When a Streamed Response Fails](./content-types.md#when-a-streamed-response-fails). For other formats, detect terminal error records or stream and decoding failures as well. Neither status 200 nor count headers prove that a page completed successfully. Each page request runs a new query, so concurrent changes can cause repeats or omissions between pages. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for search work limits and recovery guidance. The count-header guardrails do not override query-specific limits such as the full-text work budget.