Document cut-off JSON streams and storage.maxTransactionOpenTime

reference/configuration/options.md

@@ -366,6 +366,7 @@ storage:
 - `compactOnStart` — Compact all non-system databases on startup; _Default_: `false` (Added in: v4.3.0)
 - `compactOnStartKeepBackup` — Retain compaction backups; _Default_: `false`
 - `maxTransactionQueueTime` — Per-thread write-commit duration threshold for HTTP 503 backpressure, and the limit on how long a write keeps retrying conflicts with other writes (a separate 503, code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT`); see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionqueuetime); _Default_: `45s`
+- `maxTransactionOpenTime` — Milliseconds a transaction may stay open without activity before Harper aborts its uncommitted writes (which only writes renew) or releases its read-only snapshot (which reads renew; RocksDB only, LMDB uses a fixed 30 seconds); see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionopentime); _Default_: `30000`
 - `noReadAhead` — Advise OS against read-ahead; _Default_: `false`
 - `prefetchWrites` — Prefetch before write transactions; _Default_: `true`
 - `randomAccessFields` <VersionBadge version="v5.1.0" /> — Encode records as typed random-access structures, so a single field can be read without decoding the whole record. Best for tables whose records have a stable set of fields with stable types; leave off for wide, sparse, or variably typed schemas. Read by each table when its store opens, so a change applies to existing tables on restart, and ignored entirely by tables declaring the [`@table(randomAccessFields:)`](../database/schema.md#randomaccessfields) directive; _Default_: `false`. See [Storage Tuning — Record Encoding](../database/storage-tuning.md#storagerandomaccessfields)

reference/database/jobs.md

@@ -12,8 +12,10 @@ Harper uses an asynchronous job system for long-running data operations. When a
 
 Job status values:
 
+- `CREATED` — the job is queued and has not started
 - `IN_PROGRESS` — the job is currently running
 - `COMPLETE` — the job finished successfully
+- `ERROR` — the job failed; its `message` holds the error
 
 ## Bulk Operations
 
@@ -159,6 +161,8 @@ Exports table data to a local file in JSON or CSV format.
 
 - `filename` _(optional)_ — filename without extension; auto-generated from epoch timestamp if omitted
 
+<VersionBadge type="changed" version="v5.4.0" /> If the query fails partway through a `json` export, the job ends with status `ERROR` and the partial file is removed. Before v5.4.0, the job completed and the file could hold a complete-looking array ending in an `error` element.
+
 ```json
 {
 	"operation": "export_local",

reference/database/storage-tuning.md

@@ -71,6 +71,26 @@ storage:
   maxTransactionQueueTime: 30s
 ```
 
+### `storage.maxTransactionOpenTime`
+
+<EngineBadge engines="RocksDB" /> <VersionBadge version="v5.0.0" />
+
+Type: `number` (milliseconds)
+
+Default: `30000`
+
+The maximum time a transaction may stay open without activity. Give a number of milliseconds; unlike `maxTransactionQueueTime`, this option does not take a duration string such as `30s`. Available since v5.0.0; LMDB databases ignore it and use a fixed 30 seconds. Harper checks open transactions once per this interval, so a stalled transaction is closed between one and two intervals after its last qualifying activity:
+
+- A transaction with uncommitted writes from an application request must keep writing; reads alone do not renew its timeout. When it expires, the writes are aborted and the request fails with status `422`. Split long-running work into smaller transactions.
+- A read-only transaction is renewed by reads. When it expires, its read snapshot is released; on RocksDB, a read that continues afterwards fails with `ReadSnapshotExpiredError` (status `503`).
+
+A streamed response, such as a REST collection query, holds its read snapshot while the client reads. On RocksDB, if the client stops reading for longer than this limit, the read fails when the client resumes. <VersionBadge type="changed" version="v5.4.0" /> The `200` status is already sent by then, so a JSON response is cut off rather than returned as a `503`; see [When a Streamed Response Fails](../rest/content-types.md#when-a-streamed-response-fails). Clients that read large results slowly should read promptly, page with [`limit(start,end)`](../rest/querying.md#limitend-or-limitstartend), or raise this limit.
+
+```yaml
+storage:
+  maxTransactionOpenTime: 60000
+```
+
 ## Compression
 
 ### `storage.compression`

reference/full-text-search/overview.md

@@ -70,7 +70,7 @@ Tantivy 0.27 changes English stemming even though the schema analyzer remains `e
 1. Drain the node's full-text requests and stop all old Harper processes that share its storage, including background workers. Do not run old and new engine versions against one local index.
 2. Confirm authoritative records and recovery backups are available. Reserve disk headroom and time for rebuilding every local full-text index, including temporarily retained retired files. Measure rebuild duration and disk growth with representative data before setting the rollout window.
 3. Start the v5.4 Harper version. Use [`describe_table`](../operations-api/operations.md#describe_table) for each affected table, using an authorized operator account that already has read access to every indexed source field, and check every `full_text_indexes[].readiness.state`. Metadata is permission-filtered; a missing entry does not prove that no index exists. Keep full-text traffic drained until all required indexes are `ready`.
-4. Run representative searches through the same REST or permission-checked application path and roles used in production, with `maxIndexLagMilliseconds: 0` and `waitForIndexMilliseconds: 30000` on each full-text condition. These wait for coverage of writes committed on the queried node before the search; they do not wait for replication delivery, and readiness alone does not establish that coverage. Check response bodies for terminal stream errors as well as HTTP status. A JSON-array response can start with HTTP 200 and end with an `error` element.
+4. Run representative searches through the same REST or permission-checked application path and roles used in production, with `maxIndexLagMilliseconds: 0` and `waitForIndexMilliseconds: 30000` on each full-text condition. These wait for coverage of writes committed on the queried node before the search; they do not wait for replication delivery, and readiness alone does not establish that coverage. Check for stream or JSON-parse failures and terminal stream errors as well as HTTP status: from v5.4.0, a JSON-array response that fails after `200` leaves out the closing `]` and aborts the connection, and may end with a best-effort `error` element or with no terminal diagnostic if interrupted mid-record (see [When a Streamed Response Fails](../rest/content-types.md#when-a-streamed-response-fails)).
 5. Retry write responses carrying `DERIVED_INDEX_LAGGING` with bounded backoff while replay catches up. Restore full-text traffic only after readiness and current-coverage searches succeed. A node-by-node upgrade can use other ready replicas only if the application's routing supports it; the rebuilding node remains unavailable for full-text search. During a mixed-version rollout, matching and ranking can differ between replicas. Pre-v5.4 replicas also reject prefix requests whose `offset + limit` exceeds 100 with `400`. Route related searches consistently and check the application's behavior before using this approach.
 
 See [freshness controls](./querying.md#freshness-controls) for the condition fields and streaming behavior. Qualify multi-node rollout, sustained-write capacity, relevance and performance against the application's workload before production rollout. A successful rebuild alone does not establish those properties.

reference/full-text-search/querying.md

@@ -186,7 +186,7 @@ All full-text conditions combined into one query must use the same freshness val
 
 A non-waiting HTTP query can return `Harper-Index-Coverage` with the admitted state, lag upper bound, and requested tolerance. A zero-size page performs no native search and carries no coverage proof. A waiting `Table.search()` establishes coverage as its iterator is consumed. A `search_by_conditions` request completes only after its wait and search finish. Waiting queries do not emit a coverage header before completion.
 
-If acceptable coverage is not reached before `waitForIndexMilliseconds` expires, Harper returns `503` with `code: "DERIVED_INDEX_LAGGING"` and `retryable: true`. For `Table.search()`, the error is raised while the async iterator is consumed. An HTTP response may already be streaming, so clients should also handle a terminal stream error rather than relying only on the initial status.
+If acceptable coverage is not reached before `waitForIndexMilliseconds` expires, Harper returns `503` with `code: "DERIVED_INDEX_LAGGING"` and `retryable: true`. For `Table.search()`, the error is raised while the async iterator is consumed. An HTTP response may already be streaming, so clients should also handle a cut-off stream (which may end without a terminal diagnostic) or terminal stream error rather than relying only on the initial status; see [When a Streamed Response Fails](../rest/content-types.md#when-a-streamed-response-fails).
 
 The same code is also used when prolonged derived-index lag causes Harper to reject local writes. Query waits can be retried according to the caller's freshness needs; rejected writes should use backoff while the index catches up. See [Write backpressure](./overview.md#write-backpressure).
 

reference/operations-api/operations.md

@@ -343,6 +343,8 @@ All bulk import/export operations are asynchronous and return a job ID. Use [`ge
 
 Exports query results to a local path on the server. Formats: `json` or `csv`.
 
+<VersionBadge type="changed" version="v5.4.0" /> If the query fails partway through a `json` export, the job ends with status `ERROR` and the partial file is removed. Before v5.4.0, the job completed and the file could hold a complete-looking array ending in an `error` element.
+
 ```json
 {
 	"operation": "export_local",
@@ -2325,7 +2327,7 @@ Detailed documentation: [Database Jobs](../database/jobs.md)
 
 ### `get_job`
 
-Returns job status (`COMPLETE`, `IN_PROGRESS`, `ERROR`), timing, and result message for the specified job ID. Bulk import/export operations return a job ID on initiation, as does a `deploy_component` with `"restart": "rolling"`, as `restartJobId`. [`restart_service`](#restart_service) also returns one; with `"replicated": true`, its job's `status` and `result.replicated` report whether each node restarted, as [`replicated`](#restart_service-replicated) describes.
+Returns job status (`CREATED`, `IN_PROGRESS`, `COMPLETE`, `ERROR`), timing, and result message for the specified job ID. Bulk import/export operations return a job ID on initiation, as does a `deploy_component` with `"restart": "rolling"`, as `restartJobId`. [`restart_service`](#restart_service) also returns one; with `"replicated": true`, its job's `status` and `result.replicated` report whether each node restarted, as [`replicated`](#restart_service-replicated) describes.
 
 ```json
 { "operation": "get_job", "id": "4a982782-929a-4507-8794-26dae1132def" }

reference/resources/query-optimization.md

@@ -101,7 +101,7 @@ For example, with `offset: 20` and `limit: 10`, Harper skips the first 20 eligib
 
 Use a bounded `limit` for large result sets. An unbounded search that exceeds its native window, fetched-hit limit or retained-result limit throws 400; use a bounded query. Exhausting the execution-time budget instead throws 503. `offset + limit` must not exceed either `storage.fullText.maxResultsPerRequest` or `storage.fullText.maxRawHitsPerRequest`; replayed or rejected hits can exhaust the raw-work budget earlier. Raising only the raw-hit budget allows more filtering work without raising the retained-record target. Offset requests revisit earlier matches and can retain up to `offset + limit` accepted records. Raising work limits can increase CPU, reads and memory use.
 
-The fetched-hit budget includes the initial batch replay when Harper switches to native traversal. It counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. A budget failure is a failed query, not a completed short page. If a REST response has already started streaming, a later query error ends that stream instead of changing its HTTP status. JSON-array clients must check for a terminal object shaped like `{ "error": "ErrorName: message" }` and treat the response as failed. For other formats, detect stream or decoding failures; status 200 and count headers alone do not prove completion. See [REST paging](../rest/querying.md#pagination-and-total-count). When there are no further native matches, Harper returns the eligible records found, which may be an empty result. This differs from reaching a budget while more matches remain.
+The fetched-hit budget includes the initial batch replay when Harper switches to native traversal. It counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. A budget failure is a failed query, not a completed short page. If a REST response has already started streaming, a later query error cuts off a JSON-array stream and aborts the connection—with a best-effort `{ "error": "ErrorName: message" }` element or with no terminal diagnostic if interrupted mid-record—instead of changing its HTTP status, so JSON-array clients must treat a stream or JSON-parse failure, or a terminal `error` element, as a failed response; see [When a Streamed Response Fails](../rest/content-types.md#when-a-streamed-response-fails). For other formats, detect terminal error records or stream and decoding failures; status 200 and count headers alone do not prove completion. See [REST paging](../rest/querying.md#pagination-and-total-count). When there are no further native matches, Harper returns the eligible records found, which may be an empty result. This differs from reaching a budget while more matches remain.
 
 Configure the three request limits and six native process budgets under [`storage.fullText`](../configuration/options.md#storage). See [Set Configuration](../configuration/operations.md#set-configuration) for restart requirements. The execution budget is not a hard response-time guarantee: reader acquisition, query construction and some native operations can take additional time. Freshness waiting and reader acquisition use separate bounds.
 

reference/rest/content-types.md

@@ -48,6 +48,41 @@ Comma-separated values format, suitable for data export and spreadsheet import/e
 
 **When to use**: Ad-hoc data export, spreadsheet workflows, batch data processing. Not recommended for frequent or production API use.
 
+## When a Streamed Response Fails
+
+A collection query, or any resource that returns an iterable, is streamed: Harper sends the `200` status and headers, then writes records as it reads them. If reading fails after that, the status can no longer change, so JSON, NDJSON and Server-Sent Events report the failure in the body.
+
+### JSON
+
+<VersionBadge type="changed" version="v5.4.0" />
+
+When a streamed JSON array fails partway through, Harper writes an `{"error": "<Name>: <message>"}` element, leaves out the closing `]`, and aborts the response. On HTTP/1.1 the connection closes without the final chunk; on HTTP/2 the stream is reset.
+
+```
+[{"id":1},{"id":2},{"error":"ReadSnapshotExpiredError: Read scan snapshot expired; ..."}
+```
+
+Treat a cut-off body as a failed read, even though the status was `200`. A network error while reading the body, or `response.json()` rejecting, means the result is incomplete. The `error` element is a best-effort diagnostic: if the client is not reading when the failure happens, it can be lost along with unread records. The abort always happens.
+
+Before v5.4.0, Harper closed the array after the `error` element and ended the response normally, so a partial result parsed as a complete array.
+
+A request that writes and then streams a result, such as a `POST` whose handler returns query results, has already committed its writes when the stream fails. Retrying it can apply them again. The `error` element includes `replicationUnconfirmed: true` when the writes committed locally but their replication was not confirmed, and `retryable: false` when the request should not be retried.
+
+On RocksDB, a long streamed read can fail this way when the client stops reading for longer than [`storage.maxTransactionOpenTime`](../database/storage-tuning.md#storagemaxtransactionopentime).
+
+### Other Formats
+
+NDJSON and Server-Sent Events end a failed stream with a terminal record, unchanged in v5.4.0:
+
+| Format                                                              | Terminal record                                          |
+| ------------------------------------------------------------------- | -------------------------------------------------------- |
+| NDJSON (`application/x-ndjson`, `application/ndjson`)               | A final line `{"$harperStreamError": { ... }}`           |
+| [Server-Sent Events](./server-sent-events.md) (`text/event-stream`) | An `event: harper-error` event whose `data` is `{ ... }` |
+
+The object has `error` (the error code or class name), `message`, and, when they apply, `status`, `replicationUnconfirmed: true`, and `retryable: false`.
+
+A MessagePack response writes no error record. Harper aborts it, and the body can end at a record boundary and look like a shorter, valid result, so treat a network error while reading as a failed read.
+
 ## Content Type via URL Extension
 
 As an alternative to the `Accept` header, responses can be requested in a specific format using file-style URL extensions: