diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 8797a83e5..3fdf13368 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -366,6 +366,7 @@ storage: - `compactOnStart` — Compact all non-system databases on startup; _Default_: `false` (Added in: v4.3.0) - `compactOnStartKeepBackup` — Retain compaction backups; _Default_: `false` - `maxTransactionQueueTime` — Per-thread write-commit duration threshold for HTTP 503 backpressure, and the limit on how long a write keeps retrying conflicts with other writes (a separate 503, code `TRANSACTION_COMMIT_CONFLICT_TIMEOUT`); see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionqueuetime); _Default_: `45s` +- `maxTransactionOpenTime` — Milliseconds a transaction may stay open without activity before Harper aborts its uncommitted writes (which only writes renew) or releases its read-only snapshot (which reads renew; RocksDB only, LMDB uses a fixed 30 seconds); see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionopentime); _Default_: `30000` - `noReadAhead` — Advise OS against read-ahead; _Default_: `false` - `prefetchWrites` — Prefetch before write transactions; _Default_: `true` - `randomAccessFields` — Encode records as typed random-access structures, so a single field can be read without decoding the whole record. Best for tables whose records have a stable set of fields with stable types; leave off for wide, sparse, or variably typed schemas. Read by each table when its store opens, so a change applies to existing tables on restart, and ignored entirely by tables declaring the [`@table(randomAccessFields:)`](../database/schema.md#randomaccessfields) directive; _Default_: `false`. See [Storage Tuning — Record Encoding](../database/storage-tuning.md#storagerandomaccessfields) diff --git a/reference/database/jobs.md b/reference/database/jobs.md index cb5a57af0..97284cfcc 100644 --- a/reference/database/jobs.md +++ b/reference/database/jobs.md @@ -12,8 +12,10 @@ Harper uses an asynchronous job system for long-running data operations. When a Job status values: +- `CREATED` — the job is queued and has not started - `IN_PROGRESS` — the job is currently running - `COMPLETE` — the job finished successfully +- `ERROR` — the job failed; its `message` holds the error ## Bulk Operations @@ -159,6 +161,8 @@ Exports table data to a local file in JSON or CSV format. - `filename` _(optional)_ — filename without extension; auto-generated from epoch timestamp if omitted + If the query fails partway through a `json` export, the job ends with status `ERROR` and the partial file is removed. Before v5.4.0, the job completed and the file could hold a complete-looking array ending in an `error` element. + ```json { "operation": "export_local", diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index e2c3bb5ca..6cae3ab7f 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -71,6 +71,26 @@ storage: maxTransactionQueueTime: 30s ``` +### `storage.maxTransactionOpenTime` + + + +Type: `number` (milliseconds) + +Default: `30000` + +The maximum time a transaction may stay open without activity. Give a number of milliseconds; unlike `maxTransactionQueueTime`, this option does not take a duration string such as `30s`. Available since v5.0.0; LMDB databases ignore it and use a fixed 30 seconds. Harper checks open transactions once per this interval, so a stalled transaction is closed between one and two intervals after its last qualifying activity: + +- A transaction with uncommitted writes from an application request must keep writing; reads alone do not renew its timeout. When it expires, the writes are aborted and the request fails with status `422`. Split long-running work into smaller transactions. +- A read-only transaction is renewed by reads. When it expires, its read snapshot is released; on RocksDB, a read that continues afterwards fails with `ReadSnapshotExpiredError` (status `503`). + +A streamed response, such as a REST collection query, holds its read snapshot while the client reads. On RocksDB, if the client stops reading for longer than this limit, the read fails when the client resumes. The `200` status is already sent by then, so a JSON response is cut off rather than returned as a `503`; see [When a Streamed Response Fails](../rest/content-types.md#when-a-streamed-response-fails). Clients that read large results slowly should read promptly, page with [`limit(start,end)`](../rest/querying.md#limitend-or-limitstartend), or raise this limit. + +```yaml +storage: + maxTransactionOpenTime: 60000 +``` + ## Compression ### `storage.compression` diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index 39d9df04b..1be46047b 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -70,7 +70,7 @@ Tantivy 0.27 changes English stemming even though the schema analyzer remains `e 1. Drain the node's full-text requests and stop all old Harper processes that share its storage, including background workers. Do not run old and new engine versions against one local index. 2. Confirm authoritative records and recovery backups are available. Reserve disk headroom and time for rebuilding every local full-text index, including temporarily retained retired files. Measure rebuild duration and disk growth with representative data before setting the rollout window. 3. Start the v5.4 Harper version. Use [`describe_table`](../operations-api/operations.md#describe_table) for each affected table, using an authorized operator account that already has read access to every indexed source field, and check every `full_text_indexes[].readiness.state`. Metadata is permission-filtered; a missing entry does not prove that no index exists. Keep full-text traffic drained until all required indexes are `ready`. -4. Run representative searches through the same REST or permission-checked application path and roles used in production, with `maxIndexLagMilliseconds: 0` and `waitForIndexMilliseconds: 30000` on each full-text condition. These wait for coverage of writes committed on the queried node before the search; they do not wait for replication delivery, and readiness alone does not establish that coverage. Check response bodies for terminal stream errors as well as HTTP status. A JSON-array response can start with HTTP 200 and end with an `error` element. +4. Run representative searches through the same REST or permission-checked application path and roles used in production, with `maxIndexLagMilliseconds: 0` and `waitForIndexMilliseconds: 30000` on each full-text condition. These wait for coverage of writes committed on the queried node before the search; they do not wait for replication delivery, and readiness alone does not establish that coverage. Check for stream or JSON-parse failures and terminal stream errors as well as HTTP status: from v5.4.0, a JSON-array response that fails after `200` leaves out the closing `]` and aborts the connection, and may end with a best-effort `error` element or with no terminal diagnostic if interrupted mid-record (see [When a Streamed Response Fails](../rest/content-types.md#when-a-streamed-response-fails)). 5. Retry write responses carrying `DERIVED_INDEX_LAGGING` with bounded backoff while replay catches up. Restore full-text traffic only after readiness and current-coverage searches succeed. A node-by-node upgrade can use other ready replicas only if the application's routing supports it; the rebuilding node remains unavailable for full-text search. During a mixed-version rollout, matching and ranking can differ between replicas. Pre-v5.4 replicas also reject prefix requests whose `offset + limit` exceeds 100 with `400`. Route related searches consistently and check the application's behavior before using this approach. See [freshness controls](./querying.md#freshness-controls) for the condition fields and streaming behavior. Qualify multi-node rollout, sustained-write capacity, relevance and performance against the application's workload before production rollout. A successful rebuild alone does not establish those properties. diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 4980eb120..acc945669 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -186,7 +186,7 @@ All full-text conditions combined into one query must use the same freshness val A non-waiting HTTP query can return `Harper-Index-Coverage` with the admitted state, lag upper bound, and requested tolerance. A zero-size page performs no native search and carries no coverage proof. A waiting `Table.search()` establishes coverage as its iterator is consumed. A `search_by_conditions` request completes only after its wait and search finish. Waiting queries do not emit a coverage header before completion. -If acceptable coverage is not reached before `waitForIndexMilliseconds` expires, Harper returns `503` with `code: "DERIVED_INDEX_LAGGING"` and `retryable: true`. For `Table.search()`, the error is raised while the async iterator is consumed. An HTTP response may already be streaming, so clients should also handle a terminal stream error rather than relying only on the initial status. +If acceptable coverage is not reached before `waitForIndexMilliseconds` expires, Harper returns `503` with `code: "DERIVED_INDEX_LAGGING"` and `retryable: true`. For `Table.search()`, the error is raised while the async iterator is consumed. An HTTP response may already be streaming, so clients should also handle a cut-off stream (which may end without a terminal diagnostic) or terminal stream error rather than relying only on the initial status; see [When a Streamed Response Fails](../rest/content-types.md#when-a-streamed-response-fails). The same code is also used when prolonged derived-index lag causes Harper to reject local writes. Query waits can be retried according to the caller's freshness needs; rejected writes should use backoff while the index catches up. See [Write backpressure](./overview.md#write-backpressure). diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 46b66537d..1ff154085 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -343,6 +343,8 @@ All bulk import/export operations are asynchronous and return a job ID. Use [`ge Exports query results to a local path on the server. Formats: `json` or `csv`. + If the query fails partway through a `json` export, the job ends with status `ERROR` and the partial file is removed. Before v5.4.0, the job completed and the file could hold a complete-looking array ending in an `error` element. + ```json { "operation": "export_local", @@ -2325,7 +2327,7 @@ Detailed documentation: [Database Jobs](../database/jobs.md) ### `get_job` -Returns job status (`COMPLETE`, `IN_PROGRESS`, `ERROR`), timing, and result message for the specified job ID. Bulk import/export operations return a job ID on initiation, as does a `deploy_component` with `"restart": "rolling"`, as `restartJobId`. [`restart_service`](#restart_service) also returns one; with `"replicated": true`, its job's `status` and `result.replicated` report whether each node restarted, as [`replicated`](#restart_service-replicated) describes. +Returns job status (`CREATED`, `IN_PROGRESS`, `COMPLETE`, `ERROR`), timing, and result message for the specified job ID. Bulk import/export operations return a job ID on initiation, as does a `deploy_component` with `"restart": "rolling"`, as `restartJobId`. [`restart_service`](#restart_service) also returns one; with `"replicated": true`, its job's `status` and `result.replicated` report whether each node restarted, as [`replicated`](#restart_service-replicated) describes. ```json { "operation": "get_job", "id": "4a982782-929a-4507-8794-26dae1132def" } diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index 3da6e188b..369de83e1 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -101,7 +101,7 @@ For example, with `offset: 20` and `limit: 10`, Harper skips the first 20 eligib Use a bounded `limit` for large result sets. An unbounded search that exceeds its native window, fetched-hit limit or retained-result limit throws 400; use a bounded query. Exhausting the execution-time budget instead throws 503. `offset + limit` must not exceed either `storage.fullText.maxResultsPerRequest` or `storage.fullText.maxRawHitsPerRequest`; replayed or rejected hits can exhaust the raw-work budget earlier. Raising only the raw-hit budget allows more filtering work without raising the retained-record target. Offset requests revisit earlier matches and can retain up to `offset + limit` accepted records. Raising work limits can increase CPU, reads and memory use. -The fetched-hit budget includes the initial batch replay when Harper switches to native traversal. It counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. A budget failure is a failed query, not a completed short page. If a REST response has already started streaming, a later query error ends that stream instead of changing its HTTP status. JSON-array clients must check for a terminal object shaped like `{ "error": "ErrorName: message" }` and treat the response as failed. For other formats, detect stream or decoding failures; status 200 and count headers alone do not prove completion. See [REST paging](../rest/querying.md#pagination-and-total-count). When there are no further native matches, Harper returns the eligible records found, which may be an empty result. This differs from reaching a budget while more matches remain. +The fetched-hit budget includes the initial batch replay when Harper switches to native traversal. It counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. A budget failure is a failed query, not a completed short page. If a REST response has already started streaming, a later query error cuts off a JSON-array stream and aborts the connection—with a best-effort `{ "error": "ErrorName: message" }` element or with no terminal diagnostic if interrupted mid-record—instead of changing its HTTP status, so JSON-array clients must treat a stream or JSON-parse failure, or a terminal `error` element, as a failed response; see [When a Streamed Response Fails](../rest/content-types.md#when-a-streamed-response-fails). For other formats, detect terminal error records or stream and decoding failures; status 200 and count headers alone do not prove completion. See [REST paging](../rest/querying.md#pagination-and-total-count). When there are no further native matches, Harper returns the eligible records found, which may be an empty result. This differs from reaching a budget while more matches remain. Configure the three request limits and six native process budgets under [`storage.fullText`](../configuration/options.md#storage). See [Set Configuration](../configuration/operations.md#set-configuration) for restart requirements. The execution budget is not a hard response-time guarantee: reader acquisition, query construction and some native operations can take additional time. Freshness waiting and reader acquisition use separate bounds. diff --git a/reference/rest/content-types.md b/reference/rest/content-types.md index 236384725..25ed64c4b 100644 --- a/reference/rest/content-types.md +++ b/reference/rest/content-types.md @@ -48,6 +48,41 @@ Comma-separated values format, suitable for data export and spreadsheet import/e **When to use**: Ad-hoc data export, spreadsheet workflows, batch data processing. Not recommended for frequent or production API use. +## When a Streamed Response Fails + +A collection query, or any resource that returns an iterable, is streamed: Harper sends the `200` status and headers, then writes records as it reads them. If reading fails after that, the status can no longer change, so JSON, NDJSON and Server-Sent Events report the failure in the body. + +### JSON + + + +When a streamed JSON array fails partway through, Harper writes an `{"error": ": "}` element, leaves out the closing `]`, and aborts the response. On HTTP/1.1 the connection closes without the final chunk; on HTTP/2 the stream is reset. + +``` +[{"id":1},{"id":2},{"error":"ReadSnapshotExpiredError: Read scan snapshot expired; ..."} +``` + +Treat a cut-off body as a failed read, even though the status was `200`. A network error while reading the body, or `response.json()` rejecting, means the result is incomplete. The `error` element is a best-effort diagnostic: if the client is not reading when the failure happens, it can be lost along with unread records. The abort always happens. + +Before v5.4.0, Harper closed the array after the `error` element and ended the response normally, so a partial result parsed as a complete array. + +A request that writes and then streams a result, such as a `POST` whose handler returns query results, has already committed its writes when the stream fails. Retrying it can apply them again. The `error` element includes `replicationUnconfirmed: true` when the writes committed locally but their replication was not confirmed, and `retryable: false` when the request should not be retried. + +On RocksDB, a long streamed read can fail this way when the client stops reading for longer than [`storage.maxTransactionOpenTime`](../database/storage-tuning.md#storagemaxtransactionopentime). + +### Other Formats + +NDJSON and Server-Sent Events end a failed stream with a terminal record, unchanged in v5.4.0: + +| Format | Terminal record | +| ------------------------------------------------------------------- | -------------------------------------------------------- | +| NDJSON (`application/x-ndjson`, `application/ndjson`) | A final line `{"$harperStreamError": { ... }}` | +| [Server-Sent Events](./server-sent-events.md) (`text/event-stream`) | An `event: harper-error` event whose `data` is `{ ... }` | + +The object has `error` (the error code or class name), `message`, and, when they apply, `status`, `replicationUnconfirmed: true`, and `retryable: false`. + +A MessagePack response writes no error record. Harper aborts it, and the body can end at a record boundary and look like a shorter, valid result, so treat a network error while reading as a failed read. + ## Content Type via URL Extension As an alternative to the `Accept` header, responses can be requested in a specific format using file-style URL extensions: diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 72faec949..3904d04ac 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -214,7 +214,7 @@ Use `limit(start,end)` to page through a collection, and opt in to a total match Counting is opt-in: without the header, no count is computed and no count headers are returned. It applies only to `GET`/`HEAD` requests and requires a bounded page — the page size (`end - start`, or `end` in the one-argument form) must be a non-negative integer no larger than **10,000**, and the window end (`offset + page size`) no larger than **1,000,000**. A request with no `limit()`, one outside those bounds, or a non-numeric one is served normally with no count headers, since counting an unbounded page would defeat the point of paging. -Full-text and autocomplete queries use the same paging syntax. The page window counts eligible records after record filtering, rather than raw full-text hits. Harper fetches more ranked batches to fill that window when needed. A page can contain fewer records when no further matches remain; reaching a search work or time limit fails the query. A failure detected before response headers can set an error HTTP status. After streaming starts, the response can retain HTTP 200 and end with an error. For JSON-array responses, check for a terminal object shaped like `{ "error": "ErrorName: message" }` and treat the response as failed; do not render that element as a record. For other formats, detect stream or decoding failures as well. Neither status 200 nor count headers prove that a page completed successfully. Each page request runs a new query, so concurrent changes can cause repeats or omissions between pages. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for search work limits and recovery guidance. +Full-text and autocomplete queries use the same paging syntax. The page window counts eligible records after record filtering, rather than raw full-text hits. Harper fetches more ranked batches to fill that window when needed. A page can contain fewer records when no further matches remain; reaching a search work or time limit fails the query. A failure detected before response headers can set an error HTTP status. After streaming starts, the response retains HTTP 200 and a failed JSON-array stream is cut off and aborted—with a best-effort `{ "error": "ErrorName: message" }` element or with no terminal diagnostic if interrupted mid-record—so treat a stream or JSON-parse error, or a terminal `error` element, as a failed page rather than a record; see [When a Streamed Response Fails](./content-types.md#when-a-streamed-response-fails). For other formats, detect terminal error records or stream and decoding failures as well. Neither status 200 nor count headers prove that a page completed successfully. Each page request runs a new query, so concurrent changes can cause repeats or omissions between pages. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for search work limits and recovery guidance. The count-header guardrails do not override query-specific limits such as the full-text work budget. diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index dbf3b8eeb..6c0faa041 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -179,6 +179,14 @@ MCP `initialize` is now limited per client identity, on both profiles, by the ne The [Data Loader](/reference/v5/database/data-loader#change-detection) now matches and tracks records by the table's primary key. On a table with a custom `@primaryKey`, user edits to loaded records are kept and data file changes apply; versions before v5.4.0 looked up records by their `id` field, rewriting any record whose `id` field did not match its primary key on each load. A data file that names a `database` now loads only into that database, creating the table there if needed, instead of into a same-named table in the default `data` database. Reordering keys in a nested object no longer counts as a content change. After upgrading, some records loaded by an earlier version stop receiving data file updates; see [Upgrading from versions before 5.4.0](/reference/v5/database/data-loader#change-detection) ([harper#3582](https://github.com/HarperFast/harper/pull/3582)). +## REST + +### A JSON Response That Fails Mid-Stream Is Cut Off + +[Cut off a JSON response that fails mid-stream](https://github.com/HarperFast/harper/pull/3664) changes how a streamed JSON array reports a failure after its `200` status is sent. Harper still writes an `{"error": ...}` element, but now leaves out the closing `]` and aborts the connection, instead of closing the array and ending the response normally. A partial result no longer parses as a complete one. Clients should treat a cut-off body, such as `response.json()` rejecting, as a failed read. NDJSON and Server-Sent Events responses are unchanged. A `json` `export_local` whose query fails now ends its job with `ERROR` and removes the partial file. + +A common trigger is a client that stops reading a large collection query for longer than `storage.maxTransactionOpenTime` (default 30 seconds), which on RocksDB fails the read with `ReadSnapshotExpiredError`. See [When a Streamed Response Fails](/reference/v5/rest/content-types#when-a-streamed-response-fails) and [`storage.maxTransactionOpenTime`](/reference/v5/database/storage-tuning#storagemaxtransactionopentime) ([harper#2392](https://github.com/HarperFast/harper/issues/2392)). + ## Configuration ### Graceful Shutdown on SIGTERM, SIGINT and SIGQUIT