From 1782051d6fffb4d49bbf825500f9cc959a4ab548 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Sun, 27 Sep 2026 16:59:21 -0600 Subject: [PATCH 01/78] Document native full-text search --- .../full-text-search-product-catalog.mdx | 124 ++++++++++ reference/database/schema.md | 19 ++ reference/full-text-search/configuration.md | 157 +++++++++++++ reference/full-text-search/overview.md | 61 +++++ reference/full-text-search/querying.md | 217 ++++++++++++++++++ reference/operations-api/operations.md | 2 + reference/resources/query-optimization.md | 8 + reference/resources/resource-api.md | 16 +- reference/rest/headers.md | 2 + reference/rest/querying.md | 14 ++ release-notes/v5-lincoln/5.3.md | 6 + sidebarsLearn.ts | 5 + sidebarsReference.ts | 23 ++ 13 files changed, 647 insertions(+), 7 deletions(-) create mode 100644 learn/developers/full-text-search-product-catalog.mdx create mode 100644 reference/full-text-search/configuration.md create mode 100644 reference/full-text-search/overview.md create mode 100644 reference/full-text-search/querying.md diff --git a/learn/developers/full-text-search-product-catalog.mdx b/learn/developers/full-text-search-product-catalog.mdx new file mode 100644 index 000000000..e99ff6dc2 --- /dev/null +++ b/learn/developers/full-text-search-product-catalog.mdx @@ -0,0 +1,124 @@ +--- +title: Add Full-Text Search to a Product Catalog +description: Configure and query a BM25-ranked product search in Harper. +--- + +# Add Full-Text Search to a Product Catalog + +This guide adds relevance-ranked text search to a product catalog, then exposes it through Harper's JavaScript and REST APIs. + +## 1. Declare the index + +Add `audit: true` to the RocksDB-backed table and declare an `@fullText` index over the fields customers search. + +```graphql title="schema.graphql" +type Product + @table(database: "catalog", audit: true) + @export + @fullText( + name: "catalogSearch" + fields: [ + { name: "name", weight: 3, highlight: true } + { name: "description", weight: 1, highlight: true } + { name: "tags" } + ] + highlighting: { maxFragments: 2, fragmentLength: 120 } + ) { + id: ID @primaryKey + name: String + description: String + tags: [String] + category: String @indexed + price: Float @indexed +} +``` + +`weight: 3` makes a match in `name` contribute more to BM25 relevance than the same match in `description` or `tags`. Highlighting is enabled only for the fields shown to customers. + +## 2. Add records + +Use the generated REST endpoint: + +```http +POST /Product/ +Content-Type: application/json + +{ + "id": "shoe-1", + "name": "Waterproof trail running shoe", + "description": "Lightweight shoe for wet mountain trails", + "tags": ["outdoor", "trail"], + "category": "footwear", + "price": 129 +} +``` + +Harper commits the record first, then applies the committed change to the local derived full-text index. Every replica performs the same derivation from the transaction it receives. + +## 3. Search from a Harper application + +```javascript +export class SearchProducts extends tables.Product { + static async get(target) { + const text = target.searchParams.get('q') ?? ''; + return tables.Product.search({ + conditions: [ + { attribute: 'catalogSearch', comparator: 'matches', value: text, includeHighlights: true }, + { attribute: 'category', comparator: 'equals', value: 'footwear' }, + ], + select: ['id', 'name', 'price', '$score', '$highlights'], + limit: 20, + }); + } +} +``` + +The full-text index provides ranked candidates. The indexed category condition narrows the result set, and Harper loads the current records before returning them. + +## 4. Search through REST + +The exported table supports the same match comparator: + +```http +GET /Product/?catalogSearch=matches=waterproof%20trail&category=footwear&select(id,name,price,$score,$highlights)&limit(20) +``` + +For a phrase: + +```http +GET /Product/?catalogSearch=matches_phrase=trail%20running&limit(20) +``` + +For record autocomplete: + +```http +GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&select(id,name)&limit(10) +``` + +Prefix search returns matching products. It does not return a dictionary of suggested terms. + +## 5. Choose consistency behavior + +Most searches can use the default three-second lag tolerance. When the next request must find a record that was just written, use the JavaScript API and wait for current index coverage: + +```javascript +const results = await tables.Product.search({ + conditions: [ + { + attribute: 'catalogSearch', + comparator: 'matches', + value: 'new seasonal product', + maxIndexLagMilliseconds: 0, + waitForIndexMilliseconds: 10000, + }, + ], +}); +``` + +Do not apply this wait to every catalog request without measuring it. The normal bounded-staleness path keeps search traffic independent of short indexing bursts. + +## 6. Operate the index + +Use `describe_table` to inspect readiness. Harper reuses compatible local index files after restart and replays changes after their checkpoint. A node rebuilds locally when the files are missing, incompatible, or corrupt; table traffic remains available, while full-text requests return a retryable `503` until the rebuild is ready. + +See the [full-text configuration reference](/reference/v5/full-text-search/configuration) and [query reference](/reference/v5/full-text-search/querying) for every option and match mode. diff --git a/reference/database/schema.md b/reference/database/schema.md index ef96ecd9d..39afafec9 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -362,6 +362,25 @@ Write semantics: Multiple `@embed` attributes on one type are computed concurrently. +### `@fullText` + + + +Creates a BM25-ranked full-text derived index from one or more stored text fields. The directive is declared on the table type and can be repeated for multiple independent indexes. + +```graphql +type Product + @table(database: "catalog", audit: true) + @fullText(name: "catalogSearch", fields: [{ name: "name", weight: 3 }, { name: "description" }, { name: "tags" }]) { + id: ID @primaryKey + name: String + description: String + tags: [String] +} +``` + +Full-text indexes require RocksDB and an audited table. They are local derived state rebuilt from committed table data rather than authoritative record storage. See [Full-Text Search Configuration](../full-text-search/configuration.md) for fields, phrase and prefix storage, synonyms, highlighting, and rebuild behavior. + ### `@createdTime` Automatically assigns a creation timestamp (Unix epoch milliseconds) to the attribute when a record is created. diff --git a/reference/full-text-search/configuration.md b/reference/full-text-search/configuration.md new file mode 100644 index 000000000..9b6f1eb7c --- /dev/null +++ b/reference/full-text-search/configuration.md @@ -0,0 +1,157 @@ +--- +title: Configuration +--- + +# Full-Text Search Configuration + + + +Declare a full-text index with `@fullText` on a table type. The directive is repeatable, so one table can have independent indexes for different search experiences. + +```graphql +type Product + @table(database: "catalog", audit: true) + @fullText( + name: "catalogSearch" + fields: [ + { name: "name", weight: 3, highlight: true } + { name: "description", weight: 1, highlight: true } + { name: "tags", weight: 1 } + ] + highlighting: { maxFragments: 2, fragmentLength: 120 } + ) { + id: ID @primaryKey + name: String + description: String + tags: [String] + price: Float @indexed +} +``` + +The index name is the attribute used in search conditions. It does not add a stored field to each record. + +## Directive options + +| Option | Default | Description | +| -------------- | ------------- | ------------------------------------------------------------------------------------ | +| `name` | required | Unique index name on the table. | +| `fields` | required | One or more source fields. | +| `analyzer` | `"english@2"` | Versioned text analyzer. `english@2` is the only supported value. | +| `stopWords` | `true` | Removes common English terms during analysis. | +| `positions` | `true` | Stores token positions. Required for phrase queries. | +| `surfaceTerms` | `true` | Stores original analyzed terms. Required for prefix, fuzzy-prefix, and highlighting. | +| `synonyms` | `[]` | Index-time synonym expansion rules. | +| `highlighting` | omitted | Enables highlight generation and sets fragment limits. | + +Each field entry supports: + +| Option | Default | Description | +| ----------- | -------- | ------------------------------------------------------------------------------ | +| `name` | required | Stored source field name. | +| `weight` | `1` | Positive relevance weight. Higher values contribute more to BM25 ranking. | +| `highlight` | `false` | Allows highlights from this field when index highlighting is enabled. | +| `mediaType` | omitted | Required as `"text/plain"` for a `Blob` source; invalid on other source types. | + +Source fields cannot use `@computed` or `@relationship`. At write time, a source value must be a string, an array containing strings or nulls, or a matching text Blob. + +## Phrase and prefix support + +`positions` and `surfaceTerms` trade index size for query features. Keep their defaults unless storage measurements show that the features are unnecessary. + +```graphql +type Article + @table(audit: true) + @fullText(name: "bodySearch", fields: [{ name: "body" }], positions: false, surfaceTerms: false) { + id: ID @primaryKey + body: String +} +``` + +This smaller declaration supports `matches`, `matches_all`, and `matches_fuzzy`. It rejects phrase, prefix, fuzzy-prefix, and highlighting requests. + +## Highlighting + +Highlighting is off by default. To enable it: + +1. Set `highlight: true` on at least one source field. +2. Add `highlighting` to the directive. +3. Keep `surfaceTerms: true`. + +```graphql +@fullText( + name: "articleSearch" + fields: [{ name: "title", highlight: true }, { name: "body", highlight: true }] + highlighting: { maxFragments: 3, fragmentLength: 160 } +) +``` + +`maxFragments` and `fragmentLength` must be positive integers. Query results contain offsets into the original field value; see [Highlights](./querying.md#highlights). + +## Synonyms + +Synonyms expand terms while records are indexed: + +```graphql +@fullText( + name: "catalogSearch" + fields: [{ name: "description" }] + synonyms: [ + { source: "sneaker", replacements: ["shoe", "trainer"] } + { source: "tv", replacements: ["television"] } + ] +) +``` + +Rules require one source term and one or more unique replacement terms. A rule cannot replace a term with itself. + +## Multiple indexes + +Repeat `@fullText` when separate index definitions are useful: + +```graphql +type Product + @table(database: "catalog", audit: true) + @fullText(name: "titleSearch", fields: [{ name: "name", weight: 2 }]) + @fullText(name: "tagSearch", fields: [{ name: "tags" }]) { + id: ID @primaryKey + name: String + tags: [String] +} +``` + +The indexes update independently and can be queried independently. A single query cannot combine conditions from different full-text indexes. + +## Blob sources + +Only plain-text Blob fields are supported: + +```graphql +type Document + @table(audit: true) + @fullText(name: "contentSearch", fields: [{ name: "content", mediaType: "text/plain" }]) { + id: ID @primaryKey + content: Blob +} +``` + +Harper validates the Blob's media type when applying it to the index. + +## Schema changes and rebuilds + +Changes that affect indexed storage create a new local index generation and rebuild from source data. These include source names and media types, analyzer settings, stop-word handling, positions, surface terms, and synonyms. + +Changing source weights or highlighting settings changes query behavior without rebuilding indexed term storage. Removing an index retires its local files after the schema change is confirmed. + +## Inspecting an index + +`describe_table` reports `full_text_indexes`, including fields, analyzer options, supported query modes, generation, and readiness: + +```json +{ + "operation": "describe_table", + "database": "catalog", + "table": "Product" +} +``` + +An index can report states such as ready, rebuilding, or unavailable. Full-text queries are served only after readiness is established. diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md new file mode 100644 index 000000000..d357c741d --- /dev/null +++ b/reference/full-text-search/overview.md @@ -0,0 +1,61 @@ +--- +title: Overview +--- + +# Full-Text Search + + + +Harper can build a BM25-ranked full-text index from one or more fields on a table. The index is declared in the table schema, updated from committed record changes, and queried through the same `Table.search()` and REST interfaces used for other Harper queries. + +Use full-text search when users need relevance-ranked matching across product names, descriptions, tags, articles, or other natural-language fields. It supports term, all-term, phrase, prefix, fuzzy, and fuzzy-prefix matching. Prefix modes return matching records; Harper does not currently expose a separate term-suggestion API. + +## Requirements + +- The table must use RocksDB and have `audit: true`. +- Full-text source fields must be stored `String`, `[String]`, or `Blob` values. Blob sources must be declared as `text/plain`. +- LMDB tables cannot activate `@fullText` indexes. + +See [Configuration](./configuration.md) for the complete schema contract and [Querying](./querying.md) for search examples. + +## How it works + +Full-text indexes are derived local state. The table records and schema remain the source of truth. + +```mermaid +flowchart LR + W[Committed table write] --> A[Harper audit stream] + A --> F[Full-text derived-index runtime] + F --> T[Local Tantivy index] + T --> Q[Table.search and REST] + A --> R[Replication] + R --> D[Replica audit stream] + D --> U[Replica local Tantivy index] +``` + +Each node builds its own index from committed transactions. Harper does not replicate Tantivy files or include them in authoritative backups. + +This design has two consequences: + +- A committed record can be visible before its derived full-text index has applied the same transaction. Queries use a bounded staleness policy, described in [Freshness controls](./querying.md#freshness-controls). +- A missing, corrupt, or incompatible index can be rebuilt from the table and audit stream without changing source records. + +## Startup and recovery + +On restart, Harper reuses compatible local index files and replays committed changes after the saved checkpoint. If those files are missing, incompatible, or corrupt, Harper rebuilds the index locally. + +Ordinary table reads and writes remain available during a rebuild. Full-text queries return a retryable `503` until the index is ready. A new replica follows the same process before serving full-text queries. + +Use `describe_table` to inspect each index's declaration and readiness. See [Inspecting an index](./configuration.md#inspecting-an-index). + +## Ranking and result loading + +Tantivy ranks matching documents with BM25. Harper then loads the authoritative table records for the matching IDs and discards stale or deleted candidates. Selecting `$score` includes the relevance score in each result. + +Full-text search can be combined with structured filters. Harper pushes compatible filters into candidate evaluation so selective filters do not silently under-fill a requested page. + +## Index ownership + +The native index stores tokenized terms, posting lists, positions when enabled, stored surface terms when enabled, and enough metadata to align a reader with the Harper audit checkpoint. It is not a second copy of the Harper database and is never authoritative for record contents. + +Harper owns schema validation, permissions, transaction/audit coordination, replication-derived replay, readiness, and record loading. The native full-text library owns tokenization, indexing, BM25 ranking, fuzzy and prefix execution, phrase matching, and highlight spans. diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md new file mode 100644 index 000000000..0096726ab --- /dev/null +++ b/reference/full-text-search/querying.md @@ -0,0 +1,217 @@ +--- +title: Querying +--- + +# Querying Full-Text Indexes + + + +Use the full-text index name as a condition attribute in `Table.search()`. + +```javascript +const products = await Product.search({ + conditions: [{ attribute: 'catalogSearch', comparator: 'matches', value: 'waterproof trail shoes' }], + limit: 20, +}); +``` + +Results are ordered by BM25 relevance. The index declaration, not the source field name, identifies the full-text search target. + +## Match modes + +| Comparator | Behavior | Required index option | +| ---------------------- | --------------------------------------------------------------------- | --------------------- | +| `matches` | Matches any analyzed query term. | — | +| `matches_all` | Requires every analyzed query term. | — | +| `matches_phrase` | Matches analyzed terms in order as a phrase. | `positions: true` | +| `matches_prefix` | Matches records whose analyzed terms start with the final query term. | `surfaceTerms: true` | +| `matches_fuzzy` | Matches terms within the supported edit distance. | — | +| `matches_fuzzy_prefix` | Combines fuzzy matching with a final-term prefix. | `surfaceTerms: true` | + +Every comparator also has a negated form: `not_matches`, `not_matches_all`, `not_matches_phrase`, `not_matches_prefix`, `not_matches_fuzzy`, and `not_matches_fuzzy_prefix`. + +A query containing a negated full-text condition must also contain a positive condition. The positive condition bounds the candidate set; the negated condition filters it. + +```javascript +const products = await Product.search({ + operator: 'and', + conditions: [ + { attribute: 'catalogSearch', comparator: 'matches', value: 'waterproof' }, + { attribute: 'catalogSearch', comparator: 'not_matches', value: 'leather' }, + ], +}); +``` + +## Restricting source fields + +By default, a condition searches every source field in the index. Use `fields` to restrict it: + +```javascript +const products = await Product.search({ + conditions: [ + { + attribute: 'catalogSearch', + comparator: 'matches_phrase', + value: 'trail running', + fields: ['name', 'description'], + }, + ], +}); +``` + +Every requested field must belong to the index and be readable by the caller. + +## Score and ordering + +Select `$score` to include the BM25 score: + +```javascript +const products = await Product.search({ + conditions: [{ attribute: 'catalogSearch', comparator: 'matches', value: 'trail shoes' }], + select: ['id', 'name', '$score'], + limit: 20, +}); +``` + +Full-text results use descending relevance order. The only explicit full-text sort is `$score` descending; other sort keys and reverse iteration are rejected because they would discard the native ranking. + +## Highlights + +Highlighting returns source-field fragments and matching character spans. The index must enable highlighting for each requested source field. + +Selecting `$highlights` turns highlighting on for the query: + +```javascript +const products = await Product.search({ + conditions: [{ attribute: 'catalogSearch', comparator: 'matches_phrase', value: 'trail running' }], + select: ['id', 'name', '$score', '$highlights'], +}); +``` + +You can also set `includeHighlights: true` on the condition. A result has this shape: + +```json +{ + "id": "shoe-1", + "name": "Waterproof trail running shoe", + "$score": 2.41, + "$highlights": { + "name": [ + { + "valueIndex": 0, + "spans": [{ "start": 11, "end": 24 }], + "fragments": [ + { + "text": "Waterproof trail running shoe", + "start": 0, + "spans": [{ "start": 11, "end": 24 }] + } + ] + } + ] + } +} +``` + +Offsets are UTF-16 half-open ranges. Value-level spans address the original source value. A fragment's `start` addresses that value, while spans inside the fragment address its returned `text`. Harper returns offsets rather than HTML so the caller controls rendering and escaping. + +## Combining conditions + +Full-text conditions can be combined with structured filters using `and`: + +```javascript +const products = await Product.search({ + operator: 'and', + conditions: [ + { attribute: 'catalogSearch', comparator: 'matches', value: 'trail shoe' }, + { attribute: 'price', comparator: 'less_than', value: 150 }, + ], + limit: 20, +}); +``` + +An `or` group may combine conditions from the same full-text index: + +```javascript +const products = await Product.search({ + operator: 'or', + conditions: [ + { attribute: 'catalogSearch', comparator: 'matches_phrase', value: 'trail running' }, + { attribute: 'catalogSearch', comparator: 'matches', value: 'hiking boot' }, + ], +}); +``` + +An `or` group cannot mix full-text and ordinary record conditions, and one query cannot combine different full-text indexes. Run separate queries when either boundary is required. + +## Freshness controls + +A full-text index is derived from committed table changes. Each condition accepts: + +| Option | Default | Description | +| -------------------------- | ------- | ---------------------------------------------------------------------------------- | +| `maxIndexLagMilliseconds` | `3000` | Maximum accepted upper bound on index lag. Set to `0` to require current coverage. | +| `waitForIndexMilliseconds` | `0` | How long to wait for acceptable coverage, from `0` through `30000`. | + +For read-after-write behavior, require current coverage and allow a bounded wait: + +```javascript +const products = await Product.search({ + conditions: [ + { + attribute: 'catalogSearch', + comparator: 'matches', + value: 'new product', + maxIndexLagMilliseconds: 0, + waitForIndexMilliseconds: 10000, + }, + ], +}); +``` + +A non-waiting HTTP query returns `Harper-Index-Coverage` with the admitted state, lag upper bound, and requested tolerance. Waiting queries establish coverage while the response stream is consumed and do not emit that header before completion. + +## Prefix result window + +`matches_prefix` and `matches_fuzzy_prefix` are autocomplete-style record searches. Their current native result window is 100 records, and `offset + limit` cannot exceed that window. They return matching records, not a separate list of suggested terms. + +Other match modes page through the native result window as needed. + +## REST + +Use the comparator in the collection query string: + +```http +GET /Product/?catalogSearch=matches=waterproof%20trail&limit(20) +GET /Product/?catalogSearch=matches_phrase=trail%20running&select(id,name,$score,$highlights) +GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&limit(10) +``` + +REST supports all positive and negated full-text comparators. It can express the query text, selected fields, and pagination. Use `Table.search()` or `search_by_conditions` when a condition needs `fields`, highlighting flags, or freshness controls. + +## Operations API + +`search_by_conditions` uses the same condition shape: + +```json +{ + "operation": "search_by_conditions", + "database": "catalog", + "table": "Product", + "limit": 20, + "get_attributes": ["id", "name", "$score", "$highlights"], + "conditions": [ + { + "attribute": "catalogSearch", + "comparator": "matches_all", + "value": "waterproof trail", + "fields": ["name", "description"], + "includeHighlights": true, + "maxIndexLagMilliseconds": 0, + "waitForIndexMilliseconds": 10000 + } + ] +} +``` + +If an index is rebuilding or unavailable, the query returns a retryable `503`. Invalid declarations, unsupported match modes, incompatible combinations, and out-of-range options return `400` responses. diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 93b196a2b..445409684 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -284,6 +284,8 @@ Returns records matching one or more conditions. Supports `operator` (`and`/`or` } ``` +Conditions can also query a declared full-text index with the `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix` comparators. The Operations API preserves advanced condition options such as source `fields`, `includeHighlights`, `maxIndexLagMilliseconds`, and `waitForIndexMilliseconds`. See [Querying Full-Text Indexes](../full-text-search/querying.md#operations-api). + --- ## Bulk Operations diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index dfe2cf9d1..52816e338 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -11,6 +11,14 @@ title: Query Optimization Harper has powerful query functionality with excellent performance characteristics. Like any database, different queries can vary significantly in performance. Understanding how querying works helps you write queries that perform well as your dataset grows. +## Full-text search + + + +A condition using a declared `@fullText` index is executed by the native full-text engine and ranked with BM25. Compatible `and` filters are evaluated while accepting candidates so a selective structured filter can still fill the requested result page. Harper then loads the current source records and removes stale or deleted candidates. + +Keep result pages bounded. Prefix and fuzzy-prefix searches have a 100-record native window and are intended for autocomplete-style record lookup. Use source-field weights to tune ranking, and enable positions, surface terms, and highlighting only when their query features are needed. See [Full-Text Search](../full-text-search/overview.md). + ## Query Execution At a fundamental level, querying involves defining conditions to find matching data and then executing those conditions against the database. Harper supports indexed fields, and these indexes are used to speed up query execution. diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index b64c6e7df..fe9297347 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -972,13 +972,13 @@ The `Query` object is accepted by `search()` and the static `get()` method. Array of condition objects to filter records. Each condition: -| Property | Description | -| ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `attribute` | Property name, or an array for chained/joined properties (e.g. `['brand', 'name']`) | -| `value` | The value to match | -| `comparator` | `equals` (default), `greater_than`, `greater_than_equal`, `less_than`, `less_than_equal`, `starts_with`, `contains`, `ends_with`, `between`, `not_equal` | -| `conditions` | Nested conditions array | -| `operator` | `and` (default) or `or` for the nested `conditions` | +| Property | Description | +| ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `attribute` | Property name, or an array for chained/joined properties (e.g. `['brand', 'name']`) | +| `value` | The value to match | +| `comparator` | `equals` (default), `greater_than`, `greater_than_equal`, `less_than`, `less_than_equal`, `starts_with`, `contains`, `ends_with`, `between`, `not_equal`, or a [full-text comparator](../full-text-search/querying.md#match-modes) | +| `conditions` | Nested conditions array | +| `operator` | `and` (default) or `or` for the nested `conditions` | Example with nested conditions: @@ -1030,6 +1030,8 @@ Special properties: - `$id` — Returns the primary key regardless of its name - `$updatedtime` — Returns the last-updated timestamp - `$distance` — When the query ranks or filters by a vector index, returns the computed distance from the target vector. See [Vector Indexing](../database/schema.md#vector-indexing). +- `$score` — When a query uses a full-text index, returns its BM25 relevance score. +- `$highlights` — Enables and returns full-text fragments and matching spans when the index has highlighting configured. See [Highlights](../full-text-search/querying.md#highlights). #### Selecting related records diff --git a/reference/rest/headers.md b/reference/rest/headers.md index f0a98c215..34ecc3ce5 100644 --- a/reference/rest/headers.md +++ b/reference/rest/headers.md @@ -22,6 +22,8 @@ These headers are included in all Harper REST API responses: Collection responses to a [count request](./querying.md#pagination-and-total-count) additionally include `Content-Range`, `Range-Unit`, and `Preference-Applied` (). +A non-waiting request that uses a native derived index can include `Harper-Index-Coverage`. For full-text search, it reports the index state, the admitted lag upper bound, and the requested tolerance. See [Full-text freshness controls](../full-text-search/querying.md#freshness-controls). + ## Cache-Control diff --git a/reference/rest/querying.md b/reference/rest/querying.md index b8d7d0e66..a5ff5575e 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -64,6 +64,20 @@ GET /Product/?name==Keyboard* GET /Product/?category=software&price=gt=100&price=lt=200 ``` +### Full-Text Operators + + + +For a declared `@fullText` index, use its name as the query attribute and one of the full-text comparators: + +```http +GET /Product/?catalogSearch=matches=waterproof%20trail +GET /Product/?catalogSearch=matches_phrase=trail%20running +GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&limit(10) +``` + +REST supports `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix`, plus a `not_` form of each. See [Querying Full-Text Indexes](../full-text-search/querying.md) for ranking, highlights, query combinations, and freshness controls. + For date fields, colons must be URL-encoded as `%3A`: ```http diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index 7829e4aab..6522ce93f 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -4,6 +4,12 @@ title: '5.3' # 5.3 Release Notes +## Full-Text Search + +Harper tables can declare native BM25-ranked indexes with `@fullText` and query them through `Table.search()`, REST, or `search_by_conditions`. Search supports any-term, all-term, phrase, prefix, fuzzy, and fuzzy-prefix matching, with optional source weights, synonyms, scores, and highlights. + +The indexes are derived locally from committed audited RocksDB records. Each replica maintains its own files, reuses a compatible checkpoint after restart, and rebuilds locally when necessary. See [Full-Text Search](/reference/v5/full-text-search/overview) and [Add Full-Text Search to a Product Catalog](/learn/developers/full-text-search-product-catalog). + ### Patch Releases All patch release notes for 5.3.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.3&expanded=true). diff --git a/sidebarsLearn.ts b/sidebarsLearn.ts index 721b1690c..b44e01594 100644 --- a/sidebarsLearn.ts +++ b/sidebarsLearn.ts @@ -61,6 +61,11 @@ const sidebarsLearn: SidebarsConfig = { id: 'developers/semantic-caching-vector-indexing', label: 'Semantic Caching with Vector Indexing', }, + { + type: 'doc', + id: 'developers/full-text-search-product-catalog', + label: 'Full-Text Product Search', + }, { type: 'doc', id: 'developers/write-through-caching', diff --git a/sidebarsReference.ts b/sidebarsReference.ts index adb557736..1ad0aa137 100644 --- a/sidebarsReference.ts +++ b/sidebarsReference.ts @@ -89,6 +89,29 @@ const sidebars: SidebarsConfig = { }, ], }, + { + type: 'category', + label: 'Full-Text Search', + collapsible: false, + className: 'reference-category-header', + items: [ + { + type: 'doc', + id: 'full-text-search/overview', + label: 'Overview', + }, + { + type: 'doc', + id: 'full-text-search/configuration', + label: 'Configuration', + }, + { + type: 'doc', + id: 'full-text-search/querying', + label: 'Querying', + }, + ], + }, { type: 'category', label: 'AI & Models', From 02dbfe3bd56a0fc11473364826cb45d1755bcdb0 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Sun, 27 Sep 2026 20:31:18 -0600 Subject: [PATCH 02/78] Clarify full-text prefix result limits --- reference/full-text-search/querying.md | 2 +- reference/resources/query-optimization.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 0096726ab..437d25233 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -173,7 +173,7 @@ A non-waiting HTTP query returns `Harper-Index-Coverage` with the admitted state ## Prefix result window -`matches_prefix` and `matches_fuzzy_prefix` are autocomplete-style record searches. Their current native result window is 100 records, and `offset + limit` cannot exceed that window. They return matching records, not a separate list of suggested terms. +`matches_prefix` and `matches_fuzzy_prefix` are autocomplete-style record searches. Any expression containing one of these modes uses a 100-record native result window, and `offset + limit` cannot exceed that window. If an unbounded query has more than 100 matches, Harper returns `400` and requires a limit instead of silently truncating the result. These modes return matching records, not a separate list of suggested terms. Other match modes page through the native result window as needed. diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index 52816e338..cdcf7f5e2 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -17,7 +17,7 @@ Harper has powerful query functionality with excellent performance characteristi A condition using a declared `@fullText` index is executed by the native full-text engine and ranked with BM25. Compatible `and` filters are evaluated while accepting candidates so a selective structured filter can still fill the requested result page. Harper then loads the current source records and removes stale or deleted candidates. -Keep result pages bounded. Prefix and fuzzy-prefix searches have a 100-record native window and are intended for autocomplete-style record lookup. Use source-field weights to tune ranking, and enable positions, surface terms, and highlighting only when their query features are needed. See [Full-Text Search](../full-text-search/overview.md). +Keep result pages bounded. Any expression containing prefix or fuzzy-prefix matching has a 100-record native window and is intended for autocomplete-style record lookup. An unbounded query that exceeds the window fails instead of returning a partial result. Use source-field weights to tune ranking, and enable positions, surface terms, and highlighting only when their query features are needed. See [Full-Text Search](../full-text-search/overview.md). ## Query Execution From 5bb423301f93a08759406f110eb306183118522b Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Sun, 27 Sep 2026 23:04:55 -0600 Subject: [PATCH 03/78] docs(fulltext): clarify runtime and recovery contracts --- .../full-text-search-product-catalog.mdx | 25 +++++++++++- reference/backups/overview.md | 3 +- reference/database/schema.md | 40 +++++++++---------- reference/full-text-search/configuration.md | 25 +++++++++++- reference/full-text-search/overview.md | 24 +++++------ reference/full-text-search/querying.md | 12 +++--- reference/index.md | 1 + reference/operations-api/operations.md | 19 ++++++++- reference/rest/headers.md | 2 +- reference/rest/querying.md | 2 +- sidebarsLearn.ts | 8 ++-- 11 files changed, 110 insertions(+), 51 deletions(-) diff --git a/learn/developers/full-text-search-product-catalog.mdx b/learn/developers/full-text-search-product-catalog.mdx index e99ff6dc2..d99debec0 100644 --- a/learn/developers/full-text-search-product-catalog.mdx +++ b/learn/developers/full-text-search-product-catalog.mdx @@ -5,8 +5,21 @@ description: Configure and query a BM25-ranked product search in Harper. # Add Full-Text Search to a Product Catalog + + This guide adds relevance-ranked text search to a product catalog, then exposes it through Harper's JavaScript and REST APIs. +## What you will build + +- A BM25-ranked index over product names, descriptions, and tags +- A Harper application endpoint with highlights and a structured category filter +- REST queries for terms, phrases, and record autocomplete + +## Prerequisites + +- Harper 5.3 or later using RocksDB +- A component with a GraphQL schema and JavaScript resources + ## 1. Declare the index Add `audit: true` to the RocksDB-backed table and declare an `@fullText` index over the fields customers search. @@ -57,10 +70,10 @@ Harper commits the record first, then applies the committed change to the local ## 3. Search from a Harper application -```javascript +```javascript title="resources.js" export class SearchProducts extends tables.Product { static async get(target) { - const text = target.searchParams.get('q') ?? ''; + const text = target.get('q') ?? ''; return tables.Product.search({ conditions: [ { attribute: 'catalogSearch', comparator: 'matches', value: text, includeHighlights: true }, @@ -73,6 +86,8 @@ export class SearchProducts extends tables.Product { } ``` +Call the resource with `GET /SearchProducts?q=trail`. + The full-text index provides ranked candidates. The indexed category condition narrows the result set, and Harper loads the current records before returning them. ## 4. Search through REST @@ -122,3 +137,9 @@ Do not apply this wait to every catalog request without measuring it. The normal Use `describe_table` to inspect readiness. Harper reuses compatible local index files after restart and replays changes after their checkpoint. A node rebuilds locally when the files are missing, incompatible, or corrupt; table traffic remains available, while full-text requests return a retryable `503` until the rebuild is ready. See the [full-text configuration reference](/reference/v5/full-text-search/configuration) and [query reference](/reference/v5/full-text-search/querying) for every option and match mode. + +## Additional resources + +- [Full-text search overview](/reference/v5/full-text-search/overview) +- [Resource API](/reference/v5/resources/resource-api) +- [Query optimization](/reference/v5/resources/query-optimization) diff --git a/reference/backups/overview.md b/reference/backups/overview.md index 3a0ab21dd..38a1ac740 100644 --- a/reference/backups/overview.md +++ b/reference/backups/overview.md @@ -5,7 +5,7 @@ title: Backups -Harper can back up and restore its databases natively. A backup is a whole-database copy — all tables, the audit/transaction log, and any file-backed blobs — so a restored database is a consistent point-in-time image of everything in it. (`get_backup` on an LMDB database is the exception: it can stream a subset of tables, and includes the audit store only when requested.) +Harper can back up and restore its databases natively. A backup is a whole-database copy of authoritative state — all tables, the audit/transaction log, and any file-backed blobs — so a restored database is a consistent point-in-time image. Local derived state, including full-text index files, is excluded and rebuilt after restore. (`get_backup` on an LMDB database is the exception: it can stream a subset of tables, and includes the audit store only when requested.) Two complementary mechanisms are available: @@ -38,6 +38,7 @@ The offline path matters for restore. RocksDB is single-writer, so an in-place r - **Backups live on the node that created them.** The backup repository is a local directory. RocksDB shares files across backup IDs (a backup ID is not a self-contained folder), so disaster-recovery copies must take the entire per-database repository — `/` — not an individual backup, and must do so while no backup operation is running, or from an atomic filesystem snapshot. A live recursive copy can race `create_backup`/`delete_backup`/`purge_backups` and produce an unrestorable copy. Alternatively, use `get_backup` to pull a snapshot from a running server. - **`get_backup` always streams the current state.** It cannot download a historical managed backup; to move a retained backup off-host, copy its whole per-database repository as above. - **A restore is a point-in-time rollback.** In a replicated cluster, coordinate a restore with replication before bringing the node back. +- **Full-text indexes rebuild locally after restore.** Ordinary table traffic remains available, but full-text requests return `INDEX_REBUILDING` until the restored node catches up. - **An interrupted restore leaves the database unloadable.** If `restore_backup` is interrupted before completing (crash, power loss), Harper marks the database as incompletely restored and skips loading it on the next start, logging an incomplete-restore error. Rerun `restore_backup` for the same database and `backup_id` to recover; do not load or hand-repair the directory. ### When can a database be restored? diff --git a/reference/database/schema.md b/reference/database/schema.md index 39afafec9..9516119c5 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -292,6 +292,25 @@ Docstrings on `@hidden` fields are dropped from the descriptive surfaces alongsi > **Trust model.** Docstrings reach LLMs and public OpenAPI consumers verbatim. Treat them as code: don't put secrets, internal-only commentary, or speculative prose in them. Use `@hidden` to suppress fields that shouldn't surface publicly. +### `@fullText` + + + +Creates a BM25-ranked full-text derived index from one or more stored text fields. The directive is declared on the table type and can be repeated for multiple independent indexes. + +```graphql +type Product + @table(database: "catalog", audit: true) + @fullText(name: "catalogSearch", fields: [{ name: "name", weight: 3 }, { name: "description" }, { name: "tags" }]) { + id: ID @primaryKey + name: String + description: String + tags: [String] +} +``` + +Full-text indexes require RocksDB and an audited table. They are local derived state rebuilt from committed table data rather than authoritative record storage. See [Full-Text Search Configuration](../full-text-search/configuration.md) for fields, phrase and prefix storage, synonyms, highlighting, and rebuild behavior. + ## Field Directives Field directives apply to individual attributes in a type definition. @@ -362,25 +381,6 @@ Write semantics: Multiple `@embed` attributes on one type are computed concurrently. -### `@fullText` - - - -Creates a BM25-ranked full-text derived index from one or more stored text fields. The directive is declared on the table type and can be repeated for multiple independent indexes. - -```graphql -type Product - @table(database: "catalog", audit: true) - @fullText(name: "catalogSearch", fields: [{ name: "name", weight: 3 }, { name: "description" }, { name: "tags" }]) { - id: ID @primaryKey - name: String - description: String - tags: [String] -} -``` - -Full-text indexes require RocksDB and an audited table. They are local derived state rebuilt from committed table data rather than authoritative record storage. See [Full-Text Search Configuration](../full-text-search/configuration.md) for fields, phrase and prefix storage, synonyms, highlighting, and rebuild behavior. - ### `@createdTime` Automatically assigns a creation timestamp (Unix epoch milliseconds) to the attribute when a record is created. @@ -551,7 +551,7 @@ Computed properties that read other tables use the same [trusted server-side aut ### Computed Indexes -Computed properties can be indexed with `@indexed`, enabling custom indexing strategies such as composite indexes, full-text search, or vector indexing: +Computed properties can be indexed with `@indexed`, enabling custom lookup strategies such as composite keys. Use [`@fullText`](#fulltext) for native BM25-ranked text search and an HNSW `[Float]` field for vector indexing: ```graphql type Product @table { diff --git a/reference/full-text-search/configuration.md b/reference/full-text-search/configuration.md index 9b6f1eb7c..12c1c2f33 100644 --- a/reference/full-text-search/configuration.md +++ b/reference/full-text-search/configuration.md @@ -144,7 +144,7 @@ Changing source weights or highlighting settings changes query behavior without ## Inspecting an index -`describe_table` reports `full_text_indexes`, including fields, analyzer options, supported query modes, generation, and readiness: +`describe_table` reports `full_text_indexes`, including fields, analyzer options, supported query modes, and readiness: ```json { @@ -154,4 +154,25 @@ Changing source weights or highlighting settings changes query behavior without } ``` -An index can report states such as ready, rebuilding, or unavailable. Full-text queries are served only after readiness is established. +The table response includes entries shaped like this: + +```json +{ + "full_text_indexes": [ + { + "name": "catalogSearch", + "fields": [{ "name": "name", "weight": 3, "highlight": true }], + "analyzer": "english@2", + "stop_words": true, + "positions": true, + "surface_terms": true, + "synonyms": [], + "highlighting": { "maxFragments": 2, "fragmentLength": 120 }, + "query_modes": ["any", "all", "fuzzy", "phrase", "prefix", "fuzzy-prefix"], + "readiness": { "state": "ready", "owner_epoch": "4", "rebuild_attempts": 0 } + } + ] +} +``` + +`readiness.state` is `ready`, `rebuilding`, or `unavailable`. `readiness.reason` is included when Harper has more detail. Full-text queries are served only after readiness is established. diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index d357c741d..9024c291b 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -8,15 +8,15 @@ title: Overview Harper can build a BM25-ranked full-text index from one or more fields on a table. The index is declared in the table schema, updated from committed record changes, and queried through the same `Table.search()` and REST interfaces used for other Harper queries. -Use full-text search when users need relevance-ranked matching across product names, descriptions, tags, articles, or other natural-language fields. It supports term, all-term, phrase, prefix, fuzzy, and fuzzy-prefix matching. Prefix modes return matching records; Harper does not currently expose a separate term-suggestion API. +Use full-text search when users need relevance-ranked matching across product names, descriptions, tags, articles, or other natural-language fields. It supports term, all-term, phrase, prefix, fuzzy, and fuzzy-prefix matching. ## Requirements -- The table must use RocksDB and have `audit: true`. +- The table must use RocksDB and have `audit: true`. - Full-text source fields must be stored `String`, `[String]`, or `Blob` values. Blob sources must be declared as `text/plain`. -- LMDB tables cannot activate `@fullText` indexes. +- Creating or updating an LMDB table with `@fullText` fails with `400`. A persisted declaration encountered while opening LMDB is ignored with a warning. Migrate the table to RocksDB before activating the index. -See [Configuration](./configuration.md) for the complete schema contract and [Querying](./querying.md) for search examples. +See [Configuration](./configuration.md) for the complete schema contract, [Querying](./querying.md) for search examples, and [Query optimization](../resources/query-optimization.md) for performance guidance. ## How it works @@ -26,14 +26,14 @@ Full-text indexes are derived local state. The table records and schema remain t flowchart LR W[Committed table write] --> A[Harper audit stream] A --> F[Full-text derived-index runtime] - F --> T[Local Tantivy index] + F --> T[Local full-text index] T --> Q[Table.search and REST] A --> R[Replication] R --> D[Replica audit stream] - D --> U[Replica local Tantivy index] + D --> U[Replica local full-text index] ``` -Each node builds its own index from committed transactions. Harper does not replicate Tantivy files or include them in authoritative backups. +Each node builds its own index from committed transactions. Harper does not replicate full-text index files or include them in authoritative backups. After a restore or move to a node without compatible local files, Harper rebuilds the index before serving full-text queries. This design has two consequences: @@ -44,18 +44,16 @@ This design has two consequences: On restart, Harper reuses compatible local index files and replays committed changes after the saved checkpoint. If those files are missing, incompatible, or corrupt, Harper rebuilds the index locally. -Ordinary table reads and writes remain available during a rebuild. Full-text queries return a retryable `503` until the index is ready. A new replica follows the same process before serving full-text queries. +Ordinary table reads and writes remain available during a rebuild. Full-text queries return `503` with `code: "INDEX_REBUILDING"` and `retryable: true` until the index is ready. A new replica follows the same process before serving full-text queries. Use `describe_table` to inspect each index's declaration and readiness. See [Inspecting an index](./configuration.md#inspecting-an-index). ## Ranking and result loading -Tantivy ranks matching documents with BM25. Harper then loads the authoritative table records for the matching IDs and discards stale or deleted candidates. Selecting `$score` includes the relevance score in each result. +The native index ranks matching documents with BM25. Harper then loads the authoritative table records for the matching IDs and discards stale or deleted candidates. Selecting `$score` includes the relevance score in each result. Full-text search can be combined with structured filters. Harper pushes compatible filters into candidate evaluation so selective filters do not silently under-fill a requested page. -## Index ownership +## Stored index data -The native index stores tokenized terms, posting lists, positions when enabled, stored surface terms when enabled, and enough metadata to align a reader with the Harper audit checkpoint. It is not a second copy of the Harper database and is never authoritative for record contents. - -Harper owns schema validation, permissions, transaction/audit coordination, replication-derived replay, readiness, and record loading. The native full-text library owns tokenization, indexing, BM25 ranking, fuzzy and prefix execution, phrase matching, and highlight spans. +The local index stores tokenized terms, posting lists, optional positions and surface terms, and checkpoint metadata. It does not store an authoritative copy of each record. Harper always loads current table records before returning results, and it can discard and rebuild this derived state from committed data. diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 437d25233..0f99c4136 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -25,12 +25,12 @@ Results are ordered by BM25 relevance. The index declaration, not the source fie | `matches_all` | Requires every analyzed query term. | — | | `matches_phrase` | Matches analyzed terms in order as a phrase. | `positions: true` | | `matches_prefix` | Matches records whose analyzed terms start with the final query term. | `surfaceTerms: true` | -| `matches_fuzzy` | Matches terms within the supported edit distance. | — | -| `matches_fuzzy_prefix` | Combines fuzzy matching with a final-term prefix. | `surfaceTerms: true` | +| `matches_fuzzy` | Matches terms within one edit, including a transposition. | — | +| `matches_fuzzy_prefix` | Combines one-edit fuzzy matching with a final-term prefix. | `surfaceTerms: true` | Every comparator also has a negated form: `not_matches`, `not_matches_all`, `not_matches_phrase`, `not_matches_prefix`, `not_matches_fuzzy`, and `not_matches_fuzzy_prefix`. -A query containing a negated full-text condition must also contain a positive condition. The positive condition bounds the candidate set; the negated condition filters it. +A query containing a negated full-text condition must also contain a non-negated full-text condition on the same index. A structured record condition alone does not satisfy this requirement. The positive full-text condition bounds the candidate set; the negated condition filters it. ```javascript const products = await Product.search({ @@ -169,7 +169,9 @@ const products = await Product.search({ }); ``` -A non-waiting HTTP query returns `Harper-Index-Coverage` with the admitted state, lag upper bound, and requested tolerance. Waiting queries establish coverage while the response stream is consumed and do not emit that header before completion. +All full-text conditions combined into one query must use the same freshness values. Harper returns `400` when combined conditions specify different values. + +A non-waiting HTTP query returns `Harper-Index-Coverage` with the admitted state, lag upper bound, and requested tolerance. Waiting queries, available through `Table.search()` and `search_by_conditions`, establish coverage while the response stream is consumed and do not emit that header before completion. ## Prefix result window @@ -214,4 +216,4 @@ REST supports all positive and negated full-text comparators. It can express the } ``` -If an index is rebuilding or unavailable, the query returns a retryable `503`. Invalid declarations, unsupported match modes, incompatible combinations, and out-of-range options return `400` responses. +If an index is rebuilding, the query returns `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. Other unavailable or busy paths can return a generic `503`, so clients should branch on the code rather than the status alone. Invalid declarations, unsupported match modes, incompatible combinations, and out-of-range options return `400` responses. diff --git a/reference/index.md b/reference/index.md index 306c06837..a5606d1b3 100644 --- a/reference/index.md +++ b/reference/index.md @@ -42,6 +42,7 @@ For concept introductions, tutorials, and guides, see the [Learn](/learn) sectio | Section | Description | | ------------------------------------------------------------ | ------------------------------------------------------------------ | | [Backups](./backups/overview.md) | Managed incremental backups, snapshot downloads, and restore | +| [Full-Text Search](./full-text-search/overview.md) | BM25-ranked native search, schema configuration, and query modes | | [Logging](./logging/overview.md) | Log configuration, the `logger` API, and log management operations | | [Analytics](./analytics/overview.md) | Resource and storage analytics, system tables | | [MQTT](./mqtt/overview.md) | MQTT broker configuration and usage | diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 445409684..31a0e7bb6 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -63,12 +63,27 @@ Returns all table definitions within the specified database. Each table definiti ### `describe_table` -Returns the definition of a specific table. +Returns the definition of a specific table. A table with native full-text indexes includes `full_text_indexes`; each entry reports its source fields, analyzer options, enabled query modes, and `readiness` (`ready`, `rebuilding`, or `unavailable`). The readiness object also includes `owner_epoch`, `rebuild_attempts`, and an optional `reason`. ```json { "operation": "describe_table", "table": "dog", "database": "dev" } ``` +```json +{ + "name": "Product", + "full_text_indexes": [ + { + "name": "catalogSearch", + "fields": [{ "name": "name", "weight": 3 }], + "analyzer": "english@2", + "query_modes": ["any", "all", "fuzzy", "phrase", "prefix", "fuzzy-prefix"], + "readiness": { "state": "ready", "owner_epoch": "4", "rebuild_attempts": 0 } + } + ] +} +``` + Alongside the schema, the response carries the size of the table's **record-structure dictionaries** @@ -284,7 +299,7 @@ Returns records matching one or more conditions. Supports `operator` (`and`/`or` } ``` -Conditions can also query a declared full-text index with the `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix` comparators. The Operations API preserves advanced condition options such as source `fields`, `includeHighlights`, `maxIndexLagMilliseconds`, and `waitForIndexMilliseconds`. See [Querying Full-Text Indexes](../full-text-search/querying.md#operations-api). +Conditions can also query a declared full-text index with the `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix` comparators and their `not_` forms. A negated full-text condition requires a non-negated condition on the same full-text index. The Operations API preserves advanced condition options such as source `fields`, `includeHighlights`, `maxIndexLagMilliseconds`, and `waitForIndexMilliseconds`. See [Querying Full-Text Indexes](../full-text-search/querying.md#operations-api). --- diff --git a/reference/rest/headers.md b/reference/rest/headers.md index 34ecc3ce5..6fd4c17f7 100644 --- a/reference/rest/headers.md +++ b/reference/rest/headers.md @@ -22,7 +22,7 @@ These headers are included in all Harper REST API responses: Collection responses to a [count request](./querying.md#pagination-and-total-count) additionally include `Content-Range`, `Range-Unit`, and `Preference-Applied` (). -A non-waiting request that uses a native derived index can include `Harper-Index-Coverage`. For full-text search, it reports the index state, the admitted lag upper bound, and the requested tolerance. See [Full-text freshness controls](../full-text-search/querying.md#freshness-controls). +A non-waiting request that uses a native derived index can include `Harper-Index-Coverage`. For full-text search, a value such as `current; lag=0; tolerance=3000` reports the admitted state, lag upper bound, and requested tolerance in milliseconds. See [Full-text freshness controls](../full-text-search/querying.md#freshness-controls). ## Cache-Control diff --git a/reference/rest/querying.md b/reference/rest/querying.md index a5ff5575e..537fa8a6e 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -76,7 +76,7 @@ GET /Product/?catalogSearch=matches_phrase=trail%20running GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&limit(10) ``` -REST supports `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix`, plus a `not_` form of each. See [Querying Full-Text Indexes](../full-text-search/querying.md) for ranking, highlights, query combinations, and freshness controls. +REST supports `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix`, plus a `not_` form of each. A negated full-text condition requires a non-negated condition on the same full-text index, which may require `Table.search()` or `search_by_conditions` for combined conditions. See [Querying Full-Text Indexes](../full-text-search/querying.md) for ranking, highlights, query combinations, and freshness controls. For date fields, colons must be URL-encoded as `%3A`: diff --git a/sidebarsLearn.ts b/sidebarsLearn.ts index b44e01594..e77f63c0d 100644 --- a/sidebarsLearn.ts +++ b/sidebarsLearn.ts @@ -63,13 +63,13 @@ const sidebarsLearn: SidebarsConfig = { }, { type: 'doc', - id: 'developers/full-text-search-product-catalog', - label: 'Full-Text Product Search', + id: 'developers/write-through-caching', + label: 'Write-Through Caching', }, { type: 'doc', - id: 'developers/write-through-caching', - label: 'Write-Through Caching', + id: 'developers/full-text-search-product-catalog', + label: 'Full-Text Product Search', }, { type: 'doc', From b5e9b21a52c06b31894df796a4ead5e495e9ecd5 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 28 Sep 2026 07:12:17 -0600 Subject: [PATCH 04/78] docs(fulltext): address final review --- .../full-text-search-product-catalog.mdx | 10 +++-- reference/database/overview.md | 1 + reference/database/schema.md | 38 +++++++++---------- reference/full-text-search/configuration.md | 6 ++- reference/full-text-search/querying.md | 18 +++++++-- reference/resources/query-optimization.md | 16 ++++---- reference/rest/headers.md | 9 ++++- reference/rest/querying.md | 10 ++++- release-notes/v5-lincoln/5.3.md | 2 +- 9 files changed, 69 insertions(+), 41 deletions(-) diff --git a/learn/developers/full-text-search-product-catalog.mdx b/learn/developers/full-text-search-product-catalog.mdx index d99debec0..875968be6 100644 --- a/learn/developers/full-text-search-product-catalog.mdx +++ b/learn/developers/full-text-search-product-catalog.mdx @@ -71,10 +71,10 @@ Harper commits the record first, then applies the committed change to the local ## 3. Search from a Harper application ```javascript title="resources.js" -export class SearchProducts extends tables.Product { +export class SearchProducts extends databases.catalog.Product { static async get(target) { const text = target.get('q') ?? ''; - return tables.Product.search({ + return super.search({ conditions: [ { attribute: 'catalogSearch', comparator: 'matches', value: text, includeHighlights: true }, { attribute: 'category', comparator: 'equals', value: 'footwear' }, @@ -90,6 +90,8 @@ Call the resource with `GET /SearchProducts?q=trail`. The full-text index provides ranked candidates. The indexed category condition narrows the result set, and Harper loads the current records before returning them. +Calls through `tables` and `databases` use Harper's [trusted server-side context](/reference/v5/components/javascript-environment#tables). This example is appropriate when the selected catalog fields are public. If access depends on the caller, authorize the request in `get()` before returning results; the underlying table's role and attribute permissions are not reapplied automatically. + ## 4. Search through REST The exported table supports the same match comparator: @@ -117,7 +119,7 @@ Prefix search returns matching products. It does not return a dictionary of sugg Most searches can use the default three-second lag tolerance. When the next request must find a record that was just written, use the JavaScript API and wait for current index coverage: ```javascript -const results = await tables.Product.search({ +const results = await databases.catalog.Product.search({ conditions: [ { attribute: 'catalogSearch', @@ -134,7 +136,7 @@ Do not apply this wait to every catalog request without measuring it. The normal ## 6. Operate the index -Use `describe_table` to inspect readiness. Harper reuses compatible local index files after restart and replays changes after their checkpoint. A node rebuilds locally when the files are missing, incompatible, or corrupt; table traffic remains available, while full-text requests return a retryable `503` until the rebuild is ready. +Use `describe_table` to inspect readiness. Harper reuses compatible local index files after restart and replays changes after their checkpoint. A node rebuilds locally when the files are missing, incompatible, or corrupt; table traffic remains available, while full-text requests return `503` with `code: "INDEX_REBUILDING"` and `retryable: true` until the rebuild is ready. Retry this code with backoff; do not treat every `503` as an index rebuild. See the [full-text configuration reference](/reference/v5/full-text-search/configuration) and [query reference](/reference/v5/full-text-search/querying) for every option and match mode. diff --git a/reference/database/overview.md b/reference/database/overview.md index 406b251a9..f4257eec7 100644 --- a/reference/database/overview.md +++ b/reference/database/overview.md @@ -107,6 +107,7 @@ It is best practice to define schemas for production tables. Dynamic schemas are For deeper coverage of each database feature, see the dedicated pages in this section: - **[Schema](./schema.md)** — Defining table structure, types, indexes, relationships, and computed properties using GraphQL schema syntax +- **[Full-Text Search](../full-text-search/overview.md)** — BM25-ranked search over stored text fields on audited RocksDB tables - **[API](./api.md)** — The `tables`, `databases`, `transaction()`, and `createBlob()` globals for interacting with the database from code - **[Data Loader](./data-loader.md)** — Loading seed or initial data into tables as part of component deployment - **[Storage Algorithm](./storage-algorithm.md)** — How Harper stores data using LMDB with universal indexing and ACID compliance diff --git a/reference/database/schema.md b/reference/database/schema.md index 9516119c5..34514a71e 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -258,6 +258,25 @@ type InternalConfig @table @hidden { `@hidden` is also available as a [field directive](#hidden-field-directive) to suppress individual attributes. +### `@fullText` + + + +Creates a BM25-ranked full-text derived index from one or more stored text fields. The directive is declared on the table type and can be repeated for multiple independent indexes. + +```graphql +type Product + @table(database: "catalog", audit: true) + @fullText(name: "catalogSearch", fields: [{ name: "name", weight: 3 }, { name: "description" }, { name: "tags" }]) { + id: ID @primaryKey + name: String + description: String + tags: [String] +} +``` + +Full-text indexes require RocksDB and an audited table. They are local derived state rebuilt from committed table data rather than authoritative record storage. See [Full-Text Search Configuration](../full-text-search/configuration.md) for fields, phrase and prefix storage, synonyms, highlighting, and rebuild behavior. + ## Documenting Types and Fields Harper picks up GraphQL's standard triple-quoted docstrings on type and field definitions. Docstrings flow through to: @@ -292,25 +311,6 @@ Docstrings on `@hidden` fields are dropped from the descriptive surfaces alongsi > **Trust model.** Docstrings reach LLMs and public OpenAPI consumers verbatim. Treat them as code: don't put secrets, internal-only commentary, or speculative prose in them. Use `@hidden` to suppress fields that shouldn't surface publicly. -### `@fullText` - - - -Creates a BM25-ranked full-text derived index from one or more stored text fields. The directive is declared on the table type and can be repeated for multiple independent indexes. - -```graphql -type Product - @table(database: "catalog", audit: true) - @fullText(name: "catalogSearch", fields: [{ name: "name", weight: 3 }, { name: "description" }, { name: "tags" }]) { - id: ID @primaryKey - name: String - description: String - tags: [String] -} -``` - -Full-text indexes require RocksDB and an audited table. They are local derived state rebuilt from committed table data rather than authoritative record storage. See [Full-Text Search Configuration](../full-text-search/configuration.md) for fields, phrase and prefix storage, synonyms, highlighting, and rebuild behavior. - ## Field Directives Field directives apply to individual attributes in a type definition. diff --git a/reference/full-text-search/configuration.md b/reference/full-text-search/configuration.md index 12c1c2f33..2bd2e76c6 100644 --- a/reference/full-text-search/configuration.md +++ b/reference/full-text-search/configuration.md @@ -4,9 +4,9 @@ title: Configuration # Full-Text Search Configuration - + -Declare a full-text index with `@fullText` on a table type. The directive is repeatable, so one table can have independent indexes for different search experiences. +Declare a full-text index with `@fullText` on an audited RocksDB table type. The directive is repeatable, so one table can have independent indexes for different search experiences. ```graphql type Product @@ -176,3 +176,5 @@ The table response includes entries shaped like this: ``` `readiness.state` is `ready`, `rebuilding`, or `unavailable`. `readiness.reason` is included when Harper has more detail. Full-text queries are served only after readiness is established. + +The Operations API uses snake_case for response metadata such as `stop_words`, `surface_terms`, and `owner_epoch`. The nested `highlighting` value mirrors the GraphQL option names `maxFragments` and `fragmentLength`. diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 0f99c4136..b221cf80f 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -4,7 +4,7 @@ title: Querying # Querying Full-Text Indexes - + Use the full-text index name as a condition attribute in `Table.search()`. @@ -59,7 +59,9 @@ const products = await Product.search({ }); ``` -Every requested field must belong to the index and be readable by the caller. +Every requested field must belong to the index and be readable by the caller. Without `fields`, the caller must be allowed to read every source field in the index. Harper rejects an external REST or Operations API query with `403` if any searched source field is not readable. `$highlights` never bypasses this check. + +Direct calls through `tables` or `databases` run in a trusted server-side context and do not reapply the caller's role or attribute permissions. Custom resources must authorize protected searches before returning records or highlights. ## Score and ordering @@ -171,7 +173,7 @@ const products = await Product.search({ All full-text conditions combined into one query must use the same freshness values. Harper returns `400` when combined conditions specify different values. -A non-waiting HTTP query returns `Harper-Index-Coverage` with the admitted state, lag upper bound, and requested tolerance. Waiting queries, available through `Table.search()` and `search_by_conditions`, establish coverage while the response stream is consumed and do not emit that header before completion. +A non-waiting HTTP query returns `Harper-Index-Coverage` with the admitted state, lag upper bound, and requested tolerance. A waiting `Table.search()` establishes coverage as its iterator is consumed. A `search_by_conditions` request completes only after its wait and search finish. Waiting queries do not emit a coverage header before completion. ## Prefix result window @@ -189,7 +191,15 @@ GET /Product/?catalogSearch=matches_phrase=trail%20running&select(id,name,$score GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&limit(10) ``` -REST supports all positive and negated full-text comparators. It can express the query text, selected fields, and pagination. Use `Table.search()` or `search_by_conditions` when a condition needs `fields`, highlighting flags, or freshness controls. +REST supports all positive and negated full-text comparators. It can express the query text, selected fields, and pagination. + +Repeat the index parameter to combine the required positive and negated conditions: + +```http +GET /Product/?catalogSearch=matches=waterproof&catalogSearch=not_matches=leather&limit(20) +``` + +REST can return configured highlights by selecting `$highlights`. Use `Table.search()` or `search_by_conditions` for explicit `fields`, condition-level `includeHighlights`, or freshness controls. ## Operations API diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index cdcf7f5e2..34f439cc3 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -11,14 +11,6 @@ title: Query Optimization Harper has powerful query functionality with excellent performance characteristics. Like any database, different queries can vary significantly in performance. Understanding how querying works helps you write queries that perform well as your dataset grows. -## Full-text search - - - -A condition using a declared `@fullText` index is executed by the native full-text engine and ranked with BM25. Compatible `and` filters are evaluated while accepting candidates so a selective structured filter can still fill the requested result page. Harper then loads the current source records and removes stale or deleted candidates. - -Keep result pages bounded. Any expression containing prefix or fuzzy-prefix matching has a 100-record native window and is intended for autocomplete-style record lookup. An unbounded query that exceeds the window fails instead of returning a partial result. Use source-field weights to tune ranking, and enable positions, surface terms, and highlighting only when their query features are needed. See [Full-Text Search](../full-text-search/overview.md). - ## Query Execution At a fundamental level, querying involves defining conditions to find matching data and then executing those conditions against the database. Harper supports indexed fields, and these indexes are used to speed up query execution. @@ -87,6 +79,14 @@ When a secondary index is still being built — for example, just after a new `@ This is a transient condition — the index finishes building in the background, after which the same query succeeds. Clients should retry (with backoff) rather than treating it as an empty result or a permanent failure. On the operations API the error body includes `code` and `retryable`, so callers can branch on `error.code === 'INDEX_REBUILDING'` or `error.retryable`. +### Full-text search + + + +A condition using a declared `@fullText` index is executed by the native full-text engine and ranked with BM25. Compatible `and` filters are evaluated while accepting candidates so a selective structured filter can still fill the requested result page. Harper then loads the current source records and removes stale or deleted candidates. + +Keep result pages bounded. Any expression containing prefix or fuzzy-prefix matching has a 100-record native window and is intended for autocomplete-style record lookup. An unbounded query that exceeds the window fails instead of returning a partial result. Use source-field weights to tune ranking, and enable positions, surface terms, and highlighting only when their query features are needed. See [Full-Text Search](../full-text-search/overview.md). + ## Relationships and Joins Harper supports relationship-based queries that join data across tables. See [Schema documentation](../database/schema.md) for how to define relationships. diff --git a/reference/rest/headers.md b/reference/rest/headers.md index 6fd4c17f7..99bb54b97 100644 --- a/reference/rest/headers.md +++ b/reference/rest/headers.md @@ -22,7 +22,14 @@ These headers are included in all Harper REST API responses: Collection responses to a [count request](./querying.md#pagination-and-total-count) additionally include `Content-Range`, `Range-Unit`, and `Preference-Applied` (). -A non-waiting request that uses a native derived index can include `Harper-Index-Coverage`. For full-text search, a value such as `current; lag=0; tolerance=3000` reports the admitted state, lag upper bound, and requested tolerance in milliseconds. See [Full-text freshness controls](../full-text-search/querying.md#freshness-controls). + + +A non-waiting request that uses a native derived index can include `Harper-Index-Coverage`. Its value is one of: + +- `current; lag=0; tolerance=` — the index covered current committed data at admission. +- `bounded; lag=; tolerance=` — the admitted lag upper bound was within the requested tolerance. + +Multiple native searches append one value each. The header certifies coverage when the search is admitted; it does not certify a later traversal. See [Full-text freshness controls](../full-text-search/querying.md#freshness-controls). ## Cache-Control diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 537fa8a6e..e367b5270 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -66,7 +66,7 @@ GET /Product/?category=software&price=gt=100&price=lt=200 ### Full-Text Operators - + For a declared `@fullText` index, use its name as the query attribute and one of the full-text comparators: @@ -76,7 +76,13 @@ GET /Product/?catalogSearch=matches_phrase=trail%20running GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&limit(10) ``` -REST supports `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix`, plus a `not_` form of each. A negated full-text condition requires a non-negated condition on the same full-text index, which may require `Table.search()` or `search_by_conditions` for combined conditions. See [Querying Full-Text Indexes](../full-text-search/querying.md) for ranking, highlights, query combinations, and freshness controls. +REST supports `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix`, plus a `not_` form of each. A negated full-text condition requires a non-negated condition on the same index. Repeat the parameter to provide both conditions: + +```http +GET /Product/?catalogSearch=matches=waterproof&catalogSearch=not_matches=leather +``` + +Use `Table.search()` or `search_by_conditions` when conditions need source-field selection, condition-level highlighting, or freshness controls. See [Querying Full-Text Indexes](../full-text-search/querying.md) for ranking, highlights, query combinations, and freshness controls. For date fields, colons must be URL-encoded as `%3A`: diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index 6522ce93f..6c781b4b8 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -10,7 +10,7 @@ Harper tables can declare native BM25-ranked indexes with `@fullText` and query The indexes are derived locally from committed audited RocksDB records. Each replica maintains its own files, reuses a compatible checkpoint after restart, and rebuilds locally when necessary. See [Full-Text Search](/reference/v5/full-text-search/overview) and [Add Full-Text Search to a Product Catalog](/learn/developers/full-text-search-product-catalog). -### Patch Releases +## Patch Releases All patch release notes for 5.3.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.3&expanded=true). From 639e80c158a72c28a95f1755722bd583cfd36c46 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 28 Sep 2026 07:31:18 -0600 Subject: [PATCH 05/78] docs(fulltext): clarify operational behavior --- .../full-text-search-product-catalog.mdx | 17 +++++++++++------ reference/database/overview.md | 2 +- reference/full-text-search/configuration.md | 4 +++- reference/full-text-search/overview.md | 13 ++++++++++++- reference/full-text-search/querying.md | 2 +- reference/operations-api/operations.md | 10 ++++++++-- reference/resources/resource-api.md | 7 +++++++ release-notes/v5-lincoln/5.3.md | 8 ++++---- 8 files changed, 47 insertions(+), 16 deletions(-) diff --git a/learn/developers/full-text-search-product-catalog.mdx b/learn/developers/full-text-search-product-catalog.mdx index 875968be6..90fe31c7f 100644 --- a/learn/developers/full-text-search-product-catalog.mdx +++ b/learn/developers/full-text-search-product-catalog.mdx @@ -73,12 +73,17 @@ Harper commits the record first, then applies the committed change to the local ```javascript title="resources.js" export class SearchProducts extends databases.catalog.Product { static async get(target) { - const text = target.get('q') ?? ''; + const text = target.get('q')?.trim(); + if (!text) { + const error = new Error('q is required'); + error.statusCode = 400; + throw error; + } + const conditions = [{ attribute: 'catalogSearch', comparator: 'matches', value: text, includeHighlights: true }]; + const category = target.get('category'); + if (category) conditions.push({ attribute: 'category', comparator: 'equals', value: category }); return super.search({ - conditions: [ - { attribute: 'catalogSearch', comparator: 'matches', value: text, includeHighlights: true }, - { attribute: 'category', comparator: 'equals', value: 'footwear' }, - ], + conditions, select: ['id', 'name', 'price', '$score', '$highlights'], limit: 20, }); @@ -86,7 +91,7 @@ export class SearchProducts extends databases.catalog.Product { } ``` -Call the resource with `GET /SearchProducts?q=trail`. +Call the resource with `GET /SearchProducts?q=trail&category=footwear`. The category filter is optional; `q` is required. The full-text index provides ranked candidates. The indexed category condition narrows the result set, and Harper loads the current records before returning them. diff --git a/reference/database/overview.md b/reference/database/overview.md index f4257eec7..02924042f 100644 --- a/reference/database/overview.md +++ b/reference/database/overview.md @@ -104,7 +104,7 @@ It is best practice to define schemas for production tables. Dynamic schemas are ## Key Concepts -For deeper coverage of each database feature, see the dedicated pages in this section: +For deeper coverage of database features, see these related pages: - **[Schema](./schema.md)** — Defining table structure, types, indexes, relationships, and computed properties using GraphQL schema syntax - **[Full-Text Search](../full-text-search/overview.md)** — BM25-ranked search over stored text fields on audited RocksDB tables diff --git a/reference/full-text-search/configuration.md b/reference/full-text-search/configuration.md index 2bd2e76c6..5ae720aab 100644 --- a/reference/full-text-search/configuration.md +++ b/reference/full-text-search/configuration.md @@ -175,6 +175,8 @@ The table response includes entries shaped like this: } ``` -`readiness.state` is `ready`, `rebuilding`, or `unavailable`. `readiness.reason` is included when Harper has more detail. Full-text queries are served only after readiness is established. +`readiness.state` is `ready`, `rebuilding`, or `unavailable`. A rebuilding query returns retryable `INDEX_REBUILDING`. `unavailable` means activation failed or the automatic rebuild budget was exhausted; inspect `readiness.reason` and the node logs, correct the underlying storage, native-module, schema, or audit-log problem, then reload or reapply the schema to retry activation. Full-text queries are served only when the state is `ready`. + +`query_modes` maps to comparators as follows: `any` → `matches`, `all` → `matches_all`, `fuzzy` → `matches_fuzzy`, `phrase` → `matches_phrase`, `prefix` → `matches_prefix`, and `fuzzy-prefix` → `matches_fuzzy_prefix`. `owner_epoch` is an opaque decimal string used to fence work between derived-index owners; clients should not parse or persist it. `rebuild_attempts` is the current automatic-attempt count. The Operations API uses snake_case for response metadata such as `stop_words`, `surface_terms`, and `owner_epoch`. The nested `highlighting` value mirrors the GraphQL option names `maxFragments` and `fragmentLength`. diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index 9024c291b..d55cd54db 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -12,7 +12,7 @@ Use full-text search when users need relevance-ranked matching across product na ## Requirements -- The table must use RocksDB and have `audit: true`. +- The table must use RocksDB and explicitly declare [`audit: true`](../database/transaction.md#enabling-the-transaction-log-per-table). A new or updated declaration without it fails with `400`, even when transaction logging is enabled globally. - Full-text source fields must be stored `String`, `[String]`, or `Blob` values. Blob sources must be declared as `text/plain`. - Creating or updating an LMDB table with `@fullText` fails with `400`. A persisted declaration encountered while opening LMDB is ignored with a warning. Migrate the table to RocksDB before activating the index. @@ -44,6 +44,8 @@ This design has two consequences: On restart, Harper reuses compatible local index files and replays committed changes after the saved checkpoint. If those files are missing, incompatible, or corrupt, Harper rebuilds the index locally. +A rebuild scans the current table and replays changes committed during the scan. An unreadable, corrupt, or insufficient audit-log prefix can make a rebuild fail. Harper retries with backoff; after the retry budget is exhausted, readiness becomes `unavailable`. Inspect `describe_table` and the node logs for the reason before retrying activation. + Ordinary table reads and writes remain available during a rebuild. Full-text queries return `503` with `code: "INDEX_REBUILDING"` and `retryable: true` until the index is ready. A new replica follows the same process before serving full-text queries. Use `describe_table` to inspect each index's declaration and readiness. See [Inspecting an index](./configuration.md#inspecting-an-index). @@ -54,6 +56,15 @@ The native index ranks matching documents with BM25. Harper then loads the autho Full-text search can be combined with structured filters. Harper pushes compatible filters into candidate evaluation so selective filters do not silently under-fill a requested page. +## Limits and troubleshooting + +- Phrase search requires `positions: true`; prefix and fuzzy-prefix require `surfaceTerms: true`. +- Prefix expressions have a 100-record native result window. Use a bounded `limit`; requests beyond the window fail instead of truncating silently. +- One query can use only one full-text index. An `or` group cannot mix full-text and ordinary record conditions. +- `INDEX_REBUILDING` is transient and retryable. `unavailable` means automatic rebuild attempts were exhausted or activation failed; inspect readiness and logs rather than retrying every `503` indefinitely. + +See [Querying](./querying.md) for exact condition rules and [Configuration](./configuration.md#inspecting-an-index) for readiness details. + ## Stored index data The local index stores tokenized terms, posting lists, optional positions and surface terms, and checkpoint metadata. It does not store an authoritative copy of each record. Harper always loads current table records before returning results, and it can discard and rebuild this derived state from committed data. diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index b221cf80f..4c1cbc5f9 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -173,7 +173,7 @@ const products = await Product.search({ All full-text conditions combined into one query must use the same freshness values. Harper returns `400` when combined conditions specify different values. -A non-waiting HTTP query returns `Harper-Index-Coverage` with the admitted state, lag upper bound, and requested tolerance. A waiting `Table.search()` establishes coverage as its iterator is consumed. A `search_by_conditions` request completes only after its wait and search finish. Waiting queries do not emit a coverage header before completion. +A non-waiting HTTP query can return `Harper-Index-Coverage` with the admitted state, lag upper bound, and requested tolerance. A zero-size page performs no native search and carries no coverage proof. A waiting `Table.search()` establishes coverage as its iterator is consumed. A `search_by_conditions` request completes only after its wait and search finish. Waiting queries do not emit a coverage header before completion. ## Prefix result window diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 31a0e7bb6..2944b82b4 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -65,10 +65,14 @@ Returns all table definitions within the specified database. Each table definiti Returns the definition of a specific table. A table with native full-text indexes includes `full_text_indexes`; each entry reports its source fields, analyzer options, enabled query modes, and `readiness` (`ready`, `rebuilding`, or `unavailable`). The readiness object also includes `owner_epoch`, `rebuild_attempts`, and an optional `reason`. +Full-text response metadata: + ```json -{ "operation": "describe_table", "table": "dog", "database": "dev" } +{ "operation": "describe_table", "table": "Product", "database": "catalog" } ``` +The response also includes the table's ordinary schema and storage metadata. The relevant full-text fragment is: + ```json { "name": "Product", @@ -299,7 +303,9 @@ Returns records matching one or more conditions. Supports `operator` (`and`/`or` } ``` -Conditions can also query a declared full-text index with the `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix` comparators and their `not_` forms. A negated full-text condition requires a non-negated condition on the same full-text index. The Operations API preserves advanced condition options such as source `fields`, `includeHighlights`, `maxIndexLagMilliseconds`, and `waitForIndexMilliseconds`. See [Querying Full-Text Indexes](../full-text-search/querying.md#operations-api). +Full-text conditions: + +Conditions can query a declared full-text index with the `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix` comparators and their `not_` forms. A negated full-text condition requires a non-negated condition on the same full-text index. The Operations API preserves advanced condition options such as source `fields`, `includeHighlights`, `maxIndexLagMilliseconds`, and `waitForIndexMilliseconds`. See [Querying Full-Text Indexes](../full-text-search/querying.md#operations-api). --- diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index fe9297347..929e269db 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -980,6 +980,8 @@ Array of condition objects to filter records. Each condition: | `conditions` | Nested conditions array | | `operator` | `and` (default) or `or` for the nested `conditions` | +Full-text comparators and condition options: + Example with nested conditions: ```javascript @@ -1030,6 +1032,11 @@ Special properties: - `$id` — Returns the primary key regardless of its name - `$updatedtime` — Returns the last-updated timestamp - `$distance` — When the query ranks or filters by a vector index, returns the computed distance from the target vector. See [Vector Indexing](../database/schema.md#vector-indexing). + + + +Full-text queries add two special properties: + - `$score` — When a query uses a full-text index, returns its BM25 relevance score. - `$highlights` — Enables and returns full-text fragments and matching spans when the index has highlighting configured. See [Highlights](../full-text-search/querying.md#highlights). diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index 6c781b4b8..1386d7b3f 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -4,16 +4,16 @@ title: '5.3' # 5.3 Release Notes +### Patch Releases + +All patch release notes for 5.3.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.3&expanded=true). + ## Full-Text Search Harper tables can declare native BM25-ranked indexes with `@fullText` and query them through `Table.search()`, REST, or `search_by_conditions`. Search supports any-term, all-term, phrase, prefix, fuzzy, and fuzzy-prefix matching, with optional source weights, synonyms, scores, and highlights. The indexes are derived locally from committed audited RocksDB records. Each replica maintains its own files, reuses a compatible checkpoint after restart, and rebuilds locally when necessary. See [Full-Text Search](/reference/v5/full-text-search/overview) and [Add Full-Text Search to a Product Catalog](/learn/developers/full-text-search-product-catalog). -## Patch Releases - -All patch release notes for 5.3.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.3&expanded=true). - ## Node Identity `node.hostname` and `replication.hostname` must now be a bare hostname or IP literal. This value is the node's identity — it becomes the node's TLS certificate common name and the host that replication advertises to peers and dials to reach the node — so a URL or `host:port` value corrupted certificate matching and replication without surfacing an error: a node configured as `http://host:9926` advertised and dialed a host literally named `http` ([harper#2218](https://github.com/HarperFast/harper/issues/2218)). From b7ad21790b3965f8c7c600a1fd8386358ad90e36 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 28 Sep 2026 07:34:19 -0600 Subject: [PATCH 06/78] docs(release): keep patch releases at heading level two --- release-notes/v5-lincoln/5.3.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index 1386d7b3f..e05284d25 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -4,7 +4,7 @@ title: '5.3' # 5.3 Release Notes -### Patch Releases +## Patch Releases All patch release notes for 5.3.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.3&expanded=true). From e3379b8d3550450b827a312eaad16a7dda00bc0f Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 28 Sep 2026 07:49:41 -0600 Subject: [PATCH 07/78] docs(fulltext): define query edge cases --- reference/full-text-search/configuration.md | 2 ++ reference/full-text-search/overview.md | 2 +- reference/full-text-search/querying.md | 4 ++++ 3 files changed, 7 insertions(+), 1 deletion(-) diff --git a/reference/full-text-search/configuration.md b/reference/full-text-search/configuration.md index 5ae720aab..1bef52505 100644 --- a/reference/full-text-search/configuration.md +++ b/reference/full-text-search/configuration.md @@ -140,6 +140,8 @@ Harper validates the Blob's media type when applying it to the index. Changes that affect indexed storage create a new local index generation and rebuild from source data. These include source names and media types, analyzer settings, stop-word handling, positions, surface terms, and synonyms. +While a generation rebuilds, ordinary table reads and writes remain available, but full-text queries on that node return retryable `503` responses until the index is ready. + Changing source weights or highlighting settings changes query behavior without rebuilding indexed term storage. Removing an index retires its local files after the schema change is confirmed. ## Inspecting an index diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index d55cd54db..6d1f9aa8e 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -44,7 +44,7 @@ This design has two consequences: On restart, Harper reuses compatible local index files and replays committed changes after the saved checkpoint. If those files are missing, incompatible, or corrupt, Harper rebuilds the index locally. -A rebuild scans the current table and replays changes committed during the scan. An unreadable, corrupt, or insufficient audit-log prefix can make a rebuild fail. Harper retries with backoff; after the retry budget is exhausted, readiness becomes `unavailable`. Inspect `describe_table` and the node logs for the reason before retrying activation. +A rebuild scans the current table and replays changes committed during the scan. An unreadable or corrupt transaction log can make a rebuild fail. The same applies when [transaction-log cleanup](../database/transaction.md#delete_transaction_logs_before) or automatic retention removes history required to bridge the scan to current writes. Harper retries with backoff; after the retry budget is exhausted, readiness becomes `unavailable`. Inspect `describe_table` and the node logs for the reason before retrying activation. Ordinary table reads and writes remain available during a rebuild. Full-text queries return `503` with `code: "INDEX_REBUILDING"` and `retryable: true` until the index is ready. A new replica follows the same process before serving full-text queries. diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 4c1cbc5f9..0b772a6a9 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -30,6 +30,8 @@ Results are ordered by BM25 relevance. The index declaration, not the source fie Every comparator also has a negated form: `not_matches`, `not_matches_all`, `not_matches_phrase`, `not_matches_prefix`, `not_matches_fuzzy`, and `not_matches_fuzzy_prefix`. +An empty string is invalid and returns `400`. Whitespace-only text or text reduced to no terms by the analyzer, such as a query containing only enabled stop words, returns an exact empty result. + A query containing a negated full-text condition must also contain a non-negated full-text condition on the same index. A structured record condition alone does not satisfy this requirement. The positive full-text condition bounds the candidate set; the negated condition filters it. ```javascript @@ -175,6 +177,8 @@ All full-text conditions combined into one query must use the same freshness val A non-waiting HTTP query can return `Harper-Index-Coverage` with the admitted state, lag upper bound, and requested tolerance. A zero-size page performs no native search and carries no coverage proof. A waiting `Table.search()` establishes coverage as its iterator is consumed. A `search_by_conditions` request completes only after its wait and search finish. Waiting queries do not emit a coverage header before completion. +If acceptable coverage is not reached before `waitForIndexMilliseconds` expires, Harper returns `503` with `code: "DERIVED_INDEX_LAGGING"` and `retryable: true`. For `Table.search()`, the error is raised while the async iterator is consumed. An HTTP response may already be streaming, so clients should also handle a terminal stream error rather than relying only on the initial status. + ## Prefix result window `matches_prefix` and `matches_fuzzy_prefix` are autocomplete-style record searches. Any expression containing one of these modes uses a 100-record native result window, and `offset + limit` cannot exceed that window. If an unbounded query has more than 100 matches, Harper returns `400` and requires a limit instead of silently truncating the result. These modes return matching records, not a separate list of suggested terms. From 0954d11a7c9a144057f7518b3f08490b73f716b1 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 28 Sep 2026 08:10:04 -0600 Subject: [PATCH 08/78] docs(fulltext): resolve final contract gaps --- CONTRIBUTING.md | 2 +- .../full-text-search-product-catalog.mdx | 18 +++++++++++---- reference/backups/overview.md | 2 +- reference/database/schema.md | 1 + reference/database/transaction.md | 2 +- reference/full-text-search/configuration.md | 8 +++---- reference/full-text-search/overview.md | 3 ++- reference/full-text-search/querying.md | 2 +- reference/graphql-querying/overview.md | 2 ++ reference/resources/resource-api.md | 22 +++++++++++-------- reference/rest/headers.md | 4 ++++ src/components/EngineBadge.module.css | 7 ------ src/components/EngineBadge.tsx | 2 +- 13 files changed, 45 insertions(+), 30 deletions(-) delete mode 100644 src/components/EngineBadge.module.css diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dc6b96bed..98e6f2ac1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -229,7 +229,7 @@ The Plugin API is the primary way to implement additional functionality in Harpe - `logger.level` — Log level; _Default_: `"info"` (Added in: v4.1.0) ``` -For inline config option annotations inside list items, plain text `(Added in: vX.Y.Z)` is fine — using the component mid-sentence is awkward. Reserve `` for standalone placement after headings. +Use `` after headings or alongside the specific sentence, list item, or table entry it annotates. Use `` the same way when behavior is storage-engine-specific; comma-separated values such as `engines="RocksDB, LMDB"` are supported. Keep adjacent version and engine badges together on one line. For dense prose where a component would interrupt the sentence, plain text such as `(Added in: vX.Y.Z)` is fine. ## Redirects diff --git a/learn/developers/full-text-search-product-catalog.mdx b/learn/developers/full-text-search-product-catalog.mdx index 90fe31c7f..e7c4762f5 100644 --- a/learn/developers/full-text-search-product-catalog.mdx +++ b/learn/developers/full-text-search-product-catalog.mdx @@ -18,7 +18,7 @@ This guide adds relevance-ranked text search to a product catalog, then exposes ## Prerequisites - Harper 5.3 or later using RocksDB -- A component with a GraphQL schema and JavaScript resources +- A component with a GraphQL schema, JavaScript resources, and `rest: true` in its `config.yaml` ## 1. Declare the index @@ -79,7 +79,15 @@ export class SearchProducts extends databases.catalog.Product { error.statusCode = 400; throw error; } - const conditions = [{ attribute: 'catalogSearch', comparator: 'matches', value: text, includeHighlights: true }]; + const conditions = [ + { + attribute: 'catalogSearch', + comparator: 'matches', + value: text, + fields: ['name', 'description', 'tags'], + includeHighlights: true, + }, + ]; const category = target.get('category'); if (category) conditions.push({ attribute: 'category', comparator: 'equals', value: category }); return super.search({ @@ -91,11 +99,11 @@ export class SearchProducts extends databases.catalog.Product { } ``` -Call the resource with `GET /SearchProducts?q=trail&category=footwear`. The category filter is optional; `q` is required. +Call the resource with `GET /SearchProducts/?q=trail&category=footwear`. The category filter is optional; `q` is required. The full-text index provides ranked candidates. The indexed category condition narrows the result set, and Harper loads the current records before returning them. -Calls through `tables` and `databases` use Harper's [trusted server-side context](/reference/v5/components/javascript-environment#tables). This example is appropriate when the selected catalog fields are public. If access depends on the caller, authorize the request in `get()` before returning results; the underlying table's role and attribute permissions are not reapplied automatically. +Calls through `tables` and `databases` use Harper's [trusted server-side context](/reference/v5/components/javascript-environment#tables). This example is appropriate because every field in `fields` is public. If access depends on the caller, authorize the request in `get()` and restrict `fields` before searching; the underlying table's role and attribute permissions are not reapplied automatically, and highlights can reveal searched fields even when those fields are absent from `select`. ## 4. Search through REST @@ -139,6 +147,8 @@ const results = await databases.catalog.Product.search({ Do not apply this wait to every catalog request without measuring it. The normal bounded-staleness path keeps search traffic independent of short indexing bursts. +If the index does not reach current coverage within ten seconds, consuming the search returns retryable `503` code `DERIVED_INDEX_LAGGING`. See [freshness controls](/reference/v5/full-text-search/querying#freshness-controls). + ## 6. Operate the index Use `describe_table` to inspect readiness. Harper reuses compatible local index files after restart and replays changes after their checkpoint. A node rebuilds locally when the files are missing, incompatible, or corrupt; table traffic remains available, while full-text requests return `503` with `code: "INDEX_REBUILDING"` and `retryable: true` until the rebuild is ready. Retry this code with backoff; do not treat every `503` as an index rebuild. diff --git a/reference/backups/overview.md b/reference/backups/overview.md index 38a1ac740..3135ddd63 100644 --- a/reference/backups/overview.md +++ b/reference/backups/overview.md @@ -38,7 +38,7 @@ The offline path matters for restore. RocksDB is single-writer, so an in-place r - **Backups live on the node that created them.** The backup repository is a local directory. RocksDB shares files across backup IDs (a backup ID is not a self-contained folder), so disaster-recovery copies must take the entire per-database repository — `/` — not an individual backup, and must do so while no backup operation is running, or from an atomic filesystem snapshot. A live recursive copy can race `create_backup`/`delete_backup`/`purge_backups` and produce an unrestorable copy. Alternatively, use `get_backup` to pull a snapshot from a running server. - **`get_backup` always streams the current state.** It cannot download a historical managed backup; to move a retained backup off-host, copy its whole per-database repository as above. - **A restore is a point-in-time rollback.** In a replicated cluster, coordinate a restore with replication before bringing the node back. -- **Full-text indexes rebuild locally after restore.** Ordinary table traffic remains available, but full-text requests return `INDEX_REBUILDING` until the restored node catches up. +- **Full-text indexes rebuild locally after restore.** Ordinary table traffic remains available. While the rebuild is active, full-text requests return `503` with `code: "INDEX_REBUILDING"` and `retryable: true`; branch on the code because other `503` responses are not necessarily rebuilds. - **An interrupted restore leaves the database unloadable.** If `restore_backup` is interrupted before completing (crash, power loss), Harper marks the database as incompletely restored and skips loading it on the next start, logging an incomplete-restore error. Rerun `restore_backup` for the same database and `backup_id` to recover; do not load or hand-repair the directory. ### When can a database be restored? diff --git a/reference/database/schema.md b/reference/database/schema.md index 34514a71e..954370569 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -81,6 +81,7 @@ Optional arguments: | `eviction` | `Int` | `0` | Additional seconds after `expiration` before a record is physically removed | | `scanInterval` | `Int` | `(expiration + eviction) / 4` | Seconds between eviction scans | | `replicate` | `Boolean` | true | Enable replication of this table | +| `audit` | `Boolean` | `logging.auditLog` | Enable the table's transaction log; `@fullText` requires an explicit `true` | | `cacheControl` | `String` | — | `Cache-Control` header value emitted on anonymous GET/HEAD 200/304 responses for this table | | `randomAccessFields` | `Boolean` | `storage.randomAccessFields` | [Pin this table's record encoding](#randomaccessfields) | diff --git a/reference/database/transaction.md b/reference/database/transaction.md index baba4612e..005440fe8 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -134,7 +134,7 @@ Deletes transaction log entries older than the specified timestamp. Deprecated i Deletes transaction log entries older than the specified timestamp. :::warning Database-wide and irreversible -On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This purges whole log files whose entries predate the timestamp, removing [`read_audit_log`](#read_audit_log) history for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender detects that the requested start predates its retained history and forces a full base copy instead of incremental catch-up — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. +On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This purges whole log files whose entries predate the timestamp, removing [`read_audit_log`](#read_audit_log) history for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender detects that the requested start predates its retained history and forces a full base copy instead of incremental catch-up — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. The same history can be required to bridge a [full-text index rebuild](../full-text-search/overview.md#startup-and-recovery) from its table scan to current writes; aggressive cleanup can leave that index `unavailable` until its recovery source is restored and activation is retried. ::: Parameters: diff --git a/reference/full-text-search/configuration.md b/reference/full-text-search/configuration.md index 1bef52505..002a8cf6b 100644 --- a/reference/full-text-search/configuration.md +++ b/reference/full-text-search/configuration.md @@ -52,7 +52,7 @@ Each field entry supports: | `highlight` | `false` | Allows highlights from this field when index highlighting is enabled. | | `mediaType` | omitted | Required as `"text/plain"` for a `Blob` source; invalid on other source types. | -Source fields cannot use `@computed` or `@relationship`. At write time, a source value must be a string, an array containing strings or nulls, or a matching text Blob. +Source fields cannot use `@computed` or `@relationship`. Indexable source values are strings, arrays containing strings or nulls, or Blob values declared as plain text. ## Phrase and prefix support @@ -134,7 +134,7 @@ type Document } ``` -Harper validates the Blob's media type when applying it to the index. +`mediaType: "text/plain"` declares how Harper interprets the Blob; it does not validate runtime Blob metadata or reject the record write. The derived index decodes the Blob as UTF-8. A Blob that cannot be decoded or exceeds the one-MiB source-value limit is omitted from the index until a later valid write replaces it. ## Schema changes and rebuilds @@ -177,8 +177,8 @@ The table response includes entries shaped like this: } ``` -`readiness.state` is `ready`, `rebuilding`, or `unavailable`. A rebuilding query returns retryable `INDEX_REBUILDING`. `unavailable` means activation failed or the automatic rebuild budget was exhausted; inspect `readiness.reason` and the node logs, correct the underlying storage, native-module, schema, or audit-log problem, then reload or reapply the schema to retry activation. Full-text queries are served only when the state is `ready`. +`readiness.state` is `ready`, `rebuilding`, or `unavailable`. A rebuilding query returns `503` with retryable code `INDEX_REBUILDING`. `unavailable` means activation failed or the automatic rebuild budget was exhausted; queries return a generic, non-retryable `503`. Inspect the optional `readiness.reason` and the node logs, correct the underlying storage, native-module, schema, or transaction-log problem, then reload or reapply the schema to retry activation. Full-text queries are served only when the state is `ready`. -`query_modes` maps to comparators as follows: `any` → `matches`, `all` → `matches_all`, `fuzzy` → `matches_fuzzy`, `phrase` → `matches_phrase`, `prefix` → `matches_prefix`, and `fuzzy-prefix` → `matches_fuzzy_prefix`. `owner_epoch` is an opaque decimal string used to fence work between derived-index owners; clients should not parse or persist it. `rebuild_attempts` is the current automatic-attempt count. +`query_modes` maps to comparators as follows: `any` → `matches`, `all` → `matches_all`, `fuzzy` → `matches_fuzzy`, `phrase` → `matches_phrase`, `prefix` → `matches_prefix`, and `fuzzy-prefix` → `matches_fuzzy_prefix`. `owner_epoch` is an opaque decimal string used to fence work between derived-index owners; clients should not parse or persist it. `rebuild_attempts` is the current automatic-attempt count. `reason` is present when readiness has an operator-actionable explanation. The Operations API uses snake_case for response metadata such as `stop_words`, `surface_terms`, and `owner_epoch`. The nested `highlighting` value mirrors the GraphQL option names `maxFragments` and `fragmentLength`. diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index 6d1f9aa8e..c99cffa94 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -46,7 +46,7 @@ On restart, Harper reuses compatible local index files and replays committed cha A rebuild scans the current table and replays changes committed during the scan. An unreadable or corrupt transaction log can make a rebuild fail. The same applies when [transaction-log cleanup](../database/transaction.md#delete_transaction_logs_before) or automatic retention removes history required to bridge the scan to current writes. Harper retries with backoff; after the retry budget is exhausted, readiness becomes `unavailable`. Inspect `describe_table` and the node logs for the reason before retrying activation. -Ordinary table reads and writes remain available during a rebuild. Full-text queries return `503` with `code: "INDEX_REBUILDING"` and `retryable: true` until the index is ready. A new replica follows the same process before serving full-text queries. +Ordinary table reads and writes remain available during a rebuild. While readiness is `rebuilding`, full-text queries return `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. If rebuild attempts are exhausted and readiness becomes `unavailable`, queries return a generic, non-retryable `503` until an operator corrects the cause and retries activation. A new replica follows the same process before serving full-text queries. Use `describe_table` to inspect each index's declaration and readiness. See [Inspecting an index](./configuration.md#inspecting-an-index). @@ -62,6 +62,7 @@ Full-text search can be combined with structured filters. Harper pushes compatib - Prefix expressions have a 100-record native result window. Use a bounded `limit`; requests beyond the window fail instead of truncating silently. - One query can use only one full-text index. An `or` group cannot mix full-text and ordinary record conditions. - `INDEX_REBUILDING` is transient and retryable. `unavailable` means automatic rebuild attempts were exhausted or activation failed; inspect readiness and logs rather than retrying every `503` indefinitely. +- Generated GraphQL field arguments remain equality conditions; use `Table.search()`, REST, or `search_by_conditions` for full-text comparators. See [Querying](./querying.md) for exact condition rules and [Configuration](./configuration.md#inspecting-an-index) for readiness details. diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 0b772a6a9..65ff405ee 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -230,4 +230,4 @@ REST can return configured highlights by selecting `$highlights`. Use `Table.sea } ``` -If an index is rebuilding, the query returns `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. Other unavailable or busy paths can return a generic `503`, so clients should branch on the code rather than the status alone. Invalid declarations, unsupported match modes, incompatible combinations, and out-of-range options return `400` responses. +If an index is rebuilding, the query returns `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. An index in terminal `unavailable` state returns a generic, non-retryable `503`; inspect readiness and logs instead of retrying it as a rebuild. Other busy paths can also return a generic `503`, so clients should branch on the code rather than the status alone. Unreadable source fields return `403`. Invalid declarations, unsupported match modes, incompatible combinations, and out-of-range options return `400` responses. diff --git a/reference/graphql-querying/overview.md b/reference/graphql-querying/overview.md index fb56ffc31..6a429ec24 100644 --- a/reference/graphql-querying/overview.md +++ b/reference/graphql-querying/overview.md @@ -172,6 +172,8 @@ fragment sharedFields on Any { Any attribute can be used as an argument for a query. In this short form, multiple arguments is treated as multiple equivalency conditions with the default `and` operation. +These arguments always use equality. A declared `@fullText` index does not change GraphQL argument semantics; use [full-text search through `Table.search()`, REST, or the Operations API](../full-text-search/querying.md) for match comparators and BM25 ranking. + For example, the following query requires an `id` variable to be provided, and the system will search for a `Dog` record matching that id. ```graphql diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 929e269db..a218fc43b 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -972,15 +972,19 @@ The `Query` object is accepted by `search()` and the static `get()` method. Array of condition objects to filter records. Each condition: -| Property | Description | -| ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `attribute` | Property name, or an array for chained/joined properties (e.g. `['brand', 'name']`) | -| `value` | The value to match | -| `comparator` | `equals` (default), `greater_than`, `greater_than_equal`, `less_than`, `less_than_equal`, `starts_with`, `contains`, `ends_with`, `between`, `not_equal`, or a [full-text comparator](../full-text-search/querying.md#match-modes) | -| `conditions` | Nested conditions array | -| `operator` | `and` (default) or `or` for the nested `conditions` | - -Full-text comparators and condition options: +| Property | Description | +| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `attribute` | Property name, or an array for chained/joined properties (e.g. `['brand', 'name']`) | +| `value` | The value to match | +| `comparator` | `equals` (default), `greater_than`, `greater_than_equal`, `less_than`, `less_than_equal`, `starts_with`, `contains`, `ends_with`, `between`, `not_equal`, or a [full-text comparator](../full-text-search/querying.md#match-modes) | +| `conditions` | Nested conditions array | +| `operator` | `and` (default) or `or` for the nested `conditions` | +| `fields` | Full-text source fields to search; every name must belong to the selected full-text index | +| `includeHighlights` | Include configured full-text highlights | +| `maxIndexLagMilliseconds` | Maximum accepted full-text index lag; defaults to `3000` | +| `waitForIndexMilliseconds` | Time to wait for acceptable full-text coverage, from `0` through `30000` | + +The final four properties apply to full-text conditions: See [Querying Full-Text Indexes](../full-text-search/querying.md) for their constraints and error behavior. Example with nested conditions: diff --git a/reference/rest/headers.md b/reference/rest/headers.md index 99bb54b97..c0a2ce838 100644 --- a/reference/rest/headers.md +++ b/reference/rest/headers.md @@ -22,6 +22,8 @@ These headers are included in all Harper REST API responses: Collection responses to a [count request](./querying.md#pagination-and-total-count) additionally include `Content-Range`, `Range-Unit`, and `Preference-Applied` (). +### `Harper-Index-Coverage` + A non-waiting request that uses a native derived index can include `Harper-Index-Coverage`. Its value is one of: @@ -31,6 +33,8 @@ A non-waiting request that uses a native derived index can include `Harper-Index Multiple native searches append one value each. The header certifies coverage when the search is admitted; it does not certify a later traversal. See [Full-text freshness controls](../full-text-search/querying.md#freshness-controls). +When CORS is enabled, Harper adds this header to `Access-Control-Expose-Headers` so browser clients can read it. + ## Cache-Control diff --git a/src/components/EngineBadge.module.css b/src/components/EngineBadge.module.css deleted file mode 100644 index 6f748ecf4..000000000 --- a/src/components/EngineBadge.module.css +++ /dev/null @@ -1,7 +0,0 @@ -.badge { - display: inline-block; - font-size: 0.8rem; - color: var(--ifm-color-content-secondary); - font-style: italic; - margin-bottom: 0.75rem; -} diff --git a/src/components/EngineBadge.tsx b/src/components/EngineBadge.tsx index cf7e37cf2..a216f7f55 100644 --- a/src/components/EngineBadge.tsx +++ b/src/components/EngineBadge.tsx @@ -1,5 +1,5 @@ import React from 'react'; -import styles from './EngineBadge.module.css'; +import styles from './VersionBadge.module.css'; const CANONICAL_NAMES: Record = { rocksdb: 'RocksDB', From 6855bedb2009d622ae63d55499de529b71c43d01 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 28 Sep 2026 22:01:03 -0600 Subject: [PATCH 09/78] docs(fulltext): align with field-only schema --- .../full-text-search-product-catalog.mdx | 69 ++++++++---- reference/database/schema.md | 66 +++++------ reference/full-text-search/configuration.md | 105 +++++++++--------- reference/full-text-search/overview.md | 4 +- reference/full-text-search/querying.md | 10 +- reference/graphql-querying/overview.md | 2 +- reference/operations-api/operations.md | 2 +- reference/resources/query-optimization.md | 2 +- reference/rest/querying.md | 2 +- release-notes/v5-lincoln/5.3.md | 2 +- 10 files changed, 146 insertions(+), 118 deletions(-) diff --git a/learn/developers/full-text-search-product-catalog.mdx b/learn/developers/full-text-search-product-catalog.mdx index e7c4762f5..29ff17960 100644 --- a/learn/developers/full-text-search-product-catalog.mdx +++ b/learn/developers/full-text-search-product-catalog.mdx @@ -25,28 +25,26 @@ This guide adds relevance-ranked text search to a product catalog, then exposes Add `audit: true` to the RocksDB-backed table and declare an `@fullText` index over the fields customers search. ```graphql title="schema.graphql" -type Product - @table(database: "catalog", audit: true) - @export - @fullText( - name: "catalogSearch" - fields: [ - { name: "name", weight: 3, highlight: true } - { name: "description", weight: 1, highlight: true } - { name: "tags" } - ] - highlighting: { maxFragments: 2, fragmentLength: 120 } - ) { +type Product @table(database: "catalog", audit: true) @export { id: ID @primaryKey name: String description: String tags: [String] category: String @indexed price: Float @indexed + catalogSearch: FullText + @fullText( + fields: [ + { name: "name", weight: 3, highlight: true } + { name: "description", weight: 1, highlight: true } + { name: "tags" } + ] + highlighting: { maxFragments: 2, fragmentLength: 120 } + ) } ``` -`weight: 3` makes a match in `name` contribute more to BM25 relevance than the same match in `description` or `tags`. Highlighting is enabled only for the fields shown to customers. +The query-only field name `catalogSearch` identifies the index. It is not stored with each product. `weight: 3` makes a match in `name` contribute more to BM25 relevance than the same match in `description` or `tags`. Highlighting is enabled only for the fields shown to customers. ## 2. Add records @@ -71,9 +69,15 @@ Harper commits the record first, then applies the committed change to the local ## 3. Search from a Harper application ```javascript title="resources.js" -export class SearchProducts extends databases.catalog.Product { - static async get(target) { - const text = target.get('q')?.trim(); +export class SearchProducts extends Resource { + static loadAsInstance = false; + + allowCreate(user) { + return Boolean(user); + } + + async post(_target, body) { + const text = body.q?.trim(); if (!text) { const error = new Error('q is required'); error.statusCode = 400; @@ -88,22 +92,39 @@ export class SearchProducts extends databases.catalog.Product { includeHighlights: true, }, ]; - const category = target.get('category'); + const category = body.category; if (category) conditions.push({ attribute: 'category', comparator: 'equals', value: category }); - return super.search({ - conditions, - select: ['id', 'name', 'price', '$score', '$highlights'], - limit: 20, - }); + const records = []; + for await (const record of databases.catalog.Product.search( + { + conditions, + select: ['id', 'name', 'price', '$score', '$highlights'], + limit: 20, + checkPermission: true, + }, + this.getContext() + )) { + records.push(record); + } + return records; } } ``` -Call the resource with `GET /SearchProducts/?q=trail&category=footwear`. The category filter is optional; `q` is required. +Call the resource with an authenticated request: + +```http +POST /SearchProducts/ +Content-Type: application/json + +{ "q": "trail", "category": "footwear" } +``` + +The category filter is optional; `q` is required. The full-text index provides ranked candidates. The indexed category condition narrows the result set, and Harper loads the current records before returning them. -Calls through `tables` and `databases` use Harper's [trusted server-side context](/reference/v5/components/javascript-environment#tables). This example is appropriate because every field in `fields` is public. If access depends on the caller, authorize the request in `get()` and restrict `fields` before searching; the underlying table's role and attribute permissions are not reapplied automatically, and highlights can reveal searched fields even when those fields are absent from `select`. +Calls through `tables` and `databases` use Harper's [trusted server-side context](/reference/v5/components/javascript-environment#tables). Passing `this.getContext()` preserves the authenticated caller, and `checkPermission: true` asks the table to enforce that caller's table and source-field permissions. The custom resource must do both. Highlights can expose searched source text even when those fields are absent from `select`. ## 4. Search through REST diff --git a/reference/database/schema.md b/reference/database/schema.md index 954370569..c15bcf9f8 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -81,7 +81,7 @@ Optional arguments: | `eviction` | `Int` | `0` | Additional seconds after `expiration` before a record is physically removed | | `scanInterval` | `Int` | `(expiration + eviction) / 4` | Seconds between eviction scans | | `replicate` | `Boolean` | true | Enable replication of this table | -| `audit` | `Boolean` | `logging.auditLog` | Enable the table's transaction log; `@fullText` requires an explicit `true` | +| `audit` | `Boolean` | `logging.auditLog` | Enable the table's transaction log; new tables with `@fullText` must set this to `true` | | `cacheControl` | `String` | — | `Cache-Control` header value emitted on anonymous GET/HEAD 200/304 responses for this table | | `randomAccessFields` | `Boolean` | `storage.randomAccessFields` | [Pin this table's record encoding](#randomaccessfields) | @@ -259,25 +259,6 @@ type InternalConfig @table @hidden { `@hidden` is also available as a [field directive](#hidden-field-directive) to suppress individual attributes. -### `@fullText` - - - -Creates a BM25-ranked full-text derived index from one or more stored text fields. The directive is declared on the table type and can be repeated for multiple independent indexes. - -```graphql -type Product - @table(database: "catalog", audit: true) - @fullText(name: "catalogSearch", fields: [{ name: "name", weight: 3 }, { name: "description" }, { name: "tags" }]) { - id: ID @primaryKey - name: String - description: String - tags: [String] -} -``` - -Full-text indexes require RocksDB and an audited table. They are local derived state rebuilt from committed table data rather than authoritative record storage. See [Full-Text Search Configuration](../full-text-search/configuration.md) for fields, phrase and prefix storage, synonyms, highlighting, and rebuild behavior. - ## Documenting Types and Fields Harper picks up GraphQL's standard triple-quoted docstrings on type and field definitions. Docstrings flow through to: @@ -382,6 +363,24 @@ Write semantics: Multiple `@embed` attributes on one type are computed concurrently. +### `@fullText` + + + +Creates a BM25-ranked full-text derived index from one or more stored text fields. Declare the directive on a nullable `FullText` field; that field's name identifies the index. + +```graphql +type Product @table(database: "catalog", audit: true) { + id: ID @primaryKey + name: String + description: String + tags: [String] + catalogSearch: FullText @fullText(fields: [{ name: "name", weight: 3 }, { name: "description" }, { name: "tags" }]) +} +``` + +The `FullText` field is query-only. It is not stored with records or exposed as a selectable record property. Add another `FullText` field to create another independent index. Full-text indexes require RocksDB and an audited table. They are local derived state rebuilt from committed table data rather than authoritative record storage. See [Full-Text Search Configuration](../full-text-search/configuration.md) for the complete field contract, phrase and prefix storage, synonyms, highlighting, and rebuild behavior. + ### `@createdTime` Automatically assigns a creation timestamp (Unix epoch milliseconds) to the attribute when a record is created. @@ -811,19 +810,20 @@ Graph navigation runs on the quantized (approximate) distances. For nearest-neig Harper supports the following field types: -| Type | Description | -| --------- | ---------------------------------------------------------------------------------------------- | -| `String` | Unicode text, UTF-8 encoded | -| `Int` | 32-bit signed integer (−2,147,483,648 to 2,147,483,647) | -| `Long` | 54-bit signed integer (−9,007,199,254,740,992 to 9,007,199,254,740,992) | -| `Float` | 64-bit double precision floating point | -| `BigInt` | Integer up to ~300 digits. Note: distinct JavaScript type; handle appropriately in custom code | -| `Boolean` | `true` or `false` | -| `ID` | String; indicates a non-human-readable identifier | -| `Any` | Any primitive, object, or array | -| `Date` | JavaScript `Date` object | -| `Bytes` | Binary data as `Buffer` or `Uint8Array` | -| `Blob` | Binary large object; designed for streaming content >20KB | +| Type | Description | +| ---------- | ---------------------------------------------------------------------------------------------- | +| `String` | Unicode text, UTF-8 encoded | +| `Int` | 32-bit signed integer (−2,147,483,648 to 2,147,483,647) | +| `Long` | 54-bit signed integer (−9,007,199,254,740,992 to 9,007,199,254,740,992) | +| `Float` | 64-bit double precision floating point | +| `BigInt` | Integer up to ~300 digits. Note: distinct JavaScript type; handle appropriately in custom code | +| `Boolean` | `true` or `false` | +| `ID` | String; indicates a non-human-readable identifier | +| `Any` | Any primitive, object, or array | +| `Date` | JavaScript `Date` object | +| `Bytes` | Binary data as `Buffer` or `Uint8Array` | +| `Blob` | Binary large object; designed for streaming content >20KB | +| `FullText` | Query-only full-text index declaration; valid only with `@fullText` and never stored | Added `BigInt` in v4.3.0 diff --git a/reference/full-text-search/configuration.md b/reference/full-text-search/configuration.md index 002a8cf6b..ca0b33031 100644 --- a/reference/full-text-search/configuration.md +++ b/reference/full-text-search/configuration.md @@ -6,42 +6,50 @@ title: Configuration -Declare a full-text index with `@fullText` on an audited RocksDB table type. The directive is repeatable, so one table can have independent indexes for different search experiences. +Declare a nullable `FullText` field on an audited RocksDB table and attach `@fullText` to that field. The field name identifies the index. Add another `FullText` field when one table needs independent indexes for different search experiences. ```graphql -type Product - @table(database: "catalog", audit: true) - @fullText( - name: "catalogSearch" - fields: [ - { name: "name", weight: 3, highlight: true } - { name: "description", weight: 1, highlight: true } - { name: "tags", weight: 1 } - ] - highlighting: { maxFragments: 2, fragmentLength: 120 } - ) { +type Product @table(database: "catalog", audit: true) { id: ID @primaryKey name: String description: String tags: [String] price: Float @indexed + catalogSearch: FullText + @fullText( + fields: [ + { name: "name", weight: 3, highlight: true } + { name: "description", weight: 1, highlight: true } + { name: "tags", weight: 1 } + ] + highlighting: { maxFragments: 2, fragmentLength: 120 } + ) } ``` -The index name is the attribute used in search conditions. It does not add a stored field to each record. +`catalogSearch` is a query-only declaration. It is not stored, selectable, writable, or included in record, OpenAPI, or MCP schemas. Use the field name as the condition attribute and select `$score` or `$highlights` for search metadata. + +The declaration must use nullable `FullText` exactly and carry one `@fullText` directive. Lists, non-null forms, a bare `FullText` field, and additional directives such as `@computed`, `@indexed`, or `@allow` are rejected. `@fullText` is not valid on a table type or any other field type. + +New schema tables must explicitly set `@table(audit: true)`. An existing table that already has persisted audit logging can add the field without restating `audit`, but disabling audit logging while a full-text field exists is rejected. + +Writing the declaration-only field or selecting it as record data returns an error, including on unsealed tables. A read-through source response containing that name fails with `502` and is not cached, indexed, or audited. For HTTP-sourced caching tables, avoid index names that collide with response wrapper fields such as `data`, `headers`, `body`, and `status`. + +:::warning Prerelease schema declarations +Earlier prerelease type-level `@fullText(name: ...)` declarations are not supported. Move the directive to a nullable `FullText` field and use that field's name as the index identity. +::: ## Directive options -| Option | Default | Description | -| -------------- | ------------- | ------------------------------------------------------------------------------------ | -| `name` | required | Unique index name on the table. | -| `fields` | required | One or more source fields. | -| `analyzer` | `"english@2"` | Versioned text analyzer. `english@2` is the only supported value. | -| `stopWords` | `true` | Removes common English terms during analysis. | -| `positions` | `true` | Stores token positions. Required for phrase queries. | -| `surfaceTerms` | `true` | Stores original analyzed terms. Required for prefix, fuzzy-prefix, and highlighting. | -| `synonyms` | `[]` | Index-time synonym expansion rules. | -| `highlighting` | omitted | Enables highlight generation and sets fragment limits. | +| Option | Default | Description | +| -------------- | ------------- | ------------------------------------------------------------------------------------- | +| `fields` | required | One or more source fields. | +| `analyzer` | `"english@2"` | Versioned text analyzer. `english@2` is the only supported value. | +| `stopWords` | `true` | Removes common English terms during analysis. | +| `positions` | `true` | Stores token positions. Required for phrase queries. | +| `surfaceTerms` | `true` | Stores unstemmed analyzed terms. Required for prefix, fuzzy-prefix, and highlighting. | +| `synonyms` | `[]` | Index-time synonym expansion rules. | +| `highlighting` | omitted | Enables highlight generation and sets fragment limits. | Each field entry supports: @@ -59,11 +67,10 @@ Source fields cannot use `@computed` or `@relationship`. Indexable source values `positions` and `surfaceTerms` trade index size for query features. Keep their defaults unless storage measurements show that the features are unnecessary. ```graphql -type Article - @table(audit: true) - @fullText(name: "bodySearch", fields: [{ name: "body" }], positions: false, surfaceTerms: false) { +type Article @table(audit: true) { id: ID @primaryKey body: String + bodySearch: FullText @fullText(fields: [{ name: "body" }], positions: false, surfaceTerms: false) } ``` @@ -78,11 +85,11 @@ Highlighting is off by default. To enable it: 3. Keep `surfaceTerms: true`. ```graphql -@fullText( - name: "articleSearch" - fields: [{ name: "title", highlight: true }, { name: "body", highlight: true }] - highlighting: { maxFragments: 3, fragmentLength: 160 } -) +articleSearch: FullText + @fullText( + fields: [{ name: "title", highlight: true }, { name: "body", highlight: true }] + highlighting: { maxFragments: 3, fragmentLength: 160 } + ) ``` `maxFragments` and `fragmentLength` must be positive integers. Query results contain offsets into the original field value; see [Highlights](./querying.md#highlights). @@ -92,30 +99,29 @@ Highlighting is off by default. To enable it: Synonyms expand terms while records are indexed: ```graphql -@fullText( - name: "catalogSearch" - fields: [{ name: "description" }] - synonyms: [ - { source: "sneaker", replacements: ["shoe", "trainer"] } - { source: "tv", replacements: ["television"] } - ] -) +catalogSearch: FullText + @fullText( + fields: [{ name: "description" }] + synonyms: [ + { source: "sneaker", replacements: ["shoe", "trainer"] } + { source: "tv", replacements: ["television"] } + ] + ) ``` Rules require one source term and one or more unique replacement terms. A rule cannot replace a term with itself. ## Multiple indexes -Repeat `@fullText` when separate index definitions are useful: +Add separate `FullText` fields when separate index definitions are useful: ```graphql -type Product - @table(database: "catalog", audit: true) - @fullText(name: "titleSearch", fields: [{ name: "name", weight: 2 }]) - @fullText(name: "tagSearch", fields: [{ name: "tags" }]) { +type Product @table(database: "catalog", audit: true) { id: ID @primaryKey name: String tags: [String] + titleSearch: FullText @fullText(fields: [{ name: "name", weight: 2 }]) + tagSearch: FullText @fullText(fields: [{ name: "tags" }]) } ``` @@ -126,23 +132,22 @@ The indexes update independently and can be queried independently. A single quer Only plain-text Blob fields are supported: ```graphql -type Document - @table(audit: true) - @fullText(name: "contentSearch", fields: [{ name: "content", mediaType: "text/plain" }]) { +type Document @table(audit: true) { id: ID @primaryKey content: Blob + contentSearch: FullText @fullText(fields: [{ name: "content", mediaType: "text/plain" }]) } ``` -`mediaType: "text/plain"` declares how Harper interprets the Blob; it does not validate runtime Blob metadata or reject the record write. The derived index decodes the Blob as UTF-8. A Blob that cannot be decoded or exceeds the one-MiB source-value limit is omitted from the index until a later valid write replaces it. +`mediaType: "text/plain"` declares how Harper interprets the Blob; it does not validate runtime Blob metadata or reject the record write. The derived index decodes the Blob as UTF-8. If a Blob is invalid UTF-8 or exceeds the native source limit, Harper removes that whole record from the index rather than indexing only its other fields. A transient Blob read failure rolls back the accepted native batch and retries it, so index coverage does not advance past unread source data. ## Schema changes and rebuilds -Changes that affect indexed storage create a new local index generation and rebuild from source data. These include source names and media types, analyzer settings, stop-word handling, positions, surface terms, and synonyms. +Changes that affect indexed storage create a new local index generation and rebuild from source data. These include source membership, source order, media types, analyzer settings, stop-word handling, positions, surface terms, and synonyms. Keep the `fields` list in a stable order when its meaning has not changed. While a generation rebuilds, ordinary table reads and writes remain available, but full-text queries on that node return retryable `503` responses until the index is ready. -Changing source weights or highlighting settings changes query behavior without rebuilding indexed term storage. Removing an index retires its local files after the schema change is confirmed. +Changing source weights or highlighting settings changes query behavior without rebuilding indexed term storage. Removing the `FullText` field retires its local files after the schema change is confirmed. Renaming the field creates a new index identity and retires the old one. ## Inspecting an index diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index c99cffa94..77d017c62 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -12,9 +12,9 @@ Use full-text search when users need relevance-ranked matching across product na ## Requirements -- The table must use RocksDB and explicitly declare [`audit: true`](../database/transaction.md#enabling-the-transaction-log-per-table). A new or updated declaration without it fails with `400`, even when transaction logging is enabled globally. +- The table must use RocksDB and have [transaction logging enabled](../database/transaction.md#enabling-the-transaction-log-per-table). New schema tables should declare `@table(audit: true)`. An existing table that already persisted audit logging can add a full-text field without restating the option, but `audit: false` is always rejected while an index is declared. - Full-text source fields must be stored `String`, `[String]`, or `Blob` values. Blob sources must be declared as `text/plain`. -- Creating or updating an LMDB table with `@fullText` fails with `400`. A persisted declaration encountered while opening LMDB is ignored with a warning. Migrate the table to RocksDB before activating the index. +- Creating or updating an LMDB table with an `@fullText` field fails with `400`. A persisted declaration encountered while opening LMDB is ignored with a warning. Migrate the table to RocksDB before activating the index. See [Configuration](./configuration.md) for the complete schema contract, [Querying](./querying.md) for search examples, and [Query optimization](../resources/query-optimization.md) for performance guidance. diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 65ff405ee..9646322b7 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -6,7 +6,7 @@ title: Querying -Use the full-text index name as a condition attribute in `Table.search()`. +Use the query-only `FullText` field name as a condition attribute in `Table.search()`. ```javascript const products = await Product.search({ @@ -15,7 +15,7 @@ const products = await Product.search({ }); ``` -Results are ordered by BM25 relevance. The index declaration, not the source field name, identifies the full-text search target. +Results are ordered by BM25 relevance. The `FullText` field, not one of its source fields, identifies the search target. ## Match modes @@ -63,7 +63,7 @@ const products = await Product.search({ Every requested field must belong to the index and be readable by the caller. Without `fields`, the caller must be allowed to read every source field in the index. Harper rejects an external REST or Operations API query with `403` if any searched source field is not readable. `$highlights` never bypasses this check. -Direct calls through `tables` or `databases` run in a trusted server-side context and do not reapply the caller's role or attribute permissions. Custom resources must authorize protected searches before returning records or highlights. +Direct calls through `tables` or `databases` run in a trusted server-side context. From an authenticated custom resource, pass `this.getContext()` to `Table.search()` and set `checkPermission: true` in the search options to enforce the caller's table and source-field permissions. Context propagation alone does not request the table permission check. ## Score and ordering @@ -79,6 +79,8 @@ const products = await Product.search({ Full-text results use descending relevance order. The only explicit full-text sort is `$score` descending; other sort keys and reverse iteration are rejected because they would discard the native ranking. +Count requests do not report an exact full-text count after authorization, structured filtering, and current-record validation. They return `recordCount: null` and `recordCountExact: false`. + ## Highlights Highlighting returns source-field fragments and matching character spans. The index must enable highlighting for each requested source field. @@ -146,7 +148,7 @@ const products = await Product.search({ }); ``` -An `or` group cannot mix full-text and ordinary record conditions, and one query cannot combine different full-text indexes. Run separate queries when either boundary is required. +An `or` group cannot mix full-text and ordinary record conditions, and one query cannot combine different full-text indexes. Full-text conditions must name a field directly on the queried table; relationship and nested-property paths are not supported. Run separate queries when any of these boundaries is required. ## Freshness controls diff --git a/reference/graphql-querying/overview.md b/reference/graphql-querying/overview.md index 6a429ec24..54444c0a8 100644 --- a/reference/graphql-querying/overview.md +++ b/reference/graphql-querying/overview.md @@ -172,7 +172,7 @@ fragment sharedFields on Any { Any attribute can be used as an argument for a query. In this short form, multiple arguments is treated as multiple equivalency conditions with the default `and` operation. -These arguments always use equality. A declared `@fullText` index does not change GraphQL argument semantics; use [full-text search through `Table.search()`, REST, or the Operations API](../full-text-search/querying.md) for match comparators and BM25 ranking. +These arguments always use equality. A query-only `FullText` declaration field is omitted from the generated GraphQL record schema; use [full-text search through `Table.search()`, REST, or the Operations API](../full-text-search/querying.md) for match comparators and BM25 ranking. For example, the following query requires an `id` variable to be provided, and the system will search for a `Dog` record matching that id. diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 2944b82b4..b88f9c30b 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -63,7 +63,7 @@ Returns all table definitions within the specified database. Each table definiti ### `describe_table` -Returns the definition of a specific table. A table with native full-text indexes includes `full_text_indexes`; each entry reports its source fields, analyzer options, enabled query modes, and `readiness` (`ready`, `rebuilding`, or `unavailable`). The readiness object also includes `owner_epoch`, `rebuild_attempts`, and an optional `reason`. +Returns the definition of a specific table. A table with native full-text indexes includes `full_text_indexes`; each entry reports its query-only field name, source fields, analyzer options, enabled query modes, and `readiness` (`ready`, `rebuilding`, or `unavailable`). The readiness object also includes `owner_epoch`, `rebuild_attempts`, and an optional `reason`. Query-only `FullText` fields do not appear in the ordinary `attributes` list. Full-text response metadata: diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index 34f439cc3..01fbe79e2 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -83,7 +83,7 @@ This is a transient condition — the index finishes building in the background, -A condition using a declared `@fullText` index is executed by the native full-text engine and ranked with BM25. Compatible `and` filters are evaluated while accepting candidates so a selective structured filter can still fill the requested result page. Harper then loads the current source records and removes stale or deleted candidates. +A condition using a `FullText` field declared with `@fullText` is executed by the native full-text engine and ranked with BM25. Compatible `and` filters are evaluated while accepting candidates so a selective structured filter can still fill the requested result page. Harper then loads the current source records and removes stale or deleted candidates. Keep result pages bounded. Any expression containing prefix or fuzzy-prefix matching has a 100-record native window and is intended for autocomplete-style record lookup. An unbounded query that exceeds the window fails instead of returning a partial result. Use source-field weights to tune ranking, and enable positions, surface terms, and highlighting only when their query features are needed. See [Full-Text Search](../full-text-search/overview.md). diff --git a/reference/rest/querying.md b/reference/rest/querying.md index e367b5270..fb2c96955 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -68,7 +68,7 @@ GET /Product/?category=software&price=gt=100&price=lt=200 -For a declared `@fullText` index, use its name as the query attribute and one of the full-text comparators: +For a `FullText` field carrying `@fullText`, use the field name as the query attribute and choose a full-text comparator: ```http GET /Product/?catalogSearch=matches=waterproof%20trail diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index e05284d25..4ad991cec 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -10,7 +10,7 @@ All patch release notes for 5.3.x are available on the [releases page](https://g ## Full-Text Search -Harper tables can declare native BM25-ranked indexes with `@fullText` and query them through `Table.search()`, REST, or `search_by_conditions`. Search supports any-term, all-term, phrase, prefix, fuzzy, and fuzzy-prefix matching, with optional source weights, synonyms, scores, and highlights. +Harper tables can declare a query-only `FullText` field with `@fullText` and search that field through `Table.search()`, REST, or `search_by_conditions`. Search supports any-term, all-term, phrase, prefix, fuzzy, and fuzzy-prefix matching, with optional source weights, synonyms, scores, and highlights. The indexes are derived locally from committed audited RocksDB records. Each replica maintains its own files, reuses a compatible checkpoint after restart, and rebuilds locally when necessary. See [Full-Text Search](/reference/v5/full-text-search/overview) and [Add Full-Text Search to a Product Catalog](/learn/developers/full-text-search-product-catalog). From 1d3792dbf34803201c78f398254c7bdb8d8f2447 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 29 Sep 2026 09:47:59 -0600 Subject: [PATCH 10/78] docs(fulltext): address review feedback --- .../full-text-search-product-catalog.mdx | 41 ++++++++++--------- .../components/javascript-environment.md | 2 +- reference/database/overview.md | 20 ++++----- reference/database/schema.md | 2 + reference/database/transaction.md | 2 +- reference/full-text-search/configuration.md | 6 +-- reference/full-text-search/overview.md | 19 ++++++--- reference/full-text-search/querying.md | 8 ++-- reference/graphql-querying/overview.md | 2 +- reference/operations-api/operations.md | 8 ++-- reference/resources/resource-api.md | 12 +++--- reference/rest/headers.md | 6 +-- reference/rest/querying.md | 2 + 13 files changed, 74 insertions(+), 56 deletions(-) diff --git a/learn/developers/full-text-search-product-catalog.mdx b/learn/developers/full-text-search-product-catalog.mdx index 29ff17960..69781cf2b 100644 --- a/learn/developers/full-text-search-product-catalog.mdx +++ b/learn/developers/full-text-search-product-catalog.mdx @@ -70,14 +70,14 @@ Harper commits the record first, then applies the committed change to the local ```javascript title="resources.js" export class SearchProducts extends Resource { - static loadAsInstance = false; - - allowCreate(user) { - return Boolean(user); - } - - async post(_target, body) { - const text = body.q?.trim(); + static async post(_target, data, context) { + if (!context?.user) { + const error = new Error('Authentication required'); + error.statusCode = 401; + throw error; + } + const body = (await data) ?? {}; + const text = typeof body.q === 'string' ? body.q.trim() : ''; if (!text) { const error = new Error('q is required'); error.statusCode = 400; @@ -89,24 +89,19 @@ export class SearchProducts extends Resource { comparator: 'matches', value: text, fields: ['name', 'description', 'tags'], - includeHighlights: true, }, ]; const category = body.category; if (category) conditions.push({ attribute: 'category', comparator: 'equals', value: category }); - const records = []; - for await (const record of databases.catalog.Product.search( + return databases.catalog.Product.search( { conditions, select: ['id', 'name', 'price', '$score', '$highlights'], limit: 20, checkPermission: true, }, - this.getContext() - )) { - records.push(record); - } - return records; + context + ); } } ``` @@ -124,7 +119,9 @@ The category filter is optional; `q` is required. The full-text index provides ranked candidates. The indexed category condition narrows the result set, and Harper loads the current records before returning them. -Calls through `tables` and `databases` use Harper's [trusted server-side context](/reference/v5/components/javascript-environment#tables). Passing `this.getContext()` preserves the authenticated caller, and `checkPermission: true` asks the table to enforce that caller's table and source-field permissions. The custom resource must do both. Highlights can expose searched source text even when those fields are absent from `select`. +Calls through `tables` and `databases` use Harper's [trusted server-side context](/reference/v5/components/javascript-environment#tables). Forwarding the resource's `context` preserves the authenticated caller, and `checkPermission: true` asks the table to enforce that caller's table and source-field permissions. The custom resource must do both. Highlights can expose searched source text even when those fields are absent from `select`. + +The search includes `tags`, but that source does not set `highlight: true`. A tag match can rank a product, while `$highlights` omits `tags`. ## 4. Search through REST @@ -148,12 +145,14 @@ GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&select(id,name)&limi Prefix search returns matching products. It does not return a dictionary of suggested terms. +REST searches every source field in the index, so the caller must be allowed to read `name`, `description`, and `tags` for these requests. Use `Table.search()` or `search_by_conditions` when a role should search only a subset of sources. See [REST full-text operators](/reference/v5/rest/querying#full-text-operators). + ## 5. Choose consistency behavior Most searches can use the default three-second lag tolerance. When the next request must find a record that was just written, use the JavaScript API and wait for current index coverage: ```javascript -const results = await databases.catalog.Product.search({ +for await (const product of databases.catalog.Product.search({ conditions: [ { attribute: 'catalogSearch', @@ -163,7 +162,9 @@ const results = await databases.catalog.Product.search({ waitForIndexMilliseconds: 10000, }, ], -}); +})) { + console.log(product); +} ``` Do not apply this wait to every catalog request without measuring it. The normal bounded-staleness path keeps search traffic independent of short indexing bursts. @@ -172,7 +173,7 @@ If the index does not reach current coverage within ten seconds, consuming the s ## 6. Operate the index -Use `describe_table` to inspect readiness. Harper reuses compatible local index files after restart and replays changes after their checkpoint. A node rebuilds locally when the files are missing, incompatible, or corrupt; table traffic remains available, while full-text requests return `503` with `code: "INDEX_REBUILDING"` and `retryable: true` until the rebuild is ready. Retry this code with backoff; do not treat every `503` as an index rebuild. +Use `describe_table` to inspect readiness. Harper reuses compatible local index files after restart and replays changes after their checkpoint. A node rebuilds locally when the files are missing, incompatible, or corrupt; table traffic remains available, while full-text requests in `unknown`, `needs-rebuild`, or `rebuilding` state return `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. Retry this code with backoff; do not treat every `503` as an index rebuild. See the [full-text configuration reference](/reference/v5/full-text-search/configuration) and [query reference](/reference/v5/full-text-search/querying) for every option and match mode. diff --git a/reference/components/javascript-environment.md b/reference/components/javascript-environment.md index 8f9b35abd..018d01cba 100644 --- a/reference/components/javascript-environment.md +++ b/reference/components/javascript-environment.md @@ -88,7 +88,7 @@ An object whose properties are the tables in the default database (`data`). Each See [Database API](../database/api.md) for full reference. :::note Server-side table reads run in a trusted context -Calls to `tables.X.get()`, `tables.X.search()`, and equivalent `databases..` methods from within resource code execute in a **trusted system context** and do **not** re-apply the target table's `allowRead` guard or role-level `attribute_permissions`. This is by design — server-side code is responsible for its own authorization checks. A computed attribute or custom resource that cross-reads a protected table can return raw data regardless of the caller's role. Do not expose protected data through a computed attribute when access depends on the caller. Instead, use a custom resource and apply authorization explicitly (e.g. call `this.getCurrentUser()` in a `Resource` method, or use `getUser()`, and enforce the relevant restrictions before returning the data). +Calls to `tables.X.get()`, `tables.X.search()`, and equivalent `databases..
` methods from within resource code execute in a **trusted system context** and do **not** re-apply the target table's `allowRead` guard or role-level `attribute_permissions` by default. This is by design — server-side code is responsible for its own authorization checks. A custom resource can explicitly re-enter table and source-field authorization for `Table.search()` by forwarding its authenticated `context` and setting `checkPermission: true` in search options. Never accept or copy that flag or a permission object from client data. Context propagation alone does not request the check, and other direct table operations still require explicit authorization. A computed attribute or custom resource that cross-reads a protected table can otherwise return raw data regardless of the caller's role. Do not expose protected data through a computed attribute when access depends on the caller. Instead, use a custom resource and enforce the relevant restrictions before returning the data. ::: ### `databases` diff --git a/reference/database/overview.md b/reference/database/overview.md index 02924042f..82eae4d46 100644 --- a/reference/database/overview.md +++ b/reference/database/overview.md @@ -106,16 +106,16 @@ It is best practice to define schemas for production tables. Dynamic schemas are For deeper coverage of database features, see these related pages: -- **[Schema](./schema.md)** — Defining table structure, types, indexes, relationships, and computed properties using GraphQL schema syntax -- **[Full-Text Search](../full-text-search/overview.md)** — BM25-ranked search over stored text fields on audited RocksDB tables -- **[API](./api.md)** — The `tables`, `databases`, `transaction()`, and `createBlob()` globals for interacting with the database from code -- **[Data Loader](./data-loader.md)** — Loading seed or initial data into tables as part of component deployment -- **[Storage Algorithm](./storage-algorithm.md)** — How Harper stores data using LMDB with universal indexing and ACID compliance -- **[Storage Tuning](./storage-tuning.md)** — Tuning `storage.*` options for production: durability vs. throughput, compression, blob paths, reclamation -- **[Jobs](./jobs.md)** — Asynchronous bulk data operations (CSV import/export, S3 import/export) -- **[System Tables](./system-tables.md)** — Harper internal tables for analytics, data loader state, and other system features -- **[Compaction](./compaction.md)** — Reducing database file size by eliminating fragmentation and free space -- **[Transaction Logging](./transaction.md)** — Recording and querying a history of data changes in the transaction log +- **[Schema](./schema.md)**: Defining table structure, types, indexes, relationships, and computed properties using GraphQL schema syntax +- **[Full-Text Search](../full-text-search/overview.md)**: BM25-ranked search over stored text fields on audited RocksDB tables +- **[API](./api.md)**: The `tables`, `databases`, `transaction()`, and `createBlob()` globals for interacting with the database from code +- **[Data Loader](./data-loader.md)**: Loading seed or initial data into tables as part of component deployment +- **[Storage Algorithm](./storage-algorithm.md)**: How Harper stores data using LMDB with universal indexing and ACID compliance +- **[Storage Tuning](./storage-tuning.md)**: Tuning `storage.*` options for production: durability vs. throughput, compression, blob paths, reclamation +- **[Jobs](./jobs.md)**: Asynchronous bulk data operations (CSV import/export, S3 import/export) +- **[System Tables](./system-tables.md)**: Harper internal tables for analytics, data loader state, and other system features +- **[Compaction](./compaction.md)**: Reducing database file size by eliminating fragmentation and free space +- **[Transaction Logging](./transaction.md)**: Recording and querying a history of data changes in the transaction log ## Related Documentation diff --git a/reference/database/schema.md b/reference/database/schema.md index c15bcf9f8..7658002dd 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -825,6 +825,8 @@ Harper supports the following field types: | `Blob` | Binary large object; designed for streaming content >20KB | | `FullText` | Query-only full-text index declaration; valid only with `@fullText` and never stored | +Added `FullText` in v5.3.0 + Added `BigInt` in v4.3.0 Added `Blob` in v4.5.0 diff --git a/reference/database/transaction.md b/reference/database/transaction.md index 005440fe8..d69442286 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -134,7 +134,7 @@ Deletes transaction log entries older than the specified timestamp. Deprecated i Deletes transaction log entries older than the specified timestamp. :::warning Database-wide and irreversible -On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This purges whole log files whose entries predate the timestamp, removing [`read_audit_log`](#read_audit_log) history for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender detects that the requested start predates its retained history and forces a full base copy instead of incremental catch-up — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. The same history can be required to bridge a [full-text index rebuild](../full-text-search/overview.md#startup-and-recovery) from its table scan to current writes; aggressive cleanup can leave that index `unavailable` until its recovery source is restored and activation is retried. +On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This purges whole log files whose entries predate the timestamp, removing [`read_audit_log`](#read_audit_log) history for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender detects that the requested start predates its retained history and forces a full base copy instead of incremental catch-up — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. The same history can be required to bridge a [full-text index rebuild](../full-text-search/overview.md#startup-and-recovery) from its table scan to current writes. Cleanup that removes the bridge during a rebuild can fail that attempt; stop aggressive cleanup before retrying so the next attempt can scan from a fresh committed-log boundary. ::: Parameters: diff --git a/reference/full-text-search/configuration.md b/reference/full-text-search/configuration.md index ca0b33031..56dd2529d 100644 --- a/reference/full-text-search/configuration.md +++ b/reference/full-text-search/configuration.md @@ -31,7 +31,7 @@ type Product @table(database: "catalog", audit: true) { The declaration must use nullable `FullText` exactly and carry one `@fullText` directive. Lists, non-null forms, a bare `FullText` field, and additional directives such as `@computed`, `@indexed`, or `@allow` are rejected. `@fullText` is not valid on a table type or any other field type. -New schema tables must explicitly set `@table(audit: true)`. An existing table that already has persisted audit logging can add the field without restating `audit`, but disabling audit logging while a full-text field exists is rejected. +New schema tables must explicitly set `@table(audit: true)`; omitting it or setting `audit: false` returns `400`. An existing table that already has persisted audit logging can add the field without restating `audit`. A previously unaudited table can enable `audit: true` in the same schema change that adds its first full-text field; the initial build scans current records and does not require older transaction history. Disabling audit logging while a full-text field exists is rejected. Writing the declaration-only field or selecting it as record data returns an error, including on unsealed tables. A read-through source response containing that name fails with `502` and is not cached, indexed, or audited. For HTTP-sourced caching tables, avoid index names that collide with response wrapper fields such as `data`, `headers`, `body`, and `status`. @@ -139,7 +139,7 @@ type Document @table(audit: true) { } ``` -`mediaType: "text/plain"` declares how Harper interprets the Blob; it does not validate runtime Blob metadata or reject the record write. The derived index decodes the Blob as UTF-8. If a Blob is invalid UTF-8 or exceeds the native source limit, Harper removes that whole record from the index rather than indexing only its other fields. A transient Blob read failure rolls back the accepted native batch and retries it, so index coverage does not advance past unread source data. +`mediaType: "text/plain"` declares how Harper interprets the Blob; it does not validate runtime Blob metadata or reject the record write. The derived index decodes the Blob as UTF-8. A Blob source is limited to 1 MiB (1,048,576 bytes). If it exceeds that limit or is invalid UTF-8, Harper removes that whole record from the index rather than indexing only its other fields. A transient Blob read failure rolls back the accepted native batch and retries it, so index coverage does not advance past unread source data. ## Schema changes and rebuilds @@ -182,7 +182,7 @@ The table response includes entries shaped like this: } ``` -`readiness.state` is `ready`, `rebuilding`, or `unavailable`. A rebuilding query returns `503` with retryable code `INDEX_REBUILDING`. `unavailable` means activation failed or the automatic rebuild budget was exhausted; queries return a generic, non-retryable `503`. Inspect the optional `readiness.reason` and the node logs, correct the underlying storage, native-module, schema, or transaction-log problem, then reload or reapply the schema to retry activation. Full-text queries are served only when the state is `ready`. +`readiness.state` is `unknown`, `ready`, `needs-rebuild`, `rebuilding`, or `unavailable`. Only `ready` serves full-text queries. `unknown` is an activation or owner-election transition. `needs-rebuild` means the runtime has condemned the current generation and queued an automatic rebuild. Queries in `unknown`, `needs-rebuild`, or `rebuilding` return `503` with retryable code `INDEX_REBUILDING`. `unavailable` means activation failed or the automatic rebuild budget was exhausted; queries return a generic, non-retryable `503`. Inspect the optional `readiness.reason` and the node logs, correct the underlying storage, native-module, schema, or transaction-log problem, then reload or reapply the schema to retry activation. `query_modes` maps to comparators as follows: `any` → `matches`, `all` → `matches_all`, `fuzzy` → `matches_fuzzy`, `phrase` → `matches_phrase`, `prefix` → `matches_prefix`, and `fuzzy-prefix` → `matches_fuzzy_prefix`. `owner_epoch` is an opaque decimal string used to fence work between derived-index owners; clients should not parse or persist it. `rebuild_attempts` is the current automatic-attempt count. `reason` is present when readiness has an operator-actionable explanation. diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index 77d017c62..bddc82f05 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -4,7 +4,7 @@ title: Overview # Full-Text Search - + Harper can build a BM25-ranked full-text index from one or more fields on a table. The index is declared in the table schema, updated from committed record changes, and queried through the same `Table.search()` and REST interfaces used for other Harper queries. @@ -12,7 +12,7 @@ Use full-text search when users need relevance-ranked matching across product na ## Requirements -- The table must use RocksDB and have [transaction logging enabled](../database/transaction.md#enabling-the-transaction-log-per-table). New schema tables should declare `@table(audit: true)`. An existing table that already persisted audit logging can add a full-text field without restating the option, but `audit: false` is always rejected while an index is declared. +- The table must use RocksDB and have [transaction logging enabled](../database/transaction.md#enabling-the-transaction-log-per-table). New schema tables must declare `@table(audit: true)`; omitting it or setting `audit: false` returns `400`. An existing table that already persisted audit logging can add a full-text field without restating the option, but disabling audit is rejected while an index is declared. - Full-text source fields must be stored `String`, `[String]`, or `Blob` values. Blob sources must be declared as `text/plain`. - Creating or updating an LMDB table with an `@fullText` field fails with `400`. A persisted declaration encountered while opening LMDB is ignored with a warning. Migrate the table to RocksDB before activating the index. @@ -35,21 +35,30 @@ flowchart LR Each node builds its own index from committed transactions. Harper does not replicate full-text index files or include them in authoritative backups. After a restore or move to a node without compatible local files, Harper rebuilds the index before serving full-text queries. +The `@fullText` declaration must also exist on each node. A replicated component deployment or schema propagation can install it, but table-record replication alone only supplies the source records. Once the declaration is present, the node builds its own index. + This design has two consequences: - A committed record can be visible before its derived full-text index has applied the same transaction. Queries use a bounded staleness policy, described in [Freshness controls](./querying.md#freshness-controls). - A missing, corrupt, or incompatible index can be rebuilt from the table and audit stream without changing source records. +- If an index remains behind for more than 30 seconds, Harper rejects ordinary local writes to its table until the runtime catches up or changes lifecycle state. See [Write backpressure](#write-backpressure). ## Startup and recovery On restart, Harper reuses compatible local index files and replays committed changes after the saved checkpoint. If those files are missing, incompatible, or corrupt, Harper rebuilds the index locally. -A rebuild scans the current table and replays changes committed during the scan. An unreadable or corrupt transaction log can make a rebuild fail. The same applies when [transaction-log cleanup](../database/transaction.md#delete_transaction_logs_before) or automatic retention removes history required to bridge the scan to current writes. Harper retries with backoff; after the retry budget is exhausted, readiness becomes `unavailable`. Inspect `describe_table` and the node logs for the reason before retrying activation. +A rebuild captures a fresh committed-log boundary, scans the current table, and replays changes committed after that boundary. An unreadable or corrupt transaction log can make an attempt fail. The same applies when [transaction-log cleanup](../database/transaction.md#delete_transaction_logs_before) or automatic retention removes the history needed to bridge that scan to current writes. A later attempt captures a new boundary, so stopping aggressive cleanup can allow the rebuild to succeed without restoring already-purged history. Harper retries with backoff; after the retry budget is exhausted, readiness becomes `unavailable`. Inspect `describe_table` and the node logs for the reason before retrying activation. -Ordinary table reads and writes remain available during a rebuild. While readiness is `rebuilding`, full-text queries return `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. If rebuild attempts are exhausted and readiness becomes `unavailable`, queries return a generic, non-retryable `503` until an operator corrects the cause and retries activation. A new replica follows the same process before serving full-text queries. +Ordinary table reads and writes remain available during a rebuild. While readiness is `unknown`, `needs-rebuild`, or `rebuilding`, full-text queries return `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. If rebuild attempts are exhausted and readiness becomes `unavailable`, queries return a generic, non-retryable `503` until an operator corrects the cause and retries activation. A new replica follows the same process after the full-text declaration reaches that node. Use `describe_table` to inspect each index's declaration and readiness. See [Inspecting an index](./configuration.md#inspecting-an-index). +## Write backpressure + +Harper allows a full-text index up to 30 seconds to absorb committed changes. If the runtime remains behind beyond that budget, ordinary local writes to the indexed table fail with `503`, `code: "DERIVED_INDEX_LAGGING"`, and `retryable: true`. This applies to writes through REST, the Operations API, and direct table calls from application code. Replication notifications, source applies, and crash-recovery replay continue so the authoritative data can converge. + +Retry rejected writes with backoff rather than immediately adding more load. Inspect index readiness and node logs. Write admission resumes after the runtime proves durable catch-up or moves into another lifecycle state such as rebuilding or unavailable. + ## Ranking and result loading The native index ranks matching documents with BM25. Harper then loads the authoritative table records for the matching IDs and discards stale or deleted candidates. Selecting `$score` includes the relevance score in each result. @@ -61,7 +70,7 @@ Full-text search can be combined with structured filters. Harper pushes compatib - Phrase search requires `positions: true`; prefix and fuzzy-prefix require `surfaceTerms: true`. - Prefix expressions have a 100-record native result window. Use a bounded `limit`; requests beyond the window fail instead of truncating silently. - One query can use only one full-text index. An `or` group cannot mix full-text and ordinary record conditions. -- `INDEX_REBUILDING` is transient and retryable. `unavailable` means automatic rebuild attempts were exhausted or activation failed; inspect readiness and logs rather than retrying every `503` indefinitely. +- `INDEX_REBUILDING` is transient and retryable. `DERIVED_INDEX_LAGGING` can mean a query coverage wait expired or local writes were shed after prolonged index lag. `unavailable` means automatic rebuild attempts were exhausted or activation failed; inspect readiness and logs rather than retrying every `503` indefinitely. - Generated GraphQL field arguments remain equality conditions; use `Table.search()`, REST, or `search_by_conditions` for full-text comparators. See [Querying](./querying.md) for exact condition rules and [Configuration](./configuration.md#inspecting-an-index) for readiness details. diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 9646322b7..1288660ca 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -63,7 +63,7 @@ const products = await Product.search({ Every requested field must belong to the index and be readable by the caller. Without `fields`, the caller must be allowed to read every source field in the index. Harper rejects an external REST or Operations API query with `403` if any searched source field is not readable. `$highlights` never bypasses this check. -Direct calls through `tables` or `databases` run in a trusted server-side context. From an authenticated custom resource, pass `this.getContext()` to `Table.search()` and set `checkPermission: true` in the search options to enforce the caller's table and source-field permissions. Context propagation alone does not request the table permission check. +Direct calls through `tables` or `databases` run in a trusted server-side context. From an authenticated custom resource, pass the `context` received by the static resource method to `Table.search()` and set `checkPermission: true` in the search options to enforce the caller's table and source-field permissions. Context propagation alone does not request the table permission check. ## Score and ordering @@ -83,7 +83,7 @@ Count requests do not report an exact full-text count after authorization, struc ## Highlights -Highlighting returns source-field fragments and matching character spans. The index must enable highlighting for each requested source field. +Highlighting returns source-field fragments and matching character spans. The index must configure highlighting and mark at least one source with `highlight: true`. A query can also search sources without that option; those sources can affect matching and ranking but are omitted from `$highlights`. Selecting `$highlights` turns highlighting on for the query: @@ -181,6 +181,8 @@ A non-waiting HTTP query can return `Harper-Index-Coverage` with the admitted st If acceptable coverage is not reached before `waitForIndexMilliseconds` expires, Harper returns `503` with `code: "DERIVED_INDEX_LAGGING"` and `retryable: true`. For `Table.search()`, the error is raised while the async iterator is consumed. An HTTP response may already be streaming, so clients should also handle a terminal stream error rather than relying only on the initial status. +The same code is also used when prolonged derived-index lag causes Harper to reject local writes. Query waits can be retried according to the caller's freshness needs; rejected writes should use backoff while the index catches up. See [Write backpressure](./overview.md#write-backpressure). + ## Prefix result window `matches_prefix` and `matches_fuzzy_prefix` are autocomplete-style record searches. Any expression containing one of these modes uses a 100-record native result window, and `offset + limit` cannot exceed that window. If an unbounded query has more than 100 matches, Harper returns `400` and requires a limit instead of silently truncating the result. These modes return matching records, not a separate list of suggested terms. @@ -232,4 +234,4 @@ REST can return configured highlights by selecting `$highlights`. Use `Table.sea } ``` -If an index is rebuilding, the query returns `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. An index in terminal `unavailable` state returns a generic, non-retryable `503`; inspect readiness and logs instead of retrying it as a rebuild. Other busy paths can also return a generic `503`, so clients should branch on the code rather than the status alone. Unreadable source fields return `403`. Invalid declarations, unsupported match modes, incompatible combinations, and out-of-range options return `400` responses. +If an index is `unknown`, `needs-rebuild`, or `rebuilding`, the query returns `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. An index in terminal `unavailable` state returns a generic, non-retryable `503`; inspect readiness and logs instead of retrying it as a rebuild. Other busy paths can also return a generic `503`, so clients should branch on the code rather than the status alone. Unreadable source fields return `403`. Invalid declarations, unsupported match modes, incompatible combinations, and out-of-range options return `400` responses. diff --git a/reference/graphql-querying/overview.md b/reference/graphql-querying/overview.md index 54444c0a8..79b5f6c68 100644 --- a/reference/graphql-querying/overview.md +++ b/reference/graphql-querying/overview.md @@ -172,7 +172,7 @@ fragment sharedFields on Any { Any attribute can be used as an argument for a query. In this short form, multiple arguments is treated as multiple equivalency conditions with the default `and` operation. -These arguments always use equality. A query-only `FullText` declaration field is omitted from the generated GraphQL record schema; use [full-text search through `Table.search()`, REST, or the Operations API](../full-text-search/querying.md) for match comparators and BM25 ranking. +These arguments always use equality. A query-only `FullText` declaration field is omitted from the generated GraphQL record schema (Added in: v5.3.0); use [full-text search through `Table.search()`, REST, or the Operations API](../full-text-search/querying.md) for match comparators and BM25 ranking. For example, the following query requires an `id` variable to be provided, and the system will search for a `Dog` record matching that id. diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index b88f9c30b..3aa459b05 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -63,9 +63,9 @@ Returns all table definitions within the specified database. Each table definiti ### `describe_table` -Returns the definition of a specific table. A table with native full-text indexes includes `full_text_indexes`; each entry reports its query-only field name, source fields, analyzer options, enabled query modes, and `readiness` (`ready`, `rebuilding`, or `unavailable`). The readiness object also includes `owner_epoch`, `rebuild_attempts`, and an optional `reason`. Query-only `FullText` fields do not appear in the ordinary `attributes` list. +Returns the definition of a specific table. A table with native full-text indexes includes `full_text_indexes`; each entry reports its query-only field name, source fields, analyzer options, enabled query modes, and `readiness` (`unknown`, `ready`, `needs-rebuild`, `rebuilding`, or `unavailable`). The readiness object also includes `owner_epoch`, `rebuild_attempts`, and an optional `reason`. Query-only `FullText` fields do not appear in the ordinary `attributes` list. -Full-text response metadata: +Requesting a table with full-text indexes (Added in: v5.3.0): ```json { "operation": "describe_table", "table": "Product", "database": "catalog" } @@ -88,6 +88,8 @@ The response also includes the table's ordinary schema and storage metadata. The } ``` +#### Record-structure dictionaries + Alongside the schema, the response carries the size of the table's **record-structure dictionaries** @@ -303,7 +305,7 @@ Returns records matching one or more conditions. Supports `operator` (`and`/`or` } ``` -Full-text conditions: +Full-text conditions (Added in: v5.3.0): Conditions can query a declared full-text index with the `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix` comparators and their `not_` forms. A negated full-text condition requires a non-negated condition on the same full-text index. The Operations API preserves advanced condition options such as source `fields`, `includeHighlights`, `maxIndexLagMilliseconds`, and `waitForIndexMilliseconds`. See [Querying Full-Text Indexes](../full-text-search/querying.md#operations-api). diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index a218fc43b..ce504acbc 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -984,7 +984,7 @@ Array of condition objects to filter records. Each condition: | `maxIndexLagMilliseconds` | Maximum accepted full-text index lag; defaults to `3000` | | `waitForIndexMilliseconds` | Time to wait for acceptable full-text coverage, from `0` through `30000` | -The final four properties apply to full-text conditions: See [Querying Full-Text Indexes](../full-text-search/querying.md) for their constraints and error behavior. +The final four properties apply to full-text conditions (v5.3.0). See [Querying Full-Text Indexes](../full-text-search/querying.md) for their constraints and error behavior. Example with nested conditions: @@ -1033,16 +1033,16 @@ Properties to include in each returned record. Can be: Special properties: -- `$id` — Returns the primary key regardless of its name -- `$updatedtime` — Returns the last-updated timestamp -- `$distance` — When the query ranks or filters by a vector index, returns the computed distance from the target vector. See [Vector Indexing](../database/schema.md#vector-indexing). +- `$id`: Returns the primary key regardless of its name +- `$updatedtime`: Returns the last-updated timestamp +- `$distance`: When the query ranks or filters by a vector index, returns the computed distance from the target vector. See [Vector Indexing](../database/schema.md#vector-indexing). Full-text queries add two special properties: -- `$score` — When a query uses a full-text index, returns its BM25 relevance score. -- `$highlights` — Enables and returns full-text fragments and matching spans when the index has highlighting configured. See [Highlights](../full-text-search/querying.md#highlights). +- `$score`: When a query uses a full-text index, returns its BM25 relevance score. +- `$highlights`: Enables and returns full-text fragments and matching spans when the index has highlighting configured. See [Highlights](../full-text-search/querying.md#highlights). #### Selecting related records diff --git a/reference/rest/headers.md b/reference/rest/headers.md index c0a2ce838..1b43789f0 100644 --- a/reference/rest/headers.md +++ b/reference/rest/headers.md @@ -20,7 +20,7 @@ These headers are included in all Harper REST API responses: | `etag` | `"abc123"` | Encoded version/last-modification time of the returned record. Used for conditional requests. | | `location` | `/MyTable/new-id` | Returned on `POST` responses. Contains the path to the newly created record. | -Collection responses to a [count request](./querying.md#pagination-and-total-count) additionally include `Content-Range`, `Range-Unit`, and `Preference-Applied` (). +Collection responses to a [count request](./querying.md#pagination-and-total-count) additionally include `Content-Range`, `Range-Unit`, and `Preference-Applied` (Added in: v5.3.0). ### `Harper-Index-Coverage` @@ -28,8 +28,8 @@ Collection responses to a [count request](./querying.md#pagination-and-total-cou A non-waiting request that uses a native derived index can include `Harper-Index-Coverage`. Its value is one of: -- `current; lag=0; tolerance=` — the index covered current committed data at admission. -- `bounded; lag=; tolerance=` — the admitted lag upper bound was within the requested tolerance. +- `current; lag=0; tolerance=`: the index covered current committed data at admission. +- `bounded; lag=; tolerance=`: the admitted lag upper bound was within the requested tolerance. Multiple native searches append one value each. The header certifies coverage when the search is admitted; it does not certify a later traversal. See [Full-text freshness controls](../full-text-search/querying.md#freshness-controls). diff --git a/reference/rest/querying.md b/reference/rest/querying.md index fb2c96955..ed0a80e29 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -84,6 +84,8 @@ GET /Product/?catalogSearch=matches=waterproof&catalogSearch=not_matches=leather Use `Table.search()` or `search_by_conditions` when conditions need source-field selection, condition-level highlighting, or freshness controls. See [Querying Full-Text Indexes](../full-text-search/querying.md) for ranking, highlights, query combinations, and freshness controls. +The REST URL syntax searches every source field in the index, so the caller must be allowed to read every source field. Full-text queries cannot provide an exact total count; a `Prefer: count=exact` request returns an unavailable total (`Content-Range: items /*`). + For date fields, colons must be URL-encoded as `%3A`: ```http From 2fd256426a083cc0c9cf203d216e2be049525a6e Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 29 Sep 2026 10:54:54 -0600 Subject: [PATCH 11/78] docs(fulltext): align operational caveats --- .../full-text-search-product-catalog.mdx | 2 ++ reference/database/schema.md | 22 +++++++++---------- reference/full-text-search/overview.md | 2 +- reference/rest/querying.md | 12 +++++----- 4 files changed, 20 insertions(+), 18 deletions(-) diff --git a/learn/developers/full-text-search-product-catalog.mdx b/learn/developers/full-text-search-product-catalog.mdx index 69781cf2b..57fcf07ea 100644 --- a/learn/developers/full-text-search-product-catalog.mdx +++ b/learn/developers/full-text-search-product-catalog.mdx @@ -66,6 +66,8 @@ Content-Type: application/json Harper commits the record first, then applies the committed change to the local derived full-text index. Every replica performs the same derivation from the transaction it receives. +If the index remains behind for more than 30 seconds, a local write can return retryable `503` code `DERIVED_INDEX_LAGGING`. Retry writes with backoff while the index catches up; freshness options apply only to searches. See [write backpressure](/reference/v5/full-text-search/overview#write-backpressure). + ## 3. Search from a Harper application ```javascript title="resources.js" diff --git a/reference/database/schema.md b/reference/database/schema.md index 7658002dd..ad48f3d9f 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -73,17 +73,17 @@ type MyTable @table { Optional arguments: -| Argument | Type | Default | Description | -| -------------------- | --------- | ----------------------------- | ------------------------------------------------------------------------------------------- | -| `table` | `String` | type name | Override the table name | -| `database` | `String` | `"data"` | Database to place the table in | -| `expiration` | `Int` | — | Seconds until a record goes stale (useful for caching tables) | -| `eviction` | `Int` | `0` | Additional seconds after `expiration` before a record is physically removed | -| `scanInterval` | `Int` | `(expiration + eviction) / 4` | Seconds between eviction scans | -| `replicate` | `Boolean` | true | Enable replication of this table | -| `audit` | `Boolean` | `logging.auditLog` | Enable the table's transaction log; new tables with `@fullText` must set this to `true` | -| `cacheControl` | `String` | — | `Cache-Control` header value emitted on anonymous GET/HEAD 200/304 responses for this table | -| `randomAccessFields` | `Boolean` | `storage.randomAccessFields` | [Pin this table's record encoding](#randomaccessfields) | +| Argument | Type | Default | Description | +| -------------------- | --------- | -------------------------------------- | ------------------------------------------------------------------------------------------- | +| `table` | `String` | type name | Override the table name | +| `database` | `String` | `"data"` | Database to place the table in | +| `expiration` | `Int` | — | Seconds until a record goes stale (useful for caching tables) | +| `eviction` | `Int` | `0` | Additional seconds after `expiration` before a record is physically removed | +| `scanInterval` | `Int` | `(expiration + eviction) / 4` | Seconds between eviction scans | +| `replicate` | `Boolean` | true | Enable replication of this table | +| `audit` | `Boolean` | `logging.auditLog` for ordinary tables | Enable the table's transaction log; new tables with `@fullText` require explicit `true` | +| `cacheControl` | `String` | — | `Cache-Control` header value emitted on anonymous GET/HEAD 200/304 responses for this table | +| `randomAccessFields` | `Boolean` | `storage.randomAccessFields` | [Pin this table's record encoding](#randomaccessfields) | **`expiration`, `eviction`, and `scanInterval`** diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index bddc82f05..35307e319 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -37,7 +37,7 @@ Each node builds its own index from committed transactions. Harper does not repl The `@fullText` declaration must also exist on each node. A replicated component deployment or schema propagation can install it, but table-record replication alone only supplies the source records. Once the declaration is present, the node builds its own index. -This design has two consequences: +This design has several consequences: - A committed record can be visible before its derived full-text index has applied the same transaction. Queries use a bounded staleness policy, described in [Freshness controls](./querying.md#freshness-controls). - A missing, corrupt, or incompatible index can be rebuilt from the table and audit stream without changing source records. diff --git a/reference/rest/querying.md b/reference/rest/querying.md index ed0a80e29..2f4f77c92 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -84,7 +84,7 @@ GET /Product/?catalogSearch=matches=waterproof&catalogSearch=not_matches=leather Use `Table.search()` or `search_by_conditions` when conditions need source-field selection, condition-level highlighting, or freshness controls. See [Querying Full-Text Indexes](../full-text-search/querying.md) for ranking, highlights, query combinations, and freshness controls. -The REST URL syntax searches every source field in the index, so the caller must be allowed to read every source field. Full-text queries cannot provide an exact total count; a `Prefer: count=exact` request returns an unavailable total (`Content-Range: items /*`). +The REST URL syntax searches every source field in the index, so the caller must be allowed to read every source field. Prefix and fuzzy-prefix queries have a 100-record window; `offset + limit` cannot exceed 100. Full-text queries cannot provide an exact total count; a `Prefer: count=exact` request returns an unavailable total (`Content-Range: items /*`). For date fields, colons must be URL-encoded as `%3A`: @@ -207,10 +207,10 @@ Counting is opt-in: without the header, no count is computed and no count header Send a `Prefer` header on a `GET` (or `HEAD`) request to a collection: -| Value | Meaning | -| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `count=exact` | The exact number of matching records. Scans the full matched set, so it is off by default (see [below](#enabling-exact-counts)) and served as an estimate unless enabled for the REST interface. | -| `count=estimated` | A fast planner/table estimate. Cheap, approximate. | +| Value | Meaning | +| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `count=exact` | The exact number of matching records when the query plan can count them. Scans the full matched set, so it is off by default (see [below](#enabling-exact-counts)) and served as an estimate unless enabled for the REST interface. Full-text and other custom-index queries report an unavailable total instead. | +| `count=estimated` | A fast planner/table estimate. Cheap, approximate. | ```http GET /Product/?category=software&limit(0,25) @@ -238,7 +238,7 @@ The response status is always `200` — `Content-Range` is informational (Harper ### Unavailable totals -The total is reported as `*` (for example `Content-Range: items 0-24/*`) when it cannot be produced — an exact scan that hits its guardrail (counting the tail past the requested page is bounded by a 1,000,000-row cap and a ~1-second budget, and abandons the total rather than truncating the page), or a query with no cardinality estimate (for example a `!=` or `=ct=` (contains) condition). `Preference-Applied` still echoes the requested mode, so an unavailable total (`.../*`) is distinct from a request that asked for no count. +The total is reported as `*` (for example `Content-Range: items 0-24/*`) when it cannot be produced — an exact scan that hits its guardrail (counting the tail past the requested page is bounded by a 1,000,000-row cap and a ~1-second budget, and abandons the total rather than truncating the page), a full-text or other custom-index query, or a query with no cardinality estimate (for example a `!=` or `=ct=` (contains) condition). `Preference-Applied` still echoes the requested mode, so an unavailable total (`.../*`) is distinct from a request that asked for no count. ### Enabling exact counts From 5334c2a6d0398aa31f37c8621049d9d82977b47a Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 29 Sep 2026 11:35:49 -0600 Subject: [PATCH 12/78] docs(fulltext): complete security and storage guidance --- reference/full-text-search/overview.md | 3 +++ reference/resources/resource-api.md | 10 ++++++++-- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index 35307e319..d8afb805a 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -70,6 +70,7 @@ Full-text search can be combined with structured filters. Harper pushes compatib - Phrase search requires `positions: true`; prefix and fuzzy-prefix require `surfaceTerms: true`. - Prefix expressions have a 100-record native result window. Use a bounded `limit`; requests beyond the window fail instead of truncating silently. - One query can use only one full-text index. An `or` group cannot mix full-text and ordinary record conditions. +- Each Blob source is limited to 1 MiB (1,048,576 bytes). Invalid UTF-8 or an oversized Blob removes the entire record from the index, including its other full-text source fields, until valid content is indexed. - `INDEX_REBUILDING` is transient and retryable. `DERIVED_INDEX_LAGGING` can mean a query coverage wait expired or local writes were shed after prolonged index lag. `unavailable` means automatic rebuild attempts were exhausted or activation failed; inspect readiness and logs rather than retrying every `503` indefinitely. - Generated GraphQL field arguments remain equality conditions; use `Table.search()`, REST, or `search_by_conditions` for full-text comparators. @@ -78,3 +79,5 @@ See [Querying](./querying.md) for exact condition rules and [Configuration](./co ## Stored index data The local index stores tokenized terms, posting lists, optional positions and surface terms, and checkpoint metadata. It does not store an authoritative copy of each record. Harper always loads current table records before returning results, and it can discard and rebuild this derived state from committed data. + +Protect the index directory with the same filesystem controls as Harper data. Deleted source terms can remain in obsolete Tantivy segment files until normal merge and reclamation removes them; removal from query results does not imply immediate physical erasure. diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index ce504acbc..4201e8b3e 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -984,7 +984,7 @@ Array of condition objects to filter records. Each condition: | `maxIndexLagMilliseconds` | Maximum accepted full-text index lag; defaults to `3000` | | `waitForIndexMilliseconds` | Time to wait for acceptable full-text coverage, from `0` through `30000` | -The final four properties apply to full-text conditions (v5.3.0). See [Querying Full-Text Indexes](../full-text-search/querying.md) for their constraints and error behavior. +The final four properties apply to full-text conditions (v5.3.0). See [Querying Full-Text Indexes](../full-text-search/querying.md) for their constraints, error behavior, and permission requirements for direct server-side searches. Example with nested conditions: @@ -1112,6 +1112,12 @@ If `true`, returns conditions reordered as Harper will execute them (for debuggi If `true`, forces conditions to execute in the order supplied, disabling Harper's automatic re-ordering optimization. +### `checkPermission` + +Trusted server-side calls skip table authorization by default. When delegating a request to `Table.search()`, set `checkPermission: true` on the query and pass the authenticated `context` as the second argument. Harper then performs a one-shot read authorization using `context.user`, including table, row, selected-property, and full-text source-field permissions. Passing the context alone does not request this check. + +Set this flag in server-controlled code whenever the endpoint promises authorized access. Never accept its value or copy a permission object from client data. + --- ## RequestTarget @@ -1125,7 +1131,7 @@ Properties: - `id` — Primary key derived from the path - `isCollection` — `true` when the request targets a collection - `rowFilter` — Synchronous JavaScript predicate applied to candidate records during search. It cannot be set by REST or QUERY request data. -- `checkPermission` — Framework-owned, one-shot request to run legacy operation authorization. Harper arms it for permission-checked dispatches. Trusted server-side code making a subsequent direct Resource call may set it to `true`, causing built-in table gates to derive permissions from `context.user`. Never accept or copy this field or a permission object from client data. +- `checkPermission` — The same one-shot authorization request available on the [Query Object](#checkpermission). Harper arms it for permission-checked dispatches. Trusted server-side code making a direct Resource call may set it to `true`, causing built-in table gates to derive permissions from `context.user`. Never accept or copy this field or a permission object from client data. Standard `URLSearchParams` methods are available: From e50fc568d97025ace460f9a8767c511e5d55eeca Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 29 Sep 2026 11:57:27 -0600 Subject: [PATCH 13/78] docs(fulltext): clarify REST count behavior --- reference/full-text-search/querying.md | 2 +- reference/rest/querying.md | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 1288660ca..725434628 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -79,7 +79,7 @@ const products = await Product.search({ Full-text results use descending relevance order. The only explicit full-text sort is `$score` descending; other sort keys and reverse iteration are rejected because they would discard the native ranking. -Count requests do not report an exact full-text count after authorization, structured filtering, and current-record validation. They return `recordCount: null` and `recordCountExact: false`. +An exact count requested through `Table.search()` does not report a full-text total after authorization, structured filtering, and current-record validation. It returns `recordCount: null` and `recordCountExact: false`. REST behaves the same when its interface enables `exactCount`; otherwise `Prefer: count=exact` is downgraded to an estimated count. ## Highlights diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 2f4f77c92..a479e2454 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -84,7 +84,7 @@ GET /Product/?catalogSearch=matches=waterproof&catalogSearch=not_matches=leather Use `Table.search()` or `search_by_conditions` when conditions need source-field selection, condition-level highlighting, or freshness controls. See [Querying Full-Text Indexes](../full-text-search/querying.md) for ranking, highlights, query combinations, and freshness controls. -The REST URL syntax searches every source field in the index, so the caller must be allowed to read every source field. Prefix and fuzzy-prefix queries have a 100-record window; `offset + limit` cannot exceed 100. Full-text queries cannot provide an exact total count; a `Prefer: count=exact` request returns an unavailable total (`Content-Range: items /*`). +The REST URL syntax searches every source field in the index, so the caller must be allowed to read every source field. Prefix and fuzzy-prefix queries have a 100-record window; `offset + limit` cannot exceed 100. Full-text queries cannot provide an exact total count. On a REST interface with `exactCount: true`, a `Prefer: count=exact` request returns an unavailable total (`Content-Range: items /*`). Without that opt-in, Harper applies `count=estimated` instead and can return an estimate. For date fields, colons must be URL-encoded as `%3A`: @@ -207,10 +207,10 @@ Counting is opt-in: without the header, no count is computed and no count header Send a `Prefer` header on a `GET` (or `HEAD`) request to a collection: -| Value | Meaning | -| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `count=exact` | The exact number of matching records when the query plan can count them. Scans the full matched set, so it is off by default (see [below](#enabling-exact-counts)) and served as an estimate unless enabled for the REST interface. Full-text and other custom-index queries report an unavailable total instead. | -| `count=estimated` | A fast planner/table estimate. Cheap, approximate. | +| Value | Meaning | +| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `count=exact` | The exact number of matching records when the query plan can count them. Scans the full matched set, so it is off by default (see [below](#enabling-exact-counts)) and served as an estimate unless enabled for the REST interface. When enabled, full-text and other custom-index queries report an unavailable total instead. | +| `count=estimated` | A fast planner/table estimate. Cheap, approximate. | ```http GET /Product/?category=software&limit(0,25) From 2c4ab6d53d797bf531f96050e65d87e26b44f0b4 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 29 Sep 2026 13:34:40 -0600 Subject: [PATCH 14/78] docs(fulltext): correct runtime behavior --- reference/full-text-search/configuration.md | 2 +- reference/full-text-search/querying.md | 7 +++++-- reference/resources/resource-api.md | 2 +- 3 files changed, 7 insertions(+), 4 deletions(-) diff --git a/reference/full-text-search/configuration.md b/reference/full-text-search/configuration.md index 56dd2529d..adb7237fe 100644 --- a/reference/full-text-search/configuration.md +++ b/reference/full-text-search/configuration.md @@ -139,7 +139,7 @@ type Document @table(audit: true) { } ``` -`mediaType: "text/plain"` declares how Harper interprets the Blob; it does not validate runtime Blob metadata or reject the record write. The derived index decodes the Blob as UTF-8. A Blob source is limited to 1 MiB (1,048,576 bytes). If it exceeds that limit or is invalid UTF-8, Harper removes that whole record from the index rather than indexing only its other fields. A transient Blob read failure rolls back the accepted native batch and retries it, so index coverage does not advance past unread source data. +`mediaType: "text/plain"` declares the expected Blob media type. Harper does not reject the source record write, but the derived index checks a nonempty runtime `Blob.type` and excludes the record when it does not match `text/plain`. The index decodes matching Blobs as UTF-8. A Blob source is limited to 1 MiB (1,048,576 bytes). A media-type mismatch, oversized value, or invalid UTF-8 removes the whole record from the index rather than indexing only its other fields. A transient Blob read failure rolls back the accepted native batch and retries it, so index coverage does not advance past unread source data. ## Schema changes and rebuilds diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 725434628..2168d240d 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -162,7 +162,8 @@ A full-text index is derived from committed table changes. Each condition accept For read-after-write behavior, require current coverage and allow a bounded wait: ```javascript -const products = await Product.search({ +const products = []; +for await (const product of Product.search({ conditions: [ { attribute: 'catalogSearch', @@ -172,7 +173,9 @@ const products = await Product.search({ waitForIndexMilliseconds: 10000, }, ], -}); +})) { + products.push(product); +} ``` All full-text conditions combined into one query must use the same freshness values. Harper returns `400` when combined conditions specify different values. diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 4201e8b3e..ed3e2d100 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1114,7 +1114,7 @@ If `true`, forces conditions to execute in the order supplied, disabling Harper' ### `checkPermission` -Trusted server-side calls skip table authorization by default. When delegating a request to `Table.search()`, set `checkPermission: true` on the query and pass the authenticated `context` as the second argument. Harper then performs a one-shot read authorization using `context.user`, including table, row, selected-property, and full-text source-field permissions. Passing the context alone does not request this check. +Trusted server-side calls skip table authorization by default. When delegating a request to `Table.search()`, set `checkPermission: true` on the query and pass the authenticated `context` as the second argument. Harper then performs a one-shot read authorization using `context.user`, including table, selected-property, and full-text source-field permissions. Passing the context alone does not request this check. Record-level restrictions require a synchronous, server-controlled `rowFilter`. Set this flag in server-controlled code whenever the endpoint promises authorized access. Never accept its value or copy a permission object from client data. From 8bbaa43870f4020adf7a2bcb032b8450148e49e6 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 29 Sep 2026 14:48:52 -0600 Subject: [PATCH 15/78] docs(fulltext): align Blob troubleshooting --- reference/full-text-search/overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index d8afb805a..20e2d45bd 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -70,7 +70,7 @@ Full-text search can be combined with structured filters. Harper pushes compatib - Phrase search requires `positions: true`; prefix and fuzzy-prefix require `surfaceTerms: true`. - Prefix expressions have a 100-record native result window. Use a bounded `limit`; requests beyond the window fail instead of truncating silently. - One query can use only one full-text index. An `or` group cannot mix full-text and ordinary record conditions. -- Each Blob source is limited to 1 MiB (1,048,576 bytes). Invalid UTF-8 or an oversized Blob removes the entire record from the index, including its other full-text source fields, until valid content is indexed. +- Each Blob source is limited to 1 MiB (1,048,576 bytes). A media-type mismatch, oversized value, or invalid UTF-8 removes the entire record from the index, including its other full-text source fields, until valid content is indexed. - `INDEX_REBUILDING` is transient and retryable. `DERIVED_INDEX_LAGGING` can mean a query coverage wait expired or local writes were shed after prolonged index lag. `unavailable` means automatic rebuild attempts were exhausted or activation failed; inspect readiness and logs rather than retrying every `503` indefinitely. - Generated GraphQL field arguments remain equality conditions; use `Table.search()`, REST, or `search_by_conditions` for full-text comparators. From 25599cbfe286d654ddb3e7964b69a62f7bb42524 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 29 Sep 2026 14:55:15 -0600 Subject: [PATCH 16/78] docs(fulltext): clarify search iteration --- reference/full-text-search/querying.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 2168d240d..332325364 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -9,12 +9,14 @@ title: Querying Use the query-only `FullText` field name as a condition attribute in `Table.search()`. ```javascript -const products = await Product.search({ +const products = Product.search({ conditions: [{ attribute: 'catalogSearch', comparator: 'matches', value: 'waterproof trail shoes' }], limit: 20, }); ``` +`Table.search()` returns an async iterable. Consume it with `for await...of`, or return it directly from a custom Resource to stream the results. + Results are ordered by BM25 relevance. The `FullText` field, not one of its source fields, identifies the search target. ## Match modes @@ -35,7 +37,7 @@ An empty string is invalid and returns `400`. Whitespace-only text or text reduc A query containing a negated full-text condition must also contain a non-negated full-text condition on the same index. A structured record condition alone does not satisfy this requirement. The positive full-text condition bounds the candidate set; the negated condition filters it. ```javascript -const products = await Product.search({ +const products = Product.search({ operator: 'and', conditions: [ { attribute: 'catalogSearch', comparator: 'matches', value: 'waterproof' }, @@ -49,7 +51,7 @@ const products = await Product.search({ By default, a condition searches every source field in the index. Use `fields` to restrict it: ```javascript -const products = await Product.search({ +const products = Product.search({ conditions: [ { attribute: 'catalogSearch', @@ -70,7 +72,7 @@ Direct calls through `tables` or `databases` run in a trusted server-side contex Select `$score` to include the BM25 score: ```javascript -const products = await Product.search({ +const products = Product.search({ conditions: [{ attribute: 'catalogSearch', comparator: 'matches', value: 'trail shoes' }], select: ['id', 'name', '$score'], limit: 20, @@ -88,7 +90,7 @@ Highlighting returns source-field fragments and matching character spans. The in Selecting `$highlights` turns highlighting on for the query: ```javascript -const products = await Product.search({ +const products = Product.search({ conditions: [{ attribute: 'catalogSearch', comparator: 'matches_phrase', value: 'trail running' }], select: ['id', 'name', '$score', '$highlights'], }); @@ -126,7 +128,7 @@ Offsets are UTF-16 half-open ranges. Value-level spans address the original sour Full-text conditions can be combined with structured filters using `and`: ```javascript -const products = await Product.search({ +const products = Product.search({ operator: 'and', conditions: [ { attribute: 'catalogSearch', comparator: 'matches', value: 'trail shoe' }, @@ -139,7 +141,7 @@ const products = await Product.search({ An `or` group may combine conditions from the same full-text index: ```javascript -const products = await Product.search({ +const products = Product.search({ operator: 'or', conditions: [ { attribute: 'catalogSearch', comparator: 'matches_phrase', value: 'trail running' }, From 505a706b721fdf3bdca750d1ac5201b1fc3dbeac Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Thu, 1 Oct 2026 13:24:46 -0600 Subject: [PATCH 17/78] Document full-text filter metadata --- .../full-text-search-product-catalog.mdx | 10 +++++++-- reference/full-text-search/configuration.md | 22 ++++++++++++++++++- reference/full-text-search/querying.md | 7 +++++- 3 files changed, 35 insertions(+), 4 deletions(-) diff --git a/learn/developers/full-text-search-product-catalog.mdx b/learn/developers/full-text-search-product-catalog.mdx index 57fcf07ea..5e2849da5 100644 --- a/learn/developers/full-text-search-product-catalog.mdx +++ b/learn/developers/full-text-search-product-catalog.mdx @@ -32,6 +32,7 @@ type Product @table(database: "catalog", audit: true) @export { tags: [String] category: String @indexed price: Float @indexed + available: Boolean catalogSearch: FullText @fullText( fields: [ @@ -39,6 +40,7 @@ type Product @table(database: "catalog", audit: true) @export { { name: "description", weight: 1, highlight: true } { name: "tags" } ] + filterFields: ["category", "available"] highlighting: { maxFragments: 2, fragmentLength: 120 } ) } @@ -46,6 +48,8 @@ type Product @table(database: "catalog", audit: true) @export { The query-only field name `catalogSearch` identifies the index. It is not stored with each product. `weight: 3` makes a match in `name` contribute more to BM25 relevance than the same match in `description` or `tags`. Highlighting is enabled only for the fields shown to customers. +`filterFields` copies category and availability into the Tantivy index as typed, score-neutral metadata. This lets Tantivy narrow text matches before Harper loads product records. The product record remains authoritative, and Harper rechecks the conditions before returning it. The `@indexed` declarations are independent Harper secondary indexes; they are not required for Tantivy filter metadata. + ## 2. Add records Use the generated REST endpoint: @@ -60,7 +64,8 @@ Content-Type: application/json "description": "Lightweight shoe for wet mountain trails", "tags": ["outdoor", "trail"], "category": "footwear", - "price": 129 + "price": 129, + "available": true } ``` @@ -92,6 +97,7 @@ export class SearchProducts extends Resource { value: text, fields: ['name', 'description', 'tags'], }, + { attribute: 'available', comparator: 'equals', value: true }, ]; const category = body.category; if (category) conditions.push({ attribute: 'category', comparator: 'equals', value: category }); @@ -119,7 +125,7 @@ Content-Type: application/json The category filter is optional; `q` is required. -The full-text index provides ranked candidates. The indexed category condition narrows the result set, and Harper loads the current records before returning them. +The full-text index applies the configured availability and optional category metadata while ranking candidates. Harper then loads the current records and checks those conditions again before returning them. Without `filterFields`, the same query remains valid, but Harper applies the conditions after full-text search. Calls through `tables` and `databases` use Harper's [trusted server-side context](/reference/v5/components/javascript-environment#tables). Forwarding the resource's `context` preserves the authenticated caller, and `checkPermission: true` asks the table to enforce that caller's table and source-field permissions. The custom resource must do both. Highlights can expose searched source text even when those fields are absent from `select`. diff --git a/reference/full-text-search/configuration.md b/reference/full-text-search/configuration.md index adb7237fe..cb62ed87a 100644 --- a/reference/full-text-search/configuration.md +++ b/reference/full-text-search/configuration.md @@ -14,7 +14,9 @@ type Product @table(database: "catalog", audit: true) { name: String description: String tags: [String] + category: String @indexed price: Float @indexed + available: Boolean catalogSearch: FullText @fullText( fields: [ @@ -22,6 +24,7 @@ type Product @table(database: "catalog", audit: true) { { name: "description", weight: 1, highlight: true } { name: "tags", weight: 1 } ] + filterFields: ["category", "available"] highlighting: { maxFragments: 2, fragmentLength: 120 } ) } @@ -44,6 +47,7 @@ Earlier prerelease type-level `@fullText(name: ...)` declarations are not suppor | Option | Default | Description | | -------------- | ------------- | ------------------------------------------------------------------------------------- | | `fields` | required | One or more source fields. | +| `filterFields` | `[]` | Stored attributes copied into Tantivy as typed, score-neutral filter metadata. | | `analyzer` | `"english@2"` | Versioned text analyzer. `english@2` is the only supported value. | | `stopWords` | `true` | Removes common English terms during analysis. | | `positions` | `true` | Stores token positions. Required for phrase queries. | @@ -62,6 +66,22 @@ Each field entry supports: Source fields cannot use `@computed` or `@relationship`. Indexable source values are strings, arrays containing strings or nulls, or Blob values declared as plain text. +## Filter metadata + +`filterFields` copies selected stored attributes into the Tantivy index. Tantivy can apply those conditions before returning ranked IDs, reducing record reads without changing BM25 scores. Harper still loads the returned records and checks the original conditions, permissions, and row filters before returning them. + +Schema configuration controls which conditions Tantivy can apply directly. It does not restrict Harper queries. A condition on an attribute omitted from `filterFields` still works, but Harper applies it after full-text search. Query syntax does not change; Harper pushes eligible conditions automatically. + +| Declaration | Purpose | +| ----------------------------- | ----------------------------------------------------------------------------- | +| `fields` in `@fullText` | Analyzed text that contributes to BM25 scoring | +| `filterFields` in `@fullText` | Typed metadata that narrows one full-text index without changing scores | +| `@indexed` on an attribute | Independent Harper secondary index; not required for a full-text filter field | + +Each full-text index has its own filter metadata. Configure attributes commonly combined with text search rather than every table attribute. Filter metadata increases index size and write work. + +Supported attributes are `String`, `Boolean`, `Int`, `Long`, `Float`, `Date`, and lists of those types. Equality and `in` can be pushed for every supported type. Lower-bounded and bounded ranges can be pushed for `Int`, `Long`, and `Date`. Harper retains other conditions as record filters. Lists match when any element matches. Null, missing, and empty-list values are not indexed. Filter strings are limited to 65,530 UTF-8 bytes. Computed fields and relationships are not supported. + ## Phrase and prefix support `positions` and `surfaceTerms` trade index size for query features. Keep their defaults unless storage measurements show that the features are unnecessary. @@ -143,7 +163,7 @@ type Document @table(audit: true) { ## Schema changes and rebuilds -Changes that affect indexed storage create a new local index generation and rebuild from source data. These include source membership, source order, media types, analyzer settings, stop-word handling, positions, surface terms, and synonyms. Keep the `fields` list in a stable order when its meaning has not changed. +Changes that affect indexed storage create a new local index generation and rebuild from source data. These include source membership, source order, media types, filter fields, analyzer settings, stop-word handling, positions, surface terms, and synonyms. Keep the `fields` list in a stable order when its meaning has not changed. While a generation rebuilds, ordinary table reads and writes remain available, but full-text queries on that node return retryable `503` responses until the index is ready. diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 332325364..d9cbc1fd1 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -132,12 +132,17 @@ const products = Product.search({ operator: 'and', conditions: [ { attribute: 'catalogSearch', comparator: 'matches', value: 'trail shoe' }, - { attribute: 'price', comparator: 'less_than', value: 150 }, + { attribute: 'category', comparator: 'equals', value: 'footwear' }, + { attribute: 'available', comparator: 'equals', value: true }, ], limit: 20, }); ``` +When `category` and `available` are declared in the index's `filterFields`, Harper includes these conditions in the Tantivy query. Tantivy removes non-matching IDs before Harper loads records. The metadata is score-neutral, and Harper rechecks the original conditions against current records before returning them. + +The same query remains valid when an attribute is not declared in `filterFields`; Harper applies that condition after full-text search. Small, complete result sets from Harper secondary indexes can also be passed to Tantivy automatically as candidate IDs. This path does not require `filterFields`. + An `or` group may combine conditions from the same full-text index: ```javascript From 0109329a00719d5198979c91c92e92af12272add Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Sun, 18 Oct 2026 14:20:42 -0600 Subject: [PATCH 18/78] Document full-text snapshot continuation --- reference/resources/full-text-paging.md | 155 ++++++++++++++++++++++++ sidebarsReference.ts | 5 + 2 files changed, 160 insertions(+) create mode 100644 reference/resources/full-text-paging.md diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md new file mode 100644 index 000000000..dc8413084 --- /dev/null +++ b/reference/resources/full-text-paging.md @@ -0,0 +1,155 @@ +--- +title: Full-Text Paging +--- + +# Full-Text Paging + + + +Full-text continuation lets a client page through the original search matches while indexing +updates arrive. It is available through `Table.search()` and REST. Ordinary offset-based paging +continues to run each request against its own index snapshot. + +## Declare a full-text index + +Full-text indexes require RocksDB and an audited table. The query names the `FullText` declaration, +not a stored source field: + +```graphql +type Product @table(audit: true) { + id: ID @primaryKey + title: String + catalogSearch: FullText @fullText(fields: [{ name: "title" }]) +} +``` + +`catalogSearch` is a query-only declaration. Select the stored fields and optional `$score` in results. +Full-text matches use descending relevance order. + +## Page through `Table.search()` + +Set `continuation: true` to start. Continuation mode returns a promise for a bounded array; +ordinary searches keep their async iterable return value. + +```javascript +const query = { + conditions: [{ attribute: 'catalogSearch', comparator: 'matches', value: 'trail shoes' }], + select: ['id', 'title', '$score'], + limit: 25, +}; + +let continuation = true; +do { + const page = await tables.Product.search({ ...query, continuation }); + for (const record of page) console.log(record); + continuation = page.continuation; +} while (continuation); +``` + +The returned array has three additional properties: + +| Property | Meaning | +| -------------- | ------------------------------------------------------------------- | +| `continuation` | Token for the next page; absent when traversal is complete | +| `complete` | Whether the original search has been exhausted or explicitly closed | +| `expiresAt` | Absolute expiry time in milliseconds since the Unix epoch | + +These properties are not included when the array is serialized as JSON. Keep the same conditions +and selected fields when resuming. The page size may change. Do not combine continuation with a +nonzero offset, count request, or another sort order. + +In an authenticated custom Resource, pass `this.getContext()` as the second argument to +`tables.Product.search()` and set `checkPermission: true` in the query. This requests the caller's +table and searched-source permissions; passing context alone does not request that permission gate. +Application row filters still run on each page. + +## Page through REST + +Start with `continuation()`: + +```http +GET /Product/?catalogSearch=matches=trail%20shoes&select(id,title,$score)&limit(25)&continuation() +``` + +The body stays a JSON array. Read the next token from `Harper-Continuation`, URL-encode it, and +send it in `continuation(token)` with the same query: + +```javascript +const query = '/Product/?catalogSearch=matches=trail%20shoes&select(id,title,$score)&limit(25)'; +let token; +do { + const continuation = token ? `continuation(${encodeURIComponent(token)})` : 'continuation()'; + const response = await fetch(`${query}&${continuation}`); + if (!response.ok) throw new Error(await response.text()); + const records = await response.json(); + for (const record of records) console.log(record); + token = response.headers.get('Harper-Continuation'); +} while (token); +``` + +`Harper-Continuation-Complete` reports completion. `Harper-Continuation-Expires` contains the +absolute expiry time in milliseconds since the Unix epoch. These headers are exposed to browser +clients. Continuation responses use `Cache-Control: no-store`. JSON `QUERY` requests return the +same headers when their query body enables continuation. + +## Records and permissions can change + +The snapshot fixes the index matches and their ranking. It does not retain historical Harper +records or permissions. Each request loads current records and checks current access. Updated, +deleted, expired, or newly unauthorized records can be omitted, and new matches do not enter the +original search. + +Retrying a token resumes the same index boundary. The returned rows can differ if records or +permissions changed. A page may be empty when the remaining snapshot contains no eligible records; +use the continuation token or completion flag to decide when to stop. + +## Close or restart a search + +To stop early, repeat the query with its current token and `closeContinuation: true`. In REST, add +`closeContinuation()` alongside `continuation(token)`. Closing returns an empty completed page. + +Completion, close, expiry, index reset, definition changes, worker loss, or process restart can +invalidate a token. Resuming an unavailable snapshot returns `410`; start a new search. Once any +request finishes traversal, older tokens for that search also expire. + +Snapshots belong to one Harper instance. Its workers can handle continuation requests for each +other. Requests reaching a different Harper instance cannot resume that snapshot. + +## Configure resource limits + +Settings live under `storage.fullText` in `harper-config.yaml`: + +```yaml +storage: + fullText: + cursorTtlMilliseconds: 60000 + maxCursors: 4 + maxPageSize: 1000 + requestBudgetMilliseconds: 5000 + maxRawHitsPerRequest: 100000 +``` + +| Setting | Default | Scope | +| --------------------------- | ------: | ----------------------------------------------------------- | +| `cursorTtlMilliseconds` | 60,000 | Absolute search lifetime; paging does not extend it | +| `maxCursors` | 4 | Retained searches across all workers in one process | +| `maxRetainedQueryBytes` | 8 MiB | Retained native query text and identity bytes | +| `maxPageSize` | 1,000 | Returned records per page | +| `requestBudgetMilliseconds` | 5,000 | Freshness wait and page processing budget | +| `maxRawHitsPerRequest` | 100,000 | Fetched native hits per request, including replayed batches | +| `maxPendingRequests` | 32 | Pending requests per requesting worker | +| `maxPendingResponseBytes` | 16 MiB | Reserved response bytes per requesting worker | + +The usual individual configuration names also work, such as +`storage_fullText_maxRawHitsPerRequest` with `set_configuration`. Native process-wide admission +settings live under `storage.fullText.runtime` and require a restart when changed. Their defaults +are 128 resident indexes, 128 indexing threads, 256 search threads, 4 GiB configured writer memory, +8 GiB configured queue capacity, and 8 concurrent expensive searches. All limits apply together. + +A raw-hit budget failure returns `400`; narrow the query or increase the budget. Capacity exhaustion +and timeouts return errors instead of successful incomplete pages. If the snapshot itself runs out +of eligible matches, a shorter completed page is a valid result. + +Retained readers can keep old index segments on disk until released. These settings do not provide +a hard limit on total process memory or retained disk space. Native deadlines are cooperative, so the +request budget is not a hard wall-clock guarantee. diff --git a/sidebarsReference.ts b/sidebarsReference.ts index f997c1810..d7ea4b862 100644 --- a/sidebarsReference.ts +++ b/sidebarsReference.ts @@ -87,6 +87,11 @@ const sidebars: SidebarsConfig = { id: 'resources/query-optimization', label: 'Query Optimization', }, + { + type: 'doc', + id: 'resources/full-text-paging', + label: 'Full-Text Paging', + }, ], }, { From 8b0809102ff78bba38c0d874afd157c0791f52b2 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Sun, 18 Oct 2026 17:51:47 -0600 Subject: [PATCH 19/78] Clarify continuation callback and shared capacity requirements --- reference/resources/full-text-paging.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index dc8413084..8512fefa8 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -62,6 +62,8 @@ In an authenticated custom Resource, pass `this.getContext()` as the second argu `tables.Product.search()` and set `checkPermission: true` in the query. This requests the caller's table and searched-source permissions; passing context alone does not request that permission gate. Application row filters still run on each page. +The token does not store application callbacks such as `rowFilter` or `vectorFilter`. Supply them +again on every page, together with the caller's context and permission settings. ## Page through REST @@ -111,6 +113,8 @@ To stop early, repeat the query with its current token and `closeContinuation: t Completion, close, expiry, index reset, definition changes, worker loss, or process restart can invalidate a token. Resuming an unavailable snapshot returns `410`; start a new search. Once any request finishes traversal, older tokens for that search also expire. +Changing any `storage.fullText` setting also invalidates existing tokens, even when only a capacity +setting changes. Start a new search after changing these settings. Snapshots belong to one Harper instance. Its workers can handle continuation requests for each other. Requests reaching a different Harper instance cannot resume that snapshot. @@ -146,6 +150,11 @@ settings live under `storage.fullText.runtime` and require a restart when change are 128 resident indexes, 128 indexing threads, 256 search threads, 4 GiB configured writer memory, 8 GiB configured queue capacity, and 8 concurrent expensive searches. All limits apply together. +The cursor pool is shared by all callers. Four unfinished searches fill the default pool, so a fifth +receives `503` until a search finishes, closes, or expires. Ordinary searches remain available. +Configure `maxCursors` for the number of unfinished searches the application needs to retain. +There is no per-caller reservation or eviction policy. + A raw-hit budget failure returns `400`; narrow the query or increase the budget. Capacity exhaustion and timeouts return errors instead of successful incomplete pages. If the snapshot itself runs out of eligible matches, a shorter completed page is a valid result. From 06a5a9bbd6887da8ddfe312433ec4f03bd9ee0c8 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Sun, 18 Oct 2026 21:11:13 -0600 Subject: [PATCH 20/78] Document continuation transport sizes and joint admission limits --- reference/resources/full-text-paging.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index 8512fefa8..07d08b11c 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -94,6 +94,11 @@ absolute expiry time in milliseconds since the Unix epoch. These headers are exp clients. Continuation responses use `Cache-Control: no-store`. JSON `QUERY` requests return the same headers when their query body enables continuation. +Long record IDs make continuation tokens larger. Qualification with the released library produced +tokens of about 600 bytes for short IDs and nearly 8 KB near the supported ID limit. Review Harper's +`http.maxHeaderSize` setting and any proxy response-header and request-line limits. Leave room for the +token, query, authentication, and other headers; a token that fits Harper may exceed a proxy limit. + ## Records and permissions can change The snapshot fixes the index matches and their ranking. It does not retain historical Harper @@ -155,6 +160,12 @@ receives `503` until a search finishes, closes, or expires. Ordinary searches re Configure `maxCursors` for the number of unfinished searches the application needs to retain. There is no per-caller reservation or eviction policy. +Cursor slots, pending-request counts, and response-byte allowances apply separately. Raising +`maxCursors` does not raise `maxPendingRequests` or `maxPendingResponseBytes`. Across workers, +Harper reserves response space using the library's largest supported IDs and versions, rather than +the average size of the current records. Many searches may be retained while fewer pages can run +at once. + A raw-hit budget failure returns `400`; narrow the query or increase the budget. Capacity exhaustion and timeouts return errors instead of successful incomplete pages. If the snapshot itself runs out of eligible matches, a shorter completed page is a valid result. From 6923c72c2ca8b75af282db35f9e7fcb2cc0d3351 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Sun, 18 Oct 2026 21:43:22 -0600 Subject: [PATCH 21/78] Clarify close acknowledgement and completed-page metadata --- reference/resources/full-text-paging.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index 07d08b11c..996041829 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -90,8 +90,8 @@ do { ``` `Harper-Continuation-Complete` reports completion. `Harper-Continuation-Expires` contains the -absolute expiry time in milliseconds since the Unix epoch. These headers are exposed to browser -clients. Continuation responses use `Cache-Control: no-store`. JSON `QUERY` requests return the +absolute expiry time in milliseconds since the Unix epoch. Completed pages omit the token and +expiry. These headers are exposed to browser clients. Continuation responses use `Cache-Control: no-store`. JSON `QUERY` requests return the same headers when their query body enables continuation. Long record IDs make continuation tokens larger. Qualification with the released library produced @@ -115,6 +115,11 @@ use the continuation token or completion flag to decide when to stop. To stop early, repeat the query with its current token and `closeContinuation: true`. In REST, add `closeContinuation()` alongside `continuation(token)`. Closing returns an empty completed page. +Close and terminal pages wait for admission release before reporting success. If release fails, the +request returns an error instead. The revoked token may then return `410`, and its admission slot +can remain occupied until absolute expiry or worker exit. A successful close releases admission +capacity; it does not promise immediate removal of old index segments from disk. + Completion, close, expiry, index reset, definition changes, worker loss, or process restart can invalidate a token. Resuming an unavailable snapshot returns `410`; start a new search. Once any request finishes traversal, older tokens for that search also expire. From 3fbaced8521a18fa568ae4baa325462d2b266cf1 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 19 Oct 2026 06:23:45 -0600 Subject: [PATCH 22/78] Explain HEAD behavior for full-text continuation --- reference/resources/full-text-paging.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index 996041829..f51ae07d0 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -94,6 +94,8 @@ absolute expiry time in milliseconds since the Unix epoch. Completed pages omit expiry. These headers are exposed to browser clients. Continuation responses use `Cache-Control: no-store`. JSON `QUERY` requests return the same headers when their query body enables continuation. +HEAD rejects continuation requests because it cannot return the page records. Use GET or QUERY. + Long record IDs make continuation tokens larger. Qualification with the released library produced tokens of about 600 bytes for short IDs and nearly 8 KB near the supported ID limit. Review Harper's `http.maxHeaderSize` setting and any proxy response-header and request-line limits. Leave room for the From 8ea7e2c2f2f9ad252799441b2affb911c1694b26 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 19 Oct 2026 07:09:12 -0600 Subject: [PATCH 23/78] Document acknowledged continuation completion after cleanup --- reference/resources/full-text-paging.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index f51ae07d0..01a235bc9 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -121,6 +121,8 @@ Close and terminal pages wait for admission release before reporting success. If request returns an error instead. The revoked token may then return `410`, and its admission slot can remain occupied until absolute expiry or worker exit. A successful close releases admission capacity; it does not promise immediate removal of old index segments from disk. +Closing has a separate cleanup budget. Once release is acknowledged, the completed page can return +after its search-work deadline. Cancellation, transaction, and index-lifecycle checks still apply. Completion, close, expiry, index reset, definition changes, worker loss, or process restart can invalidate a token. Resuming an unavailable snapshot returns `410`; start a new search. Once any From e29315e59f6a4690d38ff98cc7870a111fe0fa59 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 19 Oct 2026 08:15:03 -0600 Subject: [PATCH 24/78] Explain continuation cleanup timing and filtering costs --- reference/resources/full-text-paging.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index 01a235bc9..967406a6a 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -158,6 +158,18 @@ storage: | `maxPendingRequests` | 32 | Pending requests per requesting worker | | `maxPendingResponseBytes` | 16 MiB | Reserved response bytes per requesting worker | +Search work and cleanup use separate windows based on `requestBudgetMilliseconds`. With the +5,000 ms default, a search that uses its full work budget and then closes can take about 10 seconds. +Waiting for other page requests already using the snapshot can add time. Native work checks its +deadline while running; these settings are not a hard limit on total response time. + +Ordinary searches can use secondary indexes to skip record reads and enlarge batches when many +matches are filtered out. Continued searches do not yet use those optimizations; they fetch up to +256 native hits per batch and check current records. A selective filter can therefore add many +record reads and, when the snapshot belongs to another worker, many worker requests. +Qualification and these optimizations are tracked in +[Qualify native full-text filtering and catalog-scale operation for v5.4](https://github.com/HarperFast/harper/issues/2512). + The usual individual configuration names also work, such as `storage_fullText_maxRawHitsPerRequest` with `set_configuration`. Native process-wide admission settings live under `storage.fullText.runtime` and require a restart when changed. Their defaults From 9dc5dc57e82806a025e14fe4c817f2f8535aab80 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 19 Oct 2026 10:31:06 -0600 Subject: [PATCH 25/78] Document continuation method restrictions --- reference/resources/full-text-paging.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index 967406a6a..c335625ed 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -94,7 +94,8 @@ absolute expiry time in milliseconds since the Unix epoch. Completed pages omit expiry. These headers are exposed to browser clients. Continuation responses use `Cache-Control: no-store`. JSON `QUERY` requests return the same headers when their query body enables continuation. -HEAD rejects continuation requests because it cannot return the page records. Use GET or QUERY. +Only GET and QUERY can return continuation pages. HEAD and mutating methods reject continuation +parameters with 400. OPTIONS still returns request metadata without running a search. Long record IDs make continuation tokens larger. Qualification with the released library produced tokens of about 600 bytes for short IDs and nearly 8 KB near the supported ID limit. Review Harper's From 8dee0125d3468923dcb6a9d8f31bea26f092c469 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 19 Oct 2026 10:43:21 -0600 Subject: [PATCH 26/78] Clarify supported continuation methods --- reference/resources/full-text-paging.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index c335625ed..0606d1626 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -94,8 +94,8 @@ absolute expiry time in milliseconds since the Unix epoch. Completed pages omit expiry. These headers are exposed to browser clients. Continuation responses use `Cache-Control: no-store`. JSON `QUERY` requests return the same headers when their query body enables continuation. -Only GET and QUERY can return continuation pages. HEAD and mutating methods reject continuation -parameters with 400. OPTIONS still returns request metadata without running a search. +GET and QUERY can return continuation pages. OPTIONS returns request metadata without running a +search. Other methods reject continuation parameters with 400. Long record IDs make continuation tokens larger. Qualification with the released library produced tokens of about 600 bytes for short IDs and nearly 8 KB near the supported ID limit. Review Harper's From f23ba40fc04d7b69867435614b4abeb5911e296e Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 07:17:14 -0600 Subject: [PATCH 27/78] Clarify continuation availability and response handling --- reference/resources/full-text-paging.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index 0606d1626..4d55ee9de 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -4,11 +4,12 @@ title: Full-Text Paging # Full-Text Paging - + Full-text continuation lets a client page through the original search matches while indexing updates arrive. It is available through `Table.search()` and REST. Ordinary offset-based paging continues to run each request against its own index snapshot. +The same continuation options work with autocomplete searches using `matches_prefix` and `matches_fuzzy_prefix`. ## Declare a full-text index @@ -28,7 +29,8 @@ Full-text matches use descending relevance order. ## Page through `Table.search()` -Set `continuation: true` to start. Continuation mode returns a promise for a bounded array; +Set `continuation: true` and a positive `limit` to start. The limit must not exceed `maxPageSize`. +Continuation mode returns a promise for a bounded array; ordinary searches keep their async iterable return value. ```javascript @@ -56,7 +58,7 @@ The returned array has three additional properties: These properties are not included when the array is serialized as JSON. Keep the same conditions and selected fields when resuming. The page size may change. Do not combine continuation with a -nonzero offset, count request, or another sort order. +nonzero offset, count request, another sort order, `explain`, or `sync`. In an authenticated custom Resource, pass `this.getContext()` as the second argument to `tables.Product.search()` and set `checkPermission: true` in the query. This requests the caller's @@ -94,6 +96,9 @@ absolute expiry time in milliseconds since the Unix epoch. Completed pages omit expiry. These headers are exposed to browser clients. Continuation responses use `Cache-Control: no-store`. JSON `QUERY` requests return the same headers when their query body enables continuation. +Read and parse the complete response body before saving its next token. A stream can fail after +headers are sent. If that happens, retry with the previous token while the snapshot remains valid. + GET and QUERY can return continuation pages. OPTIONS returns request metadata without running a search. Other methods reject continuation parameters with 400. From ebfebdbe5b9ebf961aeee98458941611293aff3e Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 07:29:20 -0600 Subject: [PATCH 28/78] Clarify continuation context and HTTP exceptions --- reference/resources/full-text-paging.md | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index 4d55ee9de..a72fa6b2c 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -29,7 +29,7 @@ Full-text matches use descending relevance order. ## Page through `Table.search()` -Set `continuation: true` and a positive `limit` to start. The limit must not exceed `maxPageSize`. +Set `continuation: true` and a positive integer `limit` to start. The limit must not exceed `maxPageSize`. Continuation mode returns a promise for a bounded array; ordinary searches keep their async iterable return value. @@ -57,14 +57,17 @@ The returned array has three additional properties: | `expiresAt` | Absolute expiry time in milliseconds since the Unix epoch | These properties are not included when the array is serialized as JSON. Keep the same conditions -and selected fields when resuming. The page size may change. Do not combine continuation with a -nonzero offset, count request, another sort order, `explain`, or `sync`. - -In an authenticated custom Resource, pass `this.getContext()` as the second argument to -`tables.Product.search()` and set `checkPermission: true` in the query. This requests the caller's +and selected fields when resuming. The page size may change. Use the default relevance order, or explicitly sort by descending +`$score` alone. A nonzero `offset`, another sort, or a truthy `count`, `explain`, or `sync` option +returns 400. + +In a static custom Resource method, pass the method's `context` argument as the second argument +to `tables.Product.search()`. In an instance method, use `this.getContext()` instead. See +[Resource context](./resource-api.md#getcontext-context) for both forms. Set `checkPermission: true` +in the query. This requests the caller's table and searched-source permissions; passing context alone does not request that permission gate. Application row filters still run on each page. -The token does not store application callbacks such as `rowFilter` or `vectorFilter`. Supply them +The token does not store application callbacks such as `rowFilter`. Supply them again on every page, together with the caller's context and permission settings. ## Page through REST @@ -93,14 +96,16 @@ do { `Harper-Continuation-Complete` reports completion. `Harper-Continuation-Expires` contains the absolute expiry time in milliseconds since the Unix epoch. Completed pages omit the token and -expiry. These headers are exposed to browser clients. Continuation responses use `Cache-Control: no-store`. JSON `QUERY` requests return the +expiry. Harper adds these names to `Access-Control-Expose-Headers` so browser clients can read them. +Continuation responses override the usual read-cache policy with `Cache-Control: no-store`. JSON `QUERY` requests return the same headers when their query body enables continuation. Read and parse the complete response body before saving its next token. A stream can fail after headers are sent. If that happens, retry with the previous token while the snapshot remains valid. GET and QUERY can return continuation pages. OPTIONS returns request metadata without running a -search. Other methods reject continuation parameters with 400. +search. HEAD is an exception to the usual GET-without-a-body behavior: it returns 400 before +opening or advancing a continuation. Other methods also reject continuation parameters with 400. Long record IDs make continuation tokens larger. Qualification with the released library produced tokens of about 600 bytes for short IDs and nearly 8 KB near the supported ID limit. Review Harper's @@ -148,6 +153,8 @@ storage: fullText: cursorTtlMilliseconds: 60000 maxCursors: 4 + maxRetainedQueryBytes: 8388608 + maxPendingResponseBytes: 16777216 maxPageSize: 1000 requestBudgetMilliseconds: 5000 maxRawHitsPerRequest: 100000 From 48411eeb8c0d2bf32a9e45aabafc058b40229742 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 07:32:27 -0600 Subject: [PATCH 29/78] Complete continuation examples and retry guidance --- reference/resources/full-text-paging.md | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index a72fa6b2c..2d2ad7be2 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -4,6 +4,8 @@ title: Full-Text Paging # Full-Text Paging +Continuation paging is available in v5.4.0. The full-text index declaration below is a prerequisite. + Full-text continuation lets a client page through the original search matches while indexing @@ -17,7 +19,7 @@ Full-text indexes require RocksDB and an audited table. The query names the `Ful not a stored source field: ```graphql -type Product @table(audit: true) { +type Product @table(audit: true) @export { id: ID @primaryKey title: String catalogSearch: FullText @fullText(fields: [{ name: "title" }]) @@ -25,7 +27,8 @@ type Product @table(audit: true) { ``` `catalogSearch` is a query-only declaration. Select the stored fields and optional `$score` in results. -Full-text matches use descending relevance order. +Full-text matches use descending relevance order. REST examples also require `rest: true` in the +application configuration; see [REST setup](../rest/overview.md). ## Page through `Table.search()` @@ -57,7 +60,7 @@ The returned array has three additional properties: | `expiresAt` | Absolute expiry time in milliseconds since the Unix epoch | These properties are not included when the array is serialized as JSON. Keep the same conditions -and selected fields when resuming. The page size may change. Use the default relevance order, or explicitly sort by descending +and selected fields, and use the same authenticated caller when resuming. The page size may change. Use the default relevance order, or explicitly sort by descending `$score` alone. A nonzero `offset`, another sort, or a truthy `count`, `explain`, or `sync` option returns 400. @@ -66,13 +69,17 @@ to `tables.Product.search()`. In an instance method, use `this.getContext()` ins [Resource context](./resource-api.md#getcontext-context) for both forms. Set `checkPermission: true` in the query. This requests the caller's table and searched-source permissions; passing context alone does not request that permission gate. -Application row filters still run on each page. +Changing the caller identity, including switching from an authenticated request to an internal +call without that context, returns 410. Permission changes for the same caller are checked on each +page. Application row filters still run on each page. The token does not store application callbacks such as `rowFilter`. Supply them again on every page, together with the caller's context and permission settings. ## Page through REST -Start with `continuation()`: +Use the single-argument `limit(n)` page size and start with `continuation()`. Do not use +`limit(start,end)`, request a count with `Prefer: count=`, or change the relevance order. The same +`explain` and `sync` restrictions as `Table.search()` apply; violations return 400. ```http GET /Product/?catalogSearch=matches=trail%20shoes&select(id,title,$score)&limit(25)&continuation() @@ -101,7 +108,9 @@ Continuation responses override the usual read-cache policy with `Cache-Control: same headers when their query body enables continuation. Read and parse the complete response body before saving its next token. A stream can fail after -headers are sent. If that happens, retry with the previous token while the snapshot remains valid. +headers are sent. If a resumed page fails this way, retry with the previous token while the snapshot remains valid. +If no token was received on the first page, start a new search. The first search may keep its cursor +slot until absolute expiry because it cannot be closed without a token. GET and QUERY can return continuation pages. OPTIONS returns request metadata without running a search. HEAD is an exception to the usual GET-without-a-body behavior: it returns 400 before From c7e263e84d9135e8cac9f9f77b5b0a2b1ffc106d Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 07:37:49 -0600 Subject: [PATCH 30/78] Link continuation from configuration and REST references --- reference/configuration/options.md | 4 ++++ reference/resources/resource-api.md | 4 ++++ reference/rest/headers.md | 6 ++++++ reference/rest/overview.md | 4 +++- reference/rest/querying.md | 10 ++++++++++ 5 files changed, 27 insertions(+), 1 deletion(-) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index d12128410..23a5bf733 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -302,6 +302,10 @@ replication: Database storage configuration. See [Storage Tuning](../database/storage-tuning.md) for guidance on tuning these options for production workloads, [Database Overview](../database/overview.md) for general database concepts, and [Compaction](../database/compaction.md) for reclaiming space inside existing files. +`storage.fullText` controls retained full-text searches and native search capacity. See +[Full-Text Paging](../resources/full-text-paging.md#configure-resource-limits) for the settings, +defaults and restart requirements. + ```yaml storage: path: ~/hdb/database diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index da064a259..d76d88fb7 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1035,6 +1035,10 @@ Array of condition objects to filter records. Each condition: | `conditions` | Nested conditions array | | `operator` | `and` (default) or `or` for the nested `conditions` | +Full-text conditions name a query-only `FullText` declaration and use comparators such as `matches`, +`matches_prefix`, and `matches_fuzzy_prefix`. See +[Full-Text Paging](./full-text-paging.md) for snapshot continuation and its query restrictions. + Example with nested conditions: ```javascript diff --git a/reference/rest/headers.md b/reference/rest/headers.md index 71603c802..8cb4675b9 100644 --- a/reference/rest/headers.md +++ b/reference/rest/headers.md @@ -22,11 +22,17 @@ These headers are included in all Harper REST API responses: Collection responses to a [count request](./querying.md#pagination-and-total-count) additionally include `Content-Range`, `Range-Unit`, and `Preference-Applied` (). +Full-text continuation responses also return `Harper-Continuation-Complete`, plus +`Harper-Continuation` and `Harper-Continuation-Expires` while the search remains open. These names +are added to `Access-Control-Expose-Headers`. See +[Full-Text Paging](../resources/full-text-paging.md#page-through-rest). + ## Cache-Control Harper applies a tiered Cache-Control policy to REST responses. +Full-text continuation responses override this policy with `Cache-Control: no-store`. ### Anonymous reads diff --git a/reference/rest/overview.md b/reference/rest/overview.md index 0f60b27c6..86b1826e4 100644 --- a/reference/rest/overview.md +++ b/reference/rest/overview.md @@ -88,7 +88,9 @@ With both in place, the exported name becomes the base path (`Product` above, or Notes on this surface: - The trailing slash is significant throughout: `/Product` addresses the resource itself, `/Product/` addresses its collection of records. See [URL Structure](#url-structure). -- `HEAD` is served exactly as `GET` with the response body omitted. `QUERY` is accepted on the collection path (`QUERY /Product/`) and runs a search taken from the request body rather than the URL. +- `HEAD` is served as `GET` with the response body omitted, except that + [full-text continuation requests](../resources/full-text-paging.md#page-through-rest) return `400` + before opening or advancing a search. `QUERY` is accepted on the collection path (`QUERY /Product/`) and runs a search taken from the request body rather than the URL. - A collection `POST` only generates a primary key when the body omits the table's primary-key property. If the body supplies one, Harper stores the record under that value rather than replacing it, and a `POST` to a key that already exists fails with `409` instead of overwriting. - On a successful `POST`, the new record's primary key — supplied or generated — is returned in the `Location` response header. The header carries the bare key value, not a URL, so it is not a link to follow. - `PUT` replaces the stored record, with three exceptions that Harper always applies: a [`@createdTime`](../database/schema.md#createdtime) attribute keeps the original record's value, an [`@updatedTime`](../database/schema.md#updatedtime) attribute is re-stamped with the time of the write, and the primary key is forced to match the `{id}` in the URL even if the body carries a different one. diff --git a/reference/rest/querying.md b/reference/rest/querying.md index b8d7d0e66..d2cd8a308 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -136,6 +136,16 @@ GET /Product/?price=lt=100|[rating=5&[tag=fast|tag=scalable|tag=efficient]&inSto Harper supports special query functions using call syntax, included in the query string separated by `&`. +### `continuation()` and `closeContinuation()` + + + +Full-text searches can retain their original index snapshot between requests. Start with +`continuation()`, resume with `continuation(token)`, and close early with `closeContinuation()` +alongside the token. Use `limit(n)` without a nonzero offset or a count request. See +[Full-Text Paging](../resources/full-text-paging.md#page-through-rest) for the token headers, +query restrictions and recovery steps. + ### `select(properties)` Specify which properties to include in the response. From f04f3e6abad83f36ed7f31a85b2615ef34c3e4e3 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 07:54:20 -0600 Subject: [PATCH 31/78] Tighten paging references and callback guidance --- reference/resources/full-text-paging.md | 11 +++++++---- reference/resources/overview.md | 1 + reference/resources/resource-api.md | 4 ++-- 3 files changed, 10 insertions(+), 6 deletions(-) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index 2d2ad7be2..110b11f29 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -55,7 +55,7 @@ The returned array has three additional properties: | Property | Meaning | | -------------- | ------------------------------------------------------------------- | -| `continuation` | Token for the next page; absent when traversal is complete | +| `continuation` | Token for the next page; `undefined` when traversal is complete | | `complete` | Whether the original search has been exhausted or explicitly closed | | `expiresAt` | Absolute expiry time in milliseconds since the Unix epoch | @@ -72,13 +72,14 @@ table and searched-source permissions; passing context alone does not request th Changing the caller identity, including switching from an authenticated request to an internal call without that context, returns 410. Permission changes for the same caller are checked on each page. Application row filters still run on each page. -The token does not store application callbacks such as `rowFilter`. Supply them +The token does not store application callbacks such as `rowFilter` or +[`vectorFilter`](../database/schema.md#filtered-vector-search-with-a-function-predicate). Supply them again on every page, together with the caller's context and permission settings. ## Page through REST Use the single-argument `limit(n)` page size and start with `continuation()`. Do not use -`limit(start,end)`, request a count with `Prefer: count=`, or change the relevance order. The same +a nonzero starting offset with `limit(start,end)`, request a count with `Prefer: count=`, or change the relevance order. The same `explain` and `sync` restrictions as `Table.search()` apply; violations return 400. ```http @@ -193,7 +194,9 @@ Qualification and these optimizations are tracked in [Qualify native full-text filtering and catalog-scale operation for v5.4](https://github.com/HarperFast/harper/issues/2512). The usual individual configuration names also work, such as -`storage_fullText_maxRawHitsPerRequest` with `set_configuration`. Native process-wide admission +`storage_fullText_maxRawHitsPerRequest` with `set_configuration`. Follow the +[configuration restart procedure](../configuration/operations.md#set-configuration) after applying +changes with that operation. Native process-wide admission settings live under `storage.fullText.runtime` and require a restart when changed. Their defaults are 128 resident indexes, 128 indexing threads, 256 search threads, 4 GiB configured writer memory, 8 GiB configured queue capacity, and 8 concurrent expensive searches. All limits apply together. diff --git a/reference/resources/overview.md b/reference/resources/overview.md index d0574af6b..4a00d8f17 100644 --- a/reference/resources/overview.md +++ b/reference/resources/overview.md @@ -172,3 +172,4 @@ Parameterized routes also surface in the generated OpenAPI document (as template | --------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | [Resource API](./resource-api.md) | Complete reference for instance methods, static methods, the Query object, RequestTarget, and response handling | | [Query Optimization](./query-optimization.md) | How Harper executes queries and how to write performant conditions | +| [Full-Text Paging](./full-text-paging.md) | Resume full-text and autocomplete searches on their original index snapshot | diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index d76d88fb7..485a89cb4 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1035,8 +1035,8 @@ Array of condition objects to filter records. Each condition: | `conditions` | Nested conditions array | | `operator` | `and` (default) or `or` for the nested `conditions` | -Full-text conditions name a query-only `FullText` declaration and use comparators such as `matches`, -`matches_prefix`, and `matches_fuzzy_prefix`. See +Full-text conditions name a query-only `FullText` declaration and use comparators `matches`, +`matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix`. See [Full-Text Paging](./full-text-paging.md) for snapshot continuation and its query restrictions. Example with nested conditions: From e22a0e6bfaa6a684ac49e207979ccba9acf134ca Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 07:55:30 -0600 Subject: [PATCH 32/78] Show copyable native runtime capacity settings --- reference/resources/full-text-paging.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index 110b11f29..54a5ef02f 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -201,6 +201,20 @@ settings live under `storage.fullText.runtime` and require a restart when change are 128 resident indexes, 128 indexing threads, 256 search threads, 4 GiB configured writer memory, 8 GiB configured queue capacity, and 8 concurrent expensive searches. All limits apply together. +The runtime keys use integer counts or bytes: + +```yaml +storage: + fullText: + runtime: + maxResidentIndexes: 128 + maxIndexingThreads: 128 + maxSearchThreads: 256 + maxWriterMemoryBytes: 4294967296 + maxQueuedBytes: 8589934592 + maxExpensiveSearches: 8 +``` + The cursor pool is shared by all callers. Four unfinished searches fill the default pool, so a fifth receives `503` until a search finishes, closes, or expires. Ordinary searches remain available. Configure `maxCursors` for the number of unfinished searches the application needs to retain. From b995dbe8ca953d8e19afbf320e20dc864af59e29 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 10:04:19 -0600 Subject: [PATCH 33/78] Place continuation version badges consistently --- reference/configuration/options.md | 4 ++-- reference/resources/full-text-paging.md | 4 ++-- reference/rest/headers.md | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 23a5bf733..cca263158 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -302,9 +302,9 @@ replication: Database storage configuration. See [Storage Tuning](../database/storage-tuning.md) for guidance on tuning these options for production workloads, [Database Overview](../database/overview.md) for general database concepts, and [Compaction](../database/compaction.md) for reclaiming space inside existing files. -`storage.fullText` controls retained full-text searches and native search capacity. See +`storage.fullText` (v5.4.0) controls retained full-text searches and native search capacity. See [Full-Text Paging](../resources/full-text-paging.md#configure-resource-limits) for the settings, -defaults and restart requirements. +defaults and restart requirements. ```yaml storage: diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index 54a5ef02f..0e9f1d002 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -4,10 +4,10 @@ title: Full-Text Paging # Full-Text Paging -Continuation paging is available in v5.4.0. The full-text index declaration below is a prerequisite. - +The full-text index declaration below is a prerequisite. + Full-text continuation lets a client page through the original search matches while indexing updates arrive. It is available through `Table.search()` and REST. Ordinary offset-based paging continues to run each request against its own index snapshot. diff --git a/reference/rest/headers.md b/reference/rest/headers.md index 8cb4675b9..7a64a1f39 100644 --- a/reference/rest/headers.md +++ b/reference/rest/headers.md @@ -25,7 +25,7 @@ Collection responses to a [count request](./querying.md#pagination-and-total-cou Full-text continuation responses also return `Harper-Continuation-Complete`, plus `Harper-Continuation` and `Harper-Continuation-Expires` while the search remains open. These names are added to `Access-Control-Expose-Headers`. See -[Full-Text Paging](../resources/full-text-paging.md#page-through-rest). +[Full-Text Paging](../resources/full-text-paging.md#page-through-rest) (v5.4.0). ## Cache-Control From 66dfb1708b112312145b23247badb535b6c10bc9 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 16:46:12 -0600 Subject: [PATCH 34/78] Document explicit full-text paging and ordinary search budgets --- reference/resources/full-text-paging.md | 44 +++++++++++++++---------- reference/resources/resource-api.md | 2 +- 2 files changed, 27 insertions(+), 19 deletions(-) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index 0e9f1d002..664ecc51f 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -9,7 +9,7 @@ title: Full-Text Paging The full-text index declaration below is a prerequisite. Full-text continuation lets a client page through the original search matches while indexing -updates arrive. It is available through `Table.search()` and REST. Ordinary offset-based paging +updates arrive. It is available through `Table.searchPage()` and REST. Ordinary offset-based paging continues to run each request against its own index snapshot. The same continuation options work with autocomplete searches using `matches_prefix` and `matches_fuzzy_prefix`. @@ -30,11 +30,11 @@ type Product @table(audit: true) @export { Full-text matches use descending relevance order. REST examples also require `rest: true` in the application configuration; see [REST setup](../rest/overview.md). -## Page through `Table.search()` +## Page through `Table.searchPage()` Set `continuation: true` and a positive integer `limit` to start. The limit must not exceed `maxPageSize`. -Continuation mode returns a promise for a bounded array; -ordinary searches keep their async iterable return value. +`searchPage()` returns a promise for a bounded array. Omit `continuation` or set it to `true` to start. +`Table.search()` keeps its async iterable return value and rejects continuation options. ```javascript const query = { @@ -45,7 +45,7 @@ const query = { let continuation = true; do { - const page = await tables.Product.search({ ...query, continuation }); + const page = await tables.Product.searchPage({ ...query, continuation }); for (const record of page) console.log(record); continuation = page.continuation; } while (continuation); @@ -65,7 +65,7 @@ and selected fields, and use the same authenticated caller when resuming. The pa returns 400. In a static custom Resource method, pass the method's `context` argument as the second argument -to `tables.Product.search()`. In an instance method, use `this.getContext()` instead. See +to `tables.Product.searchPage()`. In an instance method, use `this.getContext()` instead. See [Resource context](./resource-api.md#getcontext-context) for both forms. Set `checkPermission: true` in the query. This requests the caller's table and searched-source permissions; passing context alone does not request that permission gate. @@ -76,11 +76,15 @@ The token does not store application callbacks such as `rowFilter` or [`vectorFilter`](../database/schema.md#filtered-vector-search-with-a-function-predicate). Supply them again on every page, together with the caller's context and permission settings. +If your table overrides `search()`, also implement `searchPage()` with the same filtering policy. +This applies to static and instance methods. Otherwise paging returns 400, including through REST, +to avoid bypassing application filters. + ## Page through REST Use the single-argument `limit(n)` page size and start with `continuation()`. Do not use a nonzero starting offset with `limit(start,end)`, request a count with `Prefer: count=`, or change the relevance order. The same -`explain` and `sync` restrictions as `Table.search()` apply; violations return 400. +`explain` and `sync` restrictions as `Table.searchPage()` apply; violations return 400. ```http GET /Product/?catalogSearch=matches=trail%20shoes&select(id,title,$score)&limit(25)&continuation() @@ -168,18 +172,22 @@ storage: maxPageSize: 1000 requestBudgetMilliseconds: 5000 maxRawHitsPerRequest: 100000 + maxRawHitsPerOrdinaryRequest: 10000 ``` -| Setting | Default | Scope | -| --------------------------- | ------: | ----------------------------------------------------------- | -| `cursorTtlMilliseconds` | 60,000 | Absolute search lifetime; paging does not extend it | -| `maxCursors` | 4 | Retained searches across all workers in one process | -| `maxRetainedQueryBytes` | 8 MiB | Retained native query text and identity bytes | -| `maxPageSize` | 1,000 | Returned records per page | -| `requestBudgetMilliseconds` | 5,000 | Freshness wait and page processing budget | -| `maxRawHitsPerRequest` | 100,000 | Fetched native hits per request, including replayed batches | -| `maxPendingRequests` | 32 | Pending requests per requesting worker | -| `maxPendingResponseBytes` | 16 MiB | Reserved response bytes per requesting worker | +| Setting | Default | Scope | +| ------------------------------ | ------: | ----------------------------------------------------------- | +| `cursorTtlMilliseconds` | 60,000 | Absolute search lifetime; paging does not extend it | +| `maxCursors` | 4 | Retained searches across all workers in one process | +| `maxRetainedQueryBytes` | 8 MiB | Retained native query text and identity bytes | +| `maxPageSize` | 1,000 | Returned records per page | +| `requestBudgetMilliseconds` | 5,000 | Freshness wait and page processing budget | +| `maxRawHitsPerRequest` | 100,000 | Fetched native hits per request, including replayed batches | +| `maxRawHitsPerOrdinaryRequest` | 10,000 | Additional hit ceiling for searches without continuation | +| `maxPendingRequests` | 32 | Pending requests per requesting worker | +| `maxPendingResponseBytes` | 16 MiB | Reserved response bytes per requesting worker | + +Ordinary searches use the lower of `maxRawHitsPerRequest` and `maxRawHitsPerOrdinaryRequest`, and share the `requestBudgetMilliseconds` execution budget. Increase the ordinary ceiling explicitly if a selective query needs more work. Hitting a ceiling returns an error. Search work and cleanup use separate windows based on `requestBudgetMilliseconds`. With the 5,000 ms default, a search that uses its full work budget and then closes can take about 10 seconds. @@ -197,7 +205,7 @@ The usual individual configuration names also work, such as `storage_fullText_maxRawHitsPerRequest` with `set_configuration`. Follow the [configuration restart procedure](../configuration/operations.md#set-configuration) after applying changes with that operation. Native process-wide admission -settings live under `storage.fullText.runtime` and require a restart when changed. Their defaults +settings live under `storage.fullText.runtime` and require a full process restart when changed. Restarting only worker threads does not apply new native budgets; replacement workers keep the established process budgets. Their defaults are 128 resident indexes, 128 indexing threads, 256 search threads, 4 GiB configured writer memory, 8 GiB configured queue capacity, and 8 concurrent expensive searches. All limits apply together. diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 485a89cb4..24aa1a8f3 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1037,7 +1037,7 @@ Array of condition objects to filter records. Each condition: Full-text conditions name a query-only `FullText` declaration and use comparators `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix`. See -[Full-Text Paging](./full-text-paging.md) for snapshot continuation and its query restrictions. +`Table.searchPage()` and [Full-Text Paging](./full-text-paging.md) for snapshot continuation and its query restrictions. Example with nested conditions: From 88964e2b625c4e781d8cde6855b7dfd3999d0fb2 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 17:06:58 -0600 Subject: [PATCH 35/78] Make paging examples enforce caller permissions and complete the API reference --- reference/configuration/options.md | 5 +---- reference/resources/full-text-paging.md | 13 +++++++++---- reference/resources/resource-api.md | 14 ++++++++++++-- reference/rest/querying.md | 22 ++++++++++++---------- 4 files changed, 34 insertions(+), 20 deletions(-) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index cca263158..8e2a02528 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -302,10 +302,6 @@ replication: Database storage configuration. See [Storage Tuning](../database/storage-tuning.md) for guidance on tuning these options for production workloads, [Database Overview](../database/overview.md) for general database concepts, and [Compaction](../database/compaction.md) for reclaiming space inside existing files. -`storage.fullText` (v5.4.0) controls retained full-text searches and native search capacity. See -[Full-Text Paging](../resources/full-text-paging.md#configure-resource-limits) for the settings, -defaults and restart requirements. - ```yaml storage: path: ~/hdb/database @@ -315,6 +311,7 @@ storage: engine: rocksdb ``` +- `fullText` - Retained full-text searches and native search capacity. See [Full-Text Paging](../resources/full-text-paging.md#configure-resource-limits) for settings, defaults and restart requirements. - `engine` — The database storage engine to use for new databases. Currently supported engines are `rocksdb` and `lmdb`. The default is `rocksdb`. Existing databases will use the engine that was used when they were created. - `writeAsync` — Disable disk sync for higher throughput (**disables durability guarantees**); _Default_: `false` - `caching` — In-memory record caching; _Default_: `true` diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index 664ecc51f..c7f113808 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -36,16 +36,20 @@ Set `continuation: true` and a positive integer `limit` to start. The limit must `searchPage()` returns a promise for a bounded array. Omit `continuation` or set it to `true` to start. `Table.search()` keeps its async iterable return value and rejects continuation options. +Inside an authenticated Resource instance method: + ```javascript +const context = this.getContext(); const query = { conditions: [{ attribute: 'catalogSearch', comparator: 'matches', value: 'trail shoes' }], select: ['id', 'title', '$score'], limit: 25, + checkPermission: true, }; let continuation = true; do { - const page = await tables.Product.searchPage({ ...query, continuation }); + const page = await tables.Product.searchPage({ ...query, continuation }, context); for (const record of page) console.log(record); continuation = page.continuation; } while (continuation); @@ -62,7 +66,7 @@ The returned array has three additional properties: These properties are not included when the array is serialized as JSON. Keep the same conditions and selected fields, and use the same authenticated caller when resuming. The page size may change. Use the default relevance order, or explicitly sort by descending `$score` alone. A nonzero `offset`, another sort, or a truthy `count`, `explain`, or `sync` option -returns 400. +returns 400. In REST, the accepted explicit score order is `sort(-$score)` alone. In a static custom Resource method, pass the method's `context` argument as the second argument to `tables.Product.searchPage()`. In an instance method, use `this.getContext()` instead. See @@ -91,7 +95,7 @@ GET /Product/?catalogSearch=matches=trail%20shoes&select(id,title,$score)&limit( ``` The body stays a JSON array. Read the next token from `Harper-Continuation`, URL-encode it, and -send it in `continuation(token)` with the same query: +send it in `continuation(token)` with the same conditions and selected fields. The page size may change: ```javascript const query = '/Product/?catalogSearch=matches=trail%20shoes&select(id,title,$score)&limit(25)'; @@ -106,7 +110,7 @@ do { } while (token); ``` -`Harper-Continuation-Complete` reports completion. `Harper-Continuation-Expires` contains the +`Harper-Continuation-Complete` is present on every successful continuation page: the string `"false"` while more results remain, and `"true"` after completion or explicit close. `Harper-Continuation-Expires` contains the absolute expiry time in milliseconds since the Unix epoch. Completed pages omit the token and expiry. Harper adds these names to `Access-Control-Expose-Headers` so browser clients can read them. Continuation responses override the usual read-cache policy with `Cache-Control: no-store`. JSON `QUERY` requests return the @@ -168,6 +172,7 @@ storage: cursorTtlMilliseconds: 60000 maxCursors: 4 maxRetainedQueryBytes: 8388608 + maxPendingRequests: 32 maxPendingResponseBytes: 16777216 maxPageSize: 1000 requestBudgetMilliseconds: 5000 diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 24aa1a8f3..543060f58 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -91,7 +91,17 @@ The `get()` method returns a `RecordObject` — a frozen plain object with all r ### `search(query: RequestTarget | Query, context?): ExtendedIterable` -Performs a query on the resource or table. This is called by `get()` on collection requests and can be overridden to define custom query behavior. The default implementation on tables queries by the `conditions`, `limit`, `offset`, `select`, and `sort` properties parsed from the URL or query object. See [Query Object](#query-object) below for available query options. See the [ExtendedIterable](#extendediterable) below for how to interact with the query results. +Performs a query on the resource or table. This is called by `get()` on collection requests and can be overridden to define custom query behavior. A table overriding `search()` must also implement `searchPage()` with the same filtering policy to enable continuation, including through REST. The default implementation on tables queries by the `conditions`, `limit`, `offset`, `select`, and `sort` properties parsed from the URL or query object. See [Query Object](#query-object) below for available query options. See the [ExtendedIterable](#extendediterable) below for how to interact with the query results. + +--- + +### `searchPage(query: RequestTarget | Query, context?): Promise` + + + +Tables use `searchPage()` for full-text and autocomplete continuation. Supply a positive `limit` and omit `continuation` or set it to `true` to start; supply the returned token to resume. The result is an array with `continuation`, `complete`, and `expiresAt` properties. `search()` stays iterable and rejects continuation options. + +When serving an authenticated caller, pass the caller's context and set `checkPermission: true` in the query. A custom `search()` override must explicitly implement `searchPage()` with the same filtering policy; otherwise paging returns 400. See [Full-Text Paging](./full-text-paging.md) for examples, supported queries, closing a search, and token expiry. --- @@ -1036,7 +1046,7 @@ Array of condition objects to filter records. Each condition: | `operator` | `and` (default) or `or` for the nested `conditions` | Full-text conditions name a query-only `FullText` declaration and use comparators `matches`, -`matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix`. See +`matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix`. Each also has a `not_`-prefixed form; a negated full-text condition requires a positive full-text condition in the query. See `Table.searchPage()` and [Full-Text Paging](./full-text-paging.md) for snapshot continuation and its query restrictions. Example with nested conditions: diff --git a/reference/rest/querying.md b/reference/rest/querying.md index d2cd8a308..70850598c 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -136,16 +136,6 @@ GET /Product/?price=lt=100|[rating=5&[tag=fast|tag=scalable|tag=efficient]&inSto Harper supports special query functions using call syntax, included in the query string separated by `&`. -### `continuation()` and `closeContinuation()` - - - -Full-text searches can retain their original index snapshot between requests. Start with -`continuation()`, resume with `continuation(token)`, and close early with `closeContinuation()` -alongside the token. Use `limit(n)` without a nonzero offset or a count request. See -[Full-Text Paging](../resources/full-text-paging.md#page-through-rest) for the token headers, -query restrictions and recovery steps. - ### `select(properties)` Specify which properties to include in the response. @@ -183,10 +173,22 @@ GET /Product/?rating=gt=3&sort(+name) GET /Product/?sort(+rating,-price) ``` +### `continuation()` and `closeContinuation()` + + + +Full-text searches can retain their original index snapshot between requests. Start with +`continuation()`, resume with `continuation(token)`, and close early with `closeContinuation()` +alongside the token. Use the single-argument `limit(end)` form without a starting offset or a count request. See +[Full-Text Paging](../resources/full-text-paging.md#page-through-rest) for the token headers, +query restrictions and recovery steps. + ## Pagination and Total Count +For full-text paging across requests, use [continuation](#continuation-and-closecontinuation) instead of offset paging or a count request. + Use `limit(start,end)` to page through a collection, and opt in to a total match count with the `Prefer` request header (`Prefer: count=exact`) so a client can render pagination (for example "1-25 of 1,234") without a second request. Counting is opt-in: without the header, no count is computed and no count headers are returned. It applies only to `GET`/`HEAD` requests and requires a bounded page — the `limit()` must be a non-negative integer no larger than **10,000**, and the requested window (offset + limit) no larger than **1,000,000**. A request with no `limit()`, one outside those bounds, or a non-numeric one is served normally with no count headers, since counting an unbounded page would defeat the point of paging. From ffb08f70feaa8c6100c00b0d3df5e861b6c8513c Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 17:23:58 -0600 Subject: [PATCH 36/78] Clarify conservative paging reservations and version availability --- reference/resources/full-text-paging.md | 4 +++- reference/rest/querying.md | 4 ++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index c7f113808..4b8a4a0b4 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -32,7 +32,7 @@ application configuration; see [REST setup](../rest/overview.md). ## Page through `Table.searchPage()` -Set `continuation: true` and a positive integer `limit` to start. The limit must not exceed `maxPageSize`. +Use a positive integer `limit` no greater than `maxPageSize`. `searchPage()` returns a promise for a bounded array. Omit `continuation` or set it to `true` to start. `Table.search()` keeps its async iterable return value and rejects continuation options. @@ -194,6 +194,8 @@ storage: Ordinary searches use the lower of `maxRawHitsPerRequest` and `maxRawHitsPerOrdinaryRequest`, and share the `requestBudgetMilliseconds` execution budget. Increase the ordinary ceiling explicitly if a selective query needs more work. Hitting a ceiling returns an error. +A requesting worker reserves a conservative bound for a complete native batch before dispatch, including two-byte string encoding and message overhead. The reservation may exceed the returned payload. `maxPendingResponseBytes` can reject concurrent reads even when the cursor-count and pending-request limits have room. + Search work and cleanup use separate windows based on `requestBudgetMilliseconds`. With the 5,000 ms default, a search that uses its full work budget and then closes can take about 10 seconds. Waiting for other page requests already using the snapshot can add time. Native work checks its diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 70850598c..2a9c75797 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -187,10 +187,10 @@ query restrictions and recovery steps. -For full-text paging across requests, use [continuation](#continuation-and-closecontinuation) instead of offset paging or a count request. - Use `limit(start,end)` to page through a collection, and opt in to a total match count with the `Prefer` request header (`Prefer: count=exact`) so a client can render pagination (for example "1-25 of 1,234") without a second request. +For full-text paging across requests (), use [continuation](#continuation-and-closecontinuation) instead of offset paging or a count request. + Counting is opt-in: without the header, no count is computed and no count headers are returned. It applies only to `GET`/`HEAD` requests and requires a bounded page — the `limit()` must be a non-negative integer no larger than **10,000**, and the requested window (offset + limit) no larger than **1,000,000**. A request with no `limit()`, one outside those bounds, or a non-numeric one is served normally with no count headers, since counting an unbounded page would defeat the point of paging. ### Requesting a count From fa1fb2a3df2e0c21f9ba0566edcea17cd3d6cd63 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 17:53:28 -0600 Subject: [PATCH 37/78] Document paging permission boundaries and upgrade work limits --- reference/configuration/options.md | 2 +- reference/resources/full-text-paging.md | 22 ++++++++++++++++++---- reference/resources/query-optimization.md | 6 ++++++ reference/resources/resource-api.md | 15 ++++++++++++++- reference/rest/querying.md | 4 +++- release-notes/v5-lincoln/5.4.md | 8 ++++++++ 6 files changed, 50 insertions(+), 7 deletions(-) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 8e2a02528..14e668cec 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -311,8 +311,8 @@ storage: engine: rocksdb ``` -- `fullText` - Retained full-text searches and native search capacity. See [Full-Text Paging](../resources/full-text-paging.md#configure-resource-limits) for settings, defaults and restart requirements. - `engine` — The database storage engine to use for new databases. Currently supported engines are `rocksdb` and `lmdb`. The default is `rocksdb`. Existing databases will use the engine that was used when they were created. +- `fullText` - Full-text capacity controls (Added in: v5.4.0). See [Full-Text Paging](../resources/full-text-paging.md#configure-resource-limits) for settings, defaults and restart requirements. Native full-text search itself is available from v5.3.0. - `writeAsync` — Disable disk sync for higher throughput (**disables durability guarantees**); _Default_: `false` - `caching` — In-memory record caching; _Default_: `true` - `compression` — LZ4 record compression; _Default_: `true` (enabled by default since v4.3.0). Sub-options: `dictionary`, `threshold` diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index 4b8a4a0b4..1f2dadd3b 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -71,8 +71,8 @@ returns 400. In REST, the accepted explicit score order is `sort(-$score)` alone In a static custom Resource method, pass the method's `context` argument as the second argument to `tables.Product.searchPage()`. In an instance method, use `this.getContext()` instead. See [Resource context](./resource-api.md#getcontext-context) for both forms. Set `checkPermission: true` -in the query. This requests the caller's -table and searched-source permissions; passing context alone does not request that permission gate. +in the query on every page. Without it, a direct call does not request the caller's +table and searched-source permission checks; passing context alone does not enable them. Changing the caller identity, including switching from an authenticated request to an internal call without that context, returns 410. Permission changes for the same caller are checked on each page. Application row filters still run on each page. @@ -84,11 +84,17 @@ If your table overrides `search()`, also implement `searchPage()` with the same This applies to static and instance methods. Otherwise paging returns 400, including through REST, to avoid bypassing application filters. +Return the original page array from a custom Resource. Copying it with `map()`, `filter()` or +spread drops its paging metadata and a framework-serialized continuation response returns 400. +Use the query's `select` and filters instead. A Resource returning an explicit HTTP `Response` +owns its response headers, serialization and cache policy. + ## Page through REST Use the single-argument `limit(n)` page size and start with `continuation()`. Do not use a nonzero starting offset with `limit(start,end)`, request a count with `Prefer: count=`, or change the relevance order. The same `explain` and `sync` restrictions as `Table.searchPage()` apply; violations return 400. +The equivalent `limit(0,n)` form is accepted; it does not skip records within the retained search. ```http GET /Product/?catalogSearch=matches=trail%20shoes&select(id,title,$score)&limit(25)&continuation() @@ -116,6 +122,10 @@ expiry. Harper adds these names to `Access-Control-Expose-Headers` so browser cl Continuation responses override the usual read-cache policy with `Cache-Control: no-store`. JSON `QUERY` requests return the same headers when their query body enables continuation. +Request URL logging can record continuation tokens. Keep tokens out of logs and analytics. +QUERY accepts the token in its JSON body with the same query fields if it should stay out of the URL. +Tokens still require the same caller and query and expire with their search. + Read and parse the complete response body before saving its next token. A stream can fail after headers are sent. If a resumed page fails this way, retry with the previous token while the snapshot remains valid. If no token was received on the first page, start a new search. The first search may keep its cursor @@ -192,9 +202,9 @@ storage: | `maxPendingRequests` | 32 | Pending requests per requesting worker | | `maxPendingResponseBytes` | 16 MiB | Reserved response bytes per requesting worker | -Ordinary searches use the lower of `maxRawHitsPerRequest` and `maxRawHitsPerOrdinaryRequest`, and share the `requestBudgetMilliseconds` execution budget. Increase the ordinary ceiling explicitly if a selective query needs more work. Hitting a ceiling returns an error. +Ordinary searches use the lower of `maxRawHitsPerRequest` and `maxRawHitsPerOrdinaryRequest`, and share the `requestBudgetMilliseconds` execution budget. A raw-hit ceiling returns 400 instead of a partial page. Increasing the ordinary ceiling permits more native work and record reads per request; test selective queries before raising it. Replayed batches count toward the ceiling, so the number of distinct matches reached can be lower. -A requesting worker reserves a conservative bound for a complete native batch before dispatch, including two-byte string encoding and message overhead. The reservation may exceed the returned payload. `maxPendingResponseBytes` can reject concurrent reads even when the cursor-count and pending-request limits have room. +A requesting worker reserves a conservative bound for a complete native batch before dispatch, including two-byte string encoding and message overhead. The reservation may exceed the returned payload. `maxPendingResponseBytes` can reject concurrent reads with 503 even when the cursor-count and pending-request limits have room. A reservation rejection happens before dispatch and does not advance the search. Reduce concurrency and retry with the same token while it remains valid; this rejection does not include a `Retry-After` header. Search work and cleanup use separate windows based on `requestBudgetMilliseconds`. With the 5,000 ms default, a search that uses its full work budget and then closes can take about 10 seconds. @@ -216,6 +226,10 @@ settings live under `storage.fullText.runtime` and require a full process restar are 128 resident indexes, 128 indexing threads, 256 search threads, 4 GiB configured writer memory, 8 GiB configured queue capacity, and 8 concurrent expensive searches. All limits apply together. +Application-created workers inherit the established native settings too. Create them after +Harper initializes its full-text settings; workers created before that point return 503 when +they try to load full-text search. This prevents a worker from applying different process-wide limits. + The runtime keys use integer counts or bytes: ```yaml diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index dfe2cf9d1..bdc92d8b0 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -79,6 +79,12 @@ When a secondary index is still being built — for example, just after a new `@ This is a transient condition — the index finishes building in the background, after which the same query succeeds. Clients should retry (with backoff) rather than treating it as an empty result or a permanent failure. On the operations API the error body includes `code` and `retryable`, so callers can branch on `error.code === 'INDEX_REBUILDING'` or `error.retryable`. +### Full-Text Search Work Limits + + + +Full-text queries have configurable work limits, including ordinary searches without continuation. A selective filter can reach a limit before filling the requested page. See [Full-Text Paging](./full-text-paging.md#configure-resource-limits) for the limits and the cost of increasing them. + ## Relationships and Joins Harper supports relationship-based queries that join data across tables. See [Schema documentation](../database/schema.md) for how to define relationships. diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 543060f58..81df0068b 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1031,7 +1031,19 @@ body, so avoid putting sensitive detail in thrown error messages. ## Query Object -The `Query` object is accepted by `search()` and the static `get()` method. +The `Query` object is accepted by `search()`, `searchPage()`, and the static `get()` method. + +### `continuation` + +For `searchPage()`, omit this field or use `true` to start a search; use the returned token to resume. Keep the query and caller context consistent. See [Full-Text Paging](./full-text-paging.md#page-through-tablesearchpage). + +### `closeContinuation` + +Use `true` with an existing continuation token to close the search early. See [Close or restart a search](./full-text-paging.md#close-or-restart-a-search). + +### `checkPermission` + +Set `true` on every page when a direct table call acts for a caller, and pass that caller's context. Without it, the call does not request the caller's table and searched-source permission checks; context alone does not enable them. Automatic REST applies its permission checks. See [Full-Text Paging](./full-text-paging.md#page-through-tablesearchpage). ### `conditions` @@ -1099,6 +1111,7 @@ Special properties: - `$id` — Returns the primary key regardless of its name - `$updatedtime` — Returns the last-updated timestamp - `$distance` — When the query ranks or filters by a vector index, returns the computed distance from the target vector. See [Vector Indexing](../database/schema.md#vector-indexing). +- `$score` - When a query uses a full-text index, returns its relevance score. See [Full-Text Paging](./full-text-paging.md#page-through-tablesearchpage). #### Selecting related records diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 2a9c75797..9d252a239 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -179,7 +179,7 @@ GET /Product/?sort(+rating,-price) Full-text searches can retain their original index snapshot between requests. Start with `continuation()`, resume with `continuation(token)`, and close early with `closeContinuation()` -alongside the token. Use the single-argument `limit(end)` form without a starting offset or a count request. See +alongside the token. Use `limit(n)` to set the page size. See [Full-Text Paging](../resources/full-text-paging.md#page-through-rest) for the token headers, query restrictions and recovery steps. @@ -193,6 +193,8 @@ For full-text paging across requests (), use [c Counting is opt-in: without the header, no count is computed and no count headers are returned. It applies only to `GET`/`HEAD` requests and requires a bounded page — the `limit()` must be a non-negative integer no larger than **10,000**, and the requested window (offset + limit) no larger than **1,000,000**. A request with no `limit()`, one outside those bounds, or a non-numeric one is served normally with no count headers, since counting an unbounded page would defeat the point of paging. +Full-text queries also have [search work limits](../resources/full-text-paging.md#configure-resource-limits). A count request does not remove those limits; a full-text query can fail before returning a page or count. + ### Requesting a count Send a `Prefer` header on a `GET` (or `HEAD`) request to a collection: diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index ad0d66d61..87dae03c6 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -8,6 +8,14 @@ title: '5.4' All patch release notes for 5.4.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.4&expanded=true). +## Upgrade Notes + +### Full-Text Search Budgets and Snapshot Paging + +Full-text and autocomplete searches can retain their original index matches between requests through `Table.searchPage()` or REST continuation tokens. Ordinary `Table.search()` remains iterable. See [Full-Text Paging](/reference/v5/resources/full-text-paging). + +The new `storage.fullText` work limits also apply to searches without continuation: the ordinary fetched-hit ceiling defaults to 10,000 and the shared request work budget defaults to 5,000 ms. Replayed batches count toward the hit ceiling. Hitting the raw-hit ceiling returns 400 instead of a successful partial page. Test selective full-text queries before upgrading; increasing the ceiling permits more work and record reads per request. See [resource limits](/reference/v5/resources/full-text-paging#configure-resource-limits) before changing the defaults. + ## Replication ### Dedicated Replication Threads From 238315ff759e62c2f33b7172ca02f899c3cd3569 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 18:03:28 -0600 Subject: [PATCH 38/78] Clarify explicit paging dispatch and permanent worker initialization errors --- reference/resources/full-text-paging.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index 1f2dadd3b..a96b7aef0 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -84,7 +84,8 @@ If your table overrides `search()`, also implement `searchPage()` with the same This applies to static and instance methods. Otherwise paging returns 400, including through REST, to avoid bypassing application filters. -Return the original page array from a custom Resource. Copying it with `map()`, `filter()` or +A custom Resource supporting the framework's QUERY paging route must implement `searchPage()`. +Return the original page array. Copying it with `map()`, `filter()` or spread drops its paging metadata and a framework-serialized continuation response returns 400. Use the query's `select` and filters instead. A Resource returning an explicit HTTP `Response` owns its response headers, serialization and cache policy. @@ -227,8 +228,9 @@ are 128 resident indexes, 128 indexing threads, 256 search threads, 4 GiB config 8 GiB configured queue capacity, and 8 concurrent expensive searches. All limits apply together. Application-created workers inherit the established native settings too. Create them after -Harper initializes its full-text settings; workers created before that point return 503 when -they try to load full-text search. This prevents a worker from applying different process-wide limits. +Harper initializes its full-text settings. A worker created before that point cannot initialize +full-text search; recreate it after Harper initializes rather than retrying inside the same worker. +This prevents a worker from applying different process-wide limits. The runtime keys use integer counts or bytes: From 41b831ef25b8b70d52272debbbe089e3e0051d95 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 18:19:36 -0600 Subject: [PATCH 39/78] Clarify paging permissions, failure recovery and application startup --- reference/resources/full-text-paging.md | 21 +++++++++++++-------- reference/resources/query-optimization.md | 2 +- reference/resources/resource-api.md | 4 +++- release-notes/v5-lincoln/5.4.md | 10 +++++++--- 4 files changed, 24 insertions(+), 13 deletions(-) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index a96b7aef0..ed6cef951 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -64,7 +64,9 @@ The returned array has three additional properties: | `expiresAt` | Absolute expiry time in milliseconds since the Unix epoch | These properties are not included when the array is serialized as JSON. Keep the same conditions -and selected fields, and use the same authenticated caller when resuming. The page size may change. Use the default relevance order, or explicitly sort by descending +and selected fields, and use the same authenticated caller when resuming. You may raise or lower +the requested `limit` on later pages within `maxPageSize`. Each page contains at most its requested +limit; a shorter page alone does not mean the search is complete. Use the default relevance order, or explicitly sort by descending `$score` alone. A nonzero `offset`, another sort, or a truthy `count`, `explain`, or `sync` option returns 400. In REST, the accepted explicit score order is `sort(-$score)` alone. @@ -102,7 +104,7 @@ GET /Product/?catalogSearch=matches=trail%20shoes&select(id,title,$score)&limit( ``` The body stays a JSON array. Read the next token from `Harper-Continuation`, URL-encode it, and -send it in `continuation(token)` with the same conditions and selected fields. The page size may change: +send it in `continuation(token)` with the same conditions and selected fields. The requested page size may increase or decrease within `maxPageSize`: ```javascript const query = '/Product/?catalogSearch=matches=trail%20shoes&select(id,title,$score)&limit(25)'; @@ -216,8 +218,6 @@ Ordinary searches can use secondary indexes to skip record reads and enlarge bat matches are filtered out. Continued searches do not yet use those optimizations; they fetch up to 256 native hits per batch and check current records. A selective filter can therefore add many record reads and, when the snapshot belongs to another worker, many worker requests. -Qualification and these optimizations are tracked in -[Qualify native full-text filtering and catalog-scale operation for v5.4](https://github.com/HarperFast/harper/issues/2512). The usual individual configuration names also work, such as `storage_fullText_maxRawHitsPerRequest` with `set_configuration`. Follow the @@ -227,10 +227,11 @@ settings live under `storage.fullText.runtime` and require a full process restar are 128 resident indexes, 128 indexing threads, 256 search threads, 4 GiB configured writer memory, 8 GiB configured queue capacity, and 8 concurrent expensive searches. All limits apply together. -Application-created workers inherit the established native settings too. Create them after -Harper initializes its full-text settings. A worker created before that point cannot initialize -full-text search; recreate it after Harper initializes rather than retrying inside the same worker. -This prevents a worker from applying different process-wide limits. +Harper captures the native settings before loading application components, so workers created +by a running application inherit them. In a standalone process, initialize Harper before creating +workers that use full-text search. A worker created before capture cannot initialize full-text +search; recreate it after Harper initializes rather than retrying inside the same worker. +Its initialization failure is non-retryable, not a temporary capacity refusal. The runtime keys use integer counts or bytes: @@ -251,6 +252,10 @@ receives `503` until a search finishes, closes, or expires. Ordinary searches re Configure `maxCursors` for the number of unfinished searches the application needs to retain. There is no per-caller reservation or eviction policy. +Cursor-count and pending-response capacity refusals return 503 without `Retry-After`. Close unused +searches or reduce concurrency, then retry with backoff while the token remains valid. An execution-budget +503 calls for a smaller page or less expensive query; changing the query requires a new search. + Cursor slots, pending-request counts, and response-byte allowances apply separately. Raising `maxCursors` does not raise `maxPendingRequests` or `maxPendingResponseBytes`. Across workers, Harper reserves response space using the library's largest supported IDs and versions, rather than diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index bdc92d8b0..5e72803dd 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -83,7 +83,7 @@ This is a transient condition — the index finishes building in the background, -Full-text queries have configurable work limits, including ordinary searches without continuation. A selective filter can reach a limit before filling the requested page. See [Full-Text Paging](./full-text-paging.md#configure-resource-limits) for the limits and the cost of increasing them. +Full-text queries have configurable work limits, including ordinary searches without continuation. A selective filter can exhaust a work limit and fail the request instead of returning a partial page. See [Full-Text Paging](./full-text-paging.md#configure-resource-limits) for the limits, errors and cost of increasing them. ## Relationships and Joins diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 81df0068b..0f63da00b 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -99,7 +99,7 @@ Performs a query on the resource or table. This is called by `get()` on collecti -Tables use `searchPage()` for full-text and autocomplete continuation. Supply a positive `limit` and omit `continuation` or set it to `true` to start; supply the returned token to resume. The result is an array with `continuation`, `complete`, and `expiresAt` properties. `search()` stays iterable and rejects continuation options. +Tables use `searchPage()` for full-text and autocomplete continuation. Supply a positive `limit` and omit `continuation` or set it to `true` to start; supply the returned token to resume. `FullTextSearchPage` is an array of records with `continuation`, `complete`, and `expiresAt` properties; see the [result properties](./full-text-paging.md#page-through-tablesearchpage). `search()` stays iterable and rejects continuation options. When serving an authenticated caller, pass the caller's context and set `checkPermission: true` in the query. A custom `search()` override must explicitly implement `searchPage()` with the same filtering policy; otherwise paging returns 400. See [Full-Text Paging](./full-text-paging.md) for examples, supported queries, closing a search, and token expiry. @@ -1045,6 +1045,8 @@ Use `true` with an existing continuation token to close the search early. See [C Set `true` on every page when a direct table call acts for a caller, and pass that caller's context. Without it, the call does not request the caller's table and searched-source permission checks; context alone does not enable them. Automatic REST applies its permission checks. See [Full-Text Paging](./full-text-paging.md#page-through-tablesearchpage). +Set this field in server code; never copy permission-control fields from client data. + ### `conditions` Array of condition objects to filter records. Each condition: diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 87dae03c6..f90565f3f 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -10,11 +10,15 @@ All patch release notes for 5.4.x are available on the [releases page](https://g ## Upgrade Notes -### Full-Text Search Budgets and Snapshot Paging +### Full-Text Search Budgets -Full-text and autocomplete searches can retain their original index matches between requests through `Table.searchPage()` or REST continuation tokens. Ordinary `Table.search()` remains iterable. See [Full-Text Paging](/reference/v5/resources/full-text-paging). +The new `storage.fullText` work limits also apply to searches without continuation: the ordinary fetched-hit ceiling defaults to 10,000 and the shared execution budget defaults to 5,000 ms. Replayed batches count toward the hit ceiling. Hitting the raw-hit ceiling returns 400 instead of a successful partial page; exhausting the execution budget returns 503. Test selective full-text queries before upgrading. Narrow an expensive query or reduce its page size before increasing a work limit; higher limits permit more work and record reads per request. See [resource limits](/reference/v5/resources/full-text-paging#configure-resource-limits) before changing the defaults. + +## Full-Text Search -The new `storage.fullText` work limits also apply to searches without continuation: the ordinary fetched-hit ceiling defaults to 10,000 and the shared request work budget defaults to 5,000 ms. Replayed batches count toward the hit ceiling. Hitting the raw-hit ceiling returns 400 instead of a successful partial page. Test selective full-text queries before upgrading; increasing the ceiling permits more work and record reads per request. See [resource limits](/reference/v5/resources/full-text-paging#configure-resource-limits) before changing the defaults. +### Snapshot Paging + +Full-text and autocomplete searches can retain their original index matches between requests through `Table.searchPage()` or REST continuation tokens. Ordinary `Table.search()` remains iterable. See [Full-Text Paging](/reference/v5/resources/full-text-paging). ## Replication From fb873fc9776dd5da959ea6fbfca9cd19c091cfd5 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 18:33:02 -0600 Subject: [PATCH 40/78] Use plain language for copied paging responses --- reference/resources/full-text-paging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index ed6cef951..8b62eae54 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -88,7 +88,7 @@ to avoid bypassing application filters. A custom Resource supporting the framework's QUERY paging route must implement `searchPage()`. Return the original page array. Copying it with `map()`, `filter()` or -spread drops its paging metadata and a framework-serialized continuation response returns 400. +spread drops its paging metadata. Returning that copy through the normal Resource API returns 400. Use the query's `select` and filters instead. A Resource returning an explicit HTTP `Response` owns its response headers, serialization and cache policy. From f9ebdcd0fab439f03293ad8a84a0982002abd54c Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 18:52:15 -0600 Subject: [PATCH 41/78] Clarify paging response envelopes and managed worker routing --- reference/resources/full-text-paging.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index 8b62eae54..bdf075ef9 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -89,8 +89,9 @@ to avoid bypassing application filters. A custom Resource supporting the framework's QUERY paging route must implement `searchPage()`. Return the original page array. Copying it with `map()`, `filter()` or spread drops its paging metadata. Returning that copy through the normal Resource API returns 400. -Use the query's `select` and filters instead. A Resource returning an explicit HTTP `Response` -owns its response headers, serialization and cache policy. +Use the query's `select` and filters instead. To add custom headers, return +`{ headers, data: page }` with the original page array. A Resource returning an explicit HTTP +`Response` owns its response headers, serialization and cache policy. ## Page through REST @@ -232,6 +233,8 @@ by a running application inherit them. In a standalone process, initialize Harpe workers that use full-text search. A worker created before capture cannot initialize full-text search; recreate it after Harper initializes rather than retrying inside the same worker. Its initialization failure is non-retryable, not a temporary capacity refusal. +Retained paging still requires Harper's main thread or a managed worker. Inheriting native settings +does not add paging-message routing to application-created workers. The runtime keys use integer counts or bytes: From 1c4d88f9d0aae84f2bbe977804b8e431195c8ec9 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 19:03:27 -0600 Subject: [PATCH 42/78] Describe measured token sizes plainly --- reference/resources/full-text-paging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index bdf075ef9..9faffaec3 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -139,7 +139,7 @@ GET and QUERY can return continuation pages. OPTIONS returns request metadata wi search. HEAD is an exception to the usual GET-without-a-body behavior: it returns 400 before opening or advancing a continuation. Other methods also reject continuation parameters with 400. -Long record IDs make continuation tokens larger. Qualification with the released library produced +Long record IDs make continuation tokens larger. Local measurements produced tokens of about 600 bytes for short IDs and nearly 8 KB near the supported ID limit. Review Harper's `http.maxHeaderSize` setting and any proxy response-header and request-line limits. Leave room for the token, query, authentication, and other headers; a token that fits Harper may exceed a proxy limit. From 06d3089df0ef859591c0c3b08ee303c3ac5371a2 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 19:04:59 -0600 Subject: [PATCH 43/78] Clarify continuation entry points and native runtime recovery --- reference/configuration/operations.md | 4 ++++ reference/resources/full-text-paging.md | 6 +++++- reference/rest/querying.md | 3 +++ 3 files changed, 12 insertions(+), 1 deletion(-) diff --git a/reference/configuration/operations.md b/reference/configuration/operations.md index 75d9ed695..dfb186bd1 100644 --- a/reference/configuration/operations.md +++ b/reference/configuration/operations.md @@ -18,6 +18,10 @@ For the full list of configurable options, see [Configuration Options](./options Modifies one or more Harper configuration parameters. **Requires a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`). A change to `replication.threads` takes effect only on a full `restart`. +Changes to [`storage.fullText.runtime`](../resources/full-text-paging.md#configure-resource-limits) +(v5.4.0) require a full `restart`. `restart_service` replaces workers but keeps the established +process-wide native full-text budgets. + `operation` _(required)_ — must be `set_configuration` `replicated` _(optional)_ — set to `true` to apply the same configuration change to all cluster nodes in a single call. The origin node applies the change first; if its local write fails, nothing is sent to peers. Requires replication to be configured. diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index 9faffaec3..15e16f743 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -12,6 +12,7 @@ Full-text continuation lets a client page through the original search matches wh updates arrive. It is available through `Table.searchPage()` and REST. Ordinary offset-based paging continues to run each request against its own index snapshot. The same continuation options work with autocomplete searches using `matches_prefix` and `matches_fuzzy_prefix`. +Operations API searches do not support continuation options; use `Table.searchPage()` or REST. ## Declare a full-text index @@ -232,9 +233,12 @@ Harper captures the native settings before loading application components, so wo by a running application inherit them. In a standalone process, initialize Harper before creating workers that use full-text search. A worker created before capture cannot initialize full-text search; recreate it after Harper initializes rather than retrying inside the same worker. -Its initialization failure is non-retryable, not a temporary capacity refusal. +This throws `FullTextWorkerInitializationError` with status 500 and `retryable: false`. Retained paging still requires Harper's main thread or a managed worker. Inheriting native settings does not add paging-message routing to application-created workers. +Attempting retained paging from an unmanaged worker returns 503 with +`Full-text continuation requires a Harper-managed worker`; waiting and retrying on that worker +does not add routing. Run the page request through Harper's main thread or a managed worker. The runtime keys use integer counts or bytes: diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 9d252a239..0ae96a24c 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -55,6 +55,9 @@ Harper uses [FIQL](https://datatracker.ietf.org/doc/html/draft-nottingham-atompu | `=`, `===` | Strict equality (no type conversion) | | `!==` | Strict inequality (no type conversion) | +Declared full-text fields use `=matches=`, `=matches_prefix=` or `=matches_fuzzy_prefix=`. +See [Full-Text Paging](../resources/full-text-paging.md#page-through-rest) for examples and continuation. + **Examples**: ```http From aec291ea12e1db1c418037bdc0c504bd4667675a Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 21:46:20 -0600 Subject: [PATCH 44/78] Document full-text limit and offset paging --- reference/configuration/operations.md | 4 - reference/configuration/options.md | 3 +- reference/resources/full-text-paging.md | 293 ++++------------------ reference/resources/overview.md | 1 - reference/resources/query-optimization.md | 6 - reference/resources/resource-api.md | 33 +-- reference/rest/headers.md | 6 - reference/rest/overview.md | 4 +- reference/rest/querying.md | 19 +- release-notes/v5-lincoln/5.4.md | 18 +- sidebarsReference.ts | 6 +- 11 files changed, 60 insertions(+), 333 deletions(-) diff --git a/reference/configuration/operations.md b/reference/configuration/operations.md index dfb186bd1..75d9ed695 100644 --- a/reference/configuration/operations.md +++ b/reference/configuration/operations.md @@ -18,10 +18,6 @@ For the full list of configurable options, see [Configuration Options](./options Modifies one or more Harper configuration parameters. **Requires a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`). A change to `replication.threads` takes effect only on a full `restart`. -Changes to [`storage.fullText.runtime`](../resources/full-text-paging.md#configure-resource-limits) -(v5.4.0) require a full `restart`. `restart_service` replaces workers but keeps the established -process-wide native full-text budgets. - `operation` _(required)_ — must be `set_configuration` `replicated` _(optional)_ — set to `true` to apply the same configuration change to all cluster nodes in a single call. The origin node applies the change first; if its local write fails, nothing is sent to peers. Requires replication to be configured. diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 14e668cec..27bd93331 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -312,7 +312,6 @@ storage: ``` - `engine` — The database storage engine to use for new databases. Currently supported engines are `rocksdb` and `lmdb`. The default is `rocksdb`. Existing databases will use the engine that was used when they were created. -- `fullText` - Full-text capacity controls (Added in: v5.4.0). See [Full-Text Paging](../resources/full-text-paging.md#configure-resource-limits) for settings, defaults and restart requirements. Native full-text search itself is available from v5.3.0. - `writeAsync` — Disable disk sync for higher throughput (**disables durability guarantees**); _Default_: `false` - `caching` — In-memory record caching; _Default_: `true` - `compression` — LZ4 record compression; _Default_: `true` (enabled by default since v4.3.0). Sub-options: `dictionary`, `threshold` @@ -579,3 +578,5 @@ my-component: - `.package` — NPM package name, GitHub repo (`user/repo`), or local path - `.port` — Port for the component; _Default_: value of `http.port` + +Full-text search work limits and native runtime admission settings are described in [Full-text paging](../resources/full-text-paging.md#configure-search-work). diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index 15e16f743..53955bad3 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -1,278 +1,77 @@ --- -title: Full-Text Paging +title: Full-text paging --- -# Full-Text Paging +# Full-text paging - + -The full-text index declaration below is a prerequisite. +Full-text and autocomplete searches use Harper's existing paging options. Each request runs a new search; Harper does not retain a search between pages. -Full-text continuation lets a client page through the original search matches while indexing -updates arrive. It is available through `Table.searchPage()` and REST. Ordinary offset-based paging -continues to run each request against its own index snapshot. -The same continuation options work with autocomplete searches using `matches_prefix` and `matches_fuzzy_prefix`. -Operations API searches do not support continuation options; use `Table.searchPage()` or REST. +## Request a page -## Declare a full-text index - -Full-text indexes require RocksDB and an audited table. The query names the `FullText` declaration, -not a stored source field: - -```graphql -type Product @table(audit: true) @export { - id: ID @primaryKey - title: String - catalogSearch: FullText @fullText(fields: [{ name: "title" }]) -} -``` - -`catalogSearch` is a query-only declaration. Select the stored fields and optional `$score` in results. -Full-text matches use descending relevance order. REST examples also require `rest: true` in the -application configuration; see [REST setup](../rest/overview.md). - -## Page through `Table.searchPage()` - -Use a positive integer `limit` no greater than `maxPageSize`. -`searchPage()` returns a promise for a bounded array. Omit `continuation` or set it to `true` to start. -`Table.search()` keeps its async iterable return value and rejects continuation options. - -Inside an authenticated Resource instance method: +Inside an authenticated Resource handler, use `limit` and `offset` with `Table.search()`. `Table.search()` returns an iterable, including when a custom Resource forwards or filters it. ```javascript -const context = this.getContext(); -const query = { - conditions: [{ attribute: 'catalogSearch', comparator: 'matches', value: 'trail shoes' }], - select: ['id', 'title', '$score'], - limit: 25, - checkPermission: true, -}; - -let continuation = true; -do { - const page = await tables.Product.searchPage({ ...query, continuation }, context); - for (const record of page) console.log(record); - continuation = page.continuation; -} while (continuation); +return Product.search( + { + checkPermission: true, + conditions: [{ attribute: 'catalogSearch', comparator: 'matches', value: 'trail shoes' }], + offset: 20, + limit: 20, + }, + this.getContext() +); ``` -The returned array has three additional properties: - -| Property | Meaning | -| -------------- | ------------------------------------------------------------------- | -| `continuation` | Token for the next page; `undefined` when traversal is complete | -| `complete` | Whether the original search has been exhausted or explicitly closed | -| `expiresAt` | Absolute expiry time in milliseconds since the Unix epoch | +The equivalent REST range has an inclusive start and exclusive end. This request skips 20 matching records and returns up to 20 more: -These properties are not included when the array is serialized as JSON. Keep the same conditions -and selected fields, and use the same authenticated caller when resuming. You may raise or lower -the requested `limit` on later pages within `maxPageSize`. Each page contains at most its requested -limit; a shorter page alone does not mean the search is complete. Use the default relevance order, or explicitly sort by descending -`$score` alone. A nonzero `offset`, another sort, or a truthy `count`, `explain`, or `sync` option -returns 400. In REST, the accepted explicit score order is `sort(-$score)` alone. - -In a static custom Resource method, pass the method's `context` argument as the second argument -to `tables.Product.searchPage()`. In an instance method, use `this.getContext()` instead. See -[Resource context](./resource-api.md#getcontext-context) for both forms. Set `checkPermission: true` -in the query on every page. Without it, a direct call does not request the caller's -table and searched-source permission checks; passing context alone does not enable them. -Changing the caller identity, including switching from an authenticated request to an internal -call without that context, returns 410. Permission changes for the same caller are checked on each -page. Application row filters still run on each page. -The token does not store application callbacks such as `rowFilter` or -[`vectorFilter`](../database/schema.md#filtered-vector-search-with-a-function-predicate). Supply them -again on every page, together with the caller's context and permission settings. - -If your table overrides `search()`, also implement `searchPage()` with the same filtering policy. -This applies to static and instance methods. Otherwise paging returns 400, including through REST, -to avoid bypassing application filters. - -A custom Resource supporting the framework's QUERY paging route must implement `searchPage()`. -Return the original page array. Copying it with `map()`, `filter()` or -spread drops its paging metadata. Returning that copy through the normal Resource API returns 400. -Use the query's `select` and filters instead. To add custom headers, return -`{ headers, data: page }` with the original page array. A Resource returning an explicit HTTP -`Response` owns its response headers, serialization and cache policy. - -## Page through REST - -Use the single-argument `limit(n)` page size and start with `continuation()`. Do not use -a nonzero starting offset with `limit(start,end)`, request a count with `Prefer: count=`, or change the relevance order. The same -`explain` and `sync` restrictions as `Table.searchPage()` apply; violations return 400. -The equivalent `limit(0,n)` form is accepted; it does not skip records within the retained search. - -```http -GET /Product/?catalogSearch=matches=trail%20shoes&select(id,title,$score)&limit(25)&continuation() +```text +GET /Product/?catalogSearch=matches=trail%20shoes&limit(20,40) ``` -The body stays a JSON array. Read the next token from `Harper-Continuation`, URL-encode it, and -send it in `continuation(token)` with the same conditions and selected fields. The requested page size may increase or decrease within `maxPageSize`: +Each page runs a new query. Changes to records, permissions, or the index can move results between requests, so separate pages may omit or repeat records. There is no retained search or cross-request snapshot. -```javascript -const query = '/Product/?catalogSearch=matches=trail%20shoes&select(id,title,$score)&limit(25)'; -let token; -do { - const continuation = token ? `continuation(${encodeURIComponent(token)})` : 'continuation()'; - const response = await fetch(`${query}&${continuation}`); - if (!response.ok) throw new Error(await response.text()); - const records = await response.json(); - for (const record of records) console.log(record); - token = response.headers.get('Harper-Continuation'); -} while (token); -``` +Within one request, Harper reads native batches on the same index reader until enough eligible records have been found or the matches are exhausted. Current source versions and record filters are checked before counting records toward the offset and limit. This traversal also lets prefix and fuzzy-prefix searches return more than the initial autocomplete batch. The library's ordinary search window is not a total-result cutoff for a bounded Harper request. -`Harper-Continuation-Complete` is present on every successful continuation page: the string `"false"` while more results remain, and `"true"` after completion or explicit close. `Harper-Continuation-Expires` contains the -absolute expiry time in milliseconds since the Unix epoch. Completed pages omit the token and -expiry. Harper adds these names to `Access-Control-Expose-Headers` so browser clients can read them. -Continuation responses override the usual read-cache policy with `Cache-Control: no-store`. JSON `QUERY` requests return the -same headers when their query body enables continuation. +Use a bounded `limit` for large result sets. An unbounded query that exceeds the native result window returns 400. Offset requests revisit earlier matches; deep offsets and restrictive filters can exhaust the work or time budget. An exhausted search returns its remaining eligible records, including an empty result. A budget failure returns an error instead of a successful incomplete page. -Request URL logging can record continuation tokens. Keep tokens out of logs and analytics. -QUERY accepts the token in its JSON body with the same query fields if it should stay out of the URL. -Tokens still require the same caller and query and expire with their search. +## Configure search work -Read and parse the complete response body before saving its next token. A stream can fail after -headers are sent. If a resumed page fails this way, retry with the previous token while the snapshot remains valid. -If no token was received on the first page, start a new search. The first search may keep its cursor -slot until absolute expiry because it cannot be closed without a token. +Search settings are under `storage.fullText` in `harper-config.yaml`: -GET and QUERY can return continuation pages. OPTIONS returns request metadata without running a -search. HEAD is an exception to the usual GET-without-a-body behavior: it returns 400 before -opening or advancing a continuation. Other methods also reject continuation parameters with 400. +| Setting | Default | Meaning | +| --------------------------- | ------: | ----------------------------------------------------------------------------------------------------------------- | +| `maxRawHitsPerRequest` | 10,000 | Fetched native hits, including the initial batch replay when switching to traversal | +| `requestBudgetMilliseconds` | 5,000 | Execution budget across native batches, source checks and highlight tracing, capped by the native execution limit | -Long record IDs make continuation tokens larger. Local measurements produced -tokens of about 600 bytes for short IDs and nearly 8 KB near the supported ID limit. Review Harper's -`http.maxHeaderSize` setting and any proxy response-header and request-line limits. Leave room for the -token, query, authentication, and other headers; a token that fits Harper may exceed a proxy limit. +The hit limit counts fetched hits, not every record Tantivy visits while scoring. Reaching it before the page fills or the search is exhausted returns 400. Exceeding the execution budget returns 503. Narrow the query, reduce the offset, or increase the relevant configured budget. Freshness waiting uses `waitForIndexMilliseconds` separately. Reader acquisition, query construction and some native operations can take additional time; the execution budget is not a hard response-time guarantee. -## Records and permissions can change +Each setting has the usual configuration parameter name, such as `storage_fullText_maxRawHitsPerRequest`, for `set_configuration` and environment overrides. Query work limits apply to new searches after configuration changes. -The snapshot fixes the index matches and their ranking. It does not retain historical Harper -records or permissions. Each request loads current records and checks current access. Updated, -deleted, expired, or newly unauthorized records can be omitted, and new matches do not enter the -original search. +## Configure native runtime admission -Retrying a token resumes the same index boundary. The returned rows can differ if records or -permissions changed. A page may be empty when the remaining snapshot contains no eligible records; -use the continuation token or completion flag to decide when to stop. +Native process admission settings live under `storage.fullText.runtime`: -## Close or restart a search +| Setting | Default | +| ---------------------- | ------: | +| `maxResidentIndexes` | 128 | +| `maxIndexingThreads` | 128 | +| `maxSearchThreads` | 256 | +| `maxWriterMemoryBytes` | 4 GiB | +| `maxQueuedBytes` | 8 GiB | +| `maxExpensiveSearches` | 8 | -To stop early, repeat the query with its current token and `closeContinuation: true`. In REST, add -`closeContinuation()` alongside `continuation(token)`. Closing returns an empty completed page. +These limits apply together. For example, two indexing threads and 64 MiB of writer memory per index reach the default thread and memory budgets at 64 indexes. Readers share an index's runtime; they do not allocate another writer. Admission limits are not hard bounds on total resident memory or disk usage. -Close and terminal pages wait for admission release before reporting success. If release fails, the -request returns an error instead. The revoked token may then return `410`, and its admission slot -can remain occupied until absolute expiry or worker exit. A successful close releases admission -capacity; it does not promise immediate removal of old index segments from disk. -Closing has a separate cleanup budget. Once release is acknowledged, the completed page can return -after its search-work deadline. Cancellation, transaction, and index-lifecycle checks still apply. - -Completion, close, expiry, index reset, definition changes, worker loss, or process restart can -invalidate a token. Resuming an unavailable snapshot returns `410`; start a new search. Once any -request finishes traversal, older tokens for that search also expire. -Changing any `storage.fullText` setting also invalidates existing tokens, even when only a capacity -setting changes. Start a new search after changing these settings. - -Snapshots belong to one Harper instance. Its workers can handle continuation requests for each -other. Requests reaching a different Harper instance cannot resume that snapshot. - -## Configure resource limits - -Settings live under `storage.fullText` in `harper-config.yaml`: - -```yaml -storage: - fullText: - cursorTtlMilliseconds: 60000 - maxCursors: 4 - maxRetainedQueryBytes: 8388608 - maxPendingRequests: 32 - maxPendingResponseBytes: 16777216 - maxPageSize: 1000 - requestBudgetMilliseconds: 5000 - maxRawHitsPerRequest: 100000 - maxRawHitsPerOrdinaryRequest: 10000 -``` - -| Setting | Default | Scope | -| ------------------------------ | ------: | ----------------------------------------------------------- | -| `cursorTtlMilliseconds` | 60,000 | Absolute search lifetime; paging does not extend it | -| `maxCursors` | 4 | Retained searches across all workers in one process | -| `maxRetainedQueryBytes` | 8 MiB | Retained native query text and identity bytes | -| `maxPageSize` | 1,000 | Returned records per page | -| `requestBudgetMilliseconds` | 5,000 | Freshness wait and page processing budget | -| `maxRawHitsPerRequest` | 100,000 | Fetched native hits per request, including replayed batches | -| `maxRawHitsPerOrdinaryRequest` | 10,000 | Additional hit ceiling for searches without continuation | -| `maxPendingRequests` | 32 | Pending requests per requesting worker | -| `maxPendingResponseBytes` | 16 MiB | Reserved response bytes per requesting worker | - -Ordinary searches use the lower of `maxRawHitsPerRequest` and `maxRawHitsPerOrdinaryRequest`, and share the `requestBudgetMilliseconds` execution budget. A raw-hit ceiling returns 400 instead of a partial page. Increasing the ordinary ceiling permits more native work and record reads per request; test selective queries before raising it. Replayed batches count toward the ceiling, so the number of distinct matches reached can be lower. - -A requesting worker reserves a conservative bound for a complete native batch before dispatch, including two-byte string encoding and message overhead. The reservation may exceed the returned payload. `maxPendingResponseBytes` can reject concurrent reads with 503 even when the cursor-count and pending-request limits have room. A reservation rejection happens before dispatch and does not advance the search. Reduce concurrency and retry with the same token while it remains valid; this rejection does not include a `Retry-After` header. - -Search work and cleanup use separate windows based on `requestBudgetMilliseconds`. With the -5,000 ms default, a search that uses its full work budget and then closes can take about 10 seconds. -Waiting for other page requests already using the snapshot can add time. Native work checks its -deadline while running; these settings are not a hard limit on total response time. - -Ordinary searches can use secondary indexes to skip record reads and enlarge batches when many -matches are filtered out. Continued searches do not yet use those optimizations; they fetch up to -256 native hits per batch and check current records. A selective filter can therefore add many -record reads and, when the snapshot belongs to another worker, many worker requests. - -The usual individual configuration names also work, such as -`storage_fullText_maxRawHitsPerRequest` with `set_configuration`. Follow the -[configuration restart procedure](../configuration/operations.md#set-configuration) after applying -changes with that operation. Native process-wide admission -settings live under `storage.fullText.runtime` and require a full process restart when changed. Restarting only worker threads does not apply new native budgets; replacement workers keep the established process budgets. Their defaults -are 128 resident indexes, 128 indexing threads, 256 search threads, 4 GiB configured writer memory, -8 GiB configured queue capacity, and 8 concurrent expensive searches. All limits apply together. - -Harper captures the native settings before loading application components, so workers created -by a running application inherit them. In a standalone process, initialize Harper before creating -workers that use full-text search. A worker created before capture cannot initialize full-text -search; recreate it after Harper initializes rather than retrying inside the same worker. -This throws `FullTextWorkerInitializationError` with status 500 and `retryable: false`. -Retained paging still requires Harper's main thread or a managed worker. Inheriting native settings -does not add paging-message routing to application-created workers. -Attempting retained paging from an unmanaged worker returns 503 with -`Full-text continuation requires a Harper-managed worker`; waiting and retrying on that worker -does not add routing. Run the page request through Harper's main thread or a managed worker. - -The runtime keys use integer counts or bytes: - -```yaml -storage: - fullText: - runtime: - maxResidentIndexes: 128 - maxIndexingThreads: 128 - maxSearchThreads: 256 - maxWriterMemoryBytes: 4294967296 - maxQueuedBytes: 8589934592 - maxExpensiveSearches: 8 -``` +Harper captures native runtime budgets before loading application components, or on an earlier main-thread native load or managed worker spawn. Application-created workers inherit that configuration. A worker created before capture fails with `FullTextWorkerInitializationError` (500, non-retryable); recreate it after Harper initializes. Replacement workers keep the established budgets after a live configuration change. Restart the Harper process to apply new native runtime budgets; restarting only HTTP workers is insufficient. -The cursor pool is shared by all callers. Four unfinished searches fill the default pool, so a fifth -receives `503` until a search finishes, closes, or expires. Ordinary searches remain available. -Configure `maxCursors` for the number of unfinished searches the application needs to retain. -There is no per-caller reservation or eviction policy. +Harper does not configure the library's native filter metadata or submit native filter or candidate-ID requests. Ordinary conditions continue through Harper's secondary-index gates and authoritative record predicates. Native page, response-byte and highlight limits still apply. Query terms and clauses remain limited by the installed library. -Cursor-count and pending-response capacity refusals return 503 without `Retry-After`. Close unused -searches or reduce concurrency, then retry with backoff while the token remains valid. An execution-budget -503 calls for a smaller page or less expensive query; changing the query requires a new search. +## Upgrade considerations -Cursor slots, pending-request counts, and response-byte allowances apply separately. Raising -`maxCursors` does not raise `maxPendingRequests` or `maxPendingResponseBytes`. Across workers, -Harper reserves response space using the library's largest supported IDs and versions, rather than -the average size of the current records. Many searches may be retained while fewer pages can run -at once. +This change uses the fulltext 0.6.0 native package. A format-incompatible upgrade rebuilds the local index from stored records; queries on that node return 503 until the index is ready. Full-text indexes require RocksDB and audit logging. -A raw-hit budget failure returns `400`; narrow the query or increase the budget. Capacity exhaustion -and timeouts return errors instead of successful incomplete pages. If the snapshot itself runs out -of eligible matches, a shorter completed page is a valid result. +The initial native autocomplete batch is not a total-result cap. Harper continues through later batches when needed to satisfy the requested page after filtering. Work and time limits still apply. -Retained readers can keep old index segments on disk until released. These settings do not provide -a hard limit on total process memory or retained disk space. Native deadlines are cooperative, so the -request budget is not a hard wall-clock guarantee. +Full-text counts are unavailable: a count response does not turn a native hit total into an authorized record count. For the ordinary REST paging syntax, see [Querying](../rest/querying.md#pagination-and-total-count). The JSON Operations API does not accept REST query syntax; use exported table routes or `Table.search()`. diff --git a/reference/resources/overview.md b/reference/resources/overview.md index 4a00d8f17..d0574af6b 100644 --- a/reference/resources/overview.md +++ b/reference/resources/overview.md @@ -172,4 +172,3 @@ Parameterized routes also surface in the generated OpenAPI document (as template | --------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | [Resource API](./resource-api.md) | Complete reference for instance methods, static methods, the Query object, RequestTarget, and response handling | | [Query Optimization](./query-optimization.md) | How Harper executes queries and how to write performant conditions | -| [Full-Text Paging](./full-text-paging.md) | Resume full-text and autocomplete searches on their original index snapshot | diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index 5e72803dd..dfe2cf9d1 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -79,12 +79,6 @@ When a secondary index is still being built — for example, just after a new `@ This is a transient condition — the index finishes building in the background, after which the same query succeeds. Clients should retry (with backoff) rather than treating it as an empty result or a permanent failure. On the operations API the error body includes `code` and `retryable`, so callers can branch on `error.code === 'INDEX_REBUILDING'` or `error.retryable`. -### Full-Text Search Work Limits - - - -Full-text queries have configurable work limits, including ordinary searches without continuation. A selective filter can exhaust a work limit and fail the request instead of returning a partial page. See [Full-Text Paging](./full-text-paging.md#configure-resource-limits) for the limits, errors and cost of increasing them. - ## Relationships and Joins Harper supports relationship-based queries that join data across tables. See [Schema documentation](../database/schema.md) for how to define relationships. diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 0f63da00b..da064a259 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -91,17 +91,7 @@ The `get()` method returns a `RecordObject` — a frozen plain object with all r ### `search(query: RequestTarget | Query, context?): ExtendedIterable` -Performs a query on the resource or table. This is called by `get()` on collection requests and can be overridden to define custom query behavior. A table overriding `search()` must also implement `searchPage()` with the same filtering policy to enable continuation, including through REST. The default implementation on tables queries by the `conditions`, `limit`, `offset`, `select`, and `sort` properties parsed from the URL or query object. See [Query Object](#query-object) below for available query options. See the [ExtendedIterable](#extendediterable) below for how to interact with the query results. - ---- - -### `searchPage(query: RequestTarget | Query, context?): Promise` - - - -Tables use `searchPage()` for full-text and autocomplete continuation. Supply a positive `limit` and omit `continuation` or set it to `true` to start; supply the returned token to resume. `FullTextSearchPage` is an array of records with `continuation`, `complete`, and `expiresAt` properties; see the [result properties](./full-text-paging.md#page-through-tablesearchpage). `search()` stays iterable and rejects continuation options. - -When serving an authenticated caller, pass the caller's context and set `checkPermission: true` in the query. A custom `search()` override must explicitly implement `searchPage()` with the same filtering policy; otherwise paging returns 400. See [Full-Text Paging](./full-text-paging.md) for examples, supported queries, closing a search, and token expiry. +Performs a query on the resource or table. This is called by `get()` on collection requests and can be overridden to define custom query behavior. The default implementation on tables queries by the `conditions`, `limit`, `offset`, `select`, and `sort` properties parsed from the URL or query object. See [Query Object](#query-object) below for available query options. See the [ExtendedIterable](#extendediterable) below for how to interact with the query results. --- @@ -1031,21 +1021,7 @@ body, so avoid putting sensitive detail in thrown error messages. ## Query Object -The `Query` object is accepted by `search()`, `searchPage()`, and the static `get()` method. - -### `continuation` - -For `searchPage()`, omit this field or use `true` to start a search; use the returned token to resume. Keep the query and caller context consistent. See [Full-Text Paging](./full-text-paging.md#page-through-tablesearchpage). - -### `closeContinuation` - -Use `true` with an existing continuation token to close the search early. See [Close or restart a search](./full-text-paging.md#close-or-restart-a-search). - -### `checkPermission` - -Set `true` on every page when a direct table call acts for a caller, and pass that caller's context. Without it, the call does not request the caller's table and searched-source permission checks; context alone does not enable them. Automatic REST applies its permission checks. See [Full-Text Paging](./full-text-paging.md#page-through-tablesearchpage). - -Set this field in server code; never copy permission-control fields from client data. +The `Query` object is accepted by `search()` and the static `get()` method. ### `conditions` @@ -1059,10 +1035,6 @@ Array of condition objects to filter records. Each condition: | `conditions` | Nested conditions array | | `operator` | `and` (default) or `or` for the nested `conditions` | -Full-text conditions name a query-only `FullText` declaration and use comparators `matches`, -`matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix`. Each also has a `not_`-prefixed form; a negated full-text condition requires a positive full-text condition in the query. See -`Table.searchPage()` and [Full-Text Paging](./full-text-paging.md) for snapshot continuation and its query restrictions. - Example with nested conditions: ```javascript @@ -1113,7 +1085,6 @@ Special properties: - `$id` — Returns the primary key regardless of its name - `$updatedtime` — Returns the last-updated timestamp - `$distance` — When the query ranks or filters by a vector index, returns the computed distance from the target vector. See [Vector Indexing](../database/schema.md#vector-indexing). -- `$score` - When a query uses a full-text index, returns its relevance score. See [Full-Text Paging](./full-text-paging.md#page-through-tablesearchpage). #### Selecting related records diff --git a/reference/rest/headers.md b/reference/rest/headers.md index 7a64a1f39..71603c802 100644 --- a/reference/rest/headers.md +++ b/reference/rest/headers.md @@ -22,17 +22,11 @@ These headers are included in all Harper REST API responses: Collection responses to a [count request](./querying.md#pagination-and-total-count) additionally include `Content-Range`, `Range-Unit`, and `Preference-Applied` (). -Full-text continuation responses also return `Harper-Continuation-Complete`, plus -`Harper-Continuation` and `Harper-Continuation-Expires` while the search remains open. These names -are added to `Access-Control-Expose-Headers`. See -[Full-Text Paging](../resources/full-text-paging.md#page-through-rest) (v5.4.0). - ## Cache-Control Harper applies a tiered Cache-Control policy to REST responses. -Full-text continuation responses override this policy with `Cache-Control: no-store`. ### Anonymous reads diff --git a/reference/rest/overview.md b/reference/rest/overview.md index 86b1826e4..0f60b27c6 100644 --- a/reference/rest/overview.md +++ b/reference/rest/overview.md @@ -88,9 +88,7 @@ With both in place, the exported name becomes the base path (`Product` above, or Notes on this surface: - The trailing slash is significant throughout: `/Product` addresses the resource itself, `/Product/` addresses its collection of records. See [URL Structure](#url-structure). -- `HEAD` is served as `GET` with the response body omitted, except that - [full-text continuation requests](../resources/full-text-paging.md#page-through-rest) return `400` - before opening or advancing a search. `QUERY` is accepted on the collection path (`QUERY /Product/`) and runs a search taken from the request body rather than the URL. +- `HEAD` is served exactly as `GET` with the response body omitted. `QUERY` is accepted on the collection path (`QUERY /Product/`) and runs a search taken from the request body rather than the URL. - A collection `POST` only generates a primary key when the body omits the table's primary-key property. If the body supplies one, Harper stores the record under that value rather than replacing it, and a `POST` to a key that already exists fails with `409` instead of overwriting. - On a successful `POST`, the new record's primary key — supplied or generated — is returned in the `Location` response header. The header carries the bare key value, not a URL, so it is not a link to follow. - `PUT` replaces the stored record, with three exceptions that Harper always applies: a [`@createdTime`](../database/schema.md#createdtime) attribute keeps the original record's value, an [`@updatedTime`](../database/schema.md#updatedtime) attribute is re-stamped with the time of the write, and the primary key is forced to match the `{id}` in the URL even if the body carries a different one. diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 0ae96a24c..03743191e 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -55,9 +55,6 @@ Harper uses [FIQL](https://datatracker.ietf.org/doc/html/draft-nottingham-atompu | `=`, `===` | Strict equality (no type conversion) | | `!==` | Strict inequality (no type conversion) | -Declared full-text fields use `=matches=`, `=matches_prefix=` or `=matches_fuzzy_prefix=`. -See [Full-Text Paging](../resources/full-text-paging.md#page-through-rest) for examples and continuation. - **Examples**: ```http @@ -176,28 +173,16 @@ GET /Product/?rating=gt=3&sort(+name) GET /Product/?sort(+rating,-price) ``` -### `continuation()` and `closeContinuation()` - - - -Full-text searches can retain their original index snapshot between requests. Start with -`continuation()`, resume with `continuation(token)`, and close early with `closeContinuation()` -alongside the token. Use `limit(n)` to set the page size. See -[Full-Text Paging](../resources/full-text-paging.md#page-through-rest) for the token headers, -query restrictions and recovery steps. - ## Pagination and Total Count +Full-text and autocomplete queries use the same paging syntax. See [Full-text paging](../resources/full-text-paging.md) for search work limits and behavior when records change between requests. + Use `limit(start,end)` to page through a collection, and opt in to a total match count with the `Prefer` request header (`Prefer: count=exact`) so a client can render pagination (for example "1-25 of 1,234") without a second request. -For full-text paging across requests (), use [continuation](#continuation-and-closecontinuation) instead of offset paging or a count request. - Counting is opt-in: without the header, no count is computed and no count headers are returned. It applies only to `GET`/`HEAD` requests and requires a bounded page — the `limit()` must be a non-negative integer no larger than **10,000**, and the requested window (offset + limit) no larger than **1,000,000**. A request with no `limit()`, one outside those bounds, or a non-numeric one is served normally with no count headers, since counting an unbounded page would defeat the point of paging. -Full-text queries also have [search work limits](../resources/full-text-paging.md#configure-resource-limits). A count request does not remove those limits; a full-text query can fail before returning a page or count. - ### Requesting a count Send a `Prefer` header on a `GET` (or `HEAD`) request to a collection: diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index f90565f3f..7245d5c19 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -8,18 +8,6 @@ title: '5.4' All patch release notes for 5.4.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.4&expanded=true). -## Upgrade Notes - -### Full-Text Search Budgets - -The new `storage.fullText` work limits also apply to searches without continuation: the ordinary fetched-hit ceiling defaults to 10,000 and the shared execution budget defaults to 5,000 ms. Replayed batches count toward the hit ceiling. Hitting the raw-hit ceiling returns 400 instead of a successful partial page; exhausting the execution budget returns 503. Test selective full-text queries before upgrading. Narrow an expensive query or reduce its page size before increasing a work limit; higher limits permit more work and record reads per request. See [resource limits](/reference/v5/resources/full-text-paging#configure-resource-limits) before changing the defaults. - -## Full-Text Search - -### Snapshot Paging - -Full-text and autocomplete searches can retain their original index matches between requests through `Table.searchPage()` or REST continuation tokens. Ordinary `Table.search()` remains iterable. See [Full-Text Paging](/reference/v5/resources/full-text-paging). - ## Replication ### Dedicated Replication Threads @@ -75,3 +63,9 @@ The new `threads.v8Flags` setting takes a V8 flag, or a list of them, and applie ### Restore Checks a Backup Before Deleting the Database `restore_backup` no longer deletes a database before checking that the backup can be read. It first verifies the checksum of every database file in the backup, checks its transaction logs, opens each table file to confirm this version can read its format and compression, and, for a backup that captured blobs, checks that the blob snapshot is present. A backup that fails is refused and the database is left as it was (one an earlier restore left incomplete stays blocked); previously the restore deleted the database first and then failed, so a backup taken by a newer version, or one with a corrupt file, destroyed the database it was meant to restore. The check reads the whole backup once, so a restore takes longer, and an online restore keeps the database unavailable while it runs; it needs no extra disk space. A restore interrupted while it is still checking the backup leaves the destination's files and any preexisting incomplete-restore marker unchanged; a previously healthy database loads normally on the next start. Blob file contents are not checked. See [`restore_backup`](/reference/v5/backups/operations#restore_backup). + +### Full-text paging and search work limits + +Full-text and autocomplete queries use the existing `limit` and `offset` paging options. Harper reads additional native batches when needed after filtering, so the initial native result window no longer silently cuts off eligible results. Requests that exceed configured work or time budgets return an error. Separate page requests run independently and can see changes to records or index contents. + +See [Full-text paging](/reference/v5/resources/full-text-paging) for examples, limits and upgrade considerations. diff --git a/sidebarsReference.ts b/sidebarsReference.ts index d7ea4b862..a2306e408 100644 --- a/sidebarsReference.ts +++ b/sidebarsReference.ts @@ -87,11 +87,7 @@ const sidebars: SidebarsConfig = { id: 'resources/query-optimization', label: 'Query Optimization', }, - { - type: 'doc', - id: 'resources/full-text-paging', - label: 'Full-Text Paging', - }, + { type: 'doc', id: 'resources/full-text-paging', label: 'Full-text Paging' }, ], }, { From abcd092e086f5075ae9fe048e1c0368587e79626 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 21:50:23 -0600 Subject: [PATCH 45/78] Clarify full-text query syntax and budget recovery --- reference/configuration/operations.md | 2 ++ reference/resources/full-text-paging.md | 11 ++++++++++- reference/resources/resource-api.md | 2 ++ reference/rest/querying.md | 4 +++- 4 files changed, 17 insertions(+), 2 deletions(-) diff --git a/reference/configuration/operations.md b/reference/configuration/operations.md index 75d9ed695..134980d32 100644 --- a/reference/configuration/operations.md +++ b/reference/configuration/operations.md @@ -18,6 +18,8 @@ For the full list of configurable options, see [Configuration Options](./options Modifies one or more Harper configuration parameters. **Requires a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`). A change to `replication.threads` takes effect only on a full `restart`. +Full-text request work limits (`storage.fullText.maxRawHitsPerRequest` and `requestBudgetMilliseconds`) apply to new searches without a restart. Changes to `storage.fullText.runtime` require a full Harper process restart; `restart_service` does not replace the native process budget. See [Full-text paging](../resources/full-text-paging.md#configure-search-work). + `operation` _(required)_ — must be `set_configuration` `replicated` _(optional)_ — set to `true` to apply the same configuration change to all cluster nodes in a single call. The origin node applies the change first; if its local write fails, nothing is sent to peers. Requires replication to be configured. diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index 53955bad3..08585665a 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -74,4 +74,13 @@ This change uses the fulltext 0.6.0 native package. A format-incompatible upgrad The initial native autocomplete batch is not a total-result cap. Harper continues through later batches when needed to satisfy the requested page after filtering. Work and time limits still apply. -Full-text counts are unavailable: a count response does not turn a native hit total into an authorized record count. For the ordinary REST paging syntax, see [Querying](../rest/querying.md#pagination-and-total-count). The JSON Operations API does not accept REST query syntax; use exported table routes or `Table.search()`. +For an eligible bounded count request, full-text counts are unavailable and `Content-Range` uses `*` for the total. A native hit total is not an authorized record count. For the ordinary REST paging syntax, see [Querying](../rest/querying.md#pagination-and-total-count). The JSON Operations API does not accept REST query syntax; use exported table routes or `Table.search()`. + +## Errors and recovery + +| Situation | HTTP status | Action | +| --------------------------------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------ | +| Fetched-hit budget exhausted | 400 | Narrow the query, reduce the offset, or raise `maxRawHitsPerRequest`. | +| Execution budget exhausted | 503 | Narrow the query or adjust `requestBudgetMilliseconds`; retrying the same expensive query can hit the same limit. | +| Index rebuilding or unavailable | 503 | Wait for the local index to become ready before retrying. | +| Application worker started before native runtime configuration was captured | 500 | `FullTextWorkerInitializationError` is non-retryable in that worker. Recreate the worker after Harper initializes. | diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index da064a259..cf44bb715 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1035,6 +1035,8 @@ Array of condition objects to filter records. Each condition: | `conditions` | Nested conditions array | | `operator` | `and` (default) or `or` for the nested `conditions` | +Full-text conditions use `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, or `matches_fuzzy_prefix` against a declared full-text index. See [Full-text paging](./full-text-paging.md) for a `Table.search()` example and work limits. + Example with nested conditions: ```javascript diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 03743191e..bf705273a 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -55,6 +55,8 @@ Harper uses [FIQL](https://datatracker.ietf.org/doc/html/draft-nottingham-atompu | `=`, `===` | Strict equality (no type conversion) | | `!==` | Strict inequality (no type conversion) | +Full-text indexes support `=matches=`, `=matches_all=`, `=matches_phrase=`, `=matches_prefix=`, `=matches_fuzzy=`, and `=matches_fuzzy_prefix=`. These use the full-text index rather than ordinary string comparisons. See [Full-text paging](../resources/full-text-paging.md). + **Examples**: ```http @@ -157,7 +159,7 @@ GET /Product/?brand.name=Microsoft&select(name,brand{name}) ### `limit(end)` or `limit(start,end)` -Limit the number of results returned, with an optional starting offset. +Limit the number of results returned, with an optional starting offset. The start is inclusive and the end is exclusive: `limit(10,30)` skips 10 records and returns up to 20. ```http GET /Product/?rating=gt=3&inStock=true&select(rating,name)&limit(20) From ada02a40a50d25bafa07039ccbba435b80a5201c Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 21:51:04 -0600 Subject: [PATCH 46/78] Describe exact and estimated full-text counts accurately --- reference/resources/full-text-paging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index 08585665a..959edbd66 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -74,7 +74,7 @@ This change uses the fulltext 0.6.0 native package. A format-incompatible upgrad The initial native autocomplete batch is not a total-result cap. Harper continues through later batches when needed to satisfy the requested page after filtering. Work and time limits still apply. -For an eligible bounded count request, full-text counts are unavailable and `Content-Range` uses `*` for the total. A native hit total is not an authorized record count. For the ordinary REST paging syntax, see [Querying](../rest/querying.md#pagination-and-total-count). The JSON Operations API does not accept REST query syntax; use exported table routes or `Table.search()`. +Full-text queries do not provide an exact record count. When exact counting is enabled and requested, `Content-Range` uses `*` for the total. Estimated counting can return a planner estimate when no opaque row or vector filter suppresses it. A native hit total is not an authorized record count. See [count preferences](../rest/querying.md#pagination-and-total-count) for the default downgrade from exact to estimated counting. For the ordinary REST paging syntax, see [Querying](../rest/querying.md#pagination-and-total-count). The JSON Operations API does not accept REST query syntax; use exported table routes or `Table.search()`. ## Errors and recovery From 9643499aa189983528fb5cfc342d25845ae172bc Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 21:59:24 -0600 Subject: [PATCH 47/78] Make paging prerequisites and operating limits discoverable --- reference/configuration/options.md | 3 +-- reference/resources/full-text-paging.md | 24 ++++++++++++++++++++---- release-notes/v5-lincoln/5.4.md | 6 ++++-- 3 files changed, 25 insertions(+), 8 deletions(-) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 27bd93331..c091b86cd 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -312,6 +312,7 @@ storage: ``` - `engine` — The database storage engine to use for new databases. Currently supported engines are `rocksdb` and `lmdb`. The default is `rocksdb`. Existing databases will use the engine that was used when they were created. +- `fullText` — Full-text request work limits and native process budgets; see [Full-text paging](../resources/full-text-paging.md#configure-search-work). - `writeAsync` — Disable disk sync for higher throughput (**disables durability guarantees**); _Default_: `false` - `caching` — In-memory record caching; _Default_: `true` - `compression` — LZ4 record compression; _Default_: `true` (enabled by default since v4.3.0). Sub-options: `dictionary`, `threshold` @@ -578,5 +579,3 @@ my-component: - `.package` — NPM package name, GitHub repo (`user/repo`), or local path - `.port` — Port for the component; _Default_: value of `http.port` - -Full-text search work limits and native runtime admission settings are described in [Full-text paging](../resources/full-text-paging.md#configure-search-work). diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index 959edbd66..fd0cf159f 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -8,6 +8,20 @@ title: Full-text paging Full-text and autocomplete searches use Harper's existing paging options. Each request runs a new search; Harper does not retain a search between pages. +## Before you start + +Use an audited RocksDB table with a declared full-text field. For example: + +```graphql +type Product @table(audit: true) @export { + id: ID @primaryKey + title: String + catalogSearch: FullText @fullText(fields: [{ name: "title" }]) +} +``` + +`catalogSearch` names the index over `title`; it is not a stored record value. The examples below search that index through an exported table route or an authenticated Resource handler. + ## Request a page Inside an authenticated Resource handler, use `limit` and `offset` with `Table.search()`. `Table.search()` returns an iterable, including when a custom Resource forwards or filters it. @@ -34,7 +48,7 @@ Each page runs a new query. Changes to records, permissions, or the index can mo Within one request, Harper reads native batches on the same index reader until enough eligible records have been found or the matches are exhausted. Current source versions and record filters are checked before counting records toward the offset and limit. This traversal also lets prefix and fuzzy-prefix searches return more than the initial autocomplete batch. The library's ordinary search window is not a total-result cutoff for a bounded Harper request. -Use a bounded `limit` for large result sets. An unbounded query that exceeds the native result window returns 400. Offset requests revisit earlier matches; deep offsets and restrictive filters can exhaust the work or time budget. An exhausted search returns its remaining eligible records, including an empty result. A budget failure returns an error instead of a successful incomplete page. +Use a bounded `limit` for large result sets. An unbounded query that exceeds the native result window returns 400. `offset + limit` must not exceed `maxRawHitsPerRequest`; replay and rejected hits can exhaust that budget even earlier. Offset requests revisit earlier matches; deep offsets and restrictive filters can exhaust the work or time budget. Harper can retain up to `offset + limit` accepted records while building a bounded result, so raising the work limit can also increase memory use. An exhausted search returns its remaining eligible records, including an empty result. A budget failure returns an error instead of a successful incomplete page. ## Configure search work @@ -64,9 +78,9 @@ Native process admission settings live under `storage.fullText.runtime`: These limits apply together. For example, two indexing threads and 64 MiB of writer memory per index reach the default thread and memory budgets at 64 indexes. Readers share an index's runtime; they do not allocate another writer. Admission limits are not hard bounds on total resident memory or disk usage. -Harper captures native runtime budgets before loading application components, or on an earlier main-thread native load or managed worker spawn. Application-created workers inherit that configuration. A worker created before capture fails with `FullTextWorkerInitializationError` (500, non-retryable); recreate it after Harper initializes. Replacement workers keep the established budgets after a live configuration change. Restart the Harper process to apply new native runtime budgets; restarting only HTTP workers is insufficient. +Harper captures native runtime budgets before loading application components, or on an earlier main-thread native load or managed worker spawn. Application-created workers inherit that configuration. A worker created before capture fails on its first native full-text load with `FullTextWorkerInitializationError` (500, non-retryable). Create application workers from a Harper application component, after runtime configuration has been captured; recreate an early worker rather than retrying inside it. Replacement workers keep the established budgets after a live configuration change. Restart the Harper process to apply new native runtime budgets; restarting only HTTP workers is insufficient. -Harper does not configure the library's native filter metadata or submit native filter or candidate-ID requests. Ordinary conditions continue through Harper's secondary-index gates and authoritative record predicates. Native page, response-byte and highlight limits still apply. Query terms and clauses remain limited by the installed library. +Native page, response-byte and highlight limits still apply. Query terms and clauses remain limited by the installed library. ## Upgrade considerations @@ -74,7 +88,7 @@ This change uses the fulltext 0.6.0 native package. A format-incompatible upgrad The initial native autocomplete batch is not a total-result cap. Harper continues through later batches when needed to satisfy the requested page after filtering. Work and time limits still apply. -Full-text queries do not provide an exact record count. When exact counting is enabled and requested, `Content-Range` uses `*` for the total. Estimated counting can return a planner estimate when no opaque row or vector filter suppresses it. A native hit total is not an authorized record count. See [count preferences](../rest/querying.md#pagination-and-total-count) for the default downgrade from exact to estimated counting. For the ordinary REST paging syntax, see [Querying](../rest/querying.md#pagination-and-total-count). The JSON Operations API does not accept REST query syntax; use exported table routes or `Table.search()`. +Full-text queries do not provide an exact record count. When exact counting is enabled and requested, `Content-Range` uses `*` for the total. Estimated counting can return a planner estimate when no opaque row or vector filter suppresses it. A native hit total is not an authorized record count. See [count preferences](../rest/querying.md#pagination-and-total-count) for the default downgrade from exact to estimated counting. For the ordinary REST paging syntax, see [Querying](../rest/querying.md#pagination-and-total-count). The JSON Operations API uses `search_by_conditions` with `limit` and `offset` rather than REST URL syntax. The same per-request behavior and work limits apply. ## Errors and recovery @@ -84,3 +98,5 @@ Full-text queries do not provide an exact record count. When exact counting is e | Execution budget exhausted | 503 | Narrow the query or adjust `requestBudgetMilliseconds`; retrying the same expensive query can hit the same limit. | | Index rebuilding or unavailable | 503 | Wait for the local index to become ready before retrying. | | Application worker started before native runtime configuration was captured | 500 | `FullTextWorkerInitializationError` is non-retryable in that worker. Recreate the worker after Harper initializes. | + +`INDEX_REBUILDING` and `DERIVED_INDEX_LAGGING` identify retryable readiness or coverage failures. A native busy response also returns 503; reduce concurrency or retry with backoff. HTTP status alone does not distinguish these from an execution-budget failure. diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 7245d5c19..122cf5546 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -64,8 +64,10 @@ The new `threads.v8Flags` setting takes a V8 flag, or a list of them, and applie `restore_backup` no longer deletes a database before checking that the backup can be read. It first verifies the checksum of every database file in the backup, checks its transaction logs, opens each table file to confirm this version can read its format and compression, and, for a backup that captured blobs, checks that the blob snapshot is present. A backup that fails is refused and the database is left as it was (one an earlier restore left incomplete stays blocked); previously the restore deleted the database first and then failed, so a backup taken by a newer version, or one with a corrupt file, destroyed the database it was meant to restore. The check reads the whole backup once, so a restore takes longer, and an online restore keeps the database unavailable while it runs; it needs no extra disk space. A restore interrupted while it is still checking the backup leaves the destination's files and any preexisting incomplete-restore marker unchanged; a previously healthy database loads normally on the next start. Blob file contents are not checked. See [`restore_backup`](/reference/v5/backups/operations#restore_backup). -### Full-text paging and search work limits +## Full-Text Search -Full-text and autocomplete queries use the existing `limit` and `offset` paging options. Harper reads additional native batches when needed after filtering, so the initial native result window no longer silently cuts off eligible results. Requests that exceed configured work or time budgets return an error. Separate page requests run independently and can see changes to records or index contents. +### Paging and Search Work Limits + +Full-text and autocomplete queries use the existing `limit` and `offset` paging options. Harper reads additional native batches when needed after filtering, so the initial native result window no longer silently cuts off eligible results. The new `storage.fullText.maxRawHitsPerRequest` and `storage.fullText.requestBudgetMilliseconds` settings bound request work. Requests that exceed those budgets return an error. Separate page requests run independently and can see changes to records or index contents. See [Full-text paging](/reference/v5/resources/full-text-paging) for examples, limits and upgrade considerations. From cf7eba92d41d7105ab86e56a0a8f2e2f03c20019 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 22:10:44 -0600 Subject: [PATCH 48/78] Use the table global in the paging example --- reference/resources/full-text-paging.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index fd0cf159f..5dc2cef85 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -27,7 +27,7 @@ type Product @table(audit: true) @export { Inside an authenticated Resource handler, use `limit` and `offset` with `Table.search()`. `Table.search()` returns an iterable, including when a custom Resource forwards or filters it. ```javascript -return Product.search( +return tables.Product.search( { checkPermission: true, conditions: [{ attribute: 'catalogSearch', comparator: 'matches', value: 'trail shoes' }], @@ -84,7 +84,7 @@ Native page, response-byte and highlight limits still apply. Query terms and cla ## Upgrade considerations -This change uses the fulltext 0.6.0 native package. A format-incompatible upgrade rebuilds the local index from stored records; queries on that node return 503 until the index is ready. Full-text indexes require RocksDB and audit logging. +Harper 5.4 uses the fulltext 0.6.0 native package. A format-incompatible upgrade rebuilds the local index from stored records; queries on that node return 503 until the index is ready. Full-text indexes require RocksDB and audit logging. The initial native autocomplete batch is not a total-result cap. Harper continues through later batches when needed to satisfy the requested page after filtering. Work and time limits still apply. From 12fdb83b2fc92cef275f8b4ecb593ed2b1b6835e Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 22:12:12 -0600 Subject: [PATCH 49/78] Link the pinned native query limits --- reference/resources/full-text-paging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md index 5dc2cef85..e2496107f 100644 --- a/reference/resources/full-text-paging.md +++ b/reference/resources/full-text-paging.md @@ -80,7 +80,7 @@ These limits apply together. For example, two indexing threads and 64 MiB of wri Harper captures native runtime budgets before loading application components, or on an earlier main-thread native load or managed worker spawn. Application-created workers inherit that configuration. A worker created before capture fails on its first native full-text load with `FullTextWorkerInitializationError` (500, non-retryable). Create application workers from a Harper application component, after runtime configuration has been captured; recreate an early worker rather than retrying inside it. Replacement workers keep the established budgets after a live configuration change. Restart the Harper process to apply new native runtime budgets; restarting only HTTP workers is insufficient. -Native page, response-byte and highlight limits still apply. Query terms and clauses remain limited by the installed library. +Native page, response-byte and highlight limits still apply. Fulltext 0.6.0 also limits each analyzed query leaf to 64 terms and the compiled query to 256 clauses. Raising Harper's work budgets does not raise those limits. See the [published library limits](https://github.com/HarperFast/fulltext/blob/v0.6.0/docs/limits.md) for the full list and recovery guidance. ## Upgrade considerations From 83cc98f1e8cac907a66aed736b6034ed28f393ac Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 22:23:38 -0600 Subject: [PATCH 50/78] Clarify paging count and configuration exceptions --- reference/configuration/operations.md | 4 ++-- reference/configuration/options.md | 2 +- reference/operations-api/operations.md | 10 +++++----- reference/rest/querying.md | 2 ++ 4 files changed, 10 insertions(+), 8 deletions(-) diff --git a/reference/configuration/operations.md b/reference/configuration/operations.md index 134980d32..b74335bc0 100644 --- a/reference/configuration/operations.md +++ b/reference/configuration/operations.md @@ -16,9 +16,9 @@ For the full list of configurable options, see [Configuration Options](./options ## Set Configuration -Modifies one or more Harper configuration parameters. **Requires a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`). A change to `replication.threads` takes effect only on a full `restart`. +Modifies one or more Harper configuration parameters. **Most changes require a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`). A change to `replication.threads` takes effect only on a full `restart`. -Full-text request work limits (`storage.fullText.maxRawHitsPerRequest` and `requestBudgetMilliseconds`) apply to new searches without a restart. Changes to `storage.fullText.runtime` require a full Harper process restart; `restart_service` does not replace the native process budget. See [Full-text paging](../resources/full-text-paging.md#configure-search-work). +Full-text request work limits (`storage.fullText.maxRawHitsPerRequest` and `storage.fullText.requestBudgetMilliseconds`) apply to new searches without a restart. Changes to `storage.fullText.runtime` require a full Harper process restart; `restart_service` does not replace the native process budget. See [Full-text paging](../resources/full-text-paging.md#configure-search-work). `operation` _(required)_ — must be `set_configuration` diff --git a/reference/configuration/options.md b/reference/configuration/options.md index c091b86cd..2b93582d9 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -312,7 +312,7 @@ storage: ``` - `engine` — The database storage engine to use for new databases. Currently supported engines are `rocksdb` and `lmdb`. The default is `rocksdb`. Existing databases will use the engine that was used when they were created. -- `fullText` — Full-text request work limits and native process budgets; see [Full-text paging](../resources/full-text-paging.md#configure-search-work). +- `fullText` — Full-text request work limits and native process budgets; see [search work limits](../resources/full-text-paging.md#configure-search-work) and [native process budgets](../resources/full-text-paging.md#configure-native-runtime-admission). - `writeAsync` — Disable disk sync for higher throughput (**disables durability guarantees**); _Default_: `false` - `caching` — In-memory record caching; _Default_: `true` - `compression` — LZ4 record compression; _Default_: `true` (enabled by default since v4.3.0). Sub-options: `dictionary`, `threshold` diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 21815c98a..865848d90 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -1679,14 +1679,14 @@ Operations for reading and updating Harper configuration. Detailed documentation: [Configuration Overview](../configuration/overview.md) -| Operation | Description | Role Required | -| ------------------- | ---------------------------------------------------------------- | ------------- | -| `set_configuration` | Modifies Harper configuration file parameters (requires restart) | super_user | -| `get_configuration` | Returns the current Harper configuration | super_user | +| Operation | Description | Role Required | +| ------------------- | --------------------------------------------- | ------------- | +| `set_configuration` | Modifies Harper configuration file parameters | super_user | +| `get_configuration` | Returns the current Harper configuration | super_user | ### `set_configuration` -Updates configuration parameters in `harper-config.yaml`. A restart (`restart` or `restart_service`) is required for changes to take effect. +Updates configuration parameters in `harper-config.yaml`. Most changes require a restart (`restart` or `restart_service`) to take effect. Full-text request work limits apply to new searches without a restart; changes to the native process budgets require a full `restart`. See [Full-text paging](../resources/full-text-paging.md#configure-search-work). Supports `"replicated": true` to apply the same change to all cluster nodes in one call; per-node outcomes are returned in the response's `replicated` array. Only send cluster-appropriate parameters when replicating — node-local parameters (ports, `node.hostname`, file paths, TLS material, `replication.hostname`/`url`/`routes`) would overwrite every peer's local values. To apply the change cluster-wide, follow with `restart_service` using `"replicated": true` (which restarts nodes one at a time); `restart_service` restarts only worker threads, though, so a change to [`threads.v8Flags`](../configuration/options.md#applying-v8-flags) takes effect only after a `restart` sent to each node in turn (`restart` does not replicate). See [Configuration Operations](../configuration/operations.md#set-configuration) for details. diff --git a/reference/rest/querying.md b/reference/rest/querying.md index bf705273a..4ff839e56 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -222,6 +222,8 @@ The response status is always `200` — `Content-Range` is informational (Harper The total is reported as `*` (for example `Content-Range: items 0-24/*`) when it cannot be produced — an exact scan that hits its guardrail (counting the tail past the requested page is bounded by a 1,000,000-row cap and a ~1-second budget, and abandons the total rather than truncating the page), or a query with no cardinality estimate (for example a `!=` or `=ct=` (contains) condition). `Preference-Applied` still echoes the requested mode, so an unavailable total (`.../*`) is distinct from a request that asked for no count. +Full-text and autocomplete queries cannot provide an exact total. An estimated count may use the planner estimate when the query has no opaque row or vector filter. See [Full-text paging](../resources/full-text-paging.md#request-a-page). + ### Enabling exact counts Because an exact count scans the full matched set, it is **off by default**. Enable it in an application's REST configuration via the [`exactCount` option](./overview.md#configuration): From c7f3a79a733e0abecee9fa6494878351491fc8f4 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 22:31:51 -0600 Subject: [PATCH 51/78] Keep paging guidance in existing reference sections --- reference/configuration/operations.md | 2 +- reference/configuration/options.md | 13 ++- reference/configuration/overview.md | 2 +- reference/operations-api/operations.md | 4 +- reference/resources/full-text-paging.md | 102 ---------------------- reference/resources/query-optimization.md | 30 +++++++ reference/resources/resource-api.md | 2 +- reference/rest/querying.md | 6 +- release-notes/v5-lincoln/5.4.md | 2 +- sidebarsReference.ts | 1 - 10 files changed, 51 insertions(+), 113 deletions(-) delete mode 100644 reference/resources/full-text-paging.md diff --git a/reference/configuration/operations.md b/reference/configuration/operations.md index b74335bc0..177935496 100644 --- a/reference/configuration/operations.md +++ b/reference/configuration/operations.md @@ -18,7 +18,7 @@ For the full list of configurable options, see [Configuration Options](./options Modifies one or more Harper configuration parameters. **Most changes require a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`). A change to `replication.threads` takes effect only on a full `restart`. -Full-text request work limits (`storage.fullText.maxRawHitsPerRequest` and `storage.fullText.requestBudgetMilliseconds`) apply to new searches without a restart. Changes to `storage.fullText.runtime` require a full Harper process restart; `restart_service` does not replace the native process budget. See [Full-text paging](../resources/full-text-paging.md#configure-search-work). +Full-text request work limits (`storage.fullText.maxRawHitsPerRequest` and `storage.fullText.requestBudgetMilliseconds`) apply to new searches without a restart. Changes to `storage.fullText.runtime` require a full Harper process restart; `restart_service` does not replace the native process budget. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). `operation` _(required)_ — must be `set_configuration` diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 2b93582d9..7ab42326b 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -312,7 +312,18 @@ storage: ``` - `engine` — The database storage engine to use for new databases. Currently supported engines are `rocksdb` and `lmdb`. The default is `rocksdb`. Existing databases will use the engine that was used when they were created. -- `fullText` — Full-text request work limits and native process budgets; see [search work limits](../resources/full-text-paging.md#configure-search-work) and [native process budgets](../resources/full-text-paging.md#configure-native-runtime-admission). +- `fullText` — Full-text search work and native process limits. Sub-options: + - `maxRawHitsPerRequest` — Maximum fetched hits per request, including replay when switching to native traversal; default `10000`. + - `requestBudgetMilliseconds` — Execution budget across native batches, source checks and highlights, capped by the native execution limit; default `5000` ms. Reader acquisition and freshness waiting have separate bounds. + - `runtime.maxResidentIndexes` — Maximum resident native indexes; default `128`. + - `runtime.maxIndexingThreads` — Maximum native indexing threads; default `128`. + - `runtime.maxSearchThreads` — Maximum native search threads; default `256`. + - `runtime.maxWriterMemoryBytes` — Native writer-memory admission budget; default `4294967296` (4 GiB). + - `runtime.maxQueuedBytes` — Native queued-write admission budget; default `8589934592` (8 GiB). + - `runtime.maxExpensiveSearches` — Maximum concurrent expensive native searches; default `8`. + + The runtime budgets apply together and are not hard bounds on total process memory. Work-limit changes through `set_configuration` apply to new searches; runtime-budget changes require a full Harper process restart. Use the usual underscore-separated parameter names, such as `storage_fullText_maxRawHitsPerRequest`. See [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for costs and failure recovery. + - `writeAsync` — Disable disk sync for higher throughput (**disables durability guarantees**); _Default_: `false` - `caching` — In-memory record caching; _Default_: `true` - `compression` — LZ4 record compression; _Default_: `true` (enabled by default since v4.3.0). Sub-options: `dictionary`, `threshold` diff --git a/reference/configuration/overview.md b/reference/configuration/overview.md index 112327b24..309ef0726 100644 --- a/reference/configuration/overview.md +++ b/reference/configuration/overview.md @@ -21,7 +21,7 @@ For a complete reference of all available configuration options, see [Configurat ## The Configuration File -To change a configuration value, edit `harper-config.yaml` and save. **Harper must be restarted for changes to take effect.** +To change a configuration value, edit `harper-config.yaml` and save. **Restart Harper to apply file edits.** To change full-text request work limits without a restart, use [`set_configuration`](./operations.md#set-configuration). Native full-text runtime budgets still require a full process restart. Configuration keys use camelCase (e.g. `operationsApi`). Nested keys use dot notation conceptually (e.g. `operationsApi.network.port`). diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 865848d90..b0179399a 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -270,7 +270,7 @@ Returns records with a matching value on any attribute. Supports wildcards (e.g. ### `search_by_conditions` -Returns records matching one or more conditions. Supports `operator` (`and`/`or`), `offset`, `limit`, nested `conditions` groups, and `sort` with multi-level tie-breaking. +Returns records matching one or more conditions. Supports `operator` (`and`/`or`), `offset`, `limit`, nested `conditions` groups, and `sort` with multi-level tie-breaking. Full-text conditions use comparators such as `matches` and `matches_prefix`; see [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for paging costs and limits. ```json { @@ -1686,7 +1686,7 @@ Detailed documentation: [Configuration Overview](../configuration/overview.md) ### `set_configuration` -Updates configuration parameters in `harper-config.yaml`. Most changes require a restart (`restart` or `restart_service`) to take effect. Full-text request work limits apply to new searches without a restart; changes to the native process budgets require a full `restart`. See [Full-text paging](../resources/full-text-paging.md#configure-search-work). +Updates configuration parameters in `harper-config.yaml`. Most changes require a restart (`restart` or `restart_service`) to take effect. Full-text request work limits apply to new searches without a restart; changes to the native process budgets require a full `restart`. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). Supports `"replicated": true` to apply the same change to all cluster nodes in one call; per-node outcomes are returned in the response's `replicated` array. Only send cluster-appropriate parameters when replicating — node-local parameters (ports, `node.hostname`, file paths, TLS material, `replication.hostname`/`url`/`routes`) would overwrite every peer's local values. To apply the change cluster-wide, follow with `restart_service` using `"replicated": true` (which restarts nodes one at a time); `restart_service` restarts only worker threads, though, so a change to [`threads.v8Flags`](../configuration/options.md#applying-v8-flags) takes effect only after a `restart` sent to each node in turn (`restart` does not replicate). See [Configuration Operations](../configuration/operations.md#set-configuration) for details. diff --git a/reference/resources/full-text-paging.md b/reference/resources/full-text-paging.md deleted file mode 100644 index e2496107f..000000000 --- a/reference/resources/full-text-paging.md +++ /dev/null @@ -1,102 +0,0 @@ ---- -title: Full-text paging ---- - -# Full-text paging - - - -Full-text and autocomplete searches use Harper's existing paging options. Each request runs a new search; Harper does not retain a search between pages. - -## Before you start - -Use an audited RocksDB table with a declared full-text field. For example: - -```graphql -type Product @table(audit: true) @export { - id: ID @primaryKey - title: String - catalogSearch: FullText @fullText(fields: [{ name: "title" }]) -} -``` - -`catalogSearch` names the index over `title`; it is not a stored record value. The examples below search that index through an exported table route or an authenticated Resource handler. - -## Request a page - -Inside an authenticated Resource handler, use `limit` and `offset` with `Table.search()`. `Table.search()` returns an iterable, including when a custom Resource forwards or filters it. - -```javascript -return tables.Product.search( - { - checkPermission: true, - conditions: [{ attribute: 'catalogSearch', comparator: 'matches', value: 'trail shoes' }], - offset: 20, - limit: 20, - }, - this.getContext() -); -``` - -The equivalent REST range has an inclusive start and exclusive end. This request skips 20 matching records and returns up to 20 more: - -```text -GET /Product/?catalogSearch=matches=trail%20shoes&limit(20,40) -``` - -Each page runs a new query. Changes to records, permissions, or the index can move results between requests, so separate pages may omit or repeat records. There is no retained search or cross-request snapshot. - -Within one request, Harper reads native batches on the same index reader until enough eligible records have been found or the matches are exhausted. Current source versions and record filters are checked before counting records toward the offset and limit. This traversal also lets prefix and fuzzy-prefix searches return more than the initial autocomplete batch. The library's ordinary search window is not a total-result cutoff for a bounded Harper request. - -Use a bounded `limit` for large result sets. An unbounded query that exceeds the native result window returns 400. `offset + limit` must not exceed `maxRawHitsPerRequest`; replay and rejected hits can exhaust that budget even earlier. Offset requests revisit earlier matches; deep offsets and restrictive filters can exhaust the work or time budget. Harper can retain up to `offset + limit` accepted records while building a bounded result, so raising the work limit can also increase memory use. An exhausted search returns its remaining eligible records, including an empty result. A budget failure returns an error instead of a successful incomplete page. - -## Configure search work - -Search settings are under `storage.fullText` in `harper-config.yaml`: - -| Setting | Default | Meaning | -| --------------------------- | ------: | ----------------------------------------------------------------------------------------------------------------- | -| `maxRawHitsPerRequest` | 10,000 | Fetched native hits, including the initial batch replay when switching to traversal | -| `requestBudgetMilliseconds` | 5,000 | Execution budget across native batches, source checks and highlight tracing, capped by the native execution limit | - -The hit limit counts fetched hits, not every record Tantivy visits while scoring. Reaching it before the page fills or the search is exhausted returns 400. Exceeding the execution budget returns 503. Narrow the query, reduce the offset, or increase the relevant configured budget. Freshness waiting uses `waitForIndexMilliseconds` separately. Reader acquisition, query construction and some native operations can take additional time; the execution budget is not a hard response-time guarantee. - -Each setting has the usual configuration parameter name, such as `storage_fullText_maxRawHitsPerRequest`, for `set_configuration` and environment overrides. Query work limits apply to new searches after configuration changes. - -## Configure native runtime admission - -Native process admission settings live under `storage.fullText.runtime`: - -| Setting | Default | -| ---------------------- | ------: | -| `maxResidentIndexes` | 128 | -| `maxIndexingThreads` | 128 | -| `maxSearchThreads` | 256 | -| `maxWriterMemoryBytes` | 4 GiB | -| `maxQueuedBytes` | 8 GiB | -| `maxExpensiveSearches` | 8 | - -These limits apply together. For example, two indexing threads and 64 MiB of writer memory per index reach the default thread and memory budgets at 64 indexes. Readers share an index's runtime; they do not allocate another writer. Admission limits are not hard bounds on total resident memory or disk usage. - -Harper captures native runtime budgets before loading application components, or on an earlier main-thread native load or managed worker spawn. Application-created workers inherit that configuration. A worker created before capture fails on its first native full-text load with `FullTextWorkerInitializationError` (500, non-retryable). Create application workers from a Harper application component, after runtime configuration has been captured; recreate an early worker rather than retrying inside it. Replacement workers keep the established budgets after a live configuration change. Restart the Harper process to apply new native runtime budgets; restarting only HTTP workers is insufficient. - -Native page, response-byte and highlight limits still apply. Fulltext 0.6.0 also limits each analyzed query leaf to 64 terms and the compiled query to 256 clauses. Raising Harper's work budgets does not raise those limits. See the [published library limits](https://github.com/HarperFast/fulltext/blob/v0.6.0/docs/limits.md) for the full list and recovery guidance. - -## Upgrade considerations - -Harper 5.4 uses the fulltext 0.6.0 native package. A format-incompatible upgrade rebuilds the local index from stored records; queries on that node return 503 until the index is ready. Full-text indexes require RocksDB and audit logging. - -The initial native autocomplete batch is not a total-result cap. Harper continues through later batches when needed to satisfy the requested page after filtering. Work and time limits still apply. - -Full-text queries do not provide an exact record count. When exact counting is enabled and requested, `Content-Range` uses `*` for the total. Estimated counting can return a planner estimate when no opaque row or vector filter suppresses it. A native hit total is not an authorized record count. See [count preferences](../rest/querying.md#pagination-and-total-count) for the default downgrade from exact to estimated counting. For the ordinary REST paging syntax, see [Querying](../rest/querying.md#pagination-and-total-count). The JSON Operations API uses `search_by_conditions` with `limit` and `offset` rather than REST URL syntax. The same per-request behavior and work limits apply. - -## Errors and recovery - -| Situation | HTTP status | Action | -| --------------------------------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------ | -| Fetched-hit budget exhausted | 400 | Narrow the query, reduce the offset, or raise `maxRawHitsPerRequest`. | -| Execution budget exhausted | 503 | Narrow the query or adjust `requestBudgetMilliseconds`; retrying the same expensive query can hit the same limit. | -| Index rebuilding or unavailable | 503 | Wait for the local index to become ready before retrying. | -| Application worker started before native runtime configuration was captured | 500 | `FullTextWorkerInitializationError` is non-retryable in that worker. Recreate the worker after Harper initializes. | - -`INDEX_REBUILDING` and `DERIVED_INDEX_LAGGING` identify retryable readiness or coverage failures. A native busy response also returns 503; reduce concurrency or retry with backoff. HTTP status alone does not distinguish these from an execution-budget failure. diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index dfe2cf9d1..9cb42f545 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -79,6 +79,36 @@ When a secondary index is still being built — for example, just after a new `@ This is a transient condition — the index finishes building in the background, after which the same query succeeds. Clients should retry (with backoff) rather than treating it as an empty result or a permanent failure. On the operations API the error body includes `code` and `retryable`, so callers can branch on `error.code === 'INDEX_REBUILDING'` or `error.retryable`. +## Full-Text Search Work and Paging + + + + + +Full-text and autocomplete queries use Harper's existing `limit` and `offset` options. Each page runs a new query: changed records, permissions or index contents can cause records to repeat or be omitted between pages. There is no retained search across requests. See [REST paging and counts](../rest/querying.md#pagination-and-total-count) for URL syntax and count behavior. + +Within one request, Harper uses one native reader to fetch ranked batches. It checks source versions and record filters before counting records toward the requested offset and limit. Prefix and fuzzy-prefix searches can continue past the initial autocomplete batch. The native search window is not a total-result cutoff for bounded Harper queries. + +Use a bounded `limit` for large result sets. An unbounded search that exceeds the native result window returns an error. `offset + limit` must not exceed `storage.fullText.maxRawHitsPerRequest`; replay and rejected hits can exhaust that budget earlier. Offset requests revisit earlier matches and can retain up to `offset + limit` accepted records. Raising work limits can increase CPU, reads and memory use. + +The fetched-hit budget counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. The adapter does not return a shortened result as if the search had completed. If a custom resource has already started streaming a response, a later query error ends that stream instead of changing its HTTP status. An exhausted search returns its remaining eligible records, which may be empty. + +Configure the two work limits and six native process budgets under [`storage.fullText`](../configuration/options.md#storage). Work-limit changes through `set_configuration` apply to new queries. The execution budget is not a hard response-time guarantee: reader acquisition, query construction and some native operations can take additional time. The full-text condition's `waitForIndexMilliseconds` controls freshness waiting separately. + +### Native runtime configuration + +Harper captures native process budgets before loading application components, or on an earlier main-thread native load or managed worker spawn. Application-created workers inherit that configuration. Create workers from a Harper application component after initialization. A worker created before capture fails on its first native full-text load with `FullTextWorkerInitializationError` (500, non-retryable); recreate that worker after Harper initializes. + +Replacement workers keep the captured budgets after a live configuration change. Apply new native runtime budgets with a full Harper process restart; `restart_service` only replaces workers. + +### Readiness and recovery + +Full-text indexes require audit logging. A format-incompatible native upgrade rebuilds the local index from stored records, and queries return 503 while it rebuilds. Harper 5.4 uses `@harperfast/fulltext` 0.6.0. + +`INDEX_REBUILDING` and `DERIVED_INDEX_LAGGING` identify retryable readiness or coverage failures. Retry with backoff while the index makes progress. Lost retained transaction history can require a rebuild; retries alone do not repair an index that cannot catch up. Check index readiness and logs if it stays unavailable. A native busy response also returns 503; reduce concurrency or retry with backoff. For an execution-budget failure, narrow the query, reduce the offset or adjust the work budget instead of repeatedly submitting the same expensive query. + +Native page, response-byte and highlight limits still apply. Fulltext 0.6.0 also limits each analyzed query leaf to 64 terms and the compiled query to 256 clauses. Raising Harper's work budgets does not raise those native limits. + ## Relationships and Joins Harper supports relationship-based queries that join data across tables. See [Schema documentation](../database/schema.md) for how to define relationships. diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index cf44bb715..76767d8e2 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1035,7 +1035,7 @@ Array of condition objects to filter records. Each condition: | `conditions` | Nested conditions array | | `operator` | `and` (default) or `or` for the nested `conditions` | -Full-text conditions use `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, or `matches_fuzzy_prefix` against a declared full-text index. See [Full-text paging](./full-text-paging.md) for a `Table.search()` example and work limits. +Full-text conditions use `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, or `matches_fuzzy_prefix` against a declared full-text index. See [Full-text query execution](./query-optimization.md#full-text-search-work-and-paging) for a `Table.search()` example and work limits. Example with nested conditions: diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 4ff839e56..af4adf9d9 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -55,7 +55,7 @@ Harper uses [FIQL](https://datatracker.ietf.org/doc/html/draft-nottingham-atompu | `=`, `===` | Strict equality (no type conversion) | | `!==` | Strict inequality (no type conversion) | -Full-text indexes support `=matches=`, `=matches_all=`, `=matches_phrase=`, `=matches_prefix=`, `=matches_fuzzy=`, and `=matches_fuzzy_prefix=`. These use the full-text index rather than ordinary string comparisons. See [Full-text paging](../resources/full-text-paging.md). +Full-text indexes support `=matches=`, `=matches_all=`, `=matches_phrase=`, `=matches_prefix=`, `=matches_fuzzy=`, and `=matches_fuzzy_prefix=`. These use the full-text index rather than ordinary string comparisons. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). **Examples**: @@ -177,7 +177,7 @@ GET /Product/?sort(+rating,-price) ## Pagination and Total Count -Full-text and autocomplete queries use the same paging syntax. See [Full-text paging](../resources/full-text-paging.md) for search work limits and behavior when records change between requests. +Full-text and autocomplete queries use the same paging syntax. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for search work limits and behavior when records change between requests. @@ -222,7 +222,7 @@ The response status is always `200` — `Content-Range` is informational (Harper The total is reported as `*` (for example `Content-Range: items 0-24/*`) when it cannot be produced — an exact scan that hits its guardrail (counting the tail past the requested page is bounded by a 1,000,000-row cap and a ~1-second budget, and abandons the total rather than truncating the page), or a query with no cardinality estimate (for example a `!=` or `=ct=` (contains) condition). `Preference-Applied` still echoes the requested mode, so an unavailable total (`.../*`) is distinct from a request that asked for no count. -Full-text and autocomplete queries cannot provide an exact total. An estimated count may use the planner estimate when the query has no opaque row or vector filter. See [Full-text paging](../resources/full-text-paging.md#request-a-page). +Full-text and autocomplete queries cannot provide an exact total. An estimated count may use the planner estimate when the query has no opaque row or vector filter. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). ### Enabling exact counts diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 122cf5546..b0b203621 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -70,4 +70,4 @@ The new `threads.v8Flags` setting takes a V8 flag, or a list of them, and applie Full-text and autocomplete queries use the existing `limit` and `offset` paging options. Harper reads additional native batches when needed after filtering, so the initial native result window no longer silently cuts off eligible results. The new `storage.fullText.maxRawHitsPerRequest` and `storage.fullText.requestBudgetMilliseconds` settings bound request work. Requests that exceed those budgets return an error. Separate page requests run independently and can see changes to records or index contents. -See [Full-text paging](/reference/v5/resources/full-text-paging) for examples, limits and upgrade considerations. +See [Full-text query execution](/reference/v5/resources/query-optimization#full-text-search-work-and-paging) and [storage configuration](/reference/v5/configuration/options#storage) for limits, restart requirements and recovery guidance. diff --git a/sidebarsReference.ts b/sidebarsReference.ts index a2306e408..f997c1810 100644 --- a/sidebarsReference.ts +++ b/sidebarsReference.ts @@ -87,7 +87,6 @@ const sidebars: SidebarsConfig = { id: 'resources/query-optimization', label: 'Query Optimization', }, - { type: 'doc', id: 'resources/full-text-paging', label: 'Full-text Paging' }, ], }, { From f5f9a01feaee19f1f88168fc43ae464d8e154125 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 22:35:36 -0600 Subject: [PATCH 52/78] Document independent full-text result and work budgets --- reference/configuration/operations.md | 2 +- reference/configuration/options.md | 3 ++- reference/resources/query-optimization.md | 4 ++-- 3 files changed, 5 insertions(+), 4 deletions(-) diff --git a/reference/configuration/operations.md b/reference/configuration/operations.md index 177935496..eb36e0c23 100644 --- a/reference/configuration/operations.md +++ b/reference/configuration/operations.md @@ -18,7 +18,7 @@ For the full list of configurable options, see [Configuration Options](./options Modifies one or more Harper configuration parameters. **Most changes require a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`). A change to `replication.threads` takes effect only on a full `restart`. -Full-text request work limits (`storage.fullText.maxRawHitsPerRequest` and `storage.fullText.requestBudgetMilliseconds`) apply to new searches without a restart. Changes to `storage.fullText.runtime` require a full Harper process restart; `restart_service` does not replace the native process budget. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). +Full-text request work limits (`storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds`) apply to new searches without a restart. Changes to `storage.fullText.runtime` require a full Harper process restart; `restart_service` does not replace the native process budget. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). `operation` _(required)_ — must be `set_configuration` diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 7ab42326b..5db0d4fd8 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -313,7 +313,8 @@ storage: - `engine` — The database storage engine to use for new databases. Currently supported engines are `rocksdb` and `lmdb`. The default is `rocksdb`. Existing databases will use the engine that was used when they were created. - `fullText` — Full-text search work and native process limits. Sub-options: - - `maxRawHitsPerRequest` — Maximum fetched hits per request, including replay when switching to native traversal; default `10000`. + - `maxRawHitsPerRequest` — Maximum fetched hits per request; default `10000`. + - `maxResultsPerRequest` — Maximum retained offset plus limit, independent of fetched-hit work; default `10000`. - `requestBudgetMilliseconds` — Execution budget across native batches, source checks and highlights, capped by the native execution limit; default `5000` ms. Reader acquisition and freshness waiting have separate bounds. - `runtime.maxResidentIndexes` — Maximum resident native indexes; default `128`. - `runtime.maxIndexingThreads` — Maximum native indexing threads; default `128`. diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index 9cb42f545..253f65b54 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -89,11 +89,11 @@ Full-text and autocomplete queries use Harper's existing `limit` and `offset` op Within one request, Harper uses one native reader to fetch ranked batches. It checks source versions and record filters before counting records toward the requested offset and limit. Prefix and fuzzy-prefix searches can continue past the initial autocomplete batch. The native search window is not a total-result cutoff for bounded Harper queries. -Use a bounded `limit` for large result sets. An unbounded search that exceeds the native result window returns an error. `offset + limit` must not exceed `storage.fullText.maxRawHitsPerRequest`; replay and rejected hits can exhaust that budget earlier. Offset requests revisit earlier matches and can retain up to `offset + limit` accepted records. Raising work limits can increase CPU, reads and memory use. +Use a bounded `limit` for large result sets. An unbounded search that exceeds the native result window returns an error. `offset + limit` must not exceed either `storage.fullText.maxResultsPerRequest` or `storage.fullText.maxRawHitsPerRequest`; rejected hits can exhaust the raw-work budget earlier. Raising only the raw-hit budget allows more filtering work without raising the retained-record target. Offset requests revisit earlier matches and can retain up to `offset + limit` accepted records. Raising work limits can increase CPU, reads and memory use. The fetched-hit budget counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. The adapter does not return a shortened result as if the search had completed. If a custom resource has already started streaming a response, a later query error ends that stream instead of changing its HTTP status. An exhausted search returns its remaining eligible records, which may be empty. -Configure the two work limits and six native process budgets under [`storage.fullText`](../configuration/options.md#storage). Work-limit changes through `set_configuration` apply to new queries. The execution budget is not a hard response-time guarantee: reader acquisition, query construction and some native operations can take additional time. The full-text condition's `waitForIndexMilliseconds` controls freshness waiting separately. +Configure the three request limits and six native process budgets under [`storage.fullText`](../configuration/options.md#storage). Work-limit changes through `set_configuration` apply to new queries. The execution budget is not a hard response-time guarantee: reader acquisition, query construction and some native operations can take additional time. The full-text condition's `waitForIndexMilliseconds` controls freshness waiting separately. ### Native runtime configuration From 0444a14415fd967df8f40cbbc9498399e50db269 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 22:43:05 -0600 Subject: [PATCH 53/78] Clarify query rejection and documentation pointers --- reference/resources/query-optimization.md | 6 +++--- reference/resources/resource-api.md | 2 +- reference/rest/querying.md | 6 ++++-- 3 files changed, 8 insertions(+), 6 deletions(-) diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index 253f65b54..3aba628f9 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -87,9 +87,9 @@ This is a transient condition — the index finishes building in the background, Full-text and autocomplete queries use Harper's existing `limit` and `offset` options. Each page runs a new query: changed records, permissions or index contents can cause records to repeat or be omitted between pages. There is no retained search across requests. See [REST paging and counts](../rest/querying.md#pagination-and-total-count) for URL syntax and count behavior. -Within one request, Harper uses one native reader to fetch ranked batches. It checks source versions and record filters before counting records toward the requested offset and limit. Prefix and fuzzy-prefix searches can continue past the initial autocomplete batch. The native search window is not a total-result cutoff for bounded Harper queries. +Within one request, Harper uses one native reader to fetch ranked batches. It checks source versions and record filters before counting records toward the requested offset and limit. Prefix and fuzzy-prefix searches can return more than the native ordinary autocomplete window. The native search window is not a total-result cutoff for bounded Harper queries. -Use a bounded `limit` for large result sets. An unbounded search that exceeds the native result window returns an error. `offset + limit` must not exceed either `storage.fullText.maxResultsPerRequest` or `storage.fullText.maxRawHitsPerRequest`; rejected hits can exhaust the raw-work budget earlier. Raising only the raw-hit budget allows more filtering work without raising the retained-record target. Offset requests revisit earlier matches and can retain up to `offset + limit` accepted records. Raising work limits can increase CPU, reads and memory use. +Use a bounded `limit` for large result sets. An unbounded search that exceeds its native window or configured result/work budget throws 400; use a bounded query. `offset + limit` must not exceed either `storage.fullText.maxResultsPerRequest` or `storage.fullText.maxRawHitsPerRequest`; rejected hits can exhaust the raw-work budget earlier. Raising only the raw-hit budget allows more filtering work without raising the retained-record target. Offset requests revisit earlier matches and can retain up to `offset + limit` accepted records. Raising work limits can increase CPU, reads and memory use. The fetched-hit budget counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. The adapter does not return a shortened result as if the search had completed. If a custom resource has already started streaming a response, a later query error ends that stream instead of changing its HTTP status. An exhausted search returns its remaining eligible records, which may be empty. @@ -105,7 +105,7 @@ Replacement workers keep the captured budgets after a live configuration change. Full-text indexes require audit logging. A format-incompatible native upgrade rebuilds the local index from stored records, and queries return 503 while it rebuilds. Harper 5.4 uses `@harperfast/fulltext` 0.6.0. -`INDEX_REBUILDING` and `DERIVED_INDEX_LAGGING` identify retryable readiness or coverage failures. Retry with backoff while the index makes progress. Lost retained transaction history can require a rebuild; retries alone do not repair an index that cannot catch up. Check index readiness and logs if it stays unavailable. A native busy response also returns 503; reduce concurrency or retry with backoff. For an execution-budget failure, narrow the query, reduce the offset or adjust the work budget instead of repeatedly submitting the same expensive query. +`INDEX_REBUILDING` and `DERIVED_INDEX_LAGGING` identify retryable readiness or coverage failures. Retry with backoff while the index makes progress. Transaction history retained under [`logging.auditRetention`](../logging/configuration.md#loggingauditretention) (default `3d`) is used for catch-up. Lost history can require a rebuild; retries alone do not repair an index that cannot catch up. Check index readiness and logs if it stays unavailable. Budget and native-busy failures do not carry those readiness codes. A native busy response also returns 503; reduce concurrency or retry with backoff. For an execution-budget failure, narrow the query, reduce the offset or adjust the work budget instead of repeatedly submitting the same expensive query. Native page, response-byte and highlight limits still apply. Fulltext 0.6.0 also limits each analyzed query leaf to 64 terms and the compiled query to 256 clauses. Raising Harper's work budgets does not raise those native limits. diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 76767d8e2..202bf521c 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1035,7 +1035,7 @@ Array of condition objects to filter records. Each condition: | `conditions` | Nested conditions array | | `operator` | `and` (default) or `or` for the nested `conditions` | -Full-text conditions use `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, or `matches_fuzzy_prefix` against a declared full-text index. See [Full-text query execution](./query-optimization.md#full-text-search-work-and-paging) for a `Table.search()` example and work limits. +Full-text conditions use `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, or `matches_fuzzy_prefix` against a declared full-text index. See [Full-text query execution](./query-optimization.md#full-text-search-work-and-paging) for paging costs and work limits. Example with nested conditions: diff --git a/reference/rest/querying.md b/reference/rest/querying.md index af4adf9d9..44201c153 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -177,14 +177,16 @@ GET /Product/?sort(+rating,-price) ## Pagination and Total Count -Full-text and autocomplete queries use the same paging syntax. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for search work limits and behavior when records change between requests. - Use `limit(start,end)` to page through a collection, and opt in to a total match count with the `Prefer` request header (`Prefer: count=exact`) so a client can render pagination (for example "1-25 of 1,234") without a second request. Counting is opt-in: without the header, no count is computed and no count headers are returned. It applies only to `GET`/`HEAD` requests and requires a bounded page — the `limit()` must be a non-negative integer no larger than **10,000**, and the requested window (offset + limit) no larger than **1,000,000**. A request with no `limit()`, one outside those bounds, or a non-numeric one is served normally with no count headers, since counting an unbounded page would defeat the point of paging. +Full-text and autocomplete queries use the same paging syntax. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for search work limits and behavior when records change between requests. + +The count-header guardrails do not override query-specific limits such as the full-text work budget. + ### Requesting a count Send a `Prefer` header on a `GET` (or `HEAD`) request to a collection: From a5f332a64f99ab66c0923334d139a161af0877d5 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 22:48:04 -0600 Subject: [PATCH 54/78] Account for bounded replay in the raw-hit budget --- reference/configuration/options.md | 2 +- reference/resources/query-optimization.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 5db0d4fd8..d8cccbebb 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -313,7 +313,7 @@ storage: - `engine` — The database storage engine to use for new databases. Currently supported engines are `rocksdb` and `lmdb`. The default is `rocksdb`. Existing databases will use the engine that was used when they were created. - `fullText` — Full-text search work and native process limits. Sub-options: - - `maxRawHitsPerRequest` — Maximum fetched hits per request; default `10000`. + - `maxRawHitsPerRequest` — Maximum fetched hits per request, including bounded prefix replay when traversal is needed; default `10000`. - `maxResultsPerRequest` — Maximum retained offset plus limit, independent of fetched-hit work; default `10000`. - `requestBudgetMilliseconds` — Execution budget across native batches, source checks and highlights, capped by the native execution limit; default `5000` ms. Reader acquisition and freshness waiting have separate bounds. - `runtime.maxResidentIndexes` — Maximum resident native indexes; default `128`. diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index 3aba628f9..c01f6a739 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -89,9 +89,9 @@ Full-text and autocomplete queries use Harper's existing `limit` and `offset` op Within one request, Harper uses one native reader to fetch ranked batches. It checks source versions and record filters before counting records toward the requested offset and limit. Prefix and fuzzy-prefix searches can return more than the native ordinary autocomplete window. The native search window is not a total-result cutoff for bounded Harper queries. -Use a bounded `limit` for large result sets. An unbounded search that exceeds its native window or configured result/work budget throws 400; use a bounded query. `offset + limit` must not exceed either `storage.fullText.maxResultsPerRequest` or `storage.fullText.maxRawHitsPerRequest`; rejected hits can exhaust the raw-work budget earlier. Raising only the raw-hit budget allows more filtering work without raising the retained-record target. Offset requests revisit earlier matches and can retain up to `offset + limit` accepted records. Raising work limits can increase CPU, reads and memory use. +Use a bounded `limit` for large result sets. An unbounded search that exceeds its native window or configured result/work budget throws 400; use a bounded query. `offset + limit` must not exceed either `storage.fullText.maxResultsPerRequest` or `storage.fullText.maxRawHitsPerRequest`; replayed or rejected hits can exhaust the raw-work budget earlier. Raising only the raw-hit budget allows more filtering work without raising the retained-record target. Offset requests revisit earlier matches and can retain up to `offset + limit` accepted records. Raising work limits can increase CPU, reads and memory use. -The fetched-hit budget counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. The adapter does not return a shortened result as if the search had completed. If a custom resource has already started streaming a response, a later query error ends that stream instead of changing its HTTP status. An exhausted search returns its remaining eligible records, which may be empty. +The fetched-hit budget includes the initial batch replay when Harper switches to native traversal. It counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. The adapter does not return a shortened result as if the search had completed. If a custom resource has already started streaming a response, a later query error ends that stream instead of changing its HTTP status. An exhausted search returns its remaining eligible records, which may be empty. Configure the three request limits and six native process budgets under [`storage.fullText`](../configuration/options.md#storage). Work-limit changes through `set_configuration` apply to new queries. The execution budget is not a hard response-time guarantee: reader acquisition, query construction and some native operations can take additional time. The full-text condition's `waitForIndexMilliseconds` controls freshness waiting separately. From 481dfb09d9b327fbd9f728a0c26fe3a3052c62fc Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 22:49:33 -0600 Subject: [PATCH 55/78] Make full-text failure recovery and native bounds explicit --- reference/configuration/operations.md | 4 +++- reference/resources/query-optimization.md | 12 ++++++++++-- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/reference/configuration/operations.md b/reference/configuration/operations.md index eb36e0c23..11c0dee50 100644 --- a/reference/configuration/operations.md +++ b/reference/configuration/operations.md @@ -44,10 +44,12 @@ A replicated call sends the exact same values to every node. Do not include node ```json { - "message": "Configuration successfully set. You must restart HarperDB for new config settings to take effect." + "message": "Configuration successfully set. You must restart Harper for new config settings to take effect." } ``` +The response uses a generic restart message. The full-text request limits listed above apply to new searches without a restart. + ### Replicated body ```json diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index c01f6a739..64229296b 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -93,7 +93,7 @@ Use a bounded `limit` for large result sets. An unbounded search that exceeds it The fetched-hit budget includes the initial batch replay when Harper switches to native traversal. It counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. The adapter does not return a shortened result as if the search had completed. If a custom resource has already started streaming a response, a later query error ends that stream instead of changing its HTTP status. An exhausted search returns its remaining eligible records, which may be empty. -Configure the three request limits and six native process budgets under [`storage.fullText`](../configuration/options.md#storage). Work-limit changes through `set_configuration` apply to new queries. The execution budget is not a hard response-time guarantee: reader acquisition, query construction and some native operations can take additional time. The full-text condition's `waitForIndexMilliseconds` controls freshness waiting separately. +Configure the three request limits and six native process budgets under [`storage.fullText`](../configuration/options.md#storage). Work-limit changes through `set_configuration` apply to new queries. The execution budget is not a hard response-time guarantee: reader acquisition, query construction and some native operations can take additional time. Freshness waiting and reader acquisition use separate bounds. ### Native runtime configuration @@ -105,9 +105,17 @@ Replacement workers keep the captured budgets after a live configuration change. Full-text indexes require audit logging. A format-incompatible native upgrade rebuilds the local index from stored records, and queries return 503 while it rebuilds. Harper 5.4 uses `@harperfast/fulltext` 0.6.0. +| Failure | HTTP status | Code and retry guidance | +| -------------------------------------------------- | ----------- | -------------------------------------------------------------------------------------------------------- | +| Raw-hit or retained-result budget | 400 | No readiness code. Narrow the query or adjust the named budget. | +| Execution budget | 503 | No readiness code. Narrow the query; the same expensive query can fail again. | +| Native busy | 503 | No readiness code. Reduce concurrency or retry with backoff. | +| Index readiness or coverage | 503 | `INDEX_REBUILDING` or `DERIVED_INDEX_LAGGING`, with `retryable: true`. | +| Worker loaded native search before runtime capture | 500 | `FullTextWorkerInitializationError`, with `retryable: false`; recreate that worker after initialization. | + `INDEX_REBUILDING` and `DERIVED_INDEX_LAGGING` identify retryable readiness or coverage failures. Retry with backoff while the index makes progress. Transaction history retained under [`logging.auditRetention`](../logging/configuration.md#loggingauditretention) (default `3d`) is used for catch-up. Lost history can require a rebuild; retries alone do not repair an index that cannot catch up. Check index readiness and logs if it stays unavailable. Budget and native-busy failures do not carry those readiness codes. A native busy response also returns 503; reduce concurrency or retry with backoff. For an execution-budget failure, narrow the query, reduce the offset or adjust the work budget instead of repeatedly submitting the same expensive query. -Native page, response-byte and highlight limits still apply. Fulltext 0.6.0 also limits each analyzed query leaf to 64 terms and the compiled query to 256 clauses. Raising Harper's work budgets does not raise those native limits. +Native page, response-byte and highlight limits still apply. Fulltext 0.6.0 also limits each analyzed query leaf to 64 terms and the compiled query to 256 clauses. Raising Harper's work budgets does not raise those native limits. See the [published native limits](https://github.com/HarperFast/fulltext/blob/v0.6.0/docs/limits.md) for page, byte and highlight bounds. ## Relationships and Joins From af0ee1a6130703f23bb27129d73e9369759fbf63 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 22:56:32 -0600 Subject: [PATCH 56/78] Clarify both request ceilings and link storage tuning guidance --- reference/configuration/options.md | 2 +- reference/database/storage-tuning.md | 6 ++++++ reference/resources/query-optimization.md | 2 +- release-notes/v5-lincoln/5.4.md | 2 +- 4 files changed, 9 insertions(+), 3 deletions(-) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index d8cccbebb..5e3424d78 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -314,7 +314,7 @@ storage: - `engine` — The database storage engine to use for new databases. Currently supported engines are `rocksdb` and `lmdb`. The default is `rocksdb`. Existing databases will use the engine that was used when they were created. - `fullText` — Full-text search work and native process limits. Sub-options: - `maxRawHitsPerRequest` — Maximum fetched hits per request, including bounded prefix replay when traversal is needed; default `10000`. - - `maxResultsPerRequest` — Maximum retained offset plus limit, independent of fetched-hit work; default `10000`. + - `maxResultsPerRequest` — Maximum retained offset plus limit. A request must also fit the separate raw-hit budget; default `10000`. - `requestBudgetMilliseconds` — Execution budget across native batches, source checks and highlights, capped by the native execution limit; default `5000` ms. Reader acquisition and freshness waiting have separate bounds. - `runtime.maxResidentIndexes` — Maximum resident native indexes; default `128`. - `runtime.maxIndexingThreads` — Maximum native indexing threads; default `128`. diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index 9cade5b28..1a0b6fd32 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -403,3 +403,9 @@ storage: - [Compaction](./compaction.md) — reclaiming space inside existing database files - [Resource API — `sourcedFrom`](../resources/resource-api.md#sourcedfromresource-options) — caching tables that interact with reclamation - [Database API — `createBlob`](./api.md) — creating blobs that live under `blobPaths` + +### `storage.fullText` + +Type: `object` + +The [storage option reference](../configuration/options.md#storage) lists the full-text request and native process defaults. Raise the fetched-hit budget to allow more filtering work; raising it does not raise the separate retained-result limit. Larger limits can increase CPU, source reads and memory use for every permitted query. Native process budgets also share capacity across indexes, so tune them with concurrent workloads rather than a single query. See [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for paging costs, restart requirements and failure recovery. diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index 64229296b..2a934ab72 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -91,7 +91,7 @@ Within one request, Harper uses one native reader to fetch ranked batches. It ch Use a bounded `limit` for large result sets. An unbounded search that exceeds its native window or configured result/work budget throws 400; use a bounded query. `offset + limit` must not exceed either `storage.fullText.maxResultsPerRequest` or `storage.fullText.maxRawHitsPerRequest`; replayed or rejected hits can exhaust the raw-work budget earlier. Raising only the raw-hit budget allows more filtering work without raising the retained-record target. Offset requests revisit earlier matches and can retain up to `offset + limit` accepted records. Raising work limits can increase CPU, reads and memory use. -The fetched-hit budget includes the initial batch replay when Harper switches to native traversal. It counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. The adapter does not return a shortened result as if the search had completed. If a custom resource has already started streaming a response, a later query error ends that stream instead of changing its HTTP status. An exhausted search returns its remaining eligible records, which may be empty. +The fetched-hit budget includes the initial batch replay when Harper switches to native traversal. It counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. The adapter does not return a shortened result as if the search had completed. If a custom resource has already started streaming a response, a later query error ends that stream instead of changing its HTTP status. When there are no further native matches, Harper returns the eligible records found, which may be an empty result. This differs from reaching a budget while more matches remain. Configure the three request limits and six native process budgets under [`storage.fullText`](../configuration/options.md#storage). Work-limit changes through `set_configuration` apply to new queries. The execution budget is not a hard response-time guarantee: reader acquisition, query construction and some native operations can take additional time. Freshness waiting and reader acquisition use separate bounds. diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index b0b203621..cd676e60a 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -68,6 +68,6 @@ The new `threads.v8Flags` setting takes a V8 flag, or a list of them, and applie ### Paging and Search Work Limits -Full-text and autocomplete queries use the existing `limit` and `offset` paging options. Harper reads additional native batches when needed after filtering, so the initial native result window no longer silently cuts off eligible results. The new `storage.fullText.maxRawHitsPerRequest` and `storage.fullText.requestBudgetMilliseconds` settings bound request work. Requests that exceed those budgets return an error. Separate page requests run independently and can see changes to records or index contents. +Full-text and autocomplete queries use the existing `limit` and `offset` paging options. Harper reads additional native batches when needed after filtering, so the initial native result window no longer silently cuts off eligible results. The new `storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds` settings bound fetched hits, retained results and execution time. Requests that exceed those budgets return an error. Separate page requests run independently and can see changes to records or index contents. See [Full-text query execution](/reference/v5/resources/query-optimization#full-text-search-work-and-paging) and [storage configuration](/reference/v5/configuration/options#storage) for limits, restart requirements and recovery guidance. From 518f4ff7018b1439e8622c8eb5860f549257518e Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 22:57:00 -0600 Subject: [PATCH 57/78] Match the documented audit default to the shipped configuration --- reference/configuration/options.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 5e3424d78..37da9eeb3 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -232,7 +232,7 @@ logging: level: warn root: ~/hdb/log stdStreams: false - auditLog: false + auditLog: true rotation: interval: 1D maxSize: 100M @@ -244,7 +244,7 @@ logging: - `path` — Explicit log file path (overrides `root`) - `stdStreams` — Write to stdout/stderr; _Default_: `false` - `console` — Include `console.*` output; _Default_: `true` -- `auditLog` — Enable table transaction audit logging; _Default_: `false` +- `auditLog` — Enable table transaction audit logging; _Default_: `true` - `auditRetention` — Audit log retention duration; _Default_: `3d` - `external` — Logging for components using the logger API; sub-options: `level`, `path` - `rotation.enabled` / `rotation.compress` / `rotation.interval` / `rotation.maxSize` / `rotation.path` — Log file rotation (activates when `interval` or `maxSize` is set) From 72f5e4e415070e512b6513eb85d910e3c5b264de Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 23:00:18 -0600 Subject: [PATCH 58/78] Use existing worker restarts to apply full-text request limits --- reference/configuration/operations.md | 4 ++-- reference/configuration/options.md | 2 +- reference/configuration/overview.md | 2 +- reference/operations-api/operations.md | 2 +- reference/resources/query-optimization.md | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/reference/configuration/operations.md b/reference/configuration/operations.md index 11c0dee50..f49beffad 100644 --- a/reference/configuration/operations.md +++ b/reference/configuration/operations.md @@ -18,7 +18,7 @@ For the full list of configurable options, see [Configuration Options](./options Modifies one or more Harper configuration parameters. **Most changes require a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`). A change to `replication.threads` takes effect only on a full `restart`. -Full-text request work limits (`storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds`) apply to new searches without a restart. Changes to `storage.fullText.runtime` require a full Harper process restart; `restart_service` does not replace the native process budget. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). +Full-text request work limits (`storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds`) take effect in HTTP workers after `restart_service` or a full restart. Changes to `storage.fullText.runtime` require a full Harper process restart; `restart_service` does not replace the native process budget. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). `operation` _(required)_ — must be `set_configuration` @@ -48,7 +48,7 @@ A replicated call sends the exact same values to every node. Do not include node } ``` -The response uses a generic restart message. The full-text request limits listed above apply to new searches without a restart. +For full-text request limits, restarting the HTTP workers with `restart_service` is sufficient; the native process budgets require a full restart. ### Replicated body diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 37da9eeb3..97108aab4 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -323,7 +323,7 @@ storage: - `runtime.maxQueuedBytes` — Native queued-write admission budget; default `8589934592` (8 GiB). - `runtime.maxExpensiveSearches` — Maximum concurrent expensive native searches; default `8`. - The runtime budgets apply together and are not hard bounds on total process memory. Work-limit changes through `set_configuration` apply to new searches; runtime-budget changes require a full Harper process restart. Use the usual underscore-separated parameter names, such as `storage_fullText_maxRawHitsPerRequest`. See [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for costs and failure recovery. + The runtime budgets apply together and are not hard bounds on total process memory. After changing request limits through `set_configuration`, use `restart_service` or restart Harper to apply them to HTTP workers. Native runtime-budget changes require a full Harper process restart. Use the usual underscore-separated parameter names, such as `storage_fullText_maxRawHitsPerRequest`. See [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for costs and failure recovery. - `writeAsync` — Disable disk sync for higher throughput (**disables durability guarantees**); _Default_: `false` - `caching` — In-memory record caching; _Default_: `true` diff --git a/reference/configuration/overview.md b/reference/configuration/overview.md index 309ef0726..3358fc228 100644 --- a/reference/configuration/overview.md +++ b/reference/configuration/overview.md @@ -21,7 +21,7 @@ For a complete reference of all available configuration options, see [Configurat ## The Configuration File -To change a configuration value, edit `harper-config.yaml` and save. **Restart Harper to apply file edits.** To change full-text request work limits without a restart, use [`set_configuration`](./operations.md#set-configuration). Native full-text runtime budgets still require a full process restart. +To change a configuration value, edit `harper-config.yaml` and save. **Restart Harper to apply file edits.** For full-text request limits, `restart_service` reloads the HTTP workers. Native full-text runtime budgets require a full process restart. Configuration keys use camelCase (e.g. `operationsApi`). Nested keys use dot notation conceptually (e.g. `operationsApi.network.port`). diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index b0179399a..7a6c80d4a 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -1686,7 +1686,7 @@ Detailed documentation: [Configuration Overview](../configuration/overview.md) ### `set_configuration` -Updates configuration parameters in `harper-config.yaml`. Most changes require a restart (`restart` or `restart_service`) to take effect. Full-text request work limits apply to new searches without a restart; changes to the native process budgets require a full `restart`. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). +Updates configuration parameters in `harper-config.yaml`. Most changes require a restart (`restart` or `restart_service`) to take effect. Full-text request limits apply to HTTP workers after `restart_service` or a full restart; changes to native process budgets require a full `restart`. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). Supports `"replicated": true` to apply the same change to all cluster nodes in one call; per-node outcomes are returned in the response's `replicated` array. Only send cluster-appropriate parameters when replicating — node-local parameters (ports, `node.hostname`, file paths, TLS material, `replication.hostname`/`url`/`routes`) would overwrite every peer's local values. To apply the change cluster-wide, follow with `restart_service` using `"replicated": true` (which restarts nodes one at a time); `restart_service` restarts only worker threads, though, so a change to [`threads.v8Flags`](../configuration/options.md#applying-v8-flags) takes effect only after a `restart` sent to each node in turn (`restart` does not replicate). See [Configuration Operations](../configuration/operations.md#set-configuration) for details. diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index 2a934ab72..d86ba21c3 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -93,7 +93,7 @@ Use a bounded `limit` for large result sets. An unbounded search that exceeds it The fetched-hit budget includes the initial batch replay when Harper switches to native traversal. It counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. The adapter does not return a shortened result as if the search had completed. If a custom resource has already started streaming a response, a later query error ends that stream instead of changing its HTTP status. When there are no further native matches, Harper returns the eligible records found, which may be an empty result. This differs from reaching a budget while more matches remain. -Configure the three request limits and six native process budgets under [`storage.fullText`](../configuration/options.md#storage). Work-limit changes through `set_configuration` apply to new queries. The execution budget is not a hard response-time guarantee: reader acquisition, query construction and some native operations can take additional time. Freshness waiting and reader acquisition use separate bounds. +Configure the three request limits and six native process budgets under [`storage.fullText`](../configuration/options.md#storage). After changing request limits through `set_configuration`, use `restart_service` or restart Harper to apply them to HTTP workers. The execution budget is not a hard response-time guarantee: reader acquisition, query construction and some native operations can take additional time. Freshness waiting and reader acquisition use separate bounds. ### Native runtime configuration From ee1862d1f468f035658bfeb64d4181405f459d0a Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Wed, 21 Oct 2026 00:01:37 -0600 Subject: [PATCH 59/78] Clarify full-text admission limits and error recovery --- reference/configuration/operations.md | 2 -- reference/configuration/options.md | 10 +++++----- reference/configuration/overview.md | 2 +- reference/database/storage-tuning.md | 16 ++++++++++------ reference/database/transaction.md | 2 +- reference/operations-api/operations.md | 2 +- reference/resources/query-optimization.md | 12 +++++++++--- 7 files changed, 27 insertions(+), 19 deletions(-) diff --git a/reference/configuration/operations.md b/reference/configuration/operations.md index f49beffad..8da1c0403 100644 --- a/reference/configuration/operations.md +++ b/reference/configuration/operations.md @@ -48,8 +48,6 @@ A replicated call sends the exact same values to every node. Do not include node } ``` -For full-text request limits, restarting the HTTP workers with `restart_service` is sufficient; the native process budgets require a full restart. - ### Replicated body ```json diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 97108aab4..e1fd7cbe7 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -317,13 +317,13 @@ storage: - `maxResultsPerRequest` — Maximum retained offset plus limit. A request must also fit the separate raw-hit budget; default `10000`. - `requestBudgetMilliseconds` — Execution budget across native batches, source checks and highlights, capped by the native execution limit; default `5000` ms. Reader acquisition and freshness waiting have separate bounds. - `runtime.maxResidentIndexes` — Maximum resident native indexes; default `128`. - - `runtime.maxIndexingThreads` — Maximum native indexing threads; default `128`. - - `runtime.maxSearchThreads` — Maximum native search threads; default `256`. - - `runtime.maxWriterMemoryBytes` — Native writer-memory admission budget; default `4294967296` (4 GiB). - - `runtime.maxQueuedBytes` — Native queued-write admission budget; default `8589934592` (8 GiB). + - `runtime.maxIndexingThreads` — Maximum aggregate indexing-thread reservations across resident native indexes; default `128`. + - `runtime.maxSearchThreads` — Maximum aggregate search-thread reservations across resident native indexes; default `256`. + - `runtime.maxWriterMemoryBytes` — Aggregate writer-memory reservations across resident native indexes; default `4294967296` (4 GiB). + - `runtime.maxQueuedBytes` — Aggregate queued-write reservations across resident native indexes; default `8589934592` (8 GiB). - `runtime.maxExpensiveSearches` — Maximum concurrent expensive native searches; default `8`. - The runtime budgets apply together and are not hard bounds on total process memory. After changing request limits through `set_configuration`, use `restart_service` or restart Harper to apply them to HTTP workers. Native runtime-budget changes require a full Harper process restart. Use the usual underscore-separated parameter names, such as `storage_fullText_maxRawHitsPerRequest`. See [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for costs and failure recovery. + The runtime budgets apply together and are not hard bounds on total process memory. See [Set Configuration](./operations.md#set-configuration) for when request and native runtime changes take effect. Use the usual underscore-separated parameter names, such as `storage_fullText_maxRawHitsPerRequest`. See [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for costs and failure recovery. - `writeAsync` — Disable disk sync for higher throughput (**disables durability guarantees**); _Default_: `false` - `caching` — In-memory record caching; _Default_: `true` diff --git a/reference/configuration/overview.md b/reference/configuration/overview.md index 3358fc228..033c8006c 100644 --- a/reference/configuration/overview.md +++ b/reference/configuration/overview.md @@ -21,7 +21,7 @@ For a complete reference of all available configuration options, see [Configurat ## The Configuration File -To change a configuration value, edit `harper-config.yaml` and save. **Restart Harper to apply file edits.** For full-text request limits, `restart_service` reloads the HTTP workers. Native full-text runtime budgets require a full process restart. +To change a configuration value, edit `harper-config.yaml` and save. **Restart Harper to apply file edits.** See [Set Configuration](./operations.md#set-configuration) for settings that require a full process restart. Configuration keys use camelCase (e.g. `operationsApi`). Nested keys use dot notation conceptually (e.g. `operationsApi.network.port`). diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index 1a0b6fd32..8d22dbc8f 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -395,6 +395,16 @@ storage: compression: true ``` +### `storage.fullText` + + + +Type: `object` + +Defaults: See the [storage option reference](../configuration/options.md#storage). + +The [storage option reference](../configuration/options.md#storage) lists the full-text request and native process defaults. Raise the fetched-hit budget to allow more filtering work; raising it does not raise the separate retained-result limit. Larger limits can increase CPU, source reads and memory use for every permitted query. Native process budgets also share capacity across indexes, so tune them with concurrent workloads rather than a single query. See [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for paging costs, restart requirements and failure recovery. + ## Related - [Configuration Options](../configuration/options.md) — full list of `storage` options @@ -403,9 +413,3 @@ storage: - [Compaction](./compaction.md) — reclaiming space inside existing database files - [Resource API — `sourcedFrom`](../resources/resource-api.md#sourcedfromresource-options) — caching tables that interact with reclamation - [Database API — `createBlob`](./api.md) — creating blobs that live under `blobPaths` - -### `storage.fullText` - -Type: `object` - -The [storage option reference](../configuration/options.md#storage) lists the full-text request and native process defaults. Raise the fetched-hit budget to allow more filtering work; raising it does not raise the separate retained-result limit. Larger limits can increase CPU, source reads and memory use for every permitted query. Native process budgets also share capacity across indexes, so tune them with concurrent workloads rather than a single query. See [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for paging costs, restart requirements and failure recovery. diff --git a/reference/database/transaction.md b/reference/database/transaction.md index baba4612e..82f891561 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -23,7 +23,7 @@ On RocksDB (the default storage engine) the transaction log is physically a sing The transaction log is **enabled by default**. To disable it, set [`logging.auditLog`](../logging/configuration.md) to `false` in `harper-config.yaml` and restart Harper. -> The transaction log is required for real-time messaging (WebSocket and MQTT subscriptions) and replication. Do not disable it if real-time features or replication are in use. +> The transaction log is required for real-time messaging (WebSocket and MQTT subscriptions), replication and full-text indexing. Do not disable it while any of these features are in use. ## Operations diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 7a6c80d4a..8a41f2aeb 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -1686,7 +1686,7 @@ Detailed documentation: [Configuration Overview](../configuration/overview.md) ### `set_configuration` -Updates configuration parameters in `harper-config.yaml`. Most changes require a restart (`restart` or `restart_service`) to take effect. Full-text request limits apply to HTTP workers after `restart_service` or a full restart; changes to native process budgets require a full `restart`. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). +Updates configuration parameters in `harper-config.yaml`. Most changes require a restart (`restart` or `restart_service`) to take effect. See [Set Configuration](../configuration/operations.md#set-configuration) for full-text restart requirements. Supports `"replicated": true` to apply the same change to all cluster nodes in one call; per-node outcomes are returned in the response's `replicated` array. Only send cluster-appropriate parameters when replicating — node-local parameters (ports, `node.hostname`, file paths, TLS material, `replication.hostname`/`url`/`routes`) would overwrite every peer's local values. To apply the change cluster-wide, follow with `restart_service` using `"replicated": true` (which restarts nodes one at a time); `restart_service` restarts only worker threads, though, so a change to [`threads.v8Flags`](../configuration/options.md#applying-v8-flags) takes effect only after a `restart` sent to each node in turn (`restart` does not replicate). See [Configuration Operations](../configuration/operations.md#set-configuration) for details. diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index d86ba21c3..6a0e658f1 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -93,17 +93,17 @@ Use a bounded `limit` for large result sets. An unbounded search that exceeds it The fetched-hit budget includes the initial batch replay when Harper switches to native traversal. It counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. The adapter does not return a shortened result as if the search had completed. If a custom resource has already started streaming a response, a later query error ends that stream instead of changing its HTTP status. When there are no further native matches, Harper returns the eligible records found, which may be an empty result. This differs from reaching a budget while more matches remain. -Configure the three request limits and six native process budgets under [`storage.fullText`](../configuration/options.md#storage). After changing request limits through `set_configuration`, use `restart_service` or restart Harper to apply them to HTTP workers. The execution budget is not a hard response-time guarantee: reader acquisition, query construction and some native operations can take additional time. Freshness waiting and reader acquisition use separate bounds. +Configure the three request limits and six native process budgets under [`storage.fullText`](../configuration/options.md#storage). See [Set Configuration](../configuration/operations.md#set-configuration) for restart requirements. The execution budget is not a hard response-time guarantee: reader acquisition, query construction and some native operations can take additional time. Freshness waiting and reader acquisition use separate bounds. ### Native runtime configuration Harper captures native process budgets before loading application components, or on an earlier main-thread native load or managed worker spawn. Application-created workers inherit that configuration. Create workers from a Harper application component after initialization. A worker created before capture fails on its first native full-text load with `FullTextWorkerInitializationError` (500, non-retryable); recreate that worker after Harper initializes. -Replacement workers keep the captured budgets after a live configuration change. Apply new native runtime budgets with a full Harper process restart; `restart_service` only replaces workers. +Replacement workers keep the captured budgets after a configuration change. See [Set Configuration](../configuration/operations.md#set-configuration) before changing these budgets. ### Readiness and recovery -Full-text indexes require audit logging. A format-incompatible native upgrade rebuilds the local index from stored records, and queries return 503 while it rebuilds. Harper 5.4 uses `@harperfast/fulltext` 0.6.0. +Full-text indexes require audit logging; a table declared with `@table(audit: false)` cannot register a full-text index. A format-incompatible native upgrade rebuilds the local index from stored records, and queries return 503 while it rebuilds. Harper 5.4 uses `@harperfast/fulltext` 0.6.0. | Failure | HTTP status | Code and retry guidance | | -------------------------------------------------- | ----------- | -------------------------------------------------------------------------------------------------------- | @@ -113,6 +113,12 @@ Full-text indexes require audit logging. A format-incompatible native upgrade re | Index readiness or coverage | 503 | `INDEX_REBUILDING` or `DERIVED_INDEX_LAGGING`, with `retryable: true`. | | Worker loaded native search before runtime capture | 500 | `FullTextWorkerInitializationError`, with `retryable: false`; recreate that worker after initialization. | +A native result that cannot fit even a single-hit response returns 413. Other native failures and invalid query configuration can return 500; inspect the error and server logs rather than assuming they are retryable. The table lists common failures, not every possible error. + +For read-after-write queries, `maxIndexLagMilliseconds: 0` requires current coverage. `waitForIndexMilliseconds` lets a query wait for coverage (default `0`, maximum `30000` ms); without an override, the allowed index lag is `3000` ms. These options belong to the full-text condition. + +Concurrent requests each have their own work budget. These limits do not cap the number of concurrent Harper queries or prevent brief event-loop delays during synchronous source reads. Measure deep offsets and selective filters under the expected concurrency before raising budgets. + `INDEX_REBUILDING` and `DERIVED_INDEX_LAGGING` identify retryable readiness or coverage failures. Retry with backoff while the index makes progress. Transaction history retained under [`logging.auditRetention`](../logging/configuration.md#loggingauditretention) (default `3d`) is used for catch-up. Lost history can require a rebuild; retries alone do not repair an index that cannot catch up. Check index readiness and logs if it stays unavailable. Budget and native-busy failures do not carry those readiness codes. A native busy response also returns 503; reduce concurrency or retry with backoff. For an execution-budget failure, narrow the query, reduce the offset or adjust the work budget instead of repeatedly submitting the same expensive query. Native page, response-byte and highlight limits still apply. Fulltext 0.6.0 also limits each analyzed query leaf to 64 terms and the compiled query to 256 clauses. Raising Harper's work budgets does not raise those native limits. See the [published native limits](https://github.com/HarperFast/fulltext/blob/v0.6.0/docs/limits.md) for page, byte and highlight bounds. From ab2ae9791eed3b18a485f9fcdc1120e88fbbef6f Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Wed, 21 Oct 2026 07:21:06 -0600 Subject: [PATCH 60/78] Clarify filtered full-text paging and partial-page behavior --- reference/resources/query-optimization.md | 2 ++ reference/rest/querying.md | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index 6a0e658f1..34b31ad2c 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -89,6 +89,8 @@ Full-text and autocomplete queries use Harper's existing `limit` and `offset` op Within one request, Harper uses one native reader to fetch ranked batches. It checks source versions and record filters before counting records toward the requested offset and limit. Prefix and fuzzy-prefix searches can return more than the native ordinary autocomplete window. The native search window is not a total-result cutoff for bounded Harper queries. +For example, with `offset: 20` and `limit: 10`, Harper skips the first 20 eligible records and returns the next 10. Hits rejected by source-version checks or record filters do not count toward those 30 records, but fetching them still consumes the raw-hit budget. Harper fetches additional batches as needed until it fills the page, exhausts the matches or reaches a request limit. + Use a bounded `limit` for large result sets. An unbounded search that exceeds its native window or configured result/work budget throws 400; use a bounded query. `offset + limit` must not exceed either `storage.fullText.maxResultsPerRequest` or `storage.fullText.maxRawHitsPerRequest`; replayed or rejected hits can exhaust the raw-work budget earlier. Raising only the raw-hit budget allows more filtering work without raising the retained-record target. Offset requests revisit earlier matches and can retain up to `offset + limit` accepted records. Raising work limits can increase CPU, reads and memory use. The fetched-hit budget includes the initial batch replay when Harper switches to native traversal. It counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. The adapter does not return a shortened result as if the search had completed. If a custom resource has already started streaming a response, a later query error ends that stream instead of changing its HTTP status. When there are no further native matches, Harper returns the eligible records found, which may be an empty result. This differs from reaching a budget while more matches remain. diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 44201c153..e8674701c 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -183,7 +183,7 @@ Use `limit(start,end)` to page through a collection, and opt in to a total match Counting is opt-in: without the header, no count is computed and no count headers are returned. It applies only to `GET`/`HEAD` requests and requires a bounded page — the `limit()` must be a non-negative integer no larger than **10,000**, and the requested window (offset + limit) no larger than **1,000,000**. A request with no `limit()`, one outside those bounds, or a non-numeric one is served normally with no count headers, since counting an unbounded page would defeat the point of paging. -Full-text and autocomplete queries use the same paging syntax. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for search work limits and behavior when records change between requests. +Full-text and autocomplete queries use the same paging syntax. The page window counts eligible records after record filtering, rather than raw full-text hits. Harper fetches more ranked batches to fill that window when needed. A page can contain fewer records when no further matches remain; reaching a search work or time limit produces an error instead of a successful partial page. Each page request runs a new query, so concurrent changes can cause repeats or omissions between pages. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for search work limits and recovery guidance. The count-header guardrails do not override query-specific limits such as the full-text work budget. From a95ce001a5b721f8caff9f45fbe9c960a2e577f4 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Wed, 21 Oct 2026 10:33:03 -0600 Subject: [PATCH 61/78] Clarify full-text configuration leaf updates --- reference/configuration/operations.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/reference/configuration/operations.md b/reference/configuration/operations.md index 8da1c0403..c10e66715 100644 --- a/reference/configuration/operations.md +++ b/reference/configuration/operations.md @@ -26,6 +26,8 @@ Full-text request work limits (`storage.fullText.maxRawHitsPerRequest`, `storage Additional properties correspond to configuration keys in underscore-separated format (e.g. `logging_level` for `logging.level`, `clustering_enabled` for `clustering.enabled`). +For full-text settings, use individual parameter names such as `storage_fullText_maxRawHitsPerRequest` or `storage_fullText_runtime_maxSearchThreads`. `set_configuration` does not accept a whole `storage_fullText` object; use `storage.fullText` as the section name only in the YAML configuration file. + :::warning Replicate cluster-appropriate parameters only A replicated call sends the exact same values to every node. Do not include node-local parameters — ports, `node.hostname`, `rootPath`, storage/log file paths, TLS certificate or key paths, or `replication.hostname`/`url`/`routes` — as they would overwrite each peer's local values. ::: From afabe8dccf316e1df679c45516426e8436d5a238 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Wed, 21 Oct 2026 16:50:40 -0600 Subject: [PATCH 62/78] Clarify the pre-v5.4 prefix paging boundary --- reference/resources/query-optimization.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index 34b31ad2c..d6a6d3495 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -87,7 +87,7 @@ This is a transient condition — the index finishes building in the background, Full-text and autocomplete queries use Harper's existing `limit` and `offset` options. Each page runs a new query: changed records, permissions or index contents can cause records to repeat or be omitted between pages. There is no retained search across requests. See [REST paging and counts](../rest/querying.md#pagination-and-total-count) for URL syntax and count behavior. -Within one request, Harper uses one native reader to fetch ranked batches. It checks source versions and record filters before counting records toward the requested offset and limit. Prefix and fuzzy-prefix searches can return more than the native ordinary autocomplete window. The native search window is not a total-result cutoff for bounded Harper queries. +Within one request, Harper uses one native reader to fetch ranked batches. It checks source versions and record filters before counting records toward the requested offset and limit. Prefix and fuzzy-prefix searches can return more than the native ordinary autocomplete window. Before v5.4, prefix and fuzzy-prefix queries restrict `offset + limit` to 100. In v5.4, the native search window is not a total-result cutoff for bounded Harper queries. For example, with `offset: 20` and `limit: 10`, Harper skips the first 20 eligible records and returns the next 10. Hits rejected by source-version checks or record filters do not count toward those 30 records, but fetching them still consumes the raw-hit budget. Harper fetches additional batches as needed until it fills the page, exhausts the matches or reaches a request limit. From ee4436ea84c439bc1f5febfe7db99970be761f4c Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Wed, 21 Oct 2026 17:17:27 -0600 Subject: [PATCH 63/78] Remove unsupported full-text metadata examples and reconcile release notes --- .../full-text-search-product-catalog.mdx | 7 +++---- reference/database/schema.md | 2 +- reference/full-text-search/configuration.md | 20 +++---------------- reference/full-text-search/overview.md | 2 +- reference/full-text-search/querying.md | 8 +++----- release-notes/v5-lincoln/5.3.md | 8 ++------ 6 files changed, 13 insertions(+), 34 deletions(-) diff --git a/learn/developers/full-text-search-product-catalog.mdx b/learn/developers/full-text-search-product-catalog.mdx index 48845a5c9..1a3b8e7be 100644 --- a/learn/developers/full-text-search-product-catalog.mdx +++ b/learn/developers/full-text-search-product-catalog.mdx @@ -40,7 +40,6 @@ type Product @table(database: "catalog", audit: true) @export { { name: "description", weight: 1, highlight: true } { name: "tags" } ] - filterFields: ["category", "available"] highlighting: { maxFragments: 2, fragmentLength: 120 } ) } @@ -48,7 +47,7 @@ type Product @table(database: "catalog", audit: true) @export { The query-only field name `catalogSearch` identifies the index. It is not stored with each product. `weight: 3` makes a match in `name` contribute more to BM25 relevance than the same match in `description` or `tags`. Highlighting is enabled only for the fields shown to customers. -`filterFields` copies category and availability into the Tantivy index as typed, score-neutral metadata. This lets Tantivy narrow text matches before Harper loads product records. The product record remains authoritative, and Harper rechecks the conditions before returning it. The `@indexed` declarations are independent Harper secondary indexes; they are not required for Tantivy filter metadata. +The `@indexed` declarations create ordinary Harper secondary indexes. Full-text searches can use them to narrow candidates, and Harper checks structured conditions against current product records before returning results. The full-text declaration does not accept a separate filter-metadata option. ## 2. Add records @@ -125,7 +124,7 @@ Content-Type: application/json The category filter is optional; `q` is required. -The full-text index applies the configured availability and optional category metadata while ranking candidates. Harper then loads the current records and checks those conditions again before returning them. Without `filterFields`, the same query remains valid, but Harper applies the conditions after full-text search. +Harper applies the availability and optional category conditions while accepting current product records. Existing secondary indexes can narrow candidates; rejected records do not count toward the requested page. No separate full-text filter-metadata declaration is needed. Calls through `tables` and `databases` use Harper's [trusted server-side context](/reference/v5/components/javascript-environment#tables). Forwarding the resource's `context` preserves the authenticated caller, and `checkPermission: true` asks the table to enforce that caller's table and source-field permissions. The custom resource must do both. Highlights can expose searched source text even when those fields are absent from `select`. @@ -151,7 +150,7 @@ For record autocomplete: GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&select(id,name)&limit(10) ``` -Prefix search returns matching products. It does not return a dictionary of suggested terms. Before v5.4, prefix queries restrict `offset + limit` to 100. In v5.4, bounded pages can go beyond that window within request budgets; each page runs a new query rather than retaining a search. See [paging and prefix result windows](/reference/v5/full-text-search/querying#paging-and-prefix-result-windows). +Prefix search returns matching products. It does not return a dictionary of suggested terms. Before v5.4, prefix and fuzzy-prefix queries restrict `offset + limit` to 100. In v5.4, bounded pages can go beyond that window within request budgets; each page runs a new query rather than retaining a search. See [paging and prefix result windows](/reference/v5/full-text-search/querying#paging-and-prefix-result-windows). REST searches every source field in the index, so the caller must be allowed to read `name`, `description`, and `tags` for these requests. Use `Table.search()` or `search_by_conditions` when a role should search only a subset of sources. See [REST full-text operators](/reference/v5/rest/querying#full-text-operators). diff --git a/reference/database/schema.md b/reference/database/schema.md index 25777919e..f2c0b3545 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -380,7 +380,7 @@ type Product @table(database: "catalog", audit: true) { } ``` -The `FullText` field is query-only. It is not stored with records or exposed as a selectable record property. Add another `FullText` field to create another independent index. Full-text indexes require RocksDB and an audited table. They are local derived state rebuilt from committed table data rather than authoritative record storage. See [Full-Text Search Configuration](../full-text-search/configuration.md) for the complete field contract, filter metadata, phrase and prefix storage, synonyms, highlighting, and rebuild behavior. +The `FullText` field is query-only. It is not stored with records or exposed as a selectable record property. Add another `FullText` field to create another independent index. Full-text indexes require RocksDB and an audited table. They are local derived state rebuilt from committed table data rather than authoritative record storage. See [Full-Text Search Configuration](../full-text-search/configuration.md) for the complete field contract, structured filters, phrase and prefix storage, synonyms, highlighting, and rebuild behavior. ### `@decide` diff --git a/reference/full-text-search/configuration.md b/reference/full-text-search/configuration.md index cb62ed87a..d31fb0ebf 100644 --- a/reference/full-text-search/configuration.md +++ b/reference/full-text-search/configuration.md @@ -24,7 +24,6 @@ type Product @table(database: "catalog", audit: true) { { name: "description", weight: 1, highlight: true } { name: "tags", weight: 1 } ] - filterFields: ["category", "available"] highlighting: { maxFragments: 2, fragmentLength: 120 } ) } @@ -47,7 +46,6 @@ Earlier prerelease type-level `@fullText(name: ...)` declarations are not suppor | Option | Default | Description | | -------------- | ------------- | ------------------------------------------------------------------------------------- | | `fields` | required | One or more source fields. | -| `filterFields` | `[]` | Stored attributes copied into Tantivy as typed, score-neutral filter metadata. | | `analyzer` | `"english@2"` | Versioned text analyzer. `english@2` is the only supported value. | | `stopWords` | `true` | Removes common English terms during analysis. | | `positions` | `true` | Stores token positions. Required for phrase queries. | @@ -66,21 +64,9 @@ Each field entry supports: Source fields cannot use `@computed` or `@relationship`. Indexable source values are strings, arrays containing strings or nulls, or Blob values declared as plain text. -## Filter metadata +## Structured filters -`filterFields` copies selected stored attributes into the Tantivy index. Tantivy can apply those conditions before returning ranked IDs, reducing record reads without changing BM25 scores. Harper still loads the returned records and checks the original conditions, permissions, and row filters before returning them. - -Schema configuration controls which conditions Tantivy can apply directly. It does not restrict Harper queries. A condition on an attribute omitted from `filterFields` still works, but Harper applies it after full-text search. Query syntax does not change; Harper pushes eligible conditions automatically. - -| Declaration | Purpose | -| ----------------------------- | ----------------------------------------------------------------------------- | -| `fields` in `@fullText` | Analyzed text that contributes to BM25 scoring | -| `filterFields` in `@fullText` | Typed metadata that narrows one full-text index without changing scores | -| `@indexed` on an attribute | Independent Harper secondary index; not required for a full-text filter field | - -Each full-text index has its own filter metadata. Configure attributes commonly combined with text search rather than every table attribute. Filter metadata increases index size and write work. - -Supported attributes are `String`, `Boolean`, `Int`, `Long`, `Float`, `Date`, and lists of those types. Equality and `in` can be pushed for every supported type. Lower-bounded and bounded ranges can be pushed for `Int`, `Long`, and `Date`. Harper retains other conditions as record filters. Lists match when any element matches. Null, missing, and empty-list values are not indexed. Filter strings are limited to 65,530 UTF-8 bytes. Computed fields and relationships are not supported. +Combine full-text conditions with ordinary Harper conditions using `and`. Harper checks current source records and can use existing secondary indexes to narrow candidates. There is no `filterFields` option on `@fullText`; an unknown directive option returns `400`. See [combining conditions](./querying.md#combining-conditions). ## Phrase and prefix support @@ -163,7 +149,7 @@ type Document @table(audit: true) { ## Schema changes and rebuilds -Changes that affect indexed storage create a new local index generation and rebuild from source data. These include source membership, source order, media types, filter fields, analyzer settings, stop-word handling, positions, surface terms, and synonyms. Keep the `fields` list in a stable order when its meaning has not changed. +Changes that affect indexed storage create a new local index generation and rebuild from source data. These include source membership, source order, media types, analyzer settings, stop-word handling, positions, surface terms, and synonyms. Keep the `fields` list in a stable order when its meaning has not changed. While a generation rebuilds, ordinary table reads and writes remain available, but full-text queries on that node return retryable `503` responses until the index is ready. diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index 0fce51c1c..7bf10b84d 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -68,7 +68,7 @@ Full-text search can be combined with structured filters. Harper pushes compatib ## Limits and troubleshooting - Phrase search requires `positions: true`; prefix and fuzzy-prefix require `surfaceTerms: true`. -- Before v5.4, prefix expressions restrict `offset + limit` to 100. In v5.4, bounded full-text and autocomplete queries fetch additional batches within request budgets. Use a bounded `limit` and see [paging and prefix result windows](./querying.md#paging-and-prefix-result-windows); native windows are not total-result cutoffs for bounded v5.4 queries. +- Before v5.4, prefix and fuzzy-prefix expressions restrict `offset + limit` to 100. In v5.4, bounded full-text and autocomplete queries fetch additional batches within request budgets. Use a bounded `limit` and see [paging and prefix result windows](./querying.md#paging-and-prefix-result-windows); native windows are not total-result cutoffs for bounded v5.4 queries. - One query can use only one full-text index. An `or` group cannot mix full-text and ordinary record conditions. - Each Blob source is limited to 1 MiB (1,048,576 bytes). A media-type mismatch, oversized value, or invalid UTF-8 removes the entire record from the index, including its other full-text source fields, until valid content is indexed. - `INDEX_REBUILDING` is transient and retryable. `DERIVED_INDEX_LAGGING` can mean a query coverage wait expired or local writes were shed after prolonged index lag. `unavailable` means automatic rebuild attempts were exhausted or activation failed; inspect readiness and logs rather than retrying every `503` indefinitely. diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 611727303..e176d39de 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -139,9 +139,7 @@ const products = Product.search({ }); ``` -When `category` and `available` are declared in the index's `filterFields`, Harper includes these conditions in the Tantivy query. Tantivy removes non-matching IDs before Harper loads records. The metadata is score-neutral, and Harper rechecks the original conditions against current records before returning them. - -The same query remains valid when an attribute is not declared in `filterFields`; Harper applies that condition after full-text search. Small, complete result sets from Harper secondary indexes can also be passed to Tantivy automatically as candidate IDs. This path does not require `filterFields`. +Harper checks the structured conditions against current source records before returning them. Existing Harper secondary indexes can narrow candidate IDs or support point checks; no separate full-text filter metadata declaration is required. Rejected hits do not count toward a v5.4 page's offset and limit, but fetching them still consumes the raw-hit budget. An `or` group may combine conditions from the same full-text index: @@ -205,9 +203,9 @@ Before v5.4, any expression containing one of these modes uses a 100-record nati Full-text and autocomplete searches use Harper's existing `limit` and `offset`. Bounded queries can pass the ordinary native result window, including the earlier 100-record prefix boundary. Within one request, one native reader fetches ranked batches until Harper has enough eligible records for the offset and limit, exhausts the matches, or reaches a request budget. Source-version checks and record filters run before records count toward the requested page. -For example, `offset: 100` and `limit: 10` skip the first 100 eligible records and return up to the next 10, even for prefix matching. A short or empty page means matches were exhausted; reaching a work budget while more matches remain raises an error instead. An unbounded query can still fail when it exceeds the native window or configured limits. +For example, `offset: 100` and `limit: 10` skip the first 100 eligible records and return up to the next 10, even for prefix matching. A short or empty page means matches were exhausted. Exhausting the fetched-hit or retained-result budget returns `400`; narrow the query, reduce offset or limit, or raise the named budget. Exhausting the execution budget returns `503`; narrow the query or adjust the execution budget rather than retrying it unchanged. These budget errors carry no readiness-specific `code` or `retryable` flag. If a response has already started streaming, a later error terminates that stream instead of changing its HTTP status. An unbounded query can still fail when it exceeds the native window or configured limits. -`storage.fullText.maxRawHitsPerRequest`, `maxResultsPerRequest` and `requestBudgetMilliseconds` bound fetched work, retained results and execution time. Replayed or rejected hits consume raw work; a request can exhaust that budget before reaching its result target. See [storage configuration](../configuration/options.md#storage) for the controls and [REST paging and counts](../rest/querying.md#pagination-and-total-count) for URL syntax and count behavior. +`storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds` bound fetched work, retained results and execution time. Replayed or rejected hits consume raw work; a request can exhaust that budget before reaching its result target. The execution budget is not a hard response-time guarantee; reader acquisition and some native work can take additional time. See [storage configuration](../configuration/options.md#storage) for the controls and [REST paging and counts](../rest/querying.md#pagination-and-total-count) for URL syntax and count behavior. Each offset request runs a new query and can see changed records, permissions or index contents. Pages can therefore repeat or omit records during concurrent changes. There are no continuation tokens, close operations or retained searches across requests. diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index a92d3e203..84ba47850 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -12,12 +12,6 @@ These notes summarize what changed relative to 5.2. The [Harper Pro v5.3.0 relea All patch release notes for 5.3.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.3&expanded=true). -## Full-Text Search - -Harper tables can declare a query-only `FullText` field with `@fullText` and search that field through `Table.search()`, REST, or `search_by_conditions`. Search supports any-term, all-term, phrase, prefix, fuzzy, and fuzzy-prefix matching, with optional source weights, synonyms, scores, and highlights. - -The indexes are derived locally from committed audited RocksDB records. Each replica maintains its own files, reuses a compatible checkpoint after restart, and rebuilds locally when necessary. See [Full-Text Search](/reference/v5/full-text-search/overview) and [Add Full-Text Search to a Product Catalog](/learn/developers/full-text-search-product-catalog). - ## Upgrade Notes Read these before upgrading a production node or cluster. @@ -43,6 +37,8 @@ Read these before upgrading a production node or cluster. Tables can declare `@fullText` fields and query them with BM25-ranked search through the existing `Table.search`, REST, and Operations API paths, with the same authorization as other queries ([harper#2855](https://github.com/HarperFast/harper/pull/2855)). Declared fields are projected into local Tantivy indexes provided by the optional `@harperfast/fulltext` package. RocksDB and the audit log remain the source of truth: the index files can be rebuilt, are reused across restarts when compatible, and are caught up from the audit log before the index reports ready ([harper#2569](https://github.com/HarperFast/harper/pull/2569), [harper#2615](https://github.com/HarperFast/harper/pull/2615), [harper#2616](https://github.com/HarperFast/harper/pull/2616)). Dropping a database or table releases every worker's index handles, and a drop that is interrupted is retried. +Search supports any-term, all-term, phrase, prefix, fuzzy, and fuzzy-prefix matching, with optional source weights, synonyms, scores, and highlights. See [Full-Text Search](/reference/v5/full-text-search/overview) and [Add Full-Text Search to a Product Catalog](/learn/developers/full-text-search-product-catalog). + ## Vector Search: Native HNSW Index A newly declared HNSW index now uses a native, memory-mapped graph file (the optional `@harperfast/hnsw` package) instead of a graph stored in RocksDB, when it is eligible: the table is on RocksDB and has audit enabled (already recorded for the table, or set with `audit: true` in the same declaration), and the index uses the default int8-quantized cosine distance and default graph-structure options (`M`, `efConstruction`, `mL`, `optimizeRouting`). An ineligible index that leaves `nativePlane` unset, or any index that sets `nativePlane: false`, uses the previous graph. Setting `nativePlane: true` on an ineligible index is refused. Existing indexes keep the graph they were built with. Queries rerank only the rows they return, and index lag behind writes is bounded, with a way for a caller to wait until its prior writes are searchable ([harper#2430](https://github.com/HarperFast/harper/pull/2430), [harper#2657](https://github.com/HarperFast/harper/pull/2657), [harper#2658](https://github.com/HarperFast/harper/pull/2658)). From d0754c261221d63e7617469ac34ddec966140027 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Wed, 21 Oct 2026 17:17:47 -0600 Subject: [PATCH 64/78] Clarify native response overflow and audit requirements --- reference/database/transaction.md | 2 +- reference/resources/query-optimization.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/database/transaction.md b/reference/database/transaction.md index 82f891561..5bed6afb7 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -204,7 +204,7 @@ type Dog @table(audit: true) { } ``` -This overrides the [`logging.auditLog`](../logging/configuration.md) global configuration for that specific table. +This overrides the [`logging.auditLog`](../logging/configuration.md) global configuration for that specific table. Keep auditing enabled for tables with full-text indexes: disabling it prevents full-text index registration. ## Related Documentation diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index d6a6d3495..bbdd158bc 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -115,7 +115,7 @@ Full-text indexes require audit logging; a table declared with `@table(audit: fa | Index readiness or coverage | 503 | `INDEX_REBUILDING` or `DERIVED_INDEX_LAGGING`, with `retryable: true`. | | Worker loaded native search before runtime capture | 500 | `FullTextWorkerInitializationError`, with `retryable: false`; recreate that worker after initialization. | -A native result that cannot fit even a single-hit response returns 413. Other native failures and invalid query configuration can return 500; inspect the error and server logs rather than assuming they are retryable. The table lists common failures, not every possible error. +A native result that exceeds the response-byte envelope can return 413, including an oversized ordinary batch even when each individual hit would fit. Other native failures and invalid query configuration can return 500; inspect the error and server logs rather than assuming they are retryable. The table lists common failures, not every possible error. For read-after-write queries, `maxIndexLagMilliseconds: 0` requires current coverage. `waitForIndexMilliseconds` lets a query wait for coverage (default `0`, maximum `30000` ms); without an override, the allowed index lag is `3000` ms. These options belong to the full-text condition. From 1c86ef0bb3033f48446ee64e1a5b43ecf7a9255f Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Wed, 21 Oct 2026 17:21:11 -0600 Subject: [PATCH 65/78] Keep structured-filter guidance focused on supported options --- learn/developers/full-text-search-product-catalog.mdx | 4 ++-- reference/full-text-search/configuration.md | 2 +- reference/full-text-search/querying.md | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/learn/developers/full-text-search-product-catalog.mdx b/learn/developers/full-text-search-product-catalog.mdx index 1a3b8e7be..20ad256a9 100644 --- a/learn/developers/full-text-search-product-catalog.mdx +++ b/learn/developers/full-text-search-product-catalog.mdx @@ -47,7 +47,7 @@ type Product @table(database: "catalog", audit: true) @export { The query-only field name `catalogSearch` identifies the index. It is not stored with each product. `weight: 3` makes a match in `name` contribute more to BM25 relevance than the same match in `description` or `tags`. Highlighting is enabled only for the fields shown to customers. -The `@indexed` declarations create ordinary Harper secondary indexes. Full-text searches can use them to narrow candidates, and Harper checks structured conditions against current product records before returning results. The full-text declaration does not accept a separate filter-metadata option. +The `@indexed` declarations create ordinary Harper secondary indexes. Full-text searches can use them to narrow candidates, and Harper checks structured conditions against current product records before returning results. ## 2. Add records @@ -124,7 +124,7 @@ Content-Type: application/json The category filter is optional; `q` is required. -Harper applies the availability and optional category conditions while accepting current product records. Existing secondary indexes can narrow candidates; rejected records do not count toward the requested page. No separate full-text filter-metadata declaration is needed. +Harper applies the availability and optional category conditions while accepting current product records. Existing secondary indexes can narrow candidates; rejected records do not count toward the requested page. Calls through `tables` and `databases` use Harper's [trusted server-side context](/reference/v5/components/javascript-environment#tables). Forwarding the resource's `context` preserves the authenticated caller, and `checkPermission: true` asks the table to enforce that caller's table and source-field permissions. The custom resource must do both. Highlights can expose searched source text even when those fields are absent from `select`. diff --git a/reference/full-text-search/configuration.md b/reference/full-text-search/configuration.md index d31fb0ebf..239414ca6 100644 --- a/reference/full-text-search/configuration.md +++ b/reference/full-text-search/configuration.md @@ -66,7 +66,7 @@ Source fields cannot use `@computed` or `@relationship`. Indexable source values ## Structured filters -Combine full-text conditions with ordinary Harper conditions using `and`. Harper checks current source records and can use existing secondary indexes to narrow candidates. There is no `filterFields` option on `@fullText`; an unknown directive option returns `400`. See [combining conditions](./querying.md#combining-conditions). +Combine full-text conditions with ordinary Harper conditions using `and`. Harper checks current source records and can use existing secondary indexes to narrow candidates. Only the directive options listed above are accepted; an unknown option returns `400`. See [combining conditions](./querying.md#combining-conditions). ## Phrase and prefix support diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index e176d39de..83570cb2c 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -139,7 +139,7 @@ const products = Product.search({ }); ``` -Harper checks the structured conditions against current source records before returning them. Existing Harper secondary indexes can narrow candidate IDs or support point checks; no separate full-text filter metadata declaration is required. Rejected hits do not count toward a v5.4 page's offset and limit, but fetching them still consumes the raw-hit budget. +Harper checks the structured conditions against current source records before returning them. Existing Harper secondary indexes can narrow candidate IDs or support point checks. Rejected hits do not count toward a v5.4 page's offset and limit, but fetching them still consumes the raw-hit budget. An `or` group may combine conditions from the same full-text index: From c319f8cac39a0b1dd0100335c35417f545fb5cd0 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Wed, 21 Oct 2026 17:27:33 -0600 Subject: [PATCH 66/78] Clarify paging recovery, offset costs and condition ownership --- reference/full-text-search/overview.md | 3 ++- reference/full-text-search/querying.md | 4 ++-- reference/resources/resource-api.md | 4 ++-- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index 7bf10b84d..beac197c6 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -63,11 +63,12 @@ Retry rejected writes with backoff rather than immediately adding more load. Ins The native index ranks matching documents with BM25. Harper then loads the authoritative table records for the matching IDs and discards stale or deleted candidates. Selecting `$score` includes the relevance score in each result. -Full-text search can be combined with structured filters. Harper pushes compatible filters into candidate evaluation so selective filters do not silently under-fill a requested page. +Full-text search can be combined with structured filters. Harper checks conditions against current source records before returning them. In v5.4, additional ranked batches can fill a filtered page within request budgets; those budgets can prevent the page from filling. ## Limits and troubleshooting - Phrase search requires `positions: true`; prefix and fuzzy-prefix require `surfaceTerms: true`. +- Deep offsets revisit earlier matches and increase fetched work and source reads. Prefer narrower filters and a bounded maximum offset over repeatedly raising work budgets. - Before v5.4, prefix and fuzzy-prefix expressions restrict `offset + limit` to 100. In v5.4, bounded full-text and autocomplete queries fetch additional batches within request budgets. Use a bounded `limit` and see [paging and prefix result windows](./querying.md#paging-and-prefix-result-windows); native windows are not total-result cutoffs for bounded v5.4 queries. - One query can use only one full-text index. An `or` group cannot mix full-text and ordinary record conditions. - Each Blob source is limited to 1 MiB (1,048,576 bytes). A media-type mismatch, oversized value, or invalid UTF-8 removes the entire record from the index, including its other full-text source fields, until valid content is indexed. diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 83570cb2c..6dcaeee40 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -203,11 +203,11 @@ Before v5.4, any expression containing one of these modes uses a 100-record nati Full-text and autocomplete searches use Harper's existing `limit` and `offset`. Bounded queries can pass the ordinary native result window, including the earlier 100-record prefix boundary. Within one request, one native reader fetches ranked batches until Harper has enough eligible records for the offset and limit, exhausts the matches, or reaches a request budget. Source-version checks and record filters run before records count toward the requested page. -For example, `offset: 100` and `limit: 10` skip the first 100 eligible records and return up to the next 10, even for prefix matching. A short or empty page means matches were exhausted. Exhausting the fetched-hit or retained-result budget returns `400`; narrow the query, reduce offset or limit, or raise the named budget. Exhausting the execution budget returns `503`; narrow the query or adjust the execution budget rather than retrying it unchanged. These budget errors carry no readiness-specific `code` or `retryable` flag. If a response has already started streaming, a later error terminates that stream instead of changing its HTTP status. An unbounded query can still fail when it exceeds the native window or configured limits. +For example, `offset: 100` and `limit: 10` skip the first 100 eligible records and return up to the next 10, even for prefix matching. A short or empty page means matches were exhausted. Exhausting the fetched-hit or retained-result budget returns `400`; narrow the query, reduce offset or limit, or raise the named budget. Exhausting the execution budget returns `503`; narrow the query or adjust the execution budget rather than retrying it unchanged. These budget errors carry no readiness-specific `code` or `retryable` flag. If a response has already started streaming, a later error terminates that stream instead of changing its HTTP status. An unbounded query can still fail when it exceeds the native window or configured limits. Native response-envelope overflow can return `413`, including an oversized ordinary batch even when each hit would fit. Other native failures can return `500`. A `503` without a readiness code is not a signal to wait for an index rebuild; inspect its message and adjust the work or concurrency that caused it. `storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds` bound fetched work, retained results and execution time. Replayed or rejected hits consume raw work; a request can exhaust that budget before reaching its result target. The execution budget is not a hard response-time guarantee; reader acquisition and some native work can take additional time. See [storage configuration](../configuration/options.md#storage) for the controls and [REST paging and counts](../rest/querying.md#pagination-and-total-count) for URL syntax and count behavior. -Each offset request runs a new query and can see changed records, permissions or index contents. Pages can therefore repeat or omit records during concurrent changes. There are no continuation tokens, close operations or retained searches across requests. +Each offset request runs a new query and revisits earlier matches, so deeper offsets increase fetched work and source reads. Prefer narrower filters and a bounded maximum offset over repeatedly raising budgets. Each query can see changed records, permissions or index contents. Pages can therefore repeat or omit records during concurrent changes. There are no continuation tokens, close operations or retained searches across requests. ## REST diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index ac1d5b5fc..4fb23c18d 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1036,8 +1036,8 @@ Array of condition objects to filter records. Each condition: | `operator` | `and` (default) or `or` for the nested `conditions` | | `fields` | Full-text source fields to search; every name must belong to the selected full-text index | | `includeHighlights` | Include configured full-text highlights | -| `maxIndexLagMilliseconds` | Maximum accepted full-text index lag; defaults to `3000` | -| `waitForIndexMilliseconds` | Time to wait for acceptable full-text coverage, from `0` through `30000` | +| `maxIndexLagMilliseconds` | Maximum accepted full-text index lag; see [freshness controls](../full-text-search/querying.md#freshness-controls) | +| `waitForIndexMilliseconds` | Time to wait for acceptable full-text coverage; see [freshness controls](../full-text-search/querying.md#freshness-controls) | The final four properties apply to full-text conditions (v5.3.0). See [Querying Full-Text Indexes](../full-text-search/querying.md) for their constraints, error behavior, and permission requirements for direct server-side searches. From a629e79a6b574eb607abde04d778ec6d28e4dfd9 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Wed, 21 Oct 2026 17:46:16 -0600 Subject: [PATCH 67/78] Make paging errors explicit and keep badge styling unchanged --- reference/full-text-search/overview.md | 2 +- reference/full-text-search/querying.md | 2 ++ reference/rest/querying.md | 18 ++++++++++++------ src/components/EngineBadge.module.css | 7 +++++++ src/components/EngineBadge.tsx | 2 +- 5 files changed, 23 insertions(+), 8 deletions(-) create mode 100644 src/components/EngineBadge.module.css diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index beac197c6..c7ca211c0 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -63,7 +63,7 @@ Retry rejected writes with backoff rather than immediately adding more load. Ins The native index ranks matching documents with BM25. Harper then loads the authoritative table records for the matching IDs and discards stale or deleted candidates. Selecting `$score` includes the relevance score in each result. -Full-text search can be combined with structured filters. Harper checks conditions against current source records before returning them. In v5.4, additional ranked batches can fill a filtered page within request budgets; those budgets can prevent the page from filling. +Full-text search can be combined with structured filters. Harper checks conditions against current source records before returning them. In v5.4, additional ranked batches can fill a filtered page within request budgets; exhausting a budget returns an error rather than a budget-truncated page. ## Limits and troubleshooting diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 6dcaeee40..8ea047423 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -255,3 +255,5 @@ REST can return configured highlights by selecting `$highlights`. Use `Table.sea ``` If an index is `unknown`, `needs-rebuild`, or `rebuilding`, the query returns `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. An index in terminal `unavailable` state returns a generic, non-retryable `503`; inspect readiness and logs instead of retrying it as a rebuild. Other busy paths can also return a generic `503`, so clients should branch on the code rather than the status alone. Unreadable source fields return `403`. Invalid declarations, unsupported match modes, incompatible combinations, and out-of-range options return `400` responses. + +For paging-budget and native response-envelope failures, see [paging and prefix result windows](#paging-and-prefix-result-windows). diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 4d99b2ff1..aceaf2cc3 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -64,6 +64,12 @@ GET /Product/?name==Keyboard* GET /Product/?category=software&price=gt=100&price=lt=200 ``` +For date fields, colons must be URL-encoded as `%3A`: + +```http +GET /Product/?listDate=gt=2017-03-08T09%3A30%3A00.000Z +``` + ### Full-Text Operators @@ -82,16 +88,16 @@ REST supports `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `mat GET /Product/?catalogSearch=matches=waterproof&catalogSearch=not_matches=leather ``` -Use `Table.search()` or `search_by_conditions` when conditions need source-field selection, condition-level highlighting, or freshness controls. See [Querying Full-Text Indexes](../full-text-search/querying.md) for ranking, highlights, query combinations, and freshness controls. - -The REST URL syntax searches every source field in the index, so the caller must be allowed to read every source field. Before v5.4, prefix and fuzzy-prefix queries restrict `offset + limit` to 100. In v5.4, bounded full-text and autocomplete queries can pass that native window within request budgets; source-version checks and filters run before records count toward the offset and limit. Each page is a new query and can see concurrent changes. See [paging and prefix result windows](../full-text-search/querying.md#paging-and-prefix-result-windows). Full-text queries cannot provide an exact total count. On a REST interface with `exactCount: true`, a `Prefer: count=exact` request returns an unavailable total (`Content-Range: items /*`). Without that opt-in, Harper applies `count=estimated` instead and can return an estimate. - -For date fields, colons must be URL-encoded as `%3A`: +In v5.4, REST's `limit(start,end)` uses an exclusive end. This prefix query skips 100 eligible products and requests the next 10, within request budgets: ```http -GET /Product/?listDate=gt=2017-03-08T09%3A30%3A00.000Z +GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&limit(100,110) ``` +Use `Table.search()` or `search_by_conditions` when conditions need source-field selection, condition-level highlighting, or freshness controls. See [Querying Full-Text Indexes](../full-text-search/querying.md) for ranking, highlights, query combinations, and freshness controls. + +The REST URL syntax searches every source field in the index, so the caller must be allowed to read every source field. Before v5.4, prefix and fuzzy-prefix queries restrict `offset + limit` to 100. In v5.4, bounded full-text and autocomplete queries can pass that native window within request budgets; source-version checks and filters run before records count toward the offset and limit. Each page is a new query and can see concurrent changes. See [paging and prefix result windows](../full-text-search/querying.md#paging-and-prefix-result-windows). Full-text queries cannot provide an exact total count. On a REST interface with `exactCount: true`, a `Prefer: count=exact` request returns an unavailable total (`Content-Range: items /*`). Without that opt-in, Harper applies `count=estimated` instead and can return an estimate. + ### Chained Conditions (Range) Omit the attribute name on the second condition to chain it against the same attribute: diff --git a/src/components/EngineBadge.module.css b/src/components/EngineBadge.module.css new file mode 100644 index 000000000..6f748ecf4 --- /dev/null +++ b/src/components/EngineBadge.module.css @@ -0,0 +1,7 @@ +.badge { + display: inline-block; + font-size: 0.8rem; + color: var(--ifm-color-content-secondary); + font-style: italic; + margin-bottom: 0.75rem; +} diff --git a/src/components/EngineBadge.tsx b/src/components/EngineBadge.tsx index a216f7f55..cf7e37cf2 100644 --- a/src/components/EngineBadge.tsx +++ b/src/components/EngineBadge.tsx @@ -1,5 +1,5 @@ import React from 'react'; -import styles from './VersionBadge.module.css'; +import styles from './EngineBadge.module.css'; const CANONICAL_NAMES: Record = { rocksdb: 'RocksDB', From df5264e6be75e769c8af7bcf1d850d0450c7d5c7 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Wed, 21 Oct 2026 17:47:05 -0600 Subject: [PATCH 68/78] Clarify worker error codes and native capacity recovery --- reference/resources/query-optimization.md | 16 ++++++++-------- release-notes/v5-lincoln/5.4.md | 2 +- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index bbdd158bc..2df0d1870 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -107,13 +107,13 @@ Replacement workers keep the captured budgets after a configuration change. See Full-text indexes require audit logging; a table declared with `@table(audit: false)` cannot register a full-text index. A format-incompatible native upgrade rebuilds the local index from stored records, and queries return 503 while it rebuilds. Harper 5.4 uses `@harperfast/fulltext` 0.6.0. -| Failure | HTTP status | Code and retry guidance | -| -------------------------------------------------- | ----------- | -------------------------------------------------------------------------------------------------------- | -| Raw-hit or retained-result budget | 400 | No readiness code. Narrow the query or adjust the named budget. | -| Execution budget | 503 | No readiness code. Narrow the query; the same expensive query can fail again. | -| Native busy | 503 | No readiness code. Reduce concurrency or retry with backoff. | -| Index readiness or coverage | 503 | `INDEX_REBUILDING` or `DERIVED_INDEX_LAGGING`, with `retryable: true`. | -| Worker loaded native search before runtime capture | 500 | `FullTextWorkerInitializationError`, with `retryable: false`; recreate that worker after initialization. | +| Failure | HTTP status | Code and retry guidance | +| -------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | +| Raw-hit or retained-result budget | 400 | No readiness code. Narrow the query or adjust the named budget. | +| Execution budget | 503 | No readiness code. Narrow the query; the same expensive query can fail again. | +| Native busy | 503 | No readiness code. Reduce concurrency or retry with backoff. | +| Index readiness or coverage | 503 | `INDEX_REBUILDING` or `DERIVED_INDEX_LAGGING`, with `retryable: true`. | +| Worker loaded native search before runtime capture | 500 | REST reports `FullTextWorkerInitializationError`; the Operations API can omit that code. Non-retryable; recreate the worker after initialization. | A native result that exceeds the response-byte envelope can return 413, including an oversized ordinary batch even when each individual hit would fit. Other native failures and invalid query configuration can return 500; inspect the error and server logs rather than assuming they are retryable. The table lists common failures, not every possible error. @@ -121,7 +121,7 @@ For read-after-write queries, `maxIndexLagMilliseconds: 0` requires current cove Concurrent requests each have their own work budget. These limits do not cap the number of concurrent Harper queries or prevent brief event-loop delays during synchronous source reads. Measure deep offsets and selective filters under the expected concurrency before raising budgets. -`INDEX_REBUILDING` and `DERIVED_INDEX_LAGGING` identify retryable readiness or coverage failures. Retry with backoff while the index makes progress. Transaction history retained under [`logging.auditRetention`](../logging/configuration.md#loggingauditretention) (default `3d`) is used for catch-up. Lost history can require a rebuild; retries alone do not repair an index that cannot catch up. Check index readiness and logs if it stays unavailable. Budget and native-busy failures do not carry those readiness codes. A native busy response also returns 503; reduce concurrency or retry with backoff. For an execution-budget failure, narrow the query, reduce the offset or adjust the work budget instead of repeatedly submitting the same expensive query. +`INDEX_REBUILDING` and `DERIVED_INDEX_LAGGING` identify retryable readiness or coverage failures. Retry with backoff while the index makes progress. Transaction history retained under [`logging.auditRetention`](../logging/configuration.md#loggingauditretention) (default `3d`) is used for catch-up. Lost history can require a rebuild; retries alone do not repair an index that cannot catch up. Use [`describe_table`](../operations-api/operations.md#describe_table) to inspect `full_text_indexes[].readiness`, including its state and reason. Correct the reported configuration or storage problem before retrying an unavailable index. Budget and native-busy failures do not carry those readiness codes. A native busy response also returns 503; reduce concurrency or retry with backoff. For an execution-budget failure, narrow the query, reduce the offset or adjust the work budget instead of repeatedly submitting the same expensive query. Native page, response-byte and highlight limits still apply. Fulltext 0.6.0 also limits each analyzed query leaf to 64 terms and the compiled query to 256 clauses. Raising Harper's work budgets does not raise those native limits. See the [published native limits](https://github.com/HarperFast/fulltext/blob/v0.6.0/docs/limits.md) for page, byte and highlight bounds. diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index cd676e60a..c5c9f04ab 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -68,6 +68,6 @@ The new `threads.v8Flags` setting takes a V8 flag, or a list of them, and applie ### Paging and Search Work Limits -Full-text and autocomplete queries use the existing `limit` and `offset` paging options. Harper reads additional native batches when needed after filtering, so the initial native result window no longer silently cuts off eligible results. The new `storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds` settings bound fetched hits, retained results and execution time. Requests that exceed those budgets return an error. Separate page requests run independently and can see changes to records or index contents. +Full-text and autocomplete queries use the existing `limit` and `offset` paging options. Harper reads additional native batches when needed after filtering, so the initial native result window no longer silently cuts off eligible results. The new `storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds` settings bound fetched hits, retained results and execution time. Requests that exceed those budgets return an error. The `storage.fullText.runtime` section also provides process-wide native-capacity budgets; changes to those budgets require a full Harper process restart. Separate page requests run independently and can see changes to records or index contents. See [Full-text query execution](/reference/v5/resources/query-optimization#full-text-search-work-and-paging) and [storage configuration](/reference/v5/configuration/options#storage) for limits, restart requirements and recovery guidance. From 06b8ff07fb990f7ac1253f3d703684fb34472a55 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Thu, 22 Oct 2026 06:53:15 -0600 Subject: [PATCH 69/78] docs: address native full-text review feedback --- .../full-text-search-product-catalog.mdx | 6 +++--- reference/full-text-search/configuration.md | 6 +----- reference/full-text-search/overview.md | 2 +- reference/full-text-search/querying.md | 3 +-- reference/operations-api/operations.md | 14 ++++++++++++-- 5 files changed, 18 insertions(+), 13 deletions(-) diff --git a/learn/developers/full-text-search-product-catalog.mdx b/learn/developers/full-text-search-product-catalog.mdx index 20ad256a9..32407de41 100644 --- a/learn/developers/full-text-search-product-catalog.mdx +++ b/learn/developers/full-text-search-product-catalog.mdx @@ -22,7 +22,7 @@ This guide adds relevance-ranked text search to a product catalog, then exposes ## 1. Declare the index -Add `audit: true` to the RocksDB-backed table and declare an `@fullText` index over the fields customers search. +Declare an `@fullText` index over the fields customers search on a RocksDB-backed table. Ordinary tables inherit enabled audit logging by default, but a new full-text table requires explicit `audit: true` for its transaction-log recovery source. See the [audit requirement](/reference/v5/full-text-search/configuration) for existing tables. ```graphql title="schema.graphql" type Product @table(database: "catalog", audit: true) @export { @@ -122,13 +122,13 @@ Content-Type: application/json { "q": "trail", "category": "footwear" } ``` -The category filter is optional; `q` is required. +This custom resource accepts a structured JSON search body through POST. The category filter is optional; `q` is required. For URL-based GET searches, use the exported table endpoint in step 4. Harper applies the availability and optional category conditions while accepting current product records. Existing secondary indexes can narrow candidates; rejected records do not count toward the requested page. Calls through `tables` and `databases` use Harper's [trusted server-side context](/reference/v5/components/javascript-environment#tables). Forwarding the resource's `context` preserves the authenticated caller, and `checkPermission: true` asks the table to enforce that caller's table and source-field permissions. The custom resource must do both. Highlights can expose searched source text even when those fields are absent from `select`. -The search includes `tags`, but that source does not set `highlight: true`. A tag match can rank a product, while `$highlights` omits `tags`. +Matches in `tags` contribute to search results and ranking, but `$highlights` does not include tag matches because `tags` is not configured with `highlight: true`. ## 4. Search through REST diff --git a/reference/full-text-search/configuration.md b/reference/full-text-search/configuration.md index 239414ca6..ee9ef7cfb 100644 --- a/reference/full-text-search/configuration.md +++ b/reference/full-text-search/configuration.md @@ -33,14 +33,10 @@ type Product @table(database: "catalog", audit: true) { The declaration must use nullable `FullText` exactly and carry one `@fullText` directive. Lists, non-null forms, a bare `FullText` field, and additional directives such as `@computed`, `@indexed`, or `@allow` are rejected. `@fullText` is not valid on a table type or any other field type. -New schema tables must explicitly set `@table(audit: true)`; omitting it or setting `audit: false` returns `400`. An existing table that already has persisted audit logging can add the field without restating `audit`. A previously unaudited table can enable `audit: true` in the same schema change that adds its first full-text field; the initial build scans current records and does not require older transaction history. Disabling audit logging while a full-text field exists is rejected. +Ordinary tables inherit `logging.auditLog`, which defaults to `true`. New tables with `@fullText` instead require an explicit `@table(audit: true)`; omitting it or setting `audit: false` returns `400`, even when the global default is enabled. This table-level requirement makes transaction logging an explicit recovery prerequisite for the derived index rather than relying on the ambient default. An existing table that already has persisted audit logging can add the field without restating `audit`. A previously unaudited table can enable `audit: true` in the same schema change that adds its first full-text field; the initial build scans current records and does not require older transaction history. Disabling audit logging while a full-text field exists is rejected. Writing the declaration-only field or selecting it as record data returns an error, including on unsealed tables. A read-through source response containing that name fails with `502` and is not cached, indexed, or audited. For HTTP-sourced caching tables, avoid index names that collide with response wrapper fields such as `data`, `headers`, `body`, and `status`. -:::warning Prerelease schema declarations -Earlier prerelease type-level `@fullText(name: ...)` declarations are not supported. Move the directive to a nullable `FullText` field and use that field's name as the index identity. -::: - ## Directive options | Option | Default | Description | diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index c7ca211c0..afef74d45 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -20,7 +20,7 @@ See [Configuration](./configuration.md) for the complete schema contract, [Query ## How it works -Full-text indexes are derived local state. The table records and schema remain the source of truth. +Full-text indexes are derived local state. Harper uses Tantivy, a native text-search library, for term indexing and BM25 ranking. The table records and schema remain the source of truth. ```mermaid flowchart LR diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 8ea047423..fa6f36564 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -167,7 +167,6 @@ A full-text index is derived from committed table changes. Each condition accept For read-after-write behavior, require current coverage and allow a bounded wait: ```javascript -const products = []; for await (const product of Product.search({ conditions: [ { @@ -179,7 +178,7 @@ for await (const product of Product.search({ }, ], })) { - products.push(product); + console.log(product); } ``` diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 6da38754b..2bf2baf1c 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -63,9 +63,19 @@ Returns all table definitions within the specified database. Each table definiti ### `describe_table` -Returns the definition of a specific table. A table with native full-text indexes includes `full_text_indexes`; each entry reports its query-only field name, source fields, analyzer options, enabled query modes, and `readiness` (`unknown`, `ready`, `needs-rebuild`, `rebuilding`, or `unavailable`). The readiness object also includes `owner_epoch`, `rebuild_attempts`, and an optional `reason`. Query-only `FullText` fields do not appear in the ordinary `attributes` list. +Returns the definition of a specific table, including its schema and storage metadata. -Requesting a table with full-text indexes (Added in: v5.3.0): +```json +{ "operation": "describe_table", "table": "dog", "database": "dev" } +``` + +#### Full-text indexes + + + +A table with native full-text indexes includes `full_text_indexes`; each entry reports its query-only field name, source fields, analyzer options, enabled query modes, and `readiness` (`unknown`, `ready`, `needs-rebuild`, `rebuilding`, or `unavailable`). The readiness object also includes `owner_epoch`, `rebuild_attempts`, and an optional `reason`. Query-only `FullText` fields do not appear in the ordinary `attributes` list. + +Requesting a table with full-text indexes: ```json { "operation": "describe_table", "table": "Product", "database": "catalog" } From a047778105934cb9480fd4b02be7bd3c646b066b Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Thu, 22 Oct 2026 07:08:44 -0600 Subject: [PATCH 70/78] docs: clarify streaming errors and REST range semantics --- learn/developers/full-text-search-product-catalog.mdx | 4 +++- reference/operations-api/operations.md | 4 ++-- reference/rest/querying.md | 2 +- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/learn/developers/full-text-search-product-catalog.mdx b/learn/developers/full-text-search-product-catalog.mdx index 32407de41..3be7b7ee2 100644 --- a/learn/developers/full-text-search-product-catalog.mdx +++ b/learn/developers/full-text-search-product-catalog.mdx @@ -176,12 +176,14 @@ for await (const product of databases.catalog.Product.search({ Do not apply this wait to every catalog request without measuring it. The normal bounded-staleness path keeps search traffic independent of short indexing bursts. -If the index does not reach current coverage within ten seconds, consuming the search returns retryable `503` code `DERIVED_INDEX_LAGGING`. See [freshness controls](/reference/v5/full-text-search/querying#freshness-controls). +If the index does not reach current coverage within ten seconds, iteration throws an error with `statusCode: 503`, `code: "DERIVED_INDEX_LAGGING"`, and `retryable: true`. See [freshness controls](/reference/v5/full-text-search/querying#freshness-controls). ## 6. Operate the index Use `describe_table` to inspect readiness. Harper reuses compatible local index files after restart and replays changes after their checkpoint. A node rebuilds locally when the files are missing, incompatible, or corrupt; table traffic remains available, while full-text requests in `unknown`, `needs-rebuild`, or `rebuilding` state return `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. Retry this code with backoff; do not treat every `503` as an index rebuild. +The custom resource in step 3 streams its returned iterable. HTTP error statuses apply before response headers are sent; a failure after streaming starts is a terminal stream error and cannot change the HTTP status. Consumers must also handle that terminal error. See [freshness and streaming errors](/reference/v5/full-text-search/querying#freshness-controls). + See the [full-text configuration reference](/reference/v5/full-text-search/configuration) and [query reference](/reference/v5/full-text-search/querying) for every option and match mode. ## Additional resources diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 2bf2baf1c..24d71951d 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -47,7 +47,7 @@ Detailed documentation: [Database Overview](../database/overview.md) ### `describe_all` -Returns the definitions of all databases and tables within the database. Record counts above 5000 records are estimated; the response includes `estimated_record_range` when estimated. To force an exact count (requires full table scan), include `"exact_count": true`. Each table definition includes the [record-structure dictionary fields](#describe_table) described for `describe_table`. +Returns the definitions of all databases and tables within the database. Record counts above 5000 records are estimated; the response includes `estimated_record_range` when estimated. To force an exact count (requires full table scan), include `"exact_count": true`. Each table definition includes the [record-structure dictionary fields](#record-structure-dictionaries) described for `describe_table`. ```json { "operation": "describe_all" } @@ -55,7 +55,7 @@ Returns the definitions of all databases and tables within the database. Record ### `describe_database` -Returns all table definitions within the specified database. Each table definition includes the [record-structure dictionary fields](#describe_table) described for `describe_table`. +Returns all table definitions within the specified database. Each table definition includes the [record-structure dictionary fields](#record-structure-dictionaries) described for `describe_table`. ```json { "operation": "describe_database", "database": "dev" } diff --git a/reference/rest/querying.md b/reference/rest/querying.md index aceaf2cc3..81d4a017b 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -88,7 +88,7 @@ REST supports `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `mat GET /Product/?catalogSearch=matches=waterproof&catalogSearch=not_matches=leather ``` -In v5.4, REST's `limit(start,end)` uses an exclusive end. This prefix query skips 100 eligible products and requests the next 10, within request budgets: +REST's `limit(start,end)` uses an exclusive end. In v5.4, this prefix query skips 100 eligible products and requests the next 10, within request budgets: ```http GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&limit(100,110) From 51713cc6ec1eb6d30530647877c998f1d0e5320f Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Thu, 22 Oct 2026 07:13:16 -0600 Subject: [PATCH 71/78] docs: tighten full-text count guidance and scope --- reference/database/overview.md | 20 ++++++++++---------- reference/full-text-search/querying.md | 2 +- reference/resources/resource-api.md | 6 +++--- reference/rest/headers.md | 4 ++-- 4 files changed, 16 insertions(+), 16 deletions(-) diff --git a/reference/database/overview.md b/reference/database/overview.md index 82eae4d46..bfa0434d2 100644 --- a/reference/database/overview.md +++ b/reference/database/overview.md @@ -104,18 +104,18 @@ It is best practice to define schemas for production tables. Dynamic schemas are ## Key Concepts -For deeper coverage of database features, see these related pages: +For deeper coverage of each database feature, see the dedicated pages in this section: -- **[Schema](./schema.md)**: Defining table structure, types, indexes, relationships, and computed properties using GraphQL schema syntax +- **[Schema](./schema.md)** — Defining table structure, types, indexes, relationships, and computed properties using GraphQL schema syntax - **[Full-Text Search](../full-text-search/overview.md)**: BM25-ranked search over stored text fields on audited RocksDB tables -- **[API](./api.md)**: The `tables`, `databases`, `transaction()`, and `createBlob()` globals for interacting with the database from code -- **[Data Loader](./data-loader.md)**: Loading seed or initial data into tables as part of component deployment -- **[Storage Algorithm](./storage-algorithm.md)**: How Harper stores data using LMDB with universal indexing and ACID compliance -- **[Storage Tuning](./storage-tuning.md)**: Tuning `storage.*` options for production: durability vs. throughput, compression, blob paths, reclamation -- **[Jobs](./jobs.md)**: Asynchronous bulk data operations (CSV import/export, S3 import/export) -- **[System Tables](./system-tables.md)**: Harper internal tables for analytics, data loader state, and other system features -- **[Compaction](./compaction.md)**: Reducing database file size by eliminating fragmentation and free space -- **[Transaction Logging](./transaction.md)**: Recording and querying a history of data changes in the transaction log +- **[API](./api.md)** — The `tables`, `databases`, `transaction()`, and `createBlob()` globals for interacting with the database from code +- **[Data Loader](./data-loader.md)** — Loading seed or initial data into tables as part of component deployment +- **[Storage Algorithm](./storage-algorithm.md)** — How Harper stores data using LMDB with universal indexing and ACID compliance +- **[Storage Tuning](./storage-tuning.md)** — Tuning `storage.*` options for production: durability vs. throughput, compression, blob paths, reclamation +- **[Jobs](./jobs.md)** — Asynchronous bulk data operations (CSV import/export, S3 import/export) +- **[System Tables](./system-tables.md)** — Harper internal tables for analytics, data loader state, and other system features +- **[Compaction](./compaction.md)** — Reducing database file size by eliminating fragmentation and free space +- **[Transaction Logging](./transaction.md)** — Recording and querying a history of data changes in the transaction log ## Related Documentation diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index fa6f36564..65cb45883 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -81,7 +81,7 @@ const products = Product.search({ Full-text results use descending relevance order. The only explicit full-text sort is `$score` descending; other sort keys and reverse iteration are rejected because they would discard the native ranking. -An exact count requested through `Table.search()` does not report a full-text total after authorization, structured filtering, and current-record validation. It returns `recordCount: null` and `recordCountExact: false`. REST behaves the same when its interface enables `exactCount`; otherwise `Prefer: count=exact` is downgraded to an estimated count. +Full-text queries cannot provide an exact total after authorization, structured filtering, and current-record validation. On a REST interface with `exactCount` enabled, `Prefer: count=exact` reports an unavailable total; otherwise it is downgraded to an estimated count. See [pagination and total count](../rest/querying.md#pagination-and-total-count). `Table.getRecordCount()` counts table records, not matches for a full-text query. ## Highlights diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 4fb23c18d..d141d2a86 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1088,9 +1088,9 @@ Properties to include in each returned record. Can be: Special properties: -- `$id`: Returns the primary key regardless of its name -- `$updatedtime`: Returns the last-updated timestamp -- `$distance`: When the query ranks or filters by a vector index, returns the computed distance from the target vector. See [Vector Indexing](../database/schema.md#vector-indexing). +- `$id` — Returns the primary key regardless of its name +- `$updatedtime` — Returns the last-updated timestamp +- `$distance` — When the query ranks or filters by a vector index, returns the computed distance from the target vector. See [Vector Indexing](../database/schema.md#vector-indexing). diff --git a/reference/rest/headers.md b/reference/rest/headers.md index 3508819f6..04ca435c9 100644 --- a/reference/rest/headers.md +++ b/reference/rest/headers.md @@ -20,13 +20,13 @@ These headers are included in all Harper REST API responses: | `etag` | `"abc123"` | Encoded version/last-modification time of the returned record. Used for conditional requests. | | `location` | `/MyTable/new-id` | Returned on `POST` responses. Contains the path to the newly created record. | -Collection responses to a [count request](./querying.md#pagination-and-total-count) additionally include `Content-Range`, `Range-Unit`, and `Preference-Applied` (Added in: v5.3.0). +Collection responses to a [count request](./querying.md#pagination-and-total-count) additionally include `Content-Range`, `Range-Unit`, and `Preference-Applied` (). ### `Harper-Index-Coverage` -A non-waiting request that uses a native derived index can include `Harper-Index-Coverage`. Its value is one of: +A non-waiting request that uses a full-text index can include `Harper-Index-Coverage`. Its value is one of: - `current; lag=0; tolerance=`: the index covered current committed data at admission. - `bounded; lag=; tolerance=`: the admitted lag upper bound was within the requested tolerance. From f1f55daf9632009b7d39e344487bea7b0c37ecd0 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Sat, 24 Oct 2026 08:14:31 -0600 Subject: [PATCH 72/78] docs: address paging restart and error review feedback --- reference/logging/configuration.md | 6 ++-- reference/operations-api/operations.md | 39 ++++++++++++++++++++--- reference/resources/query-optimization.md | 27 +++++++++------- reference/resources/resource-api.md | 2 +- reference/rest/querying.md | 2 +- 5 files changed, 55 insertions(+), 21 deletions(-) diff --git a/reference/logging/configuration.md b/reference/logging/configuration.md index f204b9325..5ef541370 100644 --- a/reference/logging/configuration.md +++ b/reference/logging/configuration.md @@ -102,11 +102,11 @@ Default: `true` Enables transaction logging. When enabled, Harper records every insert, update, and delete to the database's transaction log. This history is read with the [`read_audit_log`](../database/transaction.md#read_audit_log) operation — a legacy name; there is a single transaction log, not a separate audit table. -See [Database / Transaction Logging](../database/transaction.md) for details on the transaction log. +Full-text indexing requires transaction logging. Disabling this setting prevents new tables from inheriting that prerequisite; a full-text table must also have its per-table audit setting enabled. See [Database / Transaction Logging](../database/transaction.md) for details on the transaction log. ```yaml logging: - auditLog: false + auditLog: true ``` ### `logging.auditRetention` @@ -339,7 +339,7 @@ logging: file: true stdStreams: false console: false - auditLog: false + auditLog: true auditRetention: 3d rotation: enabled: true diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 8a41f2aeb..56f789b94 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -47,7 +47,7 @@ Detailed documentation: [Database Overview](../database/overview.md) ### `describe_all` -Returns the definitions of all databases and tables within the database. Record counts above 5000 records are estimated; the response includes `estimated_record_range` when estimated. To force an exact count (requires full table scan), include `"exact_count": true`. Each table definition includes the [record-structure dictionary fields](#describe_table) described for `describe_table`. +Returns the definitions of all databases and tables within the database. Record counts above 5000 records are estimated; the response includes `estimated_record_range` when estimated. To force an exact count (requires full table scan), include `"exact_count": true`. Each table definition includes the [record-structure dictionary fields](#record-structure-dictionaries) described for `describe_table`. ```json { "operation": "describe_all" } @@ -55,7 +55,7 @@ Returns the definitions of all databases and tables within the database. Record ### `describe_database` -Returns all table definitions within the specified database. Each table definition includes the [record-structure dictionary fields](#describe_table) described for `describe_table`. +Returns all table definitions within the specified database. Each table definition includes the [record-structure dictionary fields](#record-structure-dictionaries) described for `describe_table`. ```json { "operation": "describe_database", "database": "dev" } @@ -63,12 +63,43 @@ Returns all table definitions within the specified database. Each table definiti ### `describe_table` -Returns the definition of a specific table. +Returns the definition of a specific table, including its schema and storage metadata. ```json { "operation": "describe_table", "table": "dog", "database": "dev" } ``` +#### Full-text indexes + + + +A table with native full-text indexes includes `full_text_indexes`; each entry reports its query-only field name, source fields, analyzer options, enabled query modes, and `readiness` (`unknown`, `ready`, `needs-rebuild`, `rebuilding`, or `unavailable`). The readiness object also includes `owner_epoch`, `rebuild_attempts`, and an optional `reason`. Query-only `FullText` fields do not appear in the ordinary `attributes` list. + +Requesting a table with full-text indexes: + +```json +{ "operation": "describe_table", "table": "Product", "database": "catalog" } +``` + +The response also includes the table's ordinary schema and storage metadata. The relevant full-text fragment is: + +```json +{ + "name": "Product", + "full_text_indexes": [ + { + "name": "catalogSearch", + "fields": [{ "name": "name", "weight": 3 }], + "analyzer": "english@2", + "query_modes": ["any", "all", "fuzzy", "phrase", "prefix", "fuzzy-prefix"], + "readiness": { "state": "ready", "owner_epoch": "4", "rebuild_attempts": 0 } + } + ] +} +``` + +#### Record-structure dictionaries + Alongside the schema, the response carries the size of the table's **record-structure dictionaries** @@ -1688,7 +1719,7 @@ Detailed documentation: [Configuration Overview](../configuration/overview.md) Updates configuration parameters in `harper-config.yaml`. Most changes require a restart (`restart` or `restart_service`) to take effect. See [Set Configuration](../configuration/operations.md#set-configuration) for full-text restart requirements. -Supports `"replicated": true` to apply the same change to all cluster nodes in one call; per-node outcomes are returned in the response's `replicated` array. Only send cluster-appropriate parameters when replicating — node-local parameters (ports, `node.hostname`, file paths, TLS material, `replication.hostname`/`url`/`routes`) would overwrite every peer's local values. To apply the change cluster-wide, follow with `restart_service` using `"replicated": true` (which restarts nodes one at a time); `restart_service` restarts only worker threads, though, so a change to [`threads.v8Flags`](../configuration/options.md#applying-v8-flags) takes effect only after a `restart` sent to each node in turn (`restart` does not replicate). See [Configuration Operations](../configuration/operations.md#set-configuration) for details. +Supports `"replicated": true` to apply the same change to all cluster nodes in one call; per-node outcomes are returned in the response's `replicated` array. Only send cluster-appropriate parameters when replicating — node-local parameters (ports, `node.hostname`, file paths, TLS material, `replication.hostname`/`url`/`routes`) would overwrite every peer's local values. To apply the change cluster-wide, follow with `restart_service` using `"replicated": true` (which restarts nodes one at a time); `restart_service` restarts only worker threads, though, so changes to [`threads.v8Flags`](../configuration/options.md#applying-v8-flags) or `storage.fullText.runtime.*` take effect only after a `restart` sent to each node in turn (`restart` does not replicate). See [Configuration Operations](../configuration/operations.md#set-configuration) for details. ```json { diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index 2df0d1870..9d837065e 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -87,11 +87,11 @@ This is a transient condition — the index finishes building in the background, Full-text and autocomplete queries use Harper's existing `limit` and `offset` options. Each page runs a new query: changed records, permissions or index contents can cause records to repeat or be omitted between pages. There is no retained search across requests. See [REST paging and counts](../rest/querying.md#pagination-and-total-count) for URL syntax and count behavior. -Within one request, Harper uses one native reader to fetch ranked batches. It checks source versions and record filters before counting records toward the requested offset and limit. Prefix and fuzzy-prefix searches can return more than the native ordinary autocomplete window. Before v5.4, prefix and fuzzy-prefix queries restrict `offset + limit` to 100. In v5.4, the native search window is not a total-result cutoff for bounded Harper queries. +Within one request, Harper uses one native reader to fetch ranked batches. It checks source versions and record filters before counting records toward the requested offset and limit. In v5.4, prefix and fuzzy-prefix searches can return more than the native ordinary autocomplete window. Before v5.4, prefix and fuzzy-prefix requests whose `offset + limit` exceeded 100 were rejected with 400; requests within that window could still return short pages after filtering. In v5.4, the native search window is not a total-result cutoff for bounded Harper queries. For example, with `offset: 20` and `limit: 10`, Harper skips the first 20 eligible records and returns the next 10. Hits rejected by source-version checks or record filters do not count toward those 30 records, but fetching them still consumes the raw-hit budget. Harper fetches additional batches as needed until it fills the page, exhausts the matches or reaches a request limit. -Use a bounded `limit` for large result sets. An unbounded search that exceeds its native window or configured result/work budget throws 400; use a bounded query. `offset + limit` must not exceed either `storage.fullText.maxResultsPerRequest` or `storage.fullText.maxRawHitsPerRequest`; replayed or rejected hits can exhaust the raw-work budget earlier. Raising only the raw-hit budget allows more filtering work without raising the retained-record target. Offset requests revisit earlier matches and can retain up to `offset + limit` accepted records. Raising work limits can increase CPU, reads and memory use. +Use a bounded `limit` for large result sets. An unbounded search that exceeds its native window, fetched-hit limit or retained-result limit throws 400; use a bounded query. Exhausting the execution-time budget instead throws 503. `offset + limit` must not exceed either `storage.fullText.maxResultsPerRequest` or `storage.fullText.maxRawHitsPerRequest`; replayed or rejected hits can exhaust the raw-work budget earlier. Raising only the raw-hit budget allows more filtering work without raising the retained-record target. Offset requests revisit earlier matches and can retain up to `offset + limit` accepted records. Raising work limits can increase CPU, reads and memory use. The fetched-hit budget includes the initial batch replay when Harper switches to native traversal. It counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. The adapter does not return a shortened result as if the search had completed. If a custom resource has already started streaming a response, a later query error ends that stream instead of changing its HTTP status. When there are no further native matches, Harper returns the eligible records found, which may be an empty result. This differs from reaching a budget while more matches remain. @@ -105,25 +105,28 @@ Replacement workers keep the captured budgets after a configuration change. See ### Readiness and recovery -Full-text indexes require audit logging; a table declared with `@table(audit: false)` cannot register a full-text index. A format-incompatible native upgrade rebuilds the local index from stored records, and queries return 503 while it rebuilds. Harper 5.4 uses `@harperfast/fulltext` 0.6.0. +Full-text indexes require audit logging; a table declared with `@table(audit: false)` cannot register a full-text index. A format-incompatible native upgrade rebuilds the local index from stored records, and queries return 503 while it rebuilds. The v5.4 paging implementation targets `@harperfast/fulltext` 0.6.0. -| Failure | HTTP status | Code and retry guidance | -| -------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | -| Raw-hit or retained-result budget | 400 | No readiness code. Narrow the query or adjust the named budget. | -| Execution budget | 503 | No readiness code. Narrow the query; the same expensive query can fail again. | -| Native busy | 503 | No readiness code. Reduce concurrency or retry with backoff. | -| Index readiness or coverage | 503 | `INDEX_REBUILDING` or `DERIVED_INDEX_LAGGING`, with `retryable: true`. | -| Worker loaded native search before runtime capture | 500 | REST reports `FullTextWorkerInitializationError`; the Operations API can omit that code. Non-retryable; recreate the worker after initialization. | +| Failure | HTTP status | Code and retry guidance | +| --------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | +| Raw-hit or retained-result budget | 400 | No readiness code. Narrow the query or adjust the named budget. | +| Execution budget | 503 | No readiness code. Narrow the query; the same expensive query can fail again. | +| Native busy | 503 | No readiness code. Reduce concurrency or retry with backoff. | +| Index unknown, needs-rebuild or rebuilding | 503 | `INDEX_REBUILDING`, with `retryable: true`, for a not-ready readiness check. Retry with backoff while the index rebuilds. | +| Coverage lag or wait timeout | 503 | `DERIVED_INDEX_LAGGING`, with `retryable: true`. Retry with backoff while the index catches up. | +| Index unavailable | 503 | No readiness-specific code or retryable flag. Inspect readiness and logs; fix the reported problem before retrying. | +| Query host becomes non-ready during a coverage wait | 503 | Can return no readiness-specific code or retryable flag. Inspect readiness before deciding whether to retry. | +| Worker loaded native search before runtime capture | 500 | REST reports `FullTextWorkerInitializationError`; the Operations API can omit that code. Non-retryable; recreate the worker after initialization. | A native result that exceeds the response-byte envelope can return 413, including an oversized ordinary batch even when each individual hit would fit. Other native failures and invalid query configuration can return 500; inspect the error and server logs rather than assuming they are retryable. The table lists common failures, not every possible error. -For read-after-write queries, `maxIndexLagMilliseconds: 0` requires current coverage. `waitForIndexMilliseconds` lets a query wait for coverage (default `0`, maximum `30000` ms); without an override, the allowed index lag is `3000` ms. These options belong to the full-text condition. +For read-after-write queries, `maxIndexLagMilliseconds: 0` requires current coverage. `waitForIndexMilliseconds` lets a query wait for coverage (default `0`, maximum `30000` ms); without an override, the allowed index lag is `3000` ms. These options belong to the full-text condition. They wait for coverage, not for an unavailable index to recover. Readiness is checked first; a query host that becomes rebuilding or unavailable during wait setup can fail immediately with a bare 503. Concurrent requests each have their own work budget. These limits do not cap the number of concurrent Harper queries or prevent brief event-loop delays during synchronous source reads. Measure deep offsets and selective filters under the expected concurrency before raising budgets. `INDEX_REBUILDING` and `DERIVED_INDEX_LAGGING` identify retryable readiness or coverage failures. Retry with backoff while the index makes progress. Transaction history retained under [`logging.auditRetention`](../logging/configuration.md#loggingauditretention) (default `3d`) is used for catch-up. Lost history can require a rebuild; retries alone do not repair an index that cannot catch up. Use [`describe_table`](../operations-api/operations.md#describe_table) to inspect `full_text_indexes[].readiness`, including its state and reason. Correct the reported configuration or storage problem before retrying an unavailable index. Budget and native-busy failures do not carry those readiness codes. A native busy response also returns 503; reduce concurrency or retry with backoff. For an execution-budget failure, narrow the query, reduce the offset or adjust the work budget instead of repeatedly submitting the same expensive query. -Native page, response-byte and highlight limits still apply. Fulltext 0.6.0 also limits each analyzed query leaf to 64 terms and the compiled query to 256 clauses. Raising Harper's work budgets does not raise those native limits. See the [published native limits](https://github.com/HarperFast/fulltext/blob/v0.6.0/docs/limits.md) for page, byte and highlight bounds. +Native page, response-byte and highlight limits still apply. That native version also limits each analyzed query leaf to 64 terms and the compiled query to 256 clauses. Raising Harper's work budgets does not raise those native limits. See the [published native limits](https://github.com/HarperFast/fulltext/blob/v0.6.0/docs/limits.md) for page, byte and highlight bounds. ## Relationships and Joins diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 202bf521c..36561e720 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1035,7 +1035,7 @@ Array of condition objects to filter records. Each condition: | `conditions` | Nested conditions array | | `operator` | `and` (default) or `or` for the nested `conditions` | -Full-text conditions use `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, or `matches_fuzzy_prefix` against a declared full-text index. See [Full-text query execution](./query-optimization.md#full-text-search-work-and-paging) for paging costs and work limits. +Full-text conditions use `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, or `matches_fuzzy_prefix` against a declared full-text index. Each comparator also has a `not_` form: `not_matches`, `not_matches_all`, `not_matches_phrase`, `not_matches_prefix`, `not_matches_fuzzy`, and `not_matches_fuzzy_prefix`. A negated condition requires a positive full-text condition on the same index. See [Full-text query execution](./query-optimization.md#full-text-search-work-and-paging) for paging costs and work limits. Example with nested conditions: diff --git a/reference/rest/querying.md b/reference/rest/querying.md index e8674701c..94b300ec4 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -55,7 +55,7 @@ Harper uses [FIQL](https://datatracker.ietf.org/doc/html/draft-nottingham-atompu | `=`, `===` | Strict equality (no type conversion) | | `!==` | Strict inequality (no type conversion) | -Full-text indexes support `=matches=`, `=matches_all=`, `=matches_phrase=`, `=matches_prefix=`, `=matches_fuzzy=`, and `=matches_fuzzy_prefix=`. These use the full-text index rather than ordinary string comparisons. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). +Full-text indexes support `=matches=`, `=matches_all=`, `=matches_phrase=`, `=matches_prefix=`, `=matches_fuzzy=`, and `=matches_fuzzy_prefix=`. Each has a negated form: `=not_matches=`, `=not_matches_all=`, `=not_matches_phrase=`, `=not_matches_prefix=`, `=not_matches_fuzzy=`, and `=not_matches_fuzzy_prefix=`. Combine a negated condition with a positive full-text condition on the same index; a structured filter alone does not qualify. These use the full-text index rather than ordinary string comparisons. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). **Examples**: From e92602c45cc4247df9e20722a6dd589dc5eb2818 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 26 Oct 2026 12:45:00 -0600 Subject: [PATCH 73/78] Clarify full-text replica freshness and operational limits --- learn/developers/full-text-search-product-catalog.mdx | 2 +- reference/full-text-search/configuration.md | 2 +- reference/full-text-search/overview.md | 1 + reference/full-text-search/querying.md | 2 +- 4 files changed, 4 insertions(+), 3 deletions(-) diff --git a/learn/developers/full-text-search-product-catalog.mdx b/learn/developers/full-text-search-product-catalog.mdx index 3be7b7ee2..ca34d3c04 100644 --- a/learn/developers/full-text-search-product-catalog.mdx +++ b/learn/developers/full-text-search-product-catalog.mdx @@ -68,7 +68,7 @@ Content-Type: application/json } ``` -Harper commits the record first, then applies the committed change to the local derived full-text index. Every replica performs the same derivation from the transaction it receives. +Harper commits the record first, then applies the committed change to the local derived full-text index. Each replica performs the same derivation after it has received both the full-text schema declaration and the record transactions. If the index remains behind for more than 30 seconds, a local write can return retryable `503` code `DERIVED_INDEX_LAGGING`. Retry writes with backoff while the index catches up; freshness options apply only to searches. See [write backpressure](/reference/v5/full-text-search/overview#write-backpressure). diff --git a/reference/full-text-search/configuration.md b/reference/full-text-search/configuration.md index ee9ef7cfb..5e604bdf1 100644 --- a/reference/full-text-search/configuration.md +++ b/reference/full-text-search/configuration.md @@ -163,7 +163,7 @@ Changing source weights or highlighting settings changes query behavior without } ``` -The table response includes entries shaped like this: +Metadata respects the caller's attribute permissions: unreadable source fields are omitted, and an index with no readable source fields is omitted entirely. An omitted entry does not by itself mean that the index is unconfigured. With access to its source fields, the table response includes entries shaped like this: ```json { diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index afef74d45..abafb4da8 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -70,6 +70,7 @@ Full-text search can be combined with structured filters. Harper checks conditio - Phrase search requires `positions: true`; prefix and fuzzy-prefix require `surfaceTerms: true`. - Deep offsets revisit earlier matches and increase fetched work and source reads. Prefer narrower filters and a bounded maximum offset over repeatedly raising work budgets. - Before v5.4, prefix and fuzzy-prefix expressions restrict `offset + limit` to 100. In v5.4, bounded full-text and autocomplete queries fetch additional batches within request budgets. Use a bounded `limit` and see [paging and prefix result windows](./querying.md#paging-and-prefix-result-windows); native windows are not total-result cutoffs for bounded v5.4 queries. +- `Table.clear()` returns `501` while any full-text declaration is active. To clear the table, first remove all `FullText` declarations and wait for the schema change to apply. Re-add the declarations afterward to rebuild the indexes from the remaining data. - One query can use only one full-text index. An `or` group cannot mix full-text and ordinary record conditions. - Each Blob source is limited to 1 MiB (1,048,576 bytes). A media-type mismatch, oversized value, or invalid UTF-8 removes the entire record from the index, including its other full-text source fields, until valid content is indexed. - `INDEX_REBUILDING` is transient and retryable. `DERIVED_INDEX_LAGGING` can mean a query coverage wait expired or local writes were shed after prolonged index lag. `unavailable` means automatic rebuild attempts were exhausted or activation failed; inspect readiness and logs rather than retrying every `503` indefinitely. diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 65cb45883..4980eb120 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -164,7 +164,7 @@ A full-text index is derived from committed table changes. Each condition accept | `maxIndexLagMilliseconds` | `3000` | Maximum accepted upper bound on index lag. Set to `0` to require current coverage. | | `waitForIndexMilliseconds` | `0` | How long to wait for acceptable coverage, from `0` through `30000`. | -For read-after-write behavior, require current coverage and allow a bounded wait: +To wait for committed changes already received by this node to reach its local index, require current coverage and allow a bounded wait. These controls do not wait for a write to arrive from another replica; a query routed to a different node can run before that node receives the write: ```javascript for await (const product of Product.search({ From 2c084ecf0235d12f98c5b029397405e8c9ea2a02 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 27 Oct 2026 20:33:03 -0600 Subject: [PATCH 74/78] Document Tantivy 0.27 versions and rebuild requirements --- reference/resources/query-optimization.md | 4 ++-- release-notes/v5-lincoln/5.4.md | 6 ++++++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index 9d837065e..b8ea4fd29 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -105,7 +105,7 @@ Replacement workers keep the captured budgets after a configuration change. See ### Readiness and recovery -Full-text indexes require audit logging; a table declared with `@table(audit: false)` cannot register a full-text index. A format-incompatible native upgrade rebuilds the local index from stored records, and queries return 503 while it rebuilds. The v5.4 paging implementation targets `@harperfast/fulltext` 0.6.0. +Full-text indexes require audit logging; a table declared with `@table(audit: false)` cannot register a full-text index. A format-incompatible native upgrade rebuilds the local index from stored records, and queries return 503 while it rebuilds. Harper v5.4 uses `@harperfast/fulltext` 0.7.0 with Tantivy 0.27.0. Indexes from fulltext 0.5.0 (shipped in Harper 5.3.4), as well as prerelease 0.6 indexes, require a full local rebuild. Drain the node's search traffic and stop old processes before starting the new version; do not mix engine versions against one local index. Verify readiness and current coverage before restoring search traffic. Replay after a long rebuild can also cause retryable write-503 responses; retry with bounded backoff while it catches up. Direct downgrade to fulltext 0.5.x is unsupported. | Failure | HTTP status | Code and retry guidance | | --------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -126,7 +126,7 @@ Concurrent requests each have their own work budget. These limits do not cap the `INDEX_REBUILDING` and `DERIVED_INDEX_LAGGING` identify retryable readiness or coverage failures. Retry with backoff while the index makes progress. Transaction history retained under [`logging.auditRetention`](../logging/configuration.md#loggingauditretention) (default `3d`) is used for catch-up. Lost history can require a rebuild; retries alone do not repair an index that cannot catch up. Use [`describe_table`](../operations-api/operations.md#describe_table) to inspect `full_text_indexes[].readiness`, including its state and reason. Correct the reported configuration or storage problem before retrying an unavailable index. Budget and native-busy failures do not carry those readiness codes. A native busy response also returns 503; reduce concurrency or retry with backoff. For an execution-budget failure, narrow the query, reduce the offset or adjust the work budget instead of repeatedly submitting the same expensive query. -Native page, response-byte and highlight limits still apply. That native version also limits each analyzed query leaf to 64 terms and the compiled query to 256 clauses. Raising Harper's work budgets does not raise those native limits. See the [published native limits](https://github.com/HarperFast/fulltext/blob/v0.6.0/docs/limits.md) for page, byte and highlight bounds. +Native page, response-byte and highlight limits still apply. That native version also limits each analyzed query leaf to 64 terms and the compiled query to 256 clauses. Raising Harper's work budgets does not raise those native limits. See the [published native limits](https://github.com/HarperFast/fulltext/blob/v0.7.0/docs/limits.md) for page, byte and highlight bounds. ## Relationships and Joins diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 9251d902c..de85f0611 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -70,6 +70,12 @@ The new `threads.v8Flags` setting takes a V8 flag, or a list of them, and applie ## Full-Text Search +### Tantivy 0.27 and Existing Index Rebuilds + +Harper now uses `@harperfast/fulltext` 0.7.0 with Tantivy 0.27.0. Existing fulltext 0.5.x indexes, including those created by Harper 5.3.4, and prerelease 0.6 indexes rebuild automatically from authoritative records. Plan for local full-text unavailability, disk headroom and possible retryable write failures while replay catches up. Stop old processes before starting the new version against the same storage; direct downgrade to fulltext 0.5.x is unsupported. + +English stemming changes despite the unchanged `english@2` analyzer name: `added` now stems to `add` rather than `ad`. Matching and ranking can change. Query modes and schema options are unchanged; benchmark results from the wrapper do not establish application performance parity. Review representative searches and see [readiness and recovery](/reference/v5/resources/query-optimization#readiness-and-recovery) before restoring traffic. + ### Paging and Search Work Limits Full-text and autocomplete queries use the existing `limit` and `offset` paging options. Harper reads additional native batches when needed after filtering, so the initial native result window no longer silently cuts off eligible results. The new `storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds` settings bound fetched hits, retained results and execution time. Requests that exceed those budgets return an error. The `storage.fullText.runtime` section also provides process-wide native-capacity budgets; changes to those budgets require a full Harper process restart. Separate page requests run independently and can see changes to records or index contents. From 966b0fb6010c1c8d6ef12d9eecbdc91475c61097 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 27 Oct 2026 20:33:15 -0600 Subject: [PATCH 75/78] Document full-text engine upgrades and recovery --- reference/full-text-search/configuration.md | 2 ++ reference/full-text-search/overview.md | 30 ++++++++++++++++++++- 2 files changed, 31 insertions(+), 1 deletion(-) diff --git a/reference/full-text-search/configuration.md b/reference/full-text-search/configuration.md index 5e604bdf1..bea10ebc7 100644 --- a/reference/full-text-search/configuration.md +++ b/reference/full-text-search/configuration.md @@ -49,6 +49,8 @@ Writing the declaration-only field or selecting it as record data returns an err | `synonyms` | `[]` | Index-time synonym expansion rules. | | `highlighting` | omitted | Enables highlight generation and sets fragment limits. | +The `english@2` name remains unchanged in v5.4, but Tantivy 0.27 changes some English stems. Existing indexes must rebuild, and matching or ranking can change. See [upgrading existing indexes](./overview.md#upgrading-existing-indexes). + Each field entry supports: | Option | Default | Description | diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index abafb4da8..d18a5a098 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -10,6 +10,8 @@ Harper can build a BM25-ranked full-text index from one or more fields on a tabl Use full-text search when users need relevance-ranked matching across product names, descriptions, tags, articles, or other natural-language fields. It supports term, all-term, phrase, prefix, fuzzy, and fuzzy-prefix matching. + Starting in Harper v5.4, the native engine is **`@harperfast/fulltext` 0.7.0, embedding Tantivy 0.27.0**. Existing native indexes require rebuilding. See [Upgrading existing indexes](#upgrading-existing-indexes) before upgrading a node. + ## Requirements - The table must use RocksDB and have [transaction logging enabled](../database/transaction.md#enabling-the-transaction-log-per-table). New schema tables must declare `@table(audit: true)`; omitting it or setting `audit: false` returns `400`. An existing table that already persisted audit logging can add a full-text field without restating the option, but disabling audit is rejected while an index is declared. @@ -49,10 +51,36 @@ On restart, Harper reuses compatible local index files and replays committed cha A rebuild captures a fresh committed-log boundary, scans the current table, and replays changes committed after that boundary. An unreadable or corrupt transaction log can make an attempt fail. The same applies when [transaction-log cleanup](../database/transaction.md#delete_transaction_logs_before) or automatic retention removes the history needed to bridge that scan to current writes. A later attempt captures a new boundary, so stopping aggressive cleanup can allow the rebuild to succeed without restoring already-purged history. Harper retries with backoff; after the retry budget is exhausted, readiness becomes `unavailable`. Inspect `describe_table` and the node logs for the reason before retrying activation. -Ordinary table reads and writes remain available during a rebuild. While readiness is `unknown`, `needs-rebuild`, or `rebuilding`, full-text queries return `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. If rebuild attempts are exhausted and readiness becomes `unavailable`, queries return a generic, non-retryable `503` until an operator corrects the cause and retries activation. A new replica follows the same process after the full-text declaration reaches that node. +Ordinary table reads remain available during a rebuild. Writes to indexed tables can receive retryable lag errors while replay catches up after a long rebuild; see [Write backpressure](#write-backpressure). While readiness is `unknown`, `needs-rebuild`, or `rebuilding`, full-text queries return `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. If rebuild attempts are exhausted and readiness becomes `unavailable`, queries return a generic, non-retryable `503` until an operator corrects the cause and retries activation. A new replica follows the same process after the full-text declaration reaches that node. Use `describe_table` to inspect each index's declaration and readiness. See [Inspecting an index](./configuration.md#inspecting-an-index). +## Upgrading existing indexes + + + +Harper 5.3.4 ships `@harperfast/fulltext` 0.5.0. Upgrading to the 0.7.0 engine in Harper v5.4 requires a full local rebuild of every existing 0.5.x index. Prerelease installations using fulltext 0.6 also require rebuilding. Harper initiates the rebuild automatically when the wrapper refuses the old persisted identity; no manual rebuild command or file editing is needed. + +### Matching and ranking changes + +Tantivy 0.27 changes English stemming even though the schema analyzer remains `english@2`. For example, `added` now stems to `add` instead of `ad`. Queries can match different records or rank them differently after rebuilding. Review representative searches and domain vocabulary before restoring production traffic; the upgrade does not add new Harper query modes or change the declared field options. + +### Upgrade checklist + +1. Drain the node's full-text requests and stop all old Harper processes that share its storage, including background workers. Do not run old and new engine versions against one local index. +2. Confirm authoritative records and recovery backups are available. Reserve disk headroom and time for rebuilding every local full-text index, including temporarily retained retired files. There is no universal rebuild-duration or capacity estimate. +3. Start the v5.4 Harper version. Use [`describe_table`](../operations-api/operations.md#describe_table) for each affected table and check every `full_text_indexes[].readiness.state`. Keep full-text traffic drained until all required indexes are `ready`. +4. Run representative searches with `maxIndexLagMilliseconds: 0` and `waitForIndexMilliseconds: 30000` on each full-text condition. These wait for coverage of writes committed before the search; readiness alone does not establish that coverage. Check response bodies for terminal stream errors as well as HTTP status. A JSON-array response can start with HTTP 200 and end with an `error` element. +5. Retry write responses carrying `DERIVED_INDEX_LAGGING` with bounded backoff while replay catches up. Restore full-text traffic only after readiness and current-coverage searches succeed. A node-by-node upgrade can use other ready replicas only if the application's routing supports it; the rebuilding node remains unavailable for full-text search. + +See [freshness controls](./querying.md#freshness-controls) for the condition fields and streaming behavior. The upgrade checks cover bounded workloads; they do not establish multi-node rollout safety, sustained-write capacity, customer relevance or performance parity. Measure those against the application's workload before rollout. + +### Recovering from a failed upgrade + +Keep full-text traffic drained, preserve authoritative data and logs, and correct the reported package, storage or coordination failure before restarting the same v5.4 Harper version. An interrupted rebuild can recover from authoritative records. Do not force readiness or edit, move or delete native index files by hand. + +**Direct downgrade to fulltext 0.5.x is unsupported.** That engine refuses both opening and resetting a 0.7 index identity. If an older version has already been started, stop it and restore the 0.7.0-based Harper version before checking readiness and catch-up again. This is recovery with the current engine, not a supported operator downgrade procedure. If recovery also fails, keep traffic drained and investigate the readiness reason and node logs. + ## Write backpressure Harper allows a full-text index up to 30 seconds to absorb committed changes. If the runtime remains behind beyond that budget, ordinary local writes to the indexed table fail with `503`, `code: "DERIVED_INDEX_LAGGING"`, and `retryable: true`. This applies to writes through REST, the Operations API, and direct table calls from application code. Replication notifications, source applies, and crash-recovery replay continue so the authoritative data can converge. From bbf78c0b7fe2053c8d91b1f7e60d13e19d7bfe46 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 27 Oct 2026 20:52:56 -0600 Subject: [PATCH 76/78] Clarify full-text upgrade readiness and retry guidance --- learn/developers/full-text-search-product-catalog.mdx | 4 ++-- reference/backups/overview.md | 2 +- reference/full-text-search/configuration.md | 10 ++++++---- reference/full-text-search/overview.md | 8 ++++---- reference/operations-api/operations.md | 2 +- 5 files changed, 14 insertions(+), 12 deletions(-) diff --git a/learn/developers/full-text-search-product-catalog.mdx b/learn/developers/full-text-search-product-catalog.mdx index ca34d3c04..749af6287 100644 --- a/learn/developers/full-text-search-product-catalog.mdx +++ b/learn/developers/full-text-search-product-catalog.mdx @@ -156,7 +156,7 @@ REST searches every source field in the index, so the caller must be allowed to ## 5. Choose consistency behavior -Most searches can use the default three-second lag tolerance. When the next request must find a record that was just written, use the JavaScript API and wait for current index coverage: +Most searches can use the default three-second lag tolerance. When the next request must find a record already committed on the queried node, use the JavaScript API and wait for current index coverage. This wait does not wait for replication to deliver writes from another node: ```javascript for await (const product of databases.catalog.Product.search({ @@ -180,7 +180,7 @@ If the index does not reach current coverage within ten seconds, iteration throw ## 6. Operate the index -Use `describe_table` to inspect readiness. Harper reuses compatible local index files after restart and replays changes after their checkpoint. A node rebuilds locally when the files are missing, incompatible, or corrupt; table traffic remains available, while full-text requests in `unknown`, `needs-rebuild`, or `rebuilding` state return `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. Retry this code with backoff; do not treat every `503` as an index rebuild. +Use `describe_table` to inspect readiness. Harper reuses compatible local index files after restart and replays changes after their checkpoint. A node rebuilds locally when the files are missing, incompatible, or corrupt; ordinary table reads remain available, while full-text requests in `unknown`, `needs-rebuild`, or `rebuilding` state return `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. Retry this code with backoff; do not treat every `503` as an index rebuild. Writes can also receive retryable `DERIVED_INDEX_LAGGING` responses while replay catches up after a long rebuild; see [write backpressure](/reference/v5/full-text-search/overview#write-backpressure). The custom resource in step 3 streams its returned iterable. HTTP error statuses apply before response headers are sent; a failure after streaming starts is a terminal stream error and cannot change the HTTP status. Consumers must also handle that terminal error. See [freshness and streaming errors](/reference/v5/full-text-search/querying#freshness-controls). diff --git a/reference/backups/overview.md b/reference/backups/overview.md index 52b927417..659683498 100644 --- a/reference/backups/overview.md +++ b/reference/backups/overview.md @@ -38,7 +38,7 @@ The offline path matters for restore. RocksDB is single-writer, so an in-place r - **Backups live on the node that created them.** The backup repository is a local directory. RocksDB shares files across backup IDs (a backup ID is not a self-contained folder), so disaster-recovery copies must take the entire per-database repository — `/` — not an individual backup, and must do so while no backup operation is running, or from an atomic filesystem snapshot. A live recursive copy can race `create_backup`/`delete_backup`/`purge_backups` and produce an unrestorable copy. Alternatively, use `get_backup` to pull a snapshot from a running server. - **`get_backup` always streams the current state.** It cannot download a historical managed backup; to move a retained backup off-host, copy its whole per-database repository as above. - **A restore is a point-in-time rollback.** In a replicated cluster, coordinate a restore with replication before bringing the node back. -- **Full-text indexes rebuild locally after restore.** Ordinary table traffic remains available. While the rebuild is active, full-text requests return `503` with `code: "INDEX_REBUILDING"` and `retryable: true`; branch on the code because other `503` responses are not necessarily rebuilds. +- **Full-text indexes rebuild locally after restore.** Ordinary table reads remain available. Writes can receive retryable `DERIVED_INDEX_LAGGING` responses while replay catches up after a long rebuild; see [write backpressure](../full-text-search/overview.md#write-backpressure). While the rebuild is active, full-text requests return `503` with `code: "INDEX_REBUILDING"` and `retryable: true`; branch on the code because other `503` responses are not necessarily rebuilds. - **A restore checks the backup before it deletes anything.** A backup with a corrupt database file, or one this version of Harper cannot read, is refused and the database is left as it was (a database an earlier restore left incomplete stays blocked). The check reads the whole backup, which adds to how long an online restore keeps the database unavailable. Blob file contents are not checked. See [`restore_backup`](./operations.md#restore_backup). - **An interrupted restore leaves the database unloadable.** If `restore_backup` is interrupted after it has started replacing the database's files (crash, power loss), Harper marks the database as incompletely restored and skips loading it on the next start, logging an incomplete-restore error. Rerun `restore_backup` for the same database and `backup_id` to recover; do not load or hand-repair the directory. diff --git a/reference/full-text-search/configuration.md b/reference/full-text-search/configuration.md index bea10ebc7..c750b166a 100644 --- a/reference/full-text-search/configuration.md +++ b/reference/full-text-search/configuration.md @@ -39,6 +39,8 @@ Writing the declaration-only field or selecting it as record data returns an err ## Directive options +Only the directive options listed below are accepted; an unknown option returns `400`. + | Option | Default | Description | | -------------- | ------------- | ------------------------------------------------------------------------------------- | | `fields` | required | One or more source fields. | @@ -49,8 +51,6 @@ Writing the declaration-only field or selecting it as record data returns an err | `synonyms` | `[]` | Index-time synonym expansion rules. | | `highlighting` | omitted | Enables highlight generation and sets fragment limits. | -The `english@2` name remains unchanged in v5.4, but Tantivy 0.27 changes some English stems. Existing indexes must rebuild, and matching or ranking can change. See [upgrading existing indexes](./overview.md#upgrading-existing-indexes). - Each field entry supports: | Option | Default | Description | @@ -64,7 +64,7 @@ Source fields cannot use `@computed` or `@relationship`. Indexable source values ## Structured filters -Combine full-text conditions with ordinary Harper conditions using `and`. Harper checks current source records and can use existing secondary indexes to narrow candidates. Only the directive options listed above are accepted; an unknown option returns `400`. See [combining conditions](./querying.md#combining-conditions). +Combine full-text conditions with ordinary Harper conditions using `and`. Harper checks current source records and can use existing secondary indexes to narrow candidates. See [combining conditions](./querying.md#combining-conditions). ## Phrase and prefix support @@ -147,9 +147,11 @@ type Document @table(audit: true) { ## Schema changes and rebuilds +The `english@2` name remains unchanged in v5.4, but Tantivy 0.27 changes some English stems. Existing indexes must rebuild, and matching or ranking can change. See [upgrading existing indexes](./overview.md#upgrading-existing-indexes). + Changes that affect indexed storage create a new local index generation and rebuild from source data. These include source membership, source order, media types, analyzer settings, stop-word handling, positions, surface terms, and synonyms. Keep the `fields` list in a stable order when its meaning has not changed. -While a generation rebuilds, ordinary table reads and writes remain available, but full-text queries on that node return retryable `503` responses until the index is ready. +While a generation rebuilds, ordinary table reads remain available, but full-text queries on that node return retryable `503` responses until the index is ready. Writes can receive retryable `DERIVED_INDEX_LAGGING` responses while replay catches up after a long rebuild; see [write backpressure](./overview.md#write-backpressure). Changing source weights or highlighting settings changes query behavior without rebuilding indexed term storage. Removing the `FullText` field retires its local files after the schema change is confirmed. Renaming the field creates a new index identity and retires the old one. diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index d18a5a098..ef073bc7d 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -59,7 +59,7 @@ Use `describe_table` to inspect each index's declaration and readiness. See [Ins -Harper 5.3.4 ships `@harperfast/fulltext` 0.5.0. Upgrading to the 0.7.0 engine in Harper v5.4 requires a full local rebuild of every existing 0.5.x index. Prerelease installations using fulltext 0.6 also require rebuilding. Harper initiates the rebuild automatically when the wrapper refuses the old persisted identity; no manual rebuild command or file editing is needed. +Harper 5.3.4 ships `@harperfast/fulltext` 0.5.0. Upgrading to the 0.7.0 engine in Harper v5.4 requires a full local rebuild of every existing 0.5.x index. Prerelease installations using fulltext 0.6 also require rebuilding. This procedure applies to the current field-based declarations; earlier beta declarations and indexes have no supported migration path. Harper initiates the rebuild automatically when the wrapper refuses the old persisted identity; no manual rebuild command or file editing is needed. ### Matching and ranking changes @@ -69,11 +69,11 @@ Tantivy 0.27 changes English stemming even though the schema analyzer remains `e 1. Drain the node's full-text requests and stop all old Harper processes that share its storage, including background workers. Do not run old and new engine versions against one local index. 2. Confirm authoritative records and recovery backups are available. Reserve disk headroom and time for rebuilding every local full-text index, including temporarily retained retired files. There is no universal rebuild-duration or capacity estimate. -3. Start the v5.4 Harper version. Use [`describe_table`](../operations-api/operations.md#describe_table) for each affected table and check every `full_text_indexes[].readiness.state`. Keep full-text traffic drained until all required indexes are `ready`. +3. Start the v5.4 Harper version. Use [`describe_table`](../operations-api/operations.md#describe_table) for each affected table, using an operator account that can read every indexed source field, and check every `full_text_indexes[].readiness.state`. Metadata is permission-filtered; a missing entry does not prove that no index exists. Keep full-text traffic drained until all required indexes are `ready`. 4. Run representative searches with `maxIndexLagMilliseconds: 0` and `waitForIndexMilliseconds: 30000` on each full-text condition. These wait for coverage of writes committed before the search; readiness alone does not establish that coverage. Check response bodies for terminal stream errors as well as HTTP status. A JSON-array response can start with HTTP 200 and end with an `error` element. -5. Retry write responses carrying `DERIVED_INDEX_LAGGING` with bounded backoff while replay catches up. Restore full-text traffic only after readiness and current-coverage searches succeed. A node-by-node upgrade can use other ready replicas only if the application's routing supports it; the rebuilding node remains unavailable for full-text search. +5. Retry write responses carrying `DERIVED_INDEX_LAGGING` with bounded backoff while replay catches up. Restore full-text traffic only after readiness and current-coverage searches succeed. A node-by-node upgrade can use other ready replicas only if the application's routing supports it; the rebuilding node remains unavailable for full-text search. During a mixed-version rollout, matching and ranking can differ between replicas. Route related searches consistently and check the application's behavior before using this approach. -See [freshness controls](./querying.md#freshness-controls) for the condition fields and streaming behavior. The upgrade checks cover bounded workloads; they do not establish multi-node rollout safety, sustained-write capacity, customer relevance or performance parity. Measure those against the application's workload before rollout. +See [freshness controls](./querying.md#freshness-controls) for the condition fields and streaming behavior. Qualify multi-node rollout, sustained-write capacity, relevance and performance against the application's workload before production rollout. A successful rebuild alone does not establish those properties. ### Recovering from a failed upgrade diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 24d71951d..01e84be71 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -73,7 +73,7 @@ Returns the definition of a specific table, including its schema and storage met -A table with native full-text indexes includes `full_text_indexes`; each entry reports its query-only field name, source fields, analyzer options, enabled query modes, and `readiness` (`unknown`, `ready`, `needs-rebuild`, `rebuilding`, or `unavailable`). The readiness object also includes `owner_epoch`, `rebuild_attempts`, and an optional `reason`. Query-only `FullText` fields do not appear in the ordinary `attributes` list. +A table with native full-text indexes includes `full_text_indexes`; each entry reports its query-only field name, source fields, analyzer options, enabled query modes, and `readiness` (`unknown`, `ready`, `needs-rebuild`, `rebuilding`, or `unavailable`). The readiness object also includes `owner_epoch`, `rebuild_attempts`, and an optional `reason`. Query-only `FullText` fields do not appear in the ordinary `attributes` list. Metadata respects attribute permissions: unreadable source fields are omitted, and an index with no readable source fields is omitted entirely. Use an account with access to every indexed source field when checking upgrade readiness; a missing entry does not prove that no index exists. Requesting a table with full-text indexes: From 47e6ae450da7d446ab7d84d4bf9c5a01e5be72f4 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 27 Oct 2026 20:53:01 -0600 Subject: [PATCH 77/78] Clarify streamed search failures and upgrade requirements --- reference/database/transaction.md | 2 +- reference/logging/configuration.md | 2 +- reference/operations-api/operations.md | 2 +- reference/resources/query-optimization.md | 26 +++++++++++------------ reference/rest/querying.md | 2 +- release-notes/v5-lincoln/5.4.md | 6 +++++- 6 files changed, 22 insertions(+), 18 deletions(-) diff --git a/reference/database/transaction.md b/reference/database/transaction.md index 5bed6afb7..7672351a3 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -204,7 +204,7 @@ type Dog @table(audit: true) { } ``` -This overrides the [`logging.auditLog`](../logging/configuration.md) global configuration for that specific table. Keep auditing enabled for tables with full-text indexes: disabling it prevents full-text index registration. +This overrides the [`logging.auditLog`](../logging/configuration.md) global configuration for that specific table. New schema tables with `@fullText` must explicitly declare `@table(audit: true)`, even when global audit logging is enabled. Existing audited tables can add an index without restating `audit`. Keep auditing enabled while any full-text index is declared; disabling it is rejected. ## Related Documentation diff --git a/reference/logging/configuration.md b/reference/logging/configuration.md index 5ef541370..f5f82ea96 100644 --- a/reference/logging/configuration.md +++ b/reference/logging/configuration.md @@ -102,7 +102,7 @@ Default: `true` Enables transaction logging. When enabled, Harper records every insert, update, and delete to the database's transaction log. This history is read with the [`read_audit_log`](../database/transaction.md#read_audit_log) operation — a legacy name; there is a single transaction log, not a separate audit table. -Full-text indexing requires transaction logging. Disabling this setting prevents new tables from inheriting that prerequisite; a full-text table must also have its per-table audit setting enabled. See [Database / Transaction Logging](../database/transaction.md) for details on the transaction log. +Full-text indexing requires transaction logging. New schema tables with `@fullText` must explicitly declare `@table(audit: true)`, even when this global setting is enabled. Existing audited tables can add an index without restating `audit`; disabling table auditing while an index is declared is rejected. See [Database / Transaction Logging](../database/transaction.md) for details on the transaction log. ```yaml logging: diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 56f789b94..56a5559f3 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -73,7 +73,7 @@ Returns the definition of a specific table, including its schema and storage met -A table with native full-text indexes includes `full_text_indexes`; each entry reports its query-only field name, source fields, analyzer options, enabled query modes, and `readiness` (`unknown`, `ready`, `needs-rebuild`, `rebuilding`, or `unavailable`). The readiness object also includes `owner_epoch`, `rebuild_attempts`, and an optional `reason`. Query-only `FullText` fields do not appear in the ordinary `attributes` list. +A table with native full-text indexes includes `full_text_indexes`; each entry reports its query-only field name, source fields, analyzer options, enabled query modes, and `readiness` (`unknown`, `ready`, `needs-rebuild`, `rebuilding`, or `unavailable`). The readiness object also includes `owner_epoch`, `rebuild_attempts`, and an optional `reason`. Query-only `FullText` fields do not appear in the ordinary `attributes` list. Metadata respects attribute permissions: unreadable source fields are omitted, and an index with no readable source fields is omitted entirely. Use an account with access to every indexed source field when checking upgrade readiness; a missing entry does not prove that no index exists. Requesting a table with full-text indexes: diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index b8ea4fd29..46e9e6821 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -93,7 +93,7 @@ For example, with `offset: 20` and `limit: 10`, Harper skips the first 20 eligib Use a bounded `limit` for large result sets. An unbounded search that exceeds its native window, fetched-hit limit or retained-result limit throws 400; use a bounded query. Exhausting the execution-time budget instead throws 503. `offset + limit` must not exceed either `storage.fullText.maxResultsPerRequest` or `storage.fullText.maxRawHitsPerRequest`; replayed or rejected hits can exhaust the raw-work budget earlier. Raising only the raw-hit budget allows more filtering work without raising the retained-record target. Offset requests revisit earlier matches and can retain up to `offset + limit` accepted records. Raising work limits can increase CPU, reads and memory use. -The fetched-hit budget includes the initial batch replay when Harper switches to native traversal. It counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. The adapter does not return a shortened result as if the search had completed. If a custom resource has already started streaming a response, a later query error ends that stream instead of changing its HTTP status. When there are no further native matches, Harper returns the eligible records found, which may be an empty result. This differs from reaching a budget while more matches remain. +The fetched-hit budget includes the initial batch replay when Harper switches to native traversal. It counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. The adapter does not return a shortened result as if the search had completed. If a REST response has already started streaming, a later query error ends that stream instead of changing its HTTP status. JSON-array clients must check for a terminal `error` element and treat the response as failed. For other formats, detect stream or decoding failures; status 200 and count headers alone do not prove completion. See [REST paging](../rest/querying.md#pagination-and-total-count). When there are no further native matches, Harper returns the eligible records found, which may be an empty result. This differs from reaching a budget while more matches remain. Configure the three request limits and six native process budgets under [`storage.fullText`](../configuration/options.md#storage). See [Set Configuration](../configuration/operations.md#set-configuration) for restart requirements. The execution budget is not a hard response-time guarantee: reader acquisition, query construction and some native operations can take additional time. Freshness waiting and reader acquisition use separate bounds. @@ -105,24 +105,24 @@ Replacement workers keep the captured budgets after a configuration change. See ### Readiness and recovery -Full-text indexes require audit logging; a table declared with `@table(audit: false)` cannot register a full-text index. A format-incompatible native upgrade rebuilds the local index from stored records, and queries return 503 while it rebuilds. Harper v5.4 uses `@harperfast/fulltext` 0.7.0 with Tantivy 0.27.0. Indexes from fulltext 0.5.0 (shipped in Harper 5.3.4), as well as prerelease 0.6 indexes, require a full local rebuild. Drain the node's search traffic and stop old processes before starting the new version; do not mix engine versions against one local index. Verify readiness and current coverage before restoring search traffic. Replay after a long rebuild can also cause retryable write-503 responses; retry with bounded backoff while it catches up. Direct downgrade to fulltext 0.5.x is unsupported. +Full-text indexes require audit logging. New schema tables must explicitly declare `@table(audit: true)` when adding `@fullText`, even if global audit logging is enabled; an existing audited table can add an index without restating `audit`. Disabling audit logging while an index is declared is rejected. A format-incompatible native upgrade rebuilds the local index from stored records, and queries return 503 while it rebuilds. Harper v5.4 uses `@harperfast/fulltext` 0.7.0 with Tantivy 0.27.0. Indexes from fulltext 0.5.x using current field-based declarations (including fulltext 0.5.0 shipped in Harper 5.3.4), as well as prerelease 0.6 indexes, require a full local rebuild. Earlier beta declarations and indexes have no supported migration path. Drain the node's search traffic and stop old processes before starting the new version; do not mix engine versions against one local index. Verify readiness and current coverage before restoring search traffic. Replay after a long rebuild can also reject ordinary local writes with 503, `code: "DERIVED_INDEX_LAGGING"`, and `retryable: true`; retry with bounded backoff while it catches up. Replication and crash-recovery replay continue. Direct downgrade to fulltext 0.5.x is unsupported: that engine refuses opening or resetting a 0.7 index identity. Stop an accidentally started old version and restore the 0.7.0-based Harper version, keeping search traffic drained until readiness and coverage recover. Do not edit native index files. Missing retained transaction history can require a full rebuild, as described below; this is current-engine recovery, not a general Harper downgrade procedure. English stemming also changes: `added` now stems to `add` rather than `ad`, so review matching and ranking against representative searches. -| Failure | HTTP status | Code and retry guidance | -| --------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | -| Raw-hit or retained-result budget | 400 | No readiness code. Narrow the query or adjust the named budget. | -| Execution budget | 503 | No readiness code. Narrow the query; the same expensive query can fail again. | -| Native busy | 503 | No readiness code. Reduce concurrency or retry with backoff. | -| Index unknown, needs-rebuild or rebuilding | 503 | `INDEX_REBUILDING`, with `retryable: true`, for a not-ready readiness check. Retry with backoff while the index rebuilds. | -| Coverage lag or wait timeout | 503 | `DERIVED_INDEX_LAGGING`, with `retryable: true`. Retry with backoff while the index catches up. | -| Index unavailable | 503 | No readiness-specific code or retryable flag. Inspect readiness and logs; fix the reported problem before retrying. | -| Query host becomes non-ready during a coverage wait | 503 | Can return no readiness-specific code or retryable flag. Inspect readiness before deciding whether to retry. | -| Worker loaded native search before runtime capture | 500 | REST reports `FullTextWorkerInitializationError`; the Operations API can omit that code. Non-retryable; recreate the worker after initialization. | +| Failure | HTTP status | Code and retry guidance | +| ------------------------------------------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | +| Raw-hit or retained-result budget | 400 | No readiness code. Narrow the query or adjust the named budget. | +| Execution budget | 503 | No readiness code. Narrow the query; the same expensive query can fail again. | +| Native busy | 503 | No readiness code. Reduce concurrency or retry with backoff. | +| Index unknown, needs-rebuild or rebuilding | 503 | `INDEX_REBUILDING`, with `retryable: true`, for a not-ready readiness check. Retry with backoff while the index rebuilds. | +| Coverage lag, wait timeout or local write backpressure | 503 | `DERIVED_INDEX_LAGGING`, with `retryable: true`. Retry with backoff while the index catches up. | +| Index unavailable | 503 | No readiness-specific code or retryable flag. Inspect readiness and logs; fix the reported problem before retrying. | +| Query host becomes non-ready during a coverage wait | 503 | Can return no readiness-specific code or retryable flag. Inspect readiness before deciding whether to retry. | +| Worker loaded native search before runtime capture | 500 | REST reports `FullTextWorkerInitializationError`; the Operations API can omit that code. Non-retryable; recreate the worker after initialization. | A native result that exceeds the response-byte envelope can return 413, including an oversized ordinary batch even when each individual hit would fit. Other native failures and invalid query configuration can return 500; inspect the error and server logs rather than assuming they are retryable. The table lists common failures, not every possible error. For read-after-write queries, `maxIndexLagMilliseconds: 0` requires current coverage. `waitForIndexMilliseconds` lets a query wait for coverage (default `0`, maximum `30000` ms); without an override, the allowed index lag is `3000` ms. These options belong to the full-text condition. They wait for coverage, not for an unavailable index to recover. Readiness is checked first; a query host that becomes rebuilding or unavailable during wait setup can fail immediately with a bare 503. -Concurrent requests each have their own work budget. These limits do not cap the number of concurrent Harper queries or prevent brief event-loop delays during synchronous source reads. Measure deep offsets and selective filters under the expected concurrency before raising budgets. +Concurrent requests each have their own work budget. These per-request budgets do not cap the number of concurrent Harper queries or prevent brief event-loop delays during synchronous source reads. Measure deep offsets and selective filters under the expected concurrency before raising budgets. `INDEX_REBUILDING` and `DERIVED_INDEX_LAGGING` identify retryable readiness or coverage failures. Retry with backoff while the index makes progress. Transaction history retained under [`logging.auditRetention`](../logging/configuration.md#loggingauditretention) (default `3d`) is used for catch-up. Lost history can require a rebuild; retries alone do not repair an index that cannot catch up. Use [`describe_table`](../operations-api/operations.md#describe_table) to inspect `full_text_indexes[].readiness`, including its state and reason. Correct the reported configuration or storage problem before retrying an unavailable index. Budget and native-busy failures do not carry those readiness codes. A native busy response also returns 503; reduce concurrency or retry with backoff. For an execution-budget failure, narrow the query, reduce the offset or adjust the work budget instead of repeatedly submitting the same expensive query. diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 94b300ec4..15021891f 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -183,7 +183,7 @@ Use `limit(start,end)` to page through a collection, and opt in to a total match Counting is opt-in: without the header, no count is computed and no count headers are returned. It applies only to `GET`/`HEAD` requests and requires a bounded page — the `limit()` must be a non-negative integer no larger than **10,000**, and the requested window (offset + limit) no larger than **1,000,000**. A request with no `limit()`, one outside those bounds, or a non-numeric one is served normally with no count headers, since counting an unbounded page would defeat the point of paging. -Full-text and autocomplete queries use the same paging syntax. The page window counts eligible records after record filtering, rather than raw full-text hits. Harper fetches more ranked batches to fill that window when needed. A page can contain fewer records when no further matches remain; reaching a search work or time limit produces an error instead of a successful partial page. Each page request runs a new query, so concurrent changes can cause repeats or omissions between pages. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for search work limits and recovery guidance. +Full-text and autocomplete queries use the same paging syntax. The page window counts eligible records after record filtering, rather than raw full-text hits. Harper fetches more ranked batches to fill that window when needed. A page can contain fewer records when no further matches remain; reaching a search work or time limit fails the query. A failure detected before response headers can set an error HTTP status. After streaming starts, the response can retain HTTP 200 and end with an error. For JSON-array responses, check for a terminal `error` element and treat the response as failed; do not render that element as a record. For other formats, detect stream or decoding failures as well. Neither status 200 nor count headers prove that a page completed successfully. Each page request runs a new query, so concurrent changes can cause repeats or omissions between pages. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for search work limits and recovery guidance. The count-header guardrails do not override query-specific limits such as the full-text work budget. diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index de85f0611..3e77b78a3 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -8,6 +8,10 @@ title: '5.4' All patch release notes for 5.4.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.4&expanded=true). +## Upgrade Notes + +Full-text indexes using the current 0.5.x declarations or prerelease 0.6 rebuild locally after this upgrade. Plan for search unavailability and review the [Tantivy upgrade requirements](#tantivy-027-and-existing-index-rebuilds) before deploying; direct native-engine downgrade is unsupported. + ## Replication ### Dedicated Replication Threads @@ -72,7 +76,7 @@ The new `threads.v8Flags` setting takes a V8 flag, or a list of them, and applie ### Tantivy 0.27 and Existing Index Rebuilds -Harper now uses `@harperfast/fulltext` 0.7.0 with Tantivy 0.27.0. Existing fulltext 0.5.x indexes, including those created by Harper 5.3.4, and prerelease 0.6 indexes rebuild automatically from authoritative records. Plan for local full-text unavailability, disk headroom and possible retryable write failures while replay catches up. Stop old processes before starting the new version against the same storage; direct downgrade to fulltext 0.5.x is unsupported. +Harper now uses `@harperfast/fulltext` 0.7.0 with Tantivy 0.27.0. Existing fulltext 0.5.x indexes using current field-based declarations, including fulltext 0.5.0 shipped in Harper 5.3.4, and prerelease 0.6 indexes rebuild automatically from authoritative records. Earlier beta declarations and indexes have no supported migration path. Plan for local full-text unavailability, disk headroom and possible retryable write failures while replay catches up. Stop old processes before starting the new version against the same storage; direct downgrade to fulltext 0.5.x is unsupported. English stemming changes despite the unchanged `english@2` analyzer name: `added` now stems to `add` rather than `ad`. Matching and ranking can change. Query modes and schema options are unchanged; benchmark results from the wrapper do not establish application performance parity. Review representative searches and see [readiness and recovery](/reference/v5/resources/query-optimization#readiness-and-recovery) before restoring traffic. From 3640af2652ae1357f22824992a0207a8de80dbf7 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 27 Oct 2026 20:54:50 -0600 Subject: [PATCH 78/78] Scope freshness waits and unsupported downgrade guidance --- reference/resources/query-optimization.md | 2 +- release-notes/v5-lincoln/5.4.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index 46e9e6821..b52a37288 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -120,7 +120,7 @@ Full-text indexes require audit logging. New schema tables must explicitly decla A native result that exceeds the response-byte envelope can return 413, including an oversized ordinary batch even when each individual hit would fit. Other native failures and invalid query configuration can return 500; inspect the error and server logs rather than assuming they are retryable. The table lists common failures, not every possible error. -For read-after-write queries, `maxIndexLagMilliseconds: 0` requires current coverage. `waitForIndexMilliseconds` lets a query wait for coverage (default `0`, maximum `30000` ms); without an override, the allowed index lag is `3000` ms. These options belong to the full-text condition. They wait for coverage, not for an unavailable index to recover. Readiness is checked first; a query host that becomes rebuilding or unavailable during wait setup can fail immediately with a bare 503. +For read-after-write queries, `maxIndexLagMilliseconds: 0` requires coverage of writes already committed on the queried node. It does not wait for replication to deliver writes from another node. `waitForIndexMilliseconds` lets a query wait for coverage (default `0`, maximum `30000` ms); without an override, the allowed index lag is `3000` ms. These options belong to the full-text condition. They wait for coverage, not for an unavailable index to recover. Readiness is checked first; a query host that becomes rebuilding or unavailable during wait setup can fail immediately with a bare 503. Concurrent requests each have their own work budget. These per-request budgets do not cap the number of concurrent Harper queries or prevent brief event-loop delays during synchronous source reads. Measure deep offsets and selective filters under the expected concurrency before raising budgets. diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 3e77b78a3..d39a10eb6 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -10,7 +10,7 @@ All patch release notes for 5.4.x are available on the [releases page](https://g ## Upgrade Notes -Full-text indexes using the current 0.5.x declarations or prerelease 0.6 rebuild locally after this upgrade. Plan for search unavailability and review the [Tantivy upgrade requirements](#tantivy-027-and-existing-index-rebuilds) before deploying; direct native-engine downgrade is unsupported. +Full-text indexes using the current 0.5.x declarations or prerelease 0.6 rebuild locally after this upgrade. Plan for search unavailability and review the [Tantivy upgrade requirements](#tantivy-027-and-existing-index-rebuilds) before deploying; direct downgrade to fulltext 0.5.x is unsupported. ## Replication