diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6e557a183..b2588a482 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -235,7 +235,7 @@ The Plugin API is the primary way to implement additional functionality in Harpe - `logger.level` — Log level; _Default_: `"info"` (Added in: v4.1.0) ``` -For inline config option annotations inside list items, plain text `(Added in: vX.Y.Z)` is fine — using the component mid-sentence is awkward. Reserve `` for standalone placement after headings. +Use `` after headings or alongside the specific sentence, list item, or table entry it annotates. Use `` the same way when behavior is storage-engine-specific; comma-separated values such as `engines="RocksDB, LMDB"` are supported. Keep adjacent version and engine badges together on one line. For dense prose where a component would interrupt the sentence, plain text such as `(Added in: vX.Y.Z)` is fine. ## Redirects diff --git a/learn/developers/full-text-search-product-catalog.mdx b/learn/developers/full-text-search-product-catalog.mdx new file mode 100644 index 000000000..749af6287 --- /dev/null +++ b/learn/developers/full-text-search-product-catalog.mdx @@ -0,0 +1,193 @@ +--- +title: Add Full-Text Search to a Product Catalog +description: Configure and query a BM25-ranked product search in Harper. +--- + +# Add Full-Text Search to a Product Catalog + + + +This guide adds relevance-ranked text search to a product catalog, then exposes it through Harper's JavaScript and REST APIs. + +## What you will build + +- A BM25-ranked index over product names, descriptions, and tags +- A Harper application endpoint with highlights and a structured category filter +- REST queries for terms, phrases, and record autocomplete + +## Prerequisites + +- Harper 5.3 or later using RocksDB +- A component with a GraphQL schema, JavaScript resources, and `rest: true` in its `config.yaml` + +## 1. Declare the index + +Declare an `@fullText` index over the fields customers search on a RocksDB-backed table. Ordinary tables inherit enabled audit logging by default, but a new full-text table requires explicit `audit: true` for its transaction-log recovery source. See the [audit requirement](/reference/v5/full-text-search/configuration) for existing tables. + +```graphql title="schema.graphql" +type Product @table(database: "catalog", audit: true) @export { + id: ID @primaryKey + name: String + description: String + tags: [String] + category: String @indexed + price: Float @indexed + available: Boolean + catalogSearch: FullText + @fullText( + fields: [ + { name: "name", weight: 3, highlight: true } + { name: "description", weight: 1, highlight: true } + { name: "tags" } + ] + highlighting: { maxFragments: 2, fragmentLength: 120 } + ) +} +``` + +The query-only field name `catalogSearch` identifies the index. It is not stored with each product. `weight: 3` makes a match in `name` contribute more to BM25 relevance than the same match in `description` or `tags`. Highlighting is enabled only for the fields shown to customers. + +The `@indexed` declarations create ordinary Harper secondary indexes. Full-text searches can use them to narrow candidates, and Harper checks structured conditions against current product records before returning results. + +## 2. Add records + +Use the generated REST endpoint: + +```http +POST /Product/ +Content-Type: application/json + +{ + "id": "shoe-1", + "name": "Waterproof trail running shoe", + "description": "Lightweight shoe for wet mountain trails", + "tags": ["outdoor", "trail"], + "category": "footwear", + "price": 129, + "available": true +} +``` + +Harper commits the record first, then applies the committed change to the local derived full-text index. Each replica performs the same derivation after it has received both the full-text schema declaration and the record transactions. + +If the index remains behind for more than 30 seconds, a local write can return retryable `503` code `DERIVED_INDEX_LAGGING`. Retry writes with backoff while the index catches up; freshness options apply only to searches. See [write backpressure](/reference/v5/full-text-search/overview#write-backpressure). + +## 3. Search from a Harper application + +```javascript title="resources.js" +export class SearchProducts extends Resource { + static async post(_target, data, context) { + if (!context?.user) { + const error = new Error('Authentication required'); + error.statusCode = 401; + throw error; + } + const body = (await data) ?? {}; + const text = typeof body.q === 'string' ? body.q.trim() : ''; + if (!text) { + const error = new Error('q is required'); + error.statusCode = 400; + throw error; + } + const conditions = [ + { + attribute: 'catalogSearch', + comparator: 'matches', + value: text, + fields: ['name', 'description', 'tags'], + }, + { attribute: 'available', comparator: 'equals', value: true }, + ]; + const category = body.category; + if (category) conditions.push({ attribute: 'category', comparator: 'equals', value: category }); + return databases.catalog.Product.search( + { + conditions, + select: ['id', 'name', 'price', '$score', '$highlights'], + limit: 20, + checkPermission: true, + }, + context + ); + } +} +``` + +Call the resource with an authenticated request: + +```http +POST /SearchProducts/ +Content-Type: application/json + +{ "q": "trail", "category": "footwear" } +``` + +This custom resource accepts a structured JSON search body through POST. The category filter is optional; `q` is required. For URL-based GET searches, use the exported table endpoint in step 4. + +Harper applies the availability and optional category conditions while accepting current product records. Existing secondary indexes can narrow candidates; rejected records do not count toward the requested page. + +Calls through `tables` and `databases` use Harper's [trusted server-side context](/reference/v5/components/javascript-environment#tables). Forwarding the resource's `context` preserves the authenticated caller, and `checkPermission: true` asks the table to enforce that caller's table and source-field permissions. The custom resource must do both. Highlights can expose searched source text even when those fields are absent from `select`. + +Matches in `tags` contribute to search results and ranking, but `$highlights` does not include tag matches because `tags` is not configured with `highlight: true`. + +## 4. Search through REST + +The exported table supports the same match comparator: + +```http +GET /Product/?catalogSearch=matches=waterproof%20trail&category=footwear&select(id,name,price,$score,$highlights)&limit(20) +``` + +For a phrase: + +```http +GET /Product/?catalogSearch=matches_phrase=trail%20running&limit(20) +``` + +For record autocomplete: + +```http +GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&select(id,name)&limit(10) +``` + +Prefix search returns matching products. It does not return a dictionary of suggested terms. Before v5.4, prefix and fuzzy-prefix queries restrict `offset + limit` to 100. In v5.4, bounded pages can go beyond that window within request budgets; each page runs a new query rather than retaining a search. See [paging and prefix result windows](/reference/v5/full-text-search/querying#paging-and-prefix-result-windows). + +REST searches every source field in the index, so the caller must be allowed to read `name`, `description`, and `tags` for these requests. Use `Table.search()` or `search_by_conditions` when a role should search only a subset of sources. See [REST full-text operators](/reference/v5/rest/querying#full-text-operators). + +## 5. Choose consistency behavior + +Most searches can use the default three-second lag tolerance. When the next request must find a record already committed on the queried node, use the JavaScript API and wait for current index coverage. This wait does not wait for replication to deliver writes from another node: + +```javascript +for await (const product of databases.catalog.Product.search({ + conditions: [ + { + attribute: 'catalogSearch', + comparator: 'matches', + value: 'new seasonal product', + maxIndexLagMilliseconds: 0, + waitForIndexMilliseconds: 10000, + }, + ], +})) { + console.log(product); +} +``` + +Do not apply this wait to every catalog request without measuring it. The normal bounded-staleness path keeps search traffic independent of short indexing bursts. + +If the index does not reach current coverage within ten seconds, iteration throws an error with `statusCode: 503`, `code: "DERIVED_INDEX_LAGGING"`, and `retryable: true`. See [freshness controls](/reference/v5/full-text-search/querying#freshness-controls). + +## 6. Operate the index + +Use `describe_table` to inspect readiness. Harper reuses compatible local index files after restart and replays changes after their checkpoint. A node rebuilds locally when the files are missing, incompatible, or corrupt; ordinary table reads remain available, while full-text requests in `unknown`, `needs-rebuild`, or `rebuilding` state return `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. Retry this code with backoff; do not treat every `503` as an index rebuild. Writes can also receive retryable `DERIVED_INDEX_LAGGING` responses while replay catches up after a long rebuild; see [write backpressure](/reference/v5/full-text-search/overview#write-backpressure). + +The custom resource in step 3 streams its returned iterable. HTTP error statuses apply before response headers are sent; a failure after streaming starts is a terminal stream error and cannot change the HTTP status. Consumers must also handle that terminal error. See [freshness and streaming errors](/reference/v5/full-text-search/querying#freshness-controls). + +See the [full-text configuration reference](/reference/v5/full-text-search/configuration) and [query reference](/reference/v5/full-text-search/querying) for every option and match mode. + +## Additional resources + +- [Full-text search overview](/reference/v5/full-text-search/overview) +- [Resource API](/reference/v5/resources/resource-api) +- [Query optimization](/reference/v5/resources/query-optimization) diff --git a/reference/backups/overview.md b/reference/backups/overview.md index e0f55f7d7..659683498 100644 --- a/reference/backups/overview.md +++ b/reference/backups/overview.md @@ -5,7 +5,7 @@ title: Backups -Harper can back up and restore its databases natively. A backup is a whole-database copy — all tables, the audit/transaction log, and any file-backed blobs — so a restored database is a consistent point-in-time image of everything in it. (`get_backup` on an LMDB database is the exception: it can stream a subset of tables, and includes the audit store only when requested.) +Harper can back up and restore its databases natively. A backup is a whole-database copy of authoritative state — all tables, the audit/transaction log, and any file-backed blobs — so a restored database is a consistent point-in-time image. Local derived state, including full-text index files, is excluded and rebuilt after restore. (`get_backup` on an LMDB database is the exception: it can stream a subset of tables, and includes the audit store only when requested.) Two complementary mechanisms are available: @@ -38,6 +38,7 @@ The offline path matters for restore. RocksDB is single-writer, so an in-place r - **Backups live on the node that created them.** The backup repository is a local directory. RocksDB shares files across backup IDs (a backup ID is not a self-contained folder), so disaster-recovery copies must take the entire per-database repository — `/` — not an individual backup, and must do so while no backup operation is running, or from an atomic filesystem snapshot. A live recursive copy can race `create_backup`/`delete_backup`/`purge_backups` and produce an unrestorable copy. Alternatively, use `get_backup` to pull a snapshot from a running server. - **`get_backup` always streams the current state.** It cannot download a historical managed backup; to move a retained backup off-host, copy its whole per-database repository as above. - **A restore is a point-in-time rollback.** In a replicated cluster, coordinate a restore with replication before bringing the node back. +- **Full-text indexes rebuild locally after restore.** Ordinary table reads remain available. Writes can receive retryable `DERIVED_INDEX_LAGGING` responses while replay catches up after a long rebuild; see [write backpressure](../full-text-search/overview.md#write-backpressure). While the rebuild is active, full-text requests return `503` with `code: "INDEX_REBUILDING"` and `retryable: true`; branch on the code because other `503` responses are not necessarily rebuilds. - **A restore checks the backup before it deletes anything.** A backup with a corrupt database file, or one this version of Harper cannot read, is refused and the database is left as it was (a database an earlier restore left incomplete stays blocked). The check reads the whole backup, which adds to how long an online restore keeps the database unavailable. Blob file contents are not checked. See [`restore_backup`](./operations.md#restore_backup). - **An interrupted restore leaves the database unloadable.** If `restore_backup` is interrupted after it has started replacing the database's files (crash, power loss), Harper marks the database as incompletely restored and skips loading it on the next start, logging an incomplete-restore error. Rerun `restore_backup` for the same database and `backup_id` to recover; do not load or hand-repair the directory. diff --git a/reference/components/javascript-environment.md b/reference/components/javascript-environment.md index 78f626b60..2a2d291f4 100644 --- a/reference/components/javascript-environment.md +++ b/reference/components/javascript-environment.md @@ -90,7 +90,7 @@ An object whose properties are the tables in the default database (`data`). Each See [Database API](../database/api.md) for full reference. :::note Server-side table reads run in a trusted context -Calls to `tables.X.get()`, `tables.X.search()`, and equivalent `databases..` methods from within resource code execute in a **trusted system context** and do **not** re-apply the target table's `allowRead` guard or role-level `attribute_permissions`. This is by design — server-side code is responsible for its own authorization checks. A computed attribute or custom resource that cross-reads a protected table can return raw data regardless of the caller's role. Do not expose protected data through a computed attribute when access depends on the caller. Instead, use a custom resource and apply authorization explicitly (e.g. call `this.getCurrentUser()` in a `Resource` method, or use `getUser()`, and enforce the relevant restrictions before returning the data). +Calls to `tables.X.get()`, `tables.X.search()`, and equivalent `databases..
` methods from within resource code execute in a **trusted system context** and do **not** re-apply the target table's `allowRead` guard or role-level `attribute_permissions` by default. This is by design — server-side code is responsible for its own authorization checks. A custom resource can explicitly re-enter table and source-field authorization for `Table.search()` by forwarding its authenticated `context` and setting `checkPermission: true` in search options. Never accept or copy that flag or a permission object from client data. Context propagation alone does not request the check. Other direct Resource methods also support the [checkPermission option](../resources/resource-api.md#checkpermission); use it with authenticated context when delegating those calls, or enforce the required authorization in application code. A computed attribute or custom resource that cross-reads a protected table can otherwise return raw data regardless of the caller's role. Do not expose protected data through a computed attribute when access depends on the caller. Instead, use a custom resource and enforce the relevant restrictions before returning the data. ::: ### `databases` diff --git a/reference/configuration/operations.md b/reference/configuration/operations.md index 75d9ed695..c10e66715 100644 --- a/reference/configuration/operations.md +++ b/reference/configuration/operations.md @@ -16,7 +16,9 @@ For the full list of configurable options, see [Configuration Options](./options ## Set Configuration -Modifies one or more Harper configuration parameters. **Requires a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`). A change to `replication.threads` takes effect only on a full `restart`. +Modifies one or more Harper configuration parameters. **Most changes require a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`). A change to `replication.threads` takes effect only on a full `restart`. + +Full-text request work limits (`storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds`) take effect in HTTP workers after `restart_service` or a full restart. Changes to `storage.fullText.runtime` require a full Harper process restart; `restart_service` does not replace the native process budget. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). `operation` _(required)_ — must be `set_configuration` @@ -24,6 +26,8 @@ Modifies one or more Harper configuration parameters. **Requires a [restart](../ Additional properties correspond to configuration keys in underscore-separated format (e.g. `logging_level` for `logging.level`, `clustering_enabled` for `clustering.enabled`). +For full-text settings, use individual parameter names such as `storage_fullText_maxRawHitsPerRequest` or `storage_fullText_runtime_maxSearchThreads`. `set_configuration` does not accept a whole `storage_fullText` object; use `storage.fullText` as the section name only in the YAML configuration file. + :::warning Replicate cluster-appropriate parameters only A replicated call sends the exact same values to every node. Do not include node-local parameters — ports, `node.hostname`, `rootPath`, storage/log file paths, TLS certificate or key paths, or `replication.hostname`/`url`/`routes` — as they would overwrite each peer's local values. ::: @@ -42,7 +46,7 @@ A replicated call sends the exact same values to every node. Do not include node ```json { - "message": "Configuration successfully set. You must restart HarperDB for new config settings to take effect." + "message": "Configuration successfully set. You must restart Harper for new config settings to take effect." } ``` diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 961f10592..edf7f438d 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -232,7 +232,7 @@ logging: level: warn root: ~/hdb/log stdStreams: false - auditLog: false + auditLog: true rotation: interval: 1D maxSize: 100M @@ -244,7 +244,7 @@ logging: - `path` — Explicit log file path (overrides `root`) - `stdStreams` — Write to stdout/stderr; _Default_: `false` - `console` — Include `console.*` output; _Default_: `true` -- `auditLog` — Enable table transaction audit logging; _Default_: `false` +- `auditLog` — Enable table transaction audit logging; _Default_: `true` - `auditRetention` — Audit log retention duration; _Default_: `3d` - `external` — Logging for components using the logger API; sub-options: `level`, `path` - `rotation.enabled` / `rotation.compress` / `rotation.interval` / `rotation.maxSize` / `rotation.path` — Log file rotation (activates when `interval` or `maxSize` is set) @@ -312,6 +312,19 @@ storage: ``` - `engine` — The database storage engine to use for new databases. Currently supported engines are `rocksdb` and `lmdb`. The default is `rocksdb`. Existing databases will use the engine that was used when they were created. +- `fullText` — Full-text search work and native process limits. Sub-options: + - `maxRawHitsPerRequest` — Maximum fetched hits per request, including bounded prefix replay when traversal is needed; default `10000`. + - `maxResultsPerRequest` — Maximum retained offset plus limit. A request must also fit the separate raw-hit budget; default `10000`. + - `requestBudgetMilliseconds` — Execution budget across native batches, source checks and highlights, capped by the native execution limit; default `5000` ms. Reader acquisition and freshness waiting have separate bounds. + - `runtime.maxResidentIndexes` — Maximum resident native indexes; default `128`. + - `runtime.maxIndexingThreads` — Maximum aggregate indexing-thread reservations across resident native indexes; default `128`. + - `runtime.maxSearchThreads` — Maximum aggregate search-thread reservations across resident native indexes; default `256`. + - `runtime.maxWriterMemoryBytes` — Aggregate writer-memory reservations across resident native indexes; default `4294967296` (4 GiB). + - `runtime.maxQueuedBytes` — Aggregate queued-write reservations across resident native indexes; default `8589934592` (8 GiB). + - `runtime.maxExpensiveSearches` — Maximum concurrent expensive native searches; default `8`. + + The runtime budgets apply together and are not hard bounds on total process memory. See [Set Configuration](./operations.md#set-configuration) for when request and native runtime changes take effect. Use the usual underscore-separated parameter names, such as `storage_fullText_maxRawHitsPerRequest`. See [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for costs; see [readiness and recovery](../resources/query-optimization.md#readiness-and-recovery) for failures. + - `writeAsync` — Disable disk sync for higher throughput (**disables durability guarantees**); _Default_: `false` - `caching` — In-memory record caching; _Default_: `true` - `compression` — LZ4 record compression; _Default_: `true` (enabled by default since v4.3.0). Sub-options: `dictionary`, `threshold` diff --git a/reference/configuration/overview.md b/reference/configuration/overview.md index 112327b24..033c8006c 100644 --- a/reference/configuration/overview.md +++ b/reference/configuration/overview.md @@ -21,7 +21,7 @@ For a complete reference of all available configuration options, see [Configurat ## The Configuration File -To change a configuration value, edit `harper-config.yaml` and save. **Harper must be restarted for changes to take effect.** +To change a configuration value, edit `harper-config.yaml` and save. **Restart Harper to apply file edits.** See [Set Configuration](./operations.md#set-configuration) for settings that require a full process restart. Configuration keys use camelCase (e.g. `operationsApi`). Nested keys use dot notation conceptually (e.g. `operationsApi.network.port`). diff --git a/reference/database/overview.md b/reference/database/overview.md index 406b251a9..f4257eec7 100644 --- a/reference/database/overview.md +++ b/reference/database/overview.md @@ -107,6 +107,7 @@ It is best practice to define schemas for production tables. Dynamic schemas are For deeper coverage of each database feature, see the dedicated pages in this section: - **[Schema](./schema.md)** — Defining table structure, types, indexes, relationships, and computed properties using GraphQL schema syntax +- **[Full-Text Search](../full-text-search/overview.md)** — BM25-ranked search over stored text fields on audited RocksDB tables - **[API](./api.md)** — The `tables`, `databases`, `transaction()`, and `createBlob()` globals for interacting with the database from code - **[Data Loader](./data-loader.md)** — Loading seed or initial data into tables as part of component deployment - **[Storage Algorithm](./storage-algorithm.md)** — How Harper stores data using LMDB with universal indexing and ACID compliance diff --git a/reference/database/schema.md b/reference/database/schema.md index a0faa9172..a3a54753a 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -73,18 +73,19 @@ type MyTable @table { Optional arguments: -| Argument | Type | Default | Description | -| ------------------------ | --------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | -| `table` | `String` | type name | Override the table name | -| `database` | `String` | `"data"` | Database to place the table in | -| `expiration` | `Int` | — | Seconds until a record goes stale (useful for caching tables) | -| `eviction` | `Int` | `0` | Additional seconds after `expiration` before a record is physically removed | -| `scanInterval` | `Int` | `(expiration + eviction) / 4` | Seconds between eviction scans | -| `replicate` | `Boolean` | true | Enable replication of this table | -| `replicaExpiration` | `Float` | `3600` | Seconds a record fetched from another node is kept as a [local replica](#replicaexpiration-and-replicarefreshfraction); `0` stops new admissions | -| `replicaRefreshFraction` | `Float` | `0.5` | Fraction of `replicaExpiration` after which a read of a replica renews its lifetime; `1` never renews | -| `cacheControl` | `String` | — | `Cache-Control` header value emitted on anonymous GET/HEAD 200/304 responses for this table | -| `randomAccessFields` | `Boolean` | `storage.randomAccessFields` | [Pin this table's record encoding](#randomaccessfields) | +| Argument | Type | Default | Description | +| ------------------------ | --------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | +| `table` | `String` | type name | Override the table name | +| `database` | `String` | `"data"` | Database to place the table in | +| `expiration` | `Int` | — | Seconds until a record goes stale (useful for caching tables) | +| `eviction` | `Int` | `0` | Additional seconds after `expiration` before a record is physically removed | +| `scanInterval` | `Int` | `(expiration + eviction) / 4` | Seconds between eviction scans | +| `replicate` | `Boolean` | true | Enable replication of this table | +| `replicaExpiration` | `Float` | `3600` | Seconds a record fetched from another node is kept as a [local replica](#replicaexpiration-and-replicarefreshfraction); `0` stops new admissions | +| `replicaRefreshFraction` | `Float` | `0.5` | Fraction of `replicaExpiration` after which a read of a replica renews its lifetime; `1` never renews | +| `audit` | `Boolean` | `logging.auditLog` for ordinary tables | Enable the table's transaction log; new tables with `@fullText` require explicit `true` | +| `cacheControl` | `String` | — | `Cache-Control` header value emitted on anonymous GET/HEAD 200/304 responses for this table | +| `randomAccessFields` | `Boolean` | `storage.randomAccessFields` | [Pin this table's record encoding](#randomaccessfields) | **`expiration`, `eviction`, and `scanInterval`** @@ -388,6 +389,24 @@ Write semantics: Multiple `@embed` attributes on one type are computed concurrently. +### `@fullText` + + + +Creates a BM25-ranked full-text derived index from one or more stored text fields. Declare the directive on a nullable `FullText` field; that field's name identifies the index. + +```graphql +type Product @table(database: "catalog", audit: true) { + id: ID @primaryKey + name: String + description: String + tags: [String] + catalogSearch: FullText @fullText(fields: [{ name: "name", weight: 3 }, { name: "description" }, { name: "tags" }]) +} +``` + +The `FullText` field is query-only. It is not stored with records or exposed as a selectable record property. Add another `FullText` field to create another independent index. Full-text indexes require RocksDB and an audited table. They are local derived state rebuilt from committed table data rather than authoritative record storage. See [Full-Text Search Configuration](../full-text-search/configuration.md) for the complete field contract, structured filters, phrase and prefix storage, synonyms, highlighting, and rebuild behavior. + ### `@decide` @@ -630,7 +649,7 @@ Computed properties that read other tables use the same [trusted server-side aut ### Computed Indexes -Computed properties can be indexed with `@indexed`, enabling custom indexing strategies such as composite indexes, full-text search, or vector indexing: +Computed properties can be indexed with `@indexed`, enabling custom lookup strategies such as composite keys. Use [`@fullText`](#fulltext) for native BM25-ranked text search and an HNSW `[Float]` field for vector indexing. The following computed secondary index splits a comma-separated string: ```graphql type Product @table { @@ -889,19 +908,22 @@ Graph navigation runs on the quantized (approximate) distances. For nearest-neig Harper supports the following field types: -| Type | Description | -| --------- | ---------------------------------------------------------------------------------------------- | -| `String` | Unicode text, UTF-8 encoded | -| `Int` | 32-bit signed integer (−2,147,483,648 to 2,147,483,647) | -| `Long` | 54-bit signed integer (−9,007,199,254,740,992 to 9,007,199,254,740,992) | -| `Float` | 64-bit double precision floating point | -| `BigInt` | Integer up to ~300 digits. Note: distinct JavaScript type; handle appropriately in custom code | -| `Boolean` | `true` or `false` | -| `ID` | String; indicates a non-human-readable identifier | -| `Any` | Any primitive, object, or array | -| `Date` | JavaScript `Date` object | -| `Bytes` | Binary data as `Buffer` or `Uint8Array` | -| `Blob` | Binary large object; designed for streaming content >20KB | +| Type | Description | +| ---------- | ---------------------------------------------------------------------------------------------- | +| `String` | Unicode text, UTF-8 encoded | +| `Int` | 32-bit signed integer (−2,147,483,648 to 2,147,483,647) | +| `Long` | 54-bit signed integer (−9,007,199,254,740,992 to 9,007,199,254,740,992) | +| `Float` | 64-bit double precision floating point | +| `BigInt` | Integer up to ~300 digits. Note: distinct JavaScript type; handle appropriately in custom code | +| `Boolean` | `true` or `false` | +| `ID` | String; indicates a non-human-readable identifier | +| `Any` | Any primitive, object, or array | +| `Date` | JavaScript `Date` object | +| `Bytes` | Binary data as `Buffer` or `Uint8Array` | +| `Blob` | Binary large object; designed for streaming content >20KB | +| `FullText` | Query-only full-text index declaration; valid only with `@fullText` and never stored | + +Added `FullText` in v5.3.0 Added `BigInt` in v4.3.0 diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index 9cade5b28..bf2a99806 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -202,7 +202,7 @@ Changing this setting is safe for existing data — records already written keep To pin one table's encoding regardless of the global setting, use the [`@table(randomAccessFields:)`](./schema.md#randomaccessfields) directive when creating the table; editing the directive later does not repin an existing table. Tables that carry the directive ignore this option entirely, which also means a fleet-wide change to this setting will not move them. -The typed dictionary is bounded, and reaching the bound is not an error: records with novel shapes past that point still write and read correctly, they are simply stored without random-access field encoding. [`describe_table`](../operations-api/operations.md#describe_table) reports both the current size and the ceiling in force (`typed_structure_count` against `typed_structure_limit`, 256 by default), which is the reliable way to see whether a table's shapes are staying within it. The classic dictionary is bounded far lower — 32 shared named-record structures — so a table with more than 32 distinct field-name sets sits at that number as a matter of course; it is a normal reading, not a leak. +The typed dictionary is bounded, and reaching the bound is not an error: records with novel shapes past that point still write and read correctly, they are simply stored without random-access field encoding. [`describe_table`](../operations-api/operations.md#record-structure-dictionaries) reports both the current size and the ceiling in force (`typed_structure_count` against `typed_structure_limit`, 256 by default), which is the reliable way to see whether a table's shapes are staying within it. The classic dictionary is bounded far lower — 32 shared named-record structures — so a table with more than 32 distinct field-name sets sits at that number as a matter of course; it is a normal reading, not a leak. ## RocksDB Memory @@ -395,6 +395,16 @@ storage: compression: true ``` +## `storage.fullText` + + + +Type: `object` + +Default: See the [storage option reference](../configuration/options.md#storage). + +The [storage option reference](../configuration/options.md#storage) lists the full-text request and native process defaults. Raise the fetched-hit budget to allow more filtering work; raising it does not raise the separate retained-result limit. Larger limits can increase CPU, source reads and memory use for every permitted query. Native process budgets also share capacity across indexes, so tune them with concurrent workloads rather than a single query. See [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for paging costs and restart requirements; see [readiness and recovery](../resources/query-optimization.md#readiness-and-recovery) for failures. + ## Related - [Configuration Options](../configuration/options.md) — full list of `storage` options diff --git a/reference/database/transaction.md b/reference/database/transaction.md index baba4612e..da616e19d 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -23,7 +23,7 @@ On RocksDB (the default storage engine) the transaction log is physically a sing The transaction log is **enabled by default**. To disable it, set [`logging.auditLog`](../logging/configuration.md) to `false` in `harper-config.yaml` and restart Harper. -> The transaction log is required for real-time messaging (WebSocket and MQTT subscriptions) and replication. Do not disable it if real-time features or replication are in use. +> The transaction log is required for real-time messaging (WebSocket and MQTT subscriptions), replication and full-text indexing. Do not disable it while any of these features are in use. ## Operations @@ -134,7 +134,7 @@ Deletes transaction log entries older than the specified timestamp. Deprecated i Deletes transaction log entries older than the specified timestamp. :::warning Database-wide and irreversible -On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This purges whole log files whose entries predate the timestamp, removing [`read_audit_log`](#read_audit_log) history for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender detects that the requested start predates its retained history and forces a full base copy instead of incremental catch-up — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. +On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This purges whole log files whose entries predate the timestamp, removing [`read_audit_log`](#read_audit_log) history for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender detects that the requested start predates its retained history and forces a full base copy instead of incremental catch-up — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. The same history can be required to bridge a [full-text index rebuild](../full-text-search/overview.md#startup-and-recovery) from its table scan to current writes. Cleanup that removes the bridge during a rebuild can fail that attempt; stop aggressive cleanup before retrying so the next attempt can scan from a fresh committed-log boundary. Size automatic retention for the workload. ::: Parameters: @@ -204,7 +204,7 @@ type Dog @table(audit: true) { } ``` -This overrides the [`logging.auditLog`](../logging/configuration.md) global configuration for that specific table. +This overrides the [`logging.auditLog`](../logging/configuration.md) global configuration for that specific table. New schema tables with `@fullText` must explicitly declare `@table(audit: true)`, even when global audit logging is enabled. Existing audited tables can add an index without restating `audit`. Keep auditing enabled while any full-text index is declared. Setting `@table(audit: false)` while a full-text index is declared is rejected during schema registration; this is a per-table prerequisite, independent of the global default. ## Related Documentation diff --git a/reference/full-text-search/configuration.md b/reference/full-text-search/configuration.md new file mode 100644 index 000000000..f7434c676 --- /dev/null +++ b/reference/full-text-search/configuration.md @@ -0,0 +1,195 @@ +--- +title: Configuration +--- + +# Full-Text Search Configuration + + + +Declare a nullable `FullText` field on an audited RocksDB table and attach `@fullText` to that field. The field name identifies the index. Add another `FullText` field when one table needs independent indexes for different search experiences. + +```graphql +type Product @table(database: "catalog", audit: true) { + id: ID @primaryKey + name: String + description: String + tags: [String] + category: String @indexed + price: Float @indexed + available: Boolean + catalogSearch: FullText + @fullText( + fields: [ + { name: "name", weight: 3, highlight: true } + { name: "description", weight: 1, highlight: true } + { name: "tags", weight: 1 } + ] + highlighting: { maxFragments: 2, fragmentLength: 120 } + ) +} +``` + +`catalogSearch` is a query-only declaration. It is not stored, selectable, writable, or included in record, OpenAPI, or MCP schemas. Use the field name as the condition attribute and select `$score` or `$highlights` for search metadata. + +The declaration must use nullable `FullText` exactly and carry one `@fullText` directive. Lists, non-null forms, a bare `FullText` field, and additional directives such as `@computed`, `@indexed`, or `@allow` are rejected. `@fullText` is not valid on a table type or any other field type. + +Ordinary tables inherit `logging.auditLog`, which defaults to `true`. New tables with `@fullText` instead require an explicit `@table(audit: true)`; omitting it or setting `audit: false` returns `400`, even when the global default is enabled. This table-level requirement makes transaction logging an explicit recovery prerequisite for the derived index rather than relying on the ambient default. An existing table that already has persisted audit logging can add the field without restating `audit`. A previously unaudited table can enable `audit: true` in the same schema change that adds its first full-text field; the initial build scans current records and does not require older transaction history. Disabling audit logging while a full-text field exists is rejected. + +Writing the declaration-only field or selecting it as record data returns an error, including on unsealed tables. A read-through source response containing that name fails with `502` and is not cached, indexed, or audited. For HTTP-sourced caching tables, avoid index names that collide with response wrapper fields such as `data`, `headers`, `body`, and `status`. + +## Directive options + +Only the directive options listed below are accepted; an unknown option returns `400`. + +| Option | Default | Description | +| -------------- | ------------- | ------------------------------------------------------------------------------------- | +| `fields` | required | One or more source fields. | +| `analyzer` | `"english@2"` | Versioned text analyzer. `english@2` is the only supported value. | +| `stopWords` | `true` | Removes common English terms during analysis. | +| `positions` | `true` | Stores token positions. Required for phrase queries. | +| `surfaceTerms` | `true` | Stores unstemmed analyzed terms. Required for prefix, fuzzy-prefix, and highlighting. | +| `synonyms` | `[]` | Index-time synonym expansion rules. | +| `highlighting` | omitted | Enables highlight generation and sets fragment limits. | + +Each field entry supports: + +| Option | Default | Description | +| ----------- | -------- | ------------------------------------------------------------------------------ | +| `name` | required | Stored source field name. | +| `weight` | `1` | Positive relevance weight. Higher values contribute more to BM25 ranking. | +| `highlight` | `false` | Allows highlights from this field when index highlighting is enabled. | +| `mediaType` | omitted | Required as `"text/plain"` for a `Blob` source; invalid on other source types. | + +Source fields cannot use `@computed` or `@relationship`. Indexable source values are strings, arrays containing strings or nulls, or Blob values declared as plain text. + +## Structured filters + +Combine full-text conditions with ordinary Harper conditions using `and`. Harper checks current source records and can use existing secondary indexes to narrow candidates. See [combining conditions](./querying.md#combining-conditions). + +## Phrase and prefix support + +`positions` and `surfaceTerms` trade index size for query features. Keep their defaults unless storage measurements show that the features are unnecessary. + +```graphql +type Article @table(audit: true) { + id: ID @primaryKey + body: String + bodySearch: FullText @fullText(fields: [{ name: "body" }], positions: false, surfaceTerms: false) +} +``` + +This smaller declaration supports `matches`, `matches_all`, and `matches_fuzzy`. It rejects phrase, prefix, fuzzy-prefix, and highlighting requests. + +## Highlighting + +Highlighting is off by default. To enable it: + +1. Set `highlight: true` on at least one source field. +2. Add `highlighting` to the directive. +3. Keep `surfaceTerms: true`. + +```graphql +articleSearch: FullText + @fullText( + fields: [{ name: "title", highlight: true }, { name: "body", highlight: true }] + highlighting: { maxFragments: 3, fragmentLength: 160 } + ) +``` + +`maxFragments` and `fragmentLength` must be positive integers. Query results contain offsets into the original field value; see [Highlights](./querying.md#highlights). + +## Synonyms + +Synonyms expand terms while records are indexed: + +```graphql +catalogSearch: FullText + @fullText( + fields: [{ name: "description" }] + synonyms: [ + { source: "sneaker", replacements: ["shoe", "trainer"] } + { source: "tv", replacements: ["television"] } + ] + ) +``` + +Rules require one source term and one or more unique replacement terms. A rule cannot replace a term with itself. + +## Multiple indexes + +Add separate `FullText` fields when separate index definitions are useful: + +```graphql +type Product @table(database: "catalog", audit: true) { + id: ID @primaryKey + name: String + tags: [String] + titleSearch: FullText @fullText(fields: [{ name: "name", weight: 2 }]) + tagSearch: FullText @fullText(fields: [{ name: "tags" }]) +} +``` + +The indexes update independently and can be queried independently. A single query cannot combine conditions from different full-text indexes. + +## Blob sources + +Only plain-text Blob fields are supported: + +```graphql +type Document @table(audit: true) { + id: ID @primaryKey + content: Blob + contentSearch: FullText @fullText(fields: [{ name: "content", mediaType: "text/plain" }]) +} +``` + +`mediaType: "text/plain"` declares the expected Blob media type. Harper does not reject the source record write, but the derived index checks a nonempty runtime `Blob.type` and excludes the record when it does not match `text/plain`. The index decodes matching Blobs as UTF-8. A Blob source is limited to 1 MiB (1,048,576 bytes). A media-type mismatch, oversized value, or invalid UTF-8 removes the whole record from the index rather than indexing only its other fields. A transient Blob read failure rolls back the accepted native batch and retries it, so index coverage does not advance past unread source data. + +## Schema changes and rebuilds + +The `english@2` name remains unchanged in v5.4, but Tantivy 0.27 changes some English stems. Existing indexes must rebuild, and matching or ranking can change. See [upgrading existing indexes](./overview.md#upgrading-existing-indexes). + +Changes that affect indexed storage create a new local index generation and rebuild from source data. These include source membership, source order, media types, analyzer settings, stop-word handling, positions, surface terms, and synonyms. Keep the `fields` list in a stable order when its meaning has not changed. + +While a generation rebuilds, ordinary table reads remain available, but full-text queries on that node return retryable `503` responses until the index is ready. Writes can receive retryable `DERIVED_INDEX_LAGGING` responses while replay catches up after a long rebuild; see [write backpressure](./overview.md#write-backpressure). + +Changing source weights or highlighting settings changes query behavior without rebuilding indexed term storage. Removing the `FullText` field retires its local files after the schema change is confirmed. Renaming the field creates a new index identity and retires the old one. + +## Inspecting an index + +With readable index sources, `describe_table` reports `full_text_indexes`, including fields, analyzer options, supported query modes, and readiness: + +```json +{ + "operation": "describe_table", + "database": "catalog", + "table": "Product" +} +``` + +Metadata respects the caller's attribute permissions: unreadable source fields are omitted, and an index with no readable source fields is omitted entirely. An omitted entry does not by itself mean that the index is unconfigured. With access to its source fields, the table response includes entries shaped like this: + +```json +{ + "full_text_indexes": [ + { + "name": "catalogSearch", + "fields": [{ "name": "name", "weight": 3, "highlight": true }], + "analyzer": "english@2", + "stop_words": true, + "positions": true, + "surface_terms": true, + "synonyms": [], + "highlighting": { "maxFragments": 2, "fragmentLength": 120 }, + "query_modes": ["any", "all", "fuzzy", "phrase", "prefix", "fuzzy-prefix"], + "readiness": { "state": "ready", "owner_epoch": "4", "rebuild_attempts": 0 } + } + ] +} +``` + +`readiness.state` is `unknown`, `ready`, `needs-rebuild`, `rebuilding`, or `unavailable`. Only `ready` serves full-text queries. `unknown` is an activation or owner-election transition. `needs-rebuild` means the runtime has condemned the current generation and queued an automatic rebuild. Queries in `unknown`, `needs-rebuild`, or `rebuilding` return `503` with retryable code `INDEX_REBUILDING`. `unavailable` means activation failed or the automatic rebuild budget was exhausted; queries return a generic, non-retryable `503`. Inspect the optional `readiness.reason` and the node logs, correct the underlying storage, native-module, schema, or transaction-log problem, then reload or reapply the schema to retry activation. + +`query_modes` maps to comparators as follows: `any` → `matches`, `all` → `matches_all`, `fuzzy` → `matches_fuzzy`, `phrase` → `matches_phrase`, `prefix` → `matches_prefix`, and `fuzzy-prefix` → `matches_fuzzy_prefix`. `owner_epoch` is an opaque decimal string used to fence work between derived-index owners; clients should not parse or persist it. `rebuild_attempts` is the current automatic-attempt count. `reason` is present when readiness has an operator-actionable explanation. + +The Operations API uses snake_case for response metadata such as `stop_words`, `surface_terms`, and `owner_epoch`. The nested `highlighting` value mirrors the GraphQL option names `maxFragments` and `fragmentLength`. diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md new file mode 100644 index 000000000..39d9df04b --- /dev/null +++ b/reference/full-text-search/overview.md @@ -0,0 +1,113 @@ +--- +title: Overview +--- + +# Full-Text Search + + + +Harper can build a BM25-ranked full-text index from one or more fields on a table. The index is declared in the table schema, updated from committed record changes, and queried through the same `Table.search()` and REST interfaces used for other Harper queries. + +Use full-text search when users need relevance-ranked matching across product names, descriptions, tags, articles, or other natural-language fields. It supports term, all-term, phrase, prefix, fuzzy, and fuzzy-prefix matching. + + Starting in Harper v5.4, the native engine is **`@harperfast/fulltext` 0.7.0, embedding Tantivy 0.27.0**. Existing fulltext 0.5.0 and prerelease 0.6 indexes require rebuilding. See [Upgrading existing indexes](#upgrading-existing-indexes) before upgrading a node. + +## Requirements + +- The table must use RocksDB and have [transaction logging enabled](../database/transaction.md#enabling-the-transaction-log-per-table). New schema tables must declare `@table(audit: true)`; omitting it or setting `audit: false` returns `400`. An existing table that already persisted audit logging can add a full-text field without restating the option, but disabling audit is rejected while an index is declared. +- Full-text source fields must be stored `String`, `[String]`, or `Blob` values. Blob sources must be declared as `text/plain`. +- Creating or updating an LMDB table with an `@fullText` field fails with `400`. A persisted declaration encountered while opening LMDB is ignored with a warning. Migrate the table to RocksDB before activating the index. + +See [Configuration](./configuration.md) for the complete schema contract, [Querying](./querying.md) for search examples, and [Query optimization](../resources/query-optimization.md) for performance guidance. + +## How it works + +Full-text indexes are derived local state. Harper uses Tantivy, a native text-search library, for term indexing and BM25 ranking. The table records and schema remain the source of truth. + +```mermaid +flowchart LR + W[Committed table write] --> A[Harper audit stream] + A --> F[Full-text derived-index runtime] + F --> T[Local full-text index] + T --> Q[Table.search and REST] + A --> R[Replication] + R --> D[Replica audit stream] + D --> U[Replica local full-text index] +``` + +Each node builds its own index from committed transactions. Harper does not replicate full-text index files or include them in authoritative backups. After a restore or move to a node without compatible local files, Harper rebuilds the index before serving full-text queries. + +The `@fullText` declaration must also exist on each node. A replicated component deployment or schema propagation can install it, but table-record replication alone only supplies the source records. Once the declaration is present, the node builds its own index. + +This design has several consequences: + +- A committed record can be visible before its derived full-text index has applied the same transaction. Queries use a bounded staleness policy, described in [Freshness controls](./querying.md#freshness-controls). +- A missing, corrupt, or incompatible index can be rebuilt from the table and audit stream without changing source records. +- If an index remains behind for more than 30 seconds, Harper rejects ordinary local writes to its table until the runtime catches up or changes lifecycle state. See [Write backpressure](#write-backpressure). + +## Startup and recovery + +On restart, Harper reuses compatible local index files and replays committed changes after the saved checkpoint. If those files are missing, incompatible, or corrupt, Harper rebuilds the index locally. + +A rebuild captures a fresh committed-log boundary, scans the current table, and replays changes committed after that boundary. An unreadable or corrupt transaction log can make an attempt fail. The same applies when [transaction-log cleanup](../database/transaction.md#delete_transaction_logs_before) or automatic retention removes the history needed to bridge that scan to current writes. A later attempt captures a new boundary, so stopping aggressive cleanup can allow the rebuild to succeed without restoring already-purged history. Harper retries with backoff; after the retry budget is exhausted, readiness becomes `unavailable`. Inspect `describe_table` and the node logs for the reason. After correcting the cause, restart the same Harper version to retry activation; keep full-text traffic drained until readiness and required coverage recover. + +Ordinary table reads remain available during a rebuild. Writes to indexed tables can receive retryable lag errors while replay catches up after a long rebuild; see [Write backpressure](#write-backpressure). While readiness is `unknown`, `needs-rebuild`, or `rebuilding`, full-text queries return `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. If rebuild attempts are exhausted and readiness becomes `unavailable`, queries return a generic, non-retryable `503` until an operator corrects the cause and retries activation. A new replica follows the same process after the full-text declaration reaches that node. + +Use `describe_table` to inspect each index's declaration and readiness. See [Inspecting an index](./configuration.md#inspecting-an-index). + +## Upgrading existing indexes + + + +Harper 5.3.1–5.3.4 ship `@harperfast/fulltext` 0.5.0. Upgrading their current field-based indexes to the 0.7.0 engine in Harper v5.4 requires a full local rebuild. Harper 5.3.0 ships fulltext 0.3.0 and is outside this qualified procedure; plan and validate that migration separately. Prerelease installations using fulltext 0.6 also require rebuilding. This procedure applies to the current field-based declarations; earlier beta declarations and indexes have no supported migration path. Harper initiates the rebuild automatically when the wrapper refuses the old persisted identity; no manual rebuild command or file editing is needed. + +### Matching and ranking changes + +Tantivy 0.27 changes English stemming even though the schema analyzer remains `english@2`. For example, `added` now stems to `add` instead of `ad`. Queries can match different records or rank them differently after rebuilding. Review representative searches and domain vocabulary before restoring production traffic; the upgrade does not add new Harper query modes or change the declared field options. + +### Upgrade checklist + +1. Drain the node's full-text requests and stop all old Harper processes that share its storage, including background workers. Do not run old and new engine versions against one local index. +2. Confirm authoritative records and recovery backups are available. Reserve disk headroom and time for rebuilding every local full-text index, including temporarily retained retired files. Measure rebuild duration and disk growth with representative data before setting the rollout window. +3. Start the v5.4 Harper version. Use [`describe_table`](../operations-api/operations.md#describe_table) for each affected table, using an authorized operator account that already has read access to every indexed source field, and check every `full_text_indexes[].readiness.state`. Metadata is permission-filtered; a missing entry does not prove that no index exists. Keep full-text traffic drained until all required indexes are `ready`. +4. Run representative searches through the same REST or permission-checked application path and roles used in production, with `maxIndexLagMilliseconds: 0` and `waitForIndexMilliseconds: 30000` on each full-text condition. These wait for coverage of writes committed on the queried node before the search; they do not wait for replication delivery, and readiness alone does not establish that coverage. Check response bodies for terminal stream errors as well as HTTP status. A JSON-array response can start with HTTP 200 and end with an `error` element. +5. Retry write responses carrying `DERIVED_INDEX_LAGGING` with bounded backoff while replay catches up. Restore full-text traffic only after readiness and current-coverage searches succeed. A node-by-node upgrade can use other ready replicas only if the application's routing supports it; the rebuilding node remains unavailable for full-text search. During a mixed-version rollout, matching and ranking can differ between replicas. Pre-v5.4 replicas also reject prefix requests whose `offset + limit` exceeds 100 with `400`. Route related searches consistently and check the application's behavior before using this approach. + +See [freshness controls](./querying.md#freshness-controls) for the condition fields and streaming behavior. Qualify multi-node rollout, sustained-write capacity, relevance and performance against the application's workload before production rollout. A successful rebuild alone does not establish those properties. + +### Recovering from a failed upgrade + +Keep full-text traffic drained, preserve authoritative data and logs, and correct the reported package, storage or coordination failure before restarting the same v5.4 Harper version. An interrupted rebuild can recover from authoritative records. Do not force readiness or edit, move or delete native index files by hand. + +**Direct downgrade to fulltext 0.5.x is unsupported.** That engine refuses both opening and resetting a 0.7 index identity. If an older version has already been started, stop it and restore the 0.7.0-based Harper version before checking readiness and catch-up again. This is recovery with the current engine, not a supported operator downgrade procedure. If recovery also fails, keep traffic drained and investigate the readiness reason and node logs. + +## Write backpressure + +Harper allows a full-text index up to 30 seconds to absorb committed changes. If the runtime remains behind beyond that budget, ordinary local writes to the indexed table fail with `503`, `code: "DERIVED_INDEX_LAGGING"`, and `retryable: true`. This applies to writes through REST, the Operations API, and direct table calls from application code. Replication notifications, source applies, and crash-recovery replay continue so the authoritative data can converge. + +Retry rejected writes with backoff rather than immediately adding more load. Inspect index readiness and node logs. Write admission resumes after the runtime proves durable catch-up or moves into another lifecycle state such as rebuilding or unavailable. + +## Ranking and result loading + +The native index ranks matching documents with BM25. Harper then loads the authoritative table records for the matching IDs and discards stale or deleted candidates. Selecting `$score` includes the relevance score in each result. + +Full-text search can be combined with structured filters. Harper checks conditions against current source records before returning them. In v5.4, additional ranked batches can fill a filtered page within request budgets; exhausting a budget returns an error rather than a budget-truncated page. + +## Limits and troubleshooting + +- Phrase search requires `positions: true`; prefix and fuzzy-prefix require `surfaceTerms: true`. +- Deep offsets revisit earlier matches and increase fetched work and source reads. Prefer narrower filters and a bounded maximum offset over repeatedly raising work budgets. +- Before v5.4, prefix and fuzzy-prefix expressions restrict `offset + limit` to 100. In v5.4, bounded full-text and autocomplete queries fetch additional batches within request budgets. Use a bounded `limit` and see [paging and prefix result windows](./querying.md#paging-and-prefix-result-windows); native windows are not total-result cutoffs for bounded v5.4 queries. +- `Table.clear()` returns `501` while any full-text declaration is active. To clear the table, first remove all `FullText` declarations and wait for the schema change to apply. Re-add the declarations afterward to rebuild the indexes from the remaining data. +- One query can use only one full-text index. An `or` group cannot mix full-text and ordinary record conditions. +- Each Blob source is limited to 1 MiB (1,048,576 bytes). A media-type mismatch, oversized value, or invalid UTF-8 removes the entire record from the index, including its other full-text source fields, until valid content is indexed. +- `INDEX_REBUILDING` is transient and retryable. `DERIVED_INDEX_LAGGING` can mean a query coverage wait expired or local writes were shed after prolonged index lag. `unavailable` means automatic rebuild attempts were exhausted or activation failed; inspect readiness and logs rather than retrying every `503` indefinitely. +- Generated GraphQL field arguments remain equality conditions; use `Table.search()`, REST, or `search_by_conditions` for full-text comparators. + +See [Querying](./querying.md) for exact condition rules and [Configuration](./configuration.md#inspecting-an-index) for readiness details. + +## Stored index data + +The local index stores tokenized terms, posting lists, optional positions and surface terms, and checkpoint metadata. It does not store an authoritative copy of each record. Harper always loads current table records before returning results, and it can discard and rebuild this derived state from committed data. + +Protect the index directory with the same filesystem controls as Harper data. Deleted source terms can remain in obsolete Tantivy segment files until normal merge and reclamation removes them; removal from query results does not imply immediate physical erasure. diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md new file mode 100644 index 000000000..4980eb120 --- /dev/null +++ b/reference/full-text-search/querying.md @@ -0,0 +1,258 @@ +--- +title: Querying +--- + +# Querying Full-Text Indexes + + + +Use the query-only `FullText` field name as a condition attribute in `Table.search()`. + +```javascript +const products = Product.search({ + conditions: [{ attribute: 'catalogSearch', comparator: 'matches', value: 'waterproof trail shoes' }], + limit: 20, +}); +``` + +`Table.search()` returns an async iterable. Consume it with `for await...of`, or return it directly from a custom Resource to stream the results. + +Results are ordered by BM25 relevance. The `FullText` field, not one of its source fields, identifies the search target. + +## Match modes + +| Comparator | Behavior | Required index option | +| ---------------------- | --------------------------------------------------------------------- | --------------------- | +| `matches` | Matches any analyzed query term. | — | +| `matches_all` | Requires every analyzed query term. | — | +| `matches_phrase` | Matches analyzed terms in order as a phrase. | `positions: true` | +| `matches_prefix` | Matches records whose analyzed terms start with the final query term. | `surfaceTerms: true` | +| `matches_fuzzy` | Matches terms within one edit, including a transposition. | — | +| `matches_fuzzy_prefix` | Combines one-edit fuzzy matching with a final-term prefix. | `surfaceTerms: true` | + +Every comparator also has a negated form: `not_matches`, `not_matches_all`, `not_matches_phrase`, `not_matches_prefix`, `not_matches_fuzzy`, and `not_matches_fuzzy_prefix`. + +An empty string is invalid and returns `400`. Whitespace-only text or text reduced to no terms by the analyzer, such as a query containing only enabled stop words, returns an exact empty result. + +A query containing a negated full-text condition must also contain a non-negated full-text condition on the same index. A structured record condition alone does not satisfy this requirement. The positive full-text condition bounds the candidate set; the negated condition filters it. + +```javascript +const products = Product.search({ + operator: 'and', + conditions: [ + { attribute: 'catalogSearch', comparator: 'matches', value: 'waterproof' }, + { attribute: 'catalogSearch', comparator: 'not_matches', value: 'leather' }, + ], +}); +``` + +## Restricting source fields + +By default, a condition searches every source field in the index. Use `fields` to restrict it: + +```javascript +const products = Product.search({ + conditions: [ + { + attribute: 'catalogSearch', + comparator: 'matches_phrase', + value: 'trail running', + fields: ['name', 'description'], + }, + ], +}); +``` + +Every requested field must belong to the index and be readable by the caller. Without `fields`, the caller must be allowed to read every source field in the index. Harper rejects an external REST or Operations API query with `403` if any searched source field is not readable. `$highlights` never bypasses this check. + +Direct calls through `tables` or `databases` run in a trusted server-side context. From an authenticated custom resource, pass the `context` received by the static resource method to `Table.search()` and set `checkPermission: true` in the search options to enforce the caller's table and source-field permissions. Context propagation alone does not request the table permission check. + +## Score and ordering + +Select `$score` to include the BM25 score: + +```javascript +const products = Product.search({ + conditions: [{ attribute: 'catalogSearch', comparator: 'matches', value: 'trail shoes' }], + select: ['id', 'name', '$score'], + limit: 20, +}); +``` + +Full-text results use descending relevance order. The only explicit full-text sort is `$score` descending; other sort keys and reverse iteration are rejected because they would discard the native ranking. + +Full-text queries cannot provide an exact total after authorization, structured filtering, and current-record validation. On a REST interface with `exactCount` enabled, `Prefer: count=exact` reports an unavailable total; otherwise it is downgraded to an estimated count. See [pagination and total count](../rest/querying.md#pagination-and-total-count). `Table.getRecordCount()` counts table records, not matches for a full-text query. + +## Highlights + +Highlighting returns source-field fragments and matching character spans. The index must configure highlighting and mark at least one source with `highlight: true`. A query can also search sources without that option; those sources can affect matching and ranking but are omitted from `$highlights`. + +Selecting `$highlights` turns highlighting on for the query: + +```javascript +const products = Product.search({ + conditions: [{ attribute: 'catalogSearch', comparator: 'matches_phrase', value: 'trail running' }], + select: ['id', 'name', '$score', '$highlights'], +}); +``` + +You can also set `includeHighlights: true` on the condition. A result has this shape: + +```json +{ + "id": "shoe-1", + "name": "Waterproof trail running shoe", + "$score": 2.41, + "$highlights": { + "name": [ + { + "valueIndex": 0, + "spans": [{ "start": 11, "end": 24 }], + "fragments": [ + { + "text": "Waterproof trail running shoe", + "start": 0, + "spans": [{ "start": 11, "end": 24 }] + } + ] + } + ] + } +} +``` + +Offsets are UTF-16 half-open ranges. Value-level spans address the original source value. A fragment's `start` addresses that value, while spans inside the fragment address its returned `text`. Harper returns offsets rather than HTML so the caller controls rendering and escaping. + +## Combining conditions + +Full-text conditions can be combined with structured filters using `and`: + +```javascript +const products = Product.search({ + operator: 'and', + conditions: [ + { attribute: 'catalogSearch', comparator: 'matches', value: 'trail shoe' }, + { attribute: 'category', comparator: 'equals', value: 'footwear' }, + { attribute: 'available', comparator: 'equals', value: true }, + ], + limit: 20, +}); +``` + +Harper checks the structured conditions against current source records before returning them. Existing Harper secondary indexes can narrow candidate IDs or support point checks. Rejected hits do not count toward a v5.4 page's offset and limit, but fetching them still consumes the raw-hit budget. + +An `or` group may combine conditions from the same full-text index: + +```javascript +const products = Product.search({ + operator: 'or', + conditions: [ + { attribute: 'catalogSearch', comparator: 'matches_phrase', value: 'trail running' }, + { attribute: 'catalogSearch', comparator: 'matches', value: 'hiking boot' }, + ], +}); +``` + +An `or` group cannot mix full-text and ordinary record conditions, and one query cannot combine different full-text indexes. Full-text conditions must name a field directly on the queried table; relationship and nested-property paths are not supported. Run separate queries when any of these boundaries is required. + +## Freshness controls + +A full-text index is derived from committed table changes. Each condition accepts: + +| Option | Default | Description | +| -------------------------- | ------- | ---------------------------------------------------------------------------------- | +| `maxIndexLagMilliseconds` | `3000` | Maximum accepted upper bound on index lag. Set to `0` to require current coverage. | +| `waitForIndexMilliseconds` | `0` | How long to wait for acceptable coverage, from `0` through `30000`. | + +To wait for committed changes already received by this node to reach its local index, require current coverage and allow a bounded wait. These controls do not wait for a write to arrive from another replica; a query routed to a different node can run before that node receives the write: + +```javascript +for await (const product of Product.search({ + conditions: [ + { + attribute: 'catalogSearch', + comparator: 'matches', + value: 'new product', + maxIndexLagMilliseconds: 0, + waitForIndexMilliseconds: 10000, + }, + ], +})) { + console.log(product); +} +``` + +All full-text conditions combined into one query must use the same freshness values. Harper returns `400` when combined conditions specify different values. + +A non-waiting HTTP query can return `Harper-Index-Coverage` with the admitted state, lag upper bound, and requested tolerance. A zero-size page performs no native search and carries no coverage proof. A waiting `Table.search()` establishes coverage as its iterator is consumed. A `search_by_conditions` request completes only after its wait and search finish. Waiting queries do not emit a coverage header before completion. + +If acceptable coverage is not reached before `waitForIndexMilliseconds` expires, Harper returns `503` with `code: "DERIVED_INDEX_LAGGING"` and `retryable: true`. For `Table.search()`, the error is raised while the async iterator is consumed. An HTTP response may already be streaming, so clients should also handle a terminal stream error rather than relying only on the initial status. + +The same code is also used when prolonged derived-index lag causes Harper to reject local writes. Query waits can be retried according to the caller's freshness needs; rejected writes should use backoff while the index catches up. See [Write backpressure](./overview.md#write-backpressure). + +## Paging and prefix result windows + +`matches_prefix` and `matches_fuzzy_prefix` are autocomplete-style record searches. These modes return matching records, not a separate list of suggested terms. + +Before v5.4, any expression containing one of these modes uses a 100-record native result window, and `offset + limit` cannot exceed 100. An unbounded query with more than 100 matches returns `400` rather than silently truncating the result. + +### Paging in v5.4 + + + +Full-text and autocomplete searches use Harper's existing `limit` and `offset`. Bounded queries can pass the ordinary native result window, including the earlier 100-record prefix boundary. Within one request, one native reader fetches ranked batches until Harper has enough eligible records for the offset and limit, exhausts the matches, or reaches a request budget. Source-version checks and record filters run before records count toward the requested page. + +For example, `offset: 100` and `limit: 10` skip the first 100 eligible records and return up to the next 10, even for prefix matching. A short or empty page means matches were exhausted. Exhausting the fetched-hit or retained-result budget returns `400`; narrow the query, reduce offset or limit, or raise the named budget. Exhausting the execution budget returns `503`; narrow the query or adjust the execution budget rather than retrying it unchanged. These budget errors carry no readiness-specific `code` or `retryable` flag. If a response has already started streaming, a later error terminates that stream instead of changing its HTTP status. An unbounded query can still fail when it exceeds the native window or configured limits. Native response-envelope overflow can return `413`, including an oversized ordinary batch even when each hit would fit. Other native failures can return `500`. A `503` without a readiness code is not a signal to wait for an index rebuild; inspect its message and adjust the work or concurrency that caused it. + +`storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds` bound fetched work, retained results and execution time. Replayed or rejected hits consume raw work; a request can exhaust that budget before reaching its result target. The execution budget is not a hard response-time guarantee; reader acquisition and some native work can take additional time. See [storage configuration](../configuration/options.md#storage) for the controls and [REST paging and counts](../rest/querying.md#pagination-and-total-count) for URL syntax and count behavior. + +Each offset request runs a new query and revisits earlier matches, so deeper offsets increase fetched work and source reads. Prefer narrower filters and a bounded maximum offset over repeatedly raising budgets. Each query can see changed records, permissions or index contents. Pages can therefore repeat or omit records during concurrent changes. There are no continuation tokens, close operations or retained searches across requests. + +## REST + +Use the comparator in the collection query string: + +```http +GET /Product/?catalogSearch=matches=waterproof%20trail&limit(20) +GET /Product/?catalogSearch=matches_phrase=trail%20running&select(id,name,$score,$highlights) +GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&limit(10) +``` + +REST supports all positive and negated full-text comparators. It can express the query text, selected fields, and pagination. + +Repeat the index parameter to combine the required positive and negated conditions: + +```http +GET /Product/?catalogSearch=matches=waterproof&catalogSearch=not_matches=leather&limit(20) +``` + +REST can return configured highlights by selecting `$highlights`. Use `Table.search()` or `search_by_conditions` for explicit `fields`, condition-level `includeHighlights`, or freshness controls. + +## Operations API + +`search_by_conditions` uses the same condition shape: + +```json +{ + "operation": "search_by_conditions", + "database": "catalog", + "table": "Product", + "limit": 20, + "get_attributes": ["id", "name", "$score", "$highlights"], + "conditions": [ + { + "attribute": "catalogSearch", + "comparator": "matches_all", + "value": "waterproof trail", + "fields": ["name", "description"], + "includeHighlights": true, + "maxIndexLagMilliseconds": 0, + "waitForIndexMilliseconds": 10000 + } + ] +} +``` + +If an index is `unknown`, `needs-rebuild`, or `rebuilding`, the query returns `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. An index in terminal `unavailable` state returns a generic, non-retryable `503`; inspect readiness and logs instead of retrying it as a rebuild. Other busy paths can also return a generic `503`, so clients should branch on the code rather than the status alone. Unreadable source fields return `403`. Invalid declarations, unsupported match modes, incompatible combinations, and out-of-range options return `400` responses. + +For paging-budget and native response-envelope failures, see [paging and prefix result windows](#paging-and-prefix-result-windows). diff --git a/reference/graphql-querying/overview.md b/reference/graphql-querying/overview.md index fb56ffc31..79b5f6c68 100644 --- a/reference/graphql-querying/overview.md +++ b/reference/graphql-querying/overview.md @@ -172,6 +172,8 @@ fragment sharedFields on Any { Any attribute can be used as an argument for a query. In this short form, multiple arguments is treated as multiple equivalency conditions with the default `and` operation. +These arguments always use equality. A query-only `FullText` declaration field is omitted from the generated GraphQL record schema (Added in: v5.3.0); use [full-text search through `Table.search()`, REST, or the Operations API](../full-text-search/querying.md) for match comparators and BM25 ranking. + For example, the following query requires an `id` variable to be provided, and the system will search for a `Dog` record matching that id. ```graphql diff --git a/reference/index.md b/reference/index.md index 306c06837..a5606d1b3 100644 --- a/reference/index.md +++ b/reference/index.md @@ -42,6 +42,7 @@ For concept introductions, tutorials, and guides, see the [Learn](/learn) sectio | Section | Description | | ------------------------------------------------------------ | ------------------------------------------------------------------ | | [Backups](./backups/overview.md) | Managed incremental backups, snapshot downloads, and restore | +| [Full-Text Search](./full-text-search/overview.md) | BM25-ranked native search, schema configuration, and query modes | | [Logging](./logging/overview.md) | Log configuration, the `logger` API, and log management operations | | [Analytics](./analytics/overview.md) | Resource and storage analytics, system tables | | [MQTT](./mqtt/overview.md) | MQTT broker configuration and usage | diff --git a/reference/logging/configuration.md b/reference/logging/configuration.md index f204b9325..f5f82ea96 100644 --- a/reference/logging/configuration.md +++ b/reference/logging/configuration.md @@ -102,11 +102,11 @@ Default: `true` Enables transaction logging. When enabled, Harper records every insert, update, and delete to the database's transaction log. This history is read with the [`read_audit_log`](../database/transaction.md#read_audit_log) operation — a legacy name; there is a single transaction log, not a separate audit table. -See [Database / Transaction Logging](../database/transaction.md) for details on the transaction log. +Full-text indexing requires transaction logging. New schema tables with `@fullText` must explicitly declare `@table(audit: true)`, even when this global setting is enabled. Existing audited tables can add an index without restating `audit`; disabling table auditing while an index is declared is rejected. See [Database / Transaction Logging](../database/transaction.md) for details on the transaction log. ```yaml logging: - auditLog: false + auditLog: true ``` ### `logging.auditRetention` @@ -339,7 +339,7 @@ logging: file: true stdStreams: false console: false - auditLog: false + auditLog: true auditRetention: 3d rotation: enabled: true diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 21815c98a..103d2bc18 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -47,7 +47,7 @@ Detailed documentation: [Database Overview](../database/overview.md) ### `describe_all` -Returns the definitions of all databases and tables within the database. Record counts above 5000 records are estimated; the response includes `estimated_record_range` when estimated. To force an exact count (requires full table scan), include `"exact_count": true`. Each table definition includes the [record-structure dictionary fields](#describe_table) described for `describe_table`. +Returns the definitions of all databases and tables within the database. Record counts above 5000 records are estimated; the response includes `estimated_record_range` when estimated. To force an exact count (requires full table scan), include `"exact_count": true`. Each table definition includes the [record-structure dictionary fields](#record-structure-dictionaries) described for `describe_table`. ```json { "operation": "describe_all" } @@ -55,7 +55,7 @@ Returns the definitions of all databases and tables within the database. Record ### `describe_database` -Returns all table definitions within the specified database. Each table definition includes the [record-structure dictionary fields](#describe_table) described for `describe_table`. +Returns all table definitions within the specified database. Each table definition includes the [record-structure dictionary fields](#record-structure-dictionaries) described for `describe_table`. ```json { "operation": "describe_database", "database": "dev" } @@ -63,12 +63,43 @@ Returns all table definitions within the specified database. Each table definiti ### `describe_table` -Returns the definition of a specific table. +Returns the definition of a specific table, including its schema and storage metadata. ```json { "operation": "describe_table", "table": "dog", "database": "dev" } ``` +#### Full-text indexes + + + +A table with native full-text indexes includes `full_text_indexes`; each entry reports its query-only field name, source fields, analyzer options, enabled query modes, and `readiness` (`unknown`, `ready`, `needs-rebuild`, `rebuilding`, or `unavailable`). The readiness object also includes `owner_epoch`, `rebuild_attempts`, and an optional `reason`. Query-only `FullText` fields do not appear in the ordinary `attributes` list. Metadata respects attribute permissions: unreadable source fields are omitted, and an index with no readable source fields is omitted entirely. Use an account with access to every indexed source field when checking upgrade readiness; a missing entry does not prove that no index exists. + +Requesting a table with full-text indexes: + +```json +{ "operation": "describe_table", "table": "Product", "database": "catalog" } +``` + +The response also includes the table's ordinary schema and storage metadata. The relevant full-text fragment is: + +```json +{ + "name": "Product", + "full_text_indexes": [ + { + "name": "catalogSearch", + "fields": [{ "name": "name", "weight": 3 }], + "analyzer": "english@2", + "query_modes": ["any", "all", "fuzzy", "phrase", "prefix", "fuzzy-prefix"], + "readiness": { "state": "ready", "owner_epoch": "4", "rebuild_attempts": 0 } + } + ] +} +``` + +#### Record-structure dictionaries + Alongside the schema, the response carries the size of the table's **record-structure dictionaries** @@ -270,7 +301,7 @@ Returns records with a matching value on any attribute. Supports wildcards (e.g. ### `search_by_conditions` -Returns records matching one or more conditions. Supports `operator` (`and`/`or`), `offset`, `limit`, nested `conditions` groups, and `sort` with multi-level tie-breaking. +Returns records matching one or more conditions. Supports `operator` (`and`/`or`), `offset`, `limit`, nested `conditions` groups, and `sort` with multi-level tie-breaking. Full-text conditions use comparators such as `matches` and `matches_prefix`; see [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for paging costs and limits. ```json { @@ -284,6 +315,10 @@ Returns records matching one or more conditions. Supports `operator` (`and`/`or` } ``` +Full-text conditions (Added in: v5.3.0): + +Conditions can query a declared full-text index with the `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix` comparators and their `not_` forms. A negated full-text condition requires a non-negated condition on the same full-text index. The Operations API preserves advanced condition options such as source `fields`, `includeHighlights`, `maxIndexLagMilliseconds`, and `waitForIndexMilliseconds`. See [Querying Full-Text Indexes](../full-text-search/querying.md#operations-api). + --- ## Bulk Operations @@ -1679,16 +1714,16 @@ Operations for reading and updating Harper configuration. Detailed documentation: [Configuration Overview](../configuration/overview.md) -| Operation | Description | Role Required | -| ------------------- | ---------------------------------------------------------------- | ------------- | -| `set_configuration` | Modifies Harper configuration file parameters (requires restart) | super_user | -| `get_configuration` | Returns the current Harper configuration | super_user | +| Operation | Description | Role Required | +| ------------------- | --------------------------------------------- | ------------- | +| `set_configuration` | Modifies Harper configuration file parameters | super_user | +| `get_configuration` | Returns the current Harper configuration | super_user | ### `set_configuration` -Updates configuration parameters in `harper-config.yaml`. A restart (`restart` or `restart_service`) is required for changes to take effect. +Updates configuration parameters in `harper-config.yaml`. Most changes require a restart (`restart` or `restart_service`) to take effect. See [Set Configuration](../configuration/operations.md#set-configuration) for full-text restart requirements. -Supports `"replicated": true` to apply the same change to all cluster nodes in one call; per-node outcomes are returned in the response's `replicated` array. Only send cluster-appropriate parameters when replicating — node-local parameters (ports, `node.hostname`, file paths, TLS material, `replication.hostname`/`url`/`routes`) would overwrite every peer's local values. To apply the change cluster-wide, follow with `restart_service` using `"replicated": true` (which restarts nodes one at a time); `restart_service` restarts only worker threads, though, so a change to [`threads.v8Flags`](../configuration/options.md#applying-v8-flags) takes effect only after a `restart` sent to each node in turn (`restart` does not replicate). See [Configuration Operations](../configuration/operations.md#set-configuration) for details. +Supports `"replicated": true` to apply the same change to all cluster nodes in one call; per-node outcomes are returned in the response's `replicated` array. Only send cluster-appropriate parameters when replicating — node-local parameters (ports, `node.hostname`, file paths, TLS material, `replication.hostname`/`url`/`routes`) would overwrite every peer's local values. To apply the change cluster-wide, follow with `restart_service` using `"replicated": true` (which restarts nodes one at a time); `restart_service` restarts only worker threads, though, so changes to [`threads.v8Flags`](../configuration/options.md#applying-v8-flags) or `storage.fullText.runtime.*` take effect only after a `restart` sent to each node in turn (`restart` does not replicate). See [Configuration Operations](../configuration/operations.md#set-configuration) for details. ```json { diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index dfe2cf9d1..3da6e188b 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -48,6 +48,14 @@ When a query is executed, conditions are evaluated against the database. Indexed **Rule of thumb**: Use `equals`, range operators, and `starts_with` on indexed fields. Avoid `contains`, `ends_with`, and `not_equal` as the sole or first condition in large datasets. +### Full-text search + + + +A condition using a `FullText` field declared with `@fullText` is executed by the native full-text engine and ranked with BM25. Compatible `and` filters narrow the matches. Harper loads current source records and removes stale or deleted candidates before returning them. + +Keep result pages bounded. Before v5.4, prefix and fuzzy-prefix expressions restrict `offset + limit` to 100. In v5.4, bounded queries can traverse beyond that window within request budgets; an unbounded query can still fail when it exceeds a native window or configured limit. See [paging and prefix result windows](../full-text-search/querying.md#paging-and-prefix-result-windows). Use source-field weights to tune ranking, and enable positions, surface terms, and highlighting only when their query features are needed. See [Full-Text Search](../full-text-search/overview.md). + ### Indexed vs. Non-Indexed Fields Indexed fields provide `O(log n)` lookup — fast even as the dataset grows. Non-indexed fields require `O(n)` full table scans. @@ -79,6 +87,57 @@ When a secondary index is still being built — for example, just after a new `@ This is a transient condition — the index finishes building in the background, after which the same query succeeds. Clients should retry (with backoff) rather than treating it as an empty result or a permanent failure. On the operations API the error body includes `code` and `retryable`, so callers can branch on `error.code === 'INDEX_REBUILDING'` or `error.retryable`. +## Full-Text Search Work and Paging + + + + + +Full-text and autocomplete queries use Harper's existing `limit` and `offset` options. Each page runs a new query: changed records, permissions or index contents can cause records to repeat or be omitted between pages. There is no retained search across requests. See [REST paging and counts](../rest/querying.md#pagination-and-total-count) for URL syntax and count behavior. + +Within one request, Harper uses one native reader to fetch ranked batches. It checks source versions and record filters before counting records toward the requested offset and limit. In v5.4, prefix and fuzzy-prefix searches can return more than the native ordinary autocomplete window. Before v5.4, prefix and fuzzy-prefix requests whose `offset + limit` exceeded 100 were rejected with 400; requests within that window could still return short pages after filtering. In v5.4, the native search window is not a total-result cutoff for bounded Harper queries. + +For example, with `offset: 20` and `limit: 10`, Harper skips the first 20 eligible records and returns the next 10. Hits rejected by source-version checks or record filters do not count toward those 30 records, but fetching them still consumes the raw-hit budget. Harper fetches additional batches as needed until it fills the page, exhausts the matches or reaches a request limit. + +Use a bounded `limit` for large result sets. An unbounded search that exceeds its native window, fetched-hit limit or retained-result limit throws 400; use a bounded query. Exhausting the execution-time budget instead throws 503. `offset + limit` must not exceed either `storage.fullText.maxResultsPerRequest` or `storage.fullText.maxRawHitsPerRequest`; replayed or rejected hits can exhaust the raw-work budget earlier. Raising only the raw-hit budget allows more filtering work without raising the retained-record target. Offset requests revisit earlier matches and can retain up to `offset + limit` accepted records. Raising work limits can increase CPU, reads and memory use. + +The fetched-hit budget includes the initial batch replay when Harper switches to native traversal. It counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. A budget failure is a failed query, not a completed short page. If a REST response has already started streaming, a later query error ends that stream instead of changing its HTTP status. JSON-array clients must check for a terminal object shaped like `{ "error": "ErrorName: message" }` and treat the response as failed. For other formats, detect stream or decoding failures; status 200 and count headers alone do not prove completion. See [REST paging](../rest/querying.md#pagination-and-total-count). When there are no further native matches, Harper returns the eligible records found, which may be an empty result. This differs from reaching a budget while more matches remain. + +Configure the three request limits and six native process budgets under [`storage.fullText`](../configuration/options.md#storage). See [Set Configuration](../configuration/operations.md#set-configuration) for restart requirements. The execution budget is not a hard response-time guarantee: reader acquisition, query construction and some native operations can take additional time. Freshness waiting and reader acquisition use separate bounds. + +### Native runtime configuration + +Harper captures native process budgets before loading application components, or on an earlier main-thread native load or managed worker spawn. Application-created workers inherit that configuration. Create workers from a Harper application component after initialization. A worker created before capture fails on its first native full-text load with `FullTextWorkerInitializationError` (500, non-retryable); recreate that worker after Harper initializes. + +Replacement workers keep the captured budgets after a configuration change. See [Set Configuration](../configuration/operations.md#set-configuration) before changing these budgets. + +### Readiness and recovery + +Full-text indexes require audit logging. New schema tables must explicitly declare `@table(audit: true)` when adding `@fullText`, even if global audit logging is enabled; an existing audited table can add an index without restating `audit`. Setting `@table(audit: false)` while an index is declared is rejected during schema registration. + +Harper v5.4 uses `@harperfast/fulltext` 0.7.0 with Tantivy 0.27.0. Existing 0.5.0 and prerelease 0.6 indexes using current field-based declarations need an automatic local rebuild after this engine upgrade. Follow [upgrading existing indexes](../full-text-search/overview.md#upgrading-existing-indexes) for affected versions, draining traffic, readiness checks, matching changes and current-engine recovery. Earlier beta declarations and indexes have no supported migration path. See [write backpressure](../full-text-search/overview.md#write-backpressure) for local write refusal and retry guidance. + +| Failure | HTTP status | Code and retry guidance | +| ------------------------------------------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Raw-hit or retained-result budget | 400 | No readiness code. Narrow the query or adjust the named budget. | +| Execution budget | 503 | No readiness code. Narrow the query; the same expensive query can fail again. | +| Native busy | 503 | No readiness code. Reduce concurrency or retry with backoff; inspect process-wide native capacity, including `storage.fullText.runtime.maxExpensiveSearches`, before raising limits. | +| Index unknown, needs-rebuild or rebuilding | 503 | `INDEX_REBUILDING`, with `retryable: true`, for a not-ready readiness check. Retry with backoff while the index rebuilds. | +| Coverage lag, wait timeout or local write backpressure | 503 | `DERIVED_INDEX_LAGGING`, with `retryable: true`. Retry with backoff while the index catches up. Ordinary local writes can be refused after more than 30 seconds of index lag. | +| Index unavailable | 503 | No readiness-specific code or retryable flag. Inspect readiness and logs; fix the reported problem before retrying. | +| Query host becomes non-ready during a coverage wait | 503 | Can return no readiness-specific code or retryable flag. Inspect readiness before deciding whether to retry. | +| Worker loaded native search before runtime capture | 500 | REST reports `code: "FullTextWorkerInitializationError"`; the Operations API can omit that code, so inspect the error message too. Non-retryable; recreate the worker after initialization. | + +A native result that exceeds the response-byte envelope can return 413, including an oversized ordinary batch even when each individual hit would fit. Other native failures and invalid query configuration can return 500; inspect the error and server logs rather than assuming they are retryable. The table lists common failures, not every possible error. + +For read-after-write queries, `maxIndexLagMilliseconds: 0` requires coverage of writes already committed on the queried node. It does not wait for replication to deliver writes from another node. `waitForIndexMilliseconds` lets a query wait for coverage (default `0`, maximum `30000` ms); without an override, the allowed index lag is `3000` ms. These options belong to the full-text condition. They wait for coverage, not for an unavailable index to recover. Readiness is checked first; a query host that becomes rebuilding or unavailable during wait setup can fail immediately with a bare 503. + +Concurrent requests each have their own work budget. These per-request budgets do not cap the number of concurrent Harper queries or prevent brief event-loop delays during synchronous source reads. Measure deep offsets and selective filters under the expected concurrency before raising budgets. + +`INDEX_REBUILDING` and `DERIVED_INDEX_LAGGING` identify retryable readiness or coverage failures. Retry with backoff while the index makes progress. Transaction history retained under [`logging.auditRetention`](../logging/configuration.md#loggingauditretention) (default `3d`) is used for catch-up. Lost history can require a rebuild; retries alone do not repair an index that cannot catch up. Use [`describe_table`](../operations-api/operations.md#describe_table) to inspect `full_text_indexes[].readiness`, including its state and reason. Stop aggressive transaction-log cleanup and correct the reported configuration or storage problem before restarting the same Harper version to retry activation. Keep full-text traffic drained until readiness and required coverage recover; do not edit native files. Budget and native-busy failures do not carry those readiness codes. A native busy response also returns 503; reduce concurrency or retry with backoff. For an execution-budget failure, narrow the query, reduce the offset or adjust the work budget instead of repeatedly submitting the same expensive query. + +Native page, response-byte and highlight limits still apply. The fulltext 0.7.0 library limits each analyzed query leaf to 64 terms and the compiled query to 256 clauses. Native invalid-query validation returns 400 before response headers; the streaming-error guidance above still applies. Raising Harper's work budgets does not raise those native limits. See the [published native limits](https://github.com/HarperFast/fulltext/blob/v0.7.0/docs/limits.md) for page, byte and highlight bounds. + ## Relationships and Joins Harper supports relationship-based queries that join data across tables. See [Schema documentation](../database/schema.md) for how to define relationships. diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 8412c1b0e..4d0ab3c88 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1027,13 +1027,21 @@ The `Query` object is accepted by `search()` and the static `get()` method. Array of condition objects to filter records. Each condition: -| Property | Description | -| ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `attribute` | Property name, or an array for chained/joined properties (e.g. `['brand', 'name']`) | -| `value` | The value to match | -| `comparator` | `equals` (default), `greater_than`, `greater_than_equal`, `less_than`, `less_than_equal`, `starts_with`, `contains`, `ends_with`, `between`, `not_equal` | -| `conditions` | Nested conditions array | -| `operator` | `and` (default) or `or` for the nested `conditions` | +| Property | Description | +| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `attribute` | Property name, or an array for chained/joined properties (e.g. `['brand', 'name']`) | +| `value` | The value to match | +| `comparator` | `equals` (default), `greater_than`, `greater_than_equal`, `less_than`, `less_than_equal`, `starts_with`, `contains`, `ends_with`, `between`, `not_equal`, or a [full-text comparator](../full-text-search/querying.md#match-modes) | +| `conditions` | Nested conditions array | +| `operator` | `and` (default) or `or` for the nested `conditions` | +| `fields` | Full-text source fields to search; every name must belong to the selected full-text index | +| `includeHighlights` | Include configured full-text highlights | +| `maxIndexLagMilliseconds` | Maximum accepted full-text index lag; see [freshness controls](../full-text-search/querying.md#freshness-controls) | +| `waitForIndexMilliseconds` | Time to wait for acceptable full-text coverage; see [freshness controls](../full-text-search/querying.md#freshness-controls) | + +The final four properties apply to full-text conditions (v5.3.0). See [Querying Full-Text Indexes](../full-text-search/querying.md) for their constraints, error behavior, and permission requirements for direct server-side searches. + +Full-text conditions use `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, or `matches_fuzzy_prefix` against a declared full-text index. Each comparator also has a `not_` form: `not_matches`, `not_matches_all`, `not_matches_phrase`, `not_matches_prefix`, `not_matches_fuzzy`, and `not_matches_fuzzy_prefix`. A negated condition requires a positive full-text condition on the same index. See [Full-text query execution](./query-optimization.md#full-text-search-work-and-paging) for paging costs and work limits. Example with nested conditions: @@ -1086,6 +1094,13 @@ Special properties: - `$updatedtime` — Returns the last-updated timestamp - `$distance` — When the query ranks or filters by a vector index, returns the computed distance from the target vector. See [Vector Indexing](../database/schema.md#vector-indexing). + + +Full-text queries add two special properties: + +- `$score`: When a query uses a full-text index, returns its BM25 relevance score. +- `$highlights`: Enables and returns full-text fragments and matching spans when the index has highlighting configured. See [Highlights](../full-text-search/querying.md#highlights). + #### Selecting related records When a field is defined as a [relationship](../database/schema.md#relationships), `select` can pull the related record(s) into each result as a nested property — the programmatic equivalent of the REST `select(name,author{name})` syntax. @@ -1246,6 +1261,12 @@ If `true`, returns conditions reordered as Harper will execute them (for debuggi If `true`, forces conditions to execute in the order supplied, disabling Harper's automatic re-ordering optimization. +### `checkPermission` + +Trusted server-side calls skip table authorization by default. When delegating a request to `Table.search()`, set `checkPermission: true` on the query and pass the authenticated `context` as the second argument. Harper then performs a one-shot read authorization using `context.user`, including table, selected-property, and full-text source-field permissions. Passing the context alone does not request this check. Record-level restrictions require a synchronous, server-controlled `rowFilter`. + +Set this flag in server-controlled code whenever the endpoint promises authorized access. Never accept its value or copy a permission object from client data. + --- ## RequestTarget @@ -1259,7 +1280,7 @@ Properties: - `id` — Primary key derived from the path - `isCollection` — `true` when the request targets a collection - `rowFilter` — Synchronous JavaScript predicate applied to candidate records during search. It cannot be set by REST or QUERY request data. -- `checkPermission` — Framework-owned, one-shot request to run legacy operation authorization. Harper arms it for permission-checked dispatches. Trusted server-side code making a subsequent direct Resource call may set it to `true`, causing built-in table gates to derive permissions from `context.user`. Never accept or copy this field or a permission object from client data. +- `checkPermission` — The same one-shot authorization request available on the [Query Object](#checkpermission). Harper arms it for permission-checked dispatches. Trusted server-side code making a direct Resource call may set it to `true`, causing built-in table gates to derive permissions from `context.user`. Never accept or copy this field or a permission object from client data. Standard `URLSearchParams` methods are available: diff --git a/reference/rest/headers.md b/reference/rest/headers.md index 71603c802..04ca435c9 100644 --- a/reference/rest/headers.md +++ b/reference/rest/headers.md @@ -22,6 +22,19 @@ These headers are included in all Harper REST API responses: Collection responses to a [count request](./querying.md#pagination-and-total-count) additionally include `Content-Range`, `Range-Unit`, and `Preference-Applied` (). +### `Harper-Index-Coverage` + + + +A non-waiting request that uses a full-text index can include `Harper-Index-Coverage`. Its value is one of: + +- `current; lag=0; tolerance=`: the index covered current committed data at admission. +- `bounded; lag=; tolerance=`: the admitted lag upper bound was within the requested tolerance. + +Multiple native searches append one value each. The header certifies coverage when the search is admitted; it does not certify a later traversal. See [Full-text freshness controls](../full-text-search/querying.md#freshness-controls). + +When CORS is enabled, Harper adds this header to `Access-Control-Expose-Headers` so browser clients can read it. + ## Cache-Control diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 45474a83a..9cafc8324 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -55,6 +55,8 @@ Harper uses [FIQL](https://datatracker.ietf.org/doc/html/draft-nottingham-atompu | `=`, `===` | Strict equality (no type conversion) | | `!==` | Strict inequality (no type conversion) | +Full-text indexes support `=matches=`, `=matches_all=`, `=matches_phrase=`, `=matches_prefix=`, `=matches_fuzzy=`, and `=matches_fuzzy_prefix=`. Each has a negated form: `=not_matches=`, `=not_matches_all=`, `=not_matches_phrase=`, `=not_matches_prefix=`, `=not_matches_fuzzy=`, and `=not_matches_fuzzy_prefix=`. Combine a negated condition with a positive full-text condition on the same index; a structured filter alone does not qualify. These use the full-text index rather than ordinary string comparisons. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). + **Examples**: ```http @@ -70,6 +72,34 @@ For date fields, colons must be URL-encoded as `%3A`: GET /Product/?listDate=gt=2017-03-08T09%3A30%3A00.000Z ``` +### Full-Text Operators + + + +For a `FullText` field carrying `@fullText`, use the field name as the query attribute and choose a full-text comparator: + +```http +GET /Product/?catalogSearch=matches=waterproof%20trail +GET /Product/?catalogSearch=matches_phrase=trail%20running +GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&limit(10) +``` + +REST supports `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix`, plus a `not_` form of each. A negated full-text condition requires a non-negated condition on the same index. Repeat the parameter to provide both conditions: + +```http +GET /Product/?catalogSearch=matches=waterproof&catalogSearch=not_matches=leather +``` + +REST's `limit(start,end)` uses an exclusive end. In v5.4, this prefix query skips 100 eligible products and requests the next 10, within request budgets: + +```http +GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&limit(100,110) +``` + +Use `Table.search()` or `search_by_conditions` when conditions need source-field selection, condition-level highlighting, or freshness controls. See [Querying Full-Text Indexes](../full-text-search/querying.md) for ranking, highlights, query combinations, and freshness controls. + +The REST URL syntax searches every source field in the index, so the caller must be allowed to read every source field. Before v5.4, prefix and fuzzy-prefix queries restrict `offset + limit` to 100. In v5.4, bounded full-text and autocomplete queries can pass that native window within request budgets; source-version checks and filters run before records count toward the offset and limit. Each page is a new query and can see concurrent changes. See [paging and prefix result windows](../full-text-search/querying.md#paging-and-prefix-result-windows). Full-text queries cannot provide an exact total count. On a REST interface with `exactCount: true`, a `Prefer: count=exact` request returns an unavailable total (`Content-Range: items /*`). Without that opt-in, Harper applies `count=estimated` instead and can return an estimate. + ### Chained Conditions (Range) Omit the attribute name on the second condition to chain it against the same attribute: @@ -184,14 +214,18 @@ Use `limit(start,end)` to page through a collection, and opt in to a total match Counting is opt-in: without the header, no count is computed and no count headers are returned. It applies only to `GET`/`HEAD` requests and requires a bounded page — the page size (`end - start`, or `end` in the one-argument form) must be a non-negative integer no larger than **10,000**, and the window end (`offset + page size`) no larger than **1,000,000**. A request with no `limit()`, one outside those bounds, or a non-numeric one is served normally with no count headers, since counting an unbounded page would defeat the point of paging. +Full-text and autocomplete queries use the same paging syntax. The page window counts eligible records after record filtering, rather than raw full-text hits. Harper fetches more ranked batches to fill that window when needed. A page can contain fewer records when no further matches remain; reaching a search work or time limit fails the query. A failure detected before response headers can set an error HTTP status. After streaming starts, the response can retain HTTP 200 and end with an error. For JSON-array responses, check for a terminal object shaped like `{ "error": "ErrorName: message" }` and treat the response as failed; do not render that element as a record. For other formats, detect stream or decoding failures as well. Neither status 200 nor count headers prove that a page completed successfully. Each page request runs a new query, so concurrent changes can cause repeats or omissions between pages. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for search work limits and recovery guidance. + +The count-header guardrails do not override query-specific limits such as the full-text work budget. + ### Requesting a count Send a `Prefer` header on a `GET` (or `HEAD`) request to a collection: -| Value | Meaning | -| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `count=exact` | The exact number of matching records. Scans the full matched set, so it is off by default (see [below](#enabling-exact-counts)) and served as an estimate unless enabled for the REST interface. | -| `count=estimated` | A fast planner/table estimate. Cheap, approximate. | +| Value | Meaning | +| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `count=exact` | The exact number of matching records when the query plan can count them. Scans the full matched set, so it is off by default (see [below](#enabling-exact-counts)) and served as an estimate unless enabled for the REST interface. When enabled, full-text and other custom-index queries report an unavailable total instead. | +| `count=estimated` | A fast planner/table estimate. Cheap, approximate. | ```http GET /Product/?category=software&limit(0,25) @@ -219,7 +253,9 @@ The response status is always `200` — `Content-Range` is informational (Harper ### Unavailable totals -The total is reported as `*` (for example `Content-Range: items 0-24/*`) when it cannot be produced — an exact scan that hits its guardrail (counting the tail past the requested page is bounded by a 1,000,000-row cap and a ~1-second budget, and abandons the total rather than truncating the page), or a query with no cardinality estimate (for example a `!=` or `=ct=` (contains) condition). `Preference-Applied` still echoes the requested mode, so an unavailable total (`.../*`) is distinct from a request that asked for no count. +The total is reported as `*` (for example `Content-Range: items 0-24/*`) when it cannot be produced — an exact scan that hits its guardrail (counting the tail past the requested page is bounded by a 1,000,000-row cap and a ~1-second budget, and abandons the total rather than truncating the page), an enabled exact-count request using a full-text or other custom index, or a query with no cardinality estimate (for example a `!=` or `=ct=` (contains) condition). `Preference-Applied` still echoes the requested mode, so an unavailable total (`.../*`) is distinct from a request that asked for no count. + +Full-text and autocomplete queries cannot provide an exact total. With REST exact counting enabled, `Prefer: count=exact` returns an unavailable total (`.../*`) and `Preference-Applied: count=exact`. With exact counting disabled (the default), the request uses estimated mode and `Preference-Applied: count=estimated`; its total is the planner estimate when available, or `*`. An estimated count may use the planner estimate when the query has no opaque row or vector filter. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). ### Enabling exact counts diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index 195c6baa9..84ba47850 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -8,7 +8,7 @@ Harper 5.3.0 is the first stable release of the 5.3 line. It adds native full-te These notes summarize what changed relative to 5.2. The [Harper Pro v5.3.0 release on GitHub](https://github.com/HarperFast/harper-pro/releases/tag/v5.3.0) is the complete list of changes: it includes the Harper core notes, the changes that were already backported to 5.2.x, and notes for anyone who ran a 5.3 pre-release. -### Patch Releases +## Patch Releases All patch release notes for 5.3.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.3&expanded=true). @@ -37,6 +37,8 @@ Read these before upgrading a production node or cluster. Tables can declare `@fullText` fields and query them with BM25-ranked search through the existing `Table.search`, REST, and Operations API paths, with the same authorization as other queries ([harper#2855](https://github.com/HarperFast/harper/pull/2855)). Declared fields are projected into local Tantivy indexes provided by the optional `@harperfast/fulltext` package. RocksDB and the audit log remain the source of truth: the index files can be rebuilt, are reused across restarts when compatible, and are caught up from the audit log before the index reports ready ([harper#2569](https://github.com/HarperFast/harper/pull/2569), [harper#2615](https://github.com/HarperFast/harper/pull/2615), [harper#2616](https://github.com/HarperFast/harper/pull/2616)). Dropping a database or table releases every worker's index handles, and a drop that is interrupted is retried. +Search supports any-term, all-term, phrase, prefix, fuzzy, and fuzzy-prefix matching, with optional source weights, synonyms, scores, and highlights. See [Full-Text Search](/reference/v5/full-text-search/overview) and [Add Full-Text Search to a Product Catalog](/learn/developers/full-text-search-product-catalog). + ## Vector Search: Native HNSW Index A newly declared HNSW index now uses a native, memory-mapped graph file (the optional `@harperfast/hnsw` package) instead of a graph stored in RocksDB, when it is eligible: the table is on RocksDB and has audit enabled (already recorded for the table, or set with `audit: true` in the same declaration), and the index uses the default int8-quantized cosine distance and default graph-structure options (`M`, `efConstruction`, `mL`, `optimizeRouting`). An ineligible index that leaves `nativePlane` unset, or any index that sets `nativePlane: false`, uses the previous graph. Setting `nativePlane: true` on an ineligible index is refused. Existing indexes keep the graph they were built with. Queries rerank only the rows they return, and index lag behind writes is bounded, with a way for a caller to wait until its prior writes are searchable ([harper#2430](https://github.com/HarperFast/harper/pull/2430), [harper#2657](https://github.com/HarperFast/harper/pull/2657), [harper#2658](https://github.com/HarperFast/harper/pull/2658)). diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 3b5aa53f6..2ae81453e 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -8,6 +8,10 @@ title: '5.4' All patch release notes for 5.4.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.4&expanded=true). +## Upgrade Notes + +Full-text indexes using fulltext 0.5.0 with current field-based declarations, or prerelease 0.6, rebuild locally after this upgrade. Plan for search unavailability and review the [Tantivy upgrade requirements](#tantivy-027-and-existing-index-rebuilds) before deploying; direct downgrade to fulltext 0.5.x is unsupported. + ## Nested Relationship Constraints [Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279) corrects REST and programmatic nested selections that accepted filters or pagination but returned all related records. Filters, ordering, offset, and limit apply per parent in collection and single-record reads. Parents remain present when the nested selection has no matches. Outer relationship filters also constrain the selected children, so an identical filter does not need to be repeated inside the nested selection. @@ -80,6 +84,20 @@ The new `threads.v8Flags` setting takes a V8 flag, or a list of them, and applie `restore_backup` no longer deletes a database before checking that the backup can be read. It first verifies the checksum of every database file in the backup, checks its transaction logs, opens each table file to confirm this version can read its format and compression, and, for a backup that captured blobs, checks that the blob snapshot is present. A backup that fails is refused and the database is left as it was (one an earlier restore left incomplete stays blocked); previously the restore deleted the database first and then failed, so a backup taken by a newer version, or one with a corrupt file, destroyed the database it was meant to restore. The check reads the whole backup once, so a restore takes longer, and an online restore keeps the database unavailable while it runs; it needs no extra disk space. A restore interrupted while it is still checking the backup leaves the destination's files and any preexisting incomplete-restore marker unchanged; a previously healthy database loads normally on the next start. Blob file contents are not checked. See [`restore_backup`](/reference/v5/backups/operations#restore_backup). +## Full-Text Search + +### Tantivy 0.27 and Existing Index Rebuilds + +Harper now uses `@harperfast/fulltext` 0.7.0 with Tantivy 0.27.0. Harper 5.3.1–5.3.4 ship fulltext 0.5.0. Existing 0.5.0 indexes using current field-based declarations, and prerelease 0.6 indexes, rebuild automatically from authoritative records. Harper 5.3.0 ships fulltext 0.3.0 and is outside this qualified procedure; validate that migration separately. Earlier beta declarations and indexes have no supported migration path. Plan for local full-text unavailability, disk headroom and possible retryable write failures while replay catches up. Stop old processes before starting the new version against the same storage; direct downgrade to fulltext 0.5.x is unsupported. + +English stemming changes despite the unchanged `english@2` analyzer name: `added` now stems to `add` rather than `ad`. Matching and ranking can change. Query modes and schema options are unchanged; benchmark results from the wrapper do not establish application performance parity. Review representative searches and see [upgrading existing indexes](/reference/v5/full-text-search/overview#upgrading-existing-indexes) before restoring traffic. + +### Paging and Search Work Limits + +Full-text and autocomplete queries use the existing `limit` and `offset` paging options. Harper reads additional native batches when needed after filtering, so the initial native result window no longer silently cuts off eligible results. The new `storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds` settings bound fetched hits, retained results and execution time. Requests that exceed those budgets return an error. The `storage.fullText.runtime` section also provides process-wide native-capacity budgets; changes to those budgets require a full Harper process restart. Separate page requests run independently and can see changes to records or index contents. + +See [Full-text query execution](/reference/v5/resources/query-optimization#full-text-search-work-and-paging) and [storage configuration](/reference/v5/configuration/options#storage) for limits, restart requirements and recovery guidance. + ## Native Addons ### Pointer-Compression Runtimes Refuse Incompatible Native Addons diff --git a/sidebarsLearn.ts b/sidebarsLearn.ts index 721b1690c..e77f63c0d 100644 --- a/sidebarsLearn.ts +++ b/sidebarsLearn.ts @@ -66,6 +66,11 @@ const sidebarsLearn: SidebarsConfig = { id: 'developers/write-through-caching', label: 'Write-Through Caching', }, + { + type: 'doc', + id: 'developers/full-text-search-product-catalog', + label: 'Full-Text Product Search', + }, { type: 'doc', id: 'developers/deploying-from-ci', diff --git a/sidebarsReference.ts b/sidebarsReference.ts index f997c1810..8ce7f8104 100644 --- a/sidebarsReference.ts +++ b/sidebarsReference.ts @@ -89,6 +89,29 @@ const sidebars: SidebarsConfig = { }, ], }, + { + type: 'category', + label: 'Full-Text Search', + collapsible: false, + className: 'reference-category-header', + items: [ + { + type: 'doc', + id: 'full-text-search/overview', + label: 'Overview', + }, + { + type: 'doc', + id: 'full-text-search/configuration', + label: 'Configuration', + }, + { + type: 'doc', + id: 'full-text-search/querying', + label: 'Querying', + }, + ], + }, { type: 'category', label: 'AI & Models',