Document native full-text search

CONTRIBUTING.md

@@ -235,7 +235,7 @@ The Plugin API is the primary way to implement additional functionality in Harpe
 - `logger.level` — Log level; _Default_: `"info"` (Added in: v4.1.0)
 ```
 
-For inline config option annotations inside list items, plain text `(Added in: vX.Y.Z)` is fine — using the component mid-sentence is awkward. Reserve `<VersionBadge>` for standalone placement after headings.
+Use `<VersionBadge>` after headings or alongside the specific sentence, list item, or table entry it annotates. Use `<EngineBadge engines="RocksDB" />` the same way when behavior is storage-engine-specific; comma-separated values such as `engines="RocksDB, LMDB"` are supported. Keep adjacent version and engine badges together on one line. For dense prose where a component would interrupt the sentence, plain text such as `(Added in: vX.Y.Z)` is fine.
 
 ## Redirects
 

learn/developers/full-text-search-product-catalog.mdx

@@ -0,0 +1,193 @@
+---
+title: Add Full-Text Search to a Product Catalog
+description: Configure and query a BM25-ranked product search in Harper.
+---
+
+# Add Full-Text Search to a Product Catalog
+
+<VersionBadge version="v5.3.0" />
+
+This guide adds relevance-ranked text search to a product catalog, then exposes it through Harper's JavaScript and REST APIs.
+
+## What you will build
+
+- A BM25-ranked index over product names, descriptions, and tags
+- A Harper application endpoint with highlights and a structured category filter
+- REST queries for terms, phrases, and record autocomplete
+
+## Prerequisites
+
+- Harper 5.3 or later using RocksDB
+- A component with a GraphQL schema, JavaScript resources, and `rest: true` in its `config.yaml`
+
+## 1. Declare the index
+
+Declare an `@fullText` index over the fields customers search on a RocksDB-backed table. Ordinary tables inherit enabled audit logging by default, but a new full-text table requires explicit `audit: true` for its transaction-log recovery source. See the [audit requirement](/reference/v5/full-text-search/configuration) for existing tables.
+
+```graphql title="schema.graphql"
+type Product @table(database: "catalog", audit: true) @export {
+	id: ID @primaryKey
+	name: String
+	description: String
+	tags: [String]
+	category: String @indexed
+	price: Float @indexed
+	available: Boolean
+	catalogSearch: FullText
+		@fullText(
+			fields: [
+				{ name: "name", weight: 3, highlight: true }
+				{ name: "description", weight: 1, highlight: true }
+				{ name: "tags" }
+			]
+			highlighting: { maxFragments: 2, fragmentLength: 120 }
+		)
+}
+```
+
+The query-only field name `catalogSearch` identifies the index. It is not stored with each product. `weight: 3` makes a match in `name` contribute more to BM25 relevance than the same match in `description` or `tags`. Highlighting is enabled only for the fields shown to customers.
+
+The `@indexed` declarations create ordinary Harper secondary indexes. Full-text searches can use them to narrow candidates, and Harper checks structured conditions against current product records before returning results.
+
+## 2. Add records
+
+Use the generated REST endpoint:
+
+```http
+POST /Product/
+Content-Type: application/json
+
+{
+	"id": "shoe-1",
+	"name": "Waterproof trail running shoe",
+	"description": "Lightweight shoe for wet mountain trails",
+	"tags": ["outdoor", "trail"],
+	"category": "footwear",
+	"price": 129,
+	"available": true
+}
+```
+
+Harper commits the record first, then applies the committed change to the local derived full-text index. Each replica performs the same derivation after it has received both the full-text schema declaration and the record transactions.
+
+If the index remains behind for more than 30 seconds, a local write can return retryable `503` code `DERIVED_INDEX_LAGGING`. Retry writes with backoff while the index catches up; freshness options apply only to searches. See [write backpressure](/reference/v5/full-text-search/overview#write-backpressure).
+
+## 3. Search from a Harper application
+
+```javascript title="resources.js"
+export class SearchProducts extends Resource {
+	static async post(_target, data, context) {
+		if (!context?.user) {
+			const error = new Error('Authentication required');
+			error.statusCode = 401;
+			throw error;
+		}
+		const body = (await data) ?? {};
+		const text = typeof body.q === 'string' ? body.q.trim() : '';
+		if (!text) {
+			const error = new Error('q is required');
+			error.statusCode = 400;
+			throw error;
+		}
+		const conditions = [
+			{
+				attribute: 'catalogSearch',
+				comparator: 'matches',
+				value: text,
+				fields: ['name', 'description', 'tags'],
+			},
+			{ attribute: 'available', comparator: 'equals', value: true },
+		];
+		const category = body.category;
+		if (category) conditions.push({ attribute: 'category', comparator: 'equals', value: category });
+		return databases.catalog.Product.search(
+			{
+				conditions,
+				select: ['id', 'name', 'price', '$score', '$highlights'],
+				limit: 20,
+				checkPermission: true,
+			},
+			context
+		);
+	}
+}
+```
+
+Call the resource with an authenticated request:
+
+```http
+POST /SearchProducts/
+Content-Type: application/json
+
+{ "q": "trail", "category": "footwear" }
+```
+
+This custom resource accepts a structured JSON search body through POST. The category filter is optional; `q` is required. For URL-based GET searches, use the exported table endpoint in step 4.
+
+Harper applies the availability and optional category conditions while accepting current product records. Existing secondary indexes can narrow candidates; rejected records do not count toward the requested page.
+
+Calls through `tables` and `databases` use Harper's [trusted server-side context](/reference/v5/components/javascript-environment#tables). Forwarding the resource's `context` preserves the authenticated caller, and `checkPermission: true` asks the table to enforce that caller's table and source-field permissions. The custom resource must do both. Highlights can expose searched source text even when those fields are absent from `select`.
+
+Matches in `tags` contribute to search results and ranking, but `$highlights` does not include tag matches because `tags` is not configured with `highlight: true`.
+
+## 4. Search through REST
+
+The exported table supports the same match comparator:
+
+```http
+GET /Product/?catalogSearch=matches=waterproof%20trail&category=footwear&select(id,name,price,$score,$highlights)&limit(20)
+```
+
+For a phrase:
+
+```http
+GET /Product/?catalogSearch=matches_phrase=trail%20running&limit(20)
+```
+
+For record autocomplete:
+
+```http
+GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&select(id,name)&limit(10)
+```
+
+Prefix search returns matching products. It does not return a dictionary of suggested terms. Before v5.4, prefix and fuzzy-prefix queries restrict `offset + limit` to 100. In v5.4, bounded pages can go beyond that window within request budgets; each page runs a new query rather than retaining a search. See [paging and prefix result windows](/reference/v5/full-text-search/querying#paging-and-prefix-result-windows).
+
+REST searches every source field in the index, so the caller must be allowed to read `name`, `description`, and `tags` for these requests. Use `Table.search()` or `search_by_conditions` when a role should search only a subset of sources. See [REST full-text operators](/reference/v5/rest/querying#full-text-operators).
+
+## 5. Choose consistency behavior
+
+Most searches can use the default three-second lag tolerance. When the next request must find a record already committed on the queried node, use the JavaScript API and wait for current index coverage. This wait does not wait for replication to deliver writes from another node:
+
+```javascript
+for await (const product of databases.catalog.Product.search({
+	conditions: [
+		{
+			attribute: 'catalogSearch',
+			comparator: 'matches',
+			value: 'new seasonal product',
+			maxIndexLagMilliseconds: 0,
+			waitForIndexMilliseconds: 10000,
+		},
+	],
+})) {
+	console.log(product);
+}
+```
+
+Do not apply this wait to every catalog request without measuring it. The normal bounded-staleness path keeps search traffic independent of short indexing bursts.
+
+If the index does not reach current coverage within ten seconds, iteration throws an error with `statusCode: 503`, `code: "DERIVED_INDEX_LAGGING"`, and `retryable: true`. See [freshness controls](/reference/v5/full-text-search/querying#freshness-controls).
+
+## 6. Operate the index
+
+Use `describe_table` to inspect readiness. Harper reuses compatible local index files after restart and replays changes after their checkpoint. A node rebuilds locally when the files are missing, incompatible, or corrupt; ordinary table reads remain available, while full-text requests in `unknown`, `needs-rebuild`, or `rebuilding` state return `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. Retry this code with backoff; do not treat every `503` as an index rebuild. Writes can also receive retryable `DERIVED_INDEX_LAGGING` responses while replay catches up after a long rebuild; see [write backpressure](/reference/v5/full-text-search/overview#write-backpressure).
+
+The custom resource in step 3 streams its returned iterable. HTTP error statuses apply before response headers are sent; a failure after streaming starts is a terminal stream error and cannot change the HTTP status. Consumers must also handle that terminal error. See [freshness and streaming errors](/reference/v5/full-text-search/querying#freshness-controls).
+
+See the [full-text configuration reference](/reference/v5/full-text-search/configuration) and [query reference](/reference/v5/full-text-search/querying) for every option and match mode.
+
+## Additional resources
+
+- [Full-text search overview](/reference/v5/full-text-search/overview)
+- [Resource API](/reference/v5/resources/resource-api)
+- [Query optimization](/reference/v5/resources/query-optimization)

reference/backups/overview.md

@@ -5,7 +5,7 @@ title: Backups
 
 <!-- Source: HarperFast/harper#1831 (RocksDB managed backup and restore operations) -->
 
-Harper can back up and restore its databases natively. A backup is a whole-database copy — all tables, the audit/transaction log, and any file-backed blobs — so a restored database is a consistent point-in-time image of everything in it. (`get_backup` on an LMDB database is the exception: it can stream a subset of tables, and includes the audit store only when requested.)
+Harper can back up and restore its databases natively. A backup is a whole-database copy of authoritative state — all tables, the audit/transaction log, and any file-backed blobs — so a restored database is a consistent point-in-time image. Local derived state, including full-text index files, is excluded and rebuilt after restore. (`get_backup` on an LMDB database is the exception: it can stream a subset of tables, and includes the audit store only when requested.)
 
 Two complementary mechanisms are available:
 
@@ -38,6 +38,7 @@ The offline path matters for restore. RocksDB is single-writer, so an in-place r
 - **Backups live on the node that created them.** The backup repository is a local directory. RocksDB shares files across backup IDs (a backup ID is not a self-contained folder), so disaster-recovery copies must take the entire per-database repository — `<backupPath>/<database>` — not an individual backup, and must do so while no backup operation is running, or from an atomic filesystem snapshot. A live recursive copy can race `create_backup`/`delete_backup`/`purge_backups` and produce an unrestorable copy. Alternatively, use `get_backup` to pull a snapshot from a running server.
 - **`get_backup` always streams the current state.** It cannot download a historical managed backup; to move a retained backup off-host, copy its whole per-database repository as above.
 - **A restore is a point-in-time rollback.** In a replicated cluster, coordinate a restore with replication before bringing the node back.
+- **Full-text indexes rebuild locally after restore.** Ordinary table reads remain available. Writes can receive retryable `DERIVED_INDEX_LAGGING` responses while replay catches up after a long rebuild; see [write backpressure](../full-text-search/overview.md#write-backpressure). While the rebuild is active, full-text requests return `503` with `code: "INDEX_REBUILDING"` and `retryable: true`; branch on the code because other `503` responses are not necessarily rebuilds.
 - **A restore checks the backup before it deletes anything.** <VersionBadge type="changed" version="v5.4.0" /> A backup with a corrupt database file, or one this version of Harper cannot read, is refused and the database is left as it was (a database an earlier restore left incomplete stays blocked). The check reads the whole backup, which adds to how long an online restore keeps the database unavailable. Blob file contents are not checked. See [`restore_backup`](./operations.md#restore_backup).
 - **An interrupted restore leaves the database unloadable.** If `restore_backup` is interrupted after it has started replacing the database's files (crash, power loss), Harper marks the database as incompletely restored and skips loading it on the next start, logging an incomplete-restore error. Rerun `restore_backup` for the same database and `backup_id` to recover; do not load or hand-repair the directory.
 

reference/components/javascript-environment.md

@@ -90,7 +90,7 @@ An object whose properties are the tables in the default database (`data`). Each
 See [Database API](../database/api.md) for full reference.
 
 :::note Server-side table reads run in a trusted context
-Calls to `tables.X.get()`, `tables.X.search()`, and equivalent `databases.<database>.<table>` methods from within resource code execute in a **trusted system context** and do **not** re-apply the target table's `allowRead` guard or role-level `attribute_permissions`. This is by design — server-side code is responsible for its own authorization checks. A computed attribute or custom resource that cross-reads a protected table can return raw data regardless of the caller's role. Do not expose protected data through a computed attribute when access depends on the caller. Instead, use a custom resource and apply authorization explicitly (e.g. call `this.getCurrentUser()` in a `Resource` method, or use `getUser()`, and enforce the relevant restrictions before returning the data).
+Calls to `tables.X.get()`, `tables.X.search()`, and equivalent `databases.<database>.<table>` methods from within resource code execute in a **trusted system context** and do **not** re-apply the target table's `allowRead` guard or role-level `attribute_permissions` by default. This is by design — server-side code is responsible for its own authorization checks. A custom resource can explicitly re-enter table and source-field authorization for `Table.search()` by forwarding its authenticated `context` and setting `checkPermission: true` in search options. Never accept or copy that flag or a permission object from client data. Context propagation alone does not request the check. Other direct Resource methods also support the [checkPermission option](../resources/resource-api.md#checkpermission); use it with authenticated context when delegating those calls, or enforce the required authorization in application code. A computed attribute or custom resource that cross-reads a protected table can otherwise return raw data regardless of the caller's role. Do not expose protected data through a computed attribute when access depends on the caller. Instead, use a custom resource and enforce the relevant restrictions before returning the data.
 :::
 
 ### `databases`

reference/configuration/operations.md

@@ -16,14 +16,18 @@ For the full list of configurable options, see [Configuration Options](./options
 
 ## Set Configuration
 
-Modifies one or more Harper configuration parameters. **Requires a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`). A change to `replication.threads` takes effect only on a full `restart`.
+Modifies one or more Harper configuration parameters. **Most changes require a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`). A change to `replication.threads` takes effect only on a full `restart`.
+
+Full-text request work limits (`storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds`) take effect in HTTP workers after `restart_service` or a full restart. Changes to `storage.fullText.runtime` require a full Harper process restart; `restart_service` does not replace the native process budget. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging).
 
 `operation` _(required)_ — must be `set_configuration`
 
 `replicated` _(optional)_ <VersionBadge version="v5.2.0" /> — set to `true` to apply the same configuration change to all cluster nodes in a single call. The origin node applies the change first; if its local write fails, nothing is sent to peers. Requires replication to be configured.
 
 Additional properties correspond to configuration keys in underscore-separated format (e.g. `logging_level` for `logging.level`, `clustering_enabled` for `clustering.enabled`).
 
+For full-text settings, use individual parameter names such as `storage_fullText_maxRawHitsPerRequest` or `storage_fullText_runtime_maxSearchThreads`. `set_configuration` does not accept a whole `storage_fullText` object; use `storage.fullText` as the section name only in the YAML configuration file.
+
 :::warning Replicate cluster-appropriate parameters only
 A replicated call sends the exact same values to every node. Do not include node-local parameters — ports, `node.hostname`, `rootPath`, storage/log file paths, TLS certificate or key paths, or `replication.hostname`/`url`/`routes` — as they would overwrite each peer's local values.
 :::
@@ -42,7 +46,7 @@ A replicated call sends the exact same values to every node. Do not include node
 
 ```json
 {
-	"message": "Configuration successfully set. You must restart HarperDB for new config settings to take effect."
+	"message": "Configuration successfully set. You must restart Harper for new config settings to take effect."
 }
 ```