diff --git a/reference/configuration/operations.md b/reference/configuration/operations.md index 2e9b089b3..eda982b63 100644 --- a/reference/configuration/operations.md +++ b/reference/configuration/operations.md @@ -18,7 +18,7 @@ For the full list of configurable options, see [Configuration Options](./options Modifies one or more Harper configuration parameters. **Most changes require a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`, [`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) and [`storage.rocks.transactionLogWriteIntents`](../database/storage-tuning.md#storagerockstransactionlogwriteintents)). A change to `replication.threads` takes effect only on a full `restart`. -Full-text request work limits (`storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds`) take effect in HTTP workers after `restart_service` or a full restart. Changes to `storage.fullText.runtime` require a full Harper process restart; `restart_service` does not replace the native process budget. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). +Full-text request work limits (`storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds`) take effect in HTTP workers after `restart_service` or a full restart. Changes to `storage.fullText.query` or `storage.fullText.runtime` require a full Harper process restart; `restart_service` replaces neither the captured query policy nor native process budget. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). `operation` _(required)_ — must be `set_configuration` @@ -26,7 +26,7 @@ Full-text request work limits (`storage.fullText.maxRawHitsPerRequest`, `storage Additional properties correspond to configuration keys in underscore-separated format (e.g. `logging_level` for `logging.level`, `clustering_enabled` for `clustering.enabled`). -For full-text settings, use individual parameter names such as `storage_fullText_maxRawHitsPerRequest` or `storage_fullText_runtime_maxSearchThreads`. `set_configuration` does not accept a whole `storage_fullText` object; use `storage.fullText` as the section name only in the YAML configuration file. +For full-text settings, use individual parameter names such as `storage_fullText_maxRawHitsPerRequest`, `storage_fullText_query_maxQueryTerms`, `storage_fullText_query_maxQueryClauses`, `storage_fullText_query_maxFuzzyDistance`, or `storage_fullText_runtime_maxSearchThreads`. `set_configuration` does not accept a whole `storage_fullText` object; use `storage.fullText` as the section name only in the YAML configuration file. :::warning Replicate cluster-appropriate parameters only A replicated call sends the exact same values to every node. Do not include node-local parameters — ports, `node.hostname`, `rootPath`, storage/log file paths, TLS certificate or key paths, or `replication.hostname`/`url`/`routes` — as they would overwrite each peer's local values. diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 005e5ecac..423a7b26d 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -320,6 +320,9 @@ storage: - `maxRawHitsPerRequest` — Maximum fetched hits per request, including bounded prefix replay when traversal is needed; default `10000`. - `maxResultsPerRequest` — Maximum retained offset plus limit. A request must also fit the separate raw-hit budget; default `10000`. - `requestBudgetMilliseconds` — Execution budget across native batches, source checks and highlights, capped by the native execution limit; default `5000` ms. Reader acquisition and freshness waiting have separate bounds. + - `query.maxQueryTerms` — Maximum analyzed terms per full-text leaf; integer `1`–`1024`, default `64`, also bounded by the loaded native runtime. + - `query.maxQueryClauses` — Maximum compiled clauses, including Boolean structure and field expansion; integer `1`–`4096`, default `256`, also bounded by the loaded native runtime. + - `query.maxFuzzyDistance` — Maximum accepted fuzzy edit distance; integer `0`–`2`, default `1`. A cap of `0` requires explicit `fuzzyDistance: 0` on fuzzy conditions; the omitted distance still defaults to `1` and is rejected. Distance `2` requires operator opt-in and workload qualification. - `runtime.maxResidentIndexes` — Maximum resident native indexes; default `128`. - `runtime.maxIndexingThreads` — Maximum aggregate indexing-thread reservations across resident native indexes; default `128`. - `runtime.maxSearchThreads` — Maximum aggregate search-thread reservations across resident native indexes; default `256`. @@ -327,6 +330,8 @@ storage: - `runtime.maxQueuedBytes` — Aggregate queued-write reservations across resident native indexes; default `8589934592` (8 GiB). - `runtime.maxExpensiveSearches` — Maximum concurrent expensive native searches; default `8`. + The `query` policy is captured for the whole process and inherited by application and HTTP workers. A full Harper process restart is required; restarting only workers does not change it. Omitted keys keep their defaults. Requests cannot raise the policy. It changes query admission, not persisted index identity. Harper 5.4.0-alpha.1/alpha.2 reject an explicit `storage.fullText.query` block. + The runtime budgets apply together and are not hard bounds on total process memory. See [Set Configuration](./operations.md#set-configuration) for when request and native runtime changes take effect. Use the usual underscore-separated parameter names, such as `storage_fullText_maxRawHitsPerRequest`. See [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for costs; see [readiness and recovery](../resources/query-optimization.md#readiness-and-recovery) for failures. - `writeAsync` — Disable disk sync for higher throughput (**disables durability guarantees**); _Default_: `false` diff --git a/reference/full-text-search/configuration.md b/reference/full-text-search/configuration.md index f7434c676..ffb04df2d 100644 --- a/reference/full-text-search/configuration.md +++ b/reference/full-text-search/configuration.md @@ -147,7 +147,7 @@ type Document @table(audit: true) { ## Schema changes and rebuilds -The `english@2` name remains unchanged in v5.4, but Tantivy 0.27 changes some English stems. Existing indexes must rebuild, and matching or ranking can change. See [upgrading existing indexes](./overview.md#upgrading-existing-indexes). +The `english@2` name remains unchanged in v5.4, but Tantivy 0.27 changes some English stems. Indexes from fulltext 0.5.0 or prerelease 0.6 must rebuild, and matching or ranking can change. Current Harper indexes written by fulltext 0.7.0 or 0.8.0 already use Tantivy 0.27 and can reuse compatible checkpoints with 0.9.0. See [upgrading existing indexes](./overview.md#upgrading-existing-indexes). Changes that affect indexed storage create a new local index generation and rebuild from source data. These include source membership, source order, media types, analyzer settings, stop-word handling, positions, surface terms, and synonyms. Keep the `fields` list in a stable order when its meaning has not changed. diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index 39d9df04b..b026967d9 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -10,7 +10,7 @@ Harper can build a BM25-ranked full-text index from one or more fields on a tabl Use full-text search when users need relevance-ranked matching across product names, descriptions, tags, articles, or other natural-language fields. It supports term, all-term, phrase, prefix, fuzzy, and fuzzy-prefix matching. - Starting in Harper v5.4, the native engine is **`@harperfast/fulltext` 0.7.0, embedding Tantivy 0.27.0**. Existing fulltext 0.5.0 and prerelease 0.6 indexes require rebuilding. See [Upgrading existing indexes](#upgrading-existing-indexes) before upgrading a node. + Starting in Harper v5.4, the native engine is **`@harperfast/fulltext` 0.9.0, embedding Tantivy 0.27.0**. Existing fulltext 0.5.0 and prerelease 0.6 indexes require rebuilding. See [Upgrading existing indexes](#upgrading-existing-indexes) before upgrading a node. ## Requirements @@ -59,16 +59,22 @@ Use `describe_table` to inspect each index's declaration and readiness. See [Ins -Harper 5.3.1–5.3.4 ship `@harperfast/fulltext` 0.5.0. Upgrading their current field-based indexes to the 0.7.0 engine in Harper v5.4 requires a full local rebuild. Harper 5.3.0 ships fulltext 0.3.0 and is outside this qualified procedure; plan and validate that migration separately. Prerelease installations using fulltext 0.6 also require rebuilding. This procedure applies to the current field-based declarations; earlier beta declarations and indexes have no supported migration path. Harper initiates the rebuild automatically when the wrapper refuses the old persisted identity; no manual rebuild command or file editing is needed. +Harper 5.3.1–5.3.4 ship `@harperfast/fulltext` 0.5.0. Upgrading their current field-based indexes to the 0.9.0 engine in Harper v5.4 requires a full local rebuild. Harper 5.3.0 ships fulltext 0.3.0 and is outside this qualified procedure; plan and validate that migration separately. Prerelease installations using fulltext 0.6 also require rebuilding. This procedure applies to the current field-based declarations; earlier beta declarations and indexes have no supported migration path. Harper initiates the rebuild automatically when the wrapper refuses the old persisted identity; no manual rebuild command or file editing is needed. + +### Upgrading from fulltext 0.7.0 or 0.8.0 + +Harper 5.4.0-alpha.1 and alpha.2 ship fulltext 0.7.0. Their existing indexes, and indexes created by development builds using fulltext 0.8.0, remain compatible with 0.9.0 and can resume from their saved checkpoint without rebuilding. These are indexes created through Harper's `@fullText` declaration; Harper does not create native filter metadata. Query controls do not change their stored identity. Stop all processes sharing a local index before switching versions; do not mix facade/native package versions or run different engines against one live directory. Check readiness, current coverage and representative searches before restoring traffic, as described in the checklist below. + +Returning to fulltext 0.7.0 or 0.8.0 after 0.9.0 has written to the index is not a qualified operator rollback procedure. If returning to Harper 5.4.0-alpha.1 or alpha.2, remove an explicitly configured `storage.fullText.query` block: those versions reject it. Removing that configuration alone does not qualify a downgrade. If an older build was already started, stop it, restore the 0.9.0-based Harper build, and check readiness and current coverage before restoring full-text traffic. ### Matching and ranking changes -Tantivy 0.27 changes English stemming even though the schema analyzer remains `english@2`. For example, `added` now stems to `add` instead of `ad`. Queries can match different records or rank them differently after rebuilding. Review representative searches and domain vocabulary before restoring production traffic; the upgrade does not add new Harper query modes or change the declared field options. +When upgrading from fulltext 0.5.0 or prerelease 0.6, Tantivy 0.27 changes English stemming even though the schema analyzer remains `english@2`. For example, `added` now stems to `add` instead of `ad`. Queries can match different records or rank them differently after rebuilding. Review representative searches and domain vocabulary before restoring production traffic; the engine migration does not change the declared field options. Query-time relevance controls are described separately in [Querying](./querying.md#query-time-relevance-controls). ### Upgrade checklist 1. Drain the node's full-text requests and stop all old Harper processes that share its storage, including background workers. Do not run old and new engine versions against one local index. -2. Confirm authoritative records and recovery backups are available. Reserve disk headroom and time for rebuilding every local full-text index, including temporarily retained retired files. Measure rebuild duration and disk growth with representative data before setting the rollout window. +2. Confirm authoritative records and recovery backups are available. For fulltext 0.5.0 or prerelease 0.6, reserve disk headroom and time for rebuilding every local full-text index, including temporarily retained retired files. Compatible 0.7/0.8 indexes normally resume from checkpoint; missing, corrupt or unrecoverably lagging files can still require rebuilding. Plan rebuild capacity for that recovery case. 3. Start the v5.4 Harper version. Use [`describe_table`](../operations-api/operations.md#describe_table) for each affected table, using an authorized operator account that already has read access to every indexed source field, and check every `full_text_indexes[].readiness.state`. Metadata is permission-filtered; a missing entry does not prove that no index exists. Keep full-text traffic drained until all required indexes are `ready`. 4. Run representative searches through the same REST or permission-checked application path and roles used in production, with `maxIndexLagMilliseconds: 0` and `waitForIndexMilliseconds: 30000` on each full-text condition. These wait for coverage of writes committed on the queried node before the search; they do not wait for replication delivery, and readiness alone does not establish that coverage. Check response bodies for terminal stream errors as well as HTTP status. A JSON-array response can start with HTTP 200 and end with an `error` element. 5. Retry write responses carrying `DERIVED_INDEX_LAGGING` with bounded backoff while replay catches up. Restore full-text traffic only after readiness and current-coverage searches succeed. A node-by-node upgrade can use other ready replicas only if the application's routing supports it; the rebuilding node remains unavailable for full-text search. During a mixed-version rollout, matching and ranking can differ between replicas. Pre-v5.4 replicas also reject prefix requests whose `offset + limit` exceeds 100 with `400`. Route related searches consistently and check the application's behavior before using this approach. @@ -79,7 +85,7 @@ See [freshness controls](./querying.md#freshness-controls) for the condition fie Keep full-text traffic drained, preserve authoritative data and logs, and correct the reported package, storage or coordination failure before restarting the same v5.4 Harper version. An interrupted rebuild can recover from authoritative records. Do not force readiness or edit, move or delete native index files by hand. -**Direct downgrade to fulltext 0.5.x is unsupported.** That engine refuses both opening and resetting a 0.7 index identity. If an older version has already been started, stop it and restore the 0.7.0-based Harper version before checking readiness and catch-up again. This is recovery with the current engine, not a supported operator downgrade procedure. If recovery also fails, keep traffic drained and investigate the readiness reason and node logs. +**Direct downgrade to fulltext 0.5.x is unsupported.** That engine refuses both opening and resetting the current Tantivy 0.27 index identity. If an older version has already been started, stop it and restore the 0.9.0-based Harper version before checking readiness and catch-up again. This is recovery with the current engine, not a supported operator downgrade procedure. If recovery also fails, keep traffic drained and investigate the readiness reason and node logs. ## Write backpressure @@ -89,7 +95,7 @@ Retry rejected writes with backoff rather than immediately adding more load. Ins ## Ranking and result loading -The native index ranks matching documents with BM25. Harper then loads the authoritative table records for the matching IDs and discards stale or deleted candidates. Selecting `$score` includes the relevance score in each result. +The native index ranks matching documents with BM25, adjusted by any supplied [query-time boosts and optional-clause contributions](./querying.md#query-time-relevance-controls) in v5.4. Harper then loads the authoritative table records for the matching IDs and discards stale or deleted candidates. Selecting `$score` includes the relevance score in each result. Full-text search can be combined with structured filters. Harper checks conditions against current source records before returning them. In v5.4, additional ranked batches can fill a filtered page within request budgets; exhausting a budget returns an error rather than a budget-truncated page. diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 4980eb120..0cf008252 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -17,18 +17,18 @@ const products = Product.search({ `Table.search()` returns an async iterable. Consume it with `for await...of`, or return it directly from a custom Resource to stream the results. -Results are ordered by BM25 relevance. The `FullText` field, not one of its source fields, identifies the search target. +Results are ordered by BM25 relevance. In v5.4, supplied [query-time boosts and optional clauses](#query-time-relevance-controls) adjust that ranking. The `FullText` field, not one of its source fields, identifies the search target. ## Match modes -| Comparator | Behavior | Required index option | -| ---------------------- | --------------------------------------------------------------------- | --------------------- | -| `matches` | Matches any analyzed query term. | — | -| `matches_all` | Requires every analyzed query term. | — | -| `matches_phrase` | Matches analyzed terms in order as a phrase. | `positions: true` | -| `matches_prefix` | Matches records whose analyzed terms start with the final query term. | `surfaceTerms: true` | -| `matches_fuzzy` | Matches terms within one edit, including a transposition. | — | -| `matches_fuzzy_prefix` | Combines one-edit fuzzy matching with a final-term prefix. | `surfaceTerms: true` | +| Comparator | Behavior | Required index option | +| ---------------------- | ---------------------------------------------------------------------- | --------------------- | +| `matches` | Matches any analyzed query term. | — | +| `matches_all` | Requires every analyzed query term. | — | +| `matches_phrase` | Matches analyzed terms in order as a phrase. | `positions: true` | +| `matches_prefix` | Matches records whose analyzed terms start with the final query term. | `surfaceTerms: true` | +| `matches_fuzzy` | By default, matches terms within one edit, including a transposition. | — | +| `matches_fuzzy_prefix` | By default, combines one-edit fuzzy matching with a final-term prefix. | `surfaceTerms: true` | Every comparator also has a negated form: `not_matches`, `not_matches_all`, `not_matches_phrase`, `not_matches_prefix`, `not_matches_fuzzy`, and `not_matches_fuzzy_prefix`. @@ -69,7 +69,7 @@ Direct calls through `tables` or `databases` run in a trusted server-side contex ## Score and ordering -Select `$score` to include the BM25 score: +Select `$score` to include the relevance score. Without query-time controls, this is the BM25 score. In v5.4, [boosts and optional clauses](#query-time-relevance-controls) adjust it when supplied: ```javascript const products = Product.search({ @@ -83,6 +83,81 @@ Full-text results use descending relevance order. The only explicit full-text so Full-text queries cannot provide an exact total after authorization, structured filtering, and current-record validation. On a REST interface with `exactCount` enabled, `Prefer: count=exact` reports an unavailable total; otherwise it is downgraded to an estimated count. See [pagination and total count](../rest/querying.md#pagination-and-total-count). `Table.getRecordCount()` counts table records, not matches for a full-text query. +## Fuzzy controls in v5.4 + + + +`fuzzyDistance` and `fuzzyTranspositions` can be set on structured `matches_fuzzy` and `matches_fuzzy_prefix` conditions, including their negated forms. Supplying them on another match mode or a group object returns HTTP 400; put the controls on its individual fuzzy conditions. Values with the wrong type also return HTTP 400. `fuzzyDistance` must be an integer from 0 to 2. `fuzzyTranspositions` must be a boolean. Omitted values default to `1` and `true`, respectively. + +The server setting `storage.fullText.query.maxFuzzyDistance` accepts integers from `0` to `2`, defaults to `1`, and caps accepted distances. With a cap of `0`, a fuzzy condition must explicitly set `fuzzyDistance: 0`. Omitting it still defaults to `1` and returns HTTP 400. Distance `2` requires a cap of `2`. + +Fuzzy edits apply only to eligible analyzed surface terms: at least four characters, alphabetic content, and no numeric characters, hyphens, underscores or slashes. Other terms receive no fuzzy expansion; fuzzy controls do not override native eligibility or the normal final-term prefix behavior. + +Distance two and fuzzy-prefix are preview features. Qualify the selected fuzzy distance and fuzzy-prefix workload against the actual corpus and concurrency before exposing them to callers. See [full-text configuration](../configuration/options.md#storage) for the surrounding request budgets. + +## Query-time relevance controls + + + +These controls are available in Harper 5.4 with its bundled fulltext wrapper. Earlier 5.4 prerelease builds may not include them. Do not send these keys to builds without query-scoring support: a successful request alone does not prove scoring was applied. A scoring-capable Harper build with an incompatible wrapper returns HTTP 400. + +Use `should` to reward additional text matches while retaining the parent condition's required matches: + +```javascript +const target = { + conditions: [ + { + attribute: 'catalogSearch', + comparator: 'matches_all', + value: 'waterproof boots', + should: [ + { + comparator: 'matches_phrase', + value: 'waterproof boots', + fields: ['name'], + boost: 3, + }, + ], + }, + ], + select: ['id', 'name', '$score'], + limit: 20, +}; +const products = Product.search(target); +``` + +A product that matches the required text remains eligible without the optional phrase. A product that matches only an optional clause is excluded. Optional scores add to the required score; they do not change which records match. + +Each optional descriptor accepts a positive full-text `comparator`, a nonempty string `value`, optional `fields`, `boost`, `fuzzyDistance`, and `fuzzyTranspositions`. + +Optional descriptors use the same [fuzzy controls](#fuzzy-controls-in-v54) as required fuzzy conditions, including their configured distance cap. Optional descriptors remain positive-only. + +`boost` and `should` are reserved query-object keys on all tables in v5.4. Misplaced controls return HTTP 400 even without a full-text index; applications that previously passed ignored custom keys must rename them. An attribute named `boost` or `should` can still be queried through the `attribute` property. + +Each descriptor searches the parent index. An `attribute` property is rejected even if it repeats the parent index name. The list must be nonempty and contain at most `storage.fullText.query.maxQueryClauses` descriptors (default `256`). This is a coarse host precheck: native compilation also counts required clauses, Boolean structure, boosts, analyzed terms and field expansion, so a smaller list can still exceed the compiled-clause budget and return HTTP 400. A positive full-text leaf condition inside an `and` or `or` group can use these controls. The following shapes return HTTP 400: + +- `should` inside an optional descriptor; +- a `not_*` comparator inside `should`; +- `boost` or `should` on the query target, a group object, a negated full-text condition, or an ordinary record condition. + +A positive full-text condition with scoring controls can still combine with negated full-text conditions on the same index and ordinary record filters under `and`. Full-text `or` groups cannot mix full-text and ordinary record conditions. + +Set `boost` on the parent condition to multiply its required text score, or on an optional descriptor to multiply that bonus. Boosts multiply existing configured source-field weights; they do not replace those weights or require an index rebuild. `boost` is a number and defaults to `1` when omitted. Values round to float32 and must then be finite between 0 and 100 inclusive. Invalid values, including negative zero, return HTTP 400. Zero preserves matching while contributing zero score. For a standalone required condition without optional clauses, a positive uniform boost rescales scores without changing relative ranking, apart from floating-point ties. Zero makes those scores tie. + +Every referenced source must be readable, including optional fields used only for ranking. Omitting `fields` searches all sources and requires permission for all of them. The same server query limits and result limits apply to optional clauses. Requested highlights include matching positive required and optional clauses. + +Use the same structured target through HTTP `QUERY`: + +```javascript +const response = await fetch('http://localhost:9926/Product/', { + method: 'QUERY', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(target), +}); +``` + +Include the application's normal authentication credentials. Operations `search_by_conditions` accepts the same `conditions` with its usual `database`, `table` and `get_attributes` properties. These controls add no FIQL syntax. + ## Highlights Highlighting returns source-field fragments and matching character spans. The index must configure highlighting and mark at least one source with `highlight: true`. A query can also search sources without that option; those sources can affect matching and ranking but are omitted from `$highlights`. @@ -208,6 +283,14 @@ For example, `offset: 100` and `limit: 10` skip the first 100 eligible records a Each offset request runs a new query and revisits earlier matches, so deeper offsets increase fetched work and source reads. Prefer narrower filters and a bounded maximum offset over repeatedly raising budgets. Each query can see changed records, permissions or index contents. Pages can therefore repeat or omit records during concurrent changes. There are no continuation tokens, close operations or retained searches across requests. +## Host query capacity + + + +The default native policy allows 64 analyzed terms per text condition and 256 compiled clauses per query. Operators can configure `storage.fullText.query.maxQueryTerms` (1-1,024) and `storage.fullText.query.maxQueryClauses` (1-4,096), within the loaded native runtime's ceilings. Boolean groups, NOT, field expansion, fuzzy branches and prefix expansion also consume clause capacity. Raising fetched-hit or retained-result budgets does not raise these limits. + +The host policy is captured once for the full Harper process, inherited by workers and applied to reader handles, including reopened readers. A full process restart is required; a worker-only restart does not apply changes. Requests cannot raise the policy. Overflow rejects the query rather than executing a truncated one. Query-capacity validation failures return `400` before the response starts. If an operations JSON stream has already started, it instead aborts and may include an error diagnostic; treat an incomplete response as failed and discard its rows. Other depth, byte, phrase/fuzzy construction, queue, response and deadline safeguards continue to apply. Native deadlines are cooperative; they cannot interrupt every construction step immediately. Fuzzy queries still have fixed native limits of 16 eligible terms and 256 construction-work units; distance two costs four times as much construction work per term and field as distance one. Raising term or clause policy does not raise those limits. Raised-policy handles also route non-ASCII text and larger ASCII queries through expensive-search admission, which can increase busy rejections under concurrency. See [storage configuration](../configuration/options.md#storage) and the [native limits](https://github.com/HarperFast/fulltext/blob/v0.9.0/docs/limits.md#query-capacity-policy). + ## REST Use the comparator in the collection query string: @@ -226,7 +309,7 @@ Repeat the index parameter to combine the required positive and negated conditio GET /Product/?catalogSearch=matches=waterproof&catalogSearch=not_matches=leather&limit(20) ``` -REST can return configured highlights by selecting `$highlights`. Use `Table.search()` or `search_by_conditions` for explicit `fields`, condition-level `includeHighlights`, or freshness controls. +REST can return configured highlights by selecting `$highlights`. Use `Table.search()` or `search_by_conditions` for explicit `fields`, condition-level `includeHighlights`, or freshness controls. For v5.4 [query-time relevance controls](#query-time-relevance-controls), use HTTP `QUERY`, `Table.search()` or `search_by_conditions`. ## Operations API @@ -253,6 +336,34 @@ REST can return configured highlights by selecting `$highlights`. Use `Table.sea } ``` +For v5.4 [query-time relevance controls](#query-time-relevance-controls), use the following shape: + +```json +{ + "operation": "search_by_conditions", + "database": "catalog", + "table": "Product", + "limit": 20, + "get_attributes": ["id", "name", "$score"], + "conditions": [ + { + "attribute": "catalogSearch", + "comparator": "matches_all", + "value": "waterproof boots", + "boost": 0.5, + "should": [ + { + "comparator": "matches_phrase", + "value": "waterproof boots", + "fields": ["name"], + "boost": 3 + } + ] + } + ] +} +``` + If an index is `unknown`, `needs-rebuild`, or `rebuilding`, the query returns `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. An index in terminal `unavailable` state returns a generic, non-retryable `503`; inspect readiness and logs instead of retrying it as a rebuild. Other busy paths can also return a generic `503`, so clients should branch on the code rather than the status alone. Unreadable source fields return `403`. Invalid declarations, unsupported match modes, incompatible combinations, and out-of-range options return `400` responses. For paging-budget and native response-envelope failures, see [paging and prefix result windows](#paging-and-prefix-result-windows). diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 657ab1251..b01a799c7 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -317,7 +317,7 @@ Returns records matching one or more conditions. Supports `operator` (`and`/`or` Full-text conditions (Added in: v5.3.0): -Conditions can query a declared full-text index with the `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix` comparators and their `not_` forms. A negated full-text condition requires a non-negated condition on the same full-text index. The Operations API preserves advanced condition options such as source `fields`, `includeHighlights`, `maxIndexLagMilliseconds`, and `waitForIndexMilliseconds`. See [Querying Full-Text Indexes](../full-text-search/querying.md#operations-api). +Conditions can query a declared full-text index with the `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix` comparators and their `not_` forms. A negated full-text condition requires a non-negated condition on the same full-text index. The Operations API preserves advanced condition options such as source `fields`, `includeHighlights`, `maxIndexLagMilliseconds`, and `waitForIndexMilliseconds`. It preserves the [v5.4 fuzzy controls](../full-text-search/querying.md#fuzzy-controls-in-v54), `fuzzyDistance` and `fuzzyTranspositions`, and the v5.4 [`should` and `boost` relevance controls](../full-text-search/querying.md#query-time-relevance-controls). See [Querying Full-Text Indexes](../full-text-search/querying.md#operations-api). --- @@ -1861,7 +1861,7 @@ Detailed documentation: [Configuration Overview](../configuration/overview.md) Updates configuration parameters in `harper-config.yaml`. Most changes require a restart (`restart` or `restart_service`) to take effect. See [Set Configuration](../configuration/operations.md#set-configuration) for full-text restart requirements. -Supports `"replicated": true` to apply the same change to all cluster nodes in one call; per-node outcomes are returned in the response's `replicated` array. Only send cluster-appropriate parameters when replicating — node-local parameters (ports, `node.hostname`, file paths, TLS material, `replication.hostname`/`url`/`routes`) would overwrite every peer's local values. To apply the change cluster-wide, follow with `restart_service` using `"replicated": true` (which restarts nodes one at a time); `restart_service` restarts only worker threads, though, so changes to [`threads.v8Flags`](../configuration/options.md#applying-v8-flags), `storage.fullText.runtime.*`, [`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) or [`storage.rocks.transactionLogWriteIntents`](../database/storage-tuning.md#storagerockstransactionlogwriteintents) take effect only after a `restart` sent to each node in turn (`restart` does not replicate). See [Configuration Operations](../configuration/operations.md#set-configuration) for details. +Supports `"replicated": true` to apply the same change to all cluster nodes in one call; per-node outcomes are returned in the response's `replicated` array. Only send cluster-appropriate parameters when replicating — node-local parameters (ports, `node.hostname`, file paths, TLS material, `replication.hostname`/`url`/`routes`) would overwrite every peer's local values. To apply the change cluster-wide, follow with `restart_service` using `"replicated": true` (which restarts nodes one at a time); `restart_service` restarts only worker threads, though, so changes to [`threads.v8Flags`](../configuration/options.md#applying-v8-flags), `storage.fullText.query.*`, `storage.fullText.runtime.*`, [`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) or [`storage.rocks.transactionLogWriteIntents`](../database/storage-tuning.md#storagerockstransactionlogwriteintents) take effect only after a `restart` sent to each node in turn (`restart` does not replicate). See [Configuration Operations](../configuration/operations.md#set-configuration) for details. ```json { diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index 3da6e188b..6561c04af 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -52,9 +52,9 @@ When a query is executed, conditions are evaluated against the database. Indexed -A condition using a `FullText` field declared with `@fullText` is executed by the native full-text engine and ranked with BM25. Compatible `and` filters narrow the matches. Harper loads current source records and removes stale or deleted candidates before returning them. +A condition using a `FullText` field declared with `@fullText` is executed by the native full-text engine and ranked with BM25, adjusted by any supplied [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls) in v5.4. Compatible `and` filters narrow the matches. Harper loads current source records and removes stale or deleted candidates before returning them. -Keep result pages bounded. Before v5.4, prefix and fuzzy-prefix expressions restrict `offset + limit` to 100. In v5.4, bounded queries can traverse beyond that window within request budgets; an unbounded query can still fail when it exceeds a native window or configured limit. See [paging and prefix result windows](../full-text-search/querying.md#paging-and-prefix-result-windows). Use source-field weights to tune ranking, and enable positions, surface terms, and highlighting only when their query features are needed. See [Full-Text Search](../full-text-search/overview.md). +Keep result pages bounded. Before v5.4, prefix and fuzzy-prefix expressions restrict `offset + limit` to 100. In v5.4, bounded queries can traverse beyond that window within request budgets. An unbounded query can still fail when it exceeds a native window or configured limit. See [paging and prefix result windows](../full-text-search/querying.md#paging-and-prefix-result-windows). Use source-field weights to tune index-level ranking. In v5.4, [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls) adjust scores per request without an index rebuild. Enable positions, surface terms, and highlighting only when their query features are needed. See [Full-Text Search](../full-text-search/overview.md). ### Indexed vs. Non-Indexed Fields @@ -103,7 +103,7 @@ Use a bounded `limit` for large result sets. An unbounded search that exceeds it The fetched-hit budget includes the initial batch replay when Harper switches to native traversal. It counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. A budget failure is a failed query, not a completed short page. If a REST response has already started streaming, a later query error ends that stream instead of changing its HTTP status. JSON-array clients must check for a terminal object shaped like `{ "error": "ErrorName: message" }` and treat the response as failed. For other formats, detect stream or decoding failures; status 200 and count headers alone do not prove completion. See [REST paging](../rest/querying.md#pagination-and-total-count). When there are no further native matches, Harper returns the eligible records found, which may be an empty result. This differs from reaching a budget while more matches remain. -Configure the three request limits and six native process budgets under [`storage.fullText`](../configuration/options.md#storage). See [Set Configuration](../configuration/operations.md#set-configuration) for restart requirements. The execution budget is not a hard response-time guarantee: reader acquisition, query construction and some native operations can take additional time. Freshness waiting and reader acquisition use separate bounds. +Configure the request work limits, query admission policy and native process budgets under [`storage.fullText`](../configuration/options.md#storage). See [Set Configuration](../configuration/operations.md#set-configuration) for restart requirements. The execution budget is not a hard response-time guarantee: reader acquisition, query construction and some native operations can take additional time. Freshness waiting and reader acquisition use separate bounds. ### Native runtime configuration @@ -115,7 +115,7 @@ Replacement workers keep the captured budgets after a configuration change. See Full-text indexes require audit logging. New schema tables must explicitly declare `@table(audit: true)` when adding `@fullText`, even if global audit logging is enabled; an existing audited table can add an index without restating `audit`. Setting `@table(audit: false)` while an index is declared is rejected during schema registration. -Harper v5.4 uses `@harperfast/fulltext` 0.7.0 with Tantivy 0.27.0. Existing 0.5.0 and prerelease 0.6 indexes using current field-based declarations need an automatic local rebuild after this engine upgrade. Follow [upgrading existing indexes](../full-text-search/overview.md#upgrading-existing-indexes) for affected versions, draining traffic, readiness checks, matching changes and current-engine recovery. Earlier beta declarations and indexes have no supported migration path. See [write backpressure](../full-text-search/overview.md#write-backpressure) for local write refusal and retry guidance. +Harper v5.4 uses `@harperfast/fulltext` 0.9.0 with Tantivy 0.27.0. Compatible text-only 0.7.0 and 0.8.0 checkpoints are reused without another rebuild. Existing 0.5.0 and prerelease 0.6 indexes using current field-based declarations need an automatic local rebuild after this engine upgrade. Follow [upgrading existing indexes](../full-text-search/overview.md#upgrading-existing-indexes) for affected versions, draining traffic, readiness checks, matching changes and current-engine recovery. Earlier beta declarations and indexes have no supported migration path. See [write backpressure](../full-text-search/overview.md#write-backpressure) for local write refusal and retry guidance. | Failure | HTTP status | Code and retry guidance | | ------------------------------------------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -136,7 +136,7 @@ Concurrent requests each have their own work budget. These per-request budgets d `INDEX_REBUILDING` and `DERIVED_INDEX_LAGGING` identify retryable readiness or coverage failures. Retry with backoff while the index makes progress. Transaction history retained under [`logging.auditRetention`](../logging/configuration.md#loggingauditretention) (default `3d`) is used for catch-up. Lost history can require a rebuild; retries alone do not repair an index that cannot catch up. Use [`describe_table`](../operations-api/operations.md#describe_table) to inspect `full_text_indexes[].readiness`, including its state and reason. Stop aggressive transaction-log cleanup and correct the reported configuration or storage problem before restarting the same Harper version to retry activation. Keep full-text traffic drained until readiness and required coverage recover; do not edit native files. Budget and native-busy failures do not carry those readiness codes. A native busy response also returns 503; reduce concurrency or retry with backoff. For an execution-budget failure, narrow the query, reduce the offset or adjust the work budget instead of repeatedly submitting the same expensive query. -Native page, response-byte and highlight limits still apply. The fulltext 0.7.0 library limits each analyzed query leaf to 64 terms and the compiled query to 256 clauses. Native invalid-query validation returns 400 before response headers; the streaming-error guidance above still applies. Raising Harper's work budgets does not raise those native limits. See the [published native limits](https://github.com/HarperFast/fulltext/blob/v0.7.0/docs/limits.md) for page, byte and highlight bounds. +Native page, response-byte and highlight limits still apply. The fulltext 0.9.0 defaults are 64 analyzed terms per leaf and 256 compiled clauses per query. The separate `storage.fullText.query` policy configures those ceilings and the maximum fuzzy distance; see [storage configuration](../configuration/options.md#storage). Native invalid-query validation returns 400 before response headers; the streaming-error guidance above still applies. Changing request work budgets does not change the query policy or the native page, byte and highlight limits. See the [published native limits](https://github.com/HarperFast/fulltext/blob/v0.9.0/docs/limits.md) for page, byte and highlight bounds. ## Relationships and Joins diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 9e115f41d..fcecd6151 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1037,11 +1037,15 @@ Array of condition objects to filter records. Each condition: | `conditions` | Nested conditions array | | `operator` | `and` (default) or `or` for the nested `conditions` | | `fields` | Full-text source fields to search; every name must belong to the selected full-text index | +| `boost` | Optional full-text score multiplier (`number`, default `1`); see v5.4 [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls) | +| `should` | Optional nonempty array of additional scoring clauses; omitted by default; see v5.4 [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls) | +| `fuzzyDistance` | Integer edit distance for a fuzzy condition, including negated forms; see [v5.4 fuzzy controls](../full-text-search/querying.md#fuzzy-controls-in-v54) and the server cap. | +| `fuzzyTranspositions` | Boolean controlling whether adjacent transpositions count as one edit on a fuzzy condition; see [v5.4 fuzzy controls](../full-text-search/querying.md#fuzzy-controls-in-v54). | | `includeHighlights` | Include configured full-text highlights | | `maxIndexLagMilliseconds` | Maximum accepted full-text index lag; see [freshness controls](../full-text-search/querying.md#freshness-controls) | | `waitForIndexMilliseconds` | Time to wait for acceptable full-text coverage; see [freshness controls](../full-text-search/querying.md#freshness-controls) | -The final four properties apply to full-text conditions (v5.3.0). See [Querying Full-Text Indexes](../full-text-search/querying.md) for their constraints, error behavior, and permission requirements for direct server-side searches. +The `fields`, `includeHighlights`, `maxIndexLagMilliseconds` and `waitForIndexMilliseconds` properties apply to full-text conditions (v5.3.0). The `fuzzyDistance` and `fuzzyTranspositions` properties apply to fuzzy conditions in v5.4. The `boost` and `should` controls are available in v5.4; their constraints are described in [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls). See [Querying Full-Text Indexes](../full-text-search/querying.md) for their constraints, error behavior, and permission requirements for direct server-side searches. Full-text conditions use `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, or `matches_fuzzy_prefix` against a declared full-text index. Each comparator also has a `not_` form: `not_matches`, `not_matches_all`, `not_matches_phrase`, `not_matches_prefix`, `not_matches_fuzzy`, and `not_matches_fuzzy_prefix`. A negated condition requires a positive full-text condition on the same index. See [Full-text query execution](./query-optimization.md#full-text-search-work-and-paging) for paging costs and work limits. @@ -1100,7 +1104,7 @@ Special properties: Full-text queries add two special properties: -- `$score`: When a query uses a full-text index, returns its BM25 relevance score. +- `$score`: When a query uses a full-text index, returns its BM25 relevance score. In v5.4, supplied [query-time boosts and optional clauses](../full-text-search/querying.md#query-time-relevance-controls) contribute to this score. - `$highlights`: Enables and returns full-text fragments and matching spans when the index has highlighting configured. See [Highlights](../full-text-search/querying.md#highlights). #### Selecting related records diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 72faec949..3d3e6dc79 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -96,7 +96,7 @@ REST's `limit(start,end)` uses an exclusive end. In v5.4, this prefix query skip GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&limit(100,110) ``` -Use `Table.search()` or `search_by_conditions` when conditions need source-field selection, condition-level highlighting, or freshness controls. See [Querying Full-Text Indexes](../full-text-search/querying.md) for ranking, highlights, query combinations, and freshness controls. +Use `Table.search()` or `search_by_conditions` when conditions need source-field selection, condition-level highlighting, or freshness controls. For v5.4 [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls), use HTTP `QUERY`, `Table.search()` or `search_by_conditions`. See [Querying Full-Text Indexes](../full-text-search/querying.md) for ranking, highlights, query combinations, and freshness controls. The REST URL syntax searches every source field in the index, so the caller must be allowed to read every source field. Before v5.4, prefix and fuzzy-prefix queries restrict `offset + limit` to 100. In v5.4, bounded full-text and autocomplete queries can pass that native window within request budgets; source-version checks and filters run before records count toward the offset and limit. Each page is a new query and can see concurrent changes. See [paging and prefix result windows](../full-text-search/querying.md#paging-and-prefix-result-windows). Full-text queries cannot provide an exact total count. On a REST interface with `exactCount: true`, a `Prefer: count=exact` request returns an unavailable total (`Content-Range: items /*`). Without that opt-in, Harper applies `count=estimated` instead and can return an estimate. diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 31f4bdf00..8ba529b70 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -14,6 +14,12 @@ Full-text indexes using fulltext 0.5.0 with current field-based declarations, or Harper 5.4 marks failed transactions as aborted in the RocksDB transaction log by default ([`storage.rocks.transactionLogAbortMarkers`](/reference/v5/database/storage-tuning#storagerockstransactionlogabortmarkers); see [Failed Transactions Are Marked Aborted](#failed-transactions-are-marked-aborted-in-the-transaction-log)). Harper 5.3.4 and earlier cannot read those markers and would treat the failed transactions as committed. Do not downgrade a node to them, or open its database files or backups with them, after it has run 5.4 with the setting on. Harper 5.3.5 and later read the markers. Setting it to `false` later stops new markers but does not remove ones already written. +## Full-Text Query Boosts and Optional Scoring + +Full-text conditions now accept `boost` to scale a required text score and `should` to add optional scoring clauses without expanding the matching records. Use these controls through `Table.search()`, HTTP `QUERY`, or `search_by_conditions`; select `$score` to return the adjusted relevance score. Optional clauses search the same index and require read access to their source fields. No index rebuild is needed for query-time scoring. See [Query-Time Relevance Controls](/reference/v5/full-text-search/querying#query-time-relevance-controls). + +**Upgrade impact:** `boost` and `should` are now reserved query-object keys on all tables. Misplaced controls return HTTP 400 instead of being ignored; rename custom keys that previously used these names. Attributes named `boost` or `should` remain queryable through the `attribute` property. + ## Nested Relationship Constraints [Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279) and [Fix wrong rows, unfiltered selects and ignored sorts in relationship queries](https://github.com/HarperFast/harper/pull/3775) correct REST and programmatic relationship queries that accepted nested constraints, non-leading relationship conditions, or top-level relationship sorts without applying them completely. Filters, ordering, offset, and limit inside `select` apply per parent in collection and single-record reads, and parents remain present when the nested selection has no matches. Outer relationship filters — whether they lead the query or follow another condition — constrain the selected children as well as the parents, so an identical filter does not need to be repeated inside the nested selection. When a top-level sort across a to-many relationship (such as `sort(products.name)`) runs in memory — when the target attribute is not `@indexed`, a more selective condition on another attribute leads the query, or `enforceExecutionOrder: true` is set — Harper now orders matched parent records by the list of each parent's related values in relationship order, compared element by element, instead of ignoring the sort; when an `@indexed` target attribute leads the query, Harper continues to order via the related table's index join. @@ -182,13 +188,15 @@ Harper Pro's CPU profiler used to sample every worker continuously, so each work ### Tantivy 0.27 and Existing Index Rebuilds -Harper now uses `@harperfast/fulltext` 0.7.0 with Tantivy 0.27.0. Harper 5.3.1–5.3.4 ship fulltext 0.5.0. Existing 0.5.0 indexes using current field-based declarations, and prerelease 0.6 indexes, rebuild automatically from authoritative records. Harper 5.3.0 ships fulltext 0.3.0 and is outside this qualified procedure; validate that migration separately. Earlier beta declarations and indexes have no supported migration path. Plan for local full-text unavailability, disk headroom and possible retryable write failures while replay catches up. Stop old processes before starting the new version against the same storage; direct downgrade to fulltext 0.5.x is unsupported. +Harper now uses `@harperfast/fulltext` 0.9.0 with Tantivy 0.27.0. Harper 5.3.1–5.3.4 ship fulltext 0.5.0. Existing 0.5.0 indexes using current field-based declarations, and prerelease 0.6 indexes, rebuild automatically from authoritative records. Harper 5.3.0 ships fulltext 0.3.0 and is outside this qualified procedure; validate that migration separately. Earlier beta declarations and indexes have no supported migration path. Plan for local full-text unavailability, disk headroom and possible retryable write failures while replay catches up. Stop old processes before starting the new version against the same storage; direct downgrade to fulltext 0.5.x is unsupported. + +For the 0.5/0.6 rebuild path, English stemming changes despite the unchanged `english@2` analyzer name: `added` now stems to `add` rather than `ad`. Matching and ranking can change. Query modes and schema options are unchanged; benchmark results from the wrapper do not establish application performance parity. Review representative searches and see [upgrading existing indexes](/reference/v5/full-text-search/overview#upgrading-existing-indexes) before restoring traffic. -English stemming changes despite the unchanged `english@2` analyzer name: `added` now stems to `add` rather than `ad`. Matching and ranking can change. Query modes and schema options are unchanged; benchmark results from the wrapper do not establish application performance parity. Review representative searches and see [upgrading existing indexes](/reference/v5/full-text-search/overview#upgrading-existing-indexes) before restoring traffic. +Harper 5.4.0-alpha.1/alpha.2 indexes using fulltext 0.7.0, and current Harper indexes written by development builds with 0.8.0, can reuse compatible checkpoints with 0.9.0. See [upgrading from fulltext 0.7.0 or 0.8.0](/reference/v5/full-text-search/overview#upgrading-from-fulltext-070-or-080) for process coordination and the unqualified rollback caveat. ### Paging and Search Work Limits -Full-text and autocomplete queries use the existing `limit` and `offset` paging options. Harper reads additional native batches when needed after filtering, so the initial native result window no longer silently cuts off eligible results. The new `storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds` settings bound fetched hits, retained results and execution time. Requests that exceed those budgets return an error. The `storage.fullText.runtime` section also provides process-wide native-capacity budgets; changes to those budgets require a full Harper process restart. Separate page requests run independently and can see changes to records or index contents. +Full-text and autocomplete queries use the existing `limit` and `offset` paging options. Harper reads additional native batches when needed after filtering, so the initial native result window no longer silently cuts off eligible results. The new `storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds` settings bound fetched hits, retained results and execution time. Requests that exceed those budgets return an error. The `storage.fullText.runtime` section also provides process-wide native-capacity budgets; changes to those budgets require a full Harper process restart. The `storage.fullText.query` policy configures analyzed terms per leaf, compiled clause count and maximum fuzzy distance; it also requires a full Harper process restart. Defaults remain 64 terms, 256 clauses and distance one. Harper 5.4.0-alpha.1/alpha.2 reject an explicit query-policy block. Separate page requests run independently and can see changes to records or index contents. See [Full-text query execution](/reference/v5/resources/query-optimization#full-text-search-work-and-paging) and [storage configuration](/reference/v5/configuration/options#storage) for limits, restart requirements and recovery guidance.