[kylebernhardy]

⊙ Problem

Applications can require text matches or combine alternatives, but cannot reward an extra match without changing eligibility. Index field weights also cannot express a boost for one request.

Refs Extend typed queries with optional scoring clauses, boosts, regex, and similar-document search.

🤔 Is this the right first slice of query relevance controls?

Deliver boosts and required-plus-optional scoring first. These compose with existing text modes without rebuilding indexes. Regex, similar-document retrieval and a separate per-request field-weight map remain outside this slice; the linked issue stays open.

💡 Solution

Draft / release gate: scoring and merged telemetry now share native ABI 14, query API 8 and telemetry API 1. Root, Cargo and all four native packages are synchronized at unpublished 0.9.0. Publication and Harper dependency adoption remain separate release work.

• boost multiplies a subtree's score while preserving its matches.
• optional requires one expression and adds scores from matching optional expressions. Optional-only matches never enter the result.
• Boost a field-restricted clause to adjust its contribution for one request; configured index weights still apply.

{
  query: {
    operator: 'optional',
    required: { text: 'waterproof', fields: ['title', 'description'] },
    optional: [{
      operator: 'boost',
      boost: 3,
      clause: { text: 'trail running', mode: 'phrase', fields: ['title'] }
    }]
  }
}

⚖️ Alternatives

• Existing OR queries: admit optional-only matches, changing the required result set.
• Index field weights: apply across requests and do not express a boosted phrase or arbitrary subtree.
• A custom scorer: would duplicate Tantivy's Boolean scoring and increase the risk of search/paging score drift. Native query composition keeps those semantics together.

🔧 Changes

Expression flow and compatibility

From request to result

The request keeps its required membership while optional matches contribute to ranking.

flowchart LR
  API[Typed expression] --> TS[Normalize and bound]
  TS --> Wire[Packed tags 4 and 5]
  Wire --> Rust[Validate and classify]
  Rust --> Native[Tantivy query tree]
  Native --> Results[Ranked hits and counts]

Loading

ts/native.ts exposes both expression variants, normalizes boosts once to float32, rejects conflicting shapes, and carries the normalized expression through search and continuation identity. Optional normalization captures an array length once for validation, budget charging and allocation, preventing Proxy-backed lists from expanding uncharged work. ts/codec.ts appends scoring tags while retaining tags 0–3 for existing leaves and Boolean groups.

src/protocol.rs decodes the new variants and accounts for their structure and admission class. src/request_validation.rs also validates direct Rust requests. Boosts must round to finite float32 values in 0..=100, excluding negative zero; optional lists must be nonempty. The shared native Optional-child check gives the Optional-specific diagnostic in typed validation, wire admission and decoding, while retaining Boolean diagnostics and error codes. Existing depth, byte, clause and deadline bounds apply to every child.

src/lib.rs reports ABI 14, query API 8 and telemetry API 1. ts/load-addon.ts rejects mismatched package versions and incomplete capability sets before use. The merged Expose fulltext operation telemetry and latency histograms is preserved; its ABI 14 was not published, so the combined artifact retains that ABI. package.json, package-lock.json, Cargo.toml and Cargo.lock synchronize root and native packages at 0.9.0. Registry versions for all five npm packages were checked before selecting this unused version. Persisted index identity is unchanged. docs/metadata-migration.md distinguishes published 0.8.0 from this pending combined release.

Native scoring and preserved invariants

⚠️ Look hardest: src/engine.rs builds optional scoring as one native Must branch plus native Should bonuses. The required branch alone decides eligibility for filters, candidate restrictions, counts and score/ID pagination. Only the required branch supplies a positive admission anchor; optional branches still contribute their expensive-work classification.

Only root boosts over Any/All leaves multiply selected field weights before building term queries, preserving the query topology used by native pruning. Recursive boosts retain native BoostQuery multiplication order: an inner large field weight must not overflow before an enclosing zero boost reaches it. Other root modes and complex subtrees also use native BoostQuery composition. At the root, this wrapper loses specialized Boolean pruning. Optional scoring over required All also adds work over the required-only intersection. Nested boosts multiply without a separate cumulative cap. Nonfinite result scores fail explicitly. Zero boosts preserve membership.

Compiler-owned stable routing travels in BuiltQuery.stable_scores. Term, Phrase, Fuzzy, prefix and Boolean builders determine it from the query they construct, replacing the duplicate AST-mode walk and division heuristic. Completed prefix/fuzzy-prefix and singleton exact-prefix paths can lower to the same faulty native conjunction; empty per-segment fields can also collapse unions into bare term scorers. Non-unit native Boost wrappers use generic scoring and clear the stable-routing flag; identity boosts inherit it, while candidate/filter wrappers keep their existing behavior. A four-term native canary compares raw TopDocs with scorer iteration. Stable collection scans matches and resolves IDs beyond the visible page. Both ordinary stable passes now reuse the shared deadline collector in src/catalog.rs, including the boundary-tie rerun. Its deletion-aware scorer loop checks the shared deadline every 1,024 visited documents and rejects the whole request on expiry. The engine error mapping preserves E_TIMEOUT, with no partial hits or cursor escape. These cooperative checks do not preempt a slow individual scorer operation. src/DESIGN.md records remaining cost and native-bound limits; this is not a universal pruning repair or strict latency guarantee.

Callers must authorize optional fields as well as required fields. An omitted field list searches every configured text field. traceMatches() remains a simple-clause tracing API; boosts do not change spans.

Evidence and caller guidance

test/query-scoring.test.mjs covers scoring, eligibility, restrictions, filtered count fallback, paging, validation and wire tags. Native engine/protocol/request tests cover typed Rust paths and scorer topology. New public compiled-query regressions compare bounded search with continuation scores at a distinct cutoff, including multiple segments, empty secondary fields, completed-prefix constructors and real candidate/filter restrictions. All and AND restriction controls each admit 150 of 300 documents. Two-term boosted All controls retain the native pruning path. The early-abort regression fails before the deadline fix by visiting all 4,096 matches; the fixed path aborts within 1,024 advances. Its deadline-enabled deletion control also preserves exact score bits and excludes tombstoned documents. Independent required-plus-two-bonus arithmetic rejects max-only scoring; a 127-record, varied-addend paging fixture catches real score-bit drift when stable routing is removed. Native nested-boost regressions fail before the fix for fractional weights and a large finite field weight under an outer zero boost. test/native.test.mjs and test/load-addon-resolution.test.mjs expect the new capabilities.

benchmarks/query-scoring.mjs adds single-term and two/three/four-term All baseline, root-lowered and native-wrapped controls. Paired correctness and score-bit checks precede timing; both default tied and varied modes complete 40 rows over four committed segments. The opt-in --routing-shapes mode adds 20 rows per corpus for OR, singleton prefix, short Fuzzy, one-token two-field Phrase and three-term two-field All. Complete traversal reads respect prefix limits; timed requests use limit 20. package.json adds the benchmark command. The benchmark record distinguishes route-specific results from throughput, catalog-scale memory and general pruning guarantees.

README.md and examples/query-modes.mjs show the caller contract; the README limits undefined-as-absent behavior to the listed scoring/structural properties, excluding present text: undefined. docs/limits.md documents Rust enum migration and removal of Eq. DESIGN.md indexes the invariants in src/DESIGN.md.

✅ Verification

Executed qualification results and environment limits

The reconciliation at ddd873f passed 196 Rust tests (9 ignored), 536 main Node tests, 28 metadata tests and 12 worker tests, plus TypeScript, Clippy, formatting and diff checks. Clean npm ci --ignore-scripts succeeded with the unpublished 0.9.0 pins. The optimized suite passed 12 tests. Generated native declarations match the committed source.

The combined loader regressions reject telemetry-only query-7 and scoring-only telemetry-missing artifacts with matching package/ABI values; each fails against its incomplete parent loader and passes after reconciliation. The same-handle regression verifies required eligibility, optional boost ordering, ordinary-lane classification and exactly-once queue-expiry accounting. The packed-consumer assertions verify ABI 14, query API 8 and telemetry API 1.

npm run check reached the packed-install phase but npm 12 injected EALLOWSCRIPTS. Running the repository's canonical release runner directly outside npm's inherited script environment passed 2 packed-consumer/runtime tests and 6 production-worker tests. User npm configuration was unchanged. No full npm run check success is claimed for that environment.

The deadline regression proves early cancellation instead of a post-scan error: it advances no more than 1,024 of 4,096 matching documents. A real deletion with merge disabled verifies live-document filtering, exact total 4,095 and exact score bits against the unwrapped stable collector. Existing four-term and public bounded-page regressions cover demonstrated compiler constructors, multiple segments, candidate restrictions and metadata filtering; they do not qualify every native rounding bound.

Historical pre-telemetry evidence follows. The previously qualified optimized addon SHA256 is 652d1aaafd045641c1880a6c08e1a52099df5aa599de91fabf64f1ec6191cd6b. Those root and installed ABI 14/query API 8 artifacts matched; these measurements do not qualify the newly combined telemetry build. Canonical npm run test:node reached the packed phase but npm 12 injected EALLOWSCRIPTS; the direct canonical release runner passed after removing inherited allow-scripts variables. User npm configuration was unchanged.

On this artifact, the unchanged default benchmark completed 40 rows per tied/varied mode, and opt-in routing shapes completed 20 rows per mode, each observing four committed segments. Varied exact-total-off p50 baseline/root/native-wrapped was All2 0.077/0.078/0.102 ms, All3 0.163/0.162/0.127 ms and All4 0.177/0.177/0.143 ms. Routing baseline/native-wrapper was OR 0.152/0.110 ms, prefix 0.159/0.129 ms, short Fuzzy 0.156/0.112 ms, Phrase over two fields 0.156/0.114 ms and All3 over two fields 0.251/0.227 ms. Membership, totals, scaled scores and distinct-boundary checks passed. These warm total-request measurements do not isolate pruning or deadline overhead and establish no general speedup or catalog-scale bound.

Harper's final paired deadline-aware 652d artifact passed 113 focused and 20 deployed tests, with the actual resolved addon hash verified. Local evidence: query-relevance-harper-deadline-final-focused.log and query-relevance-harper-deadline-final-deployed.log. Hosted scoring CI remains gated on wrapper release and the Harper dependency pin/lock update.

Persisted-index interoperability previously passed with real addons: ABI 13 seeded a document, ABI 14 reopened and added another, and ABI 13 reopened and read both. No persisted identity change or index rebuild is introduced.

README correction 5269d20 passes all 3 dependency/documentation tests and formatting. Hosted CI is rerunning.

Review response and stopping point

Fresh Claude and Composer review covers the merge reconciliation. Neither reviewer found a blocker or major issue. The final response clarifies migration history and nested-boost wording, names the queue-expiry test accurately, and checks telemetry directly from the packed native addon. The targeted test and rebuilt production packed/worker suites passed again; a focused Claude/Composer follow-up verified all eight fixes. Its final two README nits were then corrected, and the revised runtime example was executed; that four-line documentation response was not rereviewed.

The package-coordinate collision is resolved in source. Publishing compatible 0.9.0 artifacts and updating Harper's dependency remain release gates. Extend typed queries with optional scoring clauses, boosts, regex, and similar-document search retains the performance and compatibility deferrals; historical review records below describe earlier heads.

Coordinated guide: Document full-text boosts and optional scoring, local head c7e9a5b803aece59e702b20811757c1c514b50fa. Harper integration: Add full-text boosts and optional scoring conditions, local head a53592d9fa62df12d5a1366c5e948e188884bb78.

Comment generated by kAIle (GPT-6)

Related PRs: #81 overlaps (merged and reconciled as ABI14/query8/telemetry1: Expose fulltext operation telemetry and latency histograms), #79 overlaps (updates published-version guidance from Prepare fulltext 0.8.0 for release), #78 overlaps (extends request validation and admission from Expose fuzzy matching controls and align Rust request validation), #77 overlaps (extends structural budget paths from Make analyzed query capacity configurable per handle)

Complexity: complicated

Historical review context

The initial Grok findings below are superseded by the final response review at 2c3edf9: shape validation, native multiplication order, parity and benchmark evidence are addressed. Composite pruning remains an accepted measured tradeoff tracked in Extend typed queries with optional scoring clauses, boosts, regex, and similar-document search. The historical managed block is retained unchanged.

Late Grok findings

4 surviving (highest major) — Grok reviewed 3db8a90, ruled against 358e1bb

• major — ts/native.ts:1459 — boost and optional expressions silently discard sibling clause/clauses properties, so intended constraints never reach native search
• major — src/engine.rs:1512 — boosts around composite expressions still hide BooleanWeight pruning behind BoostQuery, making low-limit searches scan every match
• minor — README.md:574 — the upgrade guide advertises query API 7/native ABI 13 although this build requires 8/14, leading operators toward incompatible artifacts
• nit — ts/codec.ts:1098 — zero-accepting float validation retains an incorrect error message, alongside object stringification and compressed design prose

Not standing: 6 resolved. Grok ran 17m (17m from launch to report); the fold adjudication (cursor-grok@3db8a904fce8-8710a551, codex) took 2m.

Deferred findings

• 3084f89c82b555f2 none performance — src/engine.rs — Composite boosts bypass Tantivy specialized TopDocs pruning — filed Extend typed queries with optional scoring clauses, boosts, regex, and similar-document search #55
• e313f09f9410c6e0 none correctness — README.md — Boolean groups reject leaf-only fields and mode properties — filed Extend typed queries with optional scoring clauses, boosts, regex, and similar-document search #55
• e33b81c1db8c0e17 none performance — benchmarks/query-scoring.mjs — Default benchmark corpus retains score-boundary ties — filed Extend typed queries with optional scoring clauses, boosts, regex, and similar-document search #55
• e5b36a05fc329c13 none correctness — ts/native.ts — Boolean groups reject formerly ignored leaf-only properties — filed Extend typed queries with optional scoring clauses, boosts, regex, and similar-document search #55

Late Grok findings

5 surviving (highest blocker) — Grok reviewed 5d809c9

• blocker — package.json:75 — a clean install resolves published ABI-13 native packages for an ABI-14 facade and fails addon loading
• major — benchmarks/query-scoring.mjs:25 — the hot-path root-leaf boost specialization lacks a benchmark that exercises its specialized-pruning purpose
• minor — test/query-scoring.test.mjs:344 — root-boosted single-field All lacks public search-versus-traversal score parity coverage
• nit — src/protocol.rs:1311 — empty Optional frames report a Boolean-specific diagnostic through all native entry paths
• nit — README.md:510 — the undefined-property rule overstates behavior because present text: undefined still rejects

Not standing: none. Grok ran 19m (19m from launch to report); the fold adjudication (cursor-grok@5d809c96ad57-b4c84b62, codex) took 2m.

_{Review-Coverage: authored=codex; ran=cursor-composer,claude,cursor-grok; late=cursor-grok:adjudicated; adjudicated=domain,domain(claude); blocked=gemini(auth); declined=muse; rounds=11; full=6 @ 5269d20}

_{Review-Attention: deep ~25m (critical: codec.ts; decisions: stable-scan-for-term-sums, ordinary-lane-admission-unchanged, root-only-boost-lowering, keep-abi-14-for-combined-release, do-less-alternative) @ 5269d20}