/search optimization

[thehabes]

Summary

Resolves #309.

Resolves #313.

Part of #306. #308 is the only sub-issue still open.

Stacked on #312. Merge that first.

/search and /search/phrase now run one Atlas Search pipeline against one index, and paging happens in the database. They carry rel="next" the same way /query does after #312. /query and every search endpoint stop serving objects whose _id is not a string. A searchText that isn't a non-empty string, or options that aren't a JSON object, are refused with a 400 before they reach Atlas.

What changed

	Before	After
Indexes	presi3AnnotationText and presi2AnnotationText, in two aggregate() calls run through Promise.all	annotationText, which covers both vocabularies, in one call
Ranking	Two score lists merged and re-sorted in Node	Atlas's own descending score order, with each object scored once
Deduplication	mergeSearchResults() keyed on _id, collapsing every embedded-object _id onto "[object Object]"	Not needed. Atlas returns each object once
Paging	Each index fetched limit + skip records, then Node merged and sliced them	$skip and $limit in the pipeline, plus one extra record to learn whether another page exists
rel="next"	Absent	Present while another page exists, after the JSON-LD context link
Non-string _id	Served by /query. /search served the ObjectId ones, and by accident kept only one embedded-object _id per response	Excluded by both before the page is measured, so pages fill to limit
A failing index query	Swallowed by .catch(() => []) into an empty half of the results	Reaches next(err)
Invalid searchText or options	Passed to Atlas. Anything Atlas rejected became 200 []	400 before the database is called. See Input validation
Client options	Spread after query and path, so they could replace the search text or the fields searched	Spread first, so they tune the operator but can't replace either

Latency

{"searchText":"a"} at limit=100 on the dev cluster, median of three runs, measured 2026-09-18. The old two-index fan-out was replayed next to the new pipeline in the same script:

skip	Before	After
0	404ms	223ms
1000	913ms	432ms
5000	2122ms	303ms
20000	1981ms	301ms
50000	2558ms	305ms

a had 1218 servable matches on dev when this was measured, so the new pipeline's pages are empty from skip=5000 on. The old code still returned records at skip=5000 because it served the legacy objects. Atlas still walks matches up to skip, but Node no longer receives, holds, or sorts them, so an empty page past the end costs about as much as a page near the front.

The pipeline

searchPipelineFor() is shared by every search operator:

1. $search
2. $match on a string _id
3. $addFields for __rerum.score
4. $skip
5. $limit: limit + 1

serveSearchPage() trims the extra record. It sets the JSON-LD headers before calling setNextPageLink(), because configureLDHeadersFor() replaces Link and setNextPageLink() appends to it. The unmounted searchFuzzily, searchWildly and searchAlikes use the same two helpers, so they will page the same way if they are ever mounted.

Two should clauses carried over from the old IIIF 3.0 query could never match anything, so they are removed. One was embeddedDocument on items.annotations.items, and the other was embeddedDocument on annotations. An embeddedDocument clause scoped to an outer embeddedDocuments path can't see the fields of the one nested inside it, and it matches nothing without raising an error. The Manifest text the first clause was meant for is already reached through the items clause. With both removed, all 16 sampled responses on dev stayed byte-identical, scores included.

Input validation

Without the .catch(() => []), an Atlas rejection now reaches the client. Atlas reports a bad operator option as MongoServerError code 8 (UnknownError), which can't be told apart from a server fault. So the shape of the request is checked before the pipeline runs:

• searchText must be a non-empty string.
• options, when sent, must be a JSON object. readSearchOptions() refuses a string, an array, a number or a boolean. null counts as not sent. The unmounted searchFuzzily and searchWildly use the same check.

Each of these is a 400, and neither reaches the database.

Anything else Atlas rejects is a 500 that carries Atlas's message. That is intended. It includes:

• A bad option value, such as {"slop":"x"} or {"fuzzy":{"maxEdits":9}}.
• A /search searchText of more than 64 distinct words. The one query searches 16 paths, and 16 × 65 clauses exceed Atlas's maxClauseCount of 1024. The old IIIF 3.0 query searched 12 paths and the IIIF 2.1 query 8, so up to 85 words were searched in full. From 86 to 128 words the IIIF 3.0 half failed silently. Above 128 the search answered 200 []. /search/phrase isn't affected.

/query

controllers/crud.js wraps the client's filter as { $and: [props, { _id: { $type: "string" } }] }. It doesn't merge into it, so a client's own _id condition still applies. The winning plan is still an IXSCAN on _id, with the same keysExamined with and without the clause.

Documentation

public/API.html:

• Text Search and Phrase Search document rel="next", with a paged-search example that follows it.
• Result ordering drops the "Descending score is an ordering, not a tiling guarantee" alert, and replaces the two-index paragraph with the one-index statement. It makes no tiling promise for search, because there is no $sort tiebreaker. See Out of scope.
• Text Search now describes how matching actually works. A multi-word search matches any of the words, not all of them. Words are not stemmed, and stop words are not removed. It used to claim AND logic, stemming and stop-word removal. The JSDoc in controllers/search.js was corrected to match.
• Phrase Search's slop examples are corrected. Reversing two adjacent words costs a slop of 2, so "manuscript medieval" matches at the default and "manuscript from medieval times" does not. The JSDoc examples were corrected to match.
• Both paged examples warn when Pagination-Limit is lower than the requested lim.

openapi/contracts/core-provider.openapi.yaml: /api/search and /api/search/phrase declare the NextPageLink header. All three paths state their result ordering and the non-string _id exclusion. The exclusion is documented there, not in API.html.

Tests

routes/__tests__/search.test.js replaces the two-index dedup and merge tests with tests of the one pipeline. They cover:

• One aggregation against annotationText, searching the IIIF 3.0 and IIIF 2.1 fields.
• $skip and $limit: limit + 1 in the pipeline, and the string _id $match placed before $limit.
• The 400s, none of which reach the database.
• Options that can't replace searchText or the paths.
• rel="next" walks that serve every record once, including past the skip maximum.

routes/__tests__/query.test.js checks that the $and wrapper keeps a client's own _id condition. npm test passes 274 of 274.

Before deploying

• annotationText must exist and be queryable on the target cluster. A missing Atlas Search index doesn't raise an error. It matches nothing, so every search would answer 200 [] and leave nothing in the logs. Dev has the index. Production has not been confirmed.
• Search results on dev shrink sharply. 355,325 of the 463,284 objects on dev (76.7%) come from the RERUM v0 import and have a non-string _id. After this change, manuscript returns 0 records instead of 50. line goes from 4007 matches to 272 servable ones. Read-only probes of store.rerum.io for line, text and lorem returned no records with a legacy @id, so production looks unaffected.
• presi3AnnotationText and presi2AnnotationText are no longer queried, and can be dropped once this is live.

Out of scope

• Atlas Search index definitions are not in the repository #308: the annotationText definition still lives only in Atlas. When it gets checked in, its top-level items.annotations.items mapping can be dropped, because no query can reach it.
• The pagination contract is undocumented, misdocumented, and absent from the OpenAPI contract #305: the OpenAPI contract still has no limit or skip parameters and no Pagination-* headers.
• Score ties at a page boundary: there is no $sort tiebreaker, because one would give up the latency win. On dev, following rel="next" for a at page sizes 7, 100 and 500 returned the same 1218 records in the same order each time, with no repeats, across tie groups of up to 237 equal scores.
• Add cursor-based continuation to /query so paging depth is unbounded and cost is flat #303: cursor-based paging stays deferred.

[cubap]

Review: /search optimization (resolves #309, #313)

I reviewed the full diff against main (with #312 merged in), ran the complete test suite (277/277 pass), and verified the acceptance criteria from #309 and #313.

All criteria verified:

• Pagination now happens in the Atlas pipeline ($skip/$limit), not application memory; latency is flat with skip (dev measurements in the PR body confirm).
• Non-string _id objects are excluded by the $match stage for /query, /search, /search/phrase, and the unmounted variants.
• Descending score order, rel="next" present while more results and absent on the final page, over-fetched record trimmed, context link preserved, legacy /getByProperties.action alias emits correct links.
• Input validation (searchText non-empty string, options a JSON object) returns 400 before any DB call.
• Merge with main is clean; 405 Allow: POST headers are correct.

One spec gap found and fixed (commit 03f19ca):
#313 explicitly requires public/API.html to document that non-string _id objects are not served. The PR body said it was documented in OpenAPI only. I added it to the Custom Query alert and the Text Search / Phrase Search behavior lists.

Minor note (cosmetic): the PR body says the annotations embeddedDocument clause was "removed", but the code rewrote it to annotations.items. Per Atlas docs the operator path is relative to the embedded document, so the two are equivalent ? code is correct, description is just imprecise.

No blocking issues.

[cubap]

Approving after review: all #309/#313 acceptance criteria verified, full test suite passes (277/277), API.html doc gap fixed in 03f19ca.