perf(search): the coverage step of every message search, and word-end patterns

performance/search/PROCESS.md

@@ -389,3 +389,10 @@ Chat messages already keep a trigram index of their text, so cli-core now answer
 the word list. Timed through the whole search, it is no longer refused and takes 0.5-1.0 s at 1M messages; about
 0.5 s of every search at that size is the coverage step, not the pattern. Mail, transcripts and file text keep the
 word list ([infix-chat-text](infix-chat-text/README.md), 2026-11-11).
+
+## The coverage step every search paid
+
+That coverage step counted and dated every message of every chat on each search. Read off the trigger-kept count and
+each chat's newest message on its time index, it takes 0.2 ms instead of 0.45 s at 1M messages, and `договор*` falls
+from 478 ms to 31 ms. Word-end patterns (`*ени`) are now checked in the answer's order up to the page, so a frequent
+one answers with a page instead of a refusal ([coverage-step](coverage-step/README.md), 2026-11-11).

performance/search/README.md

@@ -59,6 +59,10 @@ at a million distinct words), and why it stays refused there with a hint to writ
 of message text, `*договор*` is no longer refused and takes 0.5–1.0 s at 1M messages, half of it a coverage step that
 every search pays.
 
+[The coverage step](coverage-step/README.md): every message search counted and dated every message of every chat
+for its coverage, 0.45 s at 1M messages; read off the chats' counts and time index, `договор*` takes 31 ms instead of
+478 ms, and a frequent word end (`*ени`) answers instead of being refused.
+
 [Mentions](mentions/README.md): how long finding phones, e-mail, links and documents by rules takes on one person's
 messages, and how many of each it finds, totals only.
 
@@ -90,6 +94,7 @@ archive copy, and released `memo search all` 6.6 s → 1.4 s with identical resu
 |---|---|
 | [message-search/](message-search/README.md) | Synthetic corpus, question supplement, benchmark runners, pinned model manifest and historical JSON reports |
 | [substring-notes/](substring-notes/README.md) | Notes, documents, annotations and contact notes found by any part of a word in any letter case: five indexes compared, then cli-core store version 39 measured (2026-10-30) |
+| [coverage-step/](coverage-step/README.md) | The coverage step every message search paid, and word-end patterns checked only up to the page, before and after at 100k and 1M (2026-11-11) |
 | [infix-chat-text/](infix-chat-text/README.md) | `*word*` in chat-message text through `messages_fts`, timed through `searchStore` at 100k and 1M, before and after (2026-11-11) |
 | [stemming/](stemming/README.md) | Do Russian and Spanish word forms get found: Snowball stems against trigrams on UD treebanks, with the cost and the gate of the stem index (2026-10-05) |
 | [everywhere/](everywhere/README.md) | Latency of a bare word that also looks in transcripts, file names and file text (2026-10-13) |

performance/search/coverage-step/README.md

@@ -0,0 +1,70 @@
+# The coverage step of every message search, and word-end patterns
+
+Measured 2026-11-11 for the cli-core pull request that fixes the search gaps of the 2026-11-11 audit (PERF-100,
+PERF-99).
+
+## Why we ran it
+
+A message search answers with its coverage: which chats it read, which are held in full, which need a sync. The
+audit's CPU profile of [`*word*` in chat messages](../infix-chat-text/README.md) showed that this step alone took about
+0.48 s of every search at 1M messages, whatever the query: it counted and dated every message of every chat
+(`chatStats`, a `count`/`min`/`max` grouped over all messages). That is a quarter of the 2 s limit agents and MCP
+search under.
+
+The same audit found that a word-end pattern (`*ени`, `*ость`) without `chat:` or `date:` was refused before a single
+message was checked, because the budget counted every candidate up front, and the refusal did not say what to write.
+
+The questions: how much faster is a search once coverage reads the trigger-kept `chats.message_count` and each chat's
+newest message off its time index, and does a word-end pattern answer once candidates are checked in the answer's
+order and stop at the page? The answer decided whether the change ships in the gap-fixing pull request.
+
+## Setup
+
+- AMD Ryzen AI 9 HX 470, 30 GB RAM, Linux 7.0, Node 24.19.0 (`node:sqlite`).
+- cli-core `origin/main` at 0.641.0 (`647fae685`) as "before", and the branch `fix/search-gaps` as "after", both built
+  to `dist/`.
+- The synthetic stores of the [infix-chat-text](../infix-chat-text/README.md) measurement: 100,000 and 1,000,000
+  messages in 100 chats, Russian and English syllable words, written through the store's own API. Nothing from a real
+  account.
+- [`bench.mjs`](bench.mjs) times `chatStats` (the old coverage read), `chatsNewest` (the new one, after only) and
+  `searchStore` (what `search messages` runs) with limit 50; p50 of 7 warm runs.
+
+```sh
+node ../infix-chat-text/bench.mjs gen <cli-core checkout> /var/tmp/infix-chat-text/chat-1000000.db 1000000
+node bench.mjs <cli-core checkout>/dist /var/tmp/infix-chat-text/chat-1000000.db after-1m
+```
+
+## Results
+
+p50 ms; "50" is a full first page, "refused" the reason and budget.
+
+| Step or query | 100k before | 100k after | 1M before | 1M after |
+|---|---|---|---|---|
+| coverage read (`chatStats` → `chatsNewest`) | 39.6 | 0.2 | 445.5 | 0.2 |
+| `договор*` | 50, 44.5 | 50, 5.8 | 50, 478.0 | 50, 30.7 |
+| `договор` | 50, 44.6 | 50, 6.1 | 50, 460.3 | 50, 28.6 |
+| `*договор*` | 50, 48.9 | 50, 10.0 | 50, 532.1 | 50, 80.8 |
+| `договор AND chat:12` | 50, 43.6 | 50, 5.3 | 50, 484.7 | 50, 24.6 |
+| `*ени` (word end, frequent) | refused (body bytes), 65.1 | 50, 93.1 | refused (candidate rows), 620.3 | 50, 647.3 |
+| `*ени*` | 50, 93.9 | 50, 55.0 | 50, 951.7 | 50, 511.5 |
+| `*ость` (word end) | 50, 59.8 | 50, 19.3 | refused (body bytes), 174.5 | 50, 179.1 |
+
+Raw lines: [`results/`](results/).
+
+## What it showed
+
+- The coverage read was nearly all of an ordinary search: at 1M messages `договор*` went from 478 ms to 31 ms, and
+  every query in the table lost the same ~0.45 s. The new read takes 0.2 ms at both sizes, since it reads one row per
+  chat.
+- A frequent word end now answers with a full first page instead of a refusal, at 1M in about the time the refusal
+  used to take (647 ms against 620 ms): the candidate query is the same, and checking stops at the 51st hit.
+- A word end that cannot fill a page within the budget is still refused; the refusal now names the pattern and
+  suggests `*part*`, more of the word, `chat:` or `date:`.
+
+## What was not measured
+
+- A store with many more chats: the new coverage read does one index probe per chat, so 10,000 chats cost 10,000
+  probes. It was not timed here.
+- A cold OS cache: the runs are warm, as in the earlier search pages.
+- Mail, whose coverage has no per-chat step of this kind.
+- Real archives: only synthetic stores.

performance/search/coverage-step/bench.mjs

@@ -0,0 +1,41 @@
+// Times a whole message search and the coverage step on a synthetic store, for one cli-core build.
+//   node bench.mjs <cli-core checkout>/dist <store file> <label>
+// The store comes from ../infix-chat-text/bench.mjs gen. Nothing here opens the owner's store.
+import { join } from "node:path"
+import { pathToFileURL } from "node:url"
+
+const [root, path, label] = process.argv.slice(2)
+const load = (file) => import(pathToFileURL(join(root, file)).href)
+const { openStore } = await load("store/store.js")
+const { searchStore } = await load("services/messages-archive-search.js")
+const account = { provider: "telegram", account: "1" }
+const store = await openStore({ path })
+const p50 = async (run) => {
+  const runs = []
+  let outcome
+  for (let at = 0; at < 7; at++) {
+    const began = performance.now()
+    try {
+      outcome = await run()
+    } catch (error) {
+      outcome = { refused: error?.details?.reason ?? error?.code, budget: error?.details?.budget }
+    }
+    runs.push(performance.now() - began)
+  }
+  runs.sort((a, b) => a - b)
+  return { p50_ms: Math.round(runs[3] * 10) / 10, ...outcome }
+}
+const out = (row) => process.stdout.write(`${JSON.stringify({ label, ...row })}\n`)
+out({ step: "chatStats", ...(await p50(async () => ({ chats: (await store.chats.chatStats(account)).length }))) })
+if (store.chats.chatsNewest)
+  out({ step: "chatsNewest", ...(await p50(async () => ({ chats: (await store.chats.chatsNewest(account)).length }))) })
+for (const query of ["договор*", "договор", "*договор*", "договор AND chat:12", "*ени", "*ени*", "*ость"]) {
+  out({
+    query,
+    ...(await p50(async () => {
+      const found = await searchStore(store, account, { text: query, limit: 50 })
+      return { hits: found.items.length, has_more: found.hasMore }
+    })),
+  })
+}
+await store.close()

performance/search/coverage-step/results/chat-100000.jsonl

@@ -0,0 +1,17 @@
+{"label":"before-100k","step":"chatStats","p50_ms":39.6,"chats":100}
+{"label":"before-100k","query":"договор*","p50_ms":44.5,"hits":50,"has_more":true}
+{"label":"before-100k","query":"договор","p50_ms":44.6,"hits":50,"has_more":true}
+{"label":"before-100k","query":"*договор*","p50_ms":48.9,"hits":50,"has_more":true}
+{"label":"before-100k","query":"договор AND chat:12","p50_ms":43.6,"hits":50,"has_more":true}
+{"label":"before-100k","query":"*ени","p50_ms":65.1,"refused":"query.too-large","budget":"body bytes"}
+{"label":"before-100k","query":"*ени*","p50_ms":93.9,"hits":50,"has_more":true}
+{"label":"before-100k","query":"*ость","p50_ms":59.8,"hits":50,"has_more":true}
+{"label":"after-100k","step":"chatStats","p50_ms":38.2,"chats":100}
+{"label":"after-100k","step":"chatsNewest","p50_ms":0.2,"chats":100}
+{"label":"after-100k","query":"договор*","p50_ms":5.8,"hits":50,"has_more":true}
+{"label":"after-100k","query":"договор","p50_ms":6.1,"hits":50,"has_more":true}
+{"label":"after-100k","query":"*договор*","p50_ms":10,"hits":50,"has_more":true}
+{"label":"after-100k","query":"договор AND chat:12","p50_ms":5.3,"hits":50,"has_more":true}
+{"label":"after-100k","query":"*ени","p50_ms":93.1,"hits":50,"has_more":true}
+{"label":"after-100k","query":"*ени*","p50_ms":55,"hits":50,"has_more":true}
+{"label":"after-100k","query":"*ость","p50_ms":19.3,"hits":50,"has_more":true}

performance/search/coverage-step/results/chat-1000000.jsonl

@@ -0,0 +1,17 @@
+{"label":"before-1m","step":"chatStats","p50_ms":445.5,"chats":100}
+{"label":"before-1m","query":"договор*","p50_ms":478,"hits":50,"has_more":true}
+{"label":"before-1m","query":"договор","p50_ms":460.3,"hits":50,"has_more":true}
+{"label":"before-1m","query":"*договор*","p50_ms":532.1,"hits":50,"has_more":true}
+{"label":"before-1m","query":"договор AND chat:12","p50_ms":484.7,"hits":50,"has_more":true}
+{"label":"before-1m","query":"*ени","p50_ms":620.3,"refused":"query.too-large","budget":"candidate rows"}
+{"label":"before-1m","query":"*ени*","p50_ms":951.7,"hits":50,"has_more":true}
+{"label":"before-1m","query":"*ость","p50_ms":174.5,"refused":"query.too-large","budget":"body bytes"}
+{"label":"after-1m","step":"chatStats","p50_ms":446.3,"chats":100}
+{"label":"after-1m","step":"chatsNewest","p50_ms":0.2,"chats":100}
+{"label":"after-1m","query":"договор*","p50_ms":30.7,"hits":50,"has_more":true}
+{"label":"after-1m","query":"договор","p50_ms":28.6,"hits":50,"has_more":true}
+{"label":"after-1m","query":"*договор*","p50_ms":80.8,"hits":50,"has_more":true}
+{"label":"after-1m","query":"договор AND chat:12","p50_ms":24.6,"hits":50,"has_more":true}
+{"label":"after-1m","query":"*ени","p50_ms":647.3,"hits":50,"has_more":true}
+{"label":"after-1m","query":"*ени*","p50_ms":511.5,"hits":50,"has_more":true}
+{"label":"after-1m","query":"*ость","p50_ms":179.1,"hits":50,"has_more":true}