perf(documents): catalog speed and memory at 10k, 50k and 100k entries

performance/documents/README.md

@@ -2,6 +2,10 @@
 
 Measurements of document conversion, shared retrieval and source-backed memories.
 
+- [catalog-scale](catalog-scale/README.md): catalog import, saves and a list page of one documents account at
+  10,000, 50,000 and 100,000 synthetic entries, before and after cli-core lifted its 10,000-entry ceiling
+  (2026-11-14).
+
 - [annotation-transfer](annotation-transfer/README.md): packaged old/new SDK copy, reference and identity collisions, interruption/replay and retention of source annotations.
 
 - [drive-sheets](drive-sheets/README.md): guarded executable acceptance for native spreadsheet ranges, cell writes and generated Sheets methods.

performance/documents/catalog-scale/README.md

@@ -0,0 +1,85 @@
+# Documents catalog past 10,000 entries: speed and memory
+
+## Why it was run
+
+The Notion CLI is planned to sync everything a token can see by default (owner's ruling NEED-2636, 2026-11-14). A
+mid-size Notion workspace has more than 10,000 pages. cli-core refused any documents account over 10,000 entries:
+`catalog.list` failed above 10,000 entries or 32 MB of metadata, and the catalog import and `catalog.reconcile`
+refused more than 10,000 ids. The cap was a safety bound from when the catalog was added, not a SQLite limit. The
+owner chose to lift it and watch speed and memory with a measurement (NEED-2639 A).
+
+The question: with the ceiling gone, how long do the catalog import, a batch of saves and a list page take at
+10,000, 50,000 and 100,000 entries, and how much memory does the process need? The answer feeds the cli-core pull
+request that lifts the ceiling (`feat/documents-catalog-no-ceiling`, plan item C10).
+
+## Setup
+
+- Machine: AMD Ryzen AI 9 HX 470 (24 threads), 30 GB RAM, NVMe disk, Linux 7.0, Node 24.19.0 (SQLite 3.53.3).
+- Before: cli-core `main` at `cab3953bc` (0.660.0). After: the code of cli-core
+  [#2944](https://github.com/WireCatLabs/cli-core/pull/2944) on that same `cab3953bc`, measured before the branch
+  was rebased onto a newer `main` (`7c067ff16` and later; nothing in between touches the catalog or ingestion).
+  Each built with `pnpm build`.
+- Path: cli-core's own `openStore` and `documentIngestion({ provider: "google-drive" })` on a new temporary
+  store. No network, no account, no CLI run.
+- Data, all synthetic: one documents account with 1 root folder, 100 folders and N files spread over them.
+- Steps:
+  - **import**: `catalog.import` of all N + 101 entries into an empty account;
+  - **import again**: the same list again, which is what every sync does when nothing changed;
+  - **save 200**: 200 document texts saved, spread over the account;
+  - **list page**: one page of 50 entries at the end of the list.
+- Each step is timed once. **Peak RSS** is the process's highest resident memory sampled every 5 ms during that
+  step; it includes what earlier steps left, so it is the process size, not the step's own cost. **Heap growth**
+  is the JavaScript heap after the step minus before it, after a forced GC before the step.
+
+```sh
+cd <cli-core checkout> && pnpm build
+node --expose-gc <cli-testing checkout>/performance/documents/catalog-scale/bench.mjs <cli-core checkout> <entries> [saves]
+```
+
+[`bench.mjs`](bench.mjs) prints one JSON line per run.
+
+## Results
+
+Before (`main`): an account over 10,000 entries cannot be imported at all ("catalog scope exceeds 10000
+entries"). Under the cap, at 9,800 files:
+
+| step | time | peak RSS | heap growth |
+| --- | ---: | ---: | ---: |
+| import | 987 ms | 186 MB | 44 MB |
+| import again | 894 ms | 186 MB | 25 MB |
+| save 200 | 2,427 ms | 251 MB | 24 MB |
+| list page | 12 ms | 250 MB | 0 MB |
+
+After (time · peak RSS · heap growth):
+
+| files | import | import again | save 200 | list page |
+| ---: | ---: | ---: | ---: | ---: |
+| 10,000 | 753 ms · 185 MB · 50 MB | 688 ms · 186 MB · 32 MB | 330 ms · 185 MB · 24 MB | 9 ms · 185 MB · 0 MB |
+| 50,000 | 3,354 ms · 357 MB · 121 MB | 3,365 ms · 358 MB · 45 MB | 277 ms · 354 MB · 24 MB | 29 ms · 354 MB · 0 MB |
+| 100,000 | 8,624 ms · 477 MB · 218 MB | 7,324 ms · 489 MB · 125 MB | 303 ms · 480 MB · 24 MB | 60 ms · 480 MB · 0 MB |
+
+The highest peak, 489 MB at 100,000, is from the second import.
+
+## What it showed
+
+- **The ceiling can go.** 100,000 entries import in under 9 s; the process peaks under 500 MB (second import).
+- **The import grows linearly**: about 75–85 µs per entry. It runs on every sync, also when nothing changed, so a
+  100,000-page workspace pays about 7 s per sync for it. That is the next thing to make faster (write only changed
+  entries), not a blocker.
+- **Saves no longer depend on the account size.** Before, the first save read every document of the account into
+  memory, in pages that re-read all rows before them; that is the 2.4 s at 9,800. A first version without the cap
+  but with the same listing took 37 s for 200 saves at 100,000. After, each save looks up only its own document
+  through an index: about 1.5 ms a save at every size.
+- **A list page stays fast** (60 ms at 100,000). The old guard also counted the whole account on every page.
+- **Memory grows with the entries handed in.** The caller (the sync engine) holds the whole list, about 2 KB per
+  entry here; cli-core adds no second full copy.
+
+## Not measured
+
+- The `observed` callback: the run binds none. A binding that gives one (drive records each file's Google owners
+  with it) makes the import read every entry back and run the callback once per entry, on every sync.
+
+- A real Notion or Drive account, real metadata sizes, or a sync over the network.
+- The sync engine's own state file (`SourceState`), which holds every document and is read and written whole.
+- Search and context queries over a store of this size.
+- More than one run per point; the numbers are single runs on an idle-ish machine shared with other work.

performance/documents/catalog-scale/bench.mjs

@@ -0,0 +1,91 @@
+// Usage: node --expose-gc bench.mjs <cli-core checkout with dist> <entries> [saves]
+// Synthetic data only. One Drive-shaped documents account: a catalog import of <entries> files under 100 folders,
+// the same import again (a sync with nothing new), <saves> document texts saved, and a page of the catalog list.
+import { mkdtempSync, rmSync } from "node:fs"
+import { tmpdir } from "node:os"
+import { join } from "node:path"
+import { pathToFileURL } from "node:url"
+
+const [core, entriesArg = "10000", savesArg = "200"] = process.argv.slice(2)
+const entries = Number(entriesArg)
+const saves = Number(savesArg)
+const load = (path) => import(pathToFileURL(join(core, "dist", path)).href)
+const { openStore } = await load("store/index.js")
+const { documentIngestion } = await load("services/documents/ingestion.js")
+
+const dir = mkdtempSync(join(tmpdir(), "catalog-scale-"))
+const store = await openStore({ path: join(dir, "wirecat.db") })
+const folders = Array.from({ length: 100 }, (_, i) => ({ id: `folder-${i}`, name: `Folder ${i}`, parents: ["root"] }))
+const file = (i) => ({
+  id: `file-${i}`,
+  name: `Report ${i} budget plan`,
+  parents: [`folder-${i % 100}`],
+  mime: "text/plain",
+  url: `https://example.invalid/file-${i}`,
+  trashed: false,
+  canDownload: true,
+  version: "1",
+  modifiedAt: "2026-11-01T10:00:00.000Z",
+})
+const files = [
+  { id: "root", name: "Root", parents: [], mime: "inode/directory" },
+  ...folders.map((f) => ({ ...f, mime: "inode/directory" })),
+  ...Array.from({ length: entries }, (_, i) => file(i)),
+].map((f) => ({ url: `https://example.invalid/${f.id}`, trashed: false, canDownload: true, version: "1", ...f }))
+const ingest = documentIngestion(store, {
+  provider: "google-drive",
+  externalId: (id) => `${id}.md`,
+  catalogInput: (f) => ({
+    externalId: `${f.id}.md`,
+    kind: f.mime === "inode/directory" ? "folder" : "file",
+    name: f.name,
+    parents: f.parents.map((id) => `${id}.md`),
+    url: f.url,
+    mime: f.mime,
+  }),
+  frontMatter: (content) => ({ source: "google-drive", fileId: content.metadata.id, sourceUrl: content.metadata.url }),
+})
+
+let peak = 0
+const sample = setInterval(() => {
+  peak = Math.max(peak, process.memoryUsage().rss)
+}, 5)
+const step = async (name, run) => {
+  globalThis.gc?.()
+  const heapBefore = process.memoryUsage().heapUsed
+  peak = process.memoryUsage().rss
+  const start = performance.now()
+  try {
+    await run()
+    const ms = performance.now() - start
+    peak = Math.max(peak, process.memoryUsage().rss)
+    return {
+      step: name,
+      ms: Math.round(ms),
+      peak_rss_mb: Math.round(peak / 2 ** 20),
+      heap_growth_mb: Math.round((process.memoryUsage().heapUsed - heapBefore) / 2 ** 20),
+    }
+  } catch (error) {
+    return { step: name, ms: Math.round(performance.now() - start), error: error.message }
+  }
+}
+const results = []
+results.push(await step("import", () => ingest.catalog.import("ana@example.com", files)))
+results.push(await step("import again", () => ingest.catalog.import("ana@example.com", files)))
+results.push(
+  await step(`save ${saves}`, async () => {
+    for (let i = 0; i < saves; i++) {
+      const metadata = files[101 + Math.floor((i * entries) / saves)]
+      await ingest.save("ana@example.com", { metadata, text: `budget text ${i}`, markdown: "", tabs: [] }, `hash-${i}`)
+    }
+  }),
+)
+results.push(
+  await step("list page", () =>
+    ingest.catalog.list("ana@example.com", { limit: 50, offset: Math.max(0, entries - 50) }),
+  ),
+)
+clearInterval(sample)
+await store.close()
+rmSync(dir, { recursive: true, force: true })
+process.stdout.write(`${JSON.stringify({ entries, saves, node: process.version, results })}\n`)