diff --git a/performance/documents/README.md b/performance/documents/README.md index 128d28fc..550373eb 100644 --- a/performance/documents/README.md +++ b/performance/documents/README.md @@ -2,6 +2,10 @@ Measurements of document conversion, shared retrieval and source-backed memories. +- [catalog-scale](catalog-scale/README.md): catalog import, saves and a list page of one documents account at + 10,000, 50,000 and 100,000 synthetic entries, before and after cli-core lifted its 10,000-entry ceiling + (2026-11-14). + - [annotation-transfer](annotation-transfer/README.md): packaged old/new SDK copy, reference and identity collisions, interruption/replay and retention of source annotations. - [drive-sheets](drive-sheets/README.md): guarded executable acceptance for native spreadsheet ranges, cell writes and generated Sheets methods. diff --git a/performance/documents/catalog-scale/README.md b/performance/documents/catalog-scale/README.md new file mode 100644 index 00000000..ae5ce571 --- /dev/null +++ b/performance/documents/catalog-scale/README.md @@ -0,0 +1,85 @@ +# Documents catalog past 10,000 entries: speed and memory + +## Why it was run + +The Notion CLI is planned to sync everything a token can see by default (owner's ruling NEED-2636, 2026-11-14). A +mid-size Notion workspace has more than 10,000 pages. cli-core refused any documents account over 10,000 entries: +`catalog.list` failed above 10,000 entries or 32 MB of metadata, and the catalog import and `catalog.reconcile` +refused more than 10,000 ids. The cap was a safety bound from when the catalog was added, not a SQLite limit. The +owner chose to lift it and watch speed and memory with a measurement (NEED-2639 A). + +The question: with the ceiling gone, how long do the catalog import, a batch of saves and a list page take at +10,000, 50,000 and 100,000 entries, and how much memory does the process need? The answer feeds the cli-core pull +request that lifts the ceiling (`feat/documents-catalog-no-ceiling`, plan item C10). + +## Setup + +- Machine: AMD Ryzen AI 9 HX 470 (24 threads), 30 GB RAM, NVMe disk, Linux 7.0, Node 24.19.0 (SQLite 3.53.3). +- Before: cli-core `main` at `cab3953bc` (0.660.0). After: the code of cli-core + [#2944](https://github.com/WireCatLabs/cli-core/pull/2944) on that same `cab3953bc`, measured before the branch + was rebased onto a newer `main` (`7c067ff16` and later; nothing in between touches the catalog or ingestion). + Each built with `pnpm build`. +- Path: cli-core's own `openStore` and `documentIngestion({ provider: "google-drive" })` on a new temporary + store. No network, no account, no CLI run. +- Data, all synthetic: one documents account with 1 root folder, 100 folders and N files spread over them. +- Steps: + - **import**: `catalog.import` of all N + 101 entries into an empty account; + - **import again**: the same list again, which is what every sync does when nothing changed; + - **save 200**: 200 document texts saved, spread over the account; + - **list page**: one page of 50 entries at the end of the list. +- Each step is timed once. **Peak RSS** is the process's highest resident memory sampled every 5 ms during that + step; it includes what earlier steps left, so it is the process size, not the step's own cost. **Heap growth** + is the JavaScript heap after the step minus before it, after a forced GC before the step. + +```sh +cd && pnpm build +node --expose-gc /performance/documents/catalog-scale/bench.mjs [saves] +``` + +[`bench.mjs`](bench.mjs) prints one JSON line per run. + +## Results + +Before (`main`): an account over 10,000 entries cannot be imported at all ("catalog scope exceeds 10000 +entries"). Under the cap, at 9,800 files: + +| step | time | peak RSS | heap growth | +| --- | ---: | ---: | ---: | +| import | 987 ms | 186 MB | 44 MB | +| import again | 894 ms | 186 MB | 25 MB | +| save 200 | 2,427 ms | 251 MB | 24 MB | +| list page | 12 ms | 250 MB | 0 MB | + +After (time · peak RSS · heap growth): + +| files | import | import again | save 200 | list page | +| ---: | ---: | ---: | ---: | ---: | +| 10,000 | 753 ms · 185 MB · 50 MB | 688 ms · 186 MB · 32 MB | 330 ms · 185 MB · 24 MB | 9 ms · 185 MB · 0 MB | +| 50,000 | 3,354 ms · 357 MB · 121 MB | 3,365 ms · 358 MB · 45 MB | 277 ms · 354 MB · 24 MB | 29 ms · 354 MB · 0 MB | +| 100,000 | 8,624 ms · 477 MB · 218 MB | 7,324 ms · 489 MB · 125 MB | 303 ms · 480 MB · 24 MB | 60 ms · 480 MB · 0 MB | + +The highest peak, 489 MB at 100,000, is from the second import. + +## What it showed + +- **The ceiling can go.** 100,000 entries import in under 9 s; the process peaks under 500 MB (second import). +- **The import grows linearly**: about 75–85 µs per entry. It runs on every sync, also when nothing changed, so a + 100,000-page workspace pays about 7 s per sync for it. That is the next thing to make faster (write only changed + entries), not a blocker. +- **Saves no longer depend on the account size.** Before, the first save read every document of the account into + memory, in pages that re-read all rows before them; that is the 2.4 s at 9,800. A first version without the cap + but with the same listing took 37 s for 200 saves at 100,000. After, each save looks up only its own document + through an index: about 1.5 ms a save at every size. +- **A list page stays fast** (60 ms at 100,000). The old guard also counted the whole account on every page. +- **Memory grows with the entries handed in.** The caller (the sync engine) holds the whole list, about 2 KB per + entry here; cli-core adds no second full copy. + +## Not measured + +- The `observed` callback: the run binds none. A binding that gives one (drive records each file's Google owners + with it) makes the import read every entry back and run the callback once per entry, on every sync. + +- A real Notion or Drive account, real metadata sizes, or a sync over the network. +- The sync engine's own state file (`SourceState`), which holds every document and is read and written whole. +- Search and context queries over a store of this size. +- More than one run per point; the numbers are single runs on an idle-ish machine shared with other work. diff --git a/performance/documents/catalog-scale/bench.mjs b/performance/documents/catalog-scale/bench.mjs new file mode 100644 index 00000000..7195a758 --- /dev/null +++ b/performance/documents/catalog-scale/bench.mjs @@ -0,0 +1,91 @@ +// Usage: node --expose-gc bench.mjs [saves] +// Synthetic data only. One Drive-shaped documents account: a catalog import of files under 100 folders, +// the same import again (a sync with nothing new), document texts saved, and a page of the catalog list. +import { mkdtempSync, rmSync } from "node:fs" +import { tmpdir } from "node:os" +import { join } from "node:path" +import { pathToFileURL } from "node:url" + +const [core, entriesArg = "10000", savesArg = "200"] = process.argv.slice(2) +const entries = Number(entriesArg) +const saves = Number(savesArg) +const load = (path) => import(pathToFileURL(join(core, "dist", path)).href) +const { openStore } = await load("store/index.js") +const { documentIngestion } = await load("services/documents/ingestion.js") + +const dir = mkdtempSync(join(tmpdir(), "catalog-scale-")) +const store = await openStore({ path: join(dir, "wirecat.db") }) +const folders = Array.from({ length: 100 }, (_, i) => ({ id: `folder-${i}`, name: `Folder ${i}`, parents: ["root"] })) +const file = (i) => ({ + id: `file-${i}`, + name: `Report ${i} budget plan`, + parents: [`folder-${i % 100}`], + mime: "text/plain", + url: `https://example.invalid/file-${i}`, + trashed: false, + canDownload: true, + version: "1", + modifiedAt: "2026-11-01T10:00:00.000Z", +}) +const files = [ + { id: "root", name: "Root", parents: [], mime: "inode/directory" }, + ...folders.map((f) => ({ ...f, mime: "inode/directory" })), + ...Array.from({ length: entries }, (_, i) => file(i)), +].map((f) => ({ url: `https://example.invalid/${f.id}`, trashed: false, canDownload: true, version: "1", ...f })) +const ingest = documentIngestion(store, { + provider: "google-drive", + externalId: (id) => `${id}.md`, + catalogInput: (f) => ({ + externalId: `${f.id}.md`, + kind: f.mime === "inode/directory" ? "folder" : "file", + name: f.name, + parents: f.parents.map((id) => `${id}.md`), + url: f.url, + mime: f.mime, + }), + frontMatter: (content) => ({ source: "google-drive", fileId: content.metadata.id, sourceUrl: content.metadata.url }), +}) + +let peak = 0 +const sample = setInterval(() => { + peak = Math.max(peak, process.memoryUsage().rss) +}, 5) +const step = async (name, run) => { + globalThis.gc?.() + const heapBefore = process.memoryUsage().heapUsed + peak = process.memoryUsage().rss + const start = performance.now() + try { + await run() + const ms = performance.now() - start + peak = Math.max(peak, process.memoryUsage().rss) + return { + step: name, + ms: Math.round(ms), + peak_rss_mb: Math.round(peak / 2 ** 20), + heap_growth_mb: Math.round((process.memoryUsage().heapUsed - heapBefore) / 2 ** 20), + } + } catch (error) { + return { step: name, ms: Math.round(performance.now() - start), error: error.message } + } +} +const results = [] +results.push(await step("import", () => ingest.catalog.import("ana@example.com", files))) +results.push(await step("import again", () => ingest.catalog.import("ana@example.com", files))) +results.push( + await step(`save ${saves}`, async () => { + for (let i = 0; i < saves; i++) { + const metadata = files[101 + Math.floor((i * entries) / saves)] + await ingest.save("ana@example.com", { metadata, text: `budget text ${i}`, markdown: "", tabs: [] }, `hash-${i}`) + } + }), +) +results.push( + await step("list page", () => + ingest.catalog.list("ana@example.com", { limit: 50, offset: Math.max(0, entries - 50) }), + ), +) +clearInterval(sample) +await store.close() +rmSync(dir, { recursive: true, force: true }) +process.stdout.write(`${JSON.stringify({ entries, saves, node: process.version, results })}\n`)