From db0cb273ee272078ea7ecb3d50a50b62d995d8a4 Mon Sep 17 00:00:00 2001 From: leemour Date: Sat, 14 Nov 2026 04:14:09 +0100 Subject: [PATCH 1/2] perf(documents): catalog speed and memory at 10k, 50k and 100k entries Why: the Notion CLI syncs everything a token sees by default, and cli-core refused a documents account over 10,000 entries. The owner lifted the ceiling (NEED-2639 A) on condition of a measurement. The script times a catalog import, the same import again, 200 saves and a list page on one synthetic documents account through cli-core's own documentIngestion. Before: over 10,000 entries cannot be imported. After: 100,000 entries import in 8.6 s with a 489 MB peak, 200 saves take 0.3 s at any size (2.4 s at 9,800 before), a list page 60 ms. Co-Authored-By: Claude Opus 5.5 --- performance/documents/README.md | 4 + performance/documents/catalog-scale/README.md | 76 ++++++++++++++++ performance/documents/catalog-scale/bench.mjs | 91 +++++++++++++++++++ 3 files changed, 171 insertions(+) create mode 100644 performance/documents/catalog-scale/README.md create mode 100644 performance/documents/catalog-scale/bench.mjs diff --git a/performance/documents/README.md b/performance/documents/README.md index 128d28fc..550373eb 100644 --- a/performance/documents/README.md +++ b/performance/documents/README.md @@ -2,6 +2,10 @@ Measurements of document conversion, shared retrieval and source-backed memories. +- [catalog-scale](catalog-scale/README.md): catalog import, saves and a list page of one documents account at + 10,000, 50,000 and 100,000 synthetic entries, before and after cli-core lifted its 10,000-entry ceiling + (2026-11-14). + - [annotation-transfer](annotation-transfer/README.md): packaged old/new SDK copy, reference and identity collisions, interruption/replay and retention of source annotations. - [drive-sheets](drive-sheets/README.md): guarded executable acceptance for native spreadsheet ranges, cell writes and generated Sheets methods. diff --git a/performance/documents/catalog-scale/README.md b/performance/documents/catalog-scale/README.md new file mode 100644 index 00000000..586b4d53 --- /dev/null +++ b/performance/documents/catalog-scale/README.md @@ -0,0 +1,76 @@ +# Documents catalog past 10,000 entries: speed and memory + +## Why it was run + +The Notion CLI is planned to sync everything a token can see by default (owner's ruling NEED-2636, 2026-11-14). A +mid-size Notion workspace has more than 10,000 pages. cli-core refused any documents account over 10,000 entries: +`catalog.list` failed above 10,000 entries or 32 MB of metadata, and the catalog import and `catalog.reconcile` +refused more than 10,000 ids. The cap was a safety bound from when the catalog was added, not a SQLite limit. The +owner chose to lift it and watch speed and memory with a measurement (NEED-2639 A). + +The question: with the ceiling gone, how long do the catalog import, a batch of saves and a list page take at +10,000, 50,000 and 100,000 entries, and how much memory does the process need? The answer feeds the cli-core pull +request that lifts the ceiling (`feat/documents-catalog-no-ceiling`, plan item C10). + +## Setup + +- Machine: AMD Ryzen AI 9 HX 470 (24 threads), 30 GB RAM, NVMe disk, Linux 7.0, Node 24.19.0 (SQLite 3.53.3). +- Before: cli-core `main` at `cab3953bc` (0.660.0). After: branch `feat/documents-catalog-no-ceiling` on the + same `main`. Each built with `pnpm build`. +- Path: cli-core's own `openStore` and `documentIngestion({ provider: "google-drive" })` on a new temporary + store. No network, no account, no CLI run. +- Data, all synthetic: one documents account with 1 root folder, 100 folders and N files spread over them. +- Steps: + - **import**: `catalog.import` of all N + 101 entries into an empty account; + - **import again**: the same list again, which is what every sync does when nothing changed; + - **save 200**: 200 document texts saved, spread over the account; + - **list page**: one page of 50 entries at the end of the list. +- Each step is timed once; the peak RSS is sampled every 5 ms during the step. The heap growth is after a forced GC. + +```sh +cd && pnpm build +node --expose-gc performance/documents/catalog-scale/bench.mjs [saves] +``` + +[`bench.mjs`](bench.mjs) prints one JSON line per run. + +## Results + +Before (`main`): an account over 10,000 entries cannot be imported at all ("catalog scope exceeds 10000 +entries"). Under the cap, at 9,800 files: + +| step | time | peak RSS | +| --- | ---: | ---: | +| import | 987 ms | 186 MB | +| import again | 894 ms | 186 MB | +| save 200 | 2,427 ms | 251 MB | +| list page | 12 ms | 250 MB | + +After: + +| files | import | import again | save 200 | list page | peak RSS | +| ---: | ---: | ---: | ---: | ---: | ---: | +| 10,000 | 753 ms | 688 ms | 330 ms | 9 ms | 186 MB | +| 50,000 | 3,354 ms | 3,365 ms | 277 ms | 29 ms | 358 MB | +| 100,000 | 8,624 ms | 7,324 ms | 303 ms | 60 ms | 489 MB | + +## What it showed + +- **The ceiling can go.** 100,000 entries import in under 9 s and need under 500 MB at the peak. +- **The import grows linearly**: about 75–85 µs per entry. It runs on every sync, also when nothing changed, so a + 100,000-page workspace pays about 7 s per sync for it. That is the next thing to make faster (write only changed + entries), not a blocker. +- **Saves no longer depend on the account size.** Before, the first save read every document of the account into + memory, in pages that re-read all rows before them; that is the 2.4 s at 9,800. A first version without the cap + but with the same listing took 37 s for 200 saves at 100,000. After, each save looks up only its own document + through an index: about 1.5 ms a save at every size. +- **A list page stays fast** (60 ms at 100,000). The old guard also counted the whole account on every page. +- **Memory grows with the entries handed in.** The caller (the sync engine) holds the whole list, about 2 KB per + entry here; cli-core adds no second full copy. + +## Not measured + +- A real Notion or Drive account, real metadata sizes, or a sync over the network. +- The sync engine's own state file (`SourceState`), which holds every document and is read and written whole. +- Search and context queries over a store of this size. +- More than one run per point; the numbers are single runs on an idle-ish machine shared with other work. diff --git a/performance/documents/catalog-scale/bench.mjs b/performance/documents/catalog-scale/bench.mjs new file mode 100644 index 00000000..7195a758 --- /dev/null +++ b/performance/documents/catalog-scale/bench.mjs @@ -0,0 +1,91 @@ +// Usage: node --expose-gc bench.mjs [saves] +// Synthetic data only. One Drive-shaped documents account: a catalog import of files under 100 folders, +// the same import again (a sync with nothing new), document texts saved, and a page of the catalog list. +import { mkdtempSync, rmSync } from "node:fs" +import { tmpdir } from "node:os" +import { join } from "node:path" +import { pathToFileURL } from "node:url" + +const [core, entriesArg = "10000", savesArg = "200"] = process.argv.slice(2) +const entries = Number(entriesArg) +const saves = Number(savesArg) +const load = (path) => import(pathToFileURL(join(core, "dist", path)).href) +const { openStore } = await load("store/index.js") +const { documentIngestion } = await load("services/documents/ingestion.js") + +const dir = mkdtempSync(join(tmpdir(), "catalog-scale-")) +const store = await openStore({ path: join(dir, "wirecat.db") }) +const folders = Array.from({ length: 100 }, (_, i) => ({ id: `folder-${i}`, name: `Folder ${i}`, parents: ["root"] })) +const file = (i) => ({ + id: `file-${i}`, + name: `Report ${i} budget plan`, + parents: [`folder-${i % 100}`], + mime: "text/plain", + url: `https://example.invalid/file-${i}`, + trashed: false, + canDownload: true, + version: "1", + modifiedAt: "2026-11-01T10:00:00.000Z", +}) +const files = [ + { id: "root", name: "Root", parents: [], mime: "inode/directory" }, + ...folders.map((f) => ({ ...f, mime: "inode/directory" })), + ...Array.from({ length: entries }, (_, i) => file(i)), +].map((f) => ({ url: `https://example.invalid/${f.id}`, trashed: false, canDownload: true, version: "1", ...f })) +const ingest = documentIngestion(store, { + provider: "google-drive", + externalId: (id) => `${id}.md`, + catalogInput: (f) => ({ + externalId: `${f.id}.md`, + kind: f.mime === "inode/directory" ? "folder" : "file", + name: f.name, + parents: f.parents.map((id) => `${id}.md`), + url: f.url, + mime: f.mime, + }), + frontMatter: (content) => ({ source: "google-drive", fileId: content.metadata.id, sourceUrl: content.metadata.url }), +}) + +let peak = 0 +const sample = setInterval(() => { + peak = Math.max(peak, process.memoryUsage().rss) +}, 5) +const step = async (name, run) => { + globalThis.gc?.() + const heapBefore = process.memoryUsage().heapUsed + peak = process.memoryUsage().rss + const start = performance.now() + try { + await run() + const ms = performance.now() - start + peak = Math.max(peak, process.memoryUsage().rss) + return { + step: name, + ms: Math.round(ms), + peak_rss_mb: Math.round(peak / 2 ** 20), + heap_growth_mb: Math.round((process.memoryUsage().heapUsed - heapBefore) / 2 ** 20), + } + } catch (error) { + return { step: name, ms: Math.round(performance.now() - start), error: error.message } + } +} +const results = [] +results.push(await step("import", () => ingest.catalog.import("ana@example.com", files))) +results.push(await step("import again", () => ingest.catalog.import("ana@example.com", files))) +results.push( + await step(`save ${saves}`, async () => { + for (let i = 0; i < saves; i++) { + const metadata = files[101 + Math.floor((i * entries) / saves)] + await ingest.save("ana@example.com", { metadata, text: `budget text ${i}`, markdown: "", tabs: [] }, `hash-${i}`) + } + }), +) +results.push( + await step("list page", () => + ingest.catalog.list("ana@example.com", { limit: 50, offset: Math.max(0, entries - 50) }), + ), +) +clearInterval(sample) +await store.close() +rmSync(dir, { recursive: true, force: true }) +process.stdout.write(`${JSON.stringify({ entries, saves, node: process.version, results })}\n`) From 49fbeda23dd30e96b6c736082430181175a95757 Mon Sep 17 00:00:00 2001 From: leemour Date: Sat, 14 Nov 2026 06:38:52 +0100 Subject: [PATCH 2/2] docs(performance): catalog-scale runs from cli-testing, names its base, shows heap growth Review of #375: the command now points the script into cli-testing, the base commit is the one measured, the tables give time, peak RSS and heap growth per step and say which step the highest peak is from, and the observed callback is listed as not measured. Co-Authored-By: Claude Opus 5.5 --- performance/documents/catalog-scale/README.md | 43 +++++++++++-------- 1 file changed, 26 insertions(+), 17 deletions(-) diff --git a/performance/documents/catalog-scale/README.md b/performance/documents/catalog-scale/README.md index 586b4d53..ae5ce571 100644 --- a/performance/documents/catalog-scale/README.md +++ b/performance/documents/catalog-scale/README.md @@ -15,8 +15,10 @@ request that lifts the ceiling (`feat/documents-catalog-no-ceiling`, plan item C ## Setup - Machine: AMD Ryzen AI 9 HX 470 (24 threads), 30 GB RAM, NVMe disk, Linux 7.0, Node 24.19.0 (SQLite 3.53.3). -- Before: cli-core `main` at `cab3953bc` (0.660.0). After: branch `feat/documents-catalog-no-ceiling` on the - same `main`. Each built with `pnpm build`. +- Before: cli-core `main` at `cab3953bc` (0.660.0). After: the code of cli-core + [#2944](https://github.com/WireCatLabs/cli-core/pull/2944) on that same `cab3953bc`, measured before the branch + was rebased onto a newer `main` (`7c067ff16` and later; nothing in between touches the catalog or ingestion). + Each built with `pnpm build`. - Path: cli-core's own `openStore` and `documentIngestion({ provider: "google-drive" })` on a new temporary store. No network, no account, no CLI run. - Data, all synthetic: one documents account with 1 root folder, 100 folders and N files spread over them. @@ -25,11 +27,13 @@ request that lifts the ceiling (`feat/documents-catalog-no-ceiling`, plan item C - **import again**: the same list again, which is what every sync does when nothing changed; - **save 200**: 200 document texts saved, spread over the account; - **list page**: one page of 50 entries at the end of the list. -- Each step is timed once; the peak RSS is sampled every 5 ms during the step. The heap growth is after a forced GC. +- Each step is timed once. **Peak RSS** is the process's highest resident memory sampled every 5 ms during that + step; it includes what earlier steps left, so it is the process size, not the step's own cost. **Heap growth** + is the JavaScript heap after the step minus before it, after a forced GC before the step. ```sh cd && pnpm build -node --expose-gc performance/documents/catalog-scale/bench.mjs [saves] +node --expose-gc /performance/documents/catalog-scale/bench.mjs [saves] ``` [`bench.mjs`](bench.mjs) prints one JSON line per run. @@ -39,24 +43,26 @@ node --expose-gc performance/documents/catalog-scale/bench.mjs