Repository navigation
perf(documents): catalog speed and memory at 10k, 50k and 100k entries - #375
Merged
Merged
Conversation
Why: the Notion CLI syncs everything a token sees by default, and cli-core refused a documents account over 10,000 entries. The owner lifted the ceiling (NEED-2639 A) on condition of a measurement. The script times a catalog import, the same import again, 200 saves and a list page on one synthetic documents account through cli-core's own documentIngestion. Before: over 10,000 entries cannot be imported. After: 100,000 entries import in 8.6 s with a 489 MB peak, 200 saves take 0.3 s at any size (2.4 s at 9,800 before), a list page 60 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e, shows heap growth Review of #375: the command now points the script into cli-testing, the base commit is the one measured, the tables give time, peak RSS and heap growth per step and say which step the highest peak is from, and the observed callback is listed as not measured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leemour
added a commit
to WireCatLabs/cli-core
that referenced
this pull request
Nov 14, 2026
## Task notion-cli plan C10, owner's ruling NEED-2639 A: lift the per-account catalog ceiling and make the catalog import and the local document listing work in portions, so a sync goes past any total. A single call may stay bounded. ## Why A Notion workspace synced whole (the default, NEED-2636) passes 10,000 pages. cli-core refused such an account: `catalog.list` above 10,000 entries / 32 MB, `catalog.reconcile` and the ingestion import above 10,000 ids, and the ingestion listing stopped at 10,000. Drive has the same limit. ## What - `src/store/sqlite/catalog.ts`: the per-account guard in `list` is gone (it also counted the whole account on every page); `reconcile` takes any number of ids. - `src/services/documents/ingestion.ts`: the import takes any number of files, upserting and reporting `observed` 200 at a time. The whole-account note cache is gone: it read every document (with text) in pages of 500 that each re-read all rows before them. Each save and removal now looks up its one document with the new `notes.fileNote(folderId, path)` (`src/store/sqlite/file-note.ts`, indexed by the documents key, or the files row of a local folder). - Per-call bounds stay: 200 entries per catalog write, 1000 per list page, 1 MB per entry. - `docs/dev/DOCUMENTS.md`: the `<id>.md` external id contract that the shared catalog commands rely on (lane L2 asked for it to be either documented or removed; documented here), and the bounds. - `feat!`: an own `NotesStore` implementation must add `fileNote`; the `catalog.scope-limit` error is no longer raised (the reason stays registered). ## Measured [cli-testing#375](WireCatLabs/cli-testing#375), `performance/documents/catalog-scale`: before, over 10,000 entries cannot be imported. After, 100,000 entries import in 8.6 s with a 489 MB peak; 200 saves take 0.3 s at any size (2.4 s at 9,800 before, 37 s at 100,000 with the old listing); a list page takes 60 ms. The import runs on every sync, also when nothing changed (about 7 s at 100,000): the next thing to speed up, not a blocker. ## Testing - New `src/services/documents/ingestion-scale.test.ts`: 10,050 entries import, list past 10,000, save twice to the same document, retire one by re-import, remove. - `src/store/catalog.test.ts`: the reconcile-over-10,000 case now expects success. - Locally: typecheck; `src/services/documents src/documents src/store` 2116 passed before the last test update, then the touched files 112 passed; biome, docs checks green. - Not stacked: independent of #2938 (C1) and #2941 (C2). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
cli-core refused a documents account over 10,000 entries; the Notion CLI syncs a whole workspace by default. The owner lifted the ceiling (NEED-2639 A) on condition of a measurement. This is that measurement, for cli-core's
feat/documents-catalog-no-ceiling(plan item C10).What
performance/documents/catalog-scale/:bench.mjs(synthetic data, cli-core's owndocumentIngestion, no network or account) and its README, linked fromperformance/documents/README.md.Headline: before, over 10,000 entries cannot be imported. After, 100,000 entries import in 8.6 s with a 489 MB peak; 200 saves take 0.3 s at any size (2.4 s at 9,800 before); a list page 60 ms.
🤖 Generated with Claude Code