Skip to content

Bound commit and refs-index memory for large records - #38

Merged
isTravis merged 1 commit into
mainfrom
tfk/commit-memory
Oct 15, 2026
Merged

isTravis merged 1 commit into
mainfrom
tfk/commit-memory

Conversation

@tefkah

@tefkah tefkah commented Oct 15, 2026

Copy link
Copy Markdown
Member

The Crossref load into crossref-demo/works on staging (2026-10-14/15) kept hitting exceededMemory on underlay-staging-bulk. A 5M-record commit's units died on every attempt, and with 100k-record versions the bulk queue still showed memory errors. Crossref works average 6.5 KB (up to 148 KB), much larger than anything the commit path was tested with.

Measured locally on Node by pushing real Crossref sample records through the real jobs, with a file store, no cache, and live memory sampled after forced GC (peak above the job's starting point):

job before after
refs.index (120k–250k records) +103–114 MB +29–47 MB
push.commit, serial (120k) +80 MB +46 MB
commit.unit, new collection (30k–120k a unit) +70–77 MB +21–37 MB
commit.unit, existing tree — +22–58 MB
push.compact +25 MB +13–28 MB

Add the Worker's own baseline and the isolate LRU (sharedLru, up to 32 MB), and the old numbers sit at or over 128 MB.

What changed:

  • refs/log.ts: events spill to sorted runs at 4 MB of event JSON instead of every 100,000 events. 100k events plus the spill's copy came to about 100 MB.
  • push/runs.ts: mergeRuns' prefetch budget now covers both spans a cursor holds (current and next), 4 MB in all. A two-run unit held 16 MB of spans.
  • RepoSink: drain() also waits while the raw body bytes in flight pass pendingBytes (16 MB default), and while more than one spill is compressing. Each compression holds its raw text. 16 leaves of Crossref records came to tens of MB.
  • Leaf spill size 4 MB → 1 MB (repo/build.ts, push/parallel.ts). This changes only how a body object splits into gzip members; tree, leaf and version hashes are unchanged. The measured runs produced identical version hashes before and after.

Tests: protocol 205/205 (new: drain holds while bodies in flight pass the byte bound), server 227/227, typecheck, oxlint and oxfmt clean.

Not in this PR, filed in meta planning/kf/underlay/_open.md (2026-10-15):

  • A parallel commit whose units die never settles. Lost units are requeued only when another unit finishes, so the session stays committing forever.
  • Diff pages up to limit=5000 are built in memory; with Crossref records, over about 500 runs out of memory.
  • Commit units are sized by entry count (1M), not bytes.

…unts both held spans, the repo sink caps bodies in flight by bytes and waits for compressions, leaves spill at 1 MB, refs events spill at 4 MB instead of 100k events
@isTravis
isTravis marked this pull request as ready for review October 15, 2026 11:58
@isTravis
isTravis merged commit 1c5e199 into main Oct 15, 2026
1 check passed
@isTravis
isTravis deleted the tfk/commit-memory branch October 15, 2026 11:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants