Repository navigation
Bound commit and refs-index memory for large records - #38
Merged
Merged
Conversation
…unts both held spans, the repo sink caps bodies in flight by bytes and waits for compressions, leaves spill at 1 MB, refs events spill at 4 MB instead of 100k events
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Crossref load into
crossref-demo/workson staging (2026-10-14/15) kept hittingexceededMemoryonunderlay-staging-bulk. A 5M-record commit's units died on every attempt, and with 100k-record versions the bulk queue still showed memory errors. Crossref works average 6.5 KB (up to 148 KB), much larger than anything the commit path was tested with.Measured locally on Node by pushing real Crossref sample records through the real jobs, with a file store, no cache, and live memory sampled after forced GC (peak above the job's starting point):
refs.index(120k–250k records)push.commit, serial (120k)commit.unit, new collection (30k–120k a unit)commit.unit, existing treepush.compactAdd the Worker's own baseline and the isolate LRU (
sharedLru, up to 32 MB), and the old numbers sit at or over 128 MB.What changed:
refs/log.ts: events spill to sorted runs at 4 MB of event JSON instead of every 100,000 events. 100k events plus the spill's copy came to about 100 MB.push/runs.ts:mergeRuns' prefetch budget now covers both spans a cursor holds (current and next), 4 MB in all. A two-run unit held 16 MB of spans.RepoSink:drain()also waits while the raw body bytes in flight passpendingBytes(16 MB default), and while more than one spill is compressing. Each compression holds its raw text. 16 leaves of Crossref records came to tens of MB.repo/build.ts,push/parallel.ts). This changes only how a body object splits into gzip members; tree, leaf and version hashes are unchanged. The measured runs produced identical version hashes before and after.Tests: protocol 205/205 (new:
drainholds while bodies in flight pass the byte bound), server 227/227, typecheck, oxlint and oxfmt clean.Not in this PR, filed in meta
planning/kf/underlay/_open.md(2026-10-15):committingforever.limit=5000are built in memory; with Crossref records, over about 500 runs out of memory.