Push any size as one version: paged run indexes, compaction pieces, byte-sized units, streamed leaves, parallel schema and privacy changes, stalled-commit recovery, a bulk Worker

[isTravis]

Push a collection of any size as one version, over any number of upload requests, with every step of the push bounded in memory and resumable, on Workers. This follows #38 (merged), which bounded the refs index, the repo sink and run prefetch. Loading a large bibliographic corpus on staging (2026-10-14/15), with records of several KB each, kept running out of memory: a 5M-record commit split into 34 units, and 14 of them hit the 128 MB isolate limit on every attempt. The session then sat in committing forever.

The design was already right: uploads are sorted runs in the internal bucket, merged during upload and committed in key-range units. These are the places that still held, in one isolate, memory proportional to the push or to a leaf:

Where	Before	Now
Commit planning	read every run's full block index (~500k entries for 5M records of ~6.5 KB)	reads run summaries (one entry per 4 MB part); a part's blocks are a page read only by a cursor that reaches it
Unit size	1M entries (≈ 6.5 GB at ~6.5 KB a record)	bytes of work: changes' raw bytes plus the base leaves they will rewrite (256 MB)
Parallel threshold	2M entries (so 100k-record versions committed serially, 494 s)	256 MB of estimated work
Compaction	count-based tiers up to tier 2; one job per compaction (a top-tier compaction ≈ 4 GB)	no top tier; a compaction over 512 MB splits into key-range pieces, each its own job
Runs a commit reads	unbounded (every upload, if compaction fell behind)	at most 32: the commit first compacts down (phase: compacting), and the last compaction queues it again
Plan rows	every unit's row and slice written in the planning job (D1: ~1,000 queries per invocation)	the plan is one object; rows and slices are written by commit.plan jobs, 1,000 units each
Rewriting a base leaf	whole body decompressed into strings (avg ~6.6 MB, up to ~53 MB with ~6.5 KB records, plus copies)	streamed through zlib's streaming gunzip, a line at a time (merge, assemble, single-record reads); confirmed on workerd: multi-member, in chunks
Diff pages	up to 5,000 records built in memory, each body by decoding its whole leaf	capped at 4 MB of records, each leaf the page touches read once, streamed
A lost or failing step	a unit was requeued only when another unit finished; the session never left committing	a sweep (every 10 min) re-queues stale compactions, plan chunks, units and assemblies; a unit queued 4 times, or a commit that makes no progress through 3 nudges, fails with a reason
Isolates	bulk jobs shared isolates with 16 MB upload batches and page renders, under one 32 MB cache	bulk jobs run in their own Worker (wrangler.bulk.jsonc) with an 8 MB cache; uploads are 8 MB batches
Schema change	revalidated every record of the type in one serial job: a type past a few million large records couldn't change its schema at all	a change that can't reject a record (schemaRelaxes) isn't revalidated; any other is revalidated in parallel jobs after assembly
Privacy change, type removal	serial	parallel where the serial build moves the type's tree as it is

One version from many uploads

• A session's idle timeout is 12 hours (was 1), so a client can stop and pick the same session up by its id.
• GET …/push/:sid now reports progress: runs, run_bytes and runs_compacting while open; then phase (compacting, planning, units, assembling) and units done of total.
• A client loads a large corpus by opening one session, uploading in batches (over days if need be), and committing once. The bulk loader in knowledgefutures/underlay-scrapers does this in Simple schema sharing #3.

Schema and privacy changes

• Relaxations aren't revalidated. schemaRelaxes (protocol) is true when the new schema differs only by required lists losing names, annotations (title, description, $comment), or the root private flag. A record valid before stays valid, so the serial build, the parallel planner and the async check all skip revalidation for these changes. Only positions where loosening can only accept more count: under not, oneOf, if and definitions/$defs everything must stay equal, and a schema with a $ref gets the annotation allowances only.

• Other schema changes revalidate in parallel. After assembly, the session goes to phase: validating, and commit.validate jobs each stream about 256 MB of the new tree's records. commit.publish then publishes, or fails the session with the same Schema validation failed 422 the serial commit gives (first 100 errors, total count). Revalidation counts as work in the planner, so a small push that changes a large type's schema goes parallel too. The sweep resumes this phase like the others.

• Privacy changes and removals go parallel where the serial build has a plain tree move and the base counts file references per type:

  • public → private, with nothing in the private set yet;
  • private → public, with no private upsert of the type in the push;
  • removing a type.

  Units merge into the moved tree, and the publish moves the type's file references and counts as the serial build does. The rest stays serial, with the type's size counted, and past ~2 GB it's refused with a 422 saying to push that change on its own: a privacy change with records already in both sets, private records pushed as the type turns public, or a removal from a base without per-type counts.

• test/schema-parallel.test.ts commits each case both ways, on two collections with the same base, and checks the same sets, counts, file sets and errors.

Measured

scripts/memory.ts runs every job in-process on synthetic bibliographic records (about 10 KB each, non-ASCII text, long reference lists). It reports each job's peak live memory: JS heap plus array buffers, after a forced GC, above where the job started.

The run: 60,000 records (614 MB) in one version, then 15,000 updates spread over all of them, so every base leaf is rewritten. Production sizes, except 64 MB compaction pieces (so pieces run) and a 4-run commit limit (so the commit compacts first):

Job	Peak (heap + buffers)
commit.unit (8)	40.7 MB (0.9 + 39.8)
refs.index (2)	35.4 MB
push.compactPiece (14)	30.2 MB
upload request (98, 8 MB batches)	25.3 MB
commit.assemble, commit.plan, push.commit, push.compact	< 4 MB

Every job is under a third of an isolate. Array-buffer readings vary between runs (Node frees them lazily); before the batch, part and sink changes in the last commits, the same run read 52 MB for uploads and 64 MB for units. test/memory.test.ts runs the script on ~40 MB in CI and fails if any job's JS heap passes 16 MB (32 MB for an upload).

Retries, cleanup and reads

• A failed commit can be committed again in the same session. This applies when the failure was the server's: 500, a step that kept failing or stopped moving; or 503, storage cleanup interfered. The uploads are still in storage, so POST …/commit plans afresh from them instead of the client uploading everything again. A refusal the client has to fix (422 invalid records, 409 base moved) answers 409 with the cause. Retries are allowed until an hour before cleanup would delete the session's uploads (about a day after it would have expired).
• Storage cleanup holds for a long commit while it makes progress. Before, a deletion window waited only for commits started less than 6 hours ago, so a long commit could be cut off. Now a commit that recorded progress within the hold counts too, and the stalled-commit sweep fails one that stops.
• Shared reads. Simultaneous readers of one node or leaf body fetch and decompress it once.
• Diff pages pass records through as text. A stored record line is already {"id","type","data"}, so it's copied into the page rather than parsed and serialised again.

Compatibility

• Run indexes written before this ({seq, tier, blocks}) and slices of plans already in flight still read: the blocks become inline pages. The protocol, tree shapes and version hashes are unchanged.
• batch_bytes advertised drops from 16 MiB to 8 MiB. Clients read it from the session: PubPub's delta push and the scrapers do.

Deploy (in this order)

1. wrangler d1 migrations apply underlay-staging --env staging --remote (0024: run bytes and claims, commit phase and progress, unit attempts).
2. wrangler deploy --env staging. The main Worker no longer consumes the bulk queue, so bulk jobs wait on it until step 4.
3. Bulk Worker secrets, the same values as the main Worker's: wrangler secret put <NAME> -c wrangler.bulk.jsonc --env staging for R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, SIGNING_KEY, LOCATION_KEY.
4. wrangler deploy -c wrangler.bulk.jsonc --env staging (creates underlay-staging-bulk, the bulk queue's consumer).

To hold the Worker split back, deploy without 0d244f1: everything else works with one Worker.

Not done / not checked

• Not run on Workers or staging: the measurements are Node, against a file store. Workers' runtime is the same V8, and workerd's streaming gunzip was checked separately.
• The largest run was 614 MB. At ~1 TB, compaction reaches more tiers, which is tested only at small sizes by forcing the limits down. Planning at that size is measured synthetically (scripts/plan-memory.ts: 32 runs, 50,000 parts): 8.4 MB of summaries, 12.6 MB in memory after splitting into 4,207 units, about 8 KB per unit's slice.
• A delete in the same push as a type's privacy move is dropped, on both paths, as the serial build always has. The parallel path keeps that for parity; it's worth fixing on its own.

Tests: server 246/246 (new: stream-push, recover, memory, schema-parallel, byte-capped and passed-through diff pages), protocol 214/214 (streaming gunzip, leaf streams, shared reads, schemaRelaxes), typecheck, oxlint and oxfmt clean. Both Workers build (wrangler deploy --dry-run).

🤖 Generated with Claude Code