Push any size as one version: paged run indexes, compaction pieces, byte-sized units, streamed leaves, parallel schema and privacy changes, stalled-commit recovery, a bulk Worker by isTravis · Pull Request #39 · knowledgefutures/underlay

Paged run indexes, compaction in key-range pieces and down to a bounded run count before commit, units sized by bytes with base-leaf cost, plans written in chunks

isTravis

committed

1b26c61

Stream leaf bodies when rewriting and assembling trees, and in single-record reads: zlib's streaming gunzip, a line at a time, no whole body in memory

isTravis

committed

f222ae8

Diff pages capped by record bytes, with each leaf a page touches read once, streamed

isTravis

committed

0d3fb7b

Recover stalled commits from the maintenance sweep: lost compactions and units go again, a unit out of attempts or a commit that stops moving fails with a reason

isTravis

committed

e1d07ef

Sessions idle out after 12 hours, and their status reports progress: runs, compactions, the commit's phase and units done

isTravis

committed

7857391

Bulk jobs in their own Worker (wrangler.bulk.jsonc): the bulk queue's consumer, no routes, a smaller isolate cache

isTravis

committed

0d244f1

Measure push jobs' memory on Crossref-sized records (scripts/memory.ts) and check it in CI; 8 MB upload batches, 4 MB run parts with one stored at a time, 512 MB compaction pieces

isTravis

committed

737adda

Protocol doc: one version from any number of uploads, the status route's progress, and underlay.org's new limits

isTravis

committed

4d47a1c

Sweep re-queues only compactions still wanted; a session reopened for missing files drops its commit phase

isTravis

committed

e04d8c6

Commit units keep eight repository writes in flight, not sixteen; test types

isTravis

committed

4b1f901

Describe the memory test's records by shape, not by source

isTravis

committed

fb87605

Schema changes that only relax skip revalidation; other schema changes revalidate in parallel jobs, and privacy moves and type removals commit in parallel, matching the serial build

isTravis

committed

4160333

Measure a commit's planning at full size from synthetic run summaries (scripts/plan-memory.ts)

isTravis

committed

46ce6a4

Share reads in progress: simultaneous readers of a node or leaf body fetch and decompress it once

isTravis

committed

2dc9885

Diff pages pass stored records through as text instead of parsing and re-serialising them

isTravis

committed

1842343

A commit that failed on the server's side can be committed again in the same session; storage cleanup holds for a commit while it makes progress, not only in its first six hours

isTravis

committed

e34c75b

A compaction's claim is dated afresh as its pieces finish, so the sweep leaves a long one alone; a run writer fills the next part while the previous one is stored

isTravis

committed

604d5bd

Navigation Menu

Repository navigation

Push any size as one version: paged run indexes, compaction pieces, byte-sized units, streamed leaves, parallel schema and privacy changes, stalled-commit recovery, a bulk Worker - #39

Push any size as one version: paged run indexes, compaction pieces, byte-sized units, streamed leaves, parallel schema and privacy changes, stalled-commit recovery, a bulk Worker#39
isTravis merged 17 commits into
mainfrom
streaming-push

Commits on Oct 15, 2026

Paged run indexes, compaction in key-range pieces and down to a bounded run count before commit, units sized by bytes with base-leaf cost, plans written in chunks

Stream leaf bodies when rewriting and assembling trees, and in single-record reads: zlib's streaming gunzip, a line at a time, no whole body in memory

Diff pages capped by record bytes, with each leaf a page touches read once, streamed

Recover stalled commits from the maintenance sweep: lost compactions and units go again, a unit out of attempts or a commit that stops moving fails with a reason

Sessions idle out after 12 hours, and their status reports progress: runs, compactions, the commit's phase and units done

Bulk jobs in their own Worker (wrangler.bulk.jsonc): the bulk queue's consumer, no routes, a smaller isolate cache

Measure push jobs' memory on Crossref-sized records (scripts/memory.ts) and check it in CI; 8 MB upload batches, 4 MB run parts with one stored at a time, 512 MB compaction pieces

Protocol doc: one version from any number of uploads, the status route's progress, and underlay.org's new limits

Sweep re-queues only compactions still wanted; a session reopened for missing files drops its commit phase

Commit units keep eight repository writes in flight, not sixteen; test types

Describe the memory test's records by shape, not by source

Schema changes that only relax skip revalidation; other schema changes revalidate in parallel jobs, and privacy moves and type removals commit in parallel, matching the serial build

Measure a commit's planning at full size from synthetic run summaries (scripts/plan-memory.ts)

Share reads in progress: simultaneous readers of a node or leaf body fetch and decompress it once

Diff pages pass stored records through as text instead of parsing and re-serialising them

A commit that failed on the server's side can be committed again in the same session; storage cleanup holds for a commit while it makes progress, not only in its first six hours

A compaction's claim is dated afresh as its pieces finish, so the sweep leaves a long one alone; a run writer fills the next part while the previous one is stored