Repository navigation
feat(TC-2917): stream SQL checkpoints through PostgreSQL parts - #582
Merged
Merged
Conversation
samgbafa
force-pushed
the
skgbafa/tc-2917-chunked-artifacts
branch
from
November 13, 2026 16:09
2cbb785 to
f1d08cb
Compare
This was referenced Nov 13, 2026
samgbafa
force-pushed
the
skgbafa/tc-2917-chunked-artifacts
branch
from
November 13, 2026 17:55
f1d08cb to
9e5918c
Compare
samgbafa
force-pushed
the
skgbafa/tc-2917-chunked-artifacts
branch
3 times, most recently
from
November 13, 2026 22:24
d697880 to
5759367
Compare
…e a transfer slot Rebased onto TC-2961 (#585): a database stored in parts streams in bounded statements, so the whole-artifact read cap and large-transfer permit apply only to the single-row layout. A sink that keeps the whole artifact in memory (MemorySink, the DuckDB export) still takes a large-transfer slot for a parts read over 8 MiB.
samgbafa
force-pushed
the
skgbafa/tc-2917-chunked-artifacts
branch
from
November 13, 2026 22:44
5759367 to
d033bc4
Compare
samgbafa
marked this pull request as ready for review
November 13, 2026 23:04
This was referenced Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SQL persistence can opt into PostgreSQL checkpoint parts through
storage.sql.chunked_artifacts(default false). The actor returns an owned snapshot file, payload statements carry at most 1 MiB, and a metadata-only conditional update publishes a sealed checkpoint. A fresh cache verifies each part's Blake3 CID, revision, pointer and layout generation before opening the engine. Details: TC-2917.Explicit artifact/account capacities are required. Exact space ownership is indexed and owner accounting applies to SQL, DuckDB and legacy writers after activation. Full staged payloads remain accounted, with one bounded overlapping replacement per database. The overlap credit is the smaller of current and staged sizes; a pending shrink provides no capacity to another database. Database identity foreign keys and a partial unique index scope replacements. New nodes can reopen and continue an existing parts layout even with the switch off, using its persisted capacities.
Aborts, lease-expired uploads, superseded checkpoints and deleted artifacts use a fenced deleting state. Each deletion statement removes at most one payload chunk; charges and replacement credits release when reclamation completes. Current references are checked under the same owner/part locks as publication. Database layout guards atomically refuse legacy replacement of a published parts layout. Before activation, quota triggers stay disabled and legacy operations issue no extra client layout probes.
This remains an OFF-by-default first-slice draft, not an enablement or deployment request. It publishes full SQL checkpoints. WAL-part persistence, DuckDB chunk writes, existing-artifact conversion, grant separation, snapshot disk budgeting and ceiling-load acceptance remain follow-up work. Operational inventory and configuration:
docs/operations/chunked-sql-artifacts.md. Rebased onto main including the single-statement save limit and checkpoint retry back-off. Legacy saves retain that limit; streamed checkpoints apply it to each bounded payload statement.Validation on an owned disposable PostgreSQL 18.6, private HOME/TC_HOME, assigned target, -j2/debug off:
test-ruling: TC-2917 — Naomi approved updating this PR's new smoke test to require exactly ceil(current checkpoint size / 1 MiB) stored chunks, plus explicit absence of superseded chunks. No baseline tests edited. Migration lock errors distinguish the boot default from shorter migration-specific limits.
Rebased onto TC-2961 (#585). #585 added a whole-artifact read limit (
storage.sql.max_artifact_read_bytes, 64 MiB) and a cap of two concurrent large transfers per node. Both sit where this PR adds the parts layout. How the two combine:Apart from these resolutions, every +/- line of the earlier four commits is unchanged against the previously reviewed head 9e5918c.
New tests: a parts database over 64 MiB loads while a single-row database over 64 MiB is still refused with the read limit; and a parts read into memory waits for a large-transfer slot while a streaming read does not.
Rebased onto main 931f0d0 (#592, #593, #602 and #595 merged). Only CHANGELOG conflicted; save() takes #592's single-statement form unchanged, with #585's transfer slot and start mark before the writer lock, then the parts-layout write check. No +/- line of this PR differs from ff4309b's.
Validation on 5f2f0ba: core with
duckdb1171/1171, clippy core and node, fmt pass. Earlier validation on 822ed70 (local PostgreSQL 18.6, private HOME, -j2):cargo test -p tinycloud-node --no-fail-fast): 1236 passed, 5 failed. All 5 failures come from the local environment, not this change: two PostgreSQL renewal harnesses require PostgreSQL 16 (CI runs 16), and three startup-identity tests refuse to start because this host has a/var/run/dstack.sockentry.cargo fmt --checkpasses.Flag remains off. No merge, production access or production actions.