Skip to content

feat(TC-2917): stream SQL checkpoints through PostgreSQL parts - #582

Merged
samgbafa merged 5 commits into
mainfrom
skgbafa/tc-2917-chunked-artifacts
Nov 13, 2026
Merged

samgbafa merged 5 commits into
mainfrom
skgbafa/tc-2917-chunked-artifacts

Conversation

@samgbafa

@samgbafa samgbafa commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

SQL persistence can opt into PostgreSQL checkpoint parts through storage.sql.chunked_artifacts (default false). The actor returns an owned snapshot file, payload statements carry at most 1 MiB, and a metadata-only conditional update publishes a sealed checkpoint. A fresh cache verifies each part's Blake3 CID, revision, pointer and layout generation before opening the engine. Details: TC-2917.

Explicit artifact/account capacities are required. Exact space ownership is indexed and owner accounting applies to SQL, DuckDB and legacy writers after activation. Full staged payloads remain accounted, with one bounded overlapping replacement per database. The overlap credit is the smaller of current and staged sizes; a pending shrink provides no capacity to another database. Database identity foreign keys and a partial unique index scope replacements. New nodes can reopen and continue an existing parts layout even with the switch off, using its persisted capacities.

Aborts, lease-expired uploads, superseded checkpoints and deleted artifacts use a fenced deleting state. Each deletion statement removes at most one payload chunk; charges and replacement credits release when reclamation completes. Current references are checked under the same owner/part locks as publication. Database layout guards atomically refuse legacy replacement of a published parts layout. Before activation, quota triggers stay disabled and legacy operations issue no extra client layout probes.

This remains an OFF-by-default first-slice draft, not an enablement or deployment request. It publishes full SQL checkpoints. WAL-part persistence, DuckDB chunk writes, existing-artifact conversion, grant separation, snapshot disk budgeting and ceiling-load acceptance remain follow-up work. Operational inventory and configuration: docs/operations/chunked-sql-artifacts.md. Rebased onto main including the single-statement save limit and checkpoint retry back-off. Legacy saves retain that limit; streamed checkpoints apply it to each bounded payload statement.

Validation on an owned disposable PostgreSQL 18.6, private HOME/TC_HOME, assigned target, -j2/debug off:

  • Sixteen targeted PostgreSQL tests pass, including cleanup isolation, over-quota non-growing replacements, multi-chunk abort accounting, concurrent legacy writes, statement limits and refusal back-off: save/restart/byte-exact recovery, existing-layout writes with the switch off, exact-owner and DuckDB quota, owner concurrency/integrity, conditional-publication race, cancellation, repeated replacement/deletion, at-quota shrink/growth and exclusivity, shrink abort/crash recovery, deletion during staging, and per-chunk reclamation accounting.
  • Core and server library Clippy pass with warnings denied; server wiring is included.
  • Conditional-publication, early reservation release and unlocked legacy-quota mutations each fail their regression test; implementations restored afterwards.
  • Full rebased core suite: 1,109 passed, 0 failed, 0 ignored, with local PostgreSQL configured.
  • rustfmt and diff whitespace checks pass.

test-ruling: TC-2917 — Naomi approved updating this PR's new smoke test to require exactly ceil(current checkpoint size / 1 MiB) stored chunks, plus explicit absence of superseded chunks. No baseline tests edited. Migration lock errors distinguish the boot default from shorter migration-specific limits.

Rebased onto TC-2961 (#585). #585 added a whole-artifact read limit (storage.sql.max_artifact_read_bytes, 64 MiB) and a cap of two concurrent large transfers per node. Both sit where this PR adds the parts layout. How the two combine:

New tests: a parts database over 64 MiB loads while a single-row database over 64 MiB is still refused with the read limit; and a parts read into memory waits for a large-transfer slot while a streaming read does not.

Rebased onto main 931f0d0 (#592, #593, #602 and #595 merged). Only CHANGELOG conflicted; save() takes #592's single-statement form unchanged, with #585's transfer slot and start mark before the writer lock, then the parts-layout write check. No +/- line of this PR differs from ff4309b's.

Validation on 5f2f0ba: core with duckdb 1171/1171, clippy core and node, fmt pass. Earlier validation on 822ed70 (local PostgreSQL 18.6, private HOME, -j2):

  • Core library with DuckDB features: 1166/1166, including the two new tests.
  • Node tests (cargo test -p tinycloud-node --no-fail-fast): 1236 passed, 5 failed. All 5 failures come from the local environment, not this change: two PostgreSQL renewal harnesses require PostgreSQL 16 (CI runs 16), and three startup-identity tests refuse to start because this host has a /var/run/dstack.sock entry.
  • Clippy with warnings denied passes for core (with DuckDB) and node, and cargo fmt --check passes.

Flag remains off. No merge, production access or production actions.

@samgbafa
samgbafa force-pushed the skgbafa/tc-2917-chunked-artifacts branch from 2cbb785 to f1d08cb Compare November 13, 2026 16:09
@samgbafa
samgbafa force-pushed the skgbafa/tc-2917-chunked-artifacts branch from f1d08cb to 9e5918c Compare November 13, 2026 17:55
@samgbafa samgbafa closed this Nov 13, 2026
@samgbafa samgbafa reopened this Nov 13, 2026
@samgbafa
samgbafa force-pushed the skgbafa/tc-2917-chunked-artifacts branch 3 times, most recently from d697880 to 5759367 Compare November 13, 2026 22:24
…e a transfer slot

Rebased onto TC-2961 (#585): a database stored in parts streams in
bounded statements, so the whole-artifact read cap and large-transfer
permit apply only to the single-row layout. A sink that keeps the whole
artifact in memory (MemorySink, the DuckDB export) still takes a
large-transfer slot for a parts read over 8 MiB.
@samgbafa
samgbafa force-pushed the skgbafa/tc-2917-chunked-artifacts branch from 5759367 to d033bc4 Compare November 13, 2026 22:44
@samgbafa
samgbafa marked this pull request as ready for review November 13, 2026 23:04
@samgbafa
samgbafa merged commit 4416a51 into main Nov 13, 2026
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant