Skip to content

Stream cumulative SQL WAL through bounded artifact parts - #589

Draft
samgbafa wants to merge 1 commit into
mainfrom
skgbafa/tc-2917-chunked-wal
Draft

samgbafa wants to merge 1 commit into
mainfrom
skgbafa/tc-2917-chunked-wal

Conversation

@samgbafa

@samgbafa samgbafa commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

SQL databases using artifact parts now persist a cumulative WAL without rewriting their checkpoint after each mutation. The actor hands off an owned WAL file; each payload statement carries at most 1 MiB. A short conditional publication advances the revision and WAL pointer while retaining the checkpoint. Full checkpoints replace both parts when needed. Details: TC-2917.

Stacked on #582; this PR targets its branch until that prerequisite merges. The feature remains off by default and this is not an enablement request.

  • Checkpoint plus WAL are checked against the artifact ceiling; owner quota and bounded replacement credit include staged and retained parts.
  • Reads verify both part CIDs, the combined logical CID, both pointers and the layout generation before exposing the cache. WAL publication requires the caller’s exact revision/checkpoint lineage.
  • Reclamation removes one chunk per statement. A full checkpoint retires both previous parts, releasing each part’s byte charge and overlap credit only after its last chunk is gone.
  • Legacy saves retain the existing single-statement cap and retry back-off. WAL transfers obey that cap per bounded statement.
  • One forward-only PostgreSQL migration extends the existing metadata guards and adds the second retired-part reference. Deploy WAL-capable readers everywhere before enabling; checkpoint-only readers fail closed on WAL layouts.

Validation (private HOME/TC_HOME, owned PostgreSQL 18.6, assigned slot, -j2/debug off): 18 targeted PostgreSQL tests pass, including SQL save→WAL→empty-cache restart with exact recovered bytes, unchanged checkpoint bytes, forced-checkpoint collection of both old parts, WAL corruption, stale publication, combined-size limits and quota accounting. Core/test and server Clippy pass with warnings denied. All 1,111 core tests pass with PostgreSQL configured. A deliberate omission of the WAL CID check fails the corruption assertion; source restored afterwards.

test-ruling: TC-2917 — Naomi approved updating this PR’s own checkpoint-only smoke to count checkpoint plus WAL chunks, assert a nonempty WAL and unchanged checkpoint ids/bytes, then force a checkpoint and prove both superseded parts are reclaimed. No other existing tests edited.

Remaining separate work: DuckDB chunk writes, existing-artifact conversion and ceiling-load acceptance. Flag stays off. No production actions.

@samgbafa
samgbafa force-pushed the skgbafa/tc-2917-chunked-artifacts branch 5 times, most recently from 5759367 to d033bc4 Compare November 13, 2026 22:44
@samgbafa
samgbafa force-pushed the skgbafa/tc-2917-chunked-wal branch from 1993c42 to aa76999 Compare November 14, 2026 01:41
@samgbafa
samgbafa changed the base branch from skgbafa/tc-2917-chunked-artifacts to main November 14, 2026 01:41

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant