Skip to content

Persist DuckDB files through bounded artifact parts - #591

Draft
samgbafa wants to merge 1 commit into
skgbafa/tc-2917-chunked-walfrom
skgbafa/tc-2917-chunked-duckdb
Draft

samgbafa wants to merge 1 commit into
skgbafa/tc-2917-chunked-walfrom
skgbafa/tc-2917-chunked-duckdb

Conversation

@samgbafa

Copy link
Copy Markdown
Contributor

DuckDB databases can now use the opt-in PostgreSQL artifact-part layout for checkpoints, cumulative WAL and imports. Actors hand off owned files; persistence streams payload statements of at most 1 MiB through the existing quota, integrity and conditional-publication path. Details: TC-2917.

Stacked on #589 (which depends on #582). The feature stays off by default. No activation or deployment is requested.

  • File-backed actors checkpoint/copy under their actor fence; smaller cumulative WAL files are copied without changing the base checkpoint. New chunk-enabled actors start file-backed at either memory threshold; closing/idle eviction removes their local cache after the connection closes. Legacy flag-off behavior is unchanged.
  • The existing artifact ceiling covers checkpoint plus WAL and the owner quota covers SQL and DuckDB together, including staged and retained parts. Non-growing checkpoints retain the existing growth checks.
  • Imports stream their already-buffered request into the same part writer. A replaced actor is closed before its files are replaced. Explicit export responses keep their existing format.
  • No schema change beyond the prerequisite PRs. No legacy-path layout probes are added.

Validation on private HOME/TC_HOME and owned PostgreSQL 18.6, assigned build target, -j2/debug off:

  • Three real PostgreSQL/DuckDB smoke tests pass: multi-chunk save, cumulative WAL, empty-cache recovery with a new flag-off repository, full-checkpoint reclamation, live-actor import and quota refusal with local-state discard. Covers both configured memory thresholds, with file-backed birth, explicit shutdown, idle eviction and subsequent recovery.
  • Core/test Clippy with DuckDB enabled passes with warnings denied. DuckDB-enabled core/test and server Clippy pass with warnings denied. All 1,160 DuckDB-enabled core tests pass with PostgreSQL configured.
  • No existing tests edited.

Remaining enabling work includes the separate predecessor-reclamation/contention and post-commit cleanup regression delta, existing-artifact conversion and ceiling-load acceptance.

Local disk: each successfully persisted open database uses up to its configured artifact ceiling for checkpoint plus cumulative WAL; a full checkpoint also holds an owned temporary copy. Operators must check CVM disk headroom before enabling. The enabling acceptance still needs to cover transient oversized native writes and snapshot copies before size refusal; this PR does not claim a hard pre-execution disk reservation.

@samgbafa
samgbafa force-pushed the skgbafa/tc-2917-chunked-wal branch from 1993c42 to aa76999 Compare November 14, 2026 01:41
@samgbafa
samgbafa force-pushed the skgbafa/tc-2917-chunked-duckdb branch from b793045 to c251b72 Compare November 14, 2026 01:41

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant