Skip to content

Preserve DuckDB main-file changes before acknowledging writes - #594

Draft
samgbafa wants to merge 2 commits into
mainfrom
skgbafa/tc-2990-duckdb-wal-base
Draft

samgbafa wants to merge 2 commits into
mainfrom
skgbafa/tc-2990-duckdb-wal-base

Conversation

@samgbafa

Copy link
Copy Markdown
Contributor

Large file-backed DuckDB inserts can write new blocks into the main file while the WAL records references to them. Saving that WAL beside an older checkpoint cannot recover the acknowledged data after eviction or restart. DuckDB now uses the WAL path only when the local main file's length and Blake3 CID match the stored checkpoint; otherwise it saves a complete checkpoint. Details: TC-2990.

The identity check and WAL copy run under the actor fence. Hashing uses a 64 KiB buffer; the existing revision/checkpoint publication guard remains intact. Unchanged-file small writes keep incremental persistence.

Validation:

  • RED on main 8802d3bb: an acknowledged 300,000-row insert fails fresh-cache recovery with Could not read enough bytes.
  • GREEN: small → 300,000-row → small writes recover exact counts, IDs and body checks from fresh caches, then export after eviction succeeds. Wrong-length/wrong-CID checks refuse WAL without mutating it.
  • All 47 DuckDB module tests and 15 diagnostic/telemetry integration tests pass; core/test Clippy passes with warnings denied. Existing tests unchanged.

@samgbafa
samgbafa force-pushed the skgbafa/tc-2990-duckdb-wal-base branch from 441b13b to 56a0a69 Compare November 13, 2026 23:47
…AL fast path and same-length reuse

Review of #594: with no recorded lineage a WAL is attached only to the
checkpoint it was verified against; new tests pin that repeated small
writes stay on the WAL path and that a same-length rewrite of freed
blocks recovers exact rows.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant