Skip to content

TC-2917: DuckDB disk headroom, spill limit and startup cache sweep - #603

Draft
samgbafa wants to merge 3 commits into
skgbafa/tc-2917-duckdb-fs-cleanupfrom
skgbafa/tc-2917-duckdb-disk-gate
Draft

samgbafa wants to merge 3 commits into
skgbafa/tc-2917-duckdb-fs-cleanupfrom
skgbafa/tc-2917-duckdb-disk-gate

Conversation

@samgbafa

@samgbafa samgbafa commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

TC-2917. DuckDB disk and spill admission, plus a startup sweep of cache leftovers. Stacked on #601. Chunked artifact storage stays off by default.

What changes

  • Writes. A write is any request that can change the database: executes, batches, imports, and any query that is not purely plain queries and EXPLAIN without ANALYZE (an admin's PREPARE … AS INSERT/EXECUTE, CALL, CREATE SCHEMA count as writes). SQL that fails to parse or is refused also counts, so below the floor it answers 507 rather than 400/403. Nothing is classified when no disk limit is set.
  • Headroom floor before a write. A DuckDB write reserves the size of its local database files before it runs; an import reserves twice its size. The bound is the snapshot copy its save can take. Reservations share the node-wide disk budget with SQL. A reservation that would leave less than storage.duckdb.min_free_bytes free is refused with 507 duckdb_disk_headroom, and nothing runs. If free space cannot be read, the write is also refused (500), never allowed.
  • While a write runs. A background check reads free space every interval. While free space is below what other operations hold plus the floor (or cannot be read), it interrupts the connection on every reading until the statement returns; DuckDB clears an interrupt when its next query starts. An interrupted write answers 507 duckdb_disk_headroom.
  • Nothing from a refused write is kept. On a headroom refusal the database's local copy is dropped and reloaded from its last saved artifact on the next request. This covers an explicit COMMIT that committed before DuckDB reported the interrupt, and the earlier statements of a non-transactional batch.
  • Spill limit. Connections are opened with max_temp_directory_size (storage.duckdb.max_temp_directory_size). A statement that spills past it answers 507 duckdb_disk_headroom. Read-only queries and exports reserve nothing and are not interrupted for headroom; only the spill limit applies to them.
  • Defaults. Both limits key on storage.sql.chunked_artifacts. With it enabled, the floor defaults to 2 GiB and the spill limit to 1 GiB per connection. The 2 GiB floor is provisional until the node_storage_headroom event shows real production headroom. With chunked artifacts off, both limits stay off unless they are set.
  • Startup check. With chunked artifacts enabled, the node refuses to start when the DuckDB floor is below storage.sql.chunked_artifact_max_bytes, since a write's save copies the database once more outside its reservation; so the floor cannot be 0 while chunked artifacts are on.
  • Refusal counter. Each DuckDB headroom refusal is counted as tinycloud_disk_headroom_refusals_total{engine="duckdb"}.
  • Hydration is not gated: it writes at most the artifact ceiling, and that space is freed at eviction.
  • Startup sweep. Before the node serves, it removes what a stopped process left in the DuckDB cache: spill directories (<db>.duckdb.tmp) and staging or renamed-aside names (<db>.duckdb.tmp.<pid>.<n>, <db>.duckdb.wal.tmp.<pid>.<n>). It only looks inside real space directories, never through a link. The cache files themselves stay, because a legacy database reopens them. This assumes one node process per cache directory, which is how every deployment runs.

Tests (new)

  • duckdb_write_short_of_headroom_is_refused_before_it_runs
  • duckdb_reservation_counts_other_databases_in_flight
  • duckdb_statement_is_stopped_when_the_disk_runs_short_while_it_runs: the stored revision is unchanged afterwards.
  • duckdb_statement_past_the_spill_limit_is_the_disk
  • duckdb_startup_sweep_removes_leftovers_and_keeps_cache_files, duckdb_startup_sweep_does_not_follow_a_linked_space
  • headroom::postgres_duckdb_query_shaped_write_is_held_to_the_floor
  • headroom::duckdb_headroom_refusal_keeps_nothing_from_the_request: earlier batch statements are kept neither locally nor after the next saved write.
  • headroom::postgres_duckdb_batch_refused_around_commit_is_not_kept: refusals placed at 10 points around a transactional batch's COMMIT; a refused batch is never kept and an accepted one never lost.
  • headroom::duckdb_shortfall_between_calls_still_stops_the_next_statement
  • headroom::postgres_duckdb_prepared_insert_is_held_to_the_floor: PREPARE and EXECUTE are refused at 0 free.
  • config::tests::chunked_artifacts_require_a_duckdb_floor_of_one_artifact_ceiling: default/explicit/0 floors around the ceiling, and flag off.
  • headroom::postgres_duckdb_write_reserves_its_database_size: with another operation holding room, a write is refused; after release it runs; nothing stays reserved.

Throwaway mutations, each caught: no discard on refusal (2 tests); queries never counted as writes; watcher stops after its first trip; sweep follows a linked space; reservation always 0; PREPARE/EXECUTE treated as reads; startup floor check removed.

Writes reserve their local file size against the node-wide disk budget
and are refused (507 duckdb_disk_headroom) below storage.duckdb.min_free_bytes;
a running write is interrupted if free space drops below the floor; a
statement spilling past storage.duckdb.max_temp_directory_size is the
same 507. Defaults (2 GiB floor, 1 GiB spill) apply only with chunked
artifacts enabled. At startup the node removes DuckDB spill directories
and staging or renamed-aside cache names left by a stopped process.
…d by the parser

Review of #603: a write refused for headroom drops the database's local
copy, so a COMMIT interrupted after it committed (or earlier statements of
a batch) is never kept or saved later; a query whose SQL writes reserves
and runs under the watcher; the watcher keeps interrupting while free
space is short; the startup sweep does not descend into linked space
directories; with chunked artifacts the node refuses a DuckDB floor below
the artifact ceiling. Tests adopted from the review.
…eck test

Review of #603 (32cdd06): a query counts as read-only for the disk floor
only when every statement is a plain query or an EXPLAIN that does not
run its statement, so an admin's PREPARE/EXECUTE is held to the floor;
requests are not classified when no disk limit is set; the limit defaults
key on the configured flag, like the startup check, which now has a test;
docs say the floor cannot be 0 while parts are on and that refused SQL
answers 507 below the floor.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant