Repository navigation
Conversation
Writes reserve their local file size against the node-wide disk budget and are refused (507 duckdb_disk_headroom) below storage.duckdb.min_free_bytes; a running write is interrupted if free space drops below the floor; a statement spilling past storage.duckdb.max_temp_directory_size is the same 507. Defaults (2 GiB floor, 1 GiB spill) apply only with chunked artifacts enabled. At startup the node removes DuckDB spill directories and staging or renamed-aside cache names left by a stopped process.
…d by the parser Review of #603: a write refused for headroom drops the database's local copy, so a COMMIT interrupted after it committed (or earlier statements of a batch) is never kept or saved later; a query whose SQL writes reserves and runs under the watcher; the watcher keeps interrupting while free space is short; the startup sweep does not descend into linked space directories; with chunked artifacts the node refuses a DuckDB floor below the artifact ceiling. Tests adopted from the review.
samgbafa
force-pushed
the
skgbafa/tc-2917-duckdb-fs-cleanup
branch
from
November 14, 2026 01:41
80729de to
327b09c
Compare
samgbafa
force-pushed
the
skgbafa/tc-2917-duckdb-disk-gate
branch
from
November 14, 2026 01:41
f4bd6d6 to
32cdd06
Compare
…eck test Review of #603 (32cdd06): a query counts as read-only for the disk floor only when every statement is a plain query or an EXPLAIN that does not run its statement, so an admin's PREPARE/EXECUTE is held to the floor; requests are not classified when no disk limit is set; the limit defaults key on the configured flag, like the startup check, which now has a test; docs say the floor cannot be 0 while parts are on and that refused SQL answers 507 below the floor.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TC-2917. DuckDB disk and spill admission, plus a startup sweep of cache leftovers. Stacked on #601. Chunked artifact storage stays off by default.
What changes
EXPLAINwithoutANALYZE(an admin'sPREPARE … AS INSERT/EXECUTE,CALL,CREATE SCHEMAcount as writes). SQL that fails to parse or is refused also counts, so below the floor it answers 507 rather than 400/403. Nothing is classified when no disk limit is set.storage.duckdb.min_free_bytesfree is refused with507 duckdb_disk_headroom, and nothing runs. If free space cannot be read, the write is also refused (500), never allowed.507 duckdb_disk_headroom.max_temp_directory_size(storage.duckdb.max_temp_directory_size). A statement that spills past it answers507 duckdb_disk_headroom. Read-only queries and exports reserve nothing and are not interrupted for headroom; only the spill limit applies to them.storage.sql.chunked_artifacts. With it enabled, the floor defaults to 2 GiB and the spill limit to 1 GiB per connection. The 2 GiB floor is provisional until thenode_storage_headroomevent shows real production headroom. With chunked artifacts off, both limits stay off unless they are set.storage.sql.chunked_artifact_max_bytes, since a write's save copies the database once more outside its reservation; so the floor cannot be 0 while chunked artifacts are on.tinycloud_disk_headroom_refusals_total{engine="duckdb"}.<db>.duckdb.tmp) and staging or renamed-aside names (<db>.duckdb.tmp.<pid>.<n>,<db>.duckdb.wal.tmp.<pid>.<n>). It only looks inside real space directories, never through a link. The cache files themselves stay, because a legacy database reopens them. This assumes one node process per cache directory, which is how every deployment runs.Tests (new)
duckdb_write_short_of_headroom_is_refused_before_it_runsduckdb_reservation_counts_other_databases_in_flightduckdb_statement_is_stopped_when_the_disk_runs_short_while_it_runs: the stored revision is unchanged afterwards.duckdb_statement_past_the_spill_limit_is_the_diskduckdb_startup_sweep_removes_leftovers_and_keeps_cache_files,duckdb_startup_sweep_does_not_follow_a_linked_spaceheadroom::postgres_duckdb_query_shaped_write_is_held_to_the_floorheadroom::duckdb_headroom_refusal_keeps_nothing_from_the_request: earlier batch statements are kept neither locally nor after the next saved write.headroom::postgres_duckdb_batch_refused_around_commit_is_not_kept: refusals placed at 10 points around a transactional batch's COMMIT; a refused batch is never kept and an accepted one never lost.headroom::duckdb_shortfall_between_calls_still_stops_the_next_statementheadroom::postgres_duckdb_prepared_insert_is_held_to_the_floor: PREPARE and EXECUTE are refused at 0 free.config::tests::chunked_artifacts_require_a_duckdb_floor_of_one_artifact_ceiling: default/explicit/0 floors around the ceiling, and flag off.headroom::postgres_duckdb_write_reserves_its_database_size: with another operation holding room, a write is refused; after release it runs; nothing stays reserved.Throwaway mutations, each caught: no discard on refusal (2 tests); queries never counted as writes; watcher stops after its first trip; sweep follows a linked space; reservation always 0; PREPARE/EXECUTE treated as reads; startup floor check removed.