Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@
- Policy/v3: minting a session and registering a policy no longer re-verify signatures or re-run the claim, presentation, credential admission and allow evaluations while holding their spaces' write order and row locks. Under the locks they check that the challenge, registration and roots are unchanged; those decisions were made before the locks, at the same time, from the same inputs. Refusal codes are unchanged, except that `credential-evaluation-changed` and `policy-evaluation-changed` can no longer occur. Details: TC-2986.
- Artifact-part writers prioritize their own predecessor reclamation, keep replacement contention retryable, and distinguish capacity refusals from space-growth limits; existing database layouts can checkpoint before adopting WAL parts. Details: TC-2917.
- DuckDB cache cleanup no longer holds up lookups of other databases while files are deleted; a spill directory left by a stopped process no longer blocks hydration or discard, and a failed import leaves no staged or imported file behind. Details: TC-2917.
- DuckDB keeps disk headroom: a write or import that would leave less than `storage.duckdb.min_free_bytes` free, a write still running when free space drops below it, and a statement that spills past `storage.duckdb.max_temp_directory_size` answer `507 duckdb_disk_headroom`; each is counted as `tinycloud_disk_headroom_refusals_total{engine="duckdb"}`. A write refused this way keeps nothing: the database's local copy is dropped and reloaded from its last saved artifact, so neither a transaction that committed just before the refusal nor earlier statements of its batch are kept or saved later. Only plain queries and `EXPLAIN` without `ANALYZE` sent as `query` count as reads; anything else (including `PREPARE`/`EXECUTE`) counts as a write. Reservations share the node-wide disk budget with SQL. Both limits key on `storage.sql.chunked_artifacts`: unset, the floor is 2 GiB (provisional until production headroom is measured) and the spill limit 1 GiB per connection when it is on; otherwise both are off. With chunked artifacts on, the node refuses to start when the DuckDB floor is below `storage.sql.chunked_artifact_max_bytes`, so the floor cannot be 0. Details: TC-2917.
- At startup the node removes DuckDB spill directories and staging or renamed-aside cache files left by a stopped process, before it serves; the cache files themselves stay. Details: TC-2917.
- DuckDB saves a complete checkpoint when a write changes its main file, and uses incremental WAL persistence only when that file still matches the stored checkpoint, so acknowledged writes remain recoverable after restart. Details: TC-2990.
- Opt-in SQL artifact parts also stream cumulative WAL updates, retaining the checkpoint until a full checkpoint is needed. Details: TC-2917.
- DuckDB can persist checkpoints, cumulative WAL and imports through the same opt-in bounded artifact parts. Details: TC-2917.
Expand Down
47 changes: 40 additions & 7 deletions docs/operations/chunked-sql-artifacts.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,10 +139,44 @@ For disk headroom, each successfully persisted open database retains at most
the configured artifact ceiling in checkpoint plus cumulative WAL. A full
checkpoint also needs an owned temporary copy while streaming, and old/new
versions overlap in PostgreSQL until their deletions commit. PostgreSQL can reuse deleted storage after vacuum; relation files need not
shrink. Size refusal occurs after native execution;
the separate snapshot/headroom acceptance must cover transient oversized
writes and temporary copies before enabling. Operators check CVM disk headroom
against the number and ceilings of open databases, plus that transient cost.
shrink. Size refusal occurs after native execution.

DuckDB keeps disk headroom on the filesystem behind its cache. Both limits key
on `storage.sql.chunked_artifacts`: with it on, `storage.duckdb.min_free_bytes`
defaults to 2 GiB and `storage.duckdb.max_temp_directory_size` to 1 GiB per
connection; with it off both stay off unless set. The 2 GiB floor is
provisional until the `storage_headroom` event shows real headroom on
production nodes; operators tune it by config. With chunked artifacts on, the
node refuses to start unless the floor is at least
`storage.sql.chunked_artifact_max_bytes`, so the floor cannot be turned off
(0) while they are on: a write's save copies the database once more, outside
the write's reservation. Leave a margin above that.

- A write is any request that can change the database: executes, batches,
imports, and any query that is not purely plain `SELECT`-style queries and
`EXPLAIN` without `ANALYZE` (so an admin's `PREPARE … AS INSERT` and
`EXECUTE`, `CALL`, `CREATE SCHEMA` and the like count). SQL that fails to
parse or is refused also counts, so while free space is below the floor it
answers `507 duckdb_disk_headroom` rather than its 400 or 403.
- Before a write runs, it reserves its local database files' size (the bound
of the snapshot copy its save can take); an import reserves twice its size.
Reservations share the node-wide budget with SQL. A reservation that would
leave less than the floor is refused with `507 duckdb_disk_headroom` and
nothing runs.
- While a write runs, it is interrupted whenever free space is below what
other operations hold plus the floor. It answers `507 duckdb_disk_headroom`,
and the database's local copy is dropped and reloaded from its last saved
artifact on the next request. So nothing from the request is kept: not a
transaction that committed just before the interrupt was reported, and not
the earlier statements of a non-transactional batch.
- A statement that spills past the per-connection limit also answers
`507 duckdb_disk_headroom`. Read-only queries and exports reserve nothing and
are not interrupted for headroom; only the spill limit applies to them.
- Hydration is not gated: it writes at most the artifact ceiling and is freed
at eviction.
- At startup, before it serves, the node removes what a stopped process left in
the DuckDB cache: spill directories and staging or renamed-aside files. The
cache files themselves stay, because a legacy database reopens them.

Artifact, account and single-statement capacity refusals return 507. Existing
per-space growth refusals retain 402. A concurrent replacement or contended
Expand All @@ -157,6 +191,5 @@ the local main file's length and CID with the stored checkpoint under the
actor fence. A mismatch requires a complete checkpoint, including blocks
written directly to the main file by a large insert.

The feature remains off. DuckDB disk and spill admission still needs its
separate acceptance before activation, alongside conversion and the
ceiling-load benchmark.
The feature remains off. Conversion and the ceiling-load benchmark still need
their separate acceptance before activation.
68 changes: 54 additions & 14 deletions tinycloud-core/src/duckdb/database.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,9 @@ use crate::request_telemetry::{self, QueryTimer, RequestContext};

use super::{
caveats::DuckDbCaveats,
describe, parser,
describe,
disk::{self, DuckDbDiskPolicy},
parser,
storage::{self, StorageMode},
types::*,
};
Expand Down Expand Up @@ -241,6 +243,34 @@ pub(crate) fn spawn_actor_with_parts(
idle_timeout_secs: u64,
max_memory_per_connection: String,
databases: Arc<DashMap<(String, String), DatabaseHandle>>,
) -> DatabaseHandle {
spawn_actor_with_disk(
space_id,
db_name,
base_path,
memory_threshold,
force_file,
idle_timeout_secs,
max_memory_per_connection,
databases,
DuckDbDiskPolicy::default(),
)
}

/// [`spawn_actor_with_parts`] under `disk`: its connections spill at most
/// `disk.max_temp_directory_bytes`, and each statement runs under the disk
/// watcher.
#[allow(clippy::too_many_arguments)]
pub(crate) fn spawn_actor_with_disk(
space_id: String,
db_name: String,
base_path: String,
memory_threshold: u64,
force_file: bool,
idle_timeout_secs: u64,
max_memory_per_connection: String,
databases: Arc<DashMap<(String, String), DatabaseHandle>>,
disk: DuckDbDiskPolicy,
) -> DatabaseHandle {
let (tx, mut rx) = mpsc::channel::<(Option<Arc<RequestContext>>, DbMessage)>(32);
let id = NEXT_ACTOR_ID.fetch_add(1, Ordering::Relaxed);
Expand All @@ -265,7 +295,11 @@ pub(crate) fn spawn_actor_with_parts(
};
let mut conn = {
let _context = request_telemetry::enter(initial_context);
match storage::open_connection(opening_path, &max_memory_per_connection) {
match storage::open_spill_capped_connection(
opening_path,
&max_memory_per_connection,
disk.max_temp_directory_bytes,
) {
Ok(c) => c,
Err(e) => {
tracing::error!("Failed to open database");
Expand Down Expand Up @@ -335,26 +369,32 @@ pub(crate) fn spawn_actor_with_parts(
without_growth,
response_tx,
} => {
let result = if without_growth {
handle_message_without_growth(
&conn,
&request,
&caveats,
&ability,
arrow_format,
)
} else {
handle_message(&conn, &request, &caveats, &ability, arrow_format, None)
};
let budget_key = disk::budget_key(&space_id, &db_name);
let write =
disk.headroom_checked() && disk::writes(&request, &caveats, &ability);
let result = disk.watch(&budget_key, &file_path, &conn, write, || {
if without_growth {
handle_message_without_growth(
&conn,
&request,
&caveats,
&ability,
arrow_format,
)
} else {
handle_message(&conn, &request, &caveats, &ability, arrow_format, None)
}
});

// Post-write promotion check
if result.is_ok() && matches!(mode, StorageMode::InMemory) {
if let Ok(size) = storage::database_size(&conn) {
if size > memory_threshold {
match storage::promote_to_file(
match storage::promote_to_spill_capped_file(
&conn,
&file_path,
&max_memory_per_connection,
disk.max_temp_directory_bytes,
) {
Ok(new_conn) => {
conn = new_conn;
Expand Down
215 changes: 215 additions & 0 deletions tinycloud-core/src/duckdb/disk.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
//! Disk headroom for DuckDB databases (TC-2917).
//!
//! The node keeps `min_free_bytes` free on the filesystem behind the DuckDB
//! cache. Before a write or an import runs, the bytes it can need are reserved
//! in the node-wide [`DiskBudget`] that SQL uses too, so two databases cannot
//! count the same free bytes. While a statement runs, a watcher interrupts it
//! whenever free space is below what other operations hold plus the floor. A
//! refused or interrupted write answers [`DuckDbError::DiskHeadroom`], is never
//! acknowledged, and the database's local state is discarded so nothing from
//! the request is kept or saved later.

use std::path::Path;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{mpsc, Arc};
use std::time::Duration;

use crate::sql::durability::{
DiskBudget, FreeSpace, ReservationGuard, ReservationRefused, StatvfsFreeSpace,
};

use super::caveats::DuckDbCaveats;
use super::types::{DuckDbError, DuckDbRequest};

/// How often the watcher reads free space while a statement runs.
const WATCH_INTERVAL: Duration = Duration::from_millis(50);

/// The DuckDB disk settings. The default is off: no reservations, no
/// free-space reads and DuckDB's own spill limit.
#[derive(Clone)]
pub struct DuckDbDiskPolicy {
/// Free bytes that must remain after every in-flight operation's
/// reservation. 0 turns the headroom check and the watcher off.
pub min_free_bytes: u64,
/// The most bytes one connection may spill to its temporary directory
/// (DuckDB `max_temp_directory_size`). 0 keeps DuckDB's default.
pub max_temp_directory_bytes: u64,
pub budget: Arc<DiskBudget>,
pub free_space: Arc<dyn FreeSpace>,
}

impl Default for DuckDbDiskPolicy {
fn default() -> Self {
Self {
min_free_bytes: 0,
max_temp_directory_bytes: 0,
budget: Arc::new(DiskBudget::default()),
free_space: Arc::new(StatvfsFreeSpace),
}
}
}

impl std::fmt::Debug for DuckDbDiskPolicy {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("DuckDbDiskPolicy")
.field("min_free_bytes", &self.min_free_bytes)
.field("max_temp_directory_bytes", &self.max_temp_directory_bytes)
.finish_non_exhaustive()
}
}

/// The budget key of one DuckDB database. SQL keys never start with this.
pub(crate) fn budget_key(space: &str, db_name: &str) -> String {
format!("duckdb:{space}/{db_name}")
}

/// Requests that can change the database and are saved afterwards: they
/// reserve room and run under the watcher. A `query` is read-only only when
/// the parser finds it read-only *and* every statement is a plain query or an
/// `EXPLAIN` that does not run its statement; anything else an admin may send
/// as a query (`PREPARE … AS INSERT` with `EXECUTE`, `CALL checkpoint()`,
/// `CREATE SCHEMA`, …) counts as a write, as does SQL the parser refuses.
/// Exports and `describe` never do. Only the spill limit applies to the rest.
pub(crate) fn writes(
request: &DuckDbRequest,
caveats: &Option<DuckDbCaveats>,
ability: &str,
) -> bool {
use sqlparser::ast::Statement;
match request {
DuckDbRequest::Query { sql, .. } => !super::parser::validate_sql(sql, caveats, ability)
.is_ok_and(|parsed| {
parsed.is_read_only
&& parsed.statements.iter().all(|statement| {
matches!(
statement,
Statement::Query(_)
| Statement::ExplainTable { .. }
| Statement::Explain { analyze: false, .. }
)
})
}),
DuckDbRequest::Describe | DuckDbRequest::Export | DuckDbRequest::ExportToKv { .. } => false,
_ => true,
}
}

impl DuckDbDiskPolicy {
pub(crate) fn headroom_checked(&self) -> bool {
self.min_free_bytes > 0
}

/// Whether the policy can refuse anything: a floor or a spill limit. With
/// neither, requests are not classified at all.
pub(crate) fn active(&self) -> bool {
self.headroom_checked() || self.max_temp_directory_bytes > 0
}

/// Reserve `bytes` for one operation of `key`, measured against the
/// filesystem behind `dir`. `None` when the check is off.
pub(crate) fn reserve(
&self,
key: &str,
bytes: u64,
dir: &Path,
) -> Result<Option<ReservationGuard>, DuckDbError> {
if !self.headroom_checked() {
return Ok(None);
}
match self.budget.reserve(
key,
bytes,
0,
self.free_space.as_ref(),
existing_ancestor(dir),
self.min_free_bytes,
) {
Ok(()) => Ok(Some(ReservationGuard::new(
self.budget.clone(),
key.to_string(),
))),
Err(ReservationRefused::Headroom) => Err(DuckDbError::DiskHeadroom),
// Free space that cannot be read is refused, never allowed.
Err(ReservationRefused::Unknown) => Err(DuckDbError::Internal(
"free disk space could not be read".into(),
)),
}
}

/// Run `statement` on `conn`. A write (`write`) is interrupted whenever
/// free space behind `dir` is below what other operations hold plus the
/// floor: DuckDB clears an interrupt when its next query starts, so the
/// watcher re-interrupts on every short reading until the statement
/// returns. A failed statement that was interrupted, or one that hit the
/// spill limit, is [`DuckDbError::DiskHeadroom`]; the caller discards the
/// database's local state, because a COMMIT can finish before DuckDB
/// reports the interrupt.
pub(crate) fn watch<T>(
&self,
key: &str,
dir: &Path,
conn: &duckdb::Connection,
write: bool,
statement: impl FnOnce() -> Result<T, DuckDbError>,
) -> Result<T, DuckDbError> {
if !write || !self.headroom_checked() {
return statement().map_err(|error| self.classify(error));
}
let interrupt = conn.interrupt_handle();
let tripped = AtomicBool::new(false);
let dir = existing_ancestor(dir);
let result = std::thread::scope(|scope| {
let (stop, stopped) = mpsc::channel::<()>();
let (tripped, dir) = (&tripped, &dir);
scope.spawn(move || {
while let Err(mpsc::RecvTimeoutError::Timeout) =
stopped.recv_timeout(WATCH_INTERVAL)
{
let floor = self
.budget
.reserved_excluding(key)
.saturating_add(self.min_free_bytes);
// Unreadable free space stops the statement, like the gate.
let short = self
.free_space
.free_bytes(dir)
.map_or(true, |free| free < floor);
if short {
tripped.store(true, Ordering::Relaxed);
interrupt.interrupt();
}
}
});
let result = statement();
drop(stop);
result
});
match result {
Err(_) if tripped.load(Ordering::Relaxed) => Err(DuckDbError::DiskHeadroom),
other => other.map_err(|error| self.classify(error)),
}
}

/// A statement that ran past the spill limit is the disk, not the caller.
fn classify(&self, error: DuckDbError) -> DuckDbError {
let spill_limit = self.max_temp_directory_bytes > 0
&& matches!(
&error,
DuckDbError::DuckDb(message) | DuckDbError::Internal(message)
if message.contains("max_temp_directory_size")
);
if spill_limit {
DuckDbError::DiskHeadroom
} else {
error
}
}
}

/// The nearest existing ancestor of `path`: where `statvfs` can be asked how
/// much room the filesystem has before the file or its directory exists.
fn existing_ancestor(path: &Path) -> &Path {
path.ancestors()
.find(|ancestor| ancestor.exists())
.unwrap_or(path)
}
1 change: 1 addition & 0 deletions tinycloud-core/src/duckdb/mod.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
pub mod caveats;
pub mod database;
pub mod describe;
pub mod disk;
pub mod parser;
pub mod service;
pub mod storage;
Expand Down
Loading
Loading