Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
- Add opt-in PostgreSQL checkpoint parts for SQL persistence, with explicit artifact/account capacities and streamed hydration. A database stored in parts loads in bounded statements and is not held to the single-row read limit (`storage.sql.max_artifact_read_bytes`); a load that keeps the whole database in memory (a DuckDB export) still waits for one of the large-transfer slots. Details: TC-2917.
- Policy/v3: minting a session and registering a policy no longer re-verify signatures or re-run the claim, presentation, credential admission and allow evaluations while holding their spaces' write order and row locks. Under the locks they check that the challenge, registration and roots are unchanged; those decisions were made before the locks, at the same time, from the same inputs. Refusal codes are unchanged, except that `credential-evaluation-changed` and `policy-evaluation-changed` can no longer occur. Details: TC-2986.
- Artifact-part writers prioritize their own predecessor reclamation, keep replacement contention retryable, and distinguish capacity refusals from space-growth limits; existing database layouts can checkpoint before adopting WAL parts. Details: TC-2917.
- DuckDB cache cleanup no longer holds up lookups of other databases while files are deleted; a spill directory left by a stopped process no longer blocks hydration or discard, and a failed import leaves no staged or imported file behind. Details: TC-2917.
- DuckDB saves a complete checkpoint when a write changes its main file, and uses incremental WAL persistence only when that file still matches the stored checkpoint, so acknowledged writes remain recoverable after restart. Details: TC-2990.
- Opt-in SQL artifact parts also stream cumulative WAL updates, retaining the checkpoint until a full checkpoint is needed. Details: TC-2917.
- DuckDB can persist checkpoints, cumulative WAL and imports through the same opt-in bounded artifact parts. Details: TC-2917.
Expand Down
12 changes: 8 additions & 4 deletions docs/operations/chunked-sql-artifacts.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,12 @@ New DuckDB databases with parts enabled start file-backed under the existing
node data directory, regardless of the in-memory threshold. Their actor fence
protects checkpoint/WAL copies. Closing or idle-evicting an actor removes its
cache files after the native connection is closed; the next access rehydrates
from durable parts. Legacy actors retain their existing storage behavior.
from durable parts. The files are renamed aside while the actor's registration
is held and deleted after it is released, so other databases' lookups do not
wait on the deletion; if the process stops in between, the next hydration of
that database removes them. A spill directory left by a stopped process does
not block hydration, and an import that fails leaves no staged or imported
file. Legacy actors retain their existing storage behavior.

For disk headroom, each successfully persisted open database retains at most
the configured artifact ceiling in checkpoint plus cumulative WAL. A full
Expand All @@ -152,7 +157,6 @@ the local main file's length and CID with the stored checkpoint under the
actor fence. A mismatch requires a complete checkpoint, including blocks
written directly to the main file by a large insert.

The feature remains off. Additional file-removal lock duration, spill-directory
cleanup, import failure cleanup and disk/spill admission conditions still need
their separate acceptance before activation, alongside conversion and the
The feature remains off. DuckDB disk and spill admission still needs its
separate acceptance before activation, alongside conversion and the
ceiling-load benchmark.
19 changes: 15 additions & 4 deletions tinycloud-core/src/database_artifacts.rs
Original file line number Diff line number Diff line change
Expand Up @@ -472,8 +472,10 @@ impl Drop for StagingRelease {

/// Remove the `<final>.tmp.<pid>.<n>` files a crashed hydration left beside
/// each of `finals`: exactly this sink's temp-name pattern, and never a file
/// this process still has in flight. Run before a hydration begins, under the
/// database's hydration exclusion.
/// this process still has in flight. A directory with that name is a DuckDB
/// spill directory an evicted actor moved aside, and is removed with its
/// contents. Run before a hydration begins, under the database's hydration
/// exclusion.
pub(crate) async fn sweep_stale_staging(finals: &[&std::path::Path]) {
for path in finals {
let (Some(dir), Some(name)) = (path.parent(), path.file_name()) else {
Expand Down Expand Up @@ -505,8 +507,17 @@ pub(crate) async fn sweep_stale_staging(finals: &[&std::path::Path]) {
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.contains(&temp);
if is_staging && !live && tokio::fs::remove_file(&temp).await.is_ok() {
tracing::info!("Removed a staging file left by an interrupted hydration");
if !is_staging || live {
continue;
}
let removed = match entry.file_type().await {
Ok(kind) if kind.is_dir() => tokio::fs::remove_dir_all(&temp).await,
_ => tokio::fs::remove_file(&temp).await,
};
if removed.is_ok() {
tracing::info!(
"Removed a staging file left by an interrupted hydration or cleanup"
);
}
}
}
Expand Down
284 changes: 252 additions & 32 deletions tinycloud-core/src/duckdb/database.rs
Original file line number Diff line number Diff line change
Expand Up @@ -188,18 +188,25 @@ impl DatabaseHandle {
}

/// Stop the actor and close its database before the cache is discarded.
///
/// An actor that has already exited (idle eviction, at the same moment as
/// this call) is done too: it drops its receiver only after closing its
/// database and running its cleanup.
pub(crate) async fn shutdown(&self) -> Result<(), DuckDbError> {
let (response_tx, response_rx) = oneshot::channel();
self.tx
if self
.tx
.send((
request_telemetry::current(),
DbMessage::Shutdown { response_tx },
))
.await
.map_err(|_| DuckDbError::Internal("Database actor not available".to_string()))?;
response_rx.await.map_err(|_| {
DuckDbError::Internal("Database actor dropped shutdown response".to_string())
})
.is_ok()
{
// A dropped sender means the actor exited with this message queued.
let _ = response_rx.await;
}
Ok(())
}
}

Expand Down Expand Up @@ -431,47 +438,125 @@ pub(crate) fn spawn_actor_with_parts(
DatabaseHandle { id, tx }
}

/// Keep registration until cleanup is complete. A replacement actor's cache
/// must never be removed by its predecessor. Explicit shutdown already holds
/// the service's per-database operation fence, even if it removed the entry.
type ActorEntry<'a> = dashmap::mapref::entry::Entry<'a, (String, String), DatabaseHandle>;

/// Keep registration until the cache paths are free. A replacement actor's
/// cache must never be removed by its predecessor. Explicit shutdown already
/// holds the service's per-database operation fence, even if it removed the
/// entry.
///
/// The files are only renamed aside while the map entry is held, so a lookup
/// of any other database in the same map shard never waits on file deletion.
/// The order is carried by the types in [`cleanup`]: renaming aside needs the
/// held entry, releasing consumes it, and only the released form can be
/// deleted.
fn deregister_actor(
databases: &DashMap<(String, String), DatabaseHandle>,
key: (String, String),
id: u64,
path: &Path,
remove_files: bool,
) {
use dashmap::mapref::entry::Entry;
let entry = databases.entry(key);
if matches!(&entry, Entry::Occupied(entry) if entry.get().id != id) {
if matches!(&entry, ActorEntry::Occupied(entry) if entry.get().id != id) {
return;
}
if remove_files {
let mut failed = false;
for candidate in [
path.to_path_buf(),
PathBuf::from(format!("{}.wal", path.display())),
PathBuf::from(format!("{}.tmp", path.display())),
] {
let result = if candidate.is_dir() {
std::fs::remove_dir_all(&candidate)
} else {
std::fs::remove_file(&candidate)
};
if let Err(error) = result {
failed |= error.kind() != std::io::ErrorKind::NotFound;
let moved = if remove_files {
cleanup::move_aside(&entry, path)
} else {
cleanup::MovedAside::default()
};
if cleanup::delete_aside(cleanup::release(entry, moved)) {
// Operators use this value-free event when checking cache-disk headroom.
tracing::warn!(
event = "duckdb_cache_cleanup_failed",
"Database cache cleanup failed"
);
}
}

/// The cleanup steps, in a module of their own so that a [`cleanup::Released`]
/// can only come from [`cleanup::release`].
mod cleanup {
use std::path::{Path, PathBuf};

use super::{cache_file_paths, remove_cache_entry, ActorEntry};

/// Cache files renamed to staging names, not yet deleted.
#[derive(Default)]
pub(super) struct MovedAside {
temps: Vec<PathBuf>,
failed: bool,
}

/// [`MovedAside`] once the registration is gone: the only form that can
/// be deleted.
pub(super) struct Released(MovedAside);

/// Rename the cache files at `path` aside while `_held` keeps the
/// registration, so the paths are free before it can disappear. The
/// sweep before each hydration removes the names if the process stops
/// before [`delete_aside`].
pub(super) fn move_aside(_held: &ActorEntry<'_>, path: &Path) -> MovedAside {
let mut moved = MovedAside::default();
for candidate in cache_file_paths(path) {
let temp = crate::database_artifacts::staging_path(path);
match std::fs::rename(&candidate, &temp) {
Ok(()) => moved.temps.push(temp),
Err(error) => {
moved.failed |= error.kind() != std::io::ErrorKind::NotFound;
crate::database_artifacts::release_staging(&temp);
}
}
}
if failed {
// Operators use this value-free event when checking cache-disk headroom.
tracing::warn!(
event = "duckdb_cache_cleanup_failed",
"Database cache cleanup failed"
);
moved
}

/// Drop the registration (or the vacant entry's lock), releasing the
/// shard.
pub(super) fn release(entry: ActorEntry<'_>, moved: MovedAside) -> Released {
match entry {
ActorEntry::Occupied(entry) => {
entry.remove();
}
ActorEntry::Vacant(entry) => drop(entry),
}
Released(moved)
}

/// Delete what was moved aside, outside the map lock. True if anything
/// failed, including the renames.
pub(super) fn delete_aside(Released(moved): Released) -> bool {
let mut failed = moved.failed;
for temp in moved.temps {
failed |= remove_cache_entry(&temp).is_err();
crate::database_artifacts::release_staging(&temp);
}
failed
}
if let Entry::Occupied(entry) = entry {
entry.remove();
}

/// The paths DuckDB keeps for the database at `path`: the main file, its WAL
/// and its spill directory.
pub(crate) fn cache_file_paths(path: &Path) -> [PathBuf; 3] {
[
path.to_path_buf(),
PathBuf::from(format!("{}.wal", path.display())),
PathBuf::from(format!("{}.tmp", path.display())),
]
}

/// Remove one cache file, or DuckDB's spill directory with its contents. A
/// path that is already gone counts as removed.
pub(crate) fn remove_cache_entry(path: &Path) -> std::io::Result<()> {
let removed = match std::fs::symlink_metadata(path) {
Ok(metadata) if metadata.is_dir() => std::fs::remove_dir_all(path),
Ok(_) => std::fs::remove_file(path),
Err(error) => Err(error),
};
match removed {
Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(()),
other => other,
}
}

Expand Down Expand Up @@ -1346,4 +1431,139 @@ mod tests {
));
assert_eq!(row_count(&conn, "events"), 1);
}

fn schema_write(sql: &str, schema: &str) -> DuckDbRequest {
DuckDbRequest::Execute {
sql: sql.to_string(),
params: Vec::new(),
schema: Some(vec![schema.to_string()]),
}
}

fn placeholder_handle(id: u64) -> DatabaseHandle {
let (tx, _rx) = mpsc::channel(1);
DatabaseHandle { id, tx }
}

fn file_actor(
base: &Path,
key: &(String, String),
idle_timeout_secs: u64,
databases: &Arc<DashMap<(String, String), DatabaseHandle>>,
) -> DatabaseHandle {
spawn_actor_with_parts(
key.0.clone(),
key.1.clone(),
base.to_string_lossy().into_owned(),
u64::MAX,
true,
idle_timeout_secs,
"128MB".into(),
databases.clone(),
)
}

#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn chunked_actor_close_spares_a_replacement_and_its_files() {
let base = tempfile::TempDir::new().unwrap();
let databases = Arc::new(DashMap::new());
let key = ("space-a".to_string(), "main".to_string());
let actor = file_actor(base.path(), &key, 300, &databases);
databases.insert(key.clone(), actor.clone());
actor
.execute(
schema_write("INSERT INTO t VALUES (1)", "CREATE TABLE t(i INTEGER)"),
None,
"tinycloud.duckdb/admin".into(),
false,
false,
)
.await
.unwrap();
let path = base.path().join(&key.0).join("main.duckdb");
assert!(path.exists());

// A replacement registered under the key; its files are at the same path.
databases.insert(key.clone(), placeholder_handle(u64::MAX));
actor.shutdown().await.unwrap();
assert!(
path.exists(),
"the predecessor removed the replacement's cache"
);
assert_eq!(databases.get(&key).map(|handle| handle.id), Some(u64::MAX));

// With the entry vacant (explicit shutdown under the operation fence)
// every cache path goes, the spill directory included.
databases.remove(&key);
let [main, wal, spill] = cache_file_paths(&path);
std::fs::create_dir_all(&spill).unwrap();
std::fs::write(spill.join("spill"), b"x").unwrap();
std::fs::write(&wal, b"w").unwrap();
deregister_actor(&databases, key.clone(), 7, &path, true);
assert!(!main.exists() && !wal.exists() && !spill.exists());
assert!(std::fs::read_dir(path.parent().unwrap())
.unwrap()
.next()
.is_none());

// Actors that do not own their files never delete them.
std::fs::write(&path, b"keep").unwrap();
deregister_actor(&databases, key, 7, &path, false);
assert!(path.exists());
}

#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn chunked_idle_eviction_frees_cache_paths_before_deregistering() {
let base = tempfile::TempDir::new().unwrap();
let databases = Arc::new(DashMap::new());
let key = ("space-evict".to_string(), "main".to_string());
let path = base.path().join(&key.0).join("main.duckdb");
let [main, wal, spill] = cache_file_paths(&path);
// A spill directory a crashed query left behind.
std::fs::create_dir_all(&spill).unwrap();
for i in 0..2_000 {
std::fs::write(spill.join(format!("s{i}")), b"x").unwrap();
}
let actor = file_actor(base.path(), &key, 1, &databases);
databases.insert(key.clone(), actor.clone());
actor
.execute(
schema_write("INSERT INTO t VALUES (1)", "CREATE TABLE t(i INTEGER)"),
None,
"tinycloud.duckdb/admin".into(),
false,
false,
)
.await
.unwrap();

let deadline = std::time::Instant::now() + std::time::Duration::from_secs(30);
while databases.contains_key(&key) {
assert!(
std::time::Instant::now() < deadline,
"actor must idle-evict"
);
tokio::time::sleep(std::time::Duration::from_millis(1)).await;
}
assert!(
!main.exists() && !wal.exists() && !spill.exists(),
"the registration vanished while cache paths remained"
);
// The renamed-aside files are deleted after the entry is released.
let dir = path.parent().unwrap();
while std::fs::read_dir(dir).unwrap().next().is_some() {
assert!(std::time::Instant::now() < deadline, "cleanup must finish");
tokio::time::sleep(std::time::Duration::from_millis(10)).await;
}
}

#[tokio::test]
async fn shutdown_of_an_exited_actor_succeeds() {
let base = tempfile::TempDir::new().unwrap();
let databases = Arc::new(DashMap::new());
let key = ("space-exit".to_string(), "main".to_string());
let actor = file_actor(base.path(), &key, 300, &databases);
actor.shutdown().await.unwrap();
actor.shutdown().await.unwrap();
}
}
Loading
Loading