Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
- Add opt-in PostgreSQL checkpoint parts for SQL persistence, with explicit artifact/account capacities and streamed hydration. A database stored in parts loads in bounded statements and is not held to the single-row read limit (`storage.sql.max_artifact_read_bytes`); a load that keeps the whole database in memory (a DuckDB export) still waits for one of the large-transfer slots. Details: TC-2917.
- Policy/v3: minting a session and registering a policy no longer re-verify signatures or re-run the claim, presentation, credential admission and allow evaluations while holding their spaces' write order and row locks. Under the locks they check that the challenge, registration and roots are unchanged; those decisions were made before the locks, at the same time, from the same inputs. Refusal codes are unchanged, except that `credential-evaluation-changed` and `policy-evaluation-changed` can no longer occur. Details: TC-2986.
- Opt-in SQL artifact parts also stream cumulative WAL updates, retaining the checkpoint until a full checkpoint is needed. Details: TC-2917.
- DuckDB can persist checkpoints, cumulative WAL and imports through the same opt-in bounded artifact parts. Details: TC-2917.
- SQLite stamp replay deadlines now use the same timestamp encoding as cleanup (TC-2920).
- PostgreSQL database artifacts use uncompressed external storage for new writes to avoid repeated decompression during chunked reads (TC-2917). Existing large artifacts require an operator re-toast; verify both storage settings after rollout and follow `scripts/tc2917-README.md` if lock contention deferred the migration.
- Encryption: v2 column format with key ids, per-purpose keys and recorded format metadata. Optional strict reads are off by default. Details: TC-2516, TC-2517.
Expand Down
25 changes: 21 additions & 4 deletions docs/operations/chunked-sql-artifacts.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Opt-in SQL checkpoint parts
# Opt-in database artifact parts

Opt-in SQL persistence stores checkpoints and cumulative WAL as immutable PostgreSQL parts.
Opt-in SQL and DuckDB persistence stores checkpoints and cumulative WAL as immutable PostgreSQL parts.
Each payload statement carries at most 1 MiB. A metadata-only compare-and-swap
publishes a sealed part; successful writes retain the existing durability and
lineage semantics. Hydration verifies the part CID before publishing its cache.
Expand Down Expand Up @@ -38,13 +38,15 @@ bytes are gone. A periodic collector recovers uploads whose 120-second lease
expired. Lease expiry determines eligibility; part identity and state fence
uploads against concurrent reclamation. Active sealed uploads retain their lease
until publication. Deleted artifacts are reclaimed by the same mechanism.
It does not enable chunked DuckDB writes. Legacy data remains readable. Drain
DuckDB uses owned checkpoint and WAL files under the same capacities. Its
imports stream the already-buffered public request into bounded artifact
statements. Legacy data remains readable. Drain
older writers before enabling this format; turning the switch off does not
convert published parts back to legacy blobs. Current readers understand both
layouts; legacy persistence refuses a database already using parts.

Do not enable for prolonged operation until the approved ceiling load acceptance and
remaining conversion and DuckDB slices are complete. On each test release, report
remaining conversion slice is complete. On each test release, report
reserved, staged and retained bytes from the following metadata-only inventory:

```sql
Expand Down Expand Up @@ -118,3 +120,18 @@ Roll out WAL-capable readers to every instance before enabling the switch.
Checkpoint-only part readers fail closed on WAL layouts. Schema changes are
forward-only; switching the feature off keeps existing layouts readable and
writable by the current binary, and does not downgrade their format.

New DuckDB databases with parts enabled start file-backed under the existing
node data directory, regardless of the in-memory threshold. Their actor fence
protects checkpoint/WAL copies. Closing or idle-evicting an actor removes its
cache files after the native connection is closed; the next access rehydrates
from durable parts. Legacy actors retain their existing storage behavior.

For disk headroom, each successfully persisted open database retains at most
the configured artifact ceiling in checkpoint plus cumulative WAL. A full
checkpoint also needs an owned temporary copy while streaming, and old/new
versions overlap in PostgreSQL until their deletions commit. PostgreSQL can reuse deleted storage after vacuum; relation files need not
shrink. Size refusal occurs after native execution;
the separate snapshot/headroom acceptance must cover transient oversized
writes and temporary copies before enabling. Operators check CVM disk headroom
against the number and ceilings of open databases, plus that transient cost.
204 changes: 198 additions & 6 deletions tinycloud-core/src/duckdb/database.rs
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,13 @@ enum DbMessage {
Export {
response_tx: oneshot::Sender<Result<Vec<u8>, DuckDbError>>,
},
Snapshot {
response_tx: oneshot::Sender<Result<tempfile::NamedTempFile, DuckDbError>>,
},
WalSnapshot {
max_bytes: u64,
response_tx: oneshot::Sender<Result<Option<tempfile::NamedTempFile>, DuckDbError>>,
},
Wal {
response_tx: oneshot::Sender<Result<Option<Vec<u8>>, DuckDbError>>,
},
Expand Down Expand Up @@ -96,6 +103,40 @@ impl DatabaseHandle {
.map_err(|_| DuckDbError::Internal("Database actor dropped response".to_string()))?
}

pub async fn snapshot(&self) -> Result<tempfile::NamedTempFile, DuckDbError> {
let (response_tx, response_rx) = oneshot::channel();
self.tx
.send((
request_telemetry::current(),
DbMessage::Snapshot { response_tx },
))
.await
.map_err(|_| DuckDbError::Internal("Database actor not available".into()))?;
response_rx
.await
.map_err(|_| DuckDbError::Internal("Database actor dropped response".into()))?
}

pub async fn wal_snapshot(
&self,
max_bytes: u64,
) -> Result<Option<tempfile::NamedTempFile>, DuckDbError> {
let (response_tx, response_rx) = oneshot::channel();
self.tx
.send((
request_telemetry::current(),
DbMessage::WalSnapshot {
max_bytes,
response_tx,
},
))
.await
.map_err(|_| DuckDbError::Internal("Database actor not available".into()))?;
response_rx
.await
.map_err(|_| DuckDbError::Internal("Database actor dropped response".into()))?
}

pub async fn wal(&self) -> Result<Option<Vec<u8>>, DuckDbError> {
let (response_tx, response_rx) = oneshot::channel();
self.tx
Expand Down Expand Up @@ -131,6 +172,29 @@ pub fn spawn_actor(
idle_timeout_secs: u64,
max_memory_per_connection: String,
databases: Arc<DashMap<(String, String), DatabaseHandle>>,
) -> DatabaseHandle {
spawn_actor_with_parts(
space_id,
db_name,
base_path,
memory_threshold,
false,
idle_timeout_secs,
max_memory_per_connection,
databases,
)
}

#[allow(clippy::too_many_arguments)]
pub(crate) fn spawn_actor_with_parts(
space_id: String,
db_name: String,
base_path: String,
memory_threshold: u64,
force_file: bool,
idle_timeout_secs: u64,
max_memory_per_connection: String,
databases: Arc<DashMap<(String, String), DatabaseHandle>>,
) -> DatabaseHandle {
let (tx, mut rx) = mpsc::channel::<(Option<Arc<RequestContext>>, DbMessage)>(32);
let id = NEXT_ACTOR_ID.fetch_add(1, Ordering::Relaxed);
Expand All @@ -143,8 +207,8 @@ pub fn spawn_actor(
.join(&space_id)
.join(format!("{}.duckdb", db_name));

// Check if file already exists -- if so, open from file
let mut mode = if file_path.exists() {
// Part-backed actors start file-backed; legacy actors reopen existing caches.
let mut mode = if force_file || file_path.exists() {
StorageMode::File(file_path.clone())
} else {
StorageMode::InMemory
Expand Down Expand Up @@ -175,6 +239,18 @@ pub fn spawn_actor(
e
))));
}
DbMessage::Snapshot { response_tx } => {
let _ = response_tx.send(Err(DuckDbError::Internal(format!(
"Failed to open: {}",
e
))));
}
DbMessage::WalSnapshot { response_tx, .. } => {
let _ = response_tx.send(Err(DuckDbError::Internal(format!(
"Failed to open: {}",
e
))));
}
DbMessage::Wal { response_tx } => {
let _ = response_tx.send(Err(DuckDbError::Internal(format!(
"Failed to open: {}",
Expand All @@ -186,7 +262,7 @@ pub fn spawn_actor(
}
}
}
databases.remove_if(&(space_id, db_name), |_, handle| handle.id == id);
deregister_actor(&databases, (space_id, db_name), id, &file_path, force_file);
return;
}
}
Expand Down Expand Up @@ -253,6 +329,18 @@ pub fn spawn_actor(
handle_export(&conn, &mode, &file_path, &max_memory_per_connection);
let _ = response_tx.send(result);
}
DbMessage::Snapshot { response_tx } => {
let result =
handle_snapshot(&conn, &mode, &file_path, &max_memory_per_connection);
let _ = response_tx.send(result);
}
DbMessage::WalSnapshot {
max_bytes,
response_tx,
} => {
let result = handle_wal_snapshot(&mode, &file_path, max_bytes);
let _ = response_tx.send(result);
}
DbMessage::Wal { response_tx } => {
let result = handle_wal(&mode, &file_path);
let _ = response_tx.send(result);
Expand All @@ -272,9 +360,7 @@ pub fn spawn_actor(
// been hydrated and registered for this key, and evicting it would send
// the next request through hydration while it is live — deleting and
// rewriting the files it holds open by path.
databases.remove_if(&(space_id.clone(), db_name.clone()), |_, handle| {
handle.id == id
});
deregister_actor(&databases, (space_id, db_name), id, &file_path, force_file);
tracing::debug!("Database actor shutting down");
if let Some(response_tx) = shutdown {
let _ = response_tx.send(());
Expand All @@ -284,6 +370,112 @@ pub fn spawn_actor(
DatabaseHandle { id, tx }
}

/// Keep registration until cleanup is complete. A replacement actor's cache
/// must never be removed by its predecessor. Explicit shutdown already holds
/// the service's per-database operation fence, even if it removed the entry.
fn deregister_actor(
databases: &DashMap<(String, String), DatabaseHandle>,
key: (String, String),
id: u64,
path: &Path,
remove_files: bool,
) {
use dashmap::mapref::entry::Entry;
let entry = databases.entry(key);
if matches!(&entry, Entry::Occupied(entry) if entry.get().id != id) {
return;
}
if remove_files {
let mut failed = false;
for candidate in [
path.to_path_buf(),
PathBuf::from(format!("{}.wal", path.display())),
PathBuf::from(format!("{}.tmp", path.display())),
] {
let result = if candidate.is_dir() {
std::fs::remove_dir_all(&candidate)
} else {
std::fs::remove_file(&candidate)
};
if let Err(error) = result {
failed |= error.kind() != std::io::ErrorKind::NotFound;
}
}
if failed {
// Operators use this value-free event when checking cache-disk headroom.
tracing::warn!(
event = "duckdb_cache_cleanup_failed",
"Database cache cleanup failed"
);
}
}
if let Entry::Occupied(entry) = entry {
entry.remove();
}
}

fn handle_wal_snapshot(
mode: &StorageMode,
file_path: &Path,
max_bytes: u64,
) -> Result<Option<tempfile::NamedTempFile>, DuckDbError> {
if !matches!(mode, StorageMode::File(_)) {
return Ok(None);
}
let mut source = match std::fs::File::open(format!("{}.wal", file_path.display())) {
Ok(file) => file,
Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None),
Err(error) => return Err(DuckDbError::Internal(error.to_string())),
};
let size = source
.metadata()
.map_err(|e| DuckDbError::Internal(e.to_string()))?
.len();
if size == 0 || size >= max_bytes {
return Ok(None);
}
let mut snapshot =
tempfile::NamedTempFile::new().map_err(|e| DuckDbError::Internal(e.to_string()))?;
std::io::copy(&mut source, snapshot.as_file_mut())
.map_err(|e| DuckDbError::Internal(e.to_string()))?;
Ok(Some(snapshot))
}

fn copy_snapshot(path: &Path) -> Result<tempfile::NamedTempFile, DuckDbError> {
let mut snapshot =
tempfile::NamedTempFile::new().map_err(|e| DuckDbError::Internal(e.to_string()))?;
let mut source = std::fs::File::open(path).map_err(|e| DuckDbError::Internal(e.to_string()))?;
std::io::copy(&mut source, snapshot.as_file_mut())
.map_err(|e| DuckDbError::Internal(e.to_string()))?;
Ok(snapshot)
}

fn handle_snapshot(
conn: &duckdb::Connection,
mode: &StorageMode,
file_path: &Path,
max_memory: &str,
) -> Result<tempfile::NamedTempFile, DuckDbError> {
match mode {
StorageMode::File(_) => {
{
let _timer = QueryTimer::new();
conn.execute_batch("CHECKPOINT;")
.map_err(|e| DuckDbError::Internal(e.to_string()))?;
}
copy_snapshot(file_path)
}
StorageMode::InMemory => {
let dir = tempfile::tempdir().map_err(|e| DuckDbError::Internal(e.to_string()))?;
let path = dir.path().join("snapshot.duckdb");
let dest = storage::open_connection(Some(&path), max_memory)?;
storage::copy_tables(conn, &dest)?;
drop(dest);
copy_snapshot(&path)
}
}
}

fn handle_wal(mode: &StorageMode, file_path: &Path) -> Result<Option<Vec<u8>>, DuckDbError> {
if !matches!(mode, StorageMode::File(_)) {
return Ok(None);
Expand Down
Loading
Loading