diff --git a/CHANGELOG.md b/CHANGELOG.md index 0d4785c2..26592e1a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,8 @@ - `POST /delegate` and `POST /revoke` check the signature (and, for a renewal lease, the signature of the stored original it renews) before taking any lock. A forged or slow-to-verify request no longer holds the delegation's chain guards, its spaces' write order or a database transaction while it is checked; it is refused at once. A renewal lease whose original is not stored is also refused before any lock. Inside the transaction only the signed time window is checked again. A resubmitted delegation that is already stored is now also signature-checked before it is answered as existing. Details: TC-2986. - Add opt-in PostgreSQL checkpoint parts for SQL persistence, with explicit artifact/account capacities and streamed hydration. A database stored in parts loads in bounded statements and is not held to the single-row read limit (`storage.sql.max_artifact_read_bytes`); a load that keeps the whole database in memory (a DuckDB export) still waits for one of the large-transfer slots. Details: TC-2917. - Policy/v3: minting a session and registering a policy no longer re-verify signatures or re-run the claim, presentation, credential admission and allow evaluations while holding their spaces' write order and row locks. Under the locks they check that the challenge, registration and roots are unchanged; those decisions were made before the locks, at the same time, from the same inputs. Refusal codes are unchanged, except that `credential-evaluation-changed` and `policy-evaluation-changed` can no longer occur. Details: TC-2986. +- Artifact-part writers prioritize their own predecessor reclamation, keep replacement contention retryable, and distinguish capacity refusals from space-growth limits; existing database layouts can checkpoint before adopting WAL parts. Details: TC-2917. +- DuckDB saves a complete checkpoint when a write changes its main file, and uses incremental WAL persistence only when that file still matches the stored checkpoint, so acknowledged writes remain recoverable after restart. Details: TC-2990. - Opt-in SQL artifact parts also stream cumulative WAL updates, retaining the checkpoint until a full checkpoint is needed. Details: TC-2917. - DuckDB can persist checkpoints, cumulative WAL and imports through the same opt-in bounded artifact parts. Details: TC-2917. - SQLite stamp replay deadlines now use the same timestamp encoding as cleanup (TC-2920). diff --git a/docs/operations/chunked-sql-artifacts.md b/docs/operations/chunked-sql-artifacts.md index 65aa1651..8778a22a 100644 --- a/docs/operations/chunked-sql-artifacts.md +++ b/docs/operations/chunked-sql-artifacts.md @@ -102,7 +102,10 @@ the same database's credit for a second simultaneous replacement. Published parts have a distinct state and infinite lease, and are excluded from both global and owner-local reclamation indexes. Inline cleanup is best effort, scoped to the uploading owner, at most 16 steps and 100 ms per pass; its errors -do not fail the save. Background passes are limited to 128 steps and one second +do not fail the save. A writer first prioritizes its own database's eligible +predecessors for at most two seconds; active upload leases are never stolen. +Failed owners are excluded for the remainder of a collection pass so other +owners can continue. Owner-lock waits at reservation are bounded at 50 ms. Background passes are limited to 128 steps and one second per 10-second tick. GC lock acquisition waits at most 50 ms per statement. An owner already above its quota can replace a database with an equal or smaller image; that exception admits no growth and does not free pending shrink @@ -135,3 +138,21 @@ shrink. Size refusal occurs after native execution; the separate snapshot/headroom acceptance must cover transient oversized writes and temporary copies before enabling. Operators check CVM disk headroom against the number and ceilings of open databases, plus that transient cost. + +Artifact, account and single-statement capacity refusals return 507. Existing +per-space growth refusals retain 402. A concurrent replacement or contended +reservation returns a retryable 503 without checkpoint back-off; staged +bookkeeping does not turn this positive pre-publication refusal into an +unknown write outcome. A genuinely lost publish reply retains its existing +unknown-outcome behavior. Expired uploads remain counted until collection. + +An old-format checkpoint can be replaced by a full owned snapshot before WAL +parts attach. For DuckDB, both legacy and owned-WAL persistence first compare +the local main file's length and CID with the stored checkpoint under the +actor fence. A mismatch requires a complete checkpoint, including blocks +written directly to the main file by a large insert. + +The feature remains off. Additional file-removal lock duration, spill-directory +cleanup, import failure cleanup and disk/spill admission conditions still need +their separate acceptance before activation, alongside conversion and the +ceiling-load benchmark. diff --git a/tinycloud-core/src/database_artifacts.rs b/tinycloud-core/src/database_artifacts.rs index daf0d9de..c44f8c88 100644 --- a/tinycloud-core/src/database_artifacts.rs +++ b/tinycloud-core/src/database_artifacts.rs @@ -109,6 +109,8 @@ pub enum DatabaseArtifactError { Db(DbErr), #[error("artifact storage limit reached")] StorageLimit, + #[error("database artifact replacement is in progress")] + ReplacementBusy, #[error("database artifact payload too large: {0} bytes")] PayloadTooLarge(u64), /// A payload larger than the repository's single-statement write limit @@ -145,7 +147,18 @@ pub enum DatabaseArtifactError { impl From for DatabaseArtifactError { fn from(error: DbErr) -> Self { - if error.to_string().contains("TC2917_QUOTA") { + use sea_orm::error::{RuntimeErr, SqlxError}; + let replacement_busy = match &error { + DbErr::Conn(RuntimeErr::SqlxError(SqlxError::Database(error))) + | DbErr::Exec(RuntimeErr::SqlxError(SqlxError::Database(error))) + | DbErr::Query(RuntimeErr::SqlxError(SqlxError::Database(error))) => { + error.code().as_deref() == Some("TC001") && error.message() == "TC2917_BUSY" + } + _ => false, + }; + if replacement_busy { + Self::ReplacementBusy + } else if error.to_string().contains("TC2917_QUOTA") { Self::StorageLimit } else if error.to_string().contains("TC2917_LAYOUT") { Self::StaleLineage @@ -783,7 +796,7 @@ pub struct SeaOrmDatabaseArtifactRepository { #[cfg(test)] race_barrier: Option>, #[cfg(test)] - chunk_publish_barrier: Option>, + pub(crate) chunk_publish_barrier: Option>, #[cfg(test)] chunk_reclaim_paused: bool, } diff --git a/tinycloud-core/src/database_artifacts/chunks.rs b/tinycloud-core/src/database_artifacts/chunks.rs index e4d8fa81..13503e86 100644 --- a/tinycloud-core/src/database_artifacts/chunks.rs +++ b/tinycloud-core/src/database_artifacts/chunks.rs @@ -17,8 +17,16 @@ fn statement(sql: &str, values: Vec) -> Statement { Statement::from_sql_and_values(DbBackend::Postgres, sql, values) } -const REAP_CANDIDATE: &str = "SELECT service,space,name,part_id FROM artifact_part p WHERE state<>'published' AND lease_until<=statement_timestamp() AND NOT EXISTS(SELECT 1 FROM database_artifact d WHERE d.service=p.service AND d.space=p.space AND d.name=p.name AND (d.checkpoint_part_id=p.part_id OR d.delta_part_id=p.part_id)) ORDER BY lease_until LIMIT 1"; -const OWNER_REAP_CANDIDATE: &str = "SELECT service,space,name,part_id FROM artifact_part p WHERE owner=$1 AND state<>'published' AND lease_until<=statement_timestamp() AND NOT EXISTS(SELECT 1 FROM database_artifact d WHERE d.service=p.service AND d.space=p.space AND d.name=p.name AND (d.checkpoint_part_id=p.part_id OR d.delta_part_id=p.part_id)) ORDER BY lease_until LIMIT 1"; +const REAP_CANDIDATE: &str = "SELECT owner,service,space,name,part_id FROM artifact_part p WHERE state<>'published' AND lease_until<=statement_timestamp() AND NOT EXISTS(SELECT 1 FROM database_artifact d WHERE d.service=p.service AND d.space=p.space AND d.name=p.name AND (d.checkpoint_part_id=p.part_id OR d.delta_part_id=p.part_id)) ORDER BY lease_until LIMIT 1"; +const OWNER_REAP_CANDIDATE: &str = "SELECT owner,service,space,name,part_id FROM artifact_part p WHERE owner=$1 AND state<>'published' AND lease_until<=statement_timestamp() AND NOT EXISTS(SELECT 1 FROM database_artifact d WHERE d.service=p.service AND d.space=p.space AND d.name=p.name AND (d.checkpoint_part_id=p.part_id OR d.delta_part_id=p.part_id)) ORDER BY lease_until LIMIT 1"; + +const DATABASE_REAP_CANDIDATE: &str = "SELECT owner,service,space,name,part_id FROM artifact_part p WHERE service=$1 AND space=$2 AND name=$3 AND state<>'published' AND lease_until<=statement_timestamp() AND NOT EXISTS(SELECT 1 FROM database_artifact d WHERE d.service=p.service AND d.space=p.space AND d.name=p.name AND (d.checkpoint_part_id=p.part_id OR d.delta_part_id=p.part_id)) ORDER BY lease_until LIMIT 1"; + +enum ReapScope<'a> { + All, + Owner(&'a str), + Database(&'a str, &'a str, &'a str), +} impl SeaOrmDatabaseArtifactRepository { pub(super) async fn ensure_legacy_write( @@ -68,7 +76,7 @@ impl SeaOrmDatabaseArtifactRepository { /// Each call frees at most one payload chunk per statement. Selection is /// advisory; the database rechecks pointers and takes the owner/part locks. pub async fn reap_unused_parts(&self, max_steps: usize) -> Result<(), DatabaseArtifactError> { - self.reap_parts(None, max_steps).await + self.reap_parts(ReapScope::All, max_steps).await } async fn reap_owner_best_effort(&self, owner: &str) { @@ -76,41 +84,72 @@ impl SeaOrmDatabaseArtifactRepository { // collector completes larger passes independently of request success. let _ = tokio::time::timeout( std::time::Duration::from_millis(100), - self.reap_parts(Some(owner), 16), + self.reap_parts(ReapScope::Owner(owner), 16), ) .await; } async fn reap_parts( &self, - owner: Option<&str>, + scope: ReapScope<'_>, max_steps: usize, ) -> Result<(), DatabaseArtifactError> { + let mut excluded_owners: Vec = Vec::new(); + let mut last_error = None; for _ in 0..max_steps { - let candidate = match owner { - Some(owner) => statement(OWNER_REAP_CANDIDATE, vec![owner.into()]), - None => statement(REAP_CANDIDATE, vec![]), + let (sql, mut values): (&str, Vec) = match scope { + ReapScope::Owner(owner) => (OWNER_REAP_CANDIDATE, vec![owner.into()]), + ReapScope::All => (REAP_CANDIDATE, vec![]), + ReapScope::Database(service, space, name) => ( + DATABASE_REAP_CANDIDATE, + vec![service.into(), space.into(), name.into()], + ), }; - let Some(row) = self.conn.query_one(candidate).await? else { + let mut sql = sql.to_string(); + if !excluded_owners.is_empty() { + let parameters = (values.len() + 1..=values.len() + excluded_owners.len()) + .map(|n| format!("${n}")) + .collect::>() + .join(","); + sql = sql.replace(" ORDER BY lease_until LIMIT 1", + &format!(" AND NOT (p.owner=ANY(ARRAY[{parameters}]::text[])) ORDER BY lease_until LIMIT 1")); + values.extend(excluded_owners.iter().cloned().map(Into::into)); + } + let Some(row) = self.conn.query_one(statement(&sql, values)).await? else { break; }; + let owner: String = row.try_get("", "owner")?; let values = vec![ row.try_get::("", "service")?.into(), row.try_get::("", "space")?.into(), row.try_get::("", "name")?.into(), row.try_get::("", "part_id")?.into(), ]; - let row = self + match self .conn .query_one(statement( "SELECT artifact_reap_one($1,$2,$3,$4) AS progress", values, )) - .await? - .ok_or(DatabaseArtifactError::ArtifactChanged)?; - // Another collector can finish the selected part first. A fresh - // candidate query must still reach the rest of this bounded pass. - let _progress = row.try_get::("", "progress")?; + .await + { + Ok(Some(row)) => { + let _progress = row.try_get::("", "progress")?; + } + Ok(None) => { + excluded_owners.push(owner); + last_error = Some(DatabaseArtifactError::ArtifactChanged); + } + Err(error) => { + // A locked or broken owner cannot end another owner's + // collection pass. Report one error after making progress. + excluded_owners.push(owner); + last_error = Some(error.into()); + } + } + } + if let Some(error) = last_error { + return Err(error); } Ok(()) } @@ -174,6 +213,13 @@ impl SeaOrmDatabaseArtifactRepository { .map_err(|_| DatabaseArtifactError::Backend("invalid artifact space".into()))?; let owner = space_id.did().to_string(); self.configure_chunked_storage().await?; + // Prioritize this database's retired predecessor over the shared + // collector. Active uploads retain their lease and are never stolen. + let _ = tokio::time::timeout( + std::time::Duration::from_secs(2), + self.reap_parts(ReapScope::Database(service, space, name), 4096), + ) + .await; self.reap_owner_best_effort(&owner).await; let current = self.load_meta(service, space, name).await?; expected.check( @@ -760,7 +806,7 @@ mod tests { ) .await; assert!( - matches!(second, Err(DatabaseArtifactError::StorageLimit)), + matches!(second, Err(DatabaseArtifactError::ReplacementBusy)), "second replacement must refuse: {second:?}" ); // Administrative failure injection changes the base after reservation, @@ -802,6 +848,7 @@ mod tests { .await .unwrap(); } + repo.reap_unused_parts(256).await.unwrap(); let reserved = fixture .conn .query_one(statement( @@ -1092,7 +1139,10 @@ mod tests { ArtifactExpectation::Any, ) .await; - assert!(matches!(second, Err(DatabaseArtifactError::StorageLimit))); + assert!(matches!( + second, + Err(DatabaseArtifactError::ReplacementBusy) + )); // Pending shrinkage cannot fund another database, then be aborted. let other = repo .save_snapshot( @@ -2043,6 +2093,33 @@ mod tests { smaller.len() as i64 ) ); + let previous_wal: String = fixture + .conn + .query_one(statement( + "SELECT part_id FROM artifact_part WHERE content_hash=$1", + vec![second.delta_content_hash.clone().unwrap().into()], + )) + .await + .unwrap() + .unwrap() + .try_get("", "part_id") + .unwrap(); + fixture + .conn + .execute(statement( + "SELECT artifact_reap_one('sql',$1,'main',$2)", + vec![space.into(), previous_wal.into()], + )) + .await + .unwrap(); + assert_eq!( + ledger(&fixture).await, + ( + (checkpoint.len() + wal.len() + smaller.len()) as i64, + smaller.len() as i64 + ), + "a deleted prefix does not release the old WAL's charge or credit" + ); repo.reap_unused_parts(64).await.unwrap(); assert_eq!( ledger(&fixture).await, @@ -2147,7 +2224,7 @@ mod tests { let repo = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()).with_chunk_limits( Some(ChunkLimits { artifact_bytes: 1024, - account_bytes: 1024, + account_bytes: 4096, }), ); let expectation = |meta: &ArtifactMeta| ArtifactExpectation::Derived { @@ -2226,4 +2303,252 @@ mod tests { assert_eq!(durable.delta_content_hash, third.delta_content_hash); assert_eq!(durable.revision, third.revision + 1); } + #[tokio::test] + async fn postgres_chunked_abort_after_committed_publish_keeps_all_chunks() { + use crate::database_artifacts::test_support::ReadAll; + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_abort_after_committed_publish_keeps_all_chunks", + ) + .await + else { + return; + }; + let space = "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000001:committed"; + let repo = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()).with_chunk_limits( + Some(ChunkLimits { + artifact_bytes: 4 * CHUNK_BYTES as u64, + account_bytes: 8 * CHUNK_BYTES as u64, + }), + ); + let bytes = vec![7; 2 * CHUNK_BYTES + 17]; + let saved = repo + .save_snapshot( + "sql", + space, + "main", + snapshot(&bytes), + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + let part: String = fixture + .conn + .query_one(statement( + "SELECT checkpoint_part_id FROM database_artifact", + vec![], + )) + .await + .unwrap() + .unwrap() + .try_get("", "checkpoint_part_id") + .unwrap(); + // Observe that the real Drop abort worker ran before checking durability. + // Three chunks ensure a part FK cannot mask deletion of a prefix. + fixture + .conn + .execute_unprepared( + r#" +CREATE TABLE cleanup_attempt(id integer); +ALTER FUNCTION artifact_reap_one(text,text,text,text,boolean) RENAME TO artifact_reap_checked; +CREATE FUNCTION artifact_reap_one(a text,b text,c text,d text,abort_upload boolean DEFAULT false) +RETURNS boolean LANGUAGE plpgsql AS $$ +BEGIN + INSERT INTO cleanup_attempt VALUES(1); + RETURN artifact_reap_checked(a,b,c,d,abort_upload); +END $$; +"#, + ) + .await + .unwrap(); + // Models cancellation after commit but before its response disarmed the + // upload guard. Use the production Drop path, not a separate collector. + drop(UploadCleanup { + repo: repo.clone(), + service: "sql".into(), + space: space.into(), + name: "main".into(), + part_id: part, + armed: true, + }); + let until = tokio::time::Instant::now() + std::time::Duration::from_secs(2); + loop { + let n: i64 = fixture + .conn + .query_one(statement( + "SELECT count(*)::bigint AS n FROM cleanup_attempt", + vec![], + )) + .await + .unwrap() + .unwrap() + .try_get("", "n") + .unwrap(); + if n > 0 { + break; + } + assert!(tokio::time::Instant::now() < until, "abort worker must run"); + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + let durable = repo.read_all("sql", space, "main").await.unwrap().unwrap(); + assert_eq!(durable.revision, saved.revision); + assert_eq!(durable.payload, bytes); + assert_eq!(ledger(&fixture).await, (saved.size_bytes, 0)); + let n: i64 = fixture + .conn + .query_one(statement( + "SELECT count(*)::bigint AS n FROM artifact_chunk", + vec![], + )) + .await + .unwrap() + .unwrap() + .try_get("", "n") + .unwrap(); + assert_eq!(n, 3); + } + + #[tokio::test] + async fn postgres_chunked_predecessor_progress_ignores_locked_owner() { + use sea_orm::TransactionTrait; + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_predecessor_progress_ignores_locked_owner", + ) + .await + else { + return; + }; + let a = "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000001:flood"; + let b = "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000002:live"; + let mut repo = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()) + .with_chunk_limits(Some(ChunkLimits { + artifact_bytes: 64 * CHUNK_BYTES as u64, + account_bytes: 256 * CHUNK_BYTES as u64, + })); + repo.chunk_reclaim_paused = true; + repo.save_snapshot( + "sql", + a, + "main", + snapshot(&[1]), + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + fixture.conn.execute(statement("INSERT INTO artifact_part(service,space,name,part_id,owner,size_bytes,chunk_count,state,lease_until) SELECT 'sql',$1,'garbage',md5(g::text),artifact_owner($1),1,1,'uploading','2000-01-01'::timestamptz FROM generate_series(1,1000)g",vec![a.into()])).await.unwrap(); + fixture.conn.execute(statement("INSERT INTO artifact_chunk(service,space,name,part_id,ordinal,payload) SELECT service,space,name,part_id,0,decode('01','hex') FROM artifact_part WHERE space=$1 AND name='garbage'",vec![a.into()])).await.unwrap(); + fixture + .conn + .execute(statement( + "UPDATE artifact_part SET state='deleting' WHERE space=$1 AND name='garbage'", + vec![a.into()], + )) + .await + .unwrap(); + fixture.conn.execute(statement("UPDATE artifact_account SET reserved_bytes=reserved_bytes+1000 WHERE owner=artifact_owner($1)",vec![a.into()])).await.unwrap(); + let first = repo + .save_snapshot( + "sql", + b, + "main", + snapshot(&vec![2; 40 * CHUNK_BYTES + 17]), + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + let second = repo + .save_snapshot( + "sql", + b, + "main", + snapshot(&[3; 1024]), + ArtifactExpectation::Derived { + revision: first.revision, + checkpoint_content_hash: first.checkpoint_content_hash, + }, + ) + .await + .unwrap(); + let locked = fixture.conn.begin().await.unwrap(); + locked + .query_one(statement( + "SELECT owner FROM artifact_account WHERE owner=artifact_owner($1) FOR UPDATE", + vec![a.into()], + )) + .await + .unwrap(); + // More than an inline pass remains, but this writer must not wait on + // another owner's locked/flooded collector queue. + let third = repo + .save_snapshot( + "sql", + b, + "main", + snapshot(&[4; 4096]), + ArtifactExpectation::Derived { + revision: second.revision, + checkpoint_content_hash: second.checkpoint_content_hash, + }, + ) + .await + .unwrap(); + assert_eq!(third.revision, 3); + // A global pass reports the locked owner after continuing other work. + assert!(repo.reap_unused_parts(128).await.is_err()); + let state=fixture.conn.query_one(statement("SELECT (SELECT count(*) FROM artifact_part WHERE space=$1 AND state='deleting')::bigint AS garbage,(SELECT reserved_bytes-credit_bytes FROM artifact_account WHERE owner=artifact_owner($1))::bigint AS charged",vec![b.into()])).await.unwrap().unwrap(); + assert_eq!(state.try_get::("", "garbage").unwrap(), 0); + assert_eq!(state.try_get::("", "charged").unwrap(), 4096); + let untouched:i64=fixture.conn.query_one(statement("SELECT count(*)::bigint AS n FROM artifact_chunk WHERE space=$1 AND name='garbage'",vec![a.into()])).await.unwrap().unwrap().try_get("","n").unwrap(); + assert_eq!(untouched, 1000); + locked.rollback().await.unwrap(); + } + #[tokio::test] + async fn postgres_chunked_flag_off_legacy_writes_do_not_update_parts() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_flag_off_legacy_writes_do_not_update_parts", + ) + .await + else { + return; + }; + let space = + "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000001:legacy-no-parts"; + let repo = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()); + // Statement triggers run even for zero-row UPDATEs. Foreign-key + // relation locks are inherited from the original schema and are not + // evidence that the publication function issued a payload-table query. + fixture + .conn + .execute_unprepared( + r#" +CREATE FUNCTION reject_legacy_part_update() RETURNS trigger LANGUAGE plpgsql AS $$ +BEGIN RAISE EXCEPTION 'unexpected parts UPDATE on legacy write'; END $$; +CREATE TRIGGER reject_legacy_part_update BEFORE UPDATE ON artifact_part + FOR EACH STATEMENT EXECUTE FUNCTION reject_legacy_part_update(); +"#, + ) + .await + .unwrap(); + let first = repo + .save( + "sql", + space, + "main", + vec![1; 128], + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + repo.save_delta( + "sql", + space, + "main", + vec![2; 128], + ArtifactExpectation::Derived { + revision: first.revision, + checkpoint_content_hash: first.checkpoint_content_hash, + }, + ) + .await + .unwrap(); + } } diff --git a/tinycloud-core/src/database_artifacts/part_lifecycle.sql b/tinycloud-core/src/database_artifacts/part_lifecycle.sql new file mode 100644 index 00000000..4644ed11 --- /dev/null +++ b/tinycloud-core/src/database_artifacts/part_lifecycle.sql @@ -0,0 +1,49 @@ +CREATE INDEX artifact_part_reap_database ON artifact_part(service,space,name,lease_until) WHERE state<>'published'; +CREATE OR REPLACE FUNCTION artifact_reserve_part(p_service text,p_space text,p_name text,p_id text,p_owner text, + p_size bigint,p_count integer,p_revision bigint,p_hash text,p_wal boolean) +RETURNS boolean LANGUAGE plpgsql SET lock_timeout='50ms' AS $$ +DECLARE base record; existing boolean; credit bigint:=0; cap bigint; part_cap bigint; replaced bigint:=0; +BEGIN + IF p_owner<>artifact_owner(p_space) THEN RAISE EXCEPTION 'TC2917_LAYOUT: owner mismatch'; END IF; + PERFORM 1 FROM artifact_account WHERE owner=p_owner FOR UPDATE; + SELECT revision,size_bytes,checkpoint_size_bytes,delta_size_bytes,checkpoint_content_hash,checkpoint_part_id,delta_part_id INTO base FROM database_artifact WHERE service=p_service AND space=p_space AND name=p_name; + existing:=FOUND; + IF (p_revision IS NULL AND existing) OR (p_revision IS NOT NULL AND + (NOT existing OR base.revision<>p_revision OR base.checkpoint_content_hash<>p_hash)) THEN + RAISE EXCEPTION 'TC2917_LAYOUT: superseded reservation'; + END IF; + IF p_wal AND (NOT existing OR base.checkpoint_part_id IS NULL) THEN RAISE EXCEPTION 'TC2917_LAYOUT: missing checkpoint'; END IF; + IF EXISTS(SELECT 1 FROM artifact_part WHERE service=p_service AND space=p_space AND name=p_name AND replacing) THEN RAISE EXCEPTION USING ERRCODE='TC001', MESSAGE='TC2917_BUSY'; END IF; + SELECT account_limit,artifact_limit INTO cap,part_cap FROM artifact_storage_settings WHERE id=1; + IF p_size<0 OR p_count<>((p_size+1048575)/1048576) OR + p_size+(CASE WHEN p_wal THEN base.checkpoint_size_bytes ELSE 0 END)>part_cap THEN RETURN false; END IF; + IF existing THEN replaced:=CASE WHEN p_wal THEN base.delta_size_bytes ELSE base.size_bytes END; END IF; + credit:=LEAST(replaced,p_size); + UPDATE artifact_account SET reserved_bytes=reserved_bytes+p_size,credit_bytes=credit_bytes+credit + WHERE owner=p_owner AND ((existing AND p_size<=replaced) OR + legacy_bytes+reserved_bytes-credit_bytes-replaced+p_size<=cap); + IF NOT FOUND THEN RETURN false; END IF; + INSERT INTO artifact_part(service,space,name,part_id,owner,size_bytes,chunk_count, + replacement_credit,replacing,replaces_part_id,replaces_delta_part_id) + VALUES(p_service,p_space,p_name,p_id,p_owner,p_size,p_count,credit,existing, + CASE WHEN existing THEN CASE WHEN p_wal THEN base.delta_part_id ELSE base.checkpoint_part_id END ELSE NULL END, + CASE WHEN existing AND NOT p_wal THEN base.delta_part_id ELSE NULL END); + RETURN true; +EXCEPTION WHEN lock_not_available THEN + RAISE EXCEPTION USING ERRCODE='TC001', MESSAGE='TC2917_BUSY'; +END $$; + +CREATE OR REPLACE FUNCTION artifact_metadata_publish() RETURNS trigger LANGUAGE plpgsql AS $$ +BEGIN + IF NEW.checkpoint_part_id IS NULL THEN RETURN NEW; END IF; + UPDATE artifact_part SET state='published',lease_until='infinity'::timestamptz + WHERE service=NEW.service AND space=NEW.space AND name=NEW.name + AND part_id IN (NEW.checkpoint_part_id,NEW.delta_part_id) AND state='sealed'; + IF NEW.checkpoint_part_id IS NOT NULL AND NOT EXISTS(SELECT 1 FROM artifact_part WHERE service=NEW.service AND space=NEW.space + AND name=NEW.name AND part_id=NEW.checkpoint_part_id AND state='published') THEN RAISE EXCEPTION 'TC2917_LAYOUT: unpublished checkpoint'; END IF; + IF NEW.delta_part_id IS NOT NULL AND NOT EXISTS(SELECT 1 FROM artifact_part WHERE service=NEW.service AND space=NEW.space + AND name=NEW.name AND part_id=NEW.delta_part_id AND state='published') THEN RAISE EXCEPTION 'TC2917_LAYOUT: unpublished WAL'; END IF; + UPDATE artifact_part SET replacing=false WHERE service=NEW.service AND space=NEW.space AND name=NEW.name + AND state='published' AND replaces_part_id IS NULL AND replaces_delta_part_id IS NULL AND replacement_credit=0; + RETURN NEW; +END $$; diff --git a/tinycloud-core/src/duckdb/database.rs b/tinycloud-core/src/duckdb/database.rs index f5ed1864..b07a7e62 100644 --- a/tinycloud-core/src/duckdb/database.rs +++ b/tinycloud-core/src/duckdb/database.rs @@ -4,6 +4,7 @@ //! result formatting, native connection open/close and filesystem work do not. use std::collections::BTreeSet; +use std::io::Read; use std::path::{Path, PathBuf}; use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::Arc; @@ -12,6 +13,7 @@ use dashmap::DashMap; use sqlparser::ast::Statement; use tokio::sync::{mpsc, oneshot}; +use crate::hash::Blake3Hasher; use crate::request_telemetry::{self, QueryTimer, RequestContext}; use super::{ @@ -38,13 +40,20 @@ enum DbMessage { Snapshot { response_tx: oneshot::Sender>, }, - WalSnapshot { + WalSnapshotForCheckpoint { max_bytes: u64, + checkpoint_size: u64, + checkpoint_hash: String, response_tx: oneshot::Sender, DuckDbError>>, }, Wal { response_tx: oneshot::Sender>, DuckDbError>>, }, + WalForCheckpoint { + checkpoint_size: u64, + checkpoint_hash: String, + response_tx: oneshot::Sender>, DuckDbError>>, + }, Shutdown { response_tx: oneshot::Sender<()>, }, @@ -117,16 +126,22 @@ impl DatabaseHandle { .map_err(|_| DuckDbError::Internal("Database actor dropped response".into()))? } - pub async fn wal_snapshot( + /// Copy an owned WAL only while its local main file matches the durable + /// checkpoint; hashing and copying share this actor's operation fence. + pub async fn wal_snapshot_for_checkpoint( &self, max_bytes: u64, + checkpoint_size: u64, + checkpoint_hash: String, ) -> Result, DuckDbError> { let (response_tx, response_rx) = oneshot::channel(); self.tx .send(( request_telemetry::current(), - DbMessage::WalSnapshot { + DbMessage::WalSnapshotForCheckpoint { max_bytes, + checkpoint_size, + checkpoint_hash, response_tx, }, )) @@ -148,6 +163,30 @@ impl DatabaseHandle { .map_err(|_| DuckDbError::Internal("Database actor dropped response".to_string()))? } + /// Copy a WAL only when its local main file still matches the durable + /// checkpoint. Both operations run under the same actor fence. + pub async fn wal_for_checkpoint( + &self, + checkpoint_size: u64, + checkpoint_hash: String, + ) -> Result>, DuckDbError> { + let (response_tx, response_rx) = oneshot::channel(); + self.tx + .send(( + request_telemetry::current(), + DbMessage::WalForCheckpoint { + checkpoint_size, + checkpoint_hash, + response_tx, + }, + )) + .await + .map_err(|_| DuckDbError::Internal("Database actor not available".into()))?; + response_rx + .await + .map_err(|_| DuckDbError::Internal("Database actor dropped response".into()))? + } + /// Stop the actor and close its database before the cache is discarded. pub(crate) async fn shutdown(&self) -> Result<(), DuckDbError> { let (response_tx, response_rx) = oneshot::channel(); @@ -245,13 +284,14 @@ pub(crate) fn spawn_actor_with_parts( e )))); } - DbMessage::WalSnapshot { response_tx, .. } => { + DbMessage::WalSnapshotForCheckpoint { response_tx, .. } => { let _ = response_tx.send(Err(DuckDbError::Internal(format!( "Failed to open: {}", e )))); } - DbMessage::Wal { response_tx } => { + DbMessage::Wal { response_tx } + | DbMessage::WalForCheckpoint { response_tx, .. } => { let _ = response_tx.send(Err(DuckDbError::Internal(format!( "Failed to open: {}", e @@ -334,17 +374,38 @@ pub(crate) fn spawn_actor_with_parts( handle_snapshot(&conn, &mode, &file_path, &max_memory_per_connection); let _ = response_tx.send(result); } - DbMessage::WalSnapshot { + DbMessage::WalSnapshotForCheckpoint { max_bytes, + checkpoint_size, + checkpoint_hash, response_tx, } => { - let result = handle_wal_snapshot(&mode, &file_path, max_bytes); + let result = handle_wal_snapshot_for_checkpoint( + &mode, + &file_path, + max_bytes, + checkpoint_size, + &checkpoint_hash, + ); let _ = response_tx.send(result); } DbMessage::Wal { response_tx } => { let result = handle_wal(&mode, &file_path); let _ = response_tx.send(result); } + DbMessage::WalForCheckpoint { + checkpoint_size, + checkpoint_hash, + response_tx, + } => { + let result = handle_wal_for_checkpoint( + &mode, + &file_path, + checkpoint_size, + &checkpoint_hash, + ); + let _ = response_tx.send(result); + } DbMessage::Shutdown { response_tx } => { shutdown = Some(response_tx); break; @@ -414,12 +475,14 @@ fn deregister_actor( } } -fn handle_wal_snapshot( +fn handle_wal_snapshot_for_checkpoint( mode: &StorageMode, file_path: &Path, max_bytes: u64, + checkpoint_size: u64, + checkpoint_hash: &str, ) -> Result, DuckDbError> { - if !matches!(mode, StorageMode::File(_)) { + if !checkpoint_matches(mode, file_path, checkpoint_size, checkpoint_hash)? { return Ok(None); } let mut source = match std::fs::File::open(format!("{}.wal", file_path.display())) { @@ -476,6 +539,56 @@ fn handle_snapshot( } } +fn handle_wal_for_checkpoint( + mode: &StorageMode, + file_path: &Path, + checkpoint_size: u64, + checkpoint_hash: &str, +) -> Result>, DuckDbError> { + if !checkpoint_matches(mode, file_path, checkpoint_size, checkpoint_hash)? { + return Ok(None); + } + handle_wal(mode, file_path) +} + +fn checkpoint_matches( + mode: &StorageMode, + file_path: &Path, + checkpoint_size: u64, + checkpoint_hash: &str, +) -> Result { + if !matches!(mode, StorageMode::File(_)) { + return Ok(false); + } + let mut file = + std::fs::File::open(file_path).map_err(|e| DuckDbError::Internal(e.to_string()))?; + if file + .metadata() + .map_err(|e| DuckDbError::Internal(e.to_string()))? + .len() + != checkpoint_size + { + return Ok(false); + } + // DuckDB can place new blocks in the main file while recording only their + // references in the WAL. Such a WAL cannot replay on the older checkpoint. + let mut hasher = Blake3Hasher::new(); + let mut buffer = [0u8; 64 * 1024]; + loop { + let read = file + .read(&mut buffer) + .map_err(|e| DuckDbError::Internal(e.to_string()))?; + if read == 0 { + break; + } + hasher.update(&buffer[..read]); + } + if hasher.finalize().to_cid(0x55).to_string() != checkpoint_hash { + return Ok(false); + } + Ok(true) +} + fn handle_wal(mode: &StorageMode, file_path: &Path) -> Result>, DuckDbError> { if !matches!(mode, StorageMode::File(_)) { return Ok(None); diff --git a/tinycloud-core/src/duckdb/service.rs b/tinycloud-core/src/duckdb/service.rs index 0e16e41b..a9f44122 100644 --- a/tinycloud-core/src/duckdb/service.rs +++ b/tinycloud-core/src/duckdb/service.rs @@ -545,17 +545,46 @@ impl DuckDbService { let key = (space.to_string(), db_name.to_string()); let expected = self.expectation(&key); + let checkpoint = if expected != ArtifactExpectation::Absent { + self.artifact_repository + .load_meta("duckdb", &key.0, db_name) + .await + .map_err(artifact_error_to_duckdb)? + } else { + None + }; + if self .artifact_repository .chunked_checkpoints("duckdb", &key.0, db_name) { - if expected != ArtifactExpectation::Absent { - if let Some(wal) = handle.wal_snapshot(MAX_WAL_DELTA_BYTES as u64).await? { - let saved = self + if let Some(meta) = &checkpoint { + let wal = if meta.checkpoint_size_bytes >= 0 { + handle + .wal_snapshot_for_checkpoint( + MAX_WAL_DELTA_BYTES as u64, + meta.checkpoint_size_bytes as u64, + meta.checkpoint_content_hash.clone(), + ) + .await? + } else { + None + }; + if let Some(wal) = wal { + let saved = match self .artifact_repository .save_wal_snapshot("duckdb", &key.0, db_name, wal, expected) .await - .map_err(artifact_error_to_duckdb)?; + { + Ok(saved) => saved, + Err( + DatabaseArtifactError::MissingCheckpoint + | DatabaseArtifactError::IncrementalPersistenceUnsupported, + ) => { + return self.checkpoint(space, db_name, handle, false).await; + } + Err(error) => return Err(artifact_error_to_duckdb(error)), + }; self.lineage.insert( key, ArtifactExpectation::Derived { @@ -569,11 +598,18 @@ impl DuckDbService { return self.checkpoint(space, db_name, handle, false).await; } - if let Some(wal) = handle - .wal() - .await? - .filter(|wal| wal.len() < MAX_WAL_DELTA_BYTES) - { + let wal = match checkpoint { + Some(meta) if meta.checkpoint_size_bytes >= 0 => { + handle + .wal_for_checkpoint( + meta.checkpoint_size_bytes as u64, + meta.checkpoint_content_hash, + ) + .await? + } + _ => None, + }; + if let Some(wal) = wal.filter(|wal| wal.len() < MAX_WAL_DELTA_BYTES) { match self .artifact_repository .save_delta("duckdb", &space.to_string(), db_name, wal, expected.clone()) @@ -746,9 +782,11 @@ fn duckdb_wal_path(path: &Path) -> PathBuf { /// The artifact size limits answer 507, as for SQL (TC-2961). fn artifact_error_to_duckdb(err: DatabaseArtifactError) -> DuckDbError { match err { - DatabaseArtifactError::WriteLimitExceeded { .. } => DuckDbError::DatabaseSizeLimit, + DatabaseArtifactError::StorageLimit | DatabaseArtifactError::WriteLimitExceeded { .. } => { + DuckDbError::DatabaseSizeLimit + } DatabaseArtifactError::ReadLimitExceeded { .. } => DuckDbError::DatabaseTooLargeToLoad, - DatabaseArtifactError::StorageLimit => DuckDbError::StorageWouldGrow, + DatabaseArtifactError::ReplacementBusy => DuckDbError::Unavailable, err => DuckDbError::Internal(err.to_string()), } } @@ -1729,7 +1767,7 @@ mod tests { ) .await; assert!( - matches!(outcome, Err(DuckDbError::StorageWouldGrow)), + matches!(outcome, Err(DuckDbError::DatabaseSizeLimit)), "{outcome:?}" ); assert!(repo @@ -1824,4 +1862,346 @@ mod tests { assert!(!PathBuf::from(format!("{}.tmp", path.display())).exists()); } } + #[tokio::test] + async fn postgres_chunked_duckdb_legacy_wal_falls_back_to_snapshot() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_duckdb_legacy_wal_falls_back_to_snapshot", + ) + .await + else { + return; + }; + let first = TempDir::new().unwrap(); + let second = TempDir::new().unwrap(); + let third = TempDir::new().unwrap(); + let space = test_space_id("legacy-duckdb-parts"); + let legacy = Arc::new(SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone())); + let source = file_service(&first, legacy); + source + .execute( + &space, + "main", + DuckDbRequest::Execute { + schema: Some(vec!["CREATE TABLE items(id INTEGER)".into()]), + sql: "INSERT INTO items VALUES (1)".into(), + params: vec![], + }, + None, + "tinycloud.duckdb/admin".into(), + false, + ) + .await + .unwrap(); + source + .discard_local_state(&(space.to_string(), "main".into())) + .await + .unwrap(); + let repo = Arc::new( + SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()).with_chunk_limits(Some( + crate::database_artifacts::chunks::ChunkLimits { + artifact_bytes: 4 * 1024 * 1024, + account_bytes: 16 * 1024 * 1024, + }, + )), + ); + let service = file_service(&second, repo.clone()); + for id in [2, 3] { + service + .execute( + &space, + "main", + DuckDbRequest::Execute { + schema: None, + sql: format!("INSERT INTO items VALUES ({id})"), + params: vec![], + }, + None, + "tinycloud.duckdb/write".into(), + false, + ) + .await + .unwrap(); + let meta = repo + .load_meta("duckdb", &space.to_string(), "main") + .await + .unwrap() + .unwrap(); + assert_eq!(meta.storage_mode, "checkpoint-parts-v1"); + assert_eq!(meta.revision, id); + assert_eq!(meta.has_delta(), id == 3); + } + service + .discard_local_state(&(space.to_string(), "main".into())) + .await + .unwrap(); + let reopened = file_service(&third, repo); + let result = reopened + .execute( + &space, + "main", + DuckDbRequest::Query { + sql: "SELECT id FROM items ORDER BY id".into(), + params: vec![], + }, + None, + "tinycloud.duckdb/read".into(), + false, + ) + .await + .unwrap(); + let DuckDbResponse::Query(query) = result.response else { + panic!("query required"); + }; + assert_eq!( + query.rows, + vec![ + vec![DuckDbValue::Integer(1)], + vec![DuckDbValue::Integer(2)], + vec![DuckDbValue::Integer(3)] + ] + ); + reopened + .discard_local_state(&(space.to_string(), "main".into())) + .await + .unwrap(); + } + #[tokio::test] + async fn tc2990_large_file_backed_insert_recovers_all_acknowledged_rows() { + let repo = artifact_repository().await; + let cache = TempDir::new().unwrap(); + let space = test_space_id("large-file-write-recovery"); + let key = (space.to_string(), "main".to_string()); + let service = file_service(&cache, repo.clone()); + service + .execute( + &space, + "main", + DuckDbRequest::Execute { + schema: Some(vec!["CREATE TABLE items(id BIGINT, body VARCHAR)".into()]), + sql: "INSERT INTO items VALUES (0, 'seed')".into(), + params: vec![], + }, + None, + "tinycloud.duckdb/admin".into(), + false, + ) + .await + .unwrap(); + // Hydration guarantees a file-backed actor, including when the native + // size estimate keeps a newly-created database in memory. + service.discard_local_state(&key).await.unwrap(); + let mut expected_count = 1i64; + let mut expected_sum = 0i64; + for (round, rows) in [(1i64, 5i64), (2, 300_000), (3, 5)] { + service.execute(&space,"main",DuckDbRequest::Execute { + schema:None, + sql:format!("INSERT INTO items SELECT {round}*1000000+i, md5(i::VARCHAR) FROM range({rows}) t(i)"), + params:vec![], + },None,"tinycloud.duckdb/write".into(),false).await.unwrap(); + expected_count += rows; + expected_sum += round * 1_000_000 * rows + rows * (rows - 1) / 2; + let fresh = TempDir::new().unwrap(); + let reader = file_service(&fresh, repo.clone()); + let recovered=reader.execute(&space,"main",DuckDbRequest::Query { + sql:"SELECT count(*),sum(id)::BIGINT,sum(CASE WHEN id=0 OR body=md5((id%1000000)::VARCHAR) THEN 1 ELSE 0 END)::BIGINT FROM items".into(), + params:vec![], + },None,"tinycloud.duckdb/read".into(),false).await; + let _ = reader.discard_local_state(&key).await; + let recovered = match recovered { + Ok(result) => result, + Err(error) => { + let _ = service.discard_local_state(&key).await; + panic!("acknowledged round {round} must recover from a fresh cache: {error:?}"); + } + }; + let DuckDbResponse::Query(query) = recovered.response else { + panic!("query required"); + }; + assert_eq!( + query.rows, + vec![vec![ + DuckDbValue::Integer(expected_count), + DuckDbValue::Integer(expected_sum), + DuckDbValue::Integer(expected_count) + ]] + ); + } + service.discard_local_state(&key).await.unwrap(); + let bytes = service.export(&space, "main").await.unwrap(); + let export_dir = TempDir::new().unwrap(); + let path = export_dir.path().join("recovered.duckdb"); + std::fs::write(&path, bytes).unwrap(); + let conn = storage::open_connection(Some(&path), "128MB").unwrap(); + let count: i64 = conn + .query_row("SELECT count(*) FROM items", [], |row| row.get(0)) + .unwrap(); + assert_eq!(count, expected_count); + } + #[tokio::test] + async fn tc2990_wal_requires_both_checkpoint_length_and_cid() { + let repo = artifact_repository().await; + let source_cache = TempDir::new().unwrap(); + let cache = TempDir::new().unwrap(); + let space = test_space_id("wal-checkpoint-identity"); + let key = (space.to_string(), "main".to_string()); + let source = file_service(&source_cache, repo.clone()); + source + .execute( + &space, + "main", + DuckDbRequest::Execute { + schema: Some(vec!["CREATE TABLE items(id INTEGER)".into()]), + sql: "INSERT INTO items VALUES (1)".into(), + params: vec![], + }, + None, + "tinycloud.duckdb/admin".into(), + false, + ) + .await + .unwrap(); + source.discard_local_state(&key).await.unwrap(); + let service = file_service(&cache, repo.clone()); + service + .execute( + &space, + "main", + DuckDbRequest::Execute { + schema: None, + sql: "INSERT INTO items VALUES (2)".into(), + params: vec![], + }, + None, + "tinycloud.duckdb/write".into(), + false, + ) + .await + .unwrap(); + let meta = repo + .load_meta("duckdb", &space.to_string(), "main") + .await + .unwrap() + .unwrap(); + assert!( + meta.has_delta(), + "a small write must retain the WAL fast path" + ); + let handle = service.databases.get(&key).unwrap().clone(); + let wal = handle.wal().await.unwrap().unwrap(); + assert!(handle + .wal_for_checkpoint( + meta.checkpoint_size_bytes as u64 + 1, + meta.checkpoint_content_hash.clone() + ) + .await + .unwrap() + .is_none()); + let wrong = crate::hash::hash(b"different checkpoint") + .to_cid(0x55) + .to_string(); + assert!(handle + .wal_for_checkpoint(meta.checkpoint_size_bytes as u64, wrong) + .await + .unwrap() + .is_none()); + let verified = handle + .wal_for_checkpoint( + meta.checkpoint_size_bytes as u64, + meta.checkpoint_content_hash, + ) + .await + .unwrap() + .unwrap(); + assert_eq!( + verified, wal, + "failed identity checks must not checkpoint or truncate the WAL" + ); + service.discard_local_state(&key).await.unwrap(); + } + #[tokio::test] + async fn postgres_chunked_duckdb_large_insert_recovers_owned_checkpoint() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_duckdb_large_insert_recovers_owned_checkpoint", + ) + .await + else { + return; + }; + let repo = Arc::new( + SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()).with_chunk_limits(Some( + crate::database_artifacts::chunks::ChunkLimits { + artifact_bytes: 64 * 1024 * 1024, + account_bytes: 256 * 1024 * 1024, + }, + )), + ); + let cache = TempDir::new().unwrap(); + let space = test_space_id("large-file-write-recovery"); + let key = (space.to_string(), "main".to_string()); + let service = file_service(&cache, repo.clone()); + service + .execute( + &space, + "main", + DuckDbRequest::Execute { + schema: Some(vec!["CREATE TABLE items(id BIGINT, body VARCHAR)".into()]), + sql: "INSERT INTO items VALUES (0, 'seed')".into(), + params: vec![], + }, + None, + "tinycloud.duckdb/admin".into(), + false, + ) + .await + .unwrap(); + // Hydration guarantees a file-backed actor, including when the native + // size estimate keeps a newly-created database in memory. + service.discard_local_state(&key).await.unwrap(); + let mut expected_count = 1i64; + let mut expected_sum = 0i64; + for (round, rows) in [(1i64, 5i64), (2, 300_000), (3, 5)] { + service.execute(&space,"main",DuckDbRequest::Execute { + schema:None, + sql:format!("INSERT INTO items SELECT {round}*1000000+i, md5(i::VARCHAR) FROM range({rows}) t(i)"), + params:vec![], + },None,"tinycloud.duckdb/write".into(),false).await.unwrap(); + expected_count += rows; + expected_sum += round * 1_000_000 * rows + rows * (rows - 1) / 2; + let fresh = TempDir::new().unwrap(); + let reader = file_service(&fresh, repo.clone()); + let recovered=reader.execute(&space,"main",DuckDbRequest::Query { + sql:"SELECT count(*),sum(id)::BIGINT,sum(CASE WHEN id=0 OR body=md5((id%1000000)::VARCHAR) THEN 1 ELSE 0 END)::BIGINT FROM items".into(), + params:vec![], + },None,"tinycloud.duckdb/read".into(),false).await; + let _ = reader.discard_local_state(&key).await; + let recovered = match recovered { + Ok(result) => result, + Err(error) => { + let _ = service.discard_local_state(&key).await; + panic!("acknowledged round {round} must recover from a fresh cache: {error:?}"); + } + }; + let DuckDbResponse::Query(query) = recovered.response else { + panic!("query required"); + }; + assert_eq!( + query.rows, + vec![vec![ + DuckDbValue::Integer(expected_count), + DuckDbValue::Integer(expected_sum), + DuckDbValue::Integer(expected_count) + ]] + ); + } + service.discard_local_state(&key).await.unwrap(); + let bytes = service.export(&space, "main").await.unwrap(); + let export_dir = TempDir::new().unwrap(); + let path = export_dir.path().join("recovered.duckdb"); + std::fs::write(&path, bytes).unwrap(); + let conn = storage::open_connection(Some(&path), "128MB").unwrap(); + let count: i64 = conn + .query_row("SELECT count(*) FROM items", [], |row| row.get(0)) + .unwrap(); + assert_eq!(count, expected_count); + } } diff --git a/tinycloud-core/src/duckdb/types.rs b/tinycloud-core/src/duckdb/types.rs index 9576802a..5fd4df17 100644 --- a/tinycloud-core/src/duckdb/types.rs +++ b/tinycloud-core/src/duckdb/types.rs @@ -462,13 +462,16 @@ pub enum DuckDbError { #[error("Operation would increase database storage")] StorageWouldGrow, /// The database's whole checkpoint exceeds the artifact write limit - /// (TC-2961); it was not saved. + /// (TC-2961) or the artifact or account capacity (TC-2917); it was not + /// saved. #[error("sql_database_size_limit: the write would exceed the database size limit; no change was applied")] DatabaseSizeLimit, /// The stored database exceeds the artifact read limit (TC-2961); it was /// not loaded. #[error("sql_database_size_limit: the database is larger than this node loads; no change was applied")] DatabaseTooLargeToLoad, + #[error("Database artifact replacement is temporarily unavailable")] + Unavailable, #[error("Invalid statement: {0}")] InvalidStatement(String), #[error("Schema error: {0}")] diff --git a/tinycloud-core/src/migrations/m20261113_030000_artifact_part_lifecycle.rs b/tinycloud-core/src/migrations/m20261113_030000_artifact_part_lifecycle.rs new file mode 100644 index 00000000..4456c204 --- /dev/null +++ b/tinycloud-core/src/migrations/m20261113_030000_artifact_part_lifecycle.rs @@ -0,0 +1,45 @@ +use sea_orm::{ConnectionTrait, DbBackend, Statement}; +use sea_orm_migration::prelude::*; + +#[derive(DeriveMigrationName)] +pub struct Migration; + +#[async_trait::async_trait] +impl MigrationTrait for Migration { + async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> { + if manager.get_database_backend() != DbBackend::Postgres { + return Ok(()); + } + let conn = manager.get_connection(); + let previous: String = conn + .query_one(Statement::from_string( + DbBackend::Postgres, + "SHOW lock_timeout".to_string(), + )) + .await? + .ok_or_else(|| DbErr::Migration("missing lock timeout".into()))? + .try_get("", "lock_timeout")?; + conn.execute_unprepared("SET LOCAL lock_timeout = '500ms'") + .await?; + manager + .get_connection() + .execute_unprepared(include_str!("../database_artifacts/part_lifecycle.sql")) + .await?; + conn.execute(Statement::from_sql_and_values( + DbBackend::Postgres, + "SELECT set_config('lock_timeout', $1, true)", + [previous.into()], + )) + .await?; + Ok(()) + } + + async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> { + if manager.get_database_backend() != DbBackend::Postgres { + return Ok(()); + } + Err(DbErr::Migration( + "artifact part lifecycle is forward-only".into(), + )) + } +} diff --git a/tinycloud-core/src/migrations/mod.rs b/tinycloud-core/src/migrations/mod.rs index 70335b73..a1a9491b 100644 --- a/tinycloud-core/src/migrations/mod.rs +++ b/tinycloud-core/src/migrations/mod.rs @@ -41,6 +41,7 @@ pub mod m20261113_000000_database_artifact_external; pub mod m20261113_010000_artifact_chunks; pub mod m20261113_020000_artifact_chunk_wal; +pub mod m20261113_030000_artifact_part_lifecycle; pub struct Migrator; @@ -88,6 +89,7 @@ impl MigratorTrait for Migrator { Box::new(m20261113_000000_database_artifact_external::Migration), Box::new(m20261113_010000_artifact_chunks::Migration), Box::new(m20261113_020000_artifact_chunk_wal::Migration), + Box::new(m20261113_030000_artifact_part_lifecycle::Migration), ] } } diff --git a/tinycloud-core/src/sql/service.rs b/tinycloud-core/src/sql/service.rs index 3ffdf673..21cc6af4 100644 --- a/tinycloud-core/src/sql/service.rs +++ b/tinycloud-core/src/sql/service.rs @@ -1039,7 +1039,7 @@ impl SqlService { return Err(SqlError::StorageWouldGrow); } } - let saved = match self + let outcome = self .bounded( (space, db_name), || SqlBusyStage::ArtifactTransfer, @@ -1047,20 +1047,50 @@ impl SqlService { async { if is_wal { self.artifact_repository - .save_wal_snapshot("sql", &key.0, db_name, snapshot, expected) + .save_wal_snapshot( + "sql", + &key.0, + db_name, + snapshot, + expected.clone(), + ) .await } else { self.artifact_repository - .save_snapshot("sql", &key.0, db_name, snapshot, expected) + .save_snapshot("sql", &key.0, db_name, snapshot, expected.clone()) .await } }, ) + .await; + let outcome = if is_wal + && matches!( + &outcome, + Ok(Err(DatabaseArtifactError::MissingCheckpoint + | DatabaseArtifactError::IncrementalPersistenceUnsupported)) + ) { + // Copying the WAL did not checkpoint the actor. An old-format + // base needs one complete snapshot before WAL parts can attach. + let snapshot = handle.snapshot().await?; + self.bounded( + (space, db_name), + || SqlBusyStage::ArtifactTransfer, + self.bounds.artifact_transfer, + self.artifact_repository + .save_snapshot("sql", &key.0, db_name, snapshot, expected), + ) .await - { + } else { + outcome + }; + let saved = match outcome { Ok(Ok(saved)) => saved, Ok(Err(error)) => { - let lineage_conflict = matches!(error, DatabaseArtifactError::StaleLineage); + let lineage_conflict = matches!( + error, + DatabaseArtifactError::StaleLineage + | DatabaseArtifactError::ReplacementBusy + ); let error = request_artifact_error(space, db_name)(error); if !lineage_conflict { self.back_off_checkpoint(&key, &error); @@ -1284,7 +1314,8 @@ fn request_artifact_error<'a>( fn artifact_error_to_sql(err: DatabaseArtifactError) -> SqlError { match err { - DatabaseArtifactError::StorageLimit => SqlError::StorageWouldGrow, + DatabaseArtifactError::StorageLimit => SqlError::DatabaseSizeLimit, + DatabaseArtifactError::ReplacementBusy => SqlError::ReplacementBusy, DatabaseArtifactError::Db(ref error) if is_transient_error(error) => { tracing::warn!( service = "sql", @@ -6295,4 +6326,483 @@ mod tests { ); } } + #[tokio::test] + async fn postgres_chunked_legacy_wal_falls_back_to_complete_snapshot() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_legacy_wal_falls_back_to_complete_snapshot", + ) + .await + else { + return; + }; + let first = TempDir::new().unwrap(); + let second = TempDir::new().unwrap(); + let third = TempDir::new().unwrap(); + let space = test_space_id("legacy-wal-parts"); + let legacy = Arc::new(SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone())); + let source = SqlService::new(first.path().to_string_lossy().into_owned(), 0, legacy); + source + .execute( + &space, + "main", + SqlRequest::Execute { + schema: Some(vec!["CREATE TABLE items(id INTEGER)".into()]), + sql: "INSERT INTO items VALUES (1)".into(), + params: vec![], + }, + None, + "tinycloud.sql/admin".into(), + ) + .await + .unwrap(); + source + .discard_local_state(&(space.to_string(), "main".into())) + .await + .unwrap(); + let repo = Arc::new( + SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()).with_chunk_limits(Some( + crate::database_artifacts::chunks::ChunkLimits { + artifact_bytes: 1024 * 1024, + account_bytes: 4 * 1024 * 1024, + }, + )), + ); + let service = SqlService::new( + second.path().to_string_lossy().into_owned(), + 0, + repo.clone(), + ); + for id in [2, 3] { + service + .execute( + &space, + "main", + SqlRequest::Execute { + schema: None, + sql: format!("INSERT INTO items VALUES ({id})"), + params: vec![], + }, + None, + "tinycloud.sql/write".into(), + ) + .await + .unwrap(); + let meta = repo + .load_meta("sql", &space.to_string(), "main") + .await + .unwrap() + .unwrap(); + assert_eq!(meta.storage_mode, "checkpoint-parts-v1"); + assert_eq!(meta.revision, id); + assert_eq!( + meta.has_delta(), + id == 3, + "first old-format write must checkpoint before attaching WAL" + ); + assert!(!service + .checkpoint_backoff + .contains_key(&(space.to_string(), "main".into()))); + } + service + .discard_local_state(&(space.to_string(), "main".into())) + .await + .unwrap(); + let reopened = SqlService::new(third.path().to_string_lossy().into_owned(), 0, repo); + let result = reopened + .execute( + &space, + "main", + SqlRequest::Query { + sql: "SELECT id FROM items ORDER BY id".into(), + params: vec![], + max_rows: None, + max_bytes: None, + }, + None, + "tinycloud.sql/read".into(), + ) + .await + .unwrap(); + let SqlResponse::Query(query) = result.response else { + panic!("query required"); + }; + assert_eq!( + query.rows, + vec![ + vec![SqlValue::Integer(1)], + vec![SqlValue::Integer(2)], + vec![SqlValue::Integer(3)] + ] + ); + reopened + .discard_local_state(&(space.to_string(), "main".into())) + .await + .unwrap(); + } + + #[tokio::test] + async fn postgres_chunked_replacement_contention_is_unavailable_without_backoff() { + use sea_orm::{ConnectionTrait, DbBackend, Statement, TransactionTrait}; + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_replacement_contention_is_unavailable_without_backoff", + ) + .await + else { + return; + }; + let repo = Arc::new( + SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()).with_chunk_limits(Some( + crate::database_artifacts::chunks::ChunkLimits { + artifact_bytes: 1024 * 1024, + account_bytes: 4 * 1024 * 1024, + }, + )), + ); + let cache = TempDir::new().unwrap(); + let space = test_space_id("parts-busy"); + let service = SqlService::new(cache.path().to_string_lossy().into_owned(), 0, repo.clone()); + service + .execute( + &space, + "main", + SqlRequest::Execute { + schema: Some(vec!["CREATE TABLE items(id INTEGER)".into()]), + sql: "INSERT INTO items VALUES (1)".into(), + params: vec![], + }, + None, + "tinycloud.sql/admin".into(), + ) + .await + .unwrap(); + let base = repo + .load_meta("sql", &space.to_string(), "main") + .await + .unwrap() + .unwrap(); + let mut racing = (*repo).clone(); + racing.chunk_publish_barrier = Some(Arc::new(tokio::sync::Barrier::new(2))); + let target = space.to_string(); + let upload = tokio::spawn(async move { + let mut snapshot = tempfile::NamedTempFile::new().unwrap(); + std::io::Write::write_all(&mut snapshot, &[9; 1024]).unwrap(); + racing + .save_snapshot( + "sql", + &target, + "main", + snapshot, + ArtifactExpectation::Derived { + revision: base.revision, + checkpoint_content_hash: base.checkpoint_content_hash, + }, + ) + .await + }); + let until = tokio::time::Instant::now() + std::time::Duration::from_secs(2); + loop { + if fixture + .conn + .query_one(Statement::from_string( + DbBackend::Postgres, + "SELECT 1 FROM artifact_part WHERE state='sealed'".to_string(), + )) + .await + .unwrap() + .is_some() + { + break; + } + assert!(tokio::time::Instant::now() < until); + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + let write = || SqlRequest::Execute { + schema: None, + sql: "INSERT INTO items VALUES (2)".into(), + params: vec![], + }; + let refused = service + .execute(&space, "main", write(), None, "tinycloud.sql/write".into()) + .await; + assert!( + matches!(refused, Err(SqlError::ReplacementBusy)), + "{refused:?}" + ); + assert!(!service + .checkpoint_backoff + .contains_key(&(space.to_string(), "main".into()))); + upload.abort(); + assert!(upload.await.unwrap_err().is_cancelled()); + let until = tokio::time::Instant::now() + std::time::Duration::from_secs(2); + loop { + let n: i64 = fixture + .conn + .query_one(Statement::from_string( + DbBackend::Postgres, + "SELECT count(*)::bigint AS n FROM artifact_part WHERE state<>'published'" + .to_string(), + )) + .await + .unwrap() + .unwrap() + .try_get("", "n") + .unwrap(); + if n == 0 { + break; + } + assert!(tokio::time::Instant::now() < until); + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + let locked = fixture.conn.begin().await.unwrap(); + locked + .query_one(Statement::from_sql_and_values( + DbBackend::Postgres, + "SELECT owner FROM artifact_account WHERE owner=artifact_owner($1) FOR UPDATE", + [space.to_string().into()], + )) + .await + .unwrap(); + let busy = tokio::time::timeout( + std::time::Duration::from_secs(2), + service.execute(&space, "main", write(), None, "tinycloud.sql/write".into()), + ) + .await + .unwrap(); + assert!(matches!(busy, Err(SqlError::ReplacementBusy)), "{busy:?}"); + assert!(!service + .checkpoint_backoff + .contains_key(&(space.to_string(), "main".into()))); + locked.rollback().await.unwrap(); + service + .execute(&space, "main", write(), None, "tinycloud.sql/write".into()) + .await + .unwrap(); + let result = service + .execute( + &space, + "main", + SqlRequest::Query { + sql: "SELECT count(*) FROM items".into(), + params: vec![], + max_rows: None, + max_bytes: None, + }, + None, + "tinycloud.sql/read".into(), + ) + .await + .unwrap(); + let SqlResponse::Query(query) = result.response else { + panic!("query required"); + }; + assert_eq!(query.rows, vec![vec![SqlValue::Integer(2)]]); + service + .discard_local_state(&(space.to_string(), "main".into())) + .await + .unwrap(); + } + #[tokio::test] + async fn postgres_chunked_artifact_and_account_refusals_are_size_limits() { + for (artifact_bytes, account_bytes) in [(1, 4 * 1024 * 1024), (4 * 1024 * 1024, 1)] { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_artifact_and_account_refusals_are_size_limits", + ) + .await + else { + return; + }; + let repo = Arc::new( + SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()).with_chunk_limits( + Some(crate::database_artifacts::chunks::ChunkLimits { + artifact_bytes, + account_bytes, + }), + ), + ); + let cache = TempDir::new().unwrap(); + let space = test_space_id("parts-quota-refusal"); + let service = + SqlService::new(cache.path().to_string_lossy().into_owned(), 0, repo.clone()); + for _ in 0..2 { + let result = service + .execute( + &space, + "main", + SqlRequest::Execute { + schema: Some(vec!["CREATE TABLE items(id INTEGER)".into()]), + sql: "INSERT INTO items VALUES (1)".into(), + params: vec![], + }, + None, + "tinycloud.sql/admin".into(), + ) + .await; + assert!( + matches!(result, Err(SqlError::DatabaseSizeLimit)), + "{result:?}" + ); + assert_eq!( + service + .checkpoint_backoff + .get(&(space.to_string(), "main".into())) + .unwrap() + .failures, + 1 + ); + } + assert!(repo + .load_meta("sql", &space.to_string(), "main") + .await + .unwrap() + .is_none()); + } + } + struct LostPartsReplyRepository(LostReplyArtifactRepository); + + #[async_trait] + impl DatabaseArtifactRepository for LostPartsReplyRepository { + async fn load_meta( + &self, + service: &str, + space: &str, + name: &str, + ) -> Result, DatabaseArtifactError> { + self.0.inner.load_meta(service, space, name).await + } + async fn read_chunks( + &self, + service: &str, + space: &str, + name: &str, + meta: &ArtifactMeta, + sink: &mut dyn ArtifactSink, + ) -> Result<(), DatabaseArtifactError> { + self.0 + .inner + .read_chunks(service, space, name, meta, sink) + .await + } + fn chunked_checkpoints(&self, service: &str, space: &str, name: &str) -> bool { + self.0.inner.chunked_checkpoints(service, space, name) + } + async fn save_snapshot( + &self, + service: &str, + space: &str, + name: &str, + snapshot: tempfile::NamedTempFile, + expected: ArtifactExpectation, + ) -> Result { + let saved = self + .0 + .inner + .save_snapshot(service, space, name, snapshot, expected) + .await?; + self.0.lose_reply()?; + Ok(saved) + } + async fn save_wal_snapshot( + &self, + service: &str, + space: &str, + name: &str, + snapshot: tempfile::NamedTempFile, + expected: ArtifactExpectation, + ) -> Result { + let saved = self + .0 + .inner + .save_wal_snapshot(service, space, name, snapshot, expected) + .await?; + self.0.lose_reply()?; + Ok(saved) + } + async fn save( + &self, + service: &str, + space: &str, + name: &str, + payload: Vec, + expected: ArtifactExpectation, + ) -> Result { + self.0.save(service, space, name, payload, expected).await + } + } + + #[tokio::test] + async fn postgres_chunked_lost_publish_reply_remains_outcome_unknown() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_lost_publish_reply_remains_outcome_unknown", + ) + .await + else { + return; + }; + let repo = Arc::new(LostPartsReplyRepository(LostReplyArtifactRepository { + inner: Arc::new( + SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()).with_chunk_limits( + Some(crate::database_artifacts::chunks::ChunkLimits { + artifact_bytes: 1024 * 1024, + account_bytes: 4 * 1024 * 1024, + }), + ), + ), + lost_replies: AtomicU64::new(0), + })); + let cache = TempDir::new().unwrap(); + let space = test_space_id("parts-lost-publish"); + let service = SqlService::new(cache.path().to_string_lossy().into_owned(), 0, repo.clone()) + .with_checkpoint_backoff(Duration::ZERO, Duration::ZERO); + for id in [1, 2] { + repo.0.lost_replies.store(1, Ordering::SeqCst); + let result = service + .execute( + &space, + "main", + SqlRequest::Execute { + schema: Some(vec!["CREATE TABLE IF NOT EXISTS items(id INTEGER)".into()]), + sql: format!("INSERT INTO items VALUES ({id})"), + params: vec![], + }, + None, + "tinycloud.sql/admin".into(), + ) + .await; + assert!( + matches!(result, Err(SqlError::OutcomeUnknown)), + "{result:?}" + ); + // The publish committed. The response must never advertise a safe + // retry, and a fresh cache must recover the acknowledged state. + let result = service + .execute( + &space, + "main", + SqlRequest::Query { + sql: "SELECT count(*) FROM items".into(), + params: vec![], + max_rows: None, + max_bytes: None, + }, + None, + "tinycloud.sql/read".into(), + ) + .await + .unwrap(); + let SqlResponse::Query(query) = result.response else { + panic!("query required"); + }; + assert_eq!(query.rows, vec![vec![SqlValue::Integer(id)]]); + let meta = repo + .load_meta("sql", &space.to_string(), "main") + .await + .unwrap() + .unwrap(); + assert_eq!(meta.has_delta(), id == 2); + } + service + .discard_local_state(&(space.to_string(), "main".into())) + .await + .unwrap(); + } } diff --git a/tinycloud-core/src/sql/types.rs b/tinycloud-core/src/sql/types.rs index 55fd0a70..d2cc9bc8 100644 --- a/tinycloud-core/src/sql/types.rs +++ b/tinycloud-core/src/sql/types.rs @@ -332,6 +332,12 @@ pub enum SqlError { /// A failure after the write was sent is [`SqlError::OutcomeUnknown`]. #[error("unavailable_retryable: database storage temporarily unavailable; retry the request")] Unavailable, + /// A part reservation was positively refused before publication. Staged + /// bookkeeping can have committed, but the request cannot be durable. + #[error( + "unavailable_retryable: database artifact replacement is in progress; retry the request" + )] + ReplacementBusy, /// The connection was lost after the request's write was sent: it may /// or may not have been applied. The client must read the database back /// before deciding to retry (TC-1054). diff --git a/tinycloud-node-server/src/routes/mod.rs b/tinycloud-node-server/src/routes/mod.rs index 6304df89..3d9759e0 100644 --- a/tinycloud-node-server/src/routes/mod.rs +++ b/tinycloud-node-server/src/routes/mod.rs @@ -3698,7 +3698,7 @@ pub(crate) fn sql_error_to_status(err: &SqlError) -> Status { tinycloud_core::request_telemetry::note_failure(FailureNote::internal(Service::Sql)); Status::InternalServerError } - SqlError::Unavailable => Status::ServiceUnavailable, + SqlError::Unavailable | SqlError::ReplacementBusy => Status::ServiceUnavailable, SqlError::OutcomeUnknown => Status::InternalServerError, SqlError::Busy(_) | SqlError::RequestTimeout => Status::ServiceUnavailable, } @@ -4164,6 +4164,7 @@ fn duckdb_error_to_status(err: &DuckDbError) -> Status { DuckDbError::QuotaExceeded => Status::new(429), DuckDbError::StorageWouldGrow => Status::new(402), DuckDbError::DatabaseSizeLimit | DuckDbError::DatabaseTooLargeToLoad => Status::new(507), + DuckDbError::Unavailable => Status::ServiceUnavailable, DuckDbError::IngestError(_) => Status::InternalServerError, DuckDbError::ExportError(_) => Status::InternalServerError, DuckDbError::ImportError(_) => Status::InternalServerError, @@ -8422,6 +8423,39 @@ mod tests { } } + #[test] + fn chunked_sql_refusals_keep_capacity_growth_and_contention_distinct() { + assert_eq!( + sql_error_to_status(&SqlError::DatabaseSizeLimit), + Status::new(507) + ); + assert_eq!( + sql_error_to_status(&SqlError::StorageWouldGrow), + Status::new(402) + ); + assert_eq!( + sql_error_to_status(&SqlError::ReplacementBusy), + Status::ServiceUnavailable + ); + } + + #[cfg(feature = "duckdb")] + #[test] + fn chunked_duckdb_refusals_keep_capacity_growth_and_contention_distinct() { + assert_eq!( + duckdb_error_to_status(&DuckDbError::DatabaseSizeLimit), + Status::new(507) + ); + assert_eq!( + duckdb_error_to_status(&DuckDbError::StorageWouldGrow), + Status::new(402) + ); + assert_eq!( + duckdb_error_to_status(&DuckDbError::Unavailable), + Status::ServiceUnavailable + ); + } + #[tokio::test] async fn sql_request_is_write_classifies_variants() { use tinycloud_core::sql::types::SqlStatement;