diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a25f1b9..a642328b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ - `POST /delegate` and `POST /revoke` check the signature (and, for a renewal lease, the signature of the stored original it renews) before taking any lock. A forged or slow-to-verify request no longer holds the delegation's chain guards, its spaces' write order or a database transaction while it is checked; it is refused at once. A renewal lease whose original is not stored is also refused before any lock. Inside the transaction only the signed time window is checked again. A resubmitted delegation that is already stored is now also signature-checked before it is answered as existing. Details: TC-2986. - Add opt-in PostgreSQL checkpoint parts for SQL persistence, with explicit artifact/account capacities and streamed hydration. A database stored in parts loads in bounded statements and is not held to the single-row read limit (`storage.sql.max_artifact_read_bytes`); a load that keeps the whole database in memory (a DuckDB export) still waits for one of the large-transfer slots. Details: TC-2917. - Policy/v3: minting a session and registering a policy no longer re-verify signatures or re-run the claim, presentation, credential admission and allow evaluations while holding their spaces' write order and row locks. Under the locks they check that the challenge, registration and roots are unchanged; those decisions were made before the locks, at the same time, from the same inputs. Refusal codes are unchanged, except that `credential-evaluation-changed` and `policy-evaluation-changed` can no longer occur. Details: TC-2986. +- Opt-in SQL artifact parts also stream cumulative WAL updates, retaining the checkpoint until a full checkpoint is needed. Details: TC-2917. - SQLite stamp replay deadlines now use the same timestamp encoding as cleanup (TC-2920). - PostgreSQL database artifacts use uncompressed external storage for new writes to avoid repeated decompression during chunked reads (TC-2917). Existing large artifacts require an operator re-toast; verify both storage settings after rollout and follow `scripts/tc2917-README.md` if lock contention deferred the migration. - Encryption: v2 column format with key ids, per-purpose keys and recorded format metadata. Optional strict reads are off by default. Details: TC-2516, TC-2517. diff --git a/docs/operations/chunked-sql-artifacts.md b/docs/operations/chunked-sql-artifacts.md index 142e578c..f6abedf4 100644 --- a/docs/operations/chunked-sql-artifacts.md +++ b/docs/operations/chunked-sql-artifacts.md @@ -1,6 +1,6 @@ # Opt-in SQL checkpoint parts -The first TC-2917 slice stores SQL checkpoints as immutable PostgreSQL parts. +Opt-in SQL persistence stores checkpoints and cumulative WAL as immutable PostgreSQL parts. Each payload statement carries at most 1 MiB. A metadata-only compare-and-swap publishes a sealed part; successful writes retain the existing durability and lineage semantics. Hydration verifies the part CID before publishing its cache. @@ -26,8 +26,12 @@ by this slice. The account capacity is charged to the space owner, across its spaces, including legacy live artifacts, staged uploads and retained parts. Reservations are atomic per owner. Quota refusal uses the SQL storage refusal. -This slice publishes a full checkpoint for each SQL mutation. WAL parts, -conversion of existing artifacts follow separately. +After a checkpoint, file-backed SQL mutations can publish an owned cumulative WAL +smaller than 8 MiB without rewriting the checkpoint. Larger WALs and writes +forbidden from growing use a fresh checkpoint. In-memory databases use +checkpoints. Both kinds of part use the same 1 MiB statement limit, and the +artifact ceiling covers checkpoint plus WAL together. Conversion of existing +artifacts follows separately. Aborted and superseded parts enter a fenced deleting state; reclamation deletes at most one payload chunk per statement and releases its charge only after all bytes are gone. A periodic collector recovers uploads whose 120-second lease @@ -40,7 +44,7 @@ convert published parts back to legacy blobs. Current readers understand both layouts; legacy persistence refuses a database already using parts. Do not enable for prolonged operation until the approved ceiling load acceptance and -remaining migration/WAL slices are complete. On each test release, report +remaining conversion and DuckDB slices are complete. On each test release, report reserved, staged and retained bytes from the following metadata-only inventory: ```sql @@ -101,3 +105,16 @@ per 10-second tick. GC lock acquisition waits at most 50 ms per statement. An owner already above its quota can replace a database with an equal or smaller image; that exception admits no growth and does not free pending shrink capacity for other databases. + +A WAL publication checks the exact durable revision and checkpoint CID. It +replaces only the WAL pointer and logical CID, advancing the layout generation; +the checkpoint pointer and bytes remain unchanged. Readers verify each part CID +and the combined logical CID, then recheck both pointers and the generation +before publishing the cache. A full checkpoint retires both previous parts. +Each retired part releases its charge and associated overlap credit only after +its final chunk is removed, so a partially collected pair stays accounted. + +Roll out WAL-capable readers to every instance before enabling the switch. +Checkpoint-only part readers fail closed on WAL layouts. Schema changes are +forward-only; switching the feature off keeps existing layouts readable and +writable by the current binary, and does not downgrade their format. diff --git a/tinycloud-core/src/database_artifacts.rs b/tinycloud-core/src/database_artifacts.rs index a8929a7e..daf0d9de 100644 --- a/tinycloud-core/src/database_artifacts.rs +++ b/tinycloud-core/src/database_artifacts.rs @@ -676,6 +676,18 @@ pub trait DatabaseArtifactRepository: Send + Sync { Err(DatabaseArtifactError::IncrementalPersistenceUnsupported) } + /// Persist an owned cumulative WAL against the caller's exact checkpoint. + async fn save_wal_snapshot( + &self, + _service: &str, + _space: &str, + _name: &str, + _snapshot: tempfile::NamedTempFile, + _expected: ArtifactExpectation, + ) -> Result { + Err(DatabaseArtifactError::IncrementalPersistenceUnsupported) + } + async fn save( &self, service: &str, @@ -1126,6 +1138,18 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository { .await } + async fn save_wal_snapshot( + &self, + service: &str, + space: &str, + name: &str, + snapshot: tempfile::NamedTempFile, + expected: ArtifactExpectation, + ) -> Result { + self.write_chunk_file(service, space, name, snapshot, expected, ArtifactPart::Wal) + .await + } + async fn save( &self, service: &str, diff --git a/tinycloud-core/src/database_artifacts/chunk_wal.sql b/tinycloud-core/src/database_artifacts/chunk_wal.sql new file mode 100644 index 00000000..c90ffca9 --- /dev/null +++ b/tinycloud-core/src/database_artifacts/chunk_wal.sql @@ -0,0 +1,141 @@ +ALTER TABLE artifact_part ADD COLUMN replaces_delta_part_id text; +ALTER TABLE artifact_part ADD CONSTRAINT artifact_part_replaced_delta_fk + FOREIGN KEY(service,space,name,replaces_delta_part_id) REFERENCES artifact_part(service,space,name,part_id); +CREATE OR REPLACE FUNCTION artifact_layout_guard() RETURNS trigger LANGUAGE plpgsql AS $$ +BEGIN + IF TG_OP='UPDATE' AND OLD.checkpoint_part_id IS NOT NULL THEN + IF NEW.checkpoint_part_id IS NULL OR NEW.storage_mode<>'checkpoint-parts-v1' + OR NEW.layout_generation<>OLD.layout_generation+1 + OR NEW.revision<>OLD.revision+1 + OR (NEW.checkpoint_part_id,NEW.delta_part_id) IS NOT DISTINCT FROM (OLD.checkpoint_part_id,OLD.delta_part_id) THEN + RAISE EXCEPTION 'TC2917_LAYOUT: legacy writes cannot replace checkpoint parts'; + END IF; + END IF; + IF TG_OP<>'DELETE' AND NEW.checkpoint_part_id IS NOT NULL THEN + IF NEW.storage_mode<>'checkpoint-parts-v1' OR NEW.delta_payload IS NOT NULL OR octet_length(NEW.payload)<>0 + OR (NEW.delta_part_id IS NULL)<>(NEW.delta_content_hash IS NULL) + OR (NEW.delta_part_id IS NULL AND NEW.delta_size_bytes<>0) THEN + RAISE EXCEPTION 'TC2917_LAYOUT: invalid checkpoint layout'; + END IF; + IF TG_OP='INSERT' AND NEW.layout_generation<>1 THEN RAISE EXCEPTION 'TC2917_LAYOUT: invalid generation'; END IF; + END IF; + IF TG_OP='DELETE' THEN RETURN OLD; ELSE RETURN NEW; END IF; +END $$; +CREATE FUNCTION artifact_reserve_part(p_service text,p_space text,p_name text,p_id text,p_owner text, + p_size bigint,p_count integer,p_revision bigint,p_hash text,p_wal boolean) +RETURNS boolean LANGUAGE plpgsql AS $$ +DECLARE base record; existing boolean; credit bigint:=0; cap bigint; part_cap bigint; replaced bigint:=0; +BEGIN + IF p_owner<>artifact_owner(p_space) THEN RAISE EXCEPTION 'TC2917_LAYOUT: owner mismatch'; END IF; + PERFORM 1 FROM artifact_account WHERE owner=p_owner FOR UPDATE; + SELECT revision,size_bytes,checkpoint_size_bytes,delta_size_bytes,checkpoint_content_hash,checkpoint_part_id,delta_part_id INTO base FROM database_artifact WHERE service=p_service AND space=p_space AND name=p_name; + existing:=FOUND; + IF (p_revision IS NULL AND existing) OR (p_revision IS NOT NULL AND + (NOT existing OR base.revision<>p_revision OR base.checkpoint_content_hash<>p_hash)) THEN + RAISE EXCEPTION 'TC2917_LAYOUT: superseded reservation'; + END IF; + IF p_wal AND (NOT existing OR base.checkpoint_part_id IS NULL) THEN RAISE EXCEPTION 'TC2917_LAYOUT: missing checkpoint'; END IF; + IF EXISTS(SELECT 1 FROM artifact_part WHERE service=p_service AND space=p_space AND name=p_name AND replacing) THEN RETURN false; END IF; + SELECT account_limit,artifact_limit INTO cap,part_cap FROM artifact_storage_settings WHERE id=1; + IF p_size<0 OR p_count<>((p_size+1048575)/1048576) OR + p_size+(CASE WHEN p_wal THEN base.checkpoint_size_bytes ELSE 0 END)>part_cap THEN RETURN false; END IF; + IF existing THEN replaced:=CASE WHEN p_wal THEN base.delta_size_bytes ELSE base.size_bytes END; END IF; + credit:=LEAST(replaced,p_size); + UPDATE artifact_account SET reserved_bytes=reserved_bytes+p_size,credit_bytes=credit_bytes+credit + WHERE owner=p_owner AND ((existing AND p_size<=replaced) OR + legacy_bytes+reserved_bytes-credit_bytes-replaced+p_size<=cap); + IF NOT FOUND THEN RETURN false; END IF; + INSERT INTO artifact_part(service,space,name,part_id,owner,size_bytes,chunk_count, + replacement_credit,replacing,replaces_part_id,replaces_delta_part_id) + VALUES(p_service,p_space,p_name,p_id,p_owner,p_size,p_count,credit,existing, + CASE WHEN existing THEN CASE WHEN p_wal THEN base.delta_part_id ELSE base.checkpoint_part_id END ELSE NULL END, + CASE WHEN existing AND NOT p_wal THEN base.delta_part_id ELSE NULL END); + RETURN true; +END $$; +CREATE OR REPLACE FUNCTION artifact_reserve(p_service text,p_space text,p_name text,p_id text,p_owner text, + p_size bigint,p_count integer,p_revision bigint,p_hash text) RETURNS boolean LANGUAGE sql AS $$ + SELECT artifact_reserve_part(p_service,p_space,p_name,p_id,p_owner,p_size,p_count,p_revision,p_hash,false) +$$; +CREATE OR REPLACE FUNCTION artifact_reap_one(p_service text,p_space text,p_name text,p_id text, + abort_upload boolean DEFAULT false) RETURNS boolean LANGUAGE plpgsql SET lock_timeout='50ms' AS $$ +DECLARE p artifact_part%ROWTYPE; next_ordinal integer; child_credit bigint; +BEGIN + SELECT * INTO p FROM artifact_part WHERE service=p_service AND space=p_space AND name=p_name AND part_id=p_id; + IF NOT FOUND THEN RETURN false; END IF; + PERFORM 1 FROM artifact_account WHERE owner=p.owner FOR UPDATE; + SELECT * INTO p FROM artifact_part WHERE service=p_service AND space=p_space + AND name=p_name AND part_id=p_id FOR UPDATE; + IF NOT FOUND OR EXISTS(SELECT 1 FROM database_artifact d WHERE d.service=p.service + AND d.space=p.space AND d.name=p.name AND + (d.checkpoint_part_id=p.part_id OR d.delta_part_id=p.part_id)) THEN RETURN false; END IF; + IF NOT abort_upload AND p.state<>'deleting' AND p.lease_until>clock_timestamp() THEN RETURN false; END IF; + UPDATE artifact_part SET state='deleting',lease_until=clock_timestamp() WHERE service=p.service AND space=p.space AND name=p.name AND part_id=p.part_id; + SELECT ordinal INTO next_ordinal FROM artifact_chunk WHERE service=p.service AND space=p.space + AND name=p.name AND part_id=p.part_id ORDER BY ordinal LIMIT 1; + IF FOUND THEN DELETE FROM artifact_chunk WHERE service=p.service AND space=p.space + AND name=p.name AND part_id=p.part_id AND ordinal=next_ordinal; END IF; + IF NOT EXISTS(SELECT 1 FROM artifact_chunk WHERE service=p.service AND space=p.space AND name=p.name AND part_id=p.part_id) THEN + -- Reduce overlapping credit only after this whole old part is gone. + -- Keeping its credit after removing its byte charge would undercount live + -- usage while another retired part is still awaiting collection. + SELECT COALESCE(sum(LEAST(replacement_credit,p.size_bytes)),0) INTO child_credit FROM artifact_part + WHERE service=p.service AND space=p.space AND name=p.name + AND (replaces_part_id=p.part_id OR replaces_delta_part_id=p.part_id); + UPDATE artifact_part SET replacement_credit=replacement_credit-LEAST(replacement_credit,p.size_bytes), + replaces_part_id=CASE WHEN replaces_part_id=p.part_id THEN NULL ELSE replaces_part_id END, + replaces_delta_part_id=CASE WHEN replaces_delta_part_id=p.part_id THEN NULL ELSE replaces_delta_part_id END + WHERE service=p.service AND space=p.space AND name=p.name + AND (replaces_part_id=p.part_id OR replaces_delta_part_id=p.part_id); + UPDATE artifact_part SET replacing=false + WHERE service=p.service AND space=p.space AND name=p.name AND state='published' + AND replaces_part_id IS NULL AND replaces_delta_part_id IS NULL AND replacement_credit=0; + DELETE FROM artifact_part WHERE service=p.service AND space=p.space AND name=p.name AND part_id=p.part_id; + UPDATE artifact_account SET reserved_bytes=reserved_bytes-p.size_bytes, + credit_bytes=credit_bytes-p.replacement_credit-child_credit WHERE owner=p.owner; + END IF; + RETURN true; +END $$; +CREATE OR REPLACE FUNCTION artifact_metadata_retire() RETURNS trigger LANGUAGE plpgsql AS $$ +DECLARE released bigint; +BEGIN + IF TG_OP='DELETE' THEN + SELECT COALESCE(sum(replacement_credit),0) INTO released FROM artifact_part + WHERE service=OLD.service AND space=OLD.space AND name=OLD.name; + UPDATE artifact_part SET state='deleting',lease_until=clock_timestamp(),replacement_credit=0,replacing=false,replaces_part_id=NULL,replaces_delta_part_id=NULL + WHERE service=OLD.service AND space=OLD.space AND name=OLD.name; + IF released>0 THEN UPDATE artifact_account SET credit_bytes=credit_bytes-released + WHERE owner=artifact_owner(OLD.space); END IF; + RETURN OLD; + END IF; + IF OLD.delta_part_id IS NOT NULL AND OLD.delta_part_id IS DISTINCT FROM NEW.delta_part_id THEN + UPDATE artifact_part SET state='deleting',lease_until=clock_timestamp() WHERE service=OLD.service AND space=OLD.space + AND name=OLD.name AND part_id=OLD.delta_part_id; + END IF; + IF OLD.checkpoint_part_id IS DISTINCT FROM NEW.checkpoint_part_id THEN + IF OLD.checkpoint_part_id IS NOT NULL THEN + UPDATE artifact_part SET state='deleting',lease_until=clock_timestamp() WHERE service=OLD.service AND space=OLD.space + AND name=OLD.name AND part_id=OLD.checkpoint_part_id; + ELSE + SELECT replacement_credit INTO released FROM artifact_part WHERE service=NEW.service + AND space=NEW.space AND name=NEW.name AND part_id=NEW.checkpoint_part_id; + UPDATE artifact_part SET replacement_credit=0,replacing=false WHERE service=NEW.service + AND space=NEW.space AND name=NEW.name AND part_id=NEW.checkpoint_part_id; + IF released>0 THEN UPDATE artifact_account SET credit_bytes=credit_bytes-released + WHERE owner=artifact_owner(NEW.space); END IF; + END IF; + END IF; + RETURN NEW; +END $$; +CREATE OR REPLACE FUNCTION artifact_metadata_publish() RETURNS trigger LANGUAGE plpgsql AS $$ +BEGIN + UPDATE artifact_part SET state='published',lease_until='infinity'::timestamptz + WHERE service=NEW.service AND space=NEW.space AND name=NEW.name + AND part_id IN (NEW.checkpoint_part_id,NEW.delta_part_id) AND state='sealed'; + IF NEW.checkpoint_part_id IS NOT NULL AND NOT EXISTS(SELECT 1 FROM artifact_part WHERE service=NEW.service AND space=NEW.space + AND name=NEW.name AND part_id=NEW.checkpoint_part_id AND state='published') THEN RAISE EXCEPTION 'TC2917_LAYOUT: unpublished checkpoint'; END IF; + IF NEW.delta_part_id IS NOT NULL AND NOT EXISTS(SELECT 1 FROM artifact_part WHERE service=NEW.service AND space=NEW.space + AND name=NEW.name AND part_id=NEW.delta_part_id AND state='published') THEN RAISE EXCEPTION 'TC2917_LAYOUT: unpublished WAL'; END IF; + UPDATE artifact_part SET replacing=false WHERE service=NEW.service AND space=NEW.space AND name=NEW.name + AND state='published' AND replaces_part_id IS NULL AND replaces_delta_part_id IS NULL AND replacement_credit=0; + RETURN NEW; +END $$; diff --git a/tinycloud-core/src/database_artifacts/chunks.rs b/tinycloud-core/src/database_artifacts/chunks.rs index a673cbbd..e4d8fa81 100644 --- a/tinycloud-core/src/database_artifacts/chunks.rs +++ b/tinycloud-core/src/database_artifacts/chunks.rs @@ -122,10 +122,33 @@ impl SeaOrmDatabaseArtifactRepository { name: &str, snapshot: tempfile::NamedTempFile, expected: ArtifactExpectation, + ) -> Result { + self.write_chunk_file( + service, + space, + name, + snapshot, + expected, + ArtifactPart::Checkpoint, + ) + .await + } + + pub(super) async fn write_chunk_file( + &self, + service: &str, + space: &str, + name: &str, + snapshot: tempfile::NamedTempFile, + expected: ArtifactExpectation, + kind: ArtifactPart, ) -> Result { if self.conn.get_database_backend() != DbBackend::Postgres { return Err(DatabaseArtifactError::IncrementalPersistenceUnsupported); } + if kind == ArtifactPart::Wal && !matches!(expected, ArtifactExpectation::Derived { .. }) { + return Err(DatabaseArtifactError::StaleLineage); + } let limits = match self.chunk_limits { Some(limits) => limits, None => { @@ -158,6 +181,22 @@ impl SeaOrmDatabaseArtifactRepository { .as_ref() .map(|m| (m.revision, m.checkpoint_content_hash.as_str())), )?; + if kind == ArtifactPart::Wal + && current + .as_ref() + .is_none_or(|m| m.storage_mode != "checkpoint-parts-v1") + { + return Err(DatabaseArtifactError::MissingCheckpoint); + } + let logical_size = if kind == ArtifactPart::Wal { + size.checked_add(current.as_ref().unwrap().checkpoint_size_bytes.max(0) as u64) + .ok_or(DatabaseArtifactError::StorageLimit)? + } else { + size + }; + if logical_size > limits.artifact_bytes || logical_size > i64::MAX as u64 { + return Err(DatabaseArtifactError::StorageLimit); + } let mut random = [0u8; 16]; rand::rngs::OsRng.fill_bytes(&mut random); let part_id = hex::encode(random); @@ -180,7 +219,7 @@ impl SeaOrmDatabaseArtifactRepository { let reserved = self .conn .query_one(statement( - "SELECT artifact_reserve($1,$2,$3,$4,$5,$6,$7,$8,$9) AS admitted", + "SELECT artifact_reserve_part($1,$2,$3,$4,$5,$6,$7,$8,$9,$10) AS admitted", vec![ service.into(), space.into(), @@ -194,6 +233,7 @@ impl SeaOrmDatabaseArtifactRepository { .as_ref() .map(|m| m.checkpoint_content_hash.clone()) .into(), + (kind == ArtifactPart::Wal).into(), ], )) .await? @@ -235,7 +275,7 @@ impl SeaOrmDatabaseArtifactRepository { } if total != size || ordinal != count { return Err(DatabaseArtifactError::SizeMismatch { - part: "checkpoint", + part: kind.as_str(), expected: size, actual: total, }); @@ -284,7 +324,42 @@ WHERE service=$1 AND space=$2 AND name=$3 AND part_id=$4 AND state='uploading' } // One metadata-only statement is the acknowledgement boundary. Parts // are already durable and sealed; no transaction spans the transfer. - let sql = if let Some(previous) = current { + let checkpoint_hash = if kind == ArtifactPart::Wal { + current.as_ref().unwrap().checkpoint_content_hash.clone() + } else { + cid.clone() + }; + let content_hash = if kind == ArtifactPart::Wal { + hash(format!("{checkpoint_hash}:{cid}").as_bytes()) + .to_cid(0x55) + .to_string() + } else { + cid.clone() + }; + let checkpoint_size = if kind == ArtifactPart::Wal { + current.as_ref().unwrap().checkpoint_size_bytes + } else { + size as i64 + }; + let sql = if kind == ArtifactPart::Wal { + let previous = current.as_ref().unwrap(); + values.extend([ + previous.revision.into(), + checkpoint_hash.clone().into(), + content_hash.clone().into(), + (logical_size as i64).into(), + ]); + r#"WITH owner_lock AS MATERIALIZED (SELECT owner FROM artifact_account + WHERE owner=(SELECT owner FROM artifact_part WHERE service=$1 AND space=$2 AND name=$3 AND part_id=$4) FOR UPDATE), + ready AS (SELECT p.part_id FROM artifact_part p CROSS JOIN owner_lock + WHERE service=$1 AND space=$2 AND name=$3 AND part_id=$4 AND state='sealed' FOR UPDATE OF p) +UPDATE database_artifact SET delta_part_id=$4,revision=$5,delta_content_hash=$6, + delta_size_bytes=$7,updated_at=$8,content_hash=$11,size_bytes=$12, + delta_payload=NULL,layout_generation=layout_generation+1 +WHERE service=$1 AND space=$2 AND name=$3 AND revision=$9 AND checkpoint_content_hash=$10 + AND checkpoint_part_id IS NOT NULL AND storage_mode='checkpoint-parts-v1' + AND EXISTS(SELECT 1 FROM ready)"# + } else if let Some(previous) = current { values.extend([ previous.revision.into(), previous.checkpoint_content_hash.into(), @@ -332,12 +407,16 @@ FROM ready ON CONFLICT DO NOTHING"# } Ok(ArtifactMeta { revision, - size_bytes: size as i64, - content_hash: cid.clone(), - checkpoint_content_hash: cid, - checkpoint_size_bytes: size as i64, - delta_content_hash: None, - delta_size_bytes: 0, + size_bytes: logical_size as i64, + content_hash, + checkpoint_content_hash: checkpoint_hash, + checkpoint_size_bytes: checkpoint_size, + delta_content_hash: (kind == ArtifactPart::Wal).then_some(cid), + delta_size_bytes: if kind == ArtifactPart::Wal { + size as i64 + } else { + 0 + }, updated_at: now, storage_mode: "checkpoint-parts-v1".into(), backend: "storage.database".into(), @@ -375,71 +454,93 @@ FROM ready ON CONFLICT DO NOTHING"# vec![service.into(),space.into(),name.into()], )).await?.ok_or(DatabaseArtifactError::ArtifactChanged)?; let generation: i64 = row.try_get("", "layout_generation")?; - let part: Option = row.try_get("", "checkpoint_part_id")?; - let Some(part) = part else { return Ok(false) }; + let checkpoint: Option = row.try_get("", "checkpoint_part_id")?; + let wal: Option = row.try_get("", "delta_part_id")?; + let checkpoint = checkpoint.ok_or(DatabaseArtifactError::ArtifactChanged)?; if row.try_get::("", "revision")? != meta.revision || row.try_get::("", "checkpoint_content_hash")? != meta.checkpoint_content_hash + || wal.is_some() != meta.has_delta() { return Err(DatabaseArtifactError::ArtifactChanged); } - if row - .try_get::>("", "delta_part_id")? - .is_some() - || meta.has_delta() - { - return Err(DatabaseArtifactError::IncrementalPersistenceUnsupported); + let mut parts = vec![( + ArtifactPart::Checkpoint, + checkpoint.clone(), + meta.checkpoint_size_bytes, + meta.checkpoint_content_hash.clone(), + )]; + if let Some(part) = &wal { + parts.push(( + ArtifactPart::Wal, + part.clone(), + meta.delta_size_bytes, + meta.delta_content_hash + .clone() + .ok_or(DatabaseArtifactError::ArtifactChanged)?, + )); } - let manifest=self.conn.query_one(statement( - "SELECT size_bytes,chunk_count,content_hash FROM artifact_part WHERE service=$1 AND space=$2 AND name=$3 AND part_id=$4 AND state='published'", - vec![service.into(),space.into(),name.into(),part.clone().into()], - )).await?.ok_or(DatabaseArtifactError::ArtifactChanged)?; - let size = manifest.try_get::("", "size_bytes")?; - let count = manifest.try_get::("", "chunk_count")?; - let cid = manifest.try_get::("", "content_hash")?; - if size != meta.checkpoint_size_bytes - || cid != meta.checkpoint_content_hash - || size < 0 - || count as u64 != (size as u64).div_ceil(CHUNK_BYTES as u64) - { - return Err(DatabaseArtifactError::ArtifactChanged); - } - sink.begin_part(ArtifactPart::Checkpoint).await?; - sink.size_hint(ArtifactPart::Checkpoint, size as u64); - let mut hasher = Blake3Hasher::new(); - let mut total = 0u64; - for ordinal in 0..count { - let row=self.conn.query_one(statement( - "SELECT payload FROM artifact_chunk WHERE service=$1 AND space=$2 AND name=$3 AND part_id=$4 AND ordinal=$5", - vec![service.into(),space.into(),name.into(),part.clone().into(),ordinal.into()], + for (kind, part, size, cid) in parts { + let manifest=self.conn.query_one(statement( + "SELECT size_bytes,chunk_count,content_hash FROM artifact_part WHERE service=$1 AND space=$2 AND name=$3 AND part_id=$4 AND state='published'", + vec![service.into(),space.into(),name.into(),part.clone().into()], )).await?.ok_or(DatabaseArtifactError::ArtifactChanged)?; - let bytes: Vec = row.try_get("", "payload")?; - let expected = (size as u64 - total).min(CHUNK_BYTES as u64); - if bytes.len() as u64 != expected { - return Err(DatabaseArtifactError::SizeMismatch { - part: "checkpoint", - expected, - actual: bytes.len() as u64, + let count = manifest.try_get::("", "chunk_count")?; + if size < 0 + || manifest.try_get::("", "size_bytes")? != size + || manifest.try_get::("", "content_hash")? != cid + || count as u64 != (size as u64).div_ceil(CHUNK_BYTES as u64) + { + return Err(DatabaseArtifactError::ArtifactChanged); + } + sink.begin_part(kind).await?; + sink.size_hint(kind, size as u64); + let mut hasher = Blake3Hasher::new(); + let mut total = 0u64; + for ordinal in 0..count { + let row=self.conn.query_one(statement( + "SELECT payload FROM artifact_chunk WHERE service=$1 AND space=$2 AND name=$3 AND part_id=$4 AND ordinal=$5", + vec![service.into(),space.into(),name.into(),part.clone().into(),ordinal.into()], + )).await?.ok_or(DatabaseArtifactError::ArtifactChanged)?; + let bytes: Vec = row.try_get("", "payload")?; + let expected = (size as u64 - total).min(CHUNK_BYTES as u64); + if bytes.len() as u64 != expected { + return Err(DatabaseArtifactError::SizeMismatch { + part: kind.as_str(), + expected, + actual: bytes.len() as u64, + }); + } + hasher.update(&bytes); + sink.write(&bytes).await?; + total += bytes.len() as u64; + } + if hasher.finalize().to_cid(0x55).to_string() != cid { + tracing::warn!( + event = "artifact_part_hash_mismatch", + "Artifact part integrity check failed" + ); + return Err(DatabaseArtifactError::HashMismatch { + part: kind.as_str(), }); } - hasher.update(&bytes); - sink.write(&bytes).await?; - total += bytes.len() as u64; + sink.finish_part().await?; } - if hasher.finalize().to_cid(0x55).to_string() != cid || cid != meta.content_hash { - tracing::warn!( - event = "artifact_part_hash_mismatch", - "Artifact part integrity check failed" - ); - return Err(DatabaseArtifactError::HashMismatch { part: "checkpoint" }); + let logical = match &meta.delta_content_hash { + Some(delta) => hash(format!("{}:{delta}", meta.checkpoint_content_hash).as_bytes()) + .to_cid(0x55) + .to_string(), + None => meta.checkpoint_content_hash.clone(), + }; + if logical != meta.content_hash { + return Err(DatabaseArtifactError::HashMismatch { part: "content" }); } let unchanged=self.conn.query_one(statement( - "SELECT 1 AS present FROM database_artifact WHERE service=$1 AND space=$2 AND name=$3 AND revision=$4 AND checkpoint_content_hash=$5 AND checkpoint_part_id=$6 AND layout_generation=$7", - vec![service.into(),space.into(),name.into(),meta.revision.into(),cid.into(),part.into(),generation.into()], + "SELECT 1 AS present FROM database_artifact WHERE service=$1 AND space=$2 AND name=$3 AND revision=$4 AND checkpoint_content_hash=$5 AND checkpoint_part_id=$6 AND layout_generation=$7 AND delta_part_id IS NOT DISTINCT FROM $8", + vec![service.into(),space.into(),name.into(),meta.revision.into(),meta.checkpoint_content_hash.clone().into(),checkpoint.into(),generation.into(),wal.into()], )).await?.is_some(); if !unchanged { return Err(DatabaseArtifactError::ArtifactChanged); - }; - sink.finish_part().await?; + } Ok(true) } } @@ -1862,4 +1963,267 @@ mod tests { drop(held); assert_eq!(waiting.await.unwrap().unwrap(), size); } + + #[tokio::test] + async fn postgres_chunked_wal_lifecycle_quota_and_integrity() { + use crate::database_artifacts::test_support::ReadAll; + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_wal_lifecycle_quota_and_integrity", + ) + .await + else { + return; + }; + let space = "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000001:wal"; + let mut repo = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()) + .with_chunk_limits(Some(ChunkLimits { + artifact_bytes: 8 * CHUNK_BYTES as u64, + account_bytes: 8 * CHUNK_BYTES as u64, + })); + repo.chunk_reclaim_paused = true; + let checkpoint = vec![1; CHUNK_BYTES + 17]; + let first = repo + .save_snapshot( + "sql", + space, + "main", + snapshot(&checkpoint), + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + let expectation = |meta: &ArtifactMeta| ArtifactExpectation::Derived { + revision: meta.revision, + checkpoint_content_hash: meta.checkpoint_content_hash.clone(), + }; + let wal = vec![2; CHUNK_BYTES + 23]; + let second = repo + .save_wal_snapshot("sql", space, "main", snapshot(&wal), expectation(&first)) + .await + .unwrap(); + assert_eq!( + second.checkpoint_content_hash, + first.checkpoint_content_hash + ); + assert_eq!(second.size_bytes, (checkpoint.len() + wal.len()) as i64); + let durable = repo.read_all("sql", space, "main").await.unwrap().unwrap(); + assert_eq!(durable.payload, checkpoint); + assert_eq!(durable.delta_payload.as_ref(), Some(&wal)); + assert!(matches!( + repo.save_wal_snapshot("sql", space, "main", snapshot(&[9]), expectation(&first)) + .await, + Err(DatabaseArtifactError::StaleLineage) + )); + assert!(matches!( + repo.save_wal_snapshot( + "sql", + space, + "main", + snapshot(&[9]), + ArtifactExpectation::Any + ) + .await, + Err(DatabaseArtifactError::StaleLineage) + )); + let smaller = vec![3; 127]; + let third = repo + .save_wal_snapshot( + "sql", + space, + "main", + snapshot(&smaller), + expectation(&second), + ) + .await + .unwrap(); + assert_eq!( + ledger(&fixture).await, + ( + (checkpoint.len() + wal.len() + smaller.len()) as i64, + smaller.len() as i64 + ) + ); + repo.reap_unused_parts(64).await.unwrap(); + assert_eq!( + ledger(&fixture).await, + ((checkpoint.len() + smaller.len()) as i64, 0) + ); + let replacement = vec![4; CHUNK_BYTES + 7]; + let fourth = repo + .save_snapshot( + "sql", + space, + "main", + snapshot(&replacement), + expectation(&third), + ) + .await + .unwrap(); + assert!(!fourth.has_delta()); + let credit = replacement.len().min(checkpoint.len() + smaller.len()) as i64; + assert_eq!( + ledger(&fixture).await, + ( + (checkpoint.len() + smaller.len() + replacement.len()) as i64, + credit + ) + ); + // Delete the checkpoint first. Its overlapping credit is released + // with its byte charge; the remaining old WAL stays fully charged. + let old: String = fixture + .conn + .query_one(statement( + "SELECT part_id FROM artifact_part WHERE content_hash=$1", + vec![first.checkpoint_content_hash.into()], + )) + .await + .unwrap() + .unwrap() + .try_get("", "part_id") + .unwrap(); + for _ in 0..2 { + fixture + .conn + .execute(statement( + "SELECT artifact_reap_one('sql',$1,'main',$2)", + vec![space.into(), old.clone().into()], + )) + .await + .unwrap(); + } + assert_eq!( + ledger(&fixture).await, + ((smaller.len() + replacement.len()) as i64, 0) + ); + repo.reap_unused_parts(64).await.unwrap(); + assert_eq!(ledger(&fixture).await, (replacement.len() as i64, 0)); + assert_eq!( + repo.read_all("sql", space, "main") + .await + .unwrap() + .unwrap() + .payload, + replacement + ); + // An authorized test mutation corrupts the WAL; reads refuse it. + let fifth = repo + .save_wal_snapshot( + "sql", + space, + "main", + snapshot(&[5; 127]), + expectation(&fourth), + ) + .await + .unwrap(); + fixture + .conn + .execute_unprepared("ALTER TABLE artifact_chunk DISABLE TRIGGER artifact_chunk_guard") + .await + .unwrap(); + fixture.conn.execute(statement("UPDATE artifact_chunk SET payload=decode(repeat('00',127),'hex') WHERE part_id=(SELECT delta_part_id FROM database_artifact)",vec![])).await.unwrap(); + fixture + .conn + .execute_unprepared("ALTER TABLE artifact_chunk ENABLE TRIGGER artifact_chunk_guard") + .await + .unwrap(); + let mut sink = MemorySink::default(); + assert!(matches!( + repo.read_chunks("sql", space, "main", &fifth, &mut sink) + .await, + Err(DatabaseArtifactError::HashMismatch { part: "wal" }) + )); + } + #[tokio::test] + async fn postgres_chunked_wal_combined_limits_and_publish_fence() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_wal_combined_limits_and_publish_fence", + ) + .await + else { + return; + }; + let space = "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000001:wal-cap"; + let repo = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()).with_chunk_limits( + Some(ChunkLimits { + artifact_bytes: 1024, + account_bytes: 1024, + }), + ); + let expectation = |meta: &ArtifactMeta| ArtifactExpectation::Derived { + revision: meta.revision, + checkpoint_content_hash: meta.checkpoint_content_hash.clone(), + }; + let first = repo + .save_snapshot( + "sql", + space, + "main", + snapshot(&[1; 768]), + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + assert!(matches!( + repo.save_wal_snapshot( + "sql", + space, + "main", + snapshot(&[2; 257]), + expectation(&first) + ) + .await, + Err(DatabaseArtifactError::StorageLimit) + )); + let second = repo + .save_wal_snapshot( + "sql", + space, + "main", + snapshot(&[2; 256]), + expectation(&first), + ) + .await + .unwrap(); + assert_eq!(ledger(&fixture).await, (1024, 0)); + let third = repo + .save_wal_snapshot( + "sql", + space, + "main", + snapshot(&[3; 128]), + expectation(&second), + ) + .await + .unwrap(); + repo.reap_unused_parts(64).await.unwrap(); + assert_eq!(ledger(&fixture).await, (896, 0)); + let barrier = std::sync::Arc::new(tokio::sync::Barrier::new(2)); + let mut racing = repo.clone(); + racing.chunk_publish_barrier = Some(barrier.clone()); + let expected = expectation(&third); + let upload = tokio::spawn(async move { + racing + .save_wal_snapshot("sql", space, "main", snapshot(&[4; 64]), expected) + .await + }); + await_replacement(&fixture).await; + // Administrative injection advances the durable revision after seal. + fixture.conn.execute_unprepared("ALTER TABLE database_artifact DISABLE TRIGGER artifact_layout_guard; UPDATE database_artifact SET revision=revision+1; ALTER TABLE database_artifact ENABLE TRIGGER artifact_layout_guard;").await.unwrap(); + barrier.wait().await; + let outcome = upload.await.unwrap(); + assert!( + matches!(outcome, Err(DatabaseArtifactError::StaleLineage)), + "{outcome:?}" + ); + let until = tokio::time::Instant::now() + std::time::Duration::from_secs(2); + while ledger(&fixture).await != (896, 0) { + repo.reap_unused_parts(64).await.unwrap(); + assert!(tokio::time::Instant::now() < until); + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + let durable = repo.load_meta("sql", space, "main").await.unwrap().unwrap(); + assert_eq!(durable.delta_content_hash, third.delta_content_hash); + assert_eq!(durable.revision, third.revision + 1); + } } diff --git a/tinycloud-core/src/migrations/m20261113_020000_artifact_chunk_wal.rs b/tinycloud-core/src/migrations/m20261113_020000_artifact_chunk_wal.rs new file mode 100644 index 00000000..8949a7b5 --- /dev/null +++ b/tinycloud-core/src/migrations/m20261113_020000_artifact_chunk_wal.rs @@ -0,0 +1,45 @@ +use sea_orm::{ConnectionTrait, DbBackend, Statement}; +use sea_orm_migration::prelude::*; + +#[derive(DeriveMigrationName)] +pub struct Migration; + +#[async_trait::async_trait] +impl MigrationTrait for Migration { + async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> { + if manager.get_database_backend() != DbBackend::Postgres { + return Ok(()); + } + let conn = manager.get_connection(); + let previous: String = conn + .query_one(Statement::from_string( + DbBackend::Postgres, + "SHOW lock_timeout".to_string(), + )) + .await? + .ok_or_else(|| DbErr::Migration("missing lock timeout".into()))? + .try_get("", "lock_timeout")?; + conn.execute_unprepared("SET LOCAL lock_timeout = '500ms'") + .await?; + manager + .get_connection() + .execute_unprepared(include_str!("../database_artifacts/chunk_wal.sql")) + .await?; + conn.execute(Statement::from_sql_and_values( + DbBackend::Postgres, + "SELECT set_config('lock_timeout', $1, true)", + [previous.into()], + )) + .await?; + Ok(()) + } + + async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> { + if manager.get_database_backend() != DbBackend::Postgres { + return Ok(()); + } + Err(DbErr::Migration( + "artifact WAL part storage is forward-only".into(), + )) + } +} diff --git a/tinycloud-core/src/migrations/mod.rs b/tinycloud-core/src/migrations/mod.rs index 4bbfd850..70335b73 100644 --- a/tinycloud-core/src/migrations/mod.rs +++ b/tinycloud-core/src/migrations/mod.rs @@ -40,6 +40,7 @@ pub mod m20261112_000000_renewal_identity; pub mod m20261113_000000_database_artifact_external; pub mod m20261113_010000_artifact_chunks; +pub mod m20261113_020000_artifact_chunk_wal; pub struct Migrator; @@ -86,6 +87,7 @@ impl MigratorTrait for Migrator { Box::new(m20261112_000000_renewal_identity::Migration), Box::new(m20261113_000000_database_artifact_external::Migration), Box::new(m20261113_010000_artifact_chunks::Migration), + Box::new(m20261113_020000_artifact_chunk_wal::Migration), ] } } diff --git a/tinycloud-core/src/sql/database.rs b/tinycloud-core/src/sql/database.rs index 4aeb1a52..c2cb08a4 100644 --- a/tinycloud-core/src/sql/database.rs +++ b/tinycloud-core/src/sql/database.rs @@ -51,6 +51,10 @@ enum DbMessage { Snapshot { response_tx: oneshot::Sender>, }, + WalSnapshot { + max_bytes: u64, + response_tx: oneshot::Sender, SqlError>>, + }, Checkpoint { response_tx: oneshot::Sender, SqlError>>, }, @@ -189,6 +193,26 @@ impl DatabaseHandle { .map_err(|_| SqlError::Internal("Database actor dropped response".into()))? } + pub async fn wal_snapshot( + &self, + max_bytes: u64, + ) -> Result, SqlError> { + let (response_tx, response_rx) = oneshot::channel(); + self.tx + .send(( + request_telemetry::current(), + DbMessage::WalSnapshot { + max_bytes, + response_tx, + }, + )) + .await + .map_err(|_| SqlError::Internal("Database actor not available".into()))?; + response_rx + .await + .map_err(|_| SqlError::Internal("Database actor dropped response".into()))? + } + pub async fn checkpoint(&self) -> Result, SqlError> { let (response_tx, response_rx) = oneshot::channel(); self.tx @@ -431,6 +455,14 @@ pub fn spawn_actor_with_policy( } let _ = response_tx.send(result); } + DbMessage::WalSnapshot { + max_bytes, + response_tx, + } => { + // Actor ownership fences the copy from concurrent writes/checkpoints. + let result = handle_wal_snapshot(&mode, &file_path, max_bytes); + let _ = response_tx.send(result); + } DbMessage::Checkpoint { response_tx } => { let result = handle_checkpoint(&conn, &mode, &file_path); if result.is_ok() { @@ -477,6 +509,33 @@ fn existing_ancestor(path: &Path) -> &Path { .unwrap_or_else(|| Path::new(".")) } +fn handle_wal_snapshot( + mode: &StorageMode, + file_path: &Path, + max_bytes: u64, +) -> Result, SqlError> { + if !matches!(mode, StorageMode::File(_)) { + return Ok(None); + } + let mut source = match std::fs::File::open(format!("{}-wal", file_path.display())) { + Ok(file) => file, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None), + Err(error) => return Err(SqlError::Internal(error.to_string())), + }; + let size = source + .metadata() + .map_err(|e| SqlError::Internal(e.to_string()))? + .len(); + if size == 0 || size >= max_bytes { + return Ok(None); + } + let mut snapshot = + tempfile::NamedTempFile::new().map_err(|e| SqlError::Internal(e.to_string()))?; + std::io::copy(&mut source, snapshot.as_file_mut()) + .map_err(|e| SqlError::Internal(e.to_string()))?; + Ok(Some(snapshot)) +} + fn handle_wal(mode: &StorageMode, file_path: &Path) -> Result>, SqlError> { if !matches!(mode, StorageMode::File(_)) { return Ok(None); diff --git a/tinycloud-core/src/sql/service.rs b/tinycloud-core/src/sql/service.rs index 9bd46190..3ffdf673 100644 --- a/tinycloud-core/src/sql/service.rs +++ b/tinycloud-core/src/sql/service.rs @@ -1013,7 +1013,16 @@ impl SqlService { .artifact_repository .chunked_checkpoints("sql", &key.0, db_name) { - let snapshot = handle.snapshot().await?; + let wal = if growth != GrowthPolicy::Forbid && expected != ArtifactExpectation::Absent { + handle.wal_snapshot(MAX_WAL_DELTA_BYTES as u64).await? + } else { + None + }; + let is_wal = wal.is_some(); + let snapshot = match wal { + Some(wal) => wal, + None => handle.snapshot().await?, + }; let bytes = snapshot .as_file() .metadata() @@ -1035,8 +1044,17 @@ impl SqlService { (space, db_name), || SqlBusyStage::ArtifactTransfer, self.bounds.artifact_transfer, - self.artifact_repository - .save_snapshot("sql", &key.0, db_name, snapshot, expected), + async { + if is_wal { + self.artifact_repository + .save_wal_snapshot("sql", &key.0, db_name, snapshot, expected) + .await + } else { + self.artifact_repository + .save_snapshot("sql", &key.0, db_name, snapshot, expected) + .await + } + }, ) .await { @@ -2740,6 +2758,17 @@ mod tests { .unwrap() .try_get::("", "checkpoint_part_id") .unwrap(); + let checkpoint_bytes: Vec> = fixture + .conn + .query_all(sea_orm::Statement::from_string( + sea_orm::DbBackend::Postgres, + "SELECT payload FROM artifact_chunk ORDER BY ordinal".to_string(), + )) + .await + .unwrap() + .into_iter() + .map(|row| row.try_get("", "payload").unwrap()) + .collect(); service .execute( &space, @@ -2810,21 +2839,39 @@ mod tests { )).await.unwrap().unwrap(); assert_eq!( state.try_get::("", "chunks").unwrap(), - (current_meta.checkpoint_size_bytes as u64).div_ceil(1024 * 1024) as i64 + ((current_meta.checkpoint_size_bytes as u64).div_ceil(1024 * 1024) + + (current_meta.delta_size_bytes as u64).div_ceil(1024 * 1024)) as i64 ); let old_chunks = fixture .conn .query_one(sea_orm::Statement::from_sql_and_values( sea_orm::DbBackend::Postgres, "SELECT count(*)::bigint AS chunks FROM artifact_chunk WHERE part_id=$1", - [superseded_part.into()], + [superseded_part.clone().into()], )) .await .unwrap() .unwrap() .try_get::("", "chunks") .unwrap(); - assert_eq!(old_chunks, 0, "superseded checkpoint must be reclaimed"); + assert_eq!( + old_chunks, + (current_meta.checkpoint_size_bytes as u64).div_ceil(1024 * 1024) as i64, + "WAL publication must retain the base checkpoint" + ); + let unchanged_bytes: Vec> = fixture + .conn + .query_all(sea_orm::Statement::from_sql_and_values( + sea_orm::DbBackend::Postgres, + "SELECT payload FROM artifact_chunk WHERE part_id=$1 ORDER BY ordinal", + [superseded_part.clone().into()], + )) + .await + .unwrap() + .into_iter() + .map(|row| row.try_get("", "payload").unwrap()) + .collect(); + assert_eq!(unchanged_bytes, checkpoint_bytes); assert!(state.try_get::("", "max_bytes").unwrap() <= 1024 * 1024); assert_eq!(state.try_get::("", "compressed").unwrap(), 0); let meta = repo @@ -2833,7 +2880,52 @@ mod tests { .unwrap() .unwrap(); assert_eq!(meta.revision, 2); - assert_eq!(meta.delta_size_bytes, 0); + assert!(meta.delta_size_bytes > 0); + let old_wal: String = fixture + .conn + .query_one(sea_orm::Statement::from_string( + sea_orm::DbBackend::Postgres, + "SELECT delta_part_id FROM database_artifact".to_string(), + )) + .await + .unwrap() + .unwrap() + .try_get("", "delta_part_id") + .unwrap(); + let handle = reopened + .databases + .get(&(space.to_string(), "main".into())) + .unwrap() + .clone(); + reopened + .persist_write(&space, "main", &handle, GrowthPolicy::Forbid) + .await + .unwrap(); + repo.reap_unused_parts(64).await.unwrap(); + let old_count: i64 = fixture + .conn + .query_one(sea_orm::Statement::from_sql_and_values( + sea_orm::DbBackend::Postgres, + "SELECT count(*)::bigint AS n FROM artifact_chunk WHERE part_id IN ($1,$2)", + [superseded_part.into(), old_wal.into()], + )) + .await + .unwrap() + .unwrap() + .try_get("", "n") + .unwrap(); + assert_eq!( + old_count, 0, + "forced checkpoint must reclaim both old parts" + ); + let fresh = repo + .load_meta("sql", &space.to_string(), "main") + .await + .unwrap() + .unwrap(); + assert!(!fresh.has_delta()); + assert_eq!(fresh.revision, 3); + assert_eq!(before, reopened.export(&space, "main").await.unwrap()); reopened .discard_local_state(&(space.to_string(), "main".into())) .await diff --git a/tinycloud-core/src/sql_sizes.rs b/tinycloud-core/src/sql_sizes.rs index 438f643f..3e795ec2 100644 --- a/tinycloud-core/src/sql_sizes.rs +++ b/tinycloud-core/src/sql_sizes.rs @@ -141,6 +141,24 @@ impl DatabaseArtifactRepository for SizeTrackingArtifactRepository { Ok(saved) } + async fn save_wal_snapshot( + &self, + service: &str, + space: &str, + name: &str, + snapshot: tempfile::NamedTempFile, + expected: ArtifactExpectation, + ) -> Result { + let saved = self + .inner + .save_wal_snapshot(service, space, name, snapshot, expected) + .await?; + self.sizes + .update(service, space, name, saved.size_bytes.max(0) as u64) + .await; + Ok(saved) + } + async fn save( &self, service: &str,