diff --git a/CHANGELOG.md b/CHANGELOG.md index 3d5ee5445..08b58464f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ - SQL and DuckDB: a whole-checkpoint save over 8 MiB no longer opens a transaction on PostgreSQL. Its metadata read and its compare-and-swap write each run as one autocommit statement under the base `statement_timeout` (30 s by default), so no transaction or row lock is held while the payload uploads. A save the server cancels at that timeout applied nothing and now answers a retryable `503`, as smaller saves already did, instead of `500 outcome_unknown`. `database.bulk_statement_timeout_ms` is removed: a value still set in the file or environment is ignored, and the node logs a warning naming it at startup. Details: TC-2961. - `/invoke`: a KV invocation that writes (`kv/put` or `kv/del`) and also reads (`kv/get`, `kv/list`, `kv/metadata` or `tinycloud.capabilities/read`) is refused with the same `501` as before, but now before any work: nothing is written and the signed request is not spent. Before, the write committed and only then was the invocation answered `501`, since `/invoke` returns one outcome. Internal callers that still mix the two get their reads after the write commits, holding no database transaction, row lock or write order. A `kv/get` answers the value the invocation wrote. If such a read fails, the error says the write was saved. Details: TC-2986. - `POST /delegate` and `POST /revoke` check the signature (and, for a renewal lease, the signature of the stored original it renews) before taking any lock. A forged or slow-to-verify request no longer holds the delegation's chain guards, its spaces' write order or a database transaction while it is checked; it is refused at once. A renewal lease whose original is not stored is also refused before any lock. Inside the transaction only the signed time window is checked again. A resubmitted delegation that is already stored is now also signature-checked before it is answered as existing. Details: TC-2986. +- Add opt-in PostgreSQL checkpoint parts for SQL persistence, with explicit artifact/account capacities and streamed hydration. A database stored in parts loads in bounded statements and is not held to the single-row read limit (`storage.sql.max_artifact_read_bytes`); a load that keeps the whole database in memory (a DuckDB export) still waits for one of the large-transfer slots. Details: TC-2917. - SQLite stamp replay deadlines now use the same timestamp encoding as cleanup (TC-2920). - PostgreSQL database artifacts use uncompressed external storage for new writes to avoid repeated decompression during chunked reads (TC-2917). Existing large artifacts require an operator re-toast; verify both storage settings after rollout and follow `scripts/tc2917-README.md` if lock contention deferred the migration. - Encryption: v2 column format with key ids, per-purpose keys and recorded format metadata. Optional strict reads are off by default. Details: TC-2516, TC-2517. diff --git a/docs/operations/chunked-sql-artifacts.md b/docs/operations/chunked-sql-artifacts.md new file mode 100644 index 000000000..142e578ca --- /dev/null +++ b/docs/operations/chunked-sql-artifacts.md @@ -0,0 +1,103 @@ +# Opt-in SQL checkpoint parts + +The first TC-2917 slice stores SQL checkpoints as immutable PostgreSQL parts. +Each payload statement carries at most 1 MiB. A metadata-only compare-and-swap +publishes a sealed part; successful writes retain the existing durability and +lineage semantics. Hydration verifies the part CID before publishing its cache. + +A database stored in parts is not held to the single-row read limit +(`storage.sql.max_artifact_read_bytes`, TC-2961): each statement reads at most +1 MiB, hydration stages the parts in files beside the cache, and the artifact +and account capacities bound its size. A read that keeps the whole database in +memory (a DuckDB export) still waits for one of the node's large-transfer slots +when it exceeds 8 MiB. + +The switch is off by default. Set explicit capacities before enabling: + +```toml +[storage.sql] +chunked_artifacts = false +chunked_artifact_max_bytes = 0 +chunked_account_max_bytes = 0 +``` + +Both capacities must be positive when enabled. No capacity defaults are chosen +by this slice. The account capacity is charged to the space owner, across its +spaces, including legacy live artifacts, staged uploads and retained parts. +Reservations are atomic per owner. Quota refusal uses the SQL storage refusal. + +This slice publishes a full checkpoint for each SQL mutation. WAL parts, +conversion of existing artifacts follow separately. +Aborted and superseded parts enter a fenced deleting state; reclamation deletes +at most one payload chunk per statement and releases its charge only after all +bytes are gone. A periodic collector recovers uploads whose 120-second lease +expired. Lease expiry determines eligibility; part identity and state fence +uploads against concurrent reclamation. Active sealed uploads retain their lease +until publication. Deleted artifacts are reclaimed by the same mechanism. +It does not enable chunked DuckDB writes. Legacy data remains readable. Drain +older writers before enabling this format; turning the switch off does not +convert published parts back to legacy blobs. Current readers understand both +layouts; legacy persistence refuses a database already using parts. + +Do not enable for prolonged operation until the approved ceiling load acceptance and +remaining migration/WAL slices are complete. On each test release, report +reserved, staged and retained bytes from the following metadata-only inventory: + +```sql +SELECT a.owner, a.legacy_bytes, a.reserved_bytes, a.credit_bytes, + a.legacy_bytes+a.reserved_bytes-a.credit_bytes AS quota_bytes, + (SELECT COALESCE(sum(octet_length(c.payload)),0) + FROM artifact_chunk c JOIN artifact_part stored + USING (service,space,name,part_id) WHERE stored.owner=a.owner) AS stored_chunk_bytes, + COALESCE(sum(p.size_bytes) FILTER (WHERE p.state='uploading'),0) AS uploading_bytes, + COALESCE(sum(p.size_bytes) FILTER ( + WHERE p.state IN ('sealed','deleting') AND NOT EXISTS ( + SELECT 1 FROM database_artifact d WHERE d.service=p.service + AND d.space=p.space AND d.name=p.name + AND (d.checkpoint_part_id=p.part_id OR d.delta_part_id=p.part_id) + )),0) AS retained_bytes +FROM artifact_account a LEFT JOIN artifact_part p ON p.owner=a.owner +GROUP BY a.owner,a.legacy_bytes,a.reserved_bytes,a.credit_bytes; +``` + +No content is selected by this inventory. Existing-row conversion and the +approved isolated-copy procedure are outside this first slice. Explicit SQL +exports still return their complete response body; the transfer-buffer bound +applies to persistence and hydration, not that public response format. + +Quota enforcement is activated atomically with an exact-owner metadata inventory. +After activation, the same database owner-row lock accounts for legacy SQL and +DuckDB writes, including writes made by an older binary. This activation remains +in force if a reader starts with the feature switch off. Database triggers also +refuse a legacy update of a published checkpoint-parts layout atomically; the +layout marker and generation are metadata, not payload contents. A flag-off +legacy read or write adds no client-side layout probe. + +Before the first activation, the quota trigger is disabled: legacy requests add +no client layout probes or quota queries. The field-only layout guard remains +installed for mixed-version writes. First activation enables owner accounting +under the same short writer fence used for its initial inventory. +After activation, switching off new chunked databases does not disable support +for already-published parts: current nodes continue their file-based writes +using the persisted capacities, and keep the collector running. + +Operators use the value-free `artifact_part_collection_failed` event to investigate deferred reclamation. + +A replacement may overlap its current database's bytes: at most one replacement +per database receives a credit, bounded by the smaller of current and staged +sizes. Full staged bytes remain visible in `reserved_bytes`. A pending shrink +never releases capacity for another database. Credit is released only when the +old part is reclaimed, or the staged upload is reclaimed after abort; legacy +replacement credit is released atomically with publication. Admission evaluates +account usage minus the current database size plus its proposed replacement. +Thus a full account can shrink, cannot grow above its quota, and cannot reuse +the same database's credit for a second simultaneous replacement. + +Published parts have a distinct state and infinite lease, and are excluded from +both global and owner-local reclamation indexes. Inline cleanup is best effort, +scoped to the uploading owner, at most 16 steps and 100 ms per pass; its errors +do not fail the save. Background passes are limited to 128 steps and one second +per 10-second tick. GC lock acquisition waits at most 50 ms per statement. +An owner already above its quota can replace a database with an equal or +smaller image; that exception admits no growth and does not free pending shrink +capacity for other databases. diff --git a/tinycloud-core/src/database_artifacts.rs b/tinycloud-core/src/database_artifacts.rs index b3cd71df0..a8929a7e2 100644 --- a/tinycloud-core/src/database_artifacts.rs +++ b/tinycloud-core/src/database_artifacts.rs @@ -1,4 +1,5 @@ use std::future::Future; +pub mod chunks; use async_trait::async_trait; use sea_orm::{ @@ -105,7 +106,9 @@ pub struct DeltaSave { #[derive(Debug, thiserror::Error)] pub enum DatabaseArtifactError { #[error("database artifact storage error: {0}")] - Db(#[from] DbErr), + Db(DbErr), + #[error("artifact storage limit reached")] + StorageLimit, #[error("database artifact payload too large: {0} bytes")] PayloadTooLarge(u64), /// A payload larger than the repository's single-statement write limit @@ -140,6 +143,18 @@ pub enum DatabaseArtifactError { Sink(String), } +impl From for DatabaseArtifactError { + fn from(error: DbErr) -> Self { + if error.to_string().contains("TC2917_QUOTA") { + Self::StorageLimit + } else if error.to_string().contains("TC2917_LAYOUT") { + Self::StaleLineage + } else { + Self::Db(error) + } + } +} + /// The stored metadata of one artifact row: everything but the payload bytes. /// /// Reading it never touches the `payload` / `delta_payload` columns, so its @@ -205,6 +220,11 @@ pub trait ArtifactSink: Send { fn size_hint(&mut self, _part: ArtifactPart, _bytes: u64) {} async fn write(&mut self, bytes: &[u8]) -> Result<(), DatabaseArtifactError>; async fn finish_part(&mut self) -> Result<(), DatabaseArtifactError>; + /// Whether the sink holds the whole artifact in memory. Such a read of a + /// large artifact waits for a large-transfer slot in either layout. + fn buffers_in_memory(&self) -> bool { + false + } } /// The most [`MemorySink`] allocates ahead of the bytes it is given. @@ -234,6 +254,10 @@ impl ArtifactSink for MemorySink { Ok(()) } + fn buffers_in_memory(&self) -> bool { + true + } + fn size_hint(&mut self, part: ArtifactPart, bytes: u64) { let target = match part { ArtifactPart::Checkpoint => Some(&mut self.checkpoint), @@ -637,6 +661,21 @@ pub trait DatabaseArtifactRepository: Send + Sync { sink: &mut dyn ArtifactSink, ) -> Result<(), DatabaseArtifactError>; + fn chunked_checkpoints(&self, _service: &str, _space: &str, _name: &str) -> bool { + false + } + + async fn save_snapshot( + &self, + _service: &str, + _space: &str, + _name: &str, + _snapshot: tempfile::NamedTempFile, + _expected: ArtifactExpectation, + ) -> Result { + Err(DatabaseArtifactError::IncrementalPersistenceUnsupported) + } + async fn save( &self, service: &str, @@ -724,11 +763,17 @@ pub struct SeaOrmDatabaseArtifactRepository { /// Permits for transfers over [`BASE_TIMEOUT_PAYLOAD_BYTES`]; shared by /// every clone of this repository. large_transfers: std::sync::Arc, + chunk_limits: Option, + observed_chunked: std::sync::Arc>, /// Test-only rendezvous seam (see [`wait_at_race_barrier`]) that lets two /// writers read the same base revision before either commits, so the /// full-checkpoint CAS conflict path can be exercised deterministically. #[cfg(test)] race_barrier: Option>, + #[cfg(test)] + chunk_publish_barrier: Option>, + #[cfg(test)] + chunk_reclaim_paused: bool, } impl SeaOrmDatabaseArtifactRepository { @@ -741,11 +786,22 @@ impl SeaOrmDatabaseArtifactRepository { large_transfers: std::sync::Arc::new(tokio::sync::Semaphore::new( MAX_CONCURRENT_LARGE_TRANSFERS, )), + chunk_limits: None, + observed_chunked: Default::default(), #[cfg(test)] race_barrier: None, + #[cfg(test)] + chunk_publish_barrier: None, + #[cfg(test)] + chunk_reclaim_paused: false, } } + pub fn with_chunk_limits(mut self, limits: Option) -> Self { + self.chunk_limits = limits; + self + } + /// Share the node's SQLite gate: artifact writes use the capability DB and /// must not advance its WAL beneath a delegation transaction's snapshot. pub fn with_sqlite_writer_lock( @@ -941,7 +997,7 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository { )>() .one(&self.conn) .await?; - Ok(row.map( + let meta = row.map( |( revision, size_bytes, @@ -965,7 +1021,17 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository { storage_mode, backend, }, - )) + ); + let key = (service.to_string(), space.to_string(), name.to_string()); + if meta + .as_ref() + .is_some_and(|m| m.storage_mode == "checkpoint-parts-v1") + { + self.observed_chunked.insert(key); + } else { + self.observed_chunked.remove(&key); + } + Ok(meta) } async fn read_chunks( @@ -976,6 +1042,14 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository { meta: &ArtifactMeta, sink: &mut dyn ArtifactSink, ) -> Result<(), DatabaseArtifactError> { + // Parts stream in bounded statements: the whole-artifact read cap and + // the large-transfer permit below are for the single-row layout. + if self + .read_chunk_layout(service, space, name, meta, sink) + .await? + { + return Ok(()); + } // TC-2961: refuse an oversized database before any slice is read. if self.max_read_bytes != 0 { for (part, bytes) in [ @@ -1033,6 +1107,25 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository { Ok(()) } + fn chunked_checkpoints(&self, service: &str, space: &str, name: &str) -> bool { + self.chunk_limits.is_some() + || self + .observed_chunked + .contains(&(service.into(), space.into(), name.into())) + } + + async fn save_snapshot( + &self, + service: &str, + space: &str, + name: &str, + snapshot: tempfile::NamedTempFile, + expected: ArtifactExpectation, + ) -> Result { + self.write_chunk_snapshot(service, space, name, snapshot, expected) + .await + } + async fn save( &self, service: &str, @@ -1048,6 +1141,7 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository { let _ = TRANSFER_STARTED.try_with(|started| { started.store(true, std::sync::atomic::Ordering::Release); }); + self.ensure_legacy_write(service, space, name).await?; let _writer = match &self.sqlite_writer_lock { Some(lock) => Some(lock.lock().await), None => None, @@ -1256,6 +1350,7 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository { expected: ArtifactExpectation, ) -> Result { self.check_write(payload.len())?; + self.ensure_legacy_write(service, space, name).await?; let _writer = match &self.sqlite_writer_lock { Some(lock) => Some(lock.lock().await), None => None, diff --git a/tinycloud-core/src/database_artifacts/chunks.rs b/tinycloud-core/src/database_artifacts/chunks.rs new file mode 100644 index 000000000..a673cbbdd --- /dev/null +++ b/tinycloud-core/src/database_artifacts/chunks.rs @@ -0,0 +1,1865 @@ +//! Opt-in checkpoint persistence with bounded statements and immutable parts. +use super::*; +use rand::RngCore; +use sea_orm::{ConnectionTrait, DbBackend, Statement, Value}; +use tinycloud_auth::resource::SpaceId; +use tokio::io::AsyncReadExt; + +pub const CHUNK_BYTES: usize = 1024 * 1024; + +#[derive(Debug, Clone, Copy)] +pub struct ChunkLimits { + pub artifact_bytes: u64, + pub account_bytes: u64, +} + +fn statement(sql: &str, values: Vec) -> Statement { + Statement::from_sql_and_values(DbBackend::Postgres, sql, values) +} + +const REAP_CANDIDATE: &str = "SELECT service,space,name,part_id FROM artifact_part p WHERE state<>'published' AND lease_until<=statement_timestamp() AND NOT EXISTS(SELECT 1 FROM database_artifact d WHERE d.service=p.service AND d.space=p.space AND d.name=p.name AND (d.checkpoint_part_id=p.part_id OR d.delta_part_id=p.part_id)) ORDER BY lease_until LIMIT 1"; +const OWNER_REAP_CANDIDATE: &str = "SELECT service,space,name,part_id FROM artifact_part p WHERE owner=$1 AND state<>'published' AND lease_until<=statement_timestamp() AND NOT EXISTS(SELECT 1 FROM database_artifact d WHERE d.service=p.service AND d.space=p.space AND d.name=p.name AND (d.checkpoint_part_id=p.part_id OR d.delta_part_id=p.part_id)) ORDER BY lease_until LIMIT 1"; + +impl SeaOrmDatabaseArtifactRepository { + pub(super) async fn ensure_legacy_write( + &self, + service: &str, + _space: &str, + _name: &str, + ) -> Result<(), DatabaseArtifactError> { + if self.conn.get_database_backend() != DbBackend::Postgres { + return Ok(()); + } + if self.chunk_limits.is_some() && service == "sql" { + return Err(DatabaseArtifactError::IncrementalPersistenceUnsupported); + } + Ok(()) + } + + pub async fn configure_chunked_storage(&self) -> Result<(), DatabaseArtifactError> { + if let Some(limits) = self.chunk_limits { + self.conn + .execute(statement( + "SELECT artifact_enable($1,$2)", + vec![ + (limits.account_bytes.min(i64::MAX as u64) as i64).into(), + (limits.artifact_bytes.min(i64::MAX as u64) as i64).into(), + ], + )) + .await?; + } + Ok(()) + } + + pub async fn chunk_storage_active(&self) -> Result { + if self.conn.get_database_backend() != DbBackend::Postgres { + return Ok(false); + } + Ok(self + .conn + .query_one(statement( + "SELECT account_limit>0 AS active FROM artifact_storage_settings WHERE id=1", + vec![], + )) + .await? + .is_some_and(|row| row.try_get::("", "active").unwrap_or(false))) + } + + /// Each call frees at most one payload chunk per statement. Selection is + /// advisory; the database rechecks pointers and takes the owner/part locks. + pub async fn reap_unused_parts(&self, max_steps: usize) -> Result<(), DatabaseArtifactError> { + self.reap_parts(None, max_steps).await + } + + async fn reap_owner_best_effort(&self, owner: &str) { + // A request never pays another owner's cleanup bill. The background + // collector completes larger passes independently of request success. + let _ = tokio::time::timeout( + std::time::Duration::from_millis(100), + self.reap_parts(Some(owner), 16), + ) + .await; + } + + async fn reap_parts( + &self, + owner: Option<&str>, + max_steps: usize, + ) -> Result<(), DatabaseArtifactError> { + for _ in 0..max_steps { + let candidate = match owner { + Some(owner) => statement(OWNER_REAP_CANDIDATE, vec![owner.into()]), + None => statement(REAP_CANDIDATE, vec![]), + }; + let Some(row) = self.conn.query_one(candidate).await? else { + break; + }; + let values = vec![ + row.try_get::("", "service")?.into(), + row.try_get::("", "space")?.into(), + row.try_get::("", "name")?.into(), + row.try_get::("", "part_id")?.into(), + ]; + let row = self + .conn + .query_one(statement( + "SELECT artifact_reap_one($1,$2,$3,$4) AS progress", + values, + )) + .await? + .ok_or(DatabaseArtifactError::ArtifactChanged)?; + // Another collector can finish the selected part first. A fresh + // candidate query must still reach the rest of this bounded pass. + let _progress = row.try_get::("", "progress")?; + } + Ok(()) + } + + pub(super) async fn write_chunk_snapshot( + &self, + service: &str, + space: &str, + name: &str, + snapshot: tempfile::NamedTempFile, + expected: ArtifactExpectation, + ) -> Result { + if self.conn.get_database_backend() != DbBackend::Postgres { + return Err(DatabaseArtifactError::IncrementalPersistenceUnsupported); + } + let limits = match self.chunk_limits { + Some(limits) => limits, + None => { + let row=self.conn.query_one(statement("SELECT account_limit,artifact_limit FROM artifact_storage_settings WHERE id=1",vec![])) + .await?.ok_or(DatabaseArtifactError::IncrementalPersistenceUnsupported)?; + ChunkLimits { + artifact_bytes: row.try_get::("", "artifact_limit")?.max(0) as u64, + account_bytes: row.try_get::("", "account_limit")?.max(0) as u64, + } + } + }; + let size = snapshot.as_file().metadata().map_err(io_error)?.len(); + if limits.artifact_bytes == 0 + || limits.account_bytes == 0 + || size > limits.artifact_bytes + || size > i64::MAX as u64 + { + return Err(DatabaseArtifactError::StorageLimit); + } + self.check_write(size.min(CHUNK_BYTES as u64) as usize)?; + let space_id: SpaceId = space + .parse() + .map_err(|_| DatabaseArtifactError::Backend("invalid artifact space".into()))?; + let owner = space_id.did().to_string(); + self.configure_chunked_storage().await?; + self.reap_owner_best_effort(&owner).await; + let current = self.load_meta(service, space, name).await?; + expected.check( + current + .as_ref() + .map(|m| (m.revision, m.checkpoint_content_hash.as_str())), + )?; + let mut random = [0u8; 16]; + rand::rngs::OsRng.fill_bytes(&mut random); + let part_id = hex::encode(random); + let count = i32::try_from(size.div_ceil(CHUNK_BYTES as u64)) + .map_err(|_| DatabaseArtifactError::StorageLimit)?; + self.conn + .execute(statement( + "INSERT INTO artifact_account(owner) VALUES($1) ON CONFLICT DO NOTHING", + vec![owner.clone().into()], + )) + .await?; + let mut cleanup = UploadCleanup { + repo: self.clone(), + service: service.into(), + space: space.into(), + name: name.into(), + part_id: part_id.clone(), + armed: true, + }; + let reserved = self + .conn + .query_one(statement( + "SELECT artifact_reserve($1,$2,$3,$4,$5,$6,$7,$8,$9) AS admitted", + vec![ + service.into(), + space.into(), + name.into(), + part_id.clone().into(), + owner.clone().into(), + (size as i64).into(), + count.into(), + current.as_ref().map(|m| m.revision).into(), + current + .as_ref() + .map(|m| m.checkpoint_content_hash.clone()) + .into(), + ], + )) + .await? + .ok_or(DatabaseArtifactError::ArtifactChanged)?; + if !reserved.try_get::("", "admitted")? { + cleanup.armed = false; + return Err(DatabaseArtifactError::StorageLimit); + } + + let mut file = tokio::fs::File::open(snapshot.path()) + .await + .map_err(io_error)?; + let mut buffer = vec![0u8; CHUNK_BYTES]; + let mut hasher = Blake3Hasher::new(); + let mut total = 0u64; + let mut ordinal = 0i32; + loop { + let mut used = 0; + while used < buffer.len() { + let n = file.read(&mut buffer[used..]).await.map_err(io_error)?; + if n == 0 { + break; + } + used += n; + } + if used == 0 { + break; + } + total += used as u64; + if total > size { + return Err(DatabaseArtifactError::StorageLimit); + } + hasher.update(&buffer[..used]); + self.conn.execute(statement( + "WITH lease AS (UPDATE artifact_part SET lease_until=clock_timestamp()+interval '120 seconds' WHERE service=$1 AND space=$2 AND name=$3 AND part_id=$4 AND state='uploading' RETURNING part_id) INSERT INTO artifact_chunk(service,space,name,part_id,ordinal,payload) SELECT $1,$2,$3,$4,$5,$6 FROM lease", + vec![service.into(),space.into(),name.into(),part_id.clone().into(),ordinal.into(),buffer[..used].to_vec().into()], + )).await?; + ordinal += 1; + } + if total != size || ordinal != count { + return Err(DatabaseArtifactError::SizeMismatch { + part: "checkpoint", + expected: size, + actual: total, + }); + } + let cid = hasher.finalize().to_cid(0x55).to_string(); + let sealed = self + .conn + .execute(statement( + r#" +UPDATE artifact_part p SET state='sealed',content_hash=$5 +WHERE service=$1 AND space=$2 AND name=$3 AND part_id=$4 AND state='uploading' + AND (SELECT count(*) FROM artifact_chunk c WHERE + (c.service,c.space,c.name,c.part_id)=(p.service,p.space,p.name,p.part_id))=p.chunk_count + AND (SELECT COALESCE(sum(octet_length(payload)),0) FROM artifact_chunk c WHERE + (c.service,c.space,c.name,c.part_id)=(p.service,p.space,p.name,p.part_id))=p.size_bytes +"#, + vec![ + service.into(), + space.into(), + name.into(), + part_id.clone().into(), + cid.clone().into(), + ], + )) + .await?; + if sealed.rows_affected() != 1 { + return Err(DatabaseArtifactError::ArtifactChanged); + } + let revision = current.as_ref().map_or(1, |m| m.revision + 1); + let now = OffsetDateTime::now_utc() + .format(&Rfc3339) + .expect("timestamp"); + let mut values: Vec = vec![ + service.into(), + space.into(), + name.into(), + part_id.clone().into(), + revision.into(), + cid.clone().into(), + (size as i64).into(), + now.clone().into(), + ]; + #[cfg(test)] + if let Some(barrier) = &self.chunk_publish_barrier { + barrier.wait().await; + } + // One metadata-only statement is the acknowledgement boundary. Parts + // are already durable and sealed; no transaction spans the transfer. + let sql = if let Some(previous) = current { + values.extend([ + previous.revision.into(), + previous.checkpoint_content_hash.into(), + ]); + r#"WITH owner_lock AS MATERIALIZED (SELECT owner FROM artifact_account + WHERE owner=(SELECT owner FROM artifact_part WHERE service=$1 AND space=$2 AND name=$3 AND part_id=$4) FOR UPDATE), + ready AS (SELECT p.part_id FROM artifact_part p CROSS JOIN owner_lock + WHERE service=$1 AND space=$2 AND name=$3 AND part_id=$4 AND state='sealed' FOR UPDATE OF p) +UPDATE database_artifact SET checkpoint_part_id=$4,delta_part_id=NULL, + revision=$5,content_hash=$6,checkpoint_content_hash=$6,size_bytes=$7, + checkpoint_size_bytes=$7,payload=''::bytea,delta_payload=NULL, + delta_content_hash=NULL,delta_size_bytes=0,storage_mode='checkpoint-parts-v1', + layout_generation=layout_generation+1,updated_at=$8 +WHERE service=$1 AND space=$2 AND name=$3 AND revision=$9 AND checkpoint_content_hash=$10 + AND EXISTS(SELECT 1 FROM ready)"# + } else { + r#"WITH owner_lock AS MATERIALIZED (SELECT owner FROM artifact_account + WHERE owner=(SELECT owner FROM artifact_part WHERE service=$1 AND space=$2 AND name=$3 AND part_id=$4) FOR UPDATE), + ready AS (SELECT p.part_id FROM artifact_part p CROSS JOIN owner_lock + WHERE service=$1 AND space=$2 AND name=$3 AND part_id=$4 AND state='sealed' FOR UPDATE OF p) +INSERT INTO database_artifact(service,space,name,checkpoint_part_id,revision,content_hash, + checkpoint_content_hash,size_bytes,checkpoint_size_bytes,payload,backend,storage_mode, + created_at,updated_at,delta_size_bytes,layout_generation) +SELECT $1,$2,$3,$4,$5,$6,$6,$7,$7,''::bytea,'storage.database','checkpoint-parts-v1',$8,$8,0,1 +FROM ready ON CONFLICT DO NOTHING"# + }; + if self + .conn + .execute(statement(sql, values)) + .await? + .rows_affected() + != 1 + { + return Err(DatabaseArtifactError::StaleLineage); + } + cleanup.armed = false; + // Reclamation is outside the publish statement. If this pass cannot + // finish, the periodic collector keeps the bytes charged until freed. + #[cfg(test)] + let reclaim = !self.chunk_reclaim_paused; + #[cfg(not(test))] + let reclaim = true; + if reclaim { + self.reap_owner_best_effort(&owner).await; + } + Ok(ArtifactMeta { + revision, + size_bytes: size as i64, + content_hash: cid.clone(), + checkpoint_content_hash: cid, + checkpoint_size_bytes: size as i64, + delta_content_hash: None, + delta_size_bytes: 0, + updated_at: now, + storage_mode: "checkpoint-parts-v1".into(), + backend: "storage.database".into(), + }) + } + + pub(super) async fn read_chunk_layout( + &self, + service: &str, + space: &str, + name: &str, + meta: &ArtifactMeta, + sink: &mut dyn ArtifactSink, + ) -> Result { + if self.conn.get_database_backend() != DbBackend::Postgres + || meta.storage_mode != "checkpoint-parts-v1" + { + return Ok(false); + } + // Parts stream in bounded statements, so the single-row read cap does + // not apply. A sink that keeps the whole artifact in memory still + // holds it all at once, like a single-row read (TC-2961). + let _large = if sink.buffers_in_memory() { + let total = u64::try_from( + meta.checkpoint_size_bytes + .saturating_add(meta.delta_size_bytes), + ) + .unwrap_or(0); + self.large_transfer_permit(total).await + } else { + None + }; + let row=self.conn.query_one(statement( + "SELECT checkpoint_part_id,delta_part_id,revision,checkpoint_content_hash,layout_generation FROM database_artifact WHERE service=$1 AND space=$2 AND name=$3", + vec![service.into(),space.into(),name.into()], + )).await?.ok_or(DatabaseArtifactError::ArtifactChanged)?; + let generation: i64 = row.try_get("", "layout_generation")?; + let part: Option = row.try_get("", "checkpoint_part_id")?; + let Some(part) = part else { return Ok(false) }; + if row.try_get::("", "revision")? != meta.revision + || row.try_get::("", "checkpoint_content_hash")? != meta.checkpoint_content_hash + { + return Err(DatabaseArtifactError::ArtifactChanged); + } + if row + .try_get::>("", "delta_part_id")? + .is_some() + || meta.has_delta() + { + return Err(DatabaseArtifactError::IncrementalPersistenceUnsupported); + } + let manifest=self.conn.query_one(statement( + "SELECT size_bytes,chunk_count,content_hash FROM artifact_part WHERE service=$1 AND space=$2 AND name=$3 AND part_id=$4 AND state='published'", + vec![service.into(),space.into(),name.into(),part.clone().into()], + )).await?.ok_or(DatabaseArtifactError::ArtifactChanged)?; + let size = manifest.try_get::("", "size_bytes")?; + let count = manifest.try_get::("", "chunk_count")?; + let cid = manifest.try_get::("", "content_hash")?; + if size != meta.checkpoint_size_bytes + || cid != meta.checkpoint_content_hash + || size < 0 + || count as u64 != (size as u64).div_ceil(CHUNK_BYTES as u64) + { + return Err(DatabaseArtifactError::ArtifactChanged); + } + sink.begin_part(ArtifactPart::Checkpoint).await?; + sink.size_hint(ArtifactPart::Checkpoint, size as u64); + let mut hasher = Blake3Hasher::new(); + let mut total = 0u64; + for ordinal in 0..count { + let row=self.conn.query_one(statement( + "SELECT payload FROM artifact_chunk WHERE service=$1 AND space=$2 AND name=$3 AND part_id=$4 AND ordinal=$5", + vec![service.into(),space.into(),name.into(),part.clone().into(),ordinal.into()], + )).await?.ok_or(DatabaseArtifactError::ArtifactChanged)?; + let bytes: Vec = row.try_get("", "payload")?; + let expected = (size as u64 - total).min(CHUNK_BYTES as u64); + if bytes.len() as u64 != expected { + return Err(DatabaseArtifactError::SizeMismatch { + part: "checkpoint", + expected, + actual: bytes.len() as u64, + }); + } + hasher.update(&bytes); + sink.write(&bytes).await?; + total += bytes.len() as u64; + } + if hasher.finalize().to_cid(0x55).to_string() != cid || cid != meta.content_hash { + tracing::warn!( + event = "artifact_part_hash_mismatch", + "Artifact part integrity check failed" + ); + return Err(DatabaseArtifactError::HashMismatch { part: "checkpoint" }); + } + let unchanged=self.conn.query_one(statement( + "SELECT 1 AS present FROM database_artifact WHERE service=$1 AND space=$2 AND name=$3 AND revision=$4 AND checkpoint_content_hash=$5 AND checkpoint_part_id=$6 AND layout_generation=$7", + vec![service.into(),space.into(),name.into(),meta.revision.into(),cid.into(),part.into(),generation.into()], + )).await?.is_some(); + if !unchanged { + return Err(DatabaseArtifactError::ArtifactChanged); + }; + sink.finish_part().await?; + Ok(true) + } +} + +struct UploadCleanup { + repo: SeaOrmDatabaseArtifactRepository, + service: String, + space: String, + name: String, + part_id: String, + armed: bool, +} +impl Drop for UploadCleanup { + fn drop(&mut self) { + if !self.armed { + return; + } + let Ok(runtime) = tokio::runtime::Handle::try_current() else { + return; + }; + let repo = self.repo.clone(); + let values: Vec = vec![ + self.service.clone().into(), + self.space.clone().into(), + self.name.clone().into(), + self.part_id.clone().into(), + ]; + runtime.spawn(async move { + // Publication may have committed despite a lost response: the + // database checks references before permitting any deletion. + for _ in 0..2048 { + let Ok(Some(row)) = repo + .conn + .query_one(statement( + "SELECT artifact_reap_one($1,$2,$3,$4,true) AS progress", + values.clone(), + )) + .await + else { + break; + }; + if row.try_get::("", "progress").ok() != Some(true) { + break; + } + } + }); + } +} + +fn io_error(error: std::io::Error) -> DatabaseArtifactError { + DatabaseArtifactError::Sink(error.to_string()) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::io::Write; + + fn snapshot(bytes: &[u8]) -> tempfile::NamedTempFile { + let mut file = tempfile::NamedTempFile::new().unwrap(); + file.write_all(bytes).unwrap(); + file + } + + #[tokio::test] + async fn postgres_chunked_checkpoint_owner_reservations_and_integrity() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_checkpoint_owner_reservations_and_integrity", + ) + .await + else { + return; + }; + let first = "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000001:one"; + let second = "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000001:two"; + let legacy = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()); + legacy + .save( + "sql", + first, + "legacy", + vec![1; 128], + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + let repo = legacy.clone().with_chunk_limits(Some(ChunkLimits { + artifact_bytes: CHUNK_BYTES as u64, + account_bytes: (CHUNK_BYTES + 128) as u64, + })); + let payload = vec![7; CHUNK_BYTES]; + let (a, b) = tokio::join!( + repo.save_snapshot( + "sql", + first, + "main", + snapshot(&payload), + ArtifactExpectation::Absent + ), + repo.save_snapshot( + "sql", + second, + "main", + snapshot(&payload), + ArtifactExpectation::Absent + ), + ); + assert_eq!(usize::from(a.is_ok()) + usize::from(b.is_ok()), 1); + let (space, meta) = match (a, b) { + (Ok(meta), Err(DatabaseArtifactError::StorageLimit)) => (first, meta), + (Err(DatabaseArtifactError::StorageLimit), Ok(meta)) => (second, meta), + other => panic!("unexpected {other:?}"), + }; + let reserved = fixture + .conn + .query_one(statement( + "SELECT reserved_bytes FROM artifact_account", + vec![], + )) + .await + .unwrap() + .unwrap() + .try_get::("", "reserved_bytes") + .unwrap(); + assert_eq!(reserved, CHUNK_BYTES as i64); + let mut sink = MemorySink::default(); + read_artifact(&repo, "sql", space, "main", &mut sink) + .await + .unwrap() + .unwrap(); + assert_eq!(sink.checkpoint, payload); + assert!(matches!( + legacy + .save("sql", space, "main", vec![1], ArtifactExpectation::Any) + .await, + Err(DatabaseArtifactError::StaleLineage) + )); + let immutable = fixture + .conn + .execute_unprepared("UPDATE artifact_chunk SET payload=decode('00','hex')") + .await; + assert!(immutable.is_err(), "sealed bytes must be immutable"); + // Simulate storage corruption under the fixture's administrative role. + fixture.conn.execute_unprepared("ALTER TABLE artifact_chunk DISABLE TRIGGER artifact_chunk_guard; UPDATE artifact_chunk SET payload=set_byte(payload,0,8); ALTER TABLE artifact_chunk ENABLE TRIGGER artifact_chunk_guard;").await.unwrap(); + let error = repo + .read_chunks("sql", space, "main", &meta, &mut MemorySink::default()) + .await + .unwrap_err(); + assert!( + matches!(error, DatabaseArtifactError::HashMismatch { .. }), + "{error:?}" + ); + } + #[tokio::test] + async fn postgres_chunked_checkpoint_publish_cas_race_and_reclamation() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_checkpoint_publish_cas_race_and_reclamation", + ) + .await + else { + return; + }; + let space = "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000001:race"; + let repo = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()).with_chunk_limits( + Some(ChunkLimits { + artifact_bytes: 1024, + account_bytes: 8192, + }), + ); + let legacy = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()); + let base = legacy + .save( + "sql", + space, + "main", + vec![0; 128], + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + let expected = ArtifactExpectation::Derived { + revision: base.revision, + checkpoint_content_hash: base.checkpoint_content_hash, + }; + let barrier = std::sync::Arc::new(tokio::sync::Barrier::new(2)); + let mut racing = repo.clone(); + racing.chunk_publish_barrier = Some(barrier.clone()); + let upload = tokio::spawn(async move { + racing + .save_snapshot("sql", space, "main", snapshot(&[1; 128]), expected) + .await + }); + let bound = tokio::time::Instant::now() + std::time::Duration::from_secs(2); + loop { + if fixture + .conn + .query_one(statement( + "SELECT 1 AS ready FROM artifact_part WHERE state='sealed'", + vec![], + )) + .await + .unwrap() + .is_some() + { + break; + } + assert!(tokio::time::Instant::now() < bound); + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + let second = repo + .save_snapshot( + "sql", + space, + "main", + snapshot(&[2; 128]), + ArtifactExpectation::Any, + ) + .await; + assert!( + matches!(second, Err(DatabaseArtifactError::StorageLimit)), + "second replacement must refuse: {second:?}" + ); + // Administrative failure injection changes the base after reservation, + // modelling a late stale publication independently of the credit fence. + fixture.conn.execute_unprepared("ALTER TABLE database_artifact DISABLE TRIGGER artifact_metadata_guard; UPDATE database_artifact SET revision=revision+1; ALTER TABLE database_artifact ENABLE TRIGGER artifact_metadata_guard;").await.unwrap(); + barrier.wait().await; + let outcome = upload.await.unwrap(); + assert!( + matches!(outcome, Err(DatabaseArtifactError::StaleLineage)), + "stale publication was admitted: {outcome:?}" + ); + for _ in 0..100 { + repo.reap_unused_parts(64).await.unwrap(); + let n = fixture + .conn + .query_one(statement( + "SELECT count(*)::bigint AS n FROM artifact_part", + vec![], + )) + .await + .unwrap() + .unwrap() + .try_get::("", "n") + .unwrap(); + if n == 0 { + break; + } + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + // Replacements must not accumulate quota charges or deleted bytes. + for byte in 3..15 { + repo.save_snapshot( + "sql", + space, + "main", + snapshot(&[byte; 128]), + ArtifactExpectation::Any, + ) + .await + .unwrap(); + } + let reserved = fixture + .conn + .query_one(statement( + "SELECT reserved_bytes FROM artifact_account", + vec![], + )) + .await + .unwrap() + .unwrap() + .try_get::("", "reserved_bytes") + .unwrap(); + assert_eq!(reserved, 128); + fixture + .conn + .execute_unprepared("DELETE FROM database_artifact") + .await + .unwrap(); + repo.reap_unused_parts(64).await.unwrap(); + let state=fixture.conn.query_one(statement("SELECT reserved_bytes,(SELECT count(*)::bigint FROM artifact_chunk) AS chunks FROM artifact_account",vec![])).await.unwrap().unwrap(); + assert_eq!(state.try_get::("", "reserved_bytes").unwrap(), 0); + assert_eq!(state.try_get::("", "chunks").unwrap(), 0); + // A process crash after reservation/first chunk: the next pass fences + // the expired upload, deletes it and releases its charge. + fixture.conn.execute_unprepared(&format!("UPDATE artifact_account SET reserved_bytes=512; INSERT INTO artifact_part(service,space,name,part_id,owner,size_bytes,chunk_count,lease_until) SELECT 'sql','{space}','crashed','0123456789abcdef',owner,512,1,clock_timestamp()-interval '1 second' FROM artifact_account; INSERT INTO artifact_chunk SELECT service,space,name,part_id,0,decode(repeat('00',512),'hex') FROM artifact_part;")).await.unwrap(); + repo.reap_unused_parts(64).await.unwrap(); + assert_eq!( + fixture + .conn + .query_one(statement( + "SELECT reserved_bytes FROM artifact_account", + vec![] + )) + .await + .unwrap() + .unwrap() + .try_get::("", "reserved_bytes") + .unwrap(), + 0 + ); + } + + #[tokio::test] + async fn postgres_chunked_checkpoint_exact_owner_and_legacy_duckdb_quota() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_checkpoint_exact_owner_and_legacy_duckdb_quota", + ) + .await + else { + return; + }; + let parent = "tinycloud:web:example.com:notes"; + let child = "tinycloud:web:example.com:alice:notes"; + let legacy = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()); + for space in [parent, child] { + legacy + .save( + "duckdb", + space, + "main", + vec![1; 600], + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + } + let repo = legacy.clone().with_chunk_limits(Some(ChunkLimits { + artifact_bytes: 1024, + account_bytes: 1000, + })); + repo.configure_chunked_storage().await.unwrap(); + let owners = fixture + .conn + .query_all(statement( + "SELECT owner,legacy_bytes FROM artifact_account ORDER BY owner", + vec![], + )) + .await + .unwrap(); + assert_eq!(owners.len(), 2); + assert_eq!( + owners[0].try_get::("", "owner").unwrap(), + "did:web:example.com" + ); + assert_eq!( + owners[1].try_get::("", "owner").unwrap(), + "did:web:example.com:alice" + ); + assert!(owners + .iter() + .all(|row| row.try_get::("", "legacy_bytes").unwrap() == 600)); + legacy + .save( + "duckdb", + parent, + "second", + vec![2; 400], + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + let refused = legacy + .save( + "duckdb", + parent, + "third", + vec![3; 1], + ArtifactExpectation::Absent, + ) + .await; + assert!( + matches!(refused, Err(DatabaseArtifactError::StorageLimit)), + "{refused:?}" + ); + legacy + .save( + "duckdb", + child, + "second", + vec![4; 400], + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + } + #[tokio::test] + async fn postgres_chunked_checkpoint_cancelled_upload_releases_reservation() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_checkpoint_cancelled_upload_releases_reservation", + ) + .await + else { + return; + }; + let space = "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000001:cancel"; + let repo = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()).with_chunk_limits( + Some(ChunkLimits { + artifact_bytes: 1024, + account_bytes: 1024, + }), + ); + let mut blocked = repo.clone(); + blocked.chunk_publish_barrier = Some(std::sync::Arc::new(tokio::sync::Barrier::new(2))); + let upload = tokio::spawn(async move { + blocked + .save_snapshot( + "sql", + space, + "cancelled", + snapshot(&[9; 512]), + ArtifactExpectation::Absent, + ) + .await + }); + let bound = tokio::time::Instant::now() + std::time::Duration::from_secs(2); + loop { + let staged = fixture + .conn + .query_one(statement( + "SELECT 1 AS ready FROM artifact_part WHERE state='sealed'", + vec![], + )) + .await + .unwrap() + .is_some(); + if staged { + break; + } + assert!( + tokio::time::Instant::now() < bound, + "upload did not reach publication barrier" + ); + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + upload.abort(); + assert!(upload.await.unwrap_err().is_cancelled()); + let bound = tokio::time::Instant::now() + std::time::Duration::from_secs(2); + loop { + let charge = fixture + .conn + .query_one(statement( + "SELECT reserved_bytes FROM artifact_account", + vec![], + )) + .await + .unwrap() + .unwrap() + .try_get::("", "reserved_bytes") + .unwrap(); + if charge == 0 { + break; + } + assert!( + tokio::time::Instant::now() < bound, + "cancelled reservation was not released" + ); + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + assert!(repo + .load_meta("sql", space, "cancelled") + .await + .unwrap() + .is_none()); + repo.save_snapshot( + "sql", + space, + "retry", + snapshot(&[10; 1024]), + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + } + #[tokio::test] + async fn postgres_chunked_checkpoint_at_quota_replacement_credit() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_checkpoint_at_quota_replacement_credit", + ) + .await + else { + return; + }; + let space = "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000001:credit"; + let repo = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()).with_chunk_limits( + Some(ChunkLimits { + artifact_bytes: 1024, + account_bytes: 512, + }), + ); + repo.save_snapshot( + "sql", + space, + "main", + snapshot(&[1; 512]), + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + let grow = repo + .save_snapshot( + "sql", + space, + "main", + snapshot(&[2; 513]), + ArtifactExpectation::Any, + ) + .await; + assert!( + matches!(grow, Err(DatabaseArtifactError::StorageLimit)), + "{grow:?}" + ); + let barrier = std::sync::Arc::new(tokio::sync::Barrier::new(2)); + let mut blocked = repo.clone(); + blocked.chunk_publish_barrier = Some(barrier.clone()); + let shrink = tokio::spawn(async move { + blocked + .save_snapshot( + "sql", + space, + "main", + snapshot(&[3; 256]), + ArtifactExpectation::Any, + ) + .await + }); + let bound = tokio::time::Instant::now() + std::time::Duration::from_secs(2); + loop { + if fixture + .conn + .query_one(statement( + "SELECT 1 AS ready FROM artifact_part WHERE replacing AND state='sealed'", + vec![], + )) + .await + .unwrap() + .is_some() + { + break; + } + assert!(tokio::time::Instant::now() < bound); + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + let second = repo + .save_snapshot( + "sql", + space, + "main", + snapshot(&[4; 128]), + ArtifactExpectation::Any, + ) + .await; + assert!(matches!(second, Err(DatabaseArtifactError::StorageLimit))); + // Pending shrinkage cannot fund another database, then be aborted. + let other = repo + .save_snapshot( + "sql", + space, + "other", + snapshot(&[5; 1]), + ArtifactExpectation::Absent, + ) + .await; + assert!(matches!(other, Err(DatabaseArtifactError::StorageLimit))); + barrier.wait().await; + assert_eq!(shrink.await.unwrap().unwrap().size_bytes, 256); + repo.save_snapshot( + "sql", + space, + "other", + snapshot(&[6; 256]), + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + let row = fixture + .conn + .query_one(statement( + "SELECT reserved_bytes,credit_bytes FROM artifact_account", + vec![], + )) + .await + .unwrap() + .unwrap(); + assert_eq!(row.try_get::("", "reserved_bytes").unwrap(), 512); + assert_eq!(row.try_get::("", "credit_bytes").unwrap(), 0); + } + async fn ledger(fixture: &crate::test_support::PostgresTestSchema) -> (i64, i64) { + let row = fixture + .conn + .query_one(statement( + "SELECT reserved_bytes,credit_bytes FROM artifact_account", + vec![], + )) + .await + .unwrap() + .unwrap(); + ( + row.try_get("", "reserved_bytes").unwrap(), + row.try_get("", "credit_bytes").unwrap(), + ) + } + + async fn await_replacement(fixture: &crate::test_support::PostgresTestSchema) { + let bound = tokio::time::Instant::now() + std::time::Duration::from_secs(2); + loop { + if fixture + .conn + .query_one(statement( + "SELECT 1 AS ready FROM artifact_part WHERE replacing AND state='sealed'", + vec![], + )) + .await + .unwrap() + .is_some() + { + break; + } + assert!(tokio::time::Instant::now() < bound); + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + } + + #[tokio::test] + async fn postgres_chunked_checkpoint_shrink_abort_and_staging_crash() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_checkpoint_shrink_abort_and_staging_crash", + ) + .await + else { + return; + }; + let space = + "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000001:abort-credit"; + let repo = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()).with_chunk_limits( + Some(ChunkLimits { + artifact_bytes: 1024, + account_bytes: 512, + }), + ); + let base = repo + .save_snapshot( + "sql", + space, + "main", + snapshot(&[1; 512]), + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + for _ in 0..3 { + let mut blocked = repo.clone(); + blocked.chunk_publish_barrier = Some(std::sync::Arc::new(tokio::sync::Barrier::new(2))); + let upload = tokio::spawn(async move { + blocked + .save_snapshot( + "sql", + space, + "main", + snapshot(&[2; 256]), + ArtifactExpectation::Any, + ) + .await + }); + await_replacement(&fixture).await; + assert_eq!(ledger(&fixture).await, (768, 256)); + upload.abort(); + assert!(upload.await.unwrap_err().is_cancelled()); + let bound = tokio::time::Instant::now() + std::time::Duration::from_secs(2); + while ledger(&fixture).await != (512, 0) { + assert!(tokio::time::Instant::now() < bound); + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + } + // An owner process disappears after a partial chunk upload; there is + // no Rust drop guard in the crashed process to do this cleanup. + let owner = space.parse::().unwrap().did().to_string(); + let admitted = fixture + .conn + .query_one(statement( + "SELECT artifact_reserve('sql',$1,'main','crashed-credit',$2,256,1,$3,$4) AS ok", + vec![ + space.into(), + owner.into(), + base.revision.into(), + base.checkpoint_content_hash.into(), + ], + )) + .await + .unwrap() + .unwrap() + .try_get::("", "ok") + .unwrap(); + assert!(admitted); + fixture.conn.execute(statement("INSERT INTO artifact_chunk SELECT service,space,name,part_id,0,$1 FROM artifact_part WHERE part_id='crashed-credit'",vec![vec![3u8;128].into()])).await.unwrap(); + repo.reap_unused_parts(64).await.unwrap(); + assert_eq!( + ledger(&fixture).await, + (768, 256), + "unexpired upload stays charged" + ); + fixture.conn.execute_unprepared("UPDATE artifact_part SET lease_until=clock_timestamp()-interval '1 second' WHERE part_id='crashed-credit'").await.unwrap(); + repo.reap_unused_parts(64).await.unwrap(); + assert_eq!(ledger(&fixture).await, (512, 0)); + assert!(fixture + .conn + .query_one(statement( + "SELECT 1 FROM artifact_chunk WHERE part_id='crashed-credit'", + vec![] + )) + .await + .unwrap() + .is_none()); + } + + #[tokio::test] + async fn postgres_chunked_checkpoint_delete_during_staging_fences_publish() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_checkpoint_delete_during_staging_fences_publish", + ) + .await + else { + return; + }; + let space = + "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000001:delete-credit"; + let repo = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()).with_chunk_limits( + Some(ChunkLimits { + artifact_bytes: 1024, + account_bytes: 512, + }), + ); + repo.save_snapshot( + "sql", + space, + "main", + snapshot(&[1; 512]), + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + let barrier = std::sync::Arc::new(tokio::sync::Barrier::new(2)); + let mut blocked = repo.clone(); + blocked.chunk_publish_barrier = Some(barrier.clone()); + let upload = tokio::spawn(async move { + blocked + .save_snapshot( + "sql", + space, + "main", + snapshot(&[2; 256]), + ArtifactExpectation::Any, + ) + .await + }); + await_replacement(&fixture).await; + fixture + .conn + .execute_unprepared("DELETE FROM database_artifact") + .await + .unwrap(); + assert_eq!( + ledger(&fixture).await, + (768, 0), + "all physical parts stay charged after deletion" + ); + barrier.wait().await; + assert!(matches!( + upload.await.unwrap(), + Err(DatabaseArtifactError::StaleLineage) + )); + repo.reap_unused_parts(64).await.unwrap(); + assert_eq!(ledger(&fixture).await, (0, 0)); + assert!(repo + .load_meta("sql", space, "main") + .await + .unwrap() + .is_none()); + } + + #[tokio::test] + async fn postgres_chunked_checkpoint_publish_credit_waits_for_physical_deletion() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_checkpoint_publish_credit_waits_for_physical_deletion", + ) + .await + else { + return; + }; + let space = + "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000001:physical-credit"; + let old = CHUNK_BYTES + 64; + let new = 256 * 1024; + let mut repo = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()) + .with_chunk_limits(Some(ChunkLimits { + artifact_bytes: 2 * CHUNK_BYTES as u64, + account_bytes: old as u64, + })); + repo.chunk_reclaim_paused = true; + repo.save_snapshot( + "sql", + space, + "main", + snapshot(&vec![1; old]), + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + let old_part = fixture + .conn + .query_one(statement( + "SELECT checkpoint_part_id FROM database_artifact", + vec![], + )) + .await + .unwrap() + .unwrap() + .try_get::("", "checkpoint_part_id") + .unwrap(); + repo.save_snapshot( + "sql", + space, + "main", + snapshot(&vec![2; new]), + ArtifactExpectation::Any, + ) + .await + .unwrap(); + assert_eq!(ledger(&fixture).await, ((old + new) as i64, new as i64)); + let args = vec![ + "sql".into(), + space.into(), + "main".into(), + old_part.clone().into(), + ]; + fixture + .conn + .query_one(statement( + "SELECT artifact_reap_one($1,$2,$3,$4)", + args.clone(), + )) + .await + .unwrap(); + assert_eq!( + ledger(&fixture).await, + ((old + new) as i64, new as i64), + "partial deletion must not free quota" + ); + fixture + .conn + .query_one(statement("SELECT artifact_reap_one($1,$2,$3,$4)", args)) + .await + .unwrap(); + assert_eq!(ledger(&fixture).await, (new as i64, 0)); + assert!(fixture + .conn + .query_one(statement( + "SELECT 1 FROM artifact_chunk WHERE part_id=$1", + vec![old_part.into()] + )) + .await + .unwrap() + .is_none()); + repo.save_snapshot( + "sql", + space, + "other", + snapshot(&vec![3; old - new]), + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + } + #[tokio::test] + async fn postgres_chunked_checkpoint_over_cap_owner_can_shrink() { + for initially_chunked in [false, true] { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_checkpoint_over_cap_owner_can_shrink", + ) + .await + else { + return; + }; + let space = + "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000001:over-cap"; + let legacy = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()); + if initially_chunked { + let initial = legacy.clone().with_chunk_limits(Some(ChunkLimits { + artifact_bytes: 1024, + account_bytes: 1024, + })); + initial + .save_snapshot( + "sql", + space, + "main", + snapshot(&[1; 512]), + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + } else { + legacy + .save( + "sql", + space, + "main", + vec![1; 512], + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + } + let capped = legacy.with_chunk_limits(Some(ChunkLimits { + artifact_bytes: 1024, + account_bytes: 128, + })); + capped.configure_chunked_storage().await.unwrap(); + capped + .save_snapshot( + "sql", + space, + "main", + snapshot(&[2; 512]), + ArtifactExpectation::Any, + ) + .await + .unwrap(); + capped + .save_snapshot( + "sql", + space, + "main", + snapshot(&[3; 400]), + ArtifactExpectation::Any, + ) + .await + .unwrap(); + let growth = capped + .save_snapshot( + "sql", + space, + "main", + snapshot(&[4; 401]), + ArtifactExpectation::Any, + ) + .await; + assert!(matches!(growth, Err(DatabaseArtifactError::StorageLimit))); + capped + .save_snapshot( + "sql", + space, + "main", + snapshot(&[5; 128]), + ArtifactExpectation::Any, + ) + .await + .unwrap(); + capped.reap_unused_parts(64).await.unwrap(); + assert_eq!(ledger(&fixture).await, (128, 0)); + } + } + + #[tokio::test] + async fn postgres_chunked_checkpoint_multi_chunk_abort_keeps_charge_until_gone() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_checkpoint_multi_chunk_abort_keeps_charge_until_gone", + ) + .await + else { + return; + }; + let space = + "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000001:abort-chunks"; + let repo = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()).with_chunk_limits( + Some(ChunkLimits { + artifact_bytes: 4 * CHUNK_BYTES as u64, + account_bytes: 4 * CHUNK_BYTES as u64, + }), + ); + let base = repo + .save_snapshot( + "sql", + space, + "main", + snapshot(&vec![1; CHUNK_BYTES]), + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + let owner = space.parse::().unwrap().did().to_string(); + let stage_size = 2 * CHUNK_BYTES + 64; + let admitted = fixture + .conn + .query_one(statement( + "SELECT artifact_reserve('sql',$1,'main','multi-abort',$2,$3,3,$4,$5) AS ok", + vec![ + space.into(), + owner.into(), + (stage_size as i64).into(), + base.revision.into(), + base.checkpoint_content_hash.into(), + ], + )) + .await + .unwrap() + .unwrap() + .try_get::("", "ok") + .unwrap(); + assert!(admitted); + for (ordinal, size) in [CHUNK_BYTES, CHUNK_BYTES, 64].into_iter().enumerate() { + fixture.conn.execute(statement("INSERT INTO artifact_chunk SELECT service,space,name,part_id,$1,$2 FROM artifact_part WHERE part_id='multi-abort'",vec![(ordinal as i32).into(),vec![2u8;size].into()])).await.unwrap(); + } + for remaining in [2, 1] { + fixture + .conn + .query_one(statement( + "SELECT artifact_reap_one('sql',$1,'main','multi-abort',true)", + vec![space.into()], + )) + .await + .unwrap(); + assert_eq!( + ledger(&fixture).await, + ((CHUNK_BYTES + stage_size) as i64, CHUNK_BYTES as i64) + ); + let n = fixture + .conn + .query_one(statement( + "SELECT count(*)::bigint AS n FROM artifact_chunk WHERE part_id='multi-abort'", + vec![], + )) + .await + .unwrap() + .unwrap() + .try_get::("", "n") + .unwrap(); + assert_eq!(n, remaining); + } + fixture + .conn + .query_one(statement( + "SELECT artifact_reap_one('sql',$1,'main','multi-abort',true)", + vec![space.into()], + )) + .await + .unwrap(); + assert_eq!(ledger(&fixture).await, (CHUNK_BYTES as i64, 0)); + assert!(fixture + .conn + .query_one(statement( + "SELECT 1 FROM artifact_chunk WHERE part_id='multi-abort'", + vec![] + )) + .await + .unwrap() + .is_none()); + } + + #[tokio::test] + async fn postgres_chunked_checkpoint_concurrent_legacy_quota_is_atomic() { + use sea_orm::TransactionTrait; + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_checkpoint_concurrent_legacy_quota_is_atomic", + ) + .await + else { + return; + }; + let first = "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000001:legacy-one"; + let second = "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000001:legacy-two"; + let legacy = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()); + for i in 0..8 { + legacy + .save( + if i % 2 == 0 { "sql" } else { "duckdb" }, + if i % 2 == 0 { first } else { second }, + &format!("db-{i}"), + vec![], + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + } + legacy + .clone() + .with_chunk_limits(Some(ChunkLimits { + artifact_bytes: 1024, + account_bytes: 512, + })) + .configure_chunked_storage() + .await + .unwrap(); + let blocker = fixture.connect(1).await; + let held = blocker.begin().await.unwrap(); + held.query_one(statement( + "SELECT owner FROM artifact_account FOR UPDATE", + vec![], + )) + .await + .unwrap(); + let barrier = std::sync::Arc::new(tokio::sync::Barrier::new(8)); + let tasks = (0..8) + .map(|i| { + let repo = legacy.clone(); + let barrier = barrier.clone(); + tokio::spawn(async move { + barrier.wait().await; + repo.save( + if i % 2 == 0 { "sql" } else { "duckdb" }, + if i % 2 == 0 { first } else { second }, + &format!("db-{i}"), + vec![7; 256], + ArtifactExpectation::Any, + ) + .await + }) + }) + .collect::>(); + let observer = fixture.connect(1).await; + let bound = tokio::time::Instant::now() + std::time::Duration::from_secs(3); + loop { + let n=observer.query_one(statement("SELECT count(DISTINCT l.pid)::bigint AS n FROM pg_locks l JOIN pg_stat_activity a ON a.pid=l.pid WHERE l.relation='artifact_account'::regclass AND l.mode='RowExclusiveLock' AND l.granted AND a.wait_event_type='Lock'",vec![])).await.unwrap().unwrap().try_get::("","n").unwrap(); + if n >= 4 { + break; + } + assert!( + tokio::time::Instant::now() < bound, + "legacy writers did not overlap at owner lock: {n}" + ); + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + held.rollback().await.unwrap(); + let mut admitted = 0; + for task in tasks { + match task.await.unwrap() { + Ok(_) => admitted += 1, + Err(DatabaseArtifactError::StorageLimit) => {} + other => panic!("unexpected {other:?}"), + } + } + assert_eq!( + admitted, 2, + "concurrent legacy quota check admitted too many writes" + ); + let state=fixture.conn.query_one(statement("SELECT legacy_bytes,(SELECT sum(size_bytes)::bigint FROM database_artifact) AS live FROM artifact_account",vec![])).await.unwrap().unwrap(); + assert_eq!(state.try_get::("", "legacy_bytes").unwrap(), 512); + assert_eq!(state.try_get::("", "live").unwrap(), 512); + } + + #[tokio::test] + async fn postgres_chunked_checkpoint_reclaim_is_owner_scoped_and_best_effort() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_checkpoint_reclaim_is_owner_scoped_and_best_effort", + ) + .await + else { + return; + }; + let space = + "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000001:isolated-reap"; + let owner = space.parse::().unwrap().did().to_string(); + let other = "did:pkh:eip155:1:0x0000000000000000000000000000000000000002"; + let repo = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()).with_chunk_limits( + Some(ChunkLimits { + artifact_bytes: 1024, + account_bytes: 4096, + }), + ); + repo.configure_chunked_storage().await.unwrap(); + fixture + .conn + .execute(statement( + "INSERT INTO artifact_account(owner) VALUES($1)", + vec![other.into()], + )) + .await + .unwrap(); + fixture.conn.execute(statement("INSERT INTO artifact_part(service,space,name,part_id,owner,size_bytes,chunk_count,state,lease_until) SELECT 'sql','synthetic-published','db-'||n,'part-'||n,$1,0,0,'published',clock_timestamp()-interval '1 year' FROM generate_series(1,100000) AS n",vec![other.into()])).await.unwrap(); + fixture.conn.execute(statement("INSERT INTO artifact_part(service,space,name,part_id,owner,size_bytes,chunk_count,state,lease_until) SELECT 'sql','synthetic-deleting','db-'||n,'garbage-'||n,$1,0,0,'deleting',clock_timestamp()-interval '1 day' FROM generate_series(1,4096) AS n",vec![other.into()])).await.unwrap(); + fixture + .conn + .execute_unprepared("ANALYZE artifact_part") + .await + .unwrap(); + let plan = fixture + .conn + .query_all(statement( + &format!("EXPLAIN (ANALYZE,TIMING OFF) {OWNER_REAP_CANDIDATE}"), + vec![owner.clone().into()], + )) + .await + .unwrap() + .into_iter() + .map(|row| row.try_get::("", "QUERY PLAN").unwrap()) + .collect::>() + .join("\n"); + assert!(plan.contains("artifact_part_reap_owner"), "{plan}"); + assert!(!plan.contains("Seq Scan on artifact_part"), "{plan}"); + repo.save_snapshot( + "sql", + space, + "main", + snapshot(&[1; 64]), + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + let n = fixture + .conn + .query_one(statement( + "SELECT count(*)::bigint AS n FROM artifact_part WHERE owner=$1", + vec![other.into()], + )) + .await + .unwrap() + .unwrap() + .try_get::("", "n") + .unwrap(); + assert_eq!(n, 104096, "writer reclaimed another owner's parts"); + // Real database failure injection: a failing collector must not fail + // a write with sufficient quota, even for its own owner's garbage. + fixture.conn.execute(statement("INSERT INTO artifact_part(service,space,name,part_id,owner,size_bytes,chunk_count,state,lease_until) VALUES('sql',$1,'broken','bad-reap',$2,0,0,'deleting',clock_timestamp())",vec![space.into(),owner.into()])).await.unwrap(); + fixture.conn.execute_unprepared("CREATE OR REPLACE FUNCTION artifact_reap_one(p_service text,p_space text,p_name text,p_id text,abort_upload boolean DEFAULT false) RETURNS boolean LANGUAGE plpgsql AS $$ BEGIN RAISE EXCEPTION 'synthetic collector failure'; END $$").await.unwrap(); + repo.save_snapshot( + "sql", + space, + "second", + snapshot(&[2; 64]), + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + } + #[tokio::test] + async fn postgres_chunked_checkpoint_honours_single_statement_cap() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_checkpoint_honours_single_statement_cap", + ) + .await + else { + return; + }; + let space = + "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000001:statement-cap"; + let repo = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()).with_chunk_limits( + Some(ChunkLimits { + artifact_bytes: 4 * CHUNK_BYTES as u64, + account_bytes: 8 * CHUNK_BYTES as u64, + }), + ); + let refused = repo + .clone() + .with_max_write_bytes(4096) + .save_snapshot( + "sql", + space, + "refused", + snapshot(&vec![1; 4097]), + ArtifactExpectation::Absent, + ) + .await; + assert!( + matches!( + refused, + Err(DatabaseArtifactError::WriteLimitExceeded { + bytes: 4097, + limit: 4096 + }) + ), + "{refused:?}" + ); + // The complete artifact may exceed the statement cap: each bounded + // chunk must obey it, as well as the separate artifact/account limits. + let saved = repo + .with_max_write_bytes(CHUNK_BYTES as u64) + .save_snapshot( + "sql", + space, + "streamed", + snapshot(&vec![2; CHUNK_BYTES + 1]), + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + assert_eq!(saved.size_bytes, CHUNK_BYTES as i64 + 1); + } + + /// Counts the bytes a read delivers without holding them. + #[derive(Default)] + struct CountingSink { + bytes: u64, + } + + #[async_trait] + impl ArtifactSink for CountingSink { + async fn reset(&mut self) -> Result<(), DatabaseArtifactError> { + self.bytes = 0; + Ok(()) + } + async fn begin_part(&mut self, _part: ArtifactPart) -> Result<(), DatabaseArtifactError> { + Ok(()) + } + async fn write(&mut self, bytes: &[u8]) -> Result<(), DatabaseArtifactError> { + self.bytes += bytes.len() as u64; + Ok(()) + } + async fn finish_part(&mut self) -> Result<(), DatabaseArtifactError> { + Ok(()) + } + } + + #[tokio::test] + async fn postgres_chunked_read_is_not_held_to_the_single_row_read_cap() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_read_is_not_held_to_the_single_row_read_cap", + ) + .await + else { + return; + }; + let space = "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000001:read-cap"; + let over = DEFAULT_MAX_ARTIFACT_READ_BYTES as usize + 1; + // Seed a single-row artifact over the default read cap with the write + // cap off, as an older node could have stored it. + SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()) + .with_max_write_bytes(0) + .save( + "sql", + space, + "legacy", + vec![3; over], + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + let repo = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()).with_chunk_limits( + Some(ChunkLimits { + artifact_bytes: 2 * over as u64, + account_bytes: 4 * over as u64, + }), + ); + repo.save_snapshot( + "sql", + space, + "parts", + snapshot(&vec![4; over]), + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + + let mut parts = CountingSink::default(); + read_artifact(&repo, "sql", space, "parts", &mut parts) + .await + .unwrap() + .unwrap(); + assert_eq!(parts.bytes, over as u64); + + let legacy = + read_artifact(&repo, "sql", space, "legacy", &mut CountingSink::default()).await; + assert!( + matches!(legacy, Err(DatabaseArtifactError::ReadLimitExceeded { .. })), + "{legacy:?}" + ); + } + + #[tokio::test] + async fn postgres_chunked_in_memory_read_waits_for_a_large_transfer_slot() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_in_memory_read_waits_for_a_large_transfer_slot", + ) + .await + else { + return; + }; + let space = "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000001:permit"; + let size = BASE_TIMEOUT_PAYLOAD_BYTES + 1; + let repo = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()).with_chunk_limits( + Some(ChunkLimits { + artifact_bytes: 2 * size as u64, + account_bytes: 4 * size as u64, + }), + ); + repo.save_snapshot( + "sql", + space, + "main", + snapshot(&vec![5; size]), + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + let held: Vec<_> = (0..MAX_CONCURRENT_LARGE_TRANSFERS) + .map(|_| repo.large_transfers.try_acquire().unwrap()) + .collect(); + + // A streaming sink holds no more than a statement's bytes: no slot. + let mut streamed = CountingSink::default(); + read_artifact(&repo, "sql", space, "main", &mut streamed) + .await + .unwrap() + .unwrap(); + assert_eq!(streamed.bytes, size as u64); + + // A sink holding the whole artifact waits until a slot frees. + let waiting = { + let repo = repo.clone(); + tokio::spawn(async move { + let mut sink = MemorySink::default(); + read_artifact(&repo, "sql", space, "main", &mut sink) + .await + .map(|_| sink.checkpoint.len()) + }) + }; + tokio::time::sleep(std::time::Duration::from_millis(300)).await; + assert!( + !waiting.is_finished(), + "an in-memory read took no transfer slot" + ); + drop(held); + assert_eq!(waiting.await.unwrap().unwrap(), size); + } +} diff --git a/tinycloud-core/src/database_artifacts/chunks.sql b/tinycloud-core/src/database_artifacts/chunks.sql new file mode 100644 index 000000000..bf7ea6868 --- /dev/null +++ b/tinycloud-core/src/database_artifacts/chunks.sql @@ -0,0 +1,265 @@ +CREATE TABLE artifact_storage_settings ( + id integer PRIMARY KEY CHECK(id=1), account_limit bigint NOT NULL DEFAULT 0, + artifact_limit bigint NOT NULL DEFAULT 0 +); +INSERT INTO artifact_storage_settings(id) VALUES(1); +CREATE TABLE artifact_account ( + owner text PRIMARY KEY, + legacy_bytes bigint NOT NULL DEFAULT 0 CHECK(legacy_bytes>=0), + reserved_bytes bigint NOT NULL DEFAULT 0 CHECK(reserved_bytes>=0), + credit_bytes bigint NOT NULL DEFAULT 0 CHECK(credit_bytes>=0) +); +CREATE TABLE artifact_space_owner ( + space text PRIMARY KEY, owner text NOT NULL REFERENCES artifact_account(owner) +); +CREATE INDEX artifact_space_owner_owner ON artifact_space_owner(owner); +-- Space names cannot contain colons. Remove exactly the final name component, +-- never a DID prefix (did:web owners may themselves have colon components). +CREATE FUNCTION artifact_owner(space text) RETURNS text LANGUAGE sql IMMUTABLE AS $$ + SELECT CASE WHEN space LIKE 'tinycloud:%' + THEN 'did:' || regexp_replace(substr(space,11), ':[^:]+$', '') + ELSE 'legacy:' || space END +$$; +CREATE TABLE artifact_part ( + service text NOT NULL, space text NOT NULL, name text NOT NULL, + part_id text NOT NULL, owner text NOT NULL REFERENCES artifact_account(owner), + size_bytes bigint NOT NULL CHECK(size_bytes>=0), content_hash text, + chunk_count integer NOT NULL CHECK(chunk_count>=0), + replacement_credit bigint NOT NULL DEFAULT 0 CHECK(replacement_credit>=0), + replacing boolean NOT NULL DEFAULT false, replaces_part_id text, + state text NOT NULL DEFAULT 'uploading' CHECK(state IN ('uploading','sealed','published','deleting')), + lease_until timestamptz NOT NULL DEFAULT clock_timestamp()+interval '120 seconds', + PRIMARY KEY(service,space,name,part_id), + FOREIGN KEY(service,space,name,replaces_part_id) REFERENCES artifact_part(service,space,name,part_id) +); +CREATE UNIQUE INDEX artifact_part_replacement ON artifact_part(service,space,name) WHERE replacing; +CREATE INDEX artifact_part_reap ON artifact_part(lease_until) WHERE state<>'published'; +CREATE INDEX artifact_part_reap_owner ON artifact_part(owner,lease_until) WHERE state<>'published'; +CREATE TABLE artifact_chunk ( + service text NOT NULL, space text NOT NULL, name text NOT NULL, + part_id text NOT NULL, ordinal integer NOT NULL CHECK(ordinal>=0), + payload bytea NOT NULL CHECK(octet_length(payload) BETWEEN 1 AND 1048576), + PRIMARY KEY(service,space,name,part_id,ordinal), + FOREIGN KEY(service,space,name,part_id) REFERENCES artifact_part(service,space,name,part_id) +); +ALTER TABLE artifact_chunk ALTER COLUMN payload SET STORAGE EXTERNAL; +ALTER TABLE database_artifact ADD COLUMN checkpoint_part_id text; +ALTER TABLE database_artifact ADD COLUMN delta_part_id text; +ALTER TABLE database_artifact ADD COLUMN layout_generation bigint NOT NULL DEFAULT 0; +ALTER TABLE database_artifact ADD CONSTRAINT artifact_checkpoint_part_fk + FOREIGN KEY(service,space,name,checkpoint_part_id) REFERENCES artifact_part(service,space,name,part_id); +ALTER TABLE database_artifact ADD CONSTRAINT artifact_delta_part_fk + FOREIGN KEY(service,space,name,delta_part_id) REFERENCES artifact_part(service,space,name,part_id); + +CREATE FUNCTION artifact_chunk_guard() RETURNS trigger LANGUAGE plpgsql AS $$ +DECLARE s text; +BEGIN + IF TG_OP='UPDATE' THEN RAISE EXCEPTION 'artifact chunks are immutable'; END IF; + IF TG_OP='DELETE' THEN + SELECT state INTO s FROM artifact_part WHERE service=OLD.service AND space=OLD.space + AND name=OLD.name AND part_id=OLD.part_id FOR SHARE; + IF s IS DISTINCT FROM 'deleting' THEN RAISE EXCEPTION 'artifact part is not deleting'; END IF; + RETURN OLD; + END IF; + SELECT state INTO s FROM artifact_part WHERE service=NEW.service AND space=NEW.space + AND name=NEW.name AND part_id=NEW.part_id FOR SHARE; + IF s IS DISTINCT FROM 'uploading' THEN RAISE EXCEPTION 'artifact part is not uploading'; END IF; + RETURN NEW; +END $$; +CREATE TRIGGER artifact_chunk_guard BEFORE INSERT OR UPDATE OR DELETE ON artifact_chunk + FOR EACH ROW EXECUTE FUNCTION artifact_chunk_guard(); + +-- Activation is separate from migration. With the switch off, no owner +-- inventory is scanned or maintained. A short writer fence makes the initial +-- exact-owner inventory and quota activation atomic with old-binary writes. +CREATE FUNCTION artifact_enable(limit_bytes bigint, part_limit bigint) RETURNS void LANGUAGE plpgsql AS $$ +DECLARE previous bigint; previous_timeout text:=current_setting('lock_timeout'); +BEGIN + SELECT account_limit INTO previous FROM artifact_storage_settings WHERE id=1; + IF previous=limit_bytes AND (SELECT artifact_limit FROM artifact_storage_settings WHERE id=1)=part_limit THEN RETURN; END IF; + SELECT account_limit INTO previous FROM artifact_storage_settings WHERE id=1 FOR UPDATE; + IF previous=limit_bytes AND (SELECT artifact_limit FROM artifact_storage_settings WHERE id=1)=part_limit THEN RETURN; END IF; + IF limit_bytes<=0 OR part_limit<=0 THEN RAISE EXCEPTION 'explicit artifact quota required'; END IF; + IF previous=0 THEN + PERFORM set_config('lock_timeout','500ms',true); + LOCK TABLE database_artifact IN SHARE ROW EXCLUSIVE MODE; + ALTER TABLE database_artifact ENABLE TRIGGER artifact_metadata_guard; + ALTER TABLE database_artifact ENABLE TRIGGER artifact_metadata_retire; + INSERT INTO artifact_account(owner,legacy_bytes) + SELECT artifact_owner(space), COALESCE(sum(size_bytes),0) + FROM database_artifact WHERE checkpoint_part_id IS NULL GROUP BY artifact_owner(space) + ON CONFLICT(owner) DO UPDATE SET legacy_bytes=EXCLUDED.legacy_bytes; + INSERT INTO artifact_space_owner(space,owner) + SELECT DISTINCT space,artifact_owner(space) FROM database_artifact ON CONFLICT DO NOTHING; + PERFORM set_config('lock_timeout',previous_timeout,true); + END IF; + UPDATE artifact_storage_settings SET account_limit=limit_bytes,artifact_limit=part_limit WHERE id=1; +END $$; + +CREATE FUNCTION artifact_layout_guard() RETURNS trigger LANGUAGE plpgsql AS $$ +BEGIN + IF TG_OP='UPDATE' AND OLD.checkpoint_part_id IS NOT NULL THEN + IF NEW.checkpoint_part_id IS NULL OR NEW.storage_mode<>'checkpoint-parts-v1' + OR (NEW.checkpoint_part_id=OLD.checkpoint_part_id AND + (NEW.revision,NEW.content_hash,NEW.payload,NEW.delta_payload,NEW.layout_generation) + IS DISTINCT FROM + (OLD.revision,OLD.content_hash,OLD.payload,OLD.delta_payload,OLD.layout_generation)) THEN + RAISE EXCEPTION 'TC2917_LAYOUT: legacy writes cannot replace checkpoint parts'; + END IF; + END IF; + IF TG_OP<>'DELETE' AND NEW.checkpoint_part_id IS NOT NULL THEN + IF NEW.storage_mode<>'checkpoint-parts-v1' OR NEW.delta_payload IS NOT NULL + OR octet_length(NEW.payload)<>0 OR NEW.delta_part_id IS NOT NULL THEN + RAISE EXCEPTION 'TC2917_LAYOUT: invalid checkpoint layout'; + END IF; + IF TG_OP='INSERT' THEN + IF NEW.layout_generation<>1 THEN RAISE EXCEPTION 'TC2917_LAYOUT: invalid generation'; END IF; + ELSIF NEW.checkpoint_part_id IS DISTINCT FROM OLD.checkpoint_part_id + AND NEW.layout_generation<>OLD.layout_generation+1 THEN + RAISE EXCEPTION 'TC2917_LAYOUT: invalid generation'; + END IF; + END IF; + IF TG_OP='DELETE' THEN RETURN OLD; ELSE RETURN NEW; END IF; +END $$; +CREATE TRIGGER artifact_layout_guard BEFORE INSERT OR UPDATE OR DELETE ON database_artifact + FOR EACH ROW EXECUTE FUNCTION artifact_layout_guard(); + +CREATE FUNCTION artifact_metadata_guard() RETURNS trigger LANGUAGE plpgsql AS $$ +DECLARE who text; old_bytes bigint:=0; new_bytes bigint:=0; cap bigint; +BEGIN + SELECT account_limit INTO cap FROM artifact_storage_settings WHERE id=1; + IF cap=0 THEN + IF TG_OP='DELETE' THEN RETURN OLD; ELSE RETURN NEW; END IF; + END IF; + IF TG_OP='DELETE' THEN who:=artifact_owner(OLD.space); ELSE who:=artifact_owner(NEW.space); END IF; + IF TG_OP='UPDATE' AND (NEW.service,NEW.space,NEW.name) IS DISTINCT FROM (OLD.service,OLD.space,OLD.name) THEN + RAISE EXCEPTION 'artifact identity is immutable'; + END IF; + INSERT INTO artifact_account(owner) VALUES(who) ON CONFLICT DO NOTHING; + IF TG_OP<>'DELETE' THEN + INSERT INTO artifact_space_owner(space,owner) VALUES(NEW.space,who) ON CONFLICT DO NOTHING; + IF NEW.checkpoint_part_id IS NULL THEN new_bytes:=NEW.size_bytes; END IF; + END IF; + IF TG_OP<>'INSERT' AND OLD.checkpoint_part_id IS NULL THEN old_bytes:=OLD.size_bytes; END IF; + UPDATE artifact_account SET legacy_bytes=legacy_bytes+new_bytes-old_bytes + WHERE owner=who AND (new_bytes<=old_bytes OR legacy_bytes+reserved_bytes-credit_bytes+new_bytes-old_bytes<=cap); + IF NOT FOUND THEN RAISE EXCEPTION 'TC2917_QUOTA'; END IF; + IF TG_OP='UPDATE' AND NEW.checkpoint_part_id IS NULL AND EXISTS( + SELECT 1 FROM artifact_part WHERE service=NEW.service AND space=NEW.space AND name=NEW.name AND replacing + ) THEN RAISE EXCEPTION 'TC2917_LAYOUT: replacement in progress'; END IF; + IF TG_OP='DELETE' THEN RETURN OLD; ELSE RETURN NEW; END IF; +END $$; +CREATE TRIGGER artifact_metadata_guard BEFORE INSERT OR UPDATE OR DELETE ON database_artifact + FOR EACH ROW EXECUTE FUNCTION artifact_metadata_guard(); +ALTER TABLE database_artifact DISABLE TRIGGER artifact_metadata_guard; + +-- Payload deletion is one chunk/statement. The owner lock is shared with +-- reservations, legacy writers and pointer publication. 'deleting' is a fence: +-- time determines eligibility, never permission to resume or publish a part. +CREATE FUNCTION artifact_reap_one(p_service text,p_space text,p_name text,p_id text, + abort_upload boolean DEFAULT false) RETURNS boolean LANGUAGE plpgsql SET lock_timeout='50ms' AS $$ +DECLARE p artifact_part%ROWTYPE; next_ordinal integer; child_credit bigint; +BEGIN + SELECT * INTO p FROM artifact_part WHERE service=p_service AND space=p_space AND name=p_name AND part_id=p_id; + IF NOT FOUND THEN RETURN false; END IF; + PERFORM 1 FROM artifact_account WHERE owner=p.owner FOR UPDATE; + SELECT * INTO p FROM artifact_part WHERE service=p_service AND space=p_space + AND name=p_name AND part_id=p_id FOR UPDATE; + IF NOT FOUND OR EXISTS(SELECT 1 FROM database_artifact d WHERE d.service=p.service + AND d.space=p.space AND d.name=p.name AND + (d.checkpoint_part_id=p.part_id OR d.delta_part_id=p.part_id)) THEN RETURN false; END IF; + IF NOT abort_upload AND p.state<>'deleting' AND p.lease_until>clock_timestamp() THEN RETURN false; END IF; + UPDATE artifact_part SET state='deleting',lease_until=clock_timestamp() WHERE service=p.service AND space=p.space AND name=p.name AND part_id=p.part_id; + SELECT ordinal INTO next_ordinal FROM artifact_chunk WHERE service=p.service AND space=p.space + AND name=p.name AND part_id=p.part_id ORDER BY ordinal LIMIT 1; + IF FOUND THEN DELETE FROM artifact_chunk WHERE service=p.service AND space=p.space + AND name=p.name AND part_id=p.part_id AND ordinal=next_ordinal; END IF; + IF NOT EXISTS(SELECT 1 FROM artifact_chunk WHERE service=p.service AND space=p.space AND name=p.name AND part_id=p.part_id) THEN + SELECT COALESCE(sum(replacement_credit),0) INTO child_credit FROM artifact_part + WHERE service=p.service AND space=p.space AND name=p.name AND replaces_part_id=p.part_id; + UPDATE artifact_part SET replacement_credit=0,replacing=false,replaces_part_id=NULL + WHERE service=p.service AND space=p.space AND name=p.name AND replaces_part_id=p.part_id; + DELETE FROM artifact_part WHERE service=p.service AND space=p.space AND name=p.name AND part_id=p.part_id; + UPDATE artifact_account SET reserved_bytes=reserved_bytes-p.size_bytes, + credit_bytes=credit_bytes-p.replacement_credit-child_credit WHERE owner=p.owner; + END IF; + RETURN true; +END $$; + +CREATE FUNCTION artifact_metadata_retire() RETURNS trigger LANGUAGE plpgsql AS $$ +DECLARE released bigint; +BEGIN + IF TG_OP='DELETE' THEN + SELECT COALESCE(sum(replacement_credit),0) INTO released FROM artifact_part + WHERE service=OLD.service AND space=OLD.space AND name=OLD.name; + UPDATE artifact_part SET state='deleting',lease_until=clock_timestamp(),replacement_credit=0,replacing=false,replaces_part_id=NULL + WHERE service=OLD.service AND space=OLD.space AND name=OLD.name; + IF released>0 THEN UPDATE artifact_account SET credit_bytes=credit_bytes-released + WHERE owner=artifact_owner(OLD.space); END IF; + RETURN OLD; + END IF; + IF OLD.checkpoint_part_id IS DISTINCT FROM NEW.checkpoint_part_id THEN + IF OLD.checkpoint_part_id IS NOT NULL THEN + UPDATE artifact_part SET state='deleting',lease_until=clock_timestamp() WHERE service=OLD.service AND space=OLD.space + AND name=OLD.name AND part_id=OLD.checkpoint_part_id; + ELSE + SELECT replacement_credit INTO released FROM artifact_part WHERE service=NEW.service + AND space=NEW.space AND name=NEW.name AND part_id=NEW.checkpoint_part_id; + UPDATE artifact_part SET replacement_credit=0,replacing=false WHERE service=NEW.service + AND space=NEW.space AND name=NEW.name AND part_id=NEW.checkpoint_part_id; + IF released>0 THEN UPDATE artifact_account SET credit_bytes=credit_bytes-released + WHERE owner=artifact_owner(NEW.space); END IF; + END IF; + END IF; + RETURN NEW; +END $$; +CREATE TRIGGER artifact_metadata_retire AFTER UPDATE OR DELETE ON database_artifact + FOR EACH ROW EXECUTE FUNCTION artifact_metadata_retire(); +ALTER TABLE database_artifact DISABLE TRIGGER artifact_metadata_retire; + +CREATE FUNCTION artifact_reserve(p_service text,p_space text,p_name text,p_id text,p_owner text, + p_size bigint,p_count integer,p_revision bigint,p_hash text) +RETURNS boolean LANGUAGE plpgsql AS $$ +DECLARE base record; existing boolean; credit bigint:=0; cap bigint; +BEGIN + IF p_owner<>artifact_owner(p_space) THEN RAISE EXCEPTION 'TC2917_LAYOUT: owner mismatch'; END IF; + PERFORM 1 FROM artifact_account WHERE owner=p_owner FOR UPDATE; + SELECT revision,size_bytes,checkpoint_content_hash,checkpoint_part_id INTO base FROM database_artifact WHERE service=p_service AND space=p_space AND name=p_name; + existing:=FOUND; + IF (p_revision IS NULL AND existing) OR (p_revision IS NOT NULL AND + (NOT existing OR base.revision<>p_revision OR base.checkpoint_content_hash<>p_hash)) THEN + RAISE EXCEPTION 'TC2917_LAYOUT: superseded reservation'; + END IF; + IF EXISTS(SELECT 1 FROM artifact_part WHERE service=p_service AND space=p_space + AND name=p_name AND replacing) THEN RETURN false; END IF; + IF existing THEN credit:=LEAST(base.size_bytes,p_size); END IF; + SELECT account_limit INTO cap FROM artifact_storage_settings WHERE id=1; + -- Pending shrinkage never creates spare quota for another database: the + -- ledger credits only overlapping bytes, not the shrink's eventual saving. + UPDATE artifact_account SET reserved_bytes=reserved_bytes+p_size,credit_bytes=credit_bytes+credit + WHERE owner=p_owner AND ((existing AND p_size<=base.size_bytes) OR + legacy_bytes+reserved_bytes-credit_bytes + -CASE WHEN existing THEN base.size_bytes ELSE 0 END+p_size<=cap); + IF NOT FOUND THEN RETURN false; END IF; + INSERT INTO artifact_part(service,space,name,part_id,owner,size_bytes,chunk_count, + replacement_credit,replacing,replaces_part_id) + VALUES(p_service,p_space,p_name,p_id,p_owner,p_size,p_count,credit,existing, + CASE WHEN existing THEN base.checkpoint_part_id ELSE NULL END); + RETURN true; +END $$; + +CREATE FUNCTION artifact_metadata_publish() RETURNS trigger LANGUAGE plpgsql AS $$ +BEGIN + IF NEW.checkpoint_part_id IS NOT NULL THEN + UPDATE artifact_part SET state='published',lease_until='infinity'::timestamptz + WHERE service=NEW.service AND space=NEW.space AND name=NEW.name + AND part_id=NEW.checkpoint_part_id AND state='sealed'; + IF NOT EXISTS(SELECT 1 FROM artifact_part WHERE service=NEW.service AND space=NEW.space + AND name=NEW.name AND part_id=NEW.checkpoint_part_id AND state='published') THEN + RAISE EXCEPTION 'TC2917_LAYOUT: unpublished part'; + END IF; + END IF; + RETURN NEW; +END $$; +CREATE TRIGGER artifact_metadata_publish AFTER INSERT OR UPDATE ON database_artifact + FOR EACH ROW EXECUTE FUNCTION artifact_metadata_publish(); diff --git a/tinycloud-core/src/db.rs b/tinycloud-core/src/db.rs index 9bc00325e..05c8dd884 100644 --- a/tinycloud-core/src/db.rs +++ b/tinycloud-core/src/db.rs @@ -662,8 +662,9 @@ fn migration_lock_wait_error(error: DbErr) -> DbErr { return error; } DbErr::Migration(format!( - "a migration gave up after waiting {MIGRATION_LOCK_TIMEOUT} (lock_timeout) for a table \ - lock; another session holds a conflicting lock (SQLSTATE 55P03: {database}). Nothing \ + "a migration reached its configured lock_timeout (boot default {MIGRATION_LOCK_TIMEOUT}; \ + migrations may set a shorter limit) while waiting for a table lock; another session \ + holds a conflicting lock (SQLSTATE 55P03: {database}). Nothing \ was applied; end that session and restart." )) } diff --git a/tinycloud-core/src/duckdb/service.rs b/tinycloud-core/src/duckdb/service.rs index d78ed984b..909892e34 100644 --- a/tinycloud-core/src/duckdb/service.rs +++ b/tinycloud-core/src/duckdb/service.rs @@ -666,6 +666,7 @@ fn artifact_error_to_duckdb(err: DatabaseArtifactError) -> DuckDbError { match err { DatabaseArtifactError::WriteLimitExceeded { .. } => DuckDbError::DatabaseSizeLimit, DatabaseArtifactError::ReadLimitExceeded { .. } => DuckDbError::DatabaseTooLargeToLoad, + DatabaseArtifactError::StorageLimit => DuckDbError::StorageWouldGrow, err => DuckDbError::Internal(err.to_string()), } } diff --git a/tinycloud-core/src/migrations/m20261113_010000_artifact_chunks.rs b/tinycloud-core/src/migrations/m20261113_010000_artifact_chunks.rs new file mode 100644 index 000000000..6aad07534 --- /dev/null +++ b/tinycloud-core/src/migrations/m20261113_010000_artifact_chunks.rs @@ -0,0 +1,45 @@ +use sea_orm::{ConnectionTrait, DbBackend, Statement}; +use sea_orm_migration::prelude::*; + +#[derive(DeriveMigrationName)] +pub struct Migration; + +#[async_trait::async_trait] +impl MigrationTrait for Migration { + async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> { + if manager.get_database_backend() != DbBackend::Postgres { + return Ok(()); + } + let conn = manager.get_connection(); + let previous: String = conn + .query_one(Statement::from_string( + DbBackend::Postgres, + "SHOW lock_timeout".to_string(), + )) + .await? + .ok_or_else(|| DbErr::Migration("missing lock timeout".into()))? + .try_get("", "lock_timeout")?; + conn.execute_unprepared("SET LOCAL lock_timeout = '500ms'") + .await?; + manager + .get_connection() + .execute_unprepared(include_str!("../database_artifacts/chunks.sql")) + .await?; + conn.execute(Statement::from_sql_and_values( + DbBackend::Postgres, + "SELECT set_config('lock_timeout', $1, true)", + [previous.into()], + )) + .await?; + Ok(()) + } + + async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> { + if manager.get_database_backend() != DbBackend::Postgres { + return Ok(()); + } + Err(DbErr::Migration( + "artifact chunk storage is forward-only".into(), + )) + } +} diff --git a/tinycloud-core/src/migrations/mod.rs b/tinycloud-core/src/migrations/mod.rs index 621ac34d5..4bbfd850c 100644 --- a/tinycloud-core/src/migrations/mod.rs +++ b/tinycloud-core/src/migrations/mod.rs @@ -39,6 +39,8 @@ pub mod m20261112_000000_renewal_identity; pub mod m20261113_000000_database_artifact_external; +pub mod m20261113_010000_artifact_chunks; + pub struct Migrator; #[async_trait::async_trait] @@ -83,6 +85,7 @@ impl MigratorTrait for Migrator { Box::new(m20261111_000000_delegation_renewals::Migration), Box::new(m20261112_000000_renewal_identity::Migration), Box::new(m20261113_000000_database_artifact_external::Migration), + Box::new(m20261113_010000_artifact_chunks::Migration), ] } } diff --git a/tinycloud-core/src/sql/database.rs b/tinycloud-core/src/sql/database.rs index bd428a24d..4aeb1a524 100644 --- a/tinycloud-core/src/sql/database.rs +++ b/tinycloud-core/src/sql/database.rs @@ -48,6 +48,9 @@ enum DbMessage { Export { response_tx: oneshot::Sender, SqlError>>, }, + Snapshot { + response_tx: oneshot::Sender>, + }, Checkpoint { response_tx: oneshot::Sender, SqlError>>, }, @@ -172,6 +175,20 @@ impl DatabaseHandle { .map_err(|_| SqlError::Internal("Database actor dropped response".to_string()))? } + pub async fn snapshot(&self) -> Result { + let (response_tx, response_rx) = oneshot::channel(); + self.tx + .send(( + request_telemetry::current(), + DbMessage::Snapshot { response_tx }, + )) + .await + .map_err(|_| SqlError::Internal("Database actor not available".into()))?; + response_rx + .await + .map_err(|_| SqlError::Internal("Database actor dropped response".into()))? + } + pub async fn checkpoint(&self) -> Result, SqlError> { let (response_tx, response_rx) = oneshot::channel(); self.tx @@ -407,6 +424,13 @@ pub fn spawn_actor_with_policy( let result = handle_export(&conn, &mode, &file_path); let _ = response_tx.send(result); } + DbMessage::Snapshot { response_tx } => { + let result = handle_snapshot(&conn, &mode); + if result.is_ok() { + policy.budget.forget(&key); + } + let _ = response_tx.send(result); + } DbMessage::Checkpoint { response_tx } => { let result = handle_checkpoint(&conn, &mode, &file_path); if result.is_ok() { @@ -497,6 +521,28 @@ fn handle_export( std::fs::read(&temp_path).map_err(|e| SqlError::Internal(e.to_string())) } +fn handle_snapshot( + conn: &rusqlite::Connection, + mode: &StorageMode, +) -> Result { + if matches!(mode, StorageMode::File(_)) { + let busy: i64 = conn + .query_row("PRAGMA wal_checkpoint(TRUNCATE)", [], |row| row.get(0)) + .map_err(|e| SqlError::Internal(e.to_string()))?; + if busy != 0 { + return Err(SqlError::Internal("Database checkpoint is busy".into())); + } + } + let snapshot = tempfile::NamedTempFile::new().map_err(|e| SqlError::Internal(e.to_string()))?; + let mut dest = rusqlite::Connection::open(snapshot.path()) + .map_err(|e| SqlError::Internal(e.to_string()))?; + rusqlite::backup::Backup::new(conn, &mut dest) + .and_then(|backup| backup.run_to_completion(i32::MAX, std::time::Duration::ZERO, None)) + .map_err(|e| SqlError::Internal(e.to_string()))?; + drop(dest); + Ok(snapshot) +} + fn handle_checkpoint( conn: &rusqlite::Connection, mode: &StorageMode, diff --git a/tinycloud-core/src/sql/service.rs b/tinycloud-core/src/sql/service.rs index 8dfa95962..9bd461903 100644 --- a/tinycloud-core/src/sql/service.rs +++ b/tinycloud-core/src/sql/service.rs @@ -978,7 +978,10 @@ impl SqlService { .saturating_add(1); let factor = 1_u32.checked_shl(failures - 1).unwrap_or(u32::MAX); let delay = first.saturating_mul(factor).min(max); - let refused = matches!(error, SqlError::DatabaseSizeLimit); + let refused = matches!( + error, + SqlError::DatabaseSizeLimit | SqlError::StorageWouldGrow + ); self.checkpoint_backoff.insert( key.clone(), CheckpointBackoff { @@ -1006,6 +1009,62 @@ impl SqlService { let key = (space.to_string(), db_name.to_string()); let expected = self.expectation(&key); + if self + .artifact_repository + .chunked_checkpoints("sql", &key.0, db_name) + { + let snapshot = handle.snapshot().await?; + let bytes = snapshot + .as_file() + .metadata() + .map_err(|e| SqlError::Internal(e.to_string()))? + .len(); + if growth == GrowthPolicy::Forbid { + let durable = self + .artifact_repository + .load_meta("sql", &key.0, db_name) + .await + .map_err(request_artifact_error(space, db_name))? + .map_or(0, |m| m.size_bytes.max(0) as u64); + if bytes > durable { + return Err(SqlError::StorageWouldGrow); + } + } + let saved = match self + .bounded( + (space, db_name), + || SqlBusyStage::ArtifactTransfer, + self.bounds.artifact_transfer, + self.artifact_repository + .save_snapshot("sql", &key.0, db_name, snapshot, expected), + ) + .await + { + Ok(Ok(saved)) => saved, + Ok(Err(error)) => { + let lineage_conflict = matches!(error, DatabaseArtifactError::StaleLineage); + let error = request_artifact_error(space, db_name)(error); + if !lineage_conflict { + self.back_off_checkpoint(&key, &error); + } + return Err(error); + } + Err(error) => { + self.back_off_checkpoint(&key, &error); + return Err(error); + } + }; + self.checkpoint_backoff.remove(&key); + self.lineage.insert( + key, + ArtifactExpectation::Derived { + revision: saved.revision, + checkpoint_content_hash: saved.checkpoint_content_hash, + }, + ); + return Ok(()); + } + // Repeated DELETE/UPDATE/DROP operations can append WAL frames without // adding a database page. Full spaces replace checkpoint + WAL, but // must also bound the resulting image against the durable charge: @@ -1207,6 +1266,7 @@ fn request_artifact_error<'a>( fn artifact_error_to_sql(err: DatabaseArtifactError) -> SqlError { match err { + DatabaseArtifactError::StorageLimit => SqlError::StorageWouldGrow, DatabaseArtifactError::Db(ref error) if is_transient_error(error) => { tracing::warn!( service = "sql", @@ -1331,7 +1391,7 @@ mod tests { SeaOrmDatabaseArtifactRepository, }, migrations::Migrator, - sea_orm::{ConnectOptions, Database, DbErr}, + sea_orm::{ConnectOptions, ConnectionTrait, Database, DbErr}, sea_orm_migration::MigratorTrait, sql_sizes::{SizeTrackingArtifactRepository, SqlSizes}, }; @@ -2627,6 +2687,312 @@ mod tests { ); } + #[tokio::test] + async fn postgres_chunked_checkpoint_save_restart_read() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_checkpoint_save_restart_read", + ) + .await + else { + return; + }; + let repo = Arc::new( + SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()).with_chunk_limits(Some( + crate::database_artifacts::chunks::ChunkLimits { + artifact_bytes: 16 * 1024 * 1024, + account_bytes: 64 * 1024 * 1024, + }, + )), + ); + let cache_one = TempDir::new().unwrap(); + let cache_two = TempDir::new().unwrap(); + let space = test_space_id("chunked-checkpoint"); + let service = SqlService::new( + cache_one.path().to_string_lossy().into_owned(), + 0, + repo.clone(), + ); + service + .execute( + &space, + "main", + SqlRequest::Execute { + schema: Some(vec![ + "CREATE TABLE items (id INTEGER PRIMARY KEY, value TEXT, body BLOB)".into(), + ]), + sql: "INSERT INTO items(value,body) VALUES ('durable',randomblob(2097153))" + .into(), + params: Vec::new(), + }, + None, + "tinycloud.sql/admin".into(), + ) + .await + .unwrap(); + let superseded_part = fixture + .conn + .query_one(sea_orm::Statement::from_string( + sea_orm::DbBackend::Postgres, + "SELECT checkpoint_part_id FROM database_artifact".to_string(), + )) + .await + .unwrap() + .unwrap() + .try_get::("", "checkpoint_part_id") + .unwrap(); + service + .execute( + &space, + "main", + SqlRequest::Execute { + schema: None, + sql: "INSERT INTO items(value,body) VALUES ('updated',x'01')".into(), + params: Vec::new(), + }, + None, + "tinycloud.sql/admin".into(), + ) + .await + .unwrap(); + let before = service.export(&space, "main").await.unwrap(); + service + .discard_local_state(&(space.to_string(), "main".into())) + .await + .unwrap(); + drop(service); + // A fresh repository and empty local cache must use the stored parts. + let reopened_repo = Arc::new( + SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()).with_chunk_limits(Some( + crate::database_artifacts::chunks::ChunkLimits { + artifact_bytes: 16 * 1024 * 1024, + account_bytes: 64 * 1024 * 1024, + }, + )), + ); + let reopened = SqlService::new( + cache_two.path().to_string_lossy().into_owned(), + 0, + reopened_repo, + ); + let result = reopened + .execute( + &space, + "main", + SqlRequest::Query { + sql: "SELECT value,length(body) FROM items ORDER BY id".into(), + params: Vec::new(), + max_rows: None, + max_bytes: None, + }, + None, + "tinycloud.sql/read".into(), + ) + .await + .unwrap(); + match result.response { + SqlResponse::Query(query) => { + assert_eq!(query.row_count, 2); + assert_eq!(query.rows[0][0], SqlValue::Text("durable".into())); + assert_eq!(query.rows[1][0], SqlValue::Text("updated".into())); + } + other => panic!("unexpected {other:?}"), + } + assert_eq!(before, reopened.export(&space, "main").await.unwrap()); + repo.reap_unused_parts(64).await.unwrap(); + let current_meta = repo + .load_meta("sql", &space.to_string(), "main") + .await + .unwrap() + .unwrap(); + let state = fixture.conn.query_one(sea_orm::Statement::from_string( + sea_orm::DbBackend::Postgres, + "SELECT max(octet_length(payload))::bigint AS max_bytes, count(*)::bigint AS chunks, count(*) FILTER(WHERE pg_column_compression(payload) IS NOT NULL)::bigint AS compressed FROM artifact_chunk".to_string(), + )).await.unwrap().unwrap(); + assert_eq!( + state.try_get::("", "chunks").unwrap(), + (current_meta.checkpoint_size_bytes as u64).div_ceil(1024 * 1024) as i64 + ); + let old_chunks = fixture + .conn + .query_one(sea_orm::Statement::from_sql_and_values( + sea_orm::DbBackend::Postgres, + "SELECT count(*)::bigint AS chunks FROM artifact_chunk WHERE part_id=$1", + [superseded_part.into()], + )) + .await + .unwrap() + .unwrap() + .try_get::("", "chunks") + .unwrap(); + assert_eq!(old_chunks, 0, "superseded checkpoint must be reclaimed"); + assert!(state.try_get::("", "max_bytes").unwrap() <= 1024 * 1024); + assert_eq!(state.try_get::("", "compressed").unwrap(), 0); + let meta = repo + .load_meta("sql", &space.to_string(), "main") + .await + .unwrap() + .unwrap(); + assert_eq!(meta.revision, 2); + assert_eq!(meta.delta_size_bytes, 0); + reopened + .discard_local_state(&(space.to_string(), "main".into())) + .await + .unwrap(); + } + + #[tokio::test] + async fn postgres_chunked_checkpoint_flag_off_reopen_can_write() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_checkpoint_flag_off_reopen_can_write", + ) + .await + else { + return; + }; + let repo = Arc::new( + SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()).with_chunk_limits(Some( + crate::database_artifacts::chunks::ChunkLimits { + artifact_bytes: 1024 * 1024, + account_bytes: 4 * 1024 * 1024, + }, + )), + ); + let first = TempDir::new().unwrap(); + let second = TempDir::new().unwrap(); + let space = test_space_id("flag-off"); + let service = SqlService::new(first.path().to_string_lossy().into_owned(), 0, repo); + service + .execute( + &space, + "main", + SqlRequest::Execute { + schema: Some(vec!["CREATE TABLE items(value TEXT)".into()]), + sql: "INSERT INTO items VALUES ('first')".into(), + params: vec![], + }, + None, + "tinycloud.sql/admin".into(), + ) + .await + .unwrap(); + service + .discard_local_state(&(space.to_string(), "main".into())) + .await + .unwrap(); + drop(service); + let flag_off = Arc::new(SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone())); + assert!(flag_off.chunk_storage_active().await.unwrap()); + let reopened = SqlService::new( + second.path().to_string_lossy().into_owned(), + 0, + flag_off.clone(), + ); + reopened + .execute( + &space, + "main", + SqlRequest::Execute { + schema: None, + sql: "INSERT INTO items VALUES ('second')".into(), + params: vec![], + }, + None, + "tinycloud.sql/admin".into(), + ) + .await + .unwrap(); + let result = reopened + .execute( + &space, + "main", + SqlRequest::Query { + sql: "SELECT value FROM items ORDER BY rowid".into(), + params: vec![], + max_rows: None, + max_bytes: None, + }, + None, + "tinycloud.sql/read".into(), + ) + .await + .unwrap(); + match result.response { + SqlResponse::Query(query) => { + assert_eq!(query.row_count, 2); + assert_eq!(query.rows[1][0], SqlValue::Text("second".into())); + } + other => panic!("unexpected {other:?}"), + } + assert_eq!( + flag_off + .load_meta("sql", &space.to_string(), "main") + .await + .unwrap() + .unwrap() + .storage_mode, + "checkpoint-parts-v1" + ); + reopened + .discard_local_state(&(space.to_string(), "main".into())) + .await + .unwrap(); + } + + #[tokio::test] + async fn postgres_chunked_checkpoint_refusal_preserves_backoff() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_checkpoint_refusal_preserves_backoff", + ) + .await + else { + return; + }; + let repo = Arc::new( + SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()) + .with_chunk_limits(Some(crate::database_artifacts::chunks::ChunkLimits { + artifact_bytes: 1024 * 1024, + account_bytes: 4 * 1024 * 1024, + })) + .with_max_write_bytes(1), + ); + let cache = TempDir::new().unwrap(); + let space = test_space_id("chunk-backoff"); + let service = SqlService::new(cache.path().to_string_lossy().into_owned(), 0, repo.clone()); + for _ in 0..2 { + let outcome = service + .execute( + &space, + "main", + SqlRequest::Execute { + schema: Some(vec!["CREATE TABLE items(value TEXT)".into()]), + sql: "INSERT INTO items VALUES ('not saved')".into(), + params: vec![], + }, + None, + "tinycloud.sql/admin".into(), + ) + .await; + assert!( + matches!(outcome, Err(SqlError::DatabaseSizeLimit)), + "{outcome:?}" + ); + assert_eq!( + service + .checkpoint_backoff + .get(&(space.to_string(), "main".into())) + .unwrap() + .failures, + 1, + "immediate retry must not perform another failed checkpoint" + ); + } + assert!(repo + .load_meta("sql", &space.to_string(), "main") + .await + .unwrap() + .is_none()); + } + #[tokio::test] async fn sql_write_survives_service_recreation_with_empty_cache() { let repo = artifact_repository().await; diff --git a/tinycloud-core/src/sql_sizes.rs b/tinycloud-core/src/sql_sizes.rs index 63960010c..438f643fa 100644 --- a/tinycloud-core/src/sql_sizes.rs +++ b/tinycloud-core/src/sql_sizes.rs @@ -119,6 +119,28 @@ impl DatabaseArtifactRepository for SizeTrackingArtifactRepository { .await } + fn chunked_checkpoints(&self, service: &str, space: &str, name: &str) -> bool { + self.inner.chunked_checkpoints(service, space, name) + } + + async fn save_snapshot( + &self, + service: &str, + space: &str, + name: &str, + snapshot: tempfile::NamedTempFile, + expected: ArtifactExpectation, + ) -> Result { + let saved = self + .inner + .save_snapshot(service, space, name, snapshot, expected) + .await?; + self.sizes + .update(service, space, name, saved.size_bytes.max(0) as u64) + .await; + Ok(saved) + } + async fn save( &self, service: &str, diff --git a/tinycloud-node-server/src/config.rs b/tinycloud-node-server/src/config.rs index 5a3a13d90..53e61ab95 100644 --- a/tinycloud-node-server/src/config.rs +++ b/tinycloud-node-server/src/config.rs @@ -1606,6 +1606,13 @@ pub struct SpacesConfig { #[derive(Serialize, Deserialize, Debug, Clone, Hash, PartialEq, Eq)] pub struct SqlStorageConfig { + /// Opt-in PostgreSQL checkpoint parts (TC-2917); requires explicit capacities. + #[serde(default)] + pub chunked_artifacts: bool, + #[serde(default)] + pub chunked_artifact_max_bytes: u64, + #[serde(default)] + pub chunked_account_max_bytes: u64, #[serde(default)] pub path: Option, #[serde(default = "default_sql_memory_threshold")] @@ -1743,6 +1750,9 @@ impl SqlStorageConfig { impl Default for SqlStorageConfig { fn default() -> Self { Self { + chunked_artifacts: false, + chunked_artifact_max_bytes: 0, + chunked_account_max_bytes: 0, path: None, memory_threshold: default_sql_memory_threshold(), lock_wait_timeout_ms: default_sql_lock_wait_timeout_ms(), diff --git a/tinycloud-node-server/src/lib.rs b/tinycloud-node-server/src/lib.rs index 97f090a20..e48d7e9ad 100644 --- a/tinycloud-node-server/src/lib.rs +++ b/tinycloud-node-server/src/lib.rs @@ -609,6 +609,20 @@ pub fn resolve_runtime_config(tinycloud_config: &Config) -> Result { tracing::RequestIdOrigins::new(&resolved.log.request_id_first_party_origins) .map_err(anyhow::Error::msg)?; resolved.storage.resolve(); + if resolved.storage.sql.chunked_artifacts { + anyhow::ensure!( + resolved.storage.database().starts_with("postgres"), + "chunked artifacts require PostgreSQL" + ); + anyhow::ensure!( + resolved.storage.sql.chunked_artifact_max_bytes > 0 + && resolved.storage.sql.chunked_account_max_bytes > 0 + && resolved.storage.sql.chunked_artifact_max_bytes <= i64::MAX as u64 + && resolved.storage.sql.chunked_account_max_bytes <= i64::MAX as u64, + "chunked artifacts require explicit positive capacities within int64" + ); + } + resolved .sql_store .validate_for_database(resolved.storage.database()) @@ -990,27 +1004,35 @@ pub async fn app_with_control( tc_bench_block_get, tc_bench_health, ]); - let database_artifact_repository: Arc = - Arc::new(SizeTrackingArtifactRepository::new( - Arc::new( - node_artifact_repository(&tinycloud) - .with_max_write_bytes( - tinycloud_config - .storage - .sql - .max_artifact_write_bytes - .as_u64(), - ) - .with_max_read_bytes( - tinycloud_config - .storage - .sql - .max_artifact_read_bytes - .as_u64(), - ), - ), - sql_sizes.clone(), + let raw_artifact_repository = node_artifact_repository(&tinycloud) + .with_max_write_bytes( + tinycloud_config + .storage + .sql + .max_artifact_write_bytes + .as_u64(), + ) + .with_max_read_bytes( + tinycloud_config + .storage + .sql + .max_artifact_read_bytes + .as_u64(), + ) + .with_chunk_limits(tinycloud_config.storage.sql.chunked_artifacts.then_some( + tinycloud_core::database_artifacts::chunks::ChunkLimits { + artifact_bytes: tinycloud_config.storage.sql.chunked_artifact_max_bytes, + account_bytes: tinycloud_config.storage.sql.chunked_account_max_bytes, + }, )); + raw_artifact_repository.configure_chunked_storage().await?; + let artifact_collector = raw_artifact_repository + .chunk_storage_active() + .await? + .then_some(raw_artifact_repository.clone()); + let database_artifact_repository: Arc = Arc::new( + SizeTrackingArtifactRepository::new(Arc::new(raw_artifact_repository), sql_sizes.clone()), + ); // Seed the SQL-size mirror AFTER `TinyCloud::new` ran migrations — the // `database_artifact` table now exists (seeding before migrations would @@ -1117,6 +1139,28 @@ pub async fn app_with_control( let renewal_scheduler_tinycloud = tinycloud.clone(); let renewal_scheduler_secret = key_setup.clone(); let rocket = rocket::custom(config) + .attach(AdHoc::on_liftoff("artifact-part-collector", move |rocket| { + let collector=artifact_collector.clone(); + let shutdown=rocket.shutdown(); + Box::pin(async move { + if let Some(collector)=collector { + tokio::spawn(async move { + let mut interval=tokio::time::interval(Duration::from_secs(10)); + tokio::pin!(shutdown); + loop { + tokio::select! { + _=interval.tick()=> { + if !matches!(tokio::time::timeout(Duration::from_secs(1), collector.reap_unused_parts(128)).await, Ok(Ok(()))) { + ::tracing::warn!(event="artifact_part_collection_failed", "Artifact part collection deferred"); + } + }, + _=&mut shutdown=>break, + } + } + }); + } + }) + })) .attach(AdHoc::on_liftoff("webhook-egress-refresh", { let policy = egress_policy.clone(); move |rocket| {