diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a25f1b9..b60d1435 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ - SQL and DuckDB: a whole-checkpoint save over 8 MiB no longer opens a transaction on PostgreSQL. Its metadata read and its compare-and-swap write each run as one autocommit statement under the base `statement_timeout` (30 s by default), so no transaction or row lock is held while the payload uploads. A save the server cancels at that timeout applied nothing and now answers a retryable `503`, as smaller saves already did, instead of `500 outcome_unknown`. `database.bulk_statement_timeout_ms` is removed: a value still set in the file or environment is ignored, and the node logs a warning naming it at startup. Details: TC-2961. - `/invoke`: a KV invocation that writes (`kv/put` or `kv/del`) and also reads (`kv/get`, `kv/list`, `kv/metadata` or `tinycloud.capabilities/read`) is refused with the same `501` as before, but now before any work: nothing is written and the signed request is not spent. Before, the write committed and only then was the invocation answered `501`, since `/invoke` returns one outcome. Internal callers that still mix the two get their reads after the write commits, holding no database transaction, row lock or write order. A `kv/get` answers the value the invocation wrote. If such a read fails, the error says the write was saved. Details: TC-2986. - `POST /delegate` and `POST /revoke` check the signature (and, for a renewal lease, the signature of the stored original it renews) before taking any lock. A forged or slow-to-verify request no longer holds the delegation's chain guards, its spaces' write order or a database transaction while it is checked; it is refused at once. A renewal lease whose original is not stored is also refused before any lock. Inside the transaction only the signed time window is checked again. A resubmitted delegation that is already stored is now also signature-checked before it is answered as existing. Details: TC-2986. -- Add opt-in PostgreSQL checkpoint parts for SQL persistence, with explicit artifact/account capacities and streamed hydration. A database stored in parts loads in bounded statements and is not held to the single-row read limit (`storage.sql.max_artifact_read_bytes`); a load that keeps the whole database in memory (a DuckDB export) still waits for one of the large-transfer slots. Details: TC-2917. +- Add opt-in PostgreSQL checkpoint parts for SQL persistence, with explicit artifact/account capacities (checked when a database is written) and streamed hydration. A database stored in parts loads in bounded statements and is not held to the single-row read limit (`storage.sql.max_artifact_read_bytes`); a load into memory over 8 MiB still waits for one of the large-transfer slots. Details: TC-2917. - Policy/v3: minting a session and registering a policy no longer re-verify signatures or re-run the claim, presentation, credential admission and allow evaluations while holding their spaces' write order and row locks. Under the locks they check that the challenge, registration and roots are unchanged; those decisions were made before the locks, at the same time, from the same inputs. Refusal codes are unchanged, except that `credential-evaluation-changed` and `policy-evaluation-changed` can no longer occur. Details: TC-2986. - SQLite stamp replay deadlines now use the same timestamp encoding as cleanup (TC-2920). - PostgreSQL database artifacts use uncompressed external storage for new writes to avoid repeated decompression during chunked reads (TC-2917). Existing large artifacts require an operator re-toast; verify both storage settings after rollout and follow `scripts/tc2917-README.md` if lock contention deferred the migration. diff --git a/docs/operations/chunked-sql-artifacts.md b/docs/operations/chunked-sql-artifacts.md index 142e578c..275f82cd 100644 --- a/docs/operations/chunked-sql-artifacts.md +++ b/docs/operations/chunked-sql-artifacts.md @@ -7,10 +7,13 @@ lineage semantics. Hydration verifies the part CID before publishing its cache. A database stored in parts is not held to the single-row read limit (`storage.sql.max_artifact_read_bytes`, TC-2961): each statement reads at most -1 MiB, hydration stages the parts in files beside the cache, and the artifact -and account capacities bound its size. A read that keeps the whole database in -memory (a DuckDB export) still waits for one of the node's large-transfer slots -when it exceeds 8 MiB. +1 MiB, and hydration stages the parts in files beside the cache. Its size is +bounded by the artifact and account capacities when it is written; a read does +not re-check them, so a database written under a larger capacity still loads +whole after the capacity is lowered. A read into a sink that keeps the whole +artifact in memory waits for one of the node's large-transfer slots when it +exceeds 8 MiB. A SQL export of a database in parts is held to neither the read +limit nor a slot yet; that is required before the switch is enabled. The switch is off by default. Set explicit capacities before enabling: diff --git a/tinycloud-core/src/database_artifacts.rs b/tinycloud-core/src/database_artifacts.rs index a8929a7e..aa72432d 100644 --- a/tinycloud-core/src/database_artifacts.rs +++ b/tinycloud-core/src/database_artifacts.rs @@ -221,10 +221,9 @@ pub trait ArtifactSink: Send { async fn write(&mut self, bytes: &[u8]) -> Result<(), DatabaseArtifactError>; async fn finish_part(&mut self) -> Result<(), DatabaseArtifactError>; /// Whether the sink holds the whole artifact in memory. Such a read of a - /// large artifact waits for a large-transfer slot in either layout. - fn buffers_in_memory(&self) -> bool { - false - } + /// large artifact waits for a large-transfer slot in either layout. There + /// is no default: a sink that wraps another forwards the inner answer. + fn buffers_in_memory(&self) -> bool; } /// The most [`MemorySink`] allocates ahead of the bytes it is given. @@ -501,6 +500,10 @@ pub(crate) async fn sweep_stale_staging(finals: &[&std::path::Path]) { #[async_trait] impl ArtifactSink for CacheFileSink { + fn buffers_in_memory(&self) -> bool { + false + } + async fn reset(&mut self) -> Result<(), DatabaseArtifactError> { self.discard(); Ok(()) @@ -1832,6 +1835,9 @@ mod tests { #[async_trait] impl ArtifactSink for GatedSink { + fn buffers_in_memory(&self) -> bool { + false + } async fn reset(&mut self) -> Result<(), DatabaseArtifactError> { Ok(()) } @@ -2602,6 +2608,9 @@ mod tests { #[async_trait] impl ArtifactSink for RaceSink<'_> { + fn buffers_in_memory(&self) -> bool { + false + } async fn reset(&mut self) -> Result<(), DatabaseArtifactError> { self.attempts += 1; self.disturbed_this_attempt = false; @@ -2702,6 +2711,9 @@ mod tests { struct DeltaRace<'a>(RaceSink<'a>, bool); #[async_trait] impl ArtifactSink for DeltaRace<'_> { + fn buffers_in_memory(&self) -> bool { + false + } async fn reset(&mut self) -> Result<(), DatabaseArtifactError> { self.0.reset().await } @@ -2805,6 +2817,9 @@ mod tests { ); #[async_trait] impl ArtifactSink for Overtaken<'_> { + fn buffers_in_memory(&self) -> bool { + false + } async fn reset(&mut self) -> Result<(), DatabaseArtifactError> { self.0.reset().await } diff --git a/tinycloud-core/src/database_artifacts/chunks.rs b/tinycloud-core/src/database_artifacts/chunks.rs index a673cbbd..58801b54 100644 --- a/tinycloud-core/src/database_artifacts/chunks.rs +++ b/tinycloud-core/src/database_artifacts/chunks.rs @@ -1735,6 +1735,9 @@ mod tests { #[async_trait] impl ArtifactSink for CountingSink { + fn buffers_in_memory(&self) -> bool { + false + } async fn reset(&mut self) -> Result<(), DatabaseArtifactError> { self.bytes = 0; Ok(()) @@ -1838,10 +1841,14 @@ mod tests { // A streaming sink holds no more than a statement's bytes: no slot. let mut streamed = CountingSink::default(); - read_artifact(&repo, "sql", space, "main", &mut streamed) - .await - .unwrap() - .unwrap(); + tokio::time::timeout( + std::time::Duration::from_secs(30), + read_artifact(&repo, "sql", space, "main", &mut streamed), + ) + .await + .expect("a streaming read waited for a transfer slot") + .unwrap() + .unwrap(); assert_eq!(streamed.bytes, size as u64); // A sink holding the whole artifact waits until a slot frees. @@ -1862,4 +1869,95 @@ mod tests { drop(held); assert_eq!(waiting.await.unwrap().unwrap(), size); } + + /// A row that reads as parts but turns out legacy falls back without + /// holding a parts slot, and the legacy read keeps its own cap and slot. + #[tokio::test] + async fn postgres_chunked_fallback_to_legacy_holds_no_parts_slot() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_chunked_fallback_to_legacy_holds_no_parts_slot", + ) + .await + else { + return; + }; + let space = "tinycloud:pkh:eip155:1:0x0000000000000000000000000000000000000001:fallback"; + let size = BASE_TIMEOUT_PAYLOAD_BYTES + 1; + let repo = SeaOrmDatabaseArtifactRepository::new(fixture.conn.clone()); + repo.save( + "sql", + space, + "fb", + vec![7; size], + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + // The metadata says parts while the row holds a single payload: the + // state a read sees when a parts row is replaced by a legacy one + // between its metadata and row reads. + fixture + .conn + .execute_unprepared( + "UPDATE database_artifact SET storage_mode = 'checkpoint-parts-v1' \ + WHERE name = 'fb'", + ) + .await + .unwrap(); + + // One slot free: an in-memory fallback completes, which it could not + // if the parts slot were still held when the legacy read takes its own. + let held: Vec<_> = (0..MAX_CONCURRENT_LARGE_TRANSFERS - 1) + .map(|_| repo.large_transfers.try_acquire().unwrap()) + .collect(); + let mut sink = MemorySink::default(); + tokio::time::timeout( + std::time::Duration::from_secs(10), + read_artifact(&repo, "sql", space, "fb", &mut sink), + ) + .await + .expect("the fallback held the parts slot into the legacy read") + .unwrap() + .unwrap(); + assert_eq!(sink.checkpoint.len(), size); + assert_eq!(repo.large_transfers.available_permits(), 1); + drop(held); + + // No slot free: even a streaming fallback waits for the legacy slot. + let held: Vec<_> = (0..MAX_CONCURRENT_LARGE_TRANSFERS) + .map(|_| repo.large_transfers.try_acquire().unwrap()) + .collect(); + let waiting = { + let repo = repo.clone(); + tokio::spawn(async move { + let mut sink = CountingSink::default(); + read_artifact(&repo, "sql", space, "fb", &mut sink) + .await + .map(|_| sink.bytes) + }) + }; + tokio::time::sleep(std::time::Duration::from_millis(300)).await; + assert!(!waiting.is_finished(), "the legacy fallback took no slot"); + drop(held); + let read = tokio::time::timeout(std::time::Duration::from_secs(30), waiting) + .await + .expect("the fallback did not finish once a slot freed"); + assert_eq!(read.unwrap().unwrap(), size as u64); + + // The single-row read cap still applies to the fallback. + let capped = repo.clone().with_max_read_bytes(size as u64 - 1); + let refused = + read_artifact(&capped, "sql", space, "fb", &mut CountingSink::default()).await; + assert!( + matches!( + refused, + Err(DatabaseArtifactError::ReadLimitExceeded { .. }) + ), + "{refused:?}" + ); + assert_eq!( + repo.large_transfers.available_permits(), + MAX_CONCURRENT_LARGE_TRANSFERS + ); + } } diff --git a/tinycloud-core/src/sql/service.rs b/tinycloud-core/src/sql/service.rs index 9bd46190..92f39f34 100644 --- a/tinycloud-core/src/sql/service.rs +++ b/tinycloud-core/src/sql/service.rs @@ -3555,6 +3555,9 @@ mod tests { #[async_trait] impl ArtifactSink for StalledSink { + fn buffers_in_memory(&self) -> bool { + false + } async fn reset(&mut self) -> Result<(), DatabaseArtifactError> { Ok(()) } @@ -5110,6 +5113,9 @@ mod tests { #[async_trait] impl ArtifactSink for CountingSink<'_> { + fn buffers_in_memory(&self) -> bool { + false + } async fn reset(&mut self) -> Result<(), DatabaseArtifactError> { self.0.reset().await }