Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
- SQL and DuckDB: a whole-checkpoint save over 8 MiB no longer opens a transaction on PostgreSQL. Its metadata read and its compare-and-swap write each run as one autocommit statement under the base `statement_timeout` (30 s by default), so no transaction or row lock is held while the payload uploads. A save the server cancels at that timeout applied nothing and now answers a retryable `503`, as smaller saves already did, instead of `500 outcome_unknown`. `database.bulk_statement_timeout_ms` is removed: a value still set in the file or environment is ignored, and the node logs a warning naming it at startup. Details: TC-2961.
- `/invoke`: a KV invocation that writes (`kv/put` or `kv/del`) and also reads (`kv/get`, `kv/list`, `kv/metadata` or `tinycloud.capabilities/read`) is refused with the same `501` as before, but now before any work: nothing is written and the signed request is not spent. Before, the write committed and only then was the invocation answered `501`, since `/invoke` returns one outcome. Internal callers that still mix the two get their reads after the write commits, holding no database transaction, row lock or write order. A `kv/get` answers the value the invocation wrote. If such a read fails, the error says the write was saved. Details: TC-2986.
- `POST /delegate` and `POST /revoke` check the signature (and, for a renewal lease, the signature of the stored original it renews) before taking any lock. A forged or slow-to-verify request no longer holds the delegation's chain guards, its spaces' write order or a database transaction while it is checked; it is refused at once. A renewal lease whose original is not stored is also refused before any lock. Inside the transaction only the signed time window is checked again. A resubmitted delegation that is already stored is now also signature-checked before it is answered as existing. Details: TC-2986.
- Add opt-in PostgreSQL checkpoint parts for SQL persistence, with explicit artifact/account capacities and streamed hydration. A database stored in parts loads in bounded statements and is not held to the single-row read limit (`storage.sql.max_artifact_read_bytes`); a load that keeps the whole database in memory (a DuckDB export) still waits for one of the large-transfer slots. Details: TC-2917.
- SQLite stamp replay deadlines now use the same timestamp encoding as cleanup (TC-2920).
- PostgreSQL database artifacts use uncompressed external storage for new writes to avoid repeated decompression during chunked reads (TC-2917). Existing large artifacts require an operator re-toast; verify both storage settings after rollout and follow `scripts/tc2917-README.md` if lock contention deferred the migration.
- Encryption: v2 column format with key ids, per-purpose keys and recorded format metadata. Optional strict reads are off by default. Details: TC-2516, TC-2517.
Expand Down
103 changes: 103 additions & 0 deletions docs/operations/chunked-sql-artifacts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# Opt-in SQL checkpoint parts

The first TC-2917 slice stores SQL checkpoints as immutable PostgreSQL parts.
Each payload statement carries at most 1 MiB. A metadata-only compare-and-swap
publishes a sealed part; successful writes retain the existing durability and
lineage semantics. Hydration verifies the part CID before publishing its cache.

A database stored in parts is not held to the single-row read limit
(`storage.sql.max_artifact_read_bytes`, TC-2961): each statement reads at most
1 MiB, hydration stages the parts in files beside the cache, and the artifact
and account capacities bound its size. A read that keeps the whole database in
memory (a DuckDB export) still waits for one of the node's large-transfer slots
when it exceeds 8 MiB.

The switch is off by default. Set explicit capacities before enabling:

```toml
[storage.sql]
chunked_artifacts = false
chunked_artifact_max_bytes = 0
chunked_account_max_bytes = 0
```

Both capacities must be positive when enabled. No capacity defaults are chosen
by this slice. The account capacity is charged to the space owner, across its
spaces, including legacy live artifacts, staged uploads and retained parts.
Reservations are atomic per owner. Quota refusal uses the SQL storage refusal.

This slice publishes a full checkpoint for each SQL mutation. WAL parts,
conversion of existing artifacts follow separately.
Aborted and superseded parts enter a fenced deleting state; reclamation deletes
at most one payload chunk per statement and releases its charge only after all
bytes are gone. A periodic collector recovers uploads whose 120-second lease
expired. Lease expiry determines eligibility; part identity and state fence
uploads against concurrent reclamation. Active sealed uploads retain their lease
until publication. Deleted artifacts are reclaimed by the same mechanism.
It does not enable chunked DuckDB writes. Legacy data remains readable. Drain
older writers before enabling this format; turning the switch off does not
convert published parts back to legacy blobs. Current readers understand both
layouts; legacy persistence refuses a database already using parts.

Do not enable for prolonged operation until the approved ceiling load acceptance and
remaining migration/WAL slices are complete. On each test release, report
reserved, staged and retained bytes from the following metadata-only inventory:

```sql
SELECT a.owner, a.legacy_bytes, a.reserved_bytes, a.credit_bytes,
a.legacy_bytes+a.reserved_bytes-a.credit_bytes AS quota_bytes,
(SELECT COALESCE(sum(octet_length(c.payload)),0)
FROM artifact_chunk c JOIN artifact_part stored
USING (service,space,name,part_id) WHERE stored.owner=a.owner) AS stored_chunk_bytes,
COALESCE(sum(p.size_bytes) FILTER (WHERE p.state='uploading'),0) AS uploading_bytes,
COALESCE(sum(p.size_bytes) FILTER (
WHERE p.state IN ('sealed','deleting') AND NOT EXISTS (
SELECT 1 FROM database_artifact d WHERE d.service=p.service
AND d.space=p.space AND d.name=p.name
AND (d.checkpoint_part_id=p.part_id OR d.delta_part_id=p.part_id)
)),0) AS retained_bytes
FROM artifact_account a LEFT JOIN artifact_part p ON p.owner=a.owner
GROUP BY a.owner,a.legacy_bytes,a.reserved_bytes,a.credit_bytes;
```

No content is selected by this inventory. Existing-row conversion and the
approved isolated-copy procedure are outside this first slice. Explicit SQL
exports still return their complete response body; the transfer-buffer bound
applies to persistence and hydration, not that public response format.

Quota enforcement is activated atomically with an exact-owner metadata inventory.
After activation, the same database owner-row lock accounts for legacy SQL and
DuckDB writes, including writes made by an older binary. This activation remains
in force if a reader starts with the feature switch off. Database triggers also
refuse a legacy update of a published checkpoint-parts layout atomically; the
layout marker and generation are metadata, not payload contents. A flag-off
legacy read or write adds no client-side layout probe.

Before the first activation, the quota trigger is disabled: legacy requests add
no client layout probes or quota queries. The field-only layout guard remains
installed for mixed-version writes. First activation enables owner accounting
under the same short writer fence used for its initial inventory.
After activation, switching off new chunked databases does not disable support
for already-published parts: current nodes continue their file-based writes
using the persisted capacities, and keep the collector running.

Operators use the value-free `artifact_part_collection_failed` event to investigate deferred reclamation.

A replacement may overlap its current database's bytes: at most one replacement
per database receives a credit, bounded by the smaller of current and staged
sizes. Full staged bytes remain visible in `reserved_bytes`. A pending shrink
never releases capacity for another database. Credit is released only when the
old part is reclaimed, or the staged upload is reclaimed after abort; legacy
replacement credit is released atomically with publication. Admission evaluates
account usage minus the current database size plus its proposed replacement.
Thus a full account can shrink, cannot grow above its quota, and cannot reuse
the same database's credit for a second simultaneous replacement.

Published parts have a distinct state and infinite lease, and are excluded from
both global and owner-local reclamation indexes. Inline cleanup is best effort,
scoped to the uploading owner, at most 16 steps and 100 ms per pass; its errors
do not fail the save. Background passes are limited to 128 steps and one second
per 10-second tick. GC lock acquisition waits at most 50 ms per statement.
An owner already above its quota can replace a database with an equal or
smaller image; that exception admits no growth and does not free pending shrink
capacity for other databases.
101 changes: 98 additions & 3 deletions tinycloud-core/src/database_artifacts.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
use std::future::Future;
pub mod chunks;

use async_trait::async_trait;
use sea_orm::{
Expand Down Expand Up @@ -105,7 +106,9 @@ pub struct DeltaSave {
#[derive(Debug, thiserror::Error)]
pub enum DatabaseArtifactError {
#[error("database artifact storage error: {0}")]
Db(#[from] DbErr),
Db(DbErr),
#[error("artifact storage limit reached")]
StorageLimit,
#[error("database artifact payload too large: {0} bytes")]
PayloadTooLarge(u64),
/// A payload larger than the repository's single-statement write limit
Expand Down Expand Up @@ -140,6 +143,18 @@ pub enum DatabaseArtifactError {
Sink(String),
}

impl From<DbErr> for DatabaseArtifactError {
fn from(error: DbErr) -> Self {
if error.to_string().contains("TC2917_QUOTA") {
Self::StorageLimit
} else if error.to_string().contains("TC2917_LAYOUT") {
Self::StaleLineage
} else {
Self::Db(error)
}
}
}

/// The stored metadata of one artifact row: everything but the payload bytes.
///
/// Reading it never touches the `payload` / `delta_payload` columns, so its
Expand Down Expand Up @@ -205,6 +220,11 @@ pub trait ArtifactSink: Send {
fn size_hint(&mut self, _part: ArtifactPart, _bytes: u64) {}
async fn write(&mut self, bytes: &[u8]) -> Result<(), DatabaseArtifactError>;
async fn finish_part(&mut self) -> Result<(), DatabaseArtifactError>;
/// Whether the sink holds the whole artifact in memory. Such a read of a
/// large artifact waits for a large-transfer slot in either layout.
fn buffers_in_memory(&self) -> bool {
false
}
}

/// The most [`MemorySink`] allocates ahead of the bytes it is given.
Expand Down Expand Up @@ -234,6 +254,10 @@ impl ArtifactSink for MemorySink {
Ok(())
}

fn buffers_in_memory(&self) -> bool {
true
}

fn size_hint(&mut self, part: ArtifactPart, bytes: u64) {
let target = match part {
ArtifactPart::Checkpoint => Some(&mut self.checkpoint),
Expand Down Expand Up @@ -637,6 +661,21 @@ pub trait DatabaseArtifactRepository: Send + Sync {
sink: &mut dyn ArtifactSink,
) -> Result<(), DatabaseArtifactError>;

fn chunked_checkpoints(&self, _service: &str, _space: &str, _name: &str) -> bool {
false
}

async fn save_snapshot(
&self,
_service: &str,
_space: &str,
_name: &str,
_snapshot: tempfile::NamedTempFile,
_expected: ArtifactExpectation,
) -> Result<ArtifactMeta, DatabaseArtifactError> {
Err(DatabaseArtifactError::IncrementalPersistenceUnsupported)
}

async fn save(
&self,
service: &str,
Expand Down Expand Up @@ -724,11 +763,17 @@ pub struct SeaOrmDatabaseArtifactRepository {
/// Permits for transfers over [`BASE_TIMEOUT_PAYLOAD_BYTES`]; shared by
/// every clone of this repository.
large_transfers: std::sync::Arc<tokio::sync::Semaphore>,
chunk_limits: Option<chunks::ChunkLimits>,
observed_chunked: std::sync::Arc<dashmap::DashSet<(String, String, String)>>,
/// Test-only rendezvous seam (see [`wait_at_race_barrier`]) that lets two
/// writers read the same base revision before either commits, so the
/// full-checkpoint CAS conflict path can be exercised deterministically.
#[cfg(test)]
race_barrier: Option<std::sync::Arc<tokio::sync::Barrier>>,
#[cfg(test)]
chunk_publish_barrier: Option<std::sync::Arc<tokio::sync::Barrier>>,
#[cfg(test)]
chunk_reclaim_paused: bool,
}

impl SeaOrmDatabaseArtifactRepository {
Expand All @@ -741,11 +786,22 @@ impl SeaOrmDatabaseArtifactRepository {
large_transfers: std::sync::Arc::new(tokio::sync::Semaphore::new(
MAX_CONCURRENT_LARGE_TRANSFERS,
)),
chunk_limits: None,
observed_chunked: Default::default(),
#[cfg(test)]
race_barrier: None,
#[cfg(test)]
chunk_publish_barrier: None,
#[cfg(test)]
chunk_reclaim_paused: false,
}
}

pub fn with_chunk_limits(mut self, limits: Option<chunks::ChunkLimits>) -> Self {
self.chunk_limits = limits;
self
}

/// Share the node's SQLite gate: artifact writes use the capability DB and
/// must not advance its WAL beneath a delegation transaction's snapshot.
pub fn with_sqlite_writer_lock(
Expand Down Expand Up @@ -941,7 +997,7 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository {
)>()
.one(&self.conn)
.await?;
Ok(row.map(
let meta = row.map(
|(
revision,
size_bytes,
Expand All @@ -965,7 +1021,17 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository {
storage_mode,
backend,
},
))
);
let key = (service.to_string(), space.to_string(), name.to_string());
if meta
.as_ref()
.is_some_and(|m| m.storage_mode == "checkpoint-parts-v1")
{
self.observed_chunked.insert(key);
} else {
self.observed_chunked.remove(&key);
}
Ok(meta)
}

async fn read_chunks(
Expand All @@ -976,6 +1042,14 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository {
meta: &ArtifactMeta,
sink: &mut dyn ArtifactSink,
) -> Result<(), DatabaseArtifactError> {
// Parts stream in bounded statements: the whole-artifact read cap and
// the large-transfer permit below are for the single-row layout.
if self
.read_chunk_layout(service, space, name, meta, sink)
.await?
{
return Ok(());
}
// TC-2961: refuse an oversized database before any slice is read.
if self.max_read_bytes != 0 {
for (part, bytes) in [
Expand Down Expand Up @@ -1033,6 +1107,25 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository {
Ok(())
}

fn chunked_checkpoints(&self, service: &str, space: &str, name: &str) -> bool {
self.chunk_limits.is_some()
|| self
.observed_chunked
.contains(&(service.into(), space.into(), name.into()))
}

async fn save_snapshot(
&self,
service: &str,
space: &str,
name: &str,
snapshot: tempfile::NamedTempFile,
expected: ArtifactExpectation,
) -> Result<ArtifactMeta, DatabaseArtifactError> {
self.write_chunk_snapshot(service, space, name, snapshot, expected)
.await
}

async fn save(
&self,
service: &str,
Expand All @@ -1048,6 +1141,7 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository {
let _ = TRANSFER_STARTED.try_with(|started| {
started.store(true, std::sync::atomic::Ordering::Release);
});
self.ensure_legacy_write(service, space, name).await?;
let _writer = match &self.sqlite_writer_lock {
Some(lock) => Some(lock.lock().await),
None => None,
Expand Down Expand Up @@ -1256,6 +1350,7 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository {
expected: ArtifactExpectation,
) -> Result<DeltaSave, DatabaseArtifactError> {
self.check_write(payload.len())?;
self.ensure_legacy_write(service, space, name).await?;
let _writer = match &self.sqlite_writer_lock {
Some(lock) => Some(lock.lock().await),
None => None,
Expand Down
Loading
Loading