Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,11 @@

## [Unreleased]
### Changes
- PostgreSQL: every pooled connection sets `idle_in_transaction_session_timeout` (`database.idle_in_transaction_timeout_ms`, env `TINYCLOUD_DATABASE__IDLE_IN_TRANSACTION_TIMEOUT_MS`, default 300000 = 5 min, `0` off), so the server ends a transaction left idle that long and the pool replaces the session. A non-zero value of 240 s or less is refused at startup: a protected legacy meeting write legitimately holds its transaction across an upload of up to 240 s. It is not set in `statement_timeout_mode = "local"` (transaction-pooling proxies). `database.idle_in_transaction_timeout_ms`, `statement_timeout_ms` and `bulk_statement_timeout_ms` above 2147483647 (PostgreSQL's largest) are now refused at startup instead of failing every connection. Details: TC-2954.
- PostgreSQL: every pooled connection sets `idle_in_transaction_session_timeout` (`database.idle_in_transaction_timeout_ms`, env `TINYCLOUD_DATABASE__IDLE_IN_TRANSACTION_TIMEOUT_MS`, default 300000 = 5 min, `0` off), so the server ends a transaction left idle that long and the pool replaces the session. A non-zero value of 240 s or less is refused at startup: a protected legacy meeting write legitimately holds its transaction across an upload of up to 240 s. It is not set in `statement_timeout_mode = "local"` (transaction-pooling proxies). `database.idle_in_transaction_timeout_ms` and `statement_timeout_ms` above 2147483647 (PostgreSQL's largest) are now refused at startup instead of failing every connection. Details: TC-2954.
- A protected legacy meeting write now waits for its space's write order before opening its transaction on PostgreSQL too (as on SQLite), so it no longer holds a pooled connection while queueing behind other writers to the same space. In exchange, on PostgreSQL it now holds that space's write order across its upload, up to 240 s: every other write to the same space waits for it meanwhile, including ordinary KV writes, delegations and revocations, epoch-head repair, Policy/v3 registration and session admission. Other spaces are not affected, and a client cannot slow the upload (the body is staged first). An ordinary KV write that waits long enough may be refused as `KvCandidateUnavailable` (it fails closed). Details: TC-2954.
- A SQL checkpoint save over 8 MiB reads its row's metadata before opening the transaction that uploads it, so that transaction holds no table lock while the payload is sent. A pooled connection closed because it was released inside a transaction is now logged as a warning. Details: TC-2954.
- A pooled connection closed because it was released inside a transaction is now logged as a warning. Details: TC-2954.
- SQL and DuckDB: a database whose stored checkpoint or WAL delta is larger than 64 MiB is no longer loaded; requests to it answer `507 sql_database_size_limit` and nothing runs (`storage.sql.max_artifact_read_bytes`, env `TINYCLOUD_STORAGE__SQL__MAX_ARTIFACT_READ_BYTES`; `0` off; the node refuses to start with it below the write limit). A database already stored over the limit becomes unreadable, not only unwritable. DuckDB now answers both artifact limits `507` instead of `500`. At most two database artifact transfers over 8 MiB (whole-checkpoint saves and large loads) run at once per node, for all tenants; further ones wait without holding a database connection, and a SQL save cut while still waiting starts no write back-off. The checkpoint back-off is held in the node's memory: a restart clears it. Details: TC-2961.
- SQL and DuckDB: a whole-checkpoint save over 8 MiB no longer opens a transaction on PostgreSQL. Its metadata read and its compare-and-swap write each run as one autocommit statement under the base `statement_timeout` (30 s by default), so no transaction or row lock is held while the payload uploads. A save the server cancels at that timeout applied nothing and now answers a retryable `503`, as smaller saves already did, instead of `500 outcome_unknown`. `database.bulk_statement_timeout_ms` is removed: a value still set in the file or environment is ignored, and the node logs a warning naming it at startup. Details: TC-2961.
- SQLite stamp replay deadlines now use the same timestamp encoding as cleanup (TC-2920).
- PostgreSQL database artifacts use uncompressed external storage for new writes to avoid repeated decompression during chunked reads (TC-2917). Existing large artifacts require an operator re-toast; verify both storage settings after rollout and follow `scripts/tc2917-README.md` if lock contention deferred the migration.
- Encryption: v2 column format with key ids, per-purpose keys and recorded format metadata. Optional strict reads are off by default. Details: TC-2516, TC-2517.
Expand Down
3 changes: 1 addition & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -301,8 +301,7 @@ These bound how long one PostgreSQL round trip can block a request (TC-1054). Ea
| database.tcp_keepalive_retries | `3` | Unanswered probes before the connection is dropped (15 + 3 × 5 = 30 s). |
| database.tcp_user_timeout_ms | `30000` | `TCP_USER_TIMEOUT`: how long sent data may go unacknowledged before the connection is dropped. `0` keeps the kernel default. |
| database.statement_timeout_mode | `session` | How PostgreSQL's `statement_timeout` is applied: `session` sets it once on each new connection (no per-request cost; covers every statement); `local` sets it with `SET LOCAL` in each explicit transaction (safe behind a transaction-pooling proxy, but one more round trip per transaction, and autocommit statements are not covered); `off` disables it. |
| database.statement_timeout_ms | `30000` | `statement_timeout` for every statement. `0` disables it. At most 2147483647 (PostgreSQL's largest); the node refuses to start with more. |
| database.bulk_statement_timeout_ms | `120000` | `statement_timeout` for statements on a whole SQL or DuckDB artifact larger than 8 MiB (or of unknown size): hydration loads and checkpoint saves, whose server-side work on a ~100 MiB value can outlast the base limit. PostgreSQL does not cancel a statement while it writes a row to the client, so the download itself is not limited by either value. At most 2147483647; the node refuses to start with more. |
| database.statement_timeout_ms | `30000` | `statement_timeout` for every statement, whole-checkpoint saves and hydration slices included. `0` disables it. At most 2147483647 (PostgreSQL's largest); the node refuses to start with more. |
| database.idle_in_transaction_timeout_ms | `300000` | PostgreSQL `idle_in_transaction_session_timeout`, set once on each new PostgreSQL connection (env `TINYCLOUD_DATABASE__IDLE_IN_TRANSACTION_TIMEOUT_MS`): the server ends a session left idle inside a transaction this long, so a stalled or leaked transaction cannot keep holding its locks and pooled connection; the pool replaces the session. It must exceed the 240 s a protected legacy meeting write holds its transaction across its upload; the node refuses to start with a non-zero value of 240 s or less, or more than 2147483647. `0` disables it. Not set when `statement_timeout_mode` is `local`: behind a transaction-pooling proxy a session setting would reach the proxy's other clients. SQLite is unaffected; the operator tool `tinycloud-sql-identity` opens its own pool without it. |
| database.retry_window_ms | `10000` | A read (or `BEGIN`) whose connection is lost is retried once only if that attempt failed within this window. A connection found dead at once is retried; one whose loss took a keepalive expiry to detect is not, so a lost connection costs a read at most one detection plus this window (about 40 s) instead of two detections (about 60 s). This bounds the retry, not the request or the per-database lock hold: acquisition waits, legitimate transfers and queueing behind the lock add to it, and a general deadline is later TC-1054 work. |

Expand Down
2 changes: 1 addition & 1 deletion docs/request-telemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ SQLx query timing follows the execution/logger lifetime, including network I/O a
| KV read/write/list/metadata/sync and public/signed access | Shared pool for graph, metadata, change feed and ticket work; block bytes are not SQL |
| Policy-v3, hooks/tickets/subscriptions, encryption persistence, replay INSERT | Shared pool for all request-owned database operations |
| Quota/admin | In-memory overrides and size mirrors do not invent queries; usage/hosted-space enumeration and any actual pool work are counted |
| SQL/DuckDB identity/fence/alias resolution and durable artifact load/save/CAS/delta | Shared pool, including PostgreSQL bulk scopes; cache hits perform only their actual work |
| SQL/DuckDB identity/fence/alias resolution and durable artifact load/save/CAS/delta | Shared pool; cache hits perform only their actual work |
| Request-awaited read audit | Separate measured batch context, fanned out before durable receipts; details below |
| User SQL rusqlite actor | Context on every work message; actual statement/schema/batch/growth-control SQL (a write request also counts the `SAVEPOINT` and `RELEASE`/`ROLLBACK` of its request transaction, TC-1491; the reservation's own PRAGMAs and checks are not counted), prepare/constraint errors, cold-open PRAGMAs, size/promotion/checkpoint SQL |
| SQL publication | The current withdrawn publication handler executes no SQL; it is not an uninstrumented database path |
Expand Down
43 changes: 15 additions & 28 deletions tinycloud-core/src/database_artifacts.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ use sea_orm::{
use time::{format_description::well_known::Rfc3339, OffsetDateTime};

use crate::{
db_pool::BulkScope,
hash::{hash, Blake3Hasher},
models::database_artifact,
};
Expand Down Expand Up @@ -659,14 +658,12 @@ pub trait DatabaseArtifactRepository: Send + Sync {
}
}

/// The largest artifact payload written under the pool's base statement
/// timeout; a larger checkpoint is written in a
/// [`RecoveringConnection::begin_bulk`] scope (TC-1054). It matches the SQL
/// service's WAL delta bound (8 MiB): the server's work on an 8 MiB value
/// (compression, TOAST, WAL) takes well under a second, far inside the 30 s
/// default.
///
/// [`RecoveringConnection::begin_bulk`]: crate::db_pool::RecoveringConnection::begin_bulk
/// The largest artifact payload a read moves in one part statement, and the
/// WAL delta bound of the SQL service (8 MiB): the server's work on an 8 MiB
/// value (compression, TOAST, WAL) takes well under a second, far inside the
/// 30 s base statement timeout. A whole checkpoint larger than this is saved
/// as one autocommit statement under the same timeout, up to the write limit
/// (TC-2961).
pub const BASE_TIMEOUT_PAYLOAD_BYTES: usize = 8 * 1024 * 1024;

/// The slice size of a streamed artifact read. Two slices fit well inside
Expand Down Expand Up @@ -1062,10 +1059,12 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository {
.format(&Rfc3339)
.expect("current timestamps should format as RFC3339");

// One autocommit statement per step, never a transaction (TC-2961): a
// transaction held open while a whole checkpoint uploads pins a pooled
// connection and its row lock for the whole transfer. The metadata
// read is its own statement; the write below re-checks what it read.
// For Absent, the primary key on the create below asserts absence
// atomically. Details: TC-2214. The read runs on the pool, outside the
// bulk transaction: the CAS on the write below needs no transaction,
// so no lock or snapshot is held while the payload uploads (TC-2954).
// atomically. Details: TC-2214
let existing = if expected == ArtifactExpectation::Absent {
None
} else {
Expand Down Expand Up @@ -1124,15 +1123,6 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository {
#[cfg(test)]
self.wait_at_race_barrier().await;

// A checkpoint larger than a WAL delta may take longer to upload than
// the pool's base statement timeout allows, so its write runs under
// the bulk one (TC-1054). Smaller ones, like every delta, fit the
// base timeout and pay no extra round trip.
let scope = if payload.len() > BASE_TIMEOUT_PAYLOAD_BYTES {
self.conn.begin_bulk().await?
} else {
BulkScope::Pool(self.conn.clone())
};
match existing.as_ref() {
Some(existing) => {
// Compare-and-swap on the (revision, checkpoint) this save read.
Expand Down Expand Up @@ -1198,7 +1188,7 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository {
database_artifact::Column::CheckpointContentHash
.eq(existing.checkpoint_content_hash.clone()),
)
.exec(&scope)
.exec(&self.conn)
.await?;
if update.rows_affected != 1 {
return Err(DatabaseArtifactError::Backend(
Expand All @@ -1225,7 +1215,7 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository {
delta_content_hash: Set(None),
delta_size_bytes: Set(0),
}
.insert(&scope)
.insert(&self.conn)
.await
.map_err(|error| {
// For this table the composite primary key is the only unique
Expand All @@ -1242,8 +1232,6 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository {
}
}

scope.commit().await?;

Ok(DatabaseArtifact {
payload,
content_hash: content_hash.clone(),
Expand Down Expand Up @@ -1843,13 +1831,12 @@ mod tests {
assert_eq!(entered.load(Ordering::SeqCst), 4);
}

/// The node's statement timeouts: a large save opens a real bulk
/// transaction on PostgreSQL (TC-1054).
/// The node's statement timeouts (TC-1054), as a save on PostgreSQL runs
/// under them.
fn node_statement_timeouts() -> crate::db_pool::StatementTimeouts {
crate::db_pool::StatementTimeouts {
mode: crate::db_pool::StatementTimeoutMode::Local,
base: std::time::Duration::from_secs(30),
bulk: std::time::Duration::from_secs(120),
}
}

Expand Down
Loading
Loading