diff --git a/CHANGELOG.md b/CHANGELOG.md index ba3c8be6..5400a95d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,10 +2,11 @@ ## [Unreleased] ### Changes -- PostgreSQL: every pooled connection sets `idle_in_transaction_session_timeout` (`database.idle_in_transaction_timeout_ms`, env `TINYCLOUD_DATABASE__IDLE_IN_TRANSACTION_TIMEOUT_MS`, default 300000 = 5 min, `0` off), so the server ends a transaction left idle that long and the pool replaces the session. A non-zero value of 240 s or less is refused at startup: a protected legacy meeting write legitimately holds its transaction across an upload of up to 240 s. It is not set in `statement_timeout_mode = "local"` (transaction-pooling proxies). `database.idle_in_transaction_timeout_ms`, `statement_timeout_ms` and `bulk_statement_timeout_ms` above 2147483647 (PostgreSQL's largest) are now refused at startup instead of failing every connection. Details: TC-2954. +- PostgreSQL: every pooled connection sets `idle_in_transaction_session_timeout` (`database.idle_in_transaction_timeout_ms`, env `TINYCLOUD_DATABASE__IDLE_IN_TRANSACTION_TIMEOUT_MS`, default 300000 = 5 min, `0` off), so the server ends a transaction left idle that long and the pool replaces the session. A non-zero value of 240 s or less is refused at startup: a protected legacy meeting write legitimately holds its transaction across an upload of up to 240 s. It is not set in `statement_timeout_mode = "local"` (transaction-pooling proxies). `database.idle_in_transaction_timeout_ms` and `statement_timeout_ms` above 2147483647 (PostgreSQL's largest) are now refused at startup instead of failing every connection. Details: TC-2954. - A protected legacy meeting write now waits for its space's write order before opening its transaction on PostgreSQL too (as on SQLite), so it no longer holds a pooled connection while queueing behind other writers to the same space. In exchange, on PostgreSQL it now holds that space's write order across its upload, up to 240 s: every other write to the same space waits for it meanwhile, including ordinary KV writes, delegations and revocations, epoch-head repair, Policy/v3 registration and session admission. Other spaces are not affected, and a client cannot slow the upload (the body is staged first). An ordinary KV write that waits long enough may be refused as `KvCandidateUnavailable` (it fails closed). Details: TC-2954. -- A SQL checkpoint save over 8 MiB reads its row's metadata before opening the transaction that uploads it, so that transaction holds no table lock while the payload is sent. A pooled connection closed because it was released inside a transaction is now logged as a warning. Details: TC-2954. +- A pooled connection closed because it was released inside a transaction is now logged as a warning. Details: TC-2954. - SQL and DuckDB: a database whose stored checkpoint or WAL delta is larger than 64 MiB is no longer loaded; requests to it answer `507 sql_database_size_limit` and nothing runs (`storage.sql.max_artifact_read_bytes`, env `TINYCLOUD_STORAGE__SQL__MAX_ARTIFACT_READ_BYTES`; `0` off; the node refuses to start with it below the write limit). A database already stored over the limit becomes unreadable, not only unwritable. DuckDB now answers both artifact limits `507` instead of `500`. At most two database artifact transfers over 8 MiB (whole-checkpoint saves and large loads) run at once per node, for all tenants; further ones wait without holding a database connection, and a SQL save cut while still waiting starts no write back-off. The checkpoint back-off is held in the node's memory: a restart clears it. Details: TC-2961. +- SQL and DuckDB: a whole-checkpoint save over 8 MiB no longer opens a transaction on PostgreSQL. Its metadata read and its compare-and-swap write each run as one autocommit statement under the base `statement_timeout` (30 s by default), so no transaction or row lock is held while the payload uploads. A save the server cancels at that timeout applied nothing and now answers a retryable `503`, as smaller saves already did, instead of `500 outcome_unknown`. `database.bulk_statement_timeout_ms` is removed: a value still set in the file or environment is ignored, and the node logs a warning naming it at startup. Details: TC-2961. - SQLite stamp replay deadlines now use the same timestamp encoding as cleanup (TC-2920). - PostgreSQL database artifacts use uncompressed external storage for new writes to avoid repeated decompression during chunked reads (TC-2917). Existing large artifacts require an operator re-toast; verify both storage settings after rollout and follow `scripts/tc2917-README.md` if lock contention deferred the migration. - Encryption: v2 column format with key ids, per-purpose keys and recorded format metadata. Optional strict reads are off by default. Details: TC-2516, TC-2517. diff --git a/README.md b/README.md index 5b50701d..19df4043 100644 --- a/README.md +++ b/README.md @@ -301,8 +301,7 @@ These bound how long one PostgreSQL round trip can block a request (TC-1054). Ea | database.tcp_keepalive_retries | `3` | Unanswered probes before the connection is dropped (15 + 3 × 5 = 30 s). | | database.tcp_user_timeout_ms | `30000` | `TCP_USER_TIMEOUT`: how long sent data may go unacknowledged before the connection is dropped. `0` keeps the kernel default. | | database.statement_timeout_mode | `session` | How PostgreSQL's `statement_timeout` is applied: `session` sets it once on each new connection (no per-request cost; covers every statement); `local` sets it with `SET LOCAL` in each explicit transaction (safe behind a transaction-pooling proxy, but one more round trip per transaction, and autocommit statements are not covered); `off` disables it. | -| database.statement_timeout_ms | `30000` | `statement_timeout` for every statement. `0` disables it. At most 2147483647 (PostgreSQL's largest); the node refuses to start with more. | -| database.bulk_statement_timeout_ms | `120000` | `statement_timeout` for statements on a whole SQL or DuckDB artifact larger than 8 MiB (or of unknown size): hydration loads and checkpoint saves, whose server-side work on a ~100 MiB value can outlast the base limit. PostgreSQL does not cancel a statement while it writes a row to the client, so the download itself is not limited by either value. At most 2147483647; the node refuses to start with more. | +| database.statement_timeout_ms | `30000` | `statement_timeout` for every statement, whole-checkpoint saves and hydration slices included. `0` disables it. At most 2147483647 (PostgreSQL's largest); the node refuses to start with more. | | database.idle_in_transaction_timeout_ms | `300000` | PostgreSQL `idle_in_transaction_session_timeout`, set once on each new PostgreSQL connection (env `TINYCLOUD_DATABASE__IDLE_IN_TRANSACTION_TIMEOUT_MS`): the server ends a session left idle inside a transaction this long, so a stalled or leaked transaction cannot keep holding its locks and pooled connection; the pool replaces the session. It must exceed the 240 s a protected legacy meeting write holds its transaction across its upload; the node refuses to start with a non-zero value of 240 s or less, or more than 2147483647. `0` disables it. Not set when `statement_timeout_mode` is `local`: behind a transaction-pooling proxy a session setting would reach the proxy's other clients. SQLite is unaffected; the operator tool `tinycloud-sql-identity` opens its own pool without it. | | database.retry_window_ms | `10000` | A read (or `BEGIN`) whose connection is lost is retried once only if that attempt failed within this window. A connection found dead at once is retried; one whose loss took a keepalive expiry to detect is not, so a lost connection costs a read at most one detection plus this window (about 40 s) instead of two detections (about 60 s). This bounds the retry, not the request or the per-database lock hold: acquisition waits, legitimate transfers and queueing behind the lock add to it, and a general deadline is later TC-1054 work. | diff --git a/docs/request-telemetry.md b/docs/request-telemetry.md index dc5609f2..a40c5332 100644 --- a/docs/request-telemetry.md +++ b/docs/request-telemetry.md @@ -88,7 +88,7 @@ SQLx query timing follows the execution/logger lifetime, including network I/O a | KV read/write/list/metadata/sync and public/signed access | Shared pool for graph, metadata, change feed and ticket work; block bytes are not SQL | | Policy-v3, hooks/tickets/subscriptions, encryption persistence, replay INSERT | Shared pool for all request-owned database operations | | Quota/admin | In-memory overrides and size mirrors do not invent queries; usage/hosted-space enumeration and any actual pool work are counted | -| SQL/DuckDB identity/fence/alias resolution and durable artifact load/save/CAS/delta | Shared pool, including PostgreSQL bulk scopes; cache hits perform only their actual work | +| SQL/DuckDB identity/fence/alias resolution and durable artifact load/save/CAS/delta | Shared pool; cache hits perform only their actual work | | Request-awaited read audit | Separate measured batch context, fanned out before durable receipts; details below | | User SQL rusqlite actor | Context on every work message; actual statement/schema/batch/growth-control SQL (a write request also counts the `SAVEPOINT` and `RELEASE`/`ROLLBACK` of its request transaction, TC-1491; the reservation's own PRAGMAs and checks are not counted), prepare/constraint errors, cold-open PRAGMAs, size/promotion/checkpoint SQL | | SQL publication | The current withdrawn publication handler executes no SQL; it is not an uninstrumented database path | diff --git a/tinycloud-core/src/database_artifacts.rs b/tinycloud-core/src/database_artifacts.rs index 74e91325..b3cd71df 100644 --- a/tinycloud-core/src/database_artifacts.rs +++ b/tinycloud-core/src/database_artifacts.rs @@ -10,7 +10,6 @@ use sea_orm::{ use time::{format_description::well_known::Rfc3339, OffsetDateTime}; use crate::{ - db_pool::BulkScope, hash::{hash, Blake3Hasher}, models::database_artifact, }; @@ -659,14 +658,12 @@ pub trait DatabaseArtifactRepository: Send + Sync { } } -/// The largest artifact payload written under the pool's base statement -/// timeout; a larger checkpoint is written in a -/// [`RecoveringConnection::begin_bulk`] scope (TC-1054). It matches the SQL -/// service's WAL delta bound (8 MiB): the server's work on an 8 MiB value -/// (compression, TOAST, WAL) takes well under a second, far inside the 30 s -/// default. -/// -/// [`RecoveringConnection::begin_bulk`]: crate::db_pool::RecoveringConnection::begin_bulk +/// The largest artifact payload a read moves in one part statement, and the +/// WAL delta bound of the SQL service (8 MiB): the server's work on an 8 MiB +/// value (compression, TOAST, WAL) takes well under a second, far inside the +/// 30 s base statement timeout. A whole checkpoint larger than this is saved +/// as one autocommit statement under the same timeout, up to the write limit +/// (TC-2961). pub const BASE_TIMEOUT_PAYLOAD_BYTES: usize = 8 * 1024 * 1024; /// The slice size of a streamed artifact read. Two slices fit well inside @@ -1062,10 +1059,12 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository { .format(&Rfc3339) .expect("current timestamps should format as RFC3339"); + // One autocommit statement per step, never a transaction (TC-2961): a + // transaction held open while a whole checkpoint uploads pins a pooled + // connection and its row lock for the whole transfer. The metadata + // read is its own statement; the write below re-checks what it read. // For Absent, the primary key on the create below asserts absence - // atomically. Details: TC-2214. The read runs on the pool, outside the - // bulk transaction: the CAS on the write below needs no transaction, - // so no lock or snapshot is held while the payload uploads (TC-2954). + // atomically. Details: TC-2214 let existing = if expected == ArtifactExpectation::Absent { None } else { @@ -1124,15 +1123,6 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository { #[cfg(test)] self.wait_at_race_barrier().await; - // A checkpoint larger than a WAL delta may take longer to upload than - // the pool's base statement timeout allows, so its write runs under - // the bulk one (TC-1054). Smaller ones, like every delta, fit the - // base timeout and pay no extra round trip. - let scope = if payload.len() > BASE_TIMEOUT_PAYLOAD_BYTES { - self.conn.begin_bulk().await? - } else { - BulkScope::Pool(self.conn.clone()) - }; match existing.as_ref() { Some(existing) => { // Compare-and-swap on the (revision, checkpoint) this save read. @@ -1198,7 +1188,7 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository { database_artifact::Column::CheckpointContentHash .eq(existing.checkpoint_content_hash.clone()), ) - .exec(&scope) + .exec(&self.conn) .await?; if update.rows_affected != 1 { return Err(DatabaseArtifactError::Backend( @@ -1225,7 +1215,7 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository { delta_content_hash: Set(None), delta_size_bytes: Set(0), } - .insert(&scope) + .insert(&self.conn) .await .map_err(|error| { // For this table the composite primary key is the only unique @@ -1242,8 +1232,6 @@ impl DatabaseArtifactRepository for SeaOrmDatabaseArtifactRepository { } } - scope.commit().await?; - Ok(DatabaseArtifact { payload, content_hash: content_hash.clone(), @@ -1843,13 +1831,12 @@ mod tests { assert_eq!(entered.load(Ordering::SeqCst), 4); } - /// The node's statement timeouts: a large save opens a real bulk - /// transaction on PostgreSQL (TC-1054). + /// The node's statement timeouts (TC-1054), as a save on PostgreSQL runs + /// under them. fn node_statement_timeouts() -> crate::db_pool::StatementTimeouts { crate::db_pool::StatementTimeouts { mode: crate::db_pool::StatementTimeoutMode::Local, base: std::time::Duration::from_secs(30), - bulk: std::time::Duration::from_secs(120), } } diff --git a/tinycloud-core/src/db_pool.rs b/tinycloud-core/src/db_pool.rs index cb27312c..9219e86c 100644 --- a/tinycloud-core/src/db_pool.rs +++ b/tinycloud-core/src/db_pool.rs @@ -103,8 +103,8 @@ impl RecoveringConnection { } /// Record the `statement_timeout` policy the pool was built with, so - /// [`Self::begin`] (in [`StatementTimeoutMode::Local`]) and - /// [`Self::begin_bulk`] can scope it. PostgreSQL only; other backends ignore it. + /// [`Self::begin`] can scope it in [`StatementTimeoutMode::Local`]. + /// PostgreSQL only; other backends ignore it. pub fn with_statement_timeouts(mut self, statement_timeouts: StatementTimeouts) -> Self { self.statement_timeouts = Some(statement_timeouts); self @@ -317,47 +317,6 @@ impl RecoveringConnection { } impl RecoveringConnection { - /// Open a scope whose statements are allowed the bulk `statement_timeout` - /// ([`StatementTimeouts::bulk`]), for statements that legitimately move a - /// whole database artifact (TC-1054). Finish it with - /// [`BulkScope::commit`]; dropping it rolls back. - /// - /// On PostgreSQL with a statement timeout configured, the scope is a - /// transaction opened with `SET LOCAL statement_timeout`: three more - /// round trips (`BEGIN`, `SET LOCAL`, `COMMIT`), so use it only on - /// infrequent paths. Statements in it are not retried after a lost - /// connection; a read-only scope is not retried either. Everywhere - /// else the scope is the pool itself, statement by statement, under the - /// usual retry policy. - pub async fn begin_bulk(&self) -> Result { - let scope = self.begin_bulk_read().await?; - if matches!(scope, BulkScope::Transaction(_)) { - note_write_may_have_applied(); - } - Ok(scope) - } - - /// [`Self::begin_bulk`] for a scope that only reads: it does not count as - /// a write that may have taken effect ([`track_writes`]). - pub async fn begin_bulk_read(&self) -> Result { - let Some(timeout) = self.bulk_statement_timeout() else { - return Ok(BulkScope::Pool(self.clone())); - }; - let transaction = self.run("begin", true, || self.conn.begin()).await?; - transaction - .execute_unprepared(&set_local_statement_timeout(timeout)) - .await?; - Ok(BulkScope::Transaction(transaction)) - } - - /// The `statement_timeout` a [`Self::begin_bulk`] transaction runs with, or - /// `None` when no statement timeout applies to this pool. - fn bulk_statement_timeout(&self) -> Option { - let timeouts = self.statement_timeouts?; - (self.conn.get_database_backend() == DatabaseBackend::Postgres && timeouts.is_enabled()) - .then_some(timeouts.bulk) - } - /// Apply [`StatementTimeoutMode::Local`] to a transaction just begun. async fn scope_local_timeout( &self, @@ -377,75 +336,6 @@ impl RecoveringConnection { } } -/// A [`RecoveringConnection::begin_bulk`] scope: the pool itself, or the -/// transaction that raised the statement timeout. -pub enum BulkScope { - Pool(RecoveringConnection), - Transaction(DatabaseTransaction), -} - -impl BulkScope { - /// Commit the scope's transaction, if it has one. - pub async fn commit(self) -> Result<(), DbErr> { - match self { - Self::Pool(_) => Ok(()), - Self::Transaction(transaction) => transaction.commit().await, - } - } -} - -#[async_trait::async_trait] -impl ConnectionTrait for BulkScope { - fn get_database_backend(&self) -> DatabaseBackend { - match self { - Self::Pool(conn) => conn.get_database_backend(), - Self::Transaction(transaction) => transaction.get_database_backend(), - } - } - - async fn execute(&self, stmt: Statement) -> Result { - match self { - Self::Pool(conn) => conn.execute(stmt).await, - Self::Transaction(transaction) => transaction.execute(stmt).await, - } - } - - async fn execute_unprepared(&self, sql: &str) -> Result { - match self { - Self::Pool(conn) => conn.execute_unprepared(sql).await, - Self::Transaction(transaction) => transaction.execute_unprepared(sql).await, - } - } - - async fn query_one(&self, stmt: Statement) -> Result, DbErr> { - match self { - Self::Pool(conn) => conn.query_one(stmt).await, - Self::Transaction(transaction) => transaction.query_one(stmt).await, - } - } - - async fn query_all(&self, stmt: Statement) -> Result, DbErr> { - match self { - Self::Pool(conn) => conn.query_all(stmt).await, - Self::Transaction(transaction) => transaction.query_all(stmt).await, - } - } - - fn support_returning(&self) -> bool { - match self { - Self::Pool(conn) => conn.support_returning(), - Self::Transaction(transaction) => transaction.support_returning(), - } - } - - fn is_mock_connection(&self) -> bool { - match self { - Self::Pool(conn) => conn.is_mock_connection(), - Self::Transaction(transaction) => transaction.is_mock_connection(), - } - } -} - /// How the node's PostgreSQL pool applies `statement_timeout` (TC-1054). #[derive( Clone, Copy, Debug, Default, PartialEq, Eq, Hash, serde::Serialize, serde::Deserialize, @@ -457,9 +347,9 @@ pub enum StatementTimeoutMode { /// autocommit ones included. Behind a transaction-pooling proxy /// (PgBouncer, PlanetScale PSBouncer) the `SET` lands on whichever server /// connection served it, so coverage is best-effort there and the setting - /// can reach other clients of the proxy. The scoped limits - /// ([`EXEMPT_STATEMENT_TIMEOUT`], [`RecoveringConnection::begin_bulk`]) - /// use `SET LOCAL` and stay inside their transaction. + /// can reach other clients of the proxy. The migrations' exemption + /// ([`EXEMPT_STATEMENT_TIMEOUT`]) uses `SET LOCAL` and stays inside its + /// transaction. #[default] Session, /// `SET LOCAL statement_timeout` at the start of every explicit @@ -477,9 +367,6 @@ pub struct StatementTimeouts { /// The timeout of every statement. Zero disables it, like /// [`StatementTimeoutMode::Off`]. pub base: Duration, - /// The timeout of a [`RecoveringConnection::begin_bulk`] scope's - /// statements: whole-artifact loads and full checkpoint saves. - pub bulk: Duration, } impl StatementTimeouts { @@ -1741,7 +1628,6 @@ mod tests { let timeouts = |mode, base_ms| StatementTimeouts { mode, base: Duration::from_millis(base_ms), - bulk: Duration::from_secs(120), }; assert_eq!( timeouts(StatementTimeoutMode::Session, 30_000) @@ -1792,12 +1678,6 @@ mod tests { .rollback() .await .unwrap(); - conn.begin_bulk_read() - .await - .unwrap() - .commit() - .await - .unwrap(); }) .await; assert!(!applied, "reads only"); @@ -1880,24 +1760,12 @@ mod tests { assert_eq!(attempts.load(Ordering::SeqCst), 2); } - /// Off PostgreSQL, a bulk scope is the pool itself: no transaction, and - /// statements keep their retry policy. + /// `local` mode issues nothing off PostgreSQL. #[tokio::test] - async fn bulk_scope_is_the_pool_off_postgres() { - let conn = sqlite().await.with_statement_timeouts(StatementTimeouts { - mode: StatementTimeoutMode::Session, - base: Duration::from_millis(1), - bulk: Duration::from_millis(1), - }); - let scope = conn.begin_bulk().await.unwrap(); - assert!(matches!(scope, BulkScope::Pool(_))); - assert_eq!(actor::Entity::find().count(&scope).await.unwrap(), 0); - scope.commit().await.unwrap(); - // `local` mode issues nothing on SQLite either. + async fn local_mode_issues_nothing_off_postgres() { let local = sqlite().await.with_statement_timeouts(StatementTimeouts { mode: StatementTimeoutMode::Local, base: Duration::from_millis(1), - bulk: Duration::from_millis(1), }); local.begin().await.unwrap().commit().await.unwrap(); } diff --git a/tinycloud-node-server/src/config.rs b/tinycloud-node-server/src/config.rs index 0c435115..5a3a13d9 100644 --- a/tinycloud-node-server/src/config.rs +++ b/tinycloud-node-server/src/config.rs @@ -1945,15 +1945,6 @@ pub struct DatabaseConfig { /// Zero disables it. #[serde(default = "default_database_statement_timeout_ms")] pub statement_timeout_ms: u64, - /// `statement_timeout`, in milliseconds, for the statements on a whole - /// SQL or DuckDB artifact that may be larger than a WAL delta (hydration - /// loads and checkpoint saves over 8 MiB, or of unknown size): the - /// server's work on a ~100 MiB value (detoasting and decompressing it on - /// load; compressing, TOASTing and WAL-logging it on save) can outlast - /// the base limit. PostgreSQL does not cancel a statement while it is - /// writing a row to the client, so the download itself is not limited. - #[serde(default = "default_database_bulk_statement_timeout_ms")] - pub bulk_statement_timeout_ms: u64, /// PostgreSQL `idle_in_transaction_session_timeout`, in milliseconds, set /// once on every pooled connection (TC-2954): the server ends a session /// that sits idle inside a transaction this long, so a leaked or stalled @@ -2028,10 +2019,6 @@ fn default_database_statement_timeout_ms() -> u64 { 30_000 } -fn default_database_bulk_statement_timeout_ms() -> u64 { - 120_000 -} - fn default_database_idle_in_transaction_timeout_ms() -> u64 { 300_000 } @@ -2072,7 +2059,6 @@ impl Default for DatabaseConfig { max_lifetime_secs: default_database_max_lifetime_secs(), statement_timeout_mode: StatementTimeoutMode::default(), statement_timeout_ms: default_database_statement_timeout_ms(), - bulk_statement_timeout_ms: default_database_bulk_statement_timeout_ms(), idle_in_transaction_timeout_ms: default_database_idle_in_transaction_timeout_ms(), tcp_keepalive: true, tcp_keepalive_idle_secs: default_database_tcp_keepalive_idle_secs(), @@ -2122,10 +2108,6 @@ impl DatabaseConfig { self.statement_timeout_ms, "database.statement_timeout_ms must be at most 2147483647", ), - ( - self.bulk_statement_timeout_ms, - "database.bulk_statement_timeout_ms must be at most 2147483647", - ), ] { if value > POSTGRES_MAX_TIMEOUT_MS { return Err(error); @@ -2153,7 +2135,6 @@ impl DatabaseConfig { StatementTimeouts { mode: self.statement_timeout_mode, base: Duration::from_millis(self.statement_timeout_ms), - bulk: Duration::from_millis(self.bulk_statement_timeout_ms), } } @@ -2438,19 +2419,10 @@ mod tests { /// stopped. #[test] fn startup_rejects_a_statement_timeout_postgres_cannot_hold() { - for (base, bulk, ok) in [ - (2_147_483_647, 2_147_483_647, true), - (2_147_483_648, 120_000, false), - (30_000, 2_147_483_648, false), - ] { + for (base, ok) in [(2_147_483_647, true), (2_147_483_648, false)] { let mut config = Config::default(); config.database.statement_timeout_ms = base; - config.database.bulk_statement_timeout_ms = bulk; - assert_eq!( - crate::resolve_runtime_config(&config).is_ok(), - ok, - "{base} {bulk}" - ); + assert_eq!(crate::resolve_runtime_config(&config).is_ok(), ok, "{base}"); } } diff --git a/tinycloud-node-server/src/lib.rs b/tinycloud-node-server/src/lib.rs index da1f4364..97f090a2 100644 --- a/tinycloud-node-server/src/lib.rs +++ b/tinycloud-node-server/src/lib.rs @@ -468,9 +468,8 @@ pub fn postgres_connections_opened() -> u64 { /// seconds instead of hanging for up to ~15 min. A connect whose options /// cannot be set fails; /// - in [`tinycloud_core::db_pool::StatementTimeoutMode::Session`], `SET statement_timeout`, so a -/// statement stuck on a live connection fails server-side. Migrations and -/// whole-artifact transfers scope their own limits -/// ([`tinycloud_core::db::migrate`], [`RecoveringConnection::begin_bulk`]). +/// statement stuck on a live connection fails server-side. Migrations +/// lift it for their own transaction ([`tinycloud_core::db::migrate`]). /// /// sea-orm's `Database::connect` cannot install these hooks, so the pool is /// built here with the same connect options it would have applied: the URL, @@ -551,7 +550,6 @@ pub async fn postgres_pool_with_recovery( ::tracing::info!( statement_timeout_mode = ?statement_timeouts.mode, statement_timeout_ms = statement_timeouts.base.as_millis() as u64, - bulk_statement_timeout_ms = statement_timeouts.bulk.as_millis() as u64, idle_in_transaction_timeout_ms, retry_window_ms = database.retry_window_ms, tcp_keepalive = keepalive.is_some(), @@ -703,6 +701,19 @@ fn production_routes(named_keys_v1: bool) -> Vec { routes } +/// Settings the node no longer reads that a deployment may still set, in +/// its file or environment. Configuration extraction ignores unknown keys, +/// so each is named once at startup instead of being dropped silently. +const REMOVED_SETTINGS: &[&str] = &["database.bulk_statement_timeout_ms"]; + +fn removed_settings(config: &Figment) -> Vec<&'static str> { + REMOVED_SETTINGS + .iter() + .copied() + .filter(|setting| config.find_value(setting).is_ok()) + .collect() +} + pub async fn app_with_control( config: &Figment, tinycloud_config: &Config, @@ -749,6 +760,12 @@ pub async fn app_with_control( if tinycloud_config.did_resolution.network_methods { ::tracing::warn!("network DID resolution enabled: did:web/did:tz fetches use the unhardened ssi client (TC-1080)"); } + for setting in removed_settings(config) { + ::tracing::warn!( + setting, + "removed setting is set and ignored; delete it from the configuration (TC-2961)" + ); + } let key_setup: StaticSecret = match resolve_keys(&tinycloud_config.keys, &tinycloud_config.tee).await? { @@ -2851,3 +2868,26 @@ mod no_transform_fairing_tests { } } } + +#[cfg(test)] +mod removed_settings_tests { + use super::*; + use rocket::figment::providers::Serialized; + + /// TC-2961: `database.bulk_statement_timeout_ms` is gone; a deployment + /// that still sets it is told so, and a default one is not. + #[test] + fn a_set_removed_setting_is_named() { + let defaults = Figment::from(Serialized::defaults(Config::default())); + assert!(removed_settings(&defaults).is_empty()); + let old = defaults.merge(("database.bulk_statement_timeout_ms", 120_000)); + assert_eq!( + removed_settings(&old), + ["database.bulk_statement_timeout_ms"] + ); + assert!( + old.extract::().is_ok(), + "still ignored, not refused" + ); + } +} diff --git a/tinycloud-node-server/src/runtime.rs b/tinycloud-node-server/src/runtime.rs index 7f3ecc0c..0573e9cd 100644 --- a/tinycloud-node-server/src/runtime.rs +++ b/tinycloud-node-server/src/runtime.rs @@ -627,7 +627,6 @@ mod tests { defaults.max_lifetime_secs, defaults.statement_timeout_mode, defaults.statement_timeout_ms, - defaults.bulk_statement_timeout_ms, defaults.retry_window_ms, ), ( @@ -635,7 +634,6 @@ mod tests { 1800, tinycloud_core::db_pool::StatementTimeoutMode::Session, 30_000, - 120_000, 10_000 ) ); diff --git a/tinycloud-node-server/tests/postgres_pool_timeouts.rs b/tinycloud-node-server/tests/postgres_pool_timeouts.rs index 26bf8d2d..fd867684 100644 --- a/tinycloud-node-server/tests/postgres_pool_timeouts.rs +++ b/tinycloud-node-server/tests/postgres_pool_timeouts.rs @@ -4,9 +4,9 @@ //! //! - TCP keepalive and `TCP_USER_TIMEOUT` are set on every pool socket //! (read back with `getsockopt`). -//! - `statement_timeout` applies per session, is lifted for migrations, -//! raised for whole-artifact transfers, and surfaces as a retryable -//! `SqlError::Unavailable` (503) that releases the database's lock. +//! - `statement_timeout` applies per session, is lifted for migrations, and +//! surfaces as a retryable `SqlError::Unavailable` (503) that releases the +//! database's lock. //! - Large artifacts still persist and hydrate under the default limits. //! //! PostgreSQL-backed tests connect only when `TINYCLOUD_TEST_POSTGRES_URL` is set @@ -54,9 +54,12 @@ use tinycloud_auth::resource::SpaceId; use tinycloud_auth::ssi::dids::DIDBuf; use tinycloud_auth::ssi::jwk::JWK; use tinycloud_core::database_artifacts::{ - DatabaseArtifactError, DatabaseArtifactRepository, MemorySink, SeaOrmDatabaseArtifactRepository, + ArtifactExpectation, DatabaseArtifact, DatabaseArtifactError, DatabaseArtifactRepository, + MemorySink, SeaOrmDatabaseArtifactRepository, +}; +use tinycloud_core::db_pool::{ + is_transient_error, track_writes, transient_error_class, StatementTimeoutMode, }; -use tinycloud_core::db_pool::{is_transient_error, transient_error_class, StatementTimeoutMode}; use tinycloud_core::sea_orm::{ ConnectOptions, ConnectionTrait, Database, DatabaseConnection, DbBackend, Statement, TransactionTrait, @@ -70,6 +73,11 @@ fn postgres_url(test: &str) -> Option { common::postgres_test_url(test) } +/// Held by the tests that move whole checkpoints over 8 MiB or time a +/// refusal, so they do not run at once: on a shared debug-build server one +/// test's transfer can push another's timed statement past its bound. +static LARGE_TRANSFERS: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(()); + /// The black-hole tests' URL, when `TINYCLOUD_TEST_BLACKHOLE=1` allows them. fn blackhole_url(test: &str) -> Option { if std::env::var("TINYCLOUD_TEST_BLACKHOLE").as_deref() != Ok("1") { @@ -141,6 +149,78 @@ impl Schema { .execute_unprepared(&format!("DROP SCHEMA {} CASCADE", self.name)) .await; } + + async fn exec(&self, sql: &str) { + self.admin.execute_unprepared(sql).await.unwrap(); + } + + async fn i64(&self, sql: &str) -> i64 { + self.admin + .query_one(statement(sql)) + .await + .unwrap() + .unwrap() + .try_get_by_index::(0) + .unwrap() + } + + /// The `main` artifact row's revision and checkpoint hash. + async fn main_row(&self) -> (i64, String) { + let row = self + .admin + .query_one(statement(&format!( + "SELECT revision, checkpoint_content_hash FROM {}.database_artifact \ + WHERE name = 'main'", + self.name + ))) + .await + .unwrap() + .unwrap(); + ( + row.try_get_by_index(0).unwrap(), + row.try_get_by_index(1).unwrap(), + ) + } + + /// Wait until `want` of this schema's pool sessions match `what`. + async fn wait_for_sessions(&self, what: &str, want: i64) { + let sql = format!( + "SELECT count(*) FROM pg_stat_activity WHERE application_name = '{}' AND {what}", + self.name + ); + let started = Instant::now(); + while self.i64(&sql).await < want { + assert!( + started.elapsed() < Duration::from_secs(60), + "no session with {what}" + ); + tokio::time::sleep(Duration::from_millis(20)).await; + } + } + + /// Every update of an artifact row now counts an attempt and takes + /// `seconds` inside PostgreSQL. + async fn slow_updates(&self, seconds: f64) { + let name = &self.name; + self.exec(&format!( + "CREATE SEQUENCE {name}.attempts; \ + CREATE FUNCTION {name}.slow() RETURNS trigger LANGUAGE plpgsql AS \ + $$ BEGIN PERFORM nextval('{name}.attempts'); PERFORM pg_sleep({seconds}); \ + RETURN NEW; END $$; \ + CREATE TRIGGER slow BEFORE UPDATE ON {name}.database_artifact \ + FOR EACH ROW EXECUTE FUNCTION {name}.slow();" + )) + .await; + } + + /// How many artifact updates reached the server since `slow_updates`. + async fn update_attempts(&self) -> i64 { + self.i64(&format!( + "SELECT CASE WHEN is_called THEN last_value ELSE 0 END FROM {}.attempts", + self.name + )) + .await + } } /// The production pool (`connect_database`) for `url`, with `tune` applied @@ -342,9 +422,8 @@ async fn postgres_pool_sockets_carry_keepalive_and_user_timeout() { } /// The session `statement_timeout` bounds a statement on a live connection -/// with a transient error (57014); a bulk scope raises it for its -/// transaction only; migrations run with none; `local` and `off` behave as -/// documented. +/// with a transient error (57014); migrations run with none; `local` and +/// `off` behave as documented. #[tokio::test(flavor = "multi_thread", worker_threads = 4)] async fn postgres_statement_timeout_modes_scopes_and_migrations() { let Some(url) = postgres_url("postgres_statement_timeout_modes_scopes_and_migrations") else { @@ -358,7 +437,6 @@ async fn postgres_statement_timeout_modes_scopes_and_migrations() { let session = pool(schema.url(&url, None), |config| { config.database.statement_timeout_ms = 300; - config.database.bulk_statement_timeout_ms = 5_000; }) .await; assert_eq!(show_statement_timeout(&session).await, "300ms"); @@ -375,21 +453,6 @@ async fn postgres_statement_timeout_modes_scopes_and_migrations() { assert!(is_transient_error(&error), "{error:?}"); assert_eq!(transient_error_class(&error), Some("sqlstate_57014")); - let scope = session.begin_bulk().await.unwrap(); - assert_eq!(show_statement_timeout(&scope).await, "5s"); - scope - .execute(statement("SELECT pg_sleep(0.6)")) - .await - .expect("the bulk timeout applies inside the scope"); - scope.commit().await.unwrap(); - for _ in 0..4 { - assert_eq!( - show_statement_timeout(&session).await, - "300ms", - "SET LOCAL did not leak" - ); - } - // Migrations are exempt: they apply from scratch on a pool whose every // other statement must finish within 1 ms. let strict = pool(schema.url(&url, None), |config| { @@ -429,13 +492,6 @@ async fn postgres_statement_timeout_modes_scopes_and_migrations() { }) .await; assert_eq!(show_statement_timeout(&off).await, "0"); - let scope = off.begin_bulk().await.unwrap(); - assert_eq!( - show_statement_timeout(&scope).await, - "0", - "no bulk transaction when off" - ); - scope.commit().await.unwrap(); for conn in [default, session, strict, local, off] { conn.sea_orm_connection().clone().close().await.unwrap(); @@ -520,7 +576,8 @@ async fn postgres_idle_in_transaction_timeout_ends_an_abandoned_transaction() { /// A statement timeout on the SQL persistence path is a bounded, retryable /// `Unavailable` (503): the database's operation lock is released, the /// next request on it succeeds, and other databases are unaffected. A full -/// checkpoint larger than a delta runs under the bulk timeout. +/// checkpoint over 8 MiB is no exception (TC-2961): it runs under the same +/// base timeout and nothing of it is applied. #[tokio::test(flavor = "multi_thread", worker_threads = 4)] async fn postgres_sql_statement_timeout_is_retryable_and_releases_the_lock() { let Some(url) = @@ -528,10 +585,10 @@ async fn postgres_sql_statement_timeout_is_retryable_and_releases_the_lock() { else { return; }; + let _serial = LARGE_TRANSFERS.lock().await; let schema = Schema::new(&url, "sqllock").await; let conn = pool(schema.url(&url, None), |config| { config.database.statement_timeout_ms = 500; - config.database.bulk_statement_timeout_ms = 10_000; }) .await; tinycloud_core::db::migrate(&conn).await.unwrap(); @@ -577,11 +634,17 @@ async fn postgres_sql_statement_timeout_is_retryable_and_releases_the_lock() { write(&service, &space, "b", 1024).await.unwrap(); assert_eq!(count(&service, &space, "b").await.unwrap(), 2); - // A WAL past the 8 MiB delta bound persists as a full checkpoint, whose - // write runs under the bulk timeout and survives the slow trigger. - write(&service, &space, "a", 9 * 1024 * 1024) + // TC-2961: a full checkpoint (a WAL over 8 MiB) is one autocommit UPDATE + // under the same base timeout: refused, retryable, the lock released, + // and nothing of it applied. + let error = write(&service, &space, "a", 9 * 1024 * 1024) + .await + .expect_err("the full checkpoint update exceeds the statement timeout"); + assert!(matches!(error, SqlError::Unavailable), "{error:?}"); + let next = tokio::time::timeout(Duration::from_secs(5), count(&service, &space, "a")) .await - .expect("a full checkpoint runs under the bulk statement timeout"); + .expect("the failed checkpoint save released the database lock"); + assert_eq!(next.unwrap(), 1); schema .admin @@ -593,7 +656,7 @@ async fn postgres_sql_statement_timeout_is_retryable_and_releases_the_lock() { write(&service, &space, "a", 1024).await.unwrap(); let cold_cache = tempfile::TempDir::new().unwrap(); let cold = sql_service(&conn, &cold_cache); - assert_eq!(count(&cold, &space, "a").await.unwrap(), 3); + assert_eq!(count(&cold, &space, "a").await.unwrap(), 2); conn.sea_orm_connection().clone().close().await.unwrap(); schema.drop().await; } @@ -646,6 +709,189 @@ async fn postgres_large_artifacts_persist_under_default_timeouts() { schema.drop().await; } +/// TC-2961: a whole-checkpoint save over 8 MiB, its first INSERT and a full +/// checkpoint UPDATE, sends no `BEGIN`: each is one autocommit statement on +/// the production pool, so no transaction stays open while the payload +/// uploads. The proxy sees every byte the pool sends. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn postgres_checkpoint_saves_send_no_begin() { + let Some(url) = postgres_url("postgres_checkpoint_saves_send_no_begin") else { + return; + }; + let _serial = LARGE_TRANSFERS.lock().await; + let schema = Schema::new(&url, "nobegin").await; + let direct = pool(schema.url(&url, None), |_| {}).await; + tinycloud_core::db::migrate(&direct).await.unwrap(); + let proxy = Proxy::start(upstream_addr(&url), None).await; + let conn = pool(schema.url(&url, Some(proxy.addr)), |_| {}).await; + let repository = SeaOrmDatabaseArtifactRepository::new(conn.clone()); + let large = 9 * 1024 * 1024; + + let created = repository + .save( + "sql", + "space", + "main", + vec![1; large], + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + repository + .save( + "sql", + "space", + "main", + vec![2; large], + ArtifactExpectation::Derived { + revision: created.revision, + checkpoint_content_hash: created.checkpoint_content_hash, + }, + ) + .await + .unwrap(); + assert_eq!(proxy.simple_queries(b"BEGIN"), 0, "no transaction"); + // The pool's own transactions do pass the proxy. + conn.begin().await.unwrap().commit().await.unwrap(); + assert_eq!(proxy.simple_queries(b"BEGIN"), 1); + + for conn in [direct, conn] { + conn.sea_orm_connection().clone().close().await.unwrap(); + } + schema.drop().await; +} + +fn derived(artifact: &DatabaseArtifact) -> ArtifactExpectation { + ArtifactExpectation::Derived { + revision: artifact.revision, + checkpoint_content_hash: artifact.checkpoint_content_hash.clone(), + } +} + +/// TC-2961 (Bobbie's #592 review): a whole-checkpoint save read revision 1 +/// on checkpoint A; before its UPDATE lands, a delta save moves the row to +/// revision 2 on the same checkpoint A. The checkpoint hash still matches, +/// so only the revision in the UPDATE's predicate refuses the stale save; +/// without it the delta would be lost. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn postgres_a_checkpoint_save_after_a_concurrent_delta_is_refused() { + let Some(url) = postgres_url("postgres_a_checkpoint_save_after_a_concurrent_delta_is_refused") + else { + return; + }; + let _serial = LARGE_TRANSFERS.lock().await; + let schema = Schema::new(&url, "casdelta").await; + let conn = pool(schema.url(&url, None), |_| {}).await; + tinycloud_core::db::migrate(&conn).await.unwrap(); + let repository = Arc::new(SeaOrmDatabaseArtifactRepository::new(conn.clone())); + let large = 9 * 1024 * 1024; + let seed = repository + .save( + "sql", + "space", + "main", + vec![1; large], + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + + // Hold the row so the save's UPDATE waits behind it. + let holder = schema.admin.begin().await.unwrap(); + holder + .execute_unprepared(&format!( + "SELECT 1 FROM {}.database_artifact WHERE name = 'main' FOR UPDATE", + schema.name + )) + .await + .unwrap(); + let save = { + let (repository, expected) = (repository.clone(), derived(&seed)); + tokio::spawn(async move { + repository + .save("sql", "space", "main", vec![9; large], expected) + .await + }) + }; + schema + .wait_for_sessions("wait_event_type = 'Lock' AND query LIKE 'UPDATE%'", 1) + .await; + // What a delta save does to the row: the next revision, same checkpoint. + holder + .execute_unprepared(&format!( + "UPDATE {}.database_artifact SET revision = revision + 1, \ + delta_payload = '\\x01', delta_size_bytes = 1, delta_content_hash = 'delta' \ + WHERE name = 'main'", + schema.name + )) + .await + .unwrap(); + holder.commit().await.unwrap(); + + let result = save.await.unwrap(); + assert!( + matches!(&result, Err(DatabaseArtifactError::Backend(message)) + if message == "concurrent database artifact update"), + "{result:?}" + ); + assert_eq!( + schema.main_row().await, + (seed.revision + 1, seed.checkpoint_content_hash), + "the delta's row stands" + ); + conn.sea_orm_connection().clone().close().await.unwrap(); + schema.drop().await; +} + +/// TC-2961 (Bobbie's #592 review): a whole-checkpoint UPDATE that the +/// server's statement timeout cancels is refused once. It is not retried, +/// not tracked as possibly applied, and the row is unchanged. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn postgres_a_timed_out_checkpoint_update_is_refused_once_and_not_applied() { + let Some(url) = + postgres_url("postgres_a_timed_out_checkpoint_update_is_refused_once_and_not_applied") + else { + return; + }; + let _serial = LARGE_TRANSFERS.lock().await; + let schema = Schema::new(&url, "updtimeout").await; + let conn = pool(schema.url(&url, None), |config| { + config.database.statement_timeout_ms = 500; + }) + .await; + tinycloud_core::db::migrate(&conn).await.unwrap(); + let repository = SeaOrmDatabaseArtifactRepository::new(conn.clone()); + let large = 9 * 1024 * 1024; + let seed = repository + .save( + "sql", + "space", + "main", + vec![1; large], + ArtifactExpectation::Absent, + ) + .await + .unwrap(); + schema.slow_updates(1.5).await; + + let (result, applied) = + track_writes(repository.save("sql", "space", "main", vec![2; large], derived(&seed))).await; + let class = match &result { + Err(DatabaseArtifactError::Db(error)) => transient_error_class(error), + _ => None, + }; + assert_eq!(class, Some("sqlstate_57014"), "{result:?}"); + assert!(!applied, "a refused autocommit statement did not apply"); + assert_eq!(schema.update_attempts().await, 1, "never retried"); + assert_eq!( + schema.main_row().await, + (seed.revision, seed.checkpoint_content_hash), + "nothing written" + ); + conn.sea_orm_connection().clone().close().await.unwrap(); + schema.drop().await; +} + /// Hydration is a series of 4 MiB slice statements, each under the BASE /// statement timeout (TC-822): a large artifact hydrates through a pool whose /// base timeout is far below the whole-artifact transfer, and a slice whose @@ -671,7 +917,6 @@ async fn postgres_hydration_slices_run_under_the_base_timeout() { } let strict = pool(schema.url(&url, None), |config| { config.database.statement_timeout_ms = 500; - config.database.bulk_statement_timeout_ms = 10_000; }) .await; let repository = Arc::new(SeaOrmDatabaseArtifactRepository::new(strict.clone())); @@ -779,6 +1024,8 @@ struct Link { client: SocketAddr, /// Bytes forwarded from PostgreSQL to the pool. down: Arc, + /// Bytes forwarded from the pool to PostgreSQL. + up: Arc>>, } /// A TCP proxy between a pool and PostgreSQL that records its connections @@ -800,11 +1047,13 @@ impl Proxy { continue; }; let down = Arc::new(AtomicU64::new(0)); + let up = Arc::new(Mutex::new(Vec::new())); registry.lock().unwrap().push(Link { client: peer, down: Arc::clone(&down), + up: Arc::clone(&up), }); - tokio::spawn(pipe(client, server, down, down_bytes_per_sec)); + tokio::spawn(pipe(client, server, down, up, down_bytes_per_sec)); } }); Self { addr, links } @@ -836,14 +1085,37 @@ impl Proxy { .map(|link| link.client) .collect() } + + /// How many simple-protocol queries (`Q` messages) the pool sent whose + /// text starts with `prefix`, over every connection. + fn simple_queries(&self, prefix: &[u8]) -> usize { + self.links + .lock() + .unwrap() + .iter() + .map(|link| { + let up = link.up.lock().unwrap(); + up.windows(5 + prefix.len()) + .filter(|window| window[0] == b'Q' && window[5..] == *prefix) + .count() + }) + .sum() + } } -async fn pipe(client: TcpStream, server: TcpStream, down: Arc, rate: Option) { +async fn pipe( + client: TcpStream, + server: TcpStream, + down: Arc, + up: Arc>>, + rate: Option, +) { let (mut client_read, mut client_write) = client.into_split(); let (mut server_read, mut server_write) = server.into_split(); let up = tokio::spawn(async move { let mut buf = vec![0u8; 64 * 1024]; while let Ok(n @ 1..) = client_read.read(&mut buf).await { + up.lock().unwrap().extend_from_slice(&buf[..n]); if server_write.write_all(&buf[..n]).await.is_err() { break; } diff --git a/tinycloud.toml b/tinycloud.toml index 310447ce..31de6ff7 100644 --- a/tinycloud.toml +++ b/tinycloud.toml @@ -60,7 +60,6 @@ cors = true # tcp_user_timeout_ms = 30000 # statement_timeout_mode = "session" # session | local | off # statement_timeout_ms = 30000 - # bulk_statement_timeout_ms = 120000 # retry_window_ms = 10000 ## How long an authorized read waits for its audit record before 503 ## read_audit_busy (TC-1317). Its data is never returned without the record.