Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

## [Unreleased]
### Changes
- PostgreSQL: every pooled connection sets `idle_in_transaction_session_timeout` (`database.idle_in_transaction_timeout_ms`, env `TINYCLOUD_DATABASE__IDLE_IN_TRANSACTION_TIMEOUT_MS`, default 300000 = 5 min, `0` off), so the server ends a transaction left idle that long and the pool replaces the session. A non-zero value of 240 s or less is refused at startup: a protected legacy meeting write legitimately holds its transaction across an upload of up to 240 s. It is not set in `statement_timeout_mode = "local"` (transaction-pooling proxies). `database.idle_in_transaction_timeout_ms`, `statement_timeout_ms` and `bulk_statement_timeout_ms` above 2147483647 (PostgreSQL's largest) are now refused at startup instead of failing every connection. Details: TC-2954.
- SQLite stamp replay deadlines now use the same timestamp encoding as cleanup (TC-2920).
- PostgreSQL database artifacts use uncompressed external storage for new writes to avoid repeated decompression during chunked reads (TC-2917). Existing large artifacts require an operator re-toast; verify both storage settings after rollout and follow `scripts/tc2917-README.md` if lock contention deferred the migration.
- Encryption: v2 column format with key ids, per-purpose keys and recorded format metadata. Optional strict reads are off by default. Details: TC-2516, TC-2517.
Expand Down
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -301,8 +301,9 @@ These bound how long one PostgreSQL round trip can block a request (TC-1054). Ea
| database.tcp_keepalive_retries | `3` | Unanswered probes before the connection is dropped (15 + 3 × 5 = 30 s). |
| database.tcp_user_timeout_ms | `30000` | `TCP_USER_TIMEOUT`: how long sent data may go unacknowledged before the connection is dropped. `0` keeps the kernel default. |
| database.statement_timeout_mode | `session` | How PostgreSQL's `statement_timeout` is applied: `session` sets it once on each new connection (no per-request cost; covers every statement); `local` sets it with `SET LOCAL` in each explicit transaction (safe behind a transaction-pooling proxy, but one more round trip per transaction, and autocommit statements are not covered); `off` disables it. |
| database.statement_timeout_ms | `30000` | `statement_timeout` for every statement. `0` disables it. |
| database.bulk_statement_timeout_ms | `120000` | `statement_timeout` for statements on a whole SQL or DuckDB artifact larger than 8 MiB (or of unknown size): hydration loads and checkpoint saves, whose server-side work on a ~100 MiB value can outlast the base limit. PostgreSQL does not cancel a statement while it writes a row to the client, so the download itself is not limited by either value. |
| database.statement_timeout_ms | `30000` | `statement_timeout` for every statement. `0` disables it. At most 2147483647 (PostgreSQL's largest); the node refuses to start with more. |
| database.bulk_statement_timeout_ms | `120000` | `statement_timeout` for statements on a whole SQL or DuckDB artifact larger than 8 MiB (or of unknown size): hydration loads and checkpoint saves, whose server-side work on a ~100 MiB value can outlast the base limit. PostgreSQL does not cancel a statement while it writes a row to the client, so the download itself is not limited by either value. At most 2147483647; the node refuses to start with more. |
| database.idle_in_transaction_timeout_ms | `300000` | PostgreSQL `idle_in_transaction_session_timeout`, set once on each new PostgreSQL connection (env `TINYCLOUD_DATABASE__IDLE_IN_TRANSACTION_TIMEOUT_MS`): the server ends a session left idle inside a transaction this long, so a stalled or leaked transaction cannot keep holding its locks and pooled connection; the pool replaces the session. It must exceed the 240 s a protected legacy meeting write holds its transaction across its upload; the node refuses to start with a non-zero value of 240 s or less, or more than 2147483647. `0` disables it. Not set when `statement_timeout_mode` is `local`: behind a transaction-pooling proxy a session setting would reach the proxy's other clients. SQLite is unaffected; the operator tool `tinycloud-sql-identity` opens its own pool without it. |
| database.retry_window_ms | `10000` | A read (or `BEGIN`) whose connection is lost is retried once only if that attempt failed within this window. A connection found dead at once is retried; one whose loss took a keepalive expiry to detect is not, so a lost connection costs a read at most one detection plus this window (about 40 s) instead of two detections (about 60 s). This bounds the retry, not the request or the per-database lock hold: acquisition waits, legitimate transfers and queueing behind the lock add to it, and a general deadline is later TC-1054 work. |

Schema migrations always run with no statement timeout. On PostgreSQL they wait at most 30 s for any lock (`lock_timeout`), so a session holding a conflicting table lock makes the node fail to start with an error naming the lock wait instead of hanging; the migrations roll back together and the next start applies them (TC-1176). Customer SQL (`tinycloud.sql/*`) runs in the node's local SQLite engine, never on PostgreSQL, so `statement_timeout` does not limit it.
Expand Down
4 changes: 3 additions & 1 deletion tinycloud-core/src/db.rs
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,9 @@ pub const KV_MAX_MUTATIONS_PER_INVOCATION: usize = 4096;
/// One invocation's uploads finish within 240 seconds of preflight, before
/// the five-minute orphan age. A live same-hash retry defers an older crash
/// stage until that retry finishes; repeated overlaps can extend its charge.
const KV_UPLOAD_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(240);
/// A protected legacy write holds its transaction across this upload, so the
/// pool's idle-in-transaction timeout must exceed it (TC-2954).
pub const KV_UPLOAD_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(240);
const _: () =
assert!(KV_UPLOAD_TIMEOUT.as_secs() < crate::kv_orphans::ORPHAN_GRACE.whole_seconds() as u64);

Expand Down
125 changes: 124 additions & 1 deletion tinycloud-node-server/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1929,6 +1929,18 @@ pub struct DatabaseConfig {
/// writing a row to the client, so the download itself is not limited.
#[serde(default = "default_database_bulk_statement_timeout_ms")]
pub bulk_statement_timeout_ms: u64,
/// PostgreSQL `idle_in_transaction_session_timeout`, in milliseconds, set
/// once on every pooled connection (TC-2954): the server ends a session
/// that sits idle inside a transaction this long, so a leaked or stalled
/// transaction cannot hold its locks and pooled connection indefinitely.
/// It must exceed the longest legitimate hold, a protected legacy meeting
/// write's upload (`KV_UPLOAD_TIMEOUT`, 240 s); `validate` refuses less.
/// Zero disables it. Not set in `statement_timeout_mode = "local"`: a
/// session `SET` behind a transaction-pooling proxy would land on
/// whichever server connection served it and reach the proxy's other
/// clients.
#[serde(default = "default_database_idle_in_transaction_timeout_ms")]
pub idle_in_transaction_timeout_ms: u64,
/// Enable TCP keepalive and `TCP_USER_TIMEOUT` on PostgreSQL connection
/// sockets, so a connection the network silently drops fails within tens
/// of seconds instead of the kernel's ~15 min retransmission timeout
Expand Down Expand Up @@ -1995,6 +2007,13 @@ fn default_database_bulk_statement_timeout_ms() -> u64 {
120_000
}

fn default_database_idle_in_transaction_timeout_ms() -> u64 {
300_000
}

/// PostgreSQL's largest millisecond timeout setting (`INT_MAX`).
const POSTGRES_MAX_TIMEOUT_MS: u64 = i32::MAX as u64;

fn default_database_tcp_keepalive_idle_secs() -> u64 {
15
}
Expand Down Expand Up @@ -2029,6 +2048,7 @@ impl Default for DatabaseConfig {
statement_timeout_mode: StatementTimeoutMode::default(),
statement_timeout_ms: default_database_statement_timeout_ms(),
bulk_statement_timeout_ms: default_database_bulk_statement_timeout_ms(),
idle_in_transaction_timeout_ms: default_database_idle_in_transaction_timeout_ms(),
tcp_keepalive: true,
tcp_keepalive_idle_secs: default_database_tcp_keepalive_idle_secs(),
tcp_keepalive_interval_secs: default_database_tcp_keepalive_interval_secs(),
Expand All @@ -2052,14 +2072,57 @@ pub struct DidResolutionConfig {

impl DatabaseConfig {
/// Reject settings that would make the node unusable (TC-1317): a zero
/// read-audit wait would answer every authorized read `read_audit_busy`.
/// read-audit wait would answer every authorized read `read_audit_busy`,
/// an idle-in-transaction timeout within the legacy upload hold would
/// fail legitimate writes (TC-2954), and a timeout above PostgreSQL's
/// largest (`INT_MAX` ms) would fail every connection after startup.
pub fn validate(&self) -> Result<(), &'static str> {
if self.read_audit_wait_ms == 0 {
return Err("database.read_audit_wait_ms must be greater than zero");
}
if self.idle_in_transaction_timeout_ms != 0
&& u128::from(self.idle_in_transaction_timeout_ms)
<= tinycloud_core::db::KV_UPLOAD_TIMEOUT.as_millis()
{
return Err(
"database.idle_in_transaction_timeout_ms must be 0 (off) or exceed the 240 s legacy upload hold",
);
}
for (value, error) in [
(
self.idle_in_transaction_timeout_ms,
"database.idle_in_transaction_timeout_ms must be at most 2147483647",
),
(
self.statement_timeout_ms,
"database.statement_timeout_ms must be at most 2147483647",
),
(
self.bulk_statement_timeout_ms,
"database.bulk_statement_timeout_ms must be at most 2147483647",
),
] {
if value > POSTGRES_MAX_TIMEOUT_MS {
return Err(error);
}
}
Ok(())
}

/// The statement that sets the pool's idle-in-transaction timeout on a new
/// connection, or `None` when it is off or the statement timeout mode is
/// `local` (a transaction-pooling proxy).
pub fn idle_in_transaction_statement(&self) -> Option<String> {
(self.idle_in_transaction_timeout_ms > 0
&& self.statement_timeout_mode != StatementTimeoutMode::Local)
.then(|| {
format!(
"SET idle_in_transaction_session_timeout = {}",
self.idle_in_transaction_timeout_ms
)
})
}

/// The pool's PostgreSQL `statement_timeout` policy.
pub fn statement_timeouts(&self) -> StatementTimeouts {
StatementTimeouts {
Expand Down Expand Up @@ -2293,6 +2356,66 @@ mod tests {
}
}

/// TC-2954: the idle-in-transaction timeout is on by default (300 s) and,
/// when on, must outlast the 240 s a protected legacy write holds its
/// transaction.
#[test]
fn tc2954_startup_rejects_an_idle_in_transaction_timeout_within_the_upload_hold() {
assert_eq!(
Config::default().database.idle_in_transaction_timeout_ms,
300_000
);
for (timeout_ms, ok) in [
(0, true),
(1, false),
(240_000, false),
(240_001, true),
(2_147_483_647, true),
(2_147_483_648, false),
] {
let mut config = Config::default();
config.database.idle_in_transaction_timeout_ms = timeout_ms;
assert_eq!(
crate::resolve_runtime_config(&config).is_ok(),
ok,
"{timeout_ms}"
);
}
}

/// TC-2954: a statement timeout above PostgreSQL's largest is refused at
/// startup instead of failing every connection after the old node has
/// stopped.
#[test]
fn startup_rejects_a_statement_timeout_postgres_cannot_hold() {
for (base, bulk, ok) in [
(2_147_483_647, 2_147_483_647, true),
(2_147_483_648, 120_000, false),
(30_000, 2_147_483_648, false),
] {
let mut config = Config::default();
config.database.statement_timeout_ms = base;
config.database.bulk_statement_timeout_ms = bulk;
assert_eq!(
crate::resolve_runtime_config(&config).is_ok(),
ok,
"{base} {bulk}"
);
}
}

/// TC-2954: behind a transaction-pooling proxy (`local` mode) the
/// session-level idle-in-transaction timeout is not set.
#[test]
fn the_idle_in_transaction_timeout_is_not_set_in_local_mode() {
let mut database = Config::default().database;
assert!(database.idle_in_transaction_statement().is_some());
database.statement_timeout_mode = StatementTimeoutMode::Local;
assert_eq!(database.idle_in_transaction_statement(), None);
database.statement_timeout_mode = StatementTimeoutMode::Off;
assert!(database.idle_in_transaction_statement().is_some());
}

/// TC-1491: off by default, 60 s / 8 MiB for every tier, and nothing is
/// validated while the feature is off.
#[test]
Expand Down
14 changes: 13 additions & 1 deletion tinycloud-node-server/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -516,6 +516,13 @@ pub async fn postgres_pool_with_recovery(
);
let statement_timeouts = database.statement_timeouts();
let session_statement = statement_timeouts.session_statement();
let idle_in_transaction_statement = database.idle_in_transaction_statement();
// The value in effect: 0 when off or in `local` statement-timeout mode.
let idle_in_transaction_timeout_ms = if idle_in_transaction_statement.is_some() {
database.idle_in_transaction_timeout_ms
} else {
0
};
let pool = recovery
.install(
connect_opts
Expand All @@ -524,9 +531,13 @@ pub async fn postgres_pool_with_recovery(
)
.after_connect(move |conn, _meta| {
let session_statement = session_statement.clone();
let idle_in_transaction_statement = idle_in_transaction_statement.clone();
Box::pin(async move {
POSTGRES_CONNECTIONS_OPENED.fetch_add(1, Ordering::Relaxed);
if let Some(statement) = session_statement {
for statement in [session_statement, idle_in_transaction_statement]
.into_iter()
.flatten()
{
conn.execute(statement.as_str()).await?;
}
Ok(())
Expand All @@ -541,6 +552,7 @@ pub async fn postgres_pool_with_recovery(
statement_timeout_mode = ?statement_timeouts.mode,
statement_timeout_ms = statement_timeouts.base.as_millis() as u64,
bulk_statement_timeout_ms = statement_timeouts.bulk.as_millis() as u64,
idle_in_transaction_timeout_ms,
retry_window_ms = database.retry_window_ms,
tcp_keepalive = keepalive.is_some(),
tcp_keepalive_idle_secs = keepalive.map(|k| k.idle.as_secs()),
Expand Down
83 changes: 83 additions & 0 deletions tinycloud-node-server/tests/postgres_pool_timeouts.rs
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,15 @@ async fn show_statement_timeout<C: ConnectionTrait>(conn: &C) -> String {
.unwrap()
}

async fn show<C: ConnectionTrait>(conn: &C, setting: &str) -> String {
conn.query_one(statement(&format!("SHOW {setting}")))
.await
.unwrap()
.unwrap()
.try_get_by_index::<String>(0)
.unwrap()
}

fn test_space(name: &str) -> SpaceId {
let jwk = JWK::generate_ed25519().unwrap();
let did: DIDBuf = DID_METHODS.generate(&jwk, "key").unwrap();
Expand Down Expand Up @@ -434,6 +443,80 @@ async fn postgres_statement_timeout_modes_scopes_and_migrations() {
schema.drop().await;
}

/// TC-2954: every pooled connection carries an idle-in-transaction timeout,
/// 300 s by default, so the server ends a transaction left idle that long and
/// a fresh connection serves the next request; 0 turns it off.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn postgres_idle_in_transaction_timeout_ends_an_abandoned_transaction() {
let Some(url) =
postgres_url("postgres_idle_in_transaction_timeout_ends_an_abandoned_transaction")
else {
return;
};
let schema = Schema::new(&url, "idletx").await;
let setting = "idle_in_transaction_session_timeout";

let default = pool(schema.url(&url, None), |_| {}).await;
assert_eq!(show(&default, setting).await, "5min");
let off = pool(schema.url(&url, None), |config| {
config.database.idle_in_transaction_timeout_ms = 0;
})
.await;
assert_eq!(show(&off, setting).await, "0");

// Below the legacy upload hold `validate` refuses; the pool itself takes it.
let short = pool(schema.url(&url, None), |config| {
config.database.idle_in_transaction_timeout_ms = 300;
})
.await;
let transaction = short.begin().await.unwrap();
let abandoned = transaction
.query_one(statement("SELECT pg_backend_pid()"))
.await
.unwrap()
.unwrap()
.try_get_by_index::<i32>(0)
.unwrap();
tokio::time::sleep(Duration::from_millis(1_000)).await;
// The request whose transaction was ended gets one prompt error of the
// connection class: the routes answer it 503 unavailable_retryable, or
// 500 outcome_unknown once a write began (TC-1054). Statements inside a
// transaction are never retried, so nothing reconnects in a loop.
let started = Instant::now();
let error = transaction
.execute(statement("SELECT 1"))
.await
.expect_err("the server ended the idle transaction");
assert!(
started.elapsed() < Duration::from_millis(500),
"{:?}",
started.elapsed()
);
assert!(is_transient_error(&error), "{error:?}");
drop(transaction);
let next = short
.query_one(statement("SELECT pg_backend_pid()"))
.await
.expect("a fresh connection serves the next request")
.unwrap()
.try_get_by_index::<i32>(0)
.unwrap();
assert_ne!(next, abandoned, "the ended session is not reused");
// The ended session is closed, not kept: one replacement serves the pool.
let pool = short.sea_orm_connection().get_postgres_connection_pool();
tokio::time::timeout(Duration::from_secs(5), async {
while pool.size() != 1 {
tokio::time::sleep(Duration::from_millis(20)).await;
}
})
.await
.expect("the pool holds exactly the replacement connection");
for conn in [default, off, short] {
conn.sea_orm_connection().clone().close().await.unwrap();
}
schema.drop().await;
}

/// A statement timeout on the SQL persistence path is a bounded, retryable
/// `Unavailable` (503): the database's operation lock is released, the
/// next request on it succeeds, and other databases are unaffected. A full
Expand Down
Loading