Skip to content

fix: end PostgreSQL transactions left idle for 5 min (TC-2954) - #586

Merged
samgbafa merged 4 commits into
mainfrom
feat/tc-2954-idle-tx-timeout
Nov 13, 2026
Merged

samgbafa merged 4 commits into
mainfrom
feat/tc-2954-idle-tx-timeout

Conversation

@samgbafa

@samgbafa samgbafa commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Linear: TC-2954 (sibling of #580; related: TC-2961, #583)

Behaviour

Every connection in the node's PostgreSQL pool sets idle_in_transaction_session_timeout once, when it connects. It is set next to the existing session statement_timeout.

  • When a session sits idle inside a transaction for longer than the timeout, PostgreSQL ends the session. Its locks are released, the open transaction rolls back, and the pool replaces the connection, so the next request gets a fresh one.
  • Setting: database.idle_in_transaction_timeout_ms, env TINYCLOUD_DATABASE__IDLE_IN_TRANSACTION_TIMEOUT_MS. 0 turns it off.
  • Default: 300000 ms (5 min). This bounds leaked or stalled transactions; it is not the fix for the checkpoint crash (fix: refuse SQL checkpoints over 64 MiB and back off failed checkpoint saves (TC-2961) #583).
  • The default stays above the longest legitimate hold: a protected legacy meeting write keeps its transaction open while it uploads, for up to KV_UPLOAD_TIMEOUT (240 s). A shorter timeout would fail those writes and slow whole-checkpoint saves.
  • At startup (resolve_runtime_config, also used by --validate-config), a non-zero value of 240 s or less is refused, so it cannot be set below that bound by accident.
  • The timeout also bounds a slow upload inside a transaction. PostgreSQL counts the time spent receiving the next message as idle, so a large single-parameter statement in an open transaction must arrive within the timeout. Today that is a whole-checkpoint save of up to 64 MiB in its bulk transaction: at 300 s that needs only about 220 KiB/s. fix: save whole checkpoints in autocommit, without a transaction (TC-2961) #592 removes that transaction, saving the checkpoint as one autocommit statement.
  • Not set when database.statement_timeout_mode = "local", the mode for transaction-pooling proxies. There a session SET lands on whichever server connection served it and reaches the proxy's other clients. The scoped SET LOCAL limits are unchanged.
  • Upper bound: idle_in_transaction_timeout_ms, statement_timeout_ms and bulk_statement_timeout_ms above 2147483647 (PostgreSQL's INT_MAX ms) are refused at startup. Before, such a value passed validation and every connection then failed, after a deploy had already stopped the old node.
  • PostgreSQL only; the SQLite pool is unchanged. The operator binary tinycloud-sql-identity opens its own pool without this timeout.
  • The effective value is logged with the pool's other connection limits at startup; it is 0 in local mode.

Tests

  • tinycloud-node-server/tests/postgres_pool_timeouts.rs::postgres_idle_in_transaction_timeout_ends_an_abandoned_transaction
    • The default shows 5min on a pooled connection; 0 leaves it off.
    • With an explicitly configured 300 ms timeout (the pool takes it; only startup validation refuses it), a transaction left idle past the timeout errors on its next statement.
    • That error comes back promptly (under 500 ms) and is connection-class (is_transient_error). The routes therefore answer it with the TC-1054 contract: 503 unavailable_retryable, or 500 outcome_unknown once a write began. Statements inside a transaction are never retried, so nothing reconnects in a loop.
    • The next request is served by a different backend (fresh connection), and the pool settles at exactly one connection (the replacement).
  • config::tests::tc2954_startup_rejects_an_idle_in_transaction_timeout_within_the_upload_hold: the default is 300000; 0, 240001 and 2147483647 are accepted; 1, 240000 and 2147483648 are refused.
  • config::tests::startup_rejects_a_statement_timeout_postgres_cannot_hold: a base or bulk statement_timeout of 2147483648 is refused; 2147483647 is accepted.
  • config::tests::the_idle_in_transaction_timeout_is_not_set_in_local_mode: no SET in local mode; set in session and off modes.

samgbafa pushed a commit that referenced this pull request Nov 13, 2026
…-2954)

- A protected legacy meeting write takes its space's sequence guard before
  BEGIN on PostgreSQL too, so it never holds a pooled connection while
  queueing for it.
- A checkpoint save reads its row's metadata before opening the bulk-timeout
  transaction for the write.
- The pool logs a warning when it closes a connection released inside a
  transaction.
- Regression for a deadline during a SeaORM begin through a slow link.

The idle-in-transaction timeout moved to its own PR (#586).
@samgbafa samgbafa changed the title fix: end PostgreSQL transactions left idle for 15 s (TC-2954) fix: end PostgreSQL transactions left idle for 5 min (TC-2954) Nov 13, 2026
samgbafa pushed a commit that referenced this pull request Nov 13, 2026
…-2954)

- A protected legacy meeting write takes its space's sequence guard before
  BEGIN on PostgreSQL too, so it never holds a pooled connection while
  queueing for it.
- A checkpoint save reads its row's metadata before opening the bulk-timeout
  transaction for the write.
- The pool logs a warning when it closes a connection released inside a
  transaction.
- Regression for a deadline during a SeaORM begin through a slow link.

The idle-in-transaction timeout moved to its own PR (#586).
Every connection in the node's PostgreSQL pool sets
idle_in_transaction_session_timeout once at connect
(database.idle_in_transaction_timeout_ms, default 15000, 0 off), so a
stalled or leaked transaction cannot keep holding its locks and pooled
connection; the pool replaces the ended session.
@samgbafa
samgbafa force-pushed the feat/tc-2954-idle-tx-timeout branch from 1efe968 to 1ec889e Compare November 13, 2026 18:33
samgbafa pushed a commit that referenced this pull request Nov 13, 2026
…-2954)

- A protected legacy meeting write takes its space's sequence guard before
  BEGIN on PostgreSQL too, so it never holds a pooled connection while
  queueing for it.
- A checkpoint save reads its row's metadata before opening the bulk-timeout
  transaction for the write.
- The pool logs a warning when it closes a connection released inside a
  transaction.
- Regression for a deadline during a SeaORM begin through a slow link.

The idle-in-transaction timeout moved to its own PR (#586).
@samgbafa
samgbafa marked this pull request as ready for review November 13, 2026 19:59
@samgbafa
samgbafa merged commit b625e1c into main Nov 13, 2026
28 checks passed
samgbafa pushed a commit that referenced this pull request Nov 13, 2026
…-2954)

- A protected legacy meeting write takes its space's sequence guard before
  BEGIN on PostgreSQL too, so it never holds a pooled connection while
  queueing for it.
- A checkpoint save reads its row's metadata before opening the bulk-timeout
  transaction for the write.
- The pool logs a warning when it closes a connection released inside a
  transaction.
- Regression for a deadline during a SeaORM begin through a slow link.

The idle-in-transaction timeout moved to its own PR (#586).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant