Skip to content

fix: order legacy write locks, slim the checkpoint transaction, artifact read limit and large-transfer cap (TC-2954, TC-2961) - #585

Merged
samgbafa merged 5 commits into
mainfrom
feat/tc-2954-followups
Nov 13, 2026
Merged

samgbafa merged 5 commits into
mainfrom
feat/tc-2954-followups

Conversation

@samgbafa

@samgbafa samgbafa commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Linear: TC-2954 (follow-ups to #580) and TC-2961 (read limit, large-transfer limit). The idle-in-transaction timeout is in its own PR, #586.

Behaviour

  • Legacy write lock order. A protected legacy meeting write now takes its space's in-process write-order lock before it opens its transaction on PostgreSQL too, as SQLite already did.
    • Before, it opened the transaction and then waited for that lock while holding a pooled connection. The writers holding the lock need a connection for their own BEGIN.
    • Same-space ordinary writers now wait out the legacy upload, without holding a connection.
    • Cost (PostgreSQL only; SQLite already behaved this way): the protected writer now holds its space's sequence guard across the upload, up to 240 s (KV_UPLOAD_TIMEOUT). Before, on PostgreSQL it took the guard only after the upload. Every write to the same space that takes the guard waits meanwhile: ordinary KV writes, delegations and revocations (transact), repair_epoch_heads, Policy/v3 registration and session admission. Other spaces are unaffected. The client cannot slow the upload, because the body is staged before the guard is taken.
    • An ordinary KV writer that waits long enough can pass the orphan grace and be refused KvCandidateUnavailable. This fails closed: nothing is written. [inferred from the code, not reproduced]
  • Checkpoint metadata read. A checkpoint save over 8 MiB reads its row's metadata on the pool before it opens the bulk-timeout transaction for the write. That transaction no longer holds a lock on database_artifact while the payload uploads. The compare-and-swap on the write is unchanged.
  • Backstop warning. When the pool closes a connection because it was released inside a transaction (the fix: never pool a PostgreSQL connection inside a transaction after a cancelled BEGIN (TC-2954) #580 backstop), it now logs a warning.
  • Artifact read limit (TC-2961). A SQL or DuckDB database whose stored checkpoint or WAL delta exceeds storage.sql.max_artifact_read_bytes is not loaded: the request answers 507 sql_database_size_limit before any of it is read. On by default at 64 MiB; 0 turns it off. The log line carries no values.
    • Without it, a read of an oversized database loads it whole in 4 MiB slices; on a compressed stored value each slice decompresses everything before it.
    • A refused load surfaces as the 507, not as an internal error. DuckDB now answers both artifact limits 507 as well; before, they were 500.
    • Startup (and --validate-config) refuses a read limit below the write limit, or a read limit with the write limit off. Databases between the two would be saved and then refused on every load.
    • A database already stored over the limit becomes unreadable, not just unwritable. Count such rows before deploying.
  • Large-transfer limit (TC-2961). At most two artifact transfers over 8 MiB (whole-checkpoint saves and loads of a large database) run at once per node. The limit is shared by every tenant and by SQL and DuckDB, first come first served, with no per-tenant share.
    • Further transfers wait before the writer lock and before any statement, holding no database connection. Smaller transfers are not affected.
    • How long a transfer may wait and run depends on its path:
      • a SQL save: within artifact_transfer_timeout_ms (45 s);
      • a SQL load: within hydration_max_ms (600 s), in a detached task whose callers answer 503 after 30 s;
      • DuckDB: only the /invoke deadline.
    • A SQL checkpoint save cut while it is still waiting for its slot has sent nothing. It answers busy and does NOT start the write back-off. track_transfer_start reports whether the save got its slot.
  • Back-off state (TC-2961, from fix: refuse SQL checkpoints over 64 MiB and back off failed checkpoint saves (TC-2961) #583). The checkpoint back-off is held in the node's memory: a restart clears it, and each node instance keeps its own. It logs at warn. Now stated in the CHANGELOG and the durability doc.

Tests

  • db::test::postgres_legacy_writer_queues_for_the_sequence_guard_without_a_connection: a protected writer queued behind the space lock holds no pooled connection. It fails with the old order (1 connection in use).
  • db_pool::release_tests::postgres_a_deadline_during_a_sea_orm_begin_returns_the_connection_outside_a_transaction: a slow-link proxy holds the reply to a SeaORM begin() past the request deadline. The connection is reused outside any transaction and the next borrower's write persists. It fails without the fix: never pool a PostgreSQL connection inside a transaction after a cancelled BEGIN (TC-2954) #580 transaction fix (the backstop replaces the connection).
  • database_artifacts::tests::a_part_over_the_read_limit_is_refused_before_anything_is_read: an over-limit checkpoint and an over-limit WAL delta are refused before the sink receives a byte; one at the limit is read.
  • database_artifacts::tests::large_transfers_run_at_most_two_at_a_time: of three concurrent loads over 8 MiB, the third does not start until one ends; a small load is not held back.
  • sql::service::tests::a_database_over_the_read_limit_answers_the_size_limit: a read and a write to an over-limit database both answer the size limit (507), not an internal error.
  • sql::service::tests::a_save_still_queued_for_a_permit_starts_no_back_off (Bobbie's F1 repro, final assertion inverted): another tenant's stalled loads hold both slots, the victim's large save times out still queued, and its next write succeeds with no back-off.
  • database_artifacts::tests::postgres_large_saves_hold_their_permits_through_the_upload (from Bobbie's review): two saves stalled in their UPDATE hold both slots, and a third large load waits.
  • database_artifacts::tests::postgres_a_large_save_waiting_for_a_permit_holds_no_connection (from Bobbie's review): on a one-connection pool, a small save runs while a large save waits for a slot.
  • database_artifacts::tests::the_read_and_write_limits_compare_the_same_stored_bytes (from Bobbie's review): equal at the limit is readable, a read limit one below refuses it, and 0 is off.
  • routes::tests::a_database_too_large_to_load_is_507: SQL DatabaseTooLargeToLoad, and both DuckDB limits under the duckdb feature, answer 507 with the size-limit code.
  • config::tests::startup_refuses_an_artifact_read_limit_below_the_write_limit.
  • Existing test changed under test-ruling: TC-2961 (bounded_read): tests/artifact_bounded_read.rs:96 turns the read limit off (with_max_read_bytes(0)), because that test measures the heap of a 256 MiB streaming load, not the limit.

@samgbafa
samgbafa force-pushed the feat/tc-2954-followups branch from 494ef4e to a27df8f Compare November 13, 2026 15:37
@samgbafa samgbafa changed the title fix: bound idle transactions, order legacy write locks, and slim the checkpoint transaction (TC-2954) fix: order legacy write locks and slim the checkpoint transaction (TC-2954) Nov 13, 2026
@samgbafa
samgbafa force-pushed the feat/tc-2954-followups branch from a27df8f to a090b18 Compare November 13, 2026 16:30
@samgbafa samgbafa changed the title fix: order legacy write locks and slim the checkpoint transaction (TC-2954) fix: order legacy write locks, slim the checkpoint transaction, artifact read limit and large-transfer cap (TC-2954, TC-2961) Nov 13, 2026
@samgbafa
samgbafa force-pushed the feat/tc-2954-followups branch from aceb3e0 to ea5d367 Compare November 13, 2026 18:33
@samgbafa
samgbafa marked this pull request as ready for review November 13, 2026 19:59
…-2954)

- A protected legacy meeting write takes its space's sequence guard before
  BEGIN on PostgreSQL too, so it never holds a pooled connection while
  queueing for it.
- A checkpoint save reads its row's metadata before opening the bulk-timeout
  transaction for the write.
- The pool logs a warning when it closes a connection released inside a
  transaction.
- Regression for a deadline during a SeaORM begin through a slow link.

The idle-in-transaction timeout moved to its own PR (#586).
…sfers

A database whose stored checkpoint or WAL delta exceeds
storage.sql.max_artifact_read_bytes (default 64 MiB, 0 = off) is not
loaded: 507 sql_database_size_limit before any slice is read. At most
two artifact transfers over 8 MiB run at once per repository; further
ones wait holding no connection. Documents that the checkpoint back-off
is in-process.
…ot; DuckDB 507; cap cross-check; review tests
@samgbafa
samgbafa force-pushed the feat/tc-2954-followups branch from 4a26402 to 2eab157 Compare November 13, 2026 20:01
@samgbafa
samgbafa merged commit e1df783 into main Nov 13, 2026
28 checks passed
samgbafa pushed a commit that referenced this pull request Nov 13, 2026
…e a transfer slot

Rebased onto TC-2961 (#585): a database stored in parts streams in
bounded statements, so the whole-artifact read cap and large-transfer
permit apply only to the single-row layout. A sink that keeps the whole
artifact in memory (MemorySink, the DuckDB export) still takes a
large-transfer slot for a parts read over 8 MiB.
samgbafa pushed a commit that referenced this pull request Nov 13, 2026
…e a transfer slot

Rebased onto TC-2961 (#585): a database stored in parts streams in
bounded statements, so the whole-artifact read cap and large-transfer
permit apply only to the single-row layout. A sink that keeps the whole
artifact in memory (MemorySink, the DuckDB export) still takes a
large-transfer slot for a parts read over 8 MiB.
samgbafa pushed a commit that referenced this pull request Nov 13, 2026
…e a transfer slot

Rebased onto TC-2961 (#585): a database stored in parts streams in
bounded statements, so the whole-artifact read cap and large-transfer
permit apply only to the single-row layout. A sink that keeps the whole
artifact in memory (MemorySink, the DuckDB export) still takes a
large-transfer slot for a parts read over 8 MiB.
samgbafa pushed a commit that referenced this pull request Nov 13, 2026
…e a transfer slot

Rebased onto TC-2961 (#585): a database stored in parts streams in
bounded statements, so the whole-artifact read cap and large-transfer
permit apply only to the single-row layout. A sink that keeps the whole
artifact in memory (MemorySink, the DuckDB export) still takes a
large-transfer slot for a parts read over 8 MiB.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant