Repository navigation
fix: order legacy write locks, slim the checkpoint transaction, artifact read limit and large-transfer cap (TC-2954, TC-2961) - #585
Merged
Conversation
samgbafa
force-pushed
the
feat/tc-2954-followups
branch
from
November 13, 2026 15:37
494ef4e to
a27df8f
Compare
samgbafa
force-pushed
the
feat/tc-2954-followups
branch
from
November 13, 2026 16:30
a27df8f to
a090b18
Compare
samgbafa
force-pushed
the
feat/tc-2954-followups
branch
from
November 13, 2026 18:33
aceb3e0 to
ea5d367
Compare
samgbafa
marked this pull request as ready for review
November 13, 2026 19:59
…-2954) - A protected legacy meeting write takes its space's sequence guard before BEGIN on PostgreSQL too, so it never holds a pooled connection while queueing for it. - A checkpoint save reads its row's metadata before opening the bulk-timeout transaction for the write. - The pool logs a warning when it closes a connection released inside a transaction. - Regression for a deadline during a SeaORM begin through a slow link. The idle-in-transaction timeout moved to its own PR (#586).
…sfers A database whose stored checkpoint or WAL delta exceeds storage.sql.max_artifact_read_bytes (default 64 MiB, 0 = off) is not loaded: 507 sql_database_size_limit before any slice is read. At most two artifact transfers over 8 MiB run at once per repository; further ones wait holding no connection. Documents that the checkpoint back-off is in-process.
…ot; DuckDB 507; cap cross-check; review tests
samgbafa
force-pushed
the
feat/tc-2954-followups
branch
from
November 13, 2026 20:01
4a26402 to
2eab157
Compare
samgbafa
pushed a commit
that referenced
this pull request
Nov 13, 2026
…e a transfer slot Rebased onto TC-2961 (#585): a database stored in parts streams in bounded statements, so the whole-artifact read cap and large-transfer permit apply only to the single-row layout. A sink that keeps the whole artifact in memory (MemorySink, the DuckDB export) still takes a large-transfer slot for a parts read over 8 MiB.
samgbafa
pushed a commit
that referenced
this pull request
Nov 13, 2026
…e a transfer slot Rebased onto TC-2961 (#585): a database stored in parts streams in bounded statements, so the whole-artifact read cap and large-transfer permit apply only to the single-row layout. A sink that keeps the whole artifact in memory (MemorySink, the DuckDB export) still takes a large-transfer slot for a parts read over 8 MiB.
samgbafa
pushed a commit
that referenced
this pull request
Nov 13, 2026
…e a transfer slot Rebased onto TC-2961 (#585): a database stored in parts streams in bounded statements, so the whole-artifact read cap and large-transfer permit apply only to the single-row layout. A sink that keeps the whole artifact in memory (MemorySink, the DuckDB export) still takes a large-transfer slot for a parts read over 8 MiB.
samgbafa
pushed a commit
that referenced
this pull request
Nov 13, 2026
…e a transfer slot Rebased onto TC-2961 (#585): a database stored in parts streams in bounded statements, so the whole-artifact read cap and large-transfer permit apply only to the single-row layout. A sink that keeps the whole artifact in memory (MemorySink, the DuckDB export) still takes a large-transfer slot for a parts read over 8 MiB.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linear: TC-2954 (follow-ups to #580) and TC-2961 (read limit, large-transfer limit). The idle-in-transaction timeout is in its own PR, #586.
Behaviour
BEGIN.KV_UPLOAD_TIMEOUT). Before, on PostgreSQL it took the guard only after the upload. Every write to the same space that takes the guard waits meanwhile: ordinary KV writes, delegations and revocations (transact),repair_epoch_heads, Policy/v3 registration and session admission. Other spaces are unaffected. The client cannot slow the upload, because the body is staged before the guard is taken.KvCandidateUnavailable. This fails closed: nothing is written. [inferred from the code, not reproduced]database_artifactwhile the payload uploads. The compare-and-swap on the write is unchanged.storage.sql.max_artifact_read_bytesis not loaded: the request answers507 sql_database_size_limitbefore any of it is read. On by default at 64 MiB;0turns it off. The log line carries no values.--validate-config) refuses a read limit below the write limit, or a read limit with the write limit off. Databases between the two would be saved and then refused on every load.artifact_transfer_timeout_ms(45 s);hydration_max_ms(600 s), in a detached task whose callers answer 503 after 30 s;/invokedeadline.track_transfer_startreports whether the save got its slot.Tests
db::test::postgres_legacy_writer_queues_for_the_sequence_guard_without_a_connection: a protected writer queued behind the space lock holds no pooled connection. It fails with the old order (1 connection in use).db_pool::release_tests::postgres_a_deadline_during_a_sea_orm_begin_returns_the_connection_outside_a_transaction: a slow-link proxy holds the reply to a SeaORMbegin()past the request deadline. The connection is reused outside any transaction and the next borrower's write persists. It fails without the fix: never pool a PostgreSQL connection inside a transaction after a cancelled BEGIN (TC-2954) #580 transaction fix (the backstop replaces the connection).database_artifacts::tests::a_part_over_the_read_limit_is_refused_before_anything_is_read: an over-limit checkpoint and an over-limit WAL delta are refused before the sink receives a byte; one at the limit is read.database_artifacts::tests::large_transfers_run_at_most_two_at_a_time: of three concurrent loads over 8 MiB, the third does not start until one ends; a small load is not held back.sql::service::tests::a_database_over_the_read_limit_answers_the_size_limit: a read and a write to an over-limit database both answer the size limit (507), not an internal error.sql::service::tests::a_save_still_queued_for_a_permit_starts_no_back_off(Bobbie's F1 repro, final assertion inverted): another tenant's stalled loads hold both slots, the victim's large save times out still queued, and its next write succeeds with no back-off.database_artifacts::tests::postgres_large_saves_hold_their_permits_through_the_upload(from Bobbie's review): two saves stalled in their UPDATE hold both slots, and a third large load waits.database_artifacts::tests::postgres_a_large_save_waiting_for_a_permit_holds_no_connection(from Bobbie's review): on a one-connection pool, a small save runs while a large save waits for a slot.database_artifacts::tests::the_read_and_write_limits_compare_the_same_stored_bytes(from Bobbie's review): equal at the limit is readable, a read limit one below refuses it, and 0 is off.routes::tests::a_database_too_large_to_load_is_507: SQLDatabaseTooLargeToLoad, and both DuckDB limits under theduckdbfeature, answer 507 with the size-limit code.config::tests::startup_refuses_an_artifact_read_limit_below_the_write_limit.tests/artifact_bounded_read.rs:96turns the read limit off (with_max_read_bytes(0)), because that test measures the heap of a 256 MiB streaming load, not the limit.