Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@
## [Unreleased]
### Changes
- PostgreSQL: every pooled connection sets `idle_in_transaction_session_timeout` (`database.idle_in_transaction_timeout_ms`, env `TINYCLOUD_DATABASE__IDLE_IN_TRANSACTION_TIMEOUT_MS`, default 300000 = 5 min, `0` off), so the server ends a transaction left idle that long and the pool replaces the session. A non-zero value of 240 s or less is refused at startup: a protected legacy meeting write legitimately holds its transaction across an upload of up to 240 s. It is not set in `statement_timeout_mode = "local"` (transaction-pooling proxies). `database.idle_in_transaction_timeout_ms`, `statement_timeout_ms` and `bulk_statement_timeout_ms` above 2147483647 (PostgreSQL's largest) are now refused at startup instead of failing every connection. Details: TC-2954.
- A protected legacy meeting write now waits for its space's write order before opening its transaction on PostgreSQL too (as on SQLite), so it no longer holds a pooled connection while queueing behind other writers to the same space. In exchange, on PostgreSQL it now holds that space's write order across its upload, up to 240 s: every other write to the same space waits for it meanwhile, including ordinary KV writes, delegations and revocations, epoch-head repair, Policy/v3 registration and session admission. Other spaces are not affected, and a client cannot slow the upload (the body is staged first). An ordinary KV write that waits long enough may be refused as `KvCandidateUnavailable` (it fails closed). Details: TC-2954.
- A SQL checkpoint save over 8 MiB reads its row's metadata before opening the transaction that uploads it, so that transaction holds no table lock while the payload is sent. A pooled connection closed because it was released inside a transaction is now logged as a warning. Details: TC-2954.
- SQL and DuckDB: a database whose stored checkpoint or WAL delta is larger than 64 MiB is no longer loaded; requests to it answer `507 sql_database_size_limit` and nothing runs (`storage.sql.max_artifact_read_bytes`, env `TINYCLOUD_STORAGE__SQL__MAX_ARTIFACT_READ_BYTES`; `0` off; the node refuses to start with it below the write limit). A database already stored over the limit becomes unreadable, not only unwritable. DuckDB now answers both artifact limits `507` instead of `500`. At most two database artifact transfers over 8 MiB (whole-checkpoint saves and large loads) run at once per node, for all tenants; further ones wait without holding a database connection, and a SQL save cut while still waiting starts no write back-off. The checkpoint back-off is held in the node's memory: a restart clears it. Details: TC-2961.
- SQLite stamp replay deadlines now use the same timestamp encoding as cleanup (TC-2920).
- PostgreSQL database artifacts use uncompressed external storage for new writes to avoid repeated decompression during chunked reads (TC-2917). Existing large artifacts require an operator re-toast; verify both storage settings after rollout and follow `scripts/tc2917-README.md` if lock contention deferred the migration.
- Encryption: v2 column format with key ids, per-purpose keys and recorded format metadata. Optional strict reads are off by default. Details: TC-2516, TC-2517.
Expand Down
13 changes: 11 additions & 2 deletions docs/sql-local-durability.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,10 +61,19 @@ A kill -9 test cannot inject "after COMMIT, before its fsync completed"; `before
| `max_database_bytes` | `0` (no cap) | per-database size cap, enforced by SQLite's `max_page_count` |
| `min_free_bytes` | `0` (check off) | disk kept free after all in-flight reservations; `0` disables the gate, the reservation and the free-space checks before COMMIT (a full-disk error still reads free space once, see Errors) |
| `max_artifact_write_bytes` | `64 MiB` (on) | the largest SQL or DuckDB artifact payload saved in one statement (TC-2961). A larger whole checkpoint is refused before it is sent, with `507 sql_database_size_limit`, and the local change is discarded. `0` turns it off, for local tests only |
| `max_artifact_read_bytes` | `64 MiB` (on) | the largest stored SQL or DuckDB checkpoint or WAL delta the node loads (TC-2961). A larger database is refused before any of it is read, with `507 sql_database_size_limit`, instead of being loaded whole. `0` turns it off; otherwise the node refuses to start with it below `max_artifact_write_bytes` (or with that off), since databases between the two would be saved and then unloadable |

**The two limits are off by default.** They are Sam's decision (card `sql-storage-report-read`) and are set by config once he answers; an unmeasured cap could refuse real users' databases. The mechanism is fully tested with explicit values. What stays on by default besides `synchronous`: the transaction per write request and the owner-budget limit below.
**`max_database_bytes` and `min_free_bytes` are off by default.** They are Sam's decision (card `sql-storage-report-read`) and are set by config once he answers; an unmeasured cap could refuse real users' databases. The mechanism is fully tested with explicit values. What stays on by default besides `synchronous`: the transaction per write request, the owner-budget limit and the two artifact limits below.

**The artifact write limit is on by default (TC-2961).** A whole checkpoint is saved as one parameter of one statement, and PostgreSQL needs a multiple of it in memory, so one tenant's large database could take down the shared primary. A database whose checkpoint exceeds 64 MiB cannot persist a write until artifacts are stored in bounded parts (TC-2917). After a whole-checkpoint save fails or is refused, the database's writes are refused without reloading it (`507` after a refusal, `503 unavailable_retryable` after a failure) for 30 s, doubling per further failure up to 15 min; reads are not affected, and a successful checkpoint save ends the back-off.
**The artifact write limit is on by default (TC-2961).** A whole checkpoint is saved as one parameter of one statement, and PostgreSQL needs a multiple of it in memory, so one tenant's large database could take down the shared primary. A database whose checkpoint exceeds 64 MiB cannot persist a write until artifacts are stored in bounded parts (TC-2917). After a whole-checkpoint save fails or is refused, the database's writes are refused without reloading it (`507` after a refusal, `503 unavailable_retryable` after a failure) for 30 s, doubling per further failure up to 15 min; reads are not affected, and a successful checkpoint save ends the back-off. A save cut while it still waits for its large-transfer slot (below) sent nothing and starts no back-off. The back-off is held in the node's memory: a restart clears it, and each node instance keeps its own.

**The artifact read limit is on by default (TC-2961).** A database whose stored checkpoint or WAL delta exceeds 64 MiB is not loaded: every request to it is answered `507 sql_database_size_limit` until artifacts are stored in bounded parts (TC-2917). A database already stored over the limit becomes unreadable, not only unwritable; count such rows before deploying the limit or lowering it.

**At most two artifact transfers over 8 MiB run at once per node (TC-2961)**, shared by every tenant and by SQL and DuckDB, first come first served, with no per-tenant share. Further ones wait holding no database connection. How long one can wait and run depends on its path:
- a SQL checkpoint save: wait and upload together within `artifact_transfer_timeout_ms` (45 s); a save cut while still waiting answers `503` and starts no back-off;
- a SQL load: wait and download together within `hydration_max_ms` (600 s), in a task that outlives its callers, who answer `503` after `hydration_wait_timeout_ms` (30 s);
- DuckDB: bounded only by the `/invoke` request deadline where it applies.
Transfers of 8 MiB or less are not counted.

**Owner budget is not switchable and can refuse a write today's code allows.** `max_page_count` is set from the owner's remaining quota (`limit - used`, the same `storage_check` read as before). Today a write that overshoots a non-zero remainder is allowed once and only the next one is refused with 402; now the overshooting write itself is refused with 402 before COMMIT. The comparison is the database's page growth against the remainder, the unit the quota already charges (logical database size); growth that frees pages elsewhere in the file is not counted, so it is never stricter than the charge.

Expand Down
Loading
Loading