Skip to content

TC-2917: chunked read follow-ups: fallback slot test, bounded slot test, sinks declare buffering - #604

Open
samgbafa wants to merge 1 commit into
mainfrom
fix/tc-2917-chunked-followups
Open

samgbafa wants to merge 1 commit into
mainfrom
fix/tc-2917-chunked-followups

Conversation

@samgbafa

Copy link
Copy Markdown
Contributor

TC-2917. Follow-up to #582: tests, a fail-closed sink contract, and doc wording. No production behaviour changes; chunked artifact storage stays off.

Changes

  • ArtifactSink::buffers_in_memory has no default. Every sink now has to say whether it holds the whole artifact in memory. Before, a new sink or a wrapper that forgot the method answered "no" and took no large-transfer slot. The memory sink answers yes and the cache-file sink answers no, as before; existing test sinks answer no (their behaviour is unchanged).
  • New test postgres_chunked_fallback_to_legacy_holds_no_parts_slot: a read whose metadata says parts but whose row is single-payload falls back to the single-row read. The test checks three things:
    • With one slot free, an in-memory fallback completes. This proves the parts slot is released before the legacy read takes its own.
    • With no slot free, even a streaming fallback waits for the legacy slot.
    • The single-row read cap still applies.
  • Bounded test: in postgres_chunked_in_memory_read_waits_for_a_large_transfer_slot, the streaming read is wrapped in a 30 s timeout, so a regression that makes it wait for a slot fails instead of hanging the job.
  • Docs: the artifact and account capacities are checked when a database is written; a read does not re-check them. The in-memory case is a read into a sink that buffers the whole artifact. A SQL export of a database in parts is not yet bounded by the read cap or a slot, and that is listed as required before enabling.

Validation (local PostgreSQL 18.6): core lib with duckdb 1181/1181; clippy core (duckdb) and node with -Dwarnings; fmt. Throwaway mutations, each now caught: the slot taken for streaming sinks fails the bounded test (it used to hang); the parts slot held into the legacy fallback fails the new fallback test; the legacy cap and slot taken before the parts attempt fails three tests.

Test-ruling: TC-2917. Two existing-test changes:

  • postgres_chunked_in_memory_read_waits_for_a_large_transfer_slot: its streaming read is wrapped in a 30 s tokio::time::timeout, failing with Elapsed. Nothing else in the test changes.
  • The existing test sinks gain buffers_in_memory returning false, the former default. This is a compile-only change.

… declare buffering

Review of #582: a new test pins that a parts read falling back to the
single-row layout releases its slot first and keeps the legacy cap and
slot; the streaming read in the in-memory slot test is bounded so a
regression fails instead of hanging; ArtifactSink::buffers_in_memory has
no default, so a new or wrapping sink must declare it; docs say the
capacities are checked at write time and name the in-memory case
correctly.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant