Skip to content

perf(qwp): speed up QWP ingestion of rows spread across several tables - #8195

Open
jerrinot wants to merge 61 commits into
masterfrom
jh_qwp_cross_table_commit_batch
Open

jerrinot wants to merge 61 commits into
masterfrom
jh_qwp_cross_table_commit_batch

Conversation

@jerrinot

@jerrinot jerrinot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor
  • One-row frames spread across tables ingest up to 7x faster.
  • Java client 1.3.12: one-row frames across tables up to 5x faster.
  • Interleaved frames commit once per table per receive pass, cutting WAL transactions.
  • Acks larger than http.send.buffer.size no longer drop the connection and duplicate rows.
  • A failed table commit can duplicate other non-DEDUP tables' rows, up to 2 MiB.
  • A demote waits for in-flight QWP batches: 1.2-1.7 ms per table without a group window.
  • A demote mid-batch closes the connection after its syncs: 10-13 s at 1,000 sync tables.
  • A refused or failed sync batch's replay reopens its writers: 28-43 s at 1,000 tables.
  • QWP over UDP resumes a table's ingestion after a failed WAL segment roll.

Tandem: questdb/java-questdb-client#122.

The server's commit batch now spans all tables of a receive pass: plain frames append to their tables' WAL writers, and each table commits once when the pass ends or at the first frame that cannot join. Before a frame's first block of a table appends, the table's WAL writer marks a savepoint. When the frame fails or cannot join, each writer it touched rolls back to its savepoint, the batch commits as it stands, and the frame is answered on its own. If that rollback fails, or the frame's failure leaves the writers in doubt, the server commits nothing of the batch and NACKs the failed frame, as master does.

QuestDB commits each table separately, so the server checks a table for what would refuse its commit (insert permission, a dropped, renamed or suspended table) as the table joins the batch: a refused table's first frame ends the batch, the frames before it commit, and that frame gets the refusal. The batch sequences its tables under the role-switch lock, which it holds across all of them, and reads the node's role once: a demote lands between batches and never splits one, and a read-only node refuses the whole batch. Under the sync and adaptive commit modes, each table first appends its WAL commit record and syncs its WAL files outside the lock (WalWriter.prepareCommit()), so only the sequencer appends, and without a group window their log syncs, run under it. A failed WAL file sync therefore fails the batch before any table commits. A batch refused or failed after its tables prepared discards their writers, which the replay reopens. An adaptive group flush that falls due in a batch runs after the lock is released; when it fails, the batch keeps its ack and the table's next frame is NACKed. A storage fault while sequencing, or a DROP, RENAME, SUSPEND WAL or REVOKE that lands while a table waits in the batch, can still fail a table after others committed: the ack then stops before the failed table's first frame, and the client's replay sends the committed tables' rows from the rest of that receive pass (http.recv.buffer.size, 2 MiB by default) again.

An ack names every table it covers and could outgrow the send buffer, which dropped the connection. The server now stages such a frame in connection memory and sends it in send-buffer-sized pieces.

A WAL writer that failed while it held no uncommitted rows, for example on a segment roll, stayed in the QWP table cache, because neither the commit nor the rollback path touches such a writer. The table lookup now evicts a distressed writer and the next frame or datagram gets a fresh one.

Rows/s on one connection: server pinned to 12 cores of a Ryzen 9 9950X, client on 4 other cores, tmpfs, loopback, JDK 25. Raw replays pre-encoded frames (QwpIngressMultiTableBenchmark, server only); Java is the same benchmark with Java client 1.3.12.

case raw before raw after Java before Java after
1 table, 1-row frames 8.26M 8.92M 5.13M 5.06M
2 tables alternating, 1-row frames 1.10M 7.34M 0.99M 4.42M
10 tables, 1-row frames 1.00M 7.07M 0.96M 3.95M
100 tables, 1-row frames 0.94M 5.53M 0.83M 1.14M
2 tables, 1 row of each per frame 1.17M 8.40M 1.12M 5.61M
1 table, 1,000-row frames* 61.8M 60.9M 33.4M 32.6M
2 tables, 1,000-row frames* 46.7M 59.8M 16.2M 16.3M

* Raw and Java: one 1 s warmup and three 1 s measurements on a fresh server, because the default run fills a 45 GB tmpfs; expect ±20% noise.

bluestreak01 and others added 6 commits November 2, 2026 17:46
Posted reviews state the review level but never the model that wrote
them, so a reader cannot tell which model to credit for a review or
compare the reviews that different models write.

The review-pr skill now asks for the model on the same opening line as
the level, and Step 4 makes that line the first line of the report.
The console output, fix-pr's report.md and the review that approve-pr
and reject-pr post therefore all carry it.

The skill reads the model from the harness, never from memory, since
models misstate their own name and version:
- pi: $PI_PROVIDER/$PI_MODEL and $PI_REASONING_LEVEL. pi sets them per
  session, so a review that fix-pr runs in a subagent names its own
  model.
- Claude Code: the exact model ID its system prompt states.
- Codex: the model and effort of the session's latest turn, from the
  rollout file that $CODEX_THREAD_ID names; the command needs jq.

When the harness reports no model, the line says "model unknown"
instead of a guess. The pi and Codex commands ran against a live pi
session and a real Codex rollout file, with and without the
variables set.
Reviews kept reporting missing tests for scenarios few users reach,
such as ATTACH and DETACH PARTITION edge cases or non-WAL tables that
ILP over TCP writes, even when the reviewer's own probe showed the
code worked. The risk rating that lets rare bugs ship covers only
Critical findings, so these gaps still arrived as Moderate findings,
or as never-run lines that the coverage map listed.

review-pr now splits the ways to reach a change. The main path is
the behavior the PR adds or fixes, reached the way its users
routinely reach it. It includes the trigger of a fixed bug and the
main path's own error and cleanup paths, so a missing test there is
still a gap. Every other path is exotic: a non-default setting or
table type, a rarely used operation or its edge cases, a less common
route into the change, or an unusual combination of features. The
reviewer probes an exotic path where the change may behave
differently. A failing probe is a functional finding, admitted and
rated like any other bug. Otherwise the row is EXOTIC: no finding,
and the report asks for no test.

The rule reaches every place that turned a missing test into a
finding: the Step 2.6 classification and probe rule, Agent 5's
cross-context check and mutation spot-check, the execution-mode and
cross-context checklists, the Step 3b admission of coverage gaps and
the Moderate definition. The coverage map still lists never-run
lines, as a record that asks for no test. The Claude Code and Codex
copies of the skill get the same rule.
PR bodies often pinned a tandem PR's head or a submodule pointer by
commit hash, or counted the tests a change added. Both go stale as
soon as the branch moves, and reviews then reported the stale body;
the Claude Code and Codex copies of review-pr listed it as a
Moderate finding. CLAUDE.md also banned design narrative, so a body
said what users gain but not how the change gets there.

CLAUDE.md now asks for effect bullets, then the approach: one to
three sentences per fix or feature on the root cause or design and
how the change addresses it. Bodies link tandem PRs by number and
leave out commit hashes and counts of tests, files, lines or
commits.

review-pr checks that format and asks for hashes and counts to go,
without checking them against the code. Its report opens with the
revisions it reviewed, a tandem PR's head included. The Claude Code
and Codex copies get pi's Step 2, its Title and description section,
which keeps body problems out of the findings, and its Agent 7,
which leaves the body to Step 2. fix-pr's analysis reports each
item's approach next to its effect bullet, and the PR body that
fix-pr prints carries both.
A person can push back on a review item in a PR comment, for example
to explain why they won't fix it. review-pr ignored such replies: Step
2.5f re-tested every earlier finding and re-raised each one that still
reproduced, fresh discovery agents could find the same issue again,
and the Claude Code and Codex copies had no rule for earlier reviews
at all. A follow-up review on #7263 re-raised as Critical an item its
author had already declined. Replies in review threads on the diff
never reached the review either, since gh pr view --comments leaves
them out.

An item a person pushed back on is now settled: a reply declines it,
or disputes it with a reason. The review never re-tests a settled
item and never raises it again in any section of the report, even
when it still reproduces. Admission omits a candidate that restates
it. Only a defect with another symptom, or one that a later commit
makes worse than the reply answered, counts as new.

Step 1 also reads review-thread comments. Discovery agents get the
settled items with the surface map and must not return them, Step 3b
gains an OMITTED settled outcome, Step 2 leaves out declined title and
body problems, and the out-of-scope list names settled items. The
Claude Code and Codex copies get a 2.5f that runs at every level.
A QWP connection's commit batch now holds plain frames of any tables
from one receive pass, interleaved in any order: each table commits
once when the pass drains, instead of a table switch committing the
batch. Frames with several table blocks join too.

The batch commits its tables one by one, in the order of their first
frames, and stops at the first that fails. The ack covers the frames
before the failed table's first frame, which hold rows of committed
tables only; the NACK names that frame. Rows that later frames added
to committed tables are in the WAL and arrive again on the client's
replay: a partial failure duplicates, it never loses.

A frame that appended rows next to the batch's and then fails, or then
meets a block that cannot join, leaves rows no writer can drop alone.
The processor then rolls the batch back and redoes its frames one at a
time from the receive buffer, each committed on its own, and answers
the frame as it would be answered without a batch. This replaces the
salvage branches: a frame that fails before touching any writer is a
boundary, and the batch commits before its answer.

A frame that redefines symbol dictionary ids can only open a batch:
redoing the frames before it would read the new strings.

Each table now keeps its own schema parse (QwpTableBlockCursor, keyed
by table name, bounded by the per-connection table limit) and its own
Phase 1 column mapping (on its WalTableUpdateDetails). The joinability
check is a cache hit, so the single-entry caches made every table
switch a batch boundary and every frame allocate.

QwpIngressMultiTableBenchmark measures one connection's ingestion
across 1-100 interleaved tables, with a raw pre-encoded client that
measures the server and with the Java client.

Tests that pinned single-table batching now pin the new contract;
the ack contract fuzz writes three tables, frames of two tables and
frames whose second block fails, and accepts the partial-failure
duplicates only where they are allowed.
The ack contract fuzz writes frames of two tables, so it now reaches a
group-closing frame of two tables whose own commitAll commits one table
and fails on the other, as on master. Its I2 invariant accepted such
duplicates only when the NACK named the frame after the ack; a rowless
deferred frame between them holds the ack lower. The invariant now asks
for any NACK above the ack, or a CLOSE, and takes the first frame with
rows after the ack as the one whose failed table committed nothing.

QwpIngressOracleFuzzTest gains a multi-table variant: producers cycle
their rows through three DEDUP tables while the server bounces, so the
real client's replays meet batches that span tables, and every table
must end up with exactly its rows.

A new test drives the live read-only gate's check before a frame opens
a batch, which only the fuzz reached so far.

The redo flag is set as the last statement that prepares the redo, so
the catch-all needs not reset it, and the block cursor counts every
cached schema section as retained, as an entry always holds its bytes.
@jerrinot jerrinot added Performance Performance improvements Core Related to storage, data type, etc. labels Nov 3, 2026
@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@azure-pipelines

azure-pipelines Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
There may be pipelines that require an authorized user to comment /azp run to run.

Since #7917, level-3 reviews take about four times as long, and they
raise findings that collapse as soon as a developer challenges them.
On this machine, 63 level-3 reviews in October took a median of 45
minutes; five on Nov 2 took 113 to 194. The main model's own time
did not change: the rest was waiting on agents.

#7917 gave discovery agents a shell, max thinking and two hours, so
they proved what they suggested by building, starting servers and
running stress loops, and falsifiers then proved it again. Discovery
goes back to the builtin reviewer, which reads code at high thinking
within the 30-minute default. Only falsifiers run code, with a
one-hour cap and a 30-minute budget in their task. Earlier findings
and slowdowns the PR body states go to falsifiers as candidates,
since discovery no longer runs anything.

The risk round, meant to let unlikely defects ship, escalated
instead: the assessor took "the worst consequence the defect can
reach", and the parent could "never make a rating rarer or milder".
A proved 160 MB allocation during restore became an out-of-memory
failure at startup and a Critical on #8166, until a challenge showed
that peak memory did not change. The assessor now rates the
consequence the evidence shows, and first tests the finding against
the author's strongest reply, such as a cost the server already pays
in normal operation; a reply that holds removes the finding. The
parent corrects ratings in both directions, and a Blocking finding
shows the reply it answers. fix-pr drops a bug whose reply holds and
lists it for the user.
@jerrinot

jerrinot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Review (level 1)

Verdict: request changes, for one blocking finding.

The raw-replay speedups reproduce: multi-table one-row frames ingest 5.7-8.4x faster, and no listed case got slower. With 100_000 alternating one-row frames, each table gets 14 WAL transactions instead of 50_000.

Title and description

  • Title: OK.
  • Labels: OK.
  • Body:
    • Bullet 3 understates its own regression: DEDUP only absorbs duplicates in tables that have it. Suggested:
      A commit failing on one table can duplicate other tables' rows unless they use DEDUP.
    • The "Why the Java client is slower, and how to avoid it" section is investigation output (packet rates, kernel coalescing). It also contradicts the table: it says the Rust client "gains 4-5.6x", but the 100-table row shows 0.98M -> 0.98M. Drop the section and keep the workaround in the tradeoff bullet:
      Java client sending one-row frames across tables: up to 2.5x slower; raise auto_flush_rows.
    • The conditions line and the table can stay.

Blocking

B1. The per-table schema cache keeps parsed schemas after disconnect, up to 32,768 per decoder

  • Problem: The new schema cache is never released, and clearing it frees nothing.
  • Net impact: Heap grows with every distinct table name written and stays until restart.
  • Evidence:
    • At 8123fc582e, a real HttpServer with the pinned Sender: 3 successive connections writing 40, 80 and 120 distinct tables leave 40, 80 and 120 live QwpSchema objects in one pooled connection state. At base 24293a37ee it is 1.
    • A standalone decoder at the default capacity, fed 400_000 names, ends with 32_768 live entries. At base it is 1.
  • Risk: High.
  • Population: QWP ingestion that writes many distinct table names over pooled connections. Also the QWP UDP receiver, whose decoder lives as long as the server.
  • Magnitude:
    • About 0.65 / 1.5 / 6.7 KB per table name for 3 / 10 / 50-column schemas.
    • Up to about 218 MB per decoder at 50 columns, more for wider schemas.
    • Pooled contexts keep it: up to 8 idle per network worker, never trimmed.
    • On a live server, the entry count went 25 -> 524 -> 1,024 across two connections that wrote 500 tables each, and stayed after those tables were dropped.
    • Base keeps one parsed schema per decoder.
  • Location (in-diff):
    • QwpTableBlockCursor.java:72 and :503-523 create one entry per distinct table-name byte string. ASCII case variants of one table count separately.
    • The capacity is qwp.max.tables.per.connection, 10_000 (QwpIngressProcessorState.java:379-382, QwpUdpReceiver.java:172).
    • The capacity reset (:515) and the parse-error release (:471) call Utf8SequenceObjHashMap.clear(). It resets only the keys and leaves values[] populated, so up to keys.length (32_768) entries stay reachable. The parse-error path no longer frees the wide schema as base's releaseCachedResources() did.
    • QwpIngressProcessorState.onDisconnected() (:1479-1482) only calls streamingDecoder.reset(), which leaves the cache alone, and the state stays on the pooled context.
  • Suggested fix:
    • Bound the cache by retained bytes, or by a small fixed count, independent of qwp.max.tables.per.connection. A decoder that never disconnects, like the UDP receiver's, must also stay bounded, so a disconnect hook alone is not enough.
    • Make clearing real: allocate a new map, or null its values.
    • Drop the cache in onDisconnected().
    • getRetainedCacheBytes() walks only the keys, so the retention tests cannot see this. Count the entries that are actually reachable there.

Moderate

M1. The new oracle fuzz test times out for some seeds

  • Problem: QwpIngressOracleFuzzTest#testOracleMultiTableTortureUnderServerBounces hits its 120 s client drain timeout for some seeds.
  • Net impact: CI flakes.
  • Evidence:
    • At 8123fc582e on an idle machine (ext4 NVMe), -Dfuzz.s0=4551 -Dfuzz.s1=36039382 ran 121.7 s and failed with close() drain timed out after 120000 ms [targetFsn=695, ackedFsn=609]. Other seeds took 70-78 s.
    • In 10 runs on a loaded machine, 5 timed out.
    • The failing seed passes in 4.7 s on tmpfs.
  • Cause: the suite's default commit mode is ADAPTIVE (Overrides.TEST_COMMIT_MODE), so every small commit runs fdatasync, and small receive chunks turn into many tiny commits.
  • Fix: run the test under nosync, bound its data volume or chunk sizes, or raise close_flush_timeout_millis.

M2. The same test never exercises a failed batch commit, despite its Javadoc

  • Problem: The Javadoc (QwpIngressOracleFuzzTest.java:440-447) says a batch that committed some tables before a failure is replayed and DEDUP absorbs the duplicates. The test never reaches that path.
  • Net impact: The partial-failure tradeoff has no end-to-end test with the real client.
  • Evidence:
    • Structurally, the path can't happen in this test: one worker, stop() halts only between loop iterations, a batch never outlives a receive pass, and no commit fault is configured.
    • 10 runs over 7 seeds logged no commit batch error and no WebSocket commit batch failed line. A control test does log them.
  • Fix: inject a one-shot commit failure on one table mid-stream, as denyCommitTo does in QwpIngressCommitBatchTest, and check the DEDUP oracle. Or correct the Javadoc.

Minor

  • QwpIngressCommitBatchTest.java:2155-2168: the Javadoc of assertNothingFollowsRoleChangeClose now sits stacked above assertMissingTableWithoutAutoCreate's own Javadoc, and the method at :2206 has none.

Coverage

  • Measured at 8123fc582e with the 6 changed test classes (289 tests, all green): 311/314 changed lines and 154/154 branches run.
  • Never run:
    • QwpMessageCursor.java:83-84: the delegating constructor, which has no production caller.
    • QwpUdpReceiver.java:172: the constructor argument, which the UDP suites reach.
  • No coverage gap blocks.

Summary

  • Verdict: request changes (B1).
  • Tradeoffs the PR makes on purpose:
    • Tables without DEDUP get duplicate rows when a batch commit fails partway.
    • The Java client is up to 2.5x slower on one-row frames across tables. With the pinned client (28c84830) on a shared machine I measured 2.75x on 2 and 10 tables, which is consistent.
  • Submodules: no pointer moves.
  • Findings: Blocking 1, Moderate 2, Minor 1.

jerrinot and others added 7 commits November 3, 2026 22:48
Review finding B1: the per-table schema cache in QwpTableBlockCursor
outlived its tables. Utf8SequenceObjHashMap.clear() leaves the values
array populated, so neither the reset at capacity nor the release after
a parse error freed anything, a disconnect left the cache alone on the
pooled connection state, and the UDP receiver's cursor lives as long as
the server. Up to 32,768 parses could stay reachable per decoder.

A table's last schema parse now lives on its WalTableUpdateDetails, like
its Phase 1 column mapping. QwpTudCache implements a new QwpSchemaCache
for the block cursor, so a parse is kept only while the connection holds
its table open, is bounded by the per-connection table limit like the
table's writer, and goes when the entry is freed: on disconnect, on
eviction, or when the cache distresses. Every spelling of a table name
shares one parse. A closed entry drops its QWP caches, since the table
map's clear() keeps freed entries reachable too. The block cursor keeps
no parse itself; without a cache it parses every block.

A table's first block on a connection parses before its entry exists.
The appender then leaves that parse in the new entry, so the table's
next block finds the object its column mapping was resolved for and
keeps joining the commit batch.

The schema lookup and the commit batch lookup ask the table cache for
the same table one after the other, so the cache checks the entry it
found last before hashing; the batch's own last-joined shortcut goes.

QwpParserCacheRetentionTest pinned that the block cursor retains the
schema section; it now pins that the table's entry does and the cursor
does not.

Also from the review:
- M1: the multi-table oracle fuzz runs without fsync, which made a seed
  outlast the client's drain timeout on disk.
- M2: its Javadoc no longer claims to cover a batch commit that fails
  partway; nothing in it fails a commit.
- A misplaced Javadoc in QwpIngressCommitBatchTest moves back to its
  method.
- A new test covers tables whose batch position is stale from an earlier
  pass, or that a rowless block opened.
An OK ack names every table its frames committed, and a durable ack
every table whose frontier moved. With a lowered http.send.buffer.size
either frame could outgrow the send buffer, and trySendAck and
trySendDurableAckFrame then dropped the connection after the rows had
committed. The client replayed them on reconnect, the next pass failed
the same way, and tables without DEDUP kept gaining copies. The commit
batch made this easy to reach: one ack now covers every table a receive
pass wrote, not the tables of at most 64 frames. Base already failed
the same way for a single frame carrying enough tables.

getFrameAddress now writes such a frame to staging memory on the
connection state, and sendStagedFrame hands it to the send buffer one
buffer at a time. A blocked send parks the send state as before; the
send buffer keeps the unsent part of its piece and the rest stays
staged. resumeSend and drainPendingResponse finish the staged frame
before their per-state continuation, so no other frame can interleave
with it. The frame bytes on the wire are unchanged, so clients see the
same single ack. The staging memory grows to the largest frame and is
freed when the connection ends; if it cannot be allocated, the
connection drops as before.

QwpIngressUpgradeProcessorResumeRecvTest#testAckBufferTooSmall asserted
the old disconnect for an ack larger than a 10-byte buffer. It now
asserts that the ack goes out in pieces of 10 and 3 bytes, which is the
behavior this change introduces.

QwpIngressOversizeAckTest drives the pinned client against a live server
with a 1 KiB send buffer, for OK acks of one frame and of many frames
per pass, and for local durable acks. QwpIngressStagedAckFrameTest pins
piece sizes around the buffer size, resumption after blocked pieces, an
error queued behind a staged ack, teardown, a peer lost mid-frame,
multi-byte table names and allocation failure. The multi-table oracle
fuzz now uses up to 26 tables and lowers the send buffer in half of its
runs.
Rename getFrameAddress to reserveFrame: it allocates staging memory,
and sendFrame relies on that, which its javadoc now says. Drop a reset
in stageFrame that its precondition already guarantees, split its
assert so the message matches the condition, and stop restating the
staging memory's lifetime in three places.

In the tests, Connection takes its frames up front and lets
PhasedNetworkFacade deliver them instead of overriding it, stageAck
picks name lengths in one expression, the frame-size test checks
reuse of the staging memory for every frame no larger than one
already staged instead of two hard-coded sizes, and
RestartableQwpServer no longer exposes a -1 sentinel named like a
buffer size.
Master's #7936 returns a quiet connection's idle WAL writers to the pool
on a keepalive PING: QwpTudCache.releaseIdleWriters() removes and frees
every cache entry without uncommitted rows. This branch added
lastCachedTud, the entry getCachedTableUpdateDetails() checks before the
hash lookup, and treats an entry without a writer as freed.

The textual conflicts kept both sides: the distressed clear() and
reset() forget lastCachedTud and the released tables; the processor
imports both Vect and MicrosecondClock; and QwpIngressCommitBatchTest
drops its own readLeShort and okAckSeqTxn, which master moved into
QwpIngressWireHarness, and keeps this branch's symbolFrame.

One conflict had no markers. releaseIdleWriters() swallows a failing
writer close, such as the close of a table dropped while the connection
idled, and that failure leaves the writer set on the released entry.
lastCachedTud then kept serving it: a second frame for the re-created
table found no batch entry for the stale TUD, took the table as outside
the batch while its new writer held the first frame's rows, and the
frame was refused. releaseIdleWriters() now forgets lastCachedTud when
it releases that entry.
QwpIngressIdleWriterReleaseTest#testIdleReleaseFailureDoesNotServeReleasedEntry
covers it and fails without the reset; the failing-msync facade it
shares with testIdleReleaseSurvivesWriterCloseFailure moves into a
helper.
getCachedTableUpdateDetails() checks lastCachedTud before the hash
lookup and took an entry without a writer as freed. A free that fails
leaves the writer set: evictStaleTud() swallows that failure, and the
failed-lookup path in getTableUpdateDetails() only rethrows it without
distressing the cache. The shortcut then kept serving the removed entry
for its name. When a stale entry's writer failed to close, the next two
frames of the re-created table in one pass found no batch entry for
the stale TUD; the second took the table as outside the batch while
its new writer held the first frame's rows, and the frame was refused.

Every site that removes an entry now calls onEntryRemoved(), which
updates the table count and forgets lastCachedTud, so lastCachedTud is
always null or a mapped entry and the writer check goes.

QwpIngressCommitBatchTest#testRecreatedTableAfterFailedEvictionSharesOneBatch
reproduces the refusal through a failing close of the evicted writer.
QwpIngressProcessorStateTest#testRemovedEntryIsNotServedAgain drives
each of the seven removal paths with the removed entry found last and
with another entry found last; reverting any single call site to the
old count update fails it.
Brings in the review-pr, fix-pr, review-prober and risk-assessor
changes of #8165 (head dde1223), so that reviews
on this branch use them: model name in the report, the author's
reply in risk ratings, no test demands for exotic paths, settled
items stay settled, and PR bodies carry the approach.
A review names its level and its model, but not the version of the
review-pr skill that wrote it. The skill changes often, so a reader
cannot tell which rules produced a posted review, or compare the
reviews that different versions of the skill write.

Each copy of review-pr (pi, Claude Code and Codex) now states its
version under its title, starting at 4.0, and asks for it on the
opening line, after the level and the model. Step 4 makes that line
the first line of the report, so the console output, fix-pr's
report.md and the reviews that approve-pr and reject-pr post all
carry it.

The version sits in the body, not in the frontmatter: pi strips the
frontmatter when it expands /skill:review-pr, so the model would
never see it there. Each PR that changes a copy raises its version.
@jerrinot

jerrinot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

Review (level 3)

Reviewed at level 3 with anthropic/claude-opus-5-5, thinking xhigh. Revisions: head db74b5696d, base 1c9aef6bd5. No submodule pointer moves.

Verdict: request changes, for one blocking finding (B1): when one table's commit fails, the rows of the other tables in that receive pass are stored twice.

Title and description

  • Title: OK.
  • Labels: OK.
  • Body:
    • The duplication tradeoff bullet doesn't say how many rows it duplicates. Measured: every row the other tables got for the rest of the receive pass, up to 2 MiB of frames (see B1).
    • The "Why the Java client is slower, and how to avoid it" section is investigation output (packet rates, kernel coalescing) and contains a commit hash (a6fd07f1). Keep the workaround in the tradeoff bullet and give the reason in one sentence.
    • There is no approach paragraph for the cross-table batch (including the redo and partial commits) or for the staged ack fix.

Suggested body, assuming B1 ends as a recorded, sized tradeoff:

- One QWP connection sending one-row frames to several tables ingests 4–6x faster.
- Interleaved frames commit once per table per receive pass, cutting WAL transactions.
- Acks larger than `http.send.buffer.size` no longer drop the connection and duplicate rows.
- One table's failed commit duplicates other non-DEDUP tables' rows for the rest of that pass.
- Java client sending one-row frames across tables: up to 2.4x slower; raise `auto_flush_rows`.

The server's commit batch now spans all tables of a receive pass: plain frames append to their tables' WAL writers, and each table commits once when the pass ends. A frame that appended rows next to the batch and then fails or cannot join makes the server roll the batch back and redo its frames one at a time. QuestDB commits each table separately, so when one table's commit fails after others committed, the ack stops before that table's first frame and the client's replay sends the committed tables' later rows again.

An ack names every table it covers and could outgrow the send buffer, which dropped the connection. The server now stages such a frame in connection memory and sends it in send-buffer-sized pieces.

The Java client writes each frame with its own `send()`. Now that the server drains the socket at once, one-row frames no longer share TCP packets, and the client's I/O thread caps at about 500k frames/s. The Rust client batches ready frames (c-questdb-client#235), and the sender pool (java-questdb-client#108) batches pooled leases' rows.

Rows/s on one connection: server pinned to 12 cores of a Ryzen 9 9950X, tmpfs, loopback, JDK 25. Raw replays pre-encoded frames (`QwpIngressMultiTableBenchmark`, server only); Java is the same benchmark with java-questdb-client#116; Rust is c-questdb-client `main`.

<keep the existing table>

Blocking

B1. A failed table commit duplicates the other tables' rows for the rest of the receive pass

  • Problem: When one table's commit fails, the replay stores other tables' rows twice.
  • Net impact: Tables without DEDUP get up to one receive pass of duplicate rows.
  • Evidence: a probe that models the client's replay, 1,000 alternating a/b one-row frame pairs in one pass:
    • head db74b5696d: 999 duplicate rows in a;
    • base 1c9aef6bd5: 0.
    • Duplicates for N = 1, 2, 10, 100, 1000 pairs: head 0, 1, 9, 99, 999; base 0 for every N. Combined [a,b] frames: head N, base 1.
  • Risk: Critical (Severe impact, Plausible likelihood). The rarest condition is a commit failure on a table that joined the batch after another table that commits. These all cause it:
    • another session runs DROP TABLE on a table under ingestion (executed: 9 duplicates at N=10 at head; at base no failure and no duplicates);
    • a demote lands between two tables' commits (the PR's own testDemoteAtSecondTableCommitAcksFirstTableBeforeClose);
    • an I/O error hits one table's commit.
  • Why "the body states it, and QWP is at-least-once" does not settle it:
    • The body gives no size.
    • Base duplicates nothing for the same trigger. Base's testCommitFailureOnOneTableCommitsNoLaterFramesOfAnother promised "the replay duplicates nothing"; head renamed it and now expects a duplicate.
    • A concurrent DROP that base absorbed with no error now causes a NACK plus duplicates.
    • The pinned client's design/qwp-nack-policy-v2.md states the invariant "NACK ⇒ batch atomically not applied server-side … violating it trades loss for duplication". The client treats status 0x09 as RETRIABLE and replays from ackedFsn+1.
  • Net determination:
    • Population: QWP connections writing two or more non-DEDUP tables per pass. Auto-created tables have no DEDUP.
    • Delta versus base: N−1 duplicates instead of 0 for alternating one-table frames; N instead of 1 for combined [a,b] frames.
    • Offsets: only DEDUP.
    • Net: negative.
  • Location (in-diff):
    • QwpIngressProcessorState.java:610-635 (commitBatch commits tables one by one in join order);
    • :2162-2170 (onCommitBatchFailed sets the watermark to the failed table's first frame − 1);
    • QwpIngressUpgradeProcessor.java:1806-1817, 2027-2046 (the NACK names that frame).
  • Trigger and symptom: A client streams one-row frames to a and b, and b's commit fails mid-pass (for example a concurrent DROP). After the automatic replay, a holds every row from the rest of that pass twice.
  • Base for the same trigger: a base batch holds a single table. a's batch commits at b's first frame, each b frame commits alone, and the replay adds every a row once. With the concurrent DROP, base sees no failure at all.
  • Suggested fix:
    1. Narrow the triggers. Decide read-only once for the whole batch, for example by holding the role-switch read lock across the table loop (check that it is reentrant), so a demote can't land between tables. Before committing anything, check every entry's tracker (dropped, structure version), so a DROP fails the whole batch before any table commits, as base does.
    2. For what remains (I/O faults on one table), record it as a decision: state the size in the body and amend the client's NACK invariant.

Moderate

M1. The redo's per-frame record keeps its peak size for the life of the connection context

  • Problem: commitBatchFrames never shrinks after a large receive pass.

  • Net impact: 1-4 MiB of heap per connection context, kept after the connection closes. Base keeps none.

  • Evidence: probe at db74b5696d, after one full 2 MiB pass and onConnectionClosed:

    Frames Heap kept per context
    Pinned-client one-row frames, 50 B 1,048,592 B
    47 B frames 2,097,168 B
    Zero-table frames, 18 B 4,194,320 B

    redoFrames keeps another 1 MiB per selector after a redo.

  • Location: QwpIngressProcessorState.java:155, 786-788, 801; QwpIngressUpgradeProcessor.java:231, 1844. LongList.clear() keeps the capacity, and the pool keeps up to 8 contexts per thread, plus every live connection.

  • Fix: a batch's frames are consecutive binary frames in the receive buffer. Re-walk them from the first frame's offset, using the first sequence and a frame count (constant state). At least restore the initial capacity in onDisconnected().

M2. The zero-allocation test doesn't catch a 16-byte allocation per frame

  • Problem: testInterleavedTablesAllocateNothingOnceWarm allows 21 bytes per frame.
  • Net impact: a per-row GC allocation on the batch path would go unnoticed.
  • Evidence: a mutation storing new Object() once per joinCommitBatch gave 55,080 B, under the 65,536 B bound, and the test passed. A 24 B allocation fails at 79,184 B. The baseline is about 7.1 KB.
  • Location: QwpIngressProcessorStateTest.java:286.
  • Fix: a bound of about 32 KiB for 3,000 frames, or measure 30,000 frames (baseline about 17 KB).

M3. A writer close that throws leaves the table's schema parse reachable

  • Problem: close() nulls the QWP caches only after super.close() succeeds.
  • Net impact: about 150 B per table plus its parse, kept after an I/O error. Nothing breaks for the user.
  • Evidence: 40 dropped tables whose writer close fails on msync: after onDisconnected(), 40 of 40 parses were still reachable, while getCachedSchemaCount() reported 0.
  • Location: WalTableUpdateDetails.java:114-118.
  • Fix: null the caches in a finally. Have the test hook count reachable values, not keys.

Minor

  • QwpIngressCommitBatchTest.java:1229-1235: the Javadoc of testRedefiningSymbolDictionaryEndsCommitBatch sits stacked above the one at :1236-1241, and that test has none.
  • QwpIngressProcessorStateTest.java:1077-1079: the comment that belongs to testZeroRowDeferredFrameIsNeverAckCovered now sits above testWideSchemaFramesJoinCommitBatch.

Adjacent (not blocking, already on base)

  • QwpTudCache.reset() leaves freed table entries in the map's value array, because AbstractLowerCaseUtf8SequenceHashSet.clear() blanks only the keys. Closed entries, and the closed writer when its close threw, stay on the heap until their slot is reused: 174 closed entries after three connections. Fix: null values[] in clear(). Minor.

Coverage

  • Test gate: passes. No coverage gaps admitted.
  • Measured: every changed line and branch ran (399/399 lines, 198/198 branches): 12 test classes, 452 tests, all passing at db74b5696d.

Summary

  • Verdict: request changes (B1).
  • Earlier review (level 1, at 8123fc582e):
    • The schema-cache retention (old B1) no longer reproduces: 0 parses remain after the 40-, 80- and 120-table connections. Only M3 is left of it.
    • The fuzz test's timing-out seed now passes in 7.9 s.
    • The two Javadoc items are fixed; the one under Minor above is new.
  • Tradeoffs the PR makes on purpose: the Java-client slowdown (measured in the body; not measured again here) and the cross-table duplication (B1).
  • Enterprise: compile questdb-enterprise against this head. The PR changes signatures it may call: appendToWalStreaming now takes WalTableUpdateDetails, isSchemaSettled has a new signature, and the old single-table batch methods are gone.
  • Submodules: no pointer moves.
  • Findings: Blocking 1 (Critical, in the diff), Moderate 3, Minor 2.

A QWP table entry whose writer close threw kept its column mapping and
its schema slot with the parse. WalTableUpdateDetails.close() cleared
the two fields only after super.close() returned, and closing the
writer of a dropped or distressed table does file I/O (rollback,
msync) that can throw out of TableUpdateDetails.closeNoLock(). Every
QWP free path goes through this close(), and a freed entry can stay
reachable, for example from a table map that does not clear its values.

close() now clears the fields in a finally around super.close(). The
writer still closes first, and its failure reaches the caller
unchanged: reset(), clear(), releaseIdleWriters() and the UDP commit
loops still log it, commitAll() and commitIfMaxUncommittedRowsReached()
still propagate it, and a failed lookup still adds it to its own
failure as suppressed.

The new QwpIngressProcessorStateTest test
testEntryFreedByFailingWriterCloseKeepsNoQwpCache frees an entry
through each of the nine free paths while msync fails. It asserts the
failure each path reports, that the freed entry holds no QWP cache,
and that the same cache then writes and commits the re-created table.
The QWP and ILP/HTTP table caches free every table entry when a
connection ends or a cache is distressed, then clear their table map.
LowerCaseUtf8SequenceObjHashMap.clear() blanked only the keys and the
key list, so its value array kept every freed entry reachable, with
the closed writer of an entry whose writer close threw, until a later
table name took the same slot or the map grew. Both caches live in the
pooled HTTP connection context, so the heap kept the entries of past
connections.

clear() now also fills the value array with null, as erase() does for
one removal and as IntObjHashMap.clear() and
LowerCaseCharSequenceObjHashMap.clear() already do. The fill allocates
nothing and keeps the arrays. No lookup reads a value through a blank
key, so every lookup returns what it returned before.

LowerCaseUtf8SequenceObjHashMapTest#testClearReleasesValues asserts,
for maps of 0 to 10,000 keys, after removals and on reuse, that no slot
that held a key still holds a value after clear().
testClearAllocatesNothing asserts that clearing a map grown to 10,000
keys allocates nothing. testResetAndDistressedClearReleaseFreedEntries,
in QwpIngressProcessorStateTest and in the new LineHttpTudCacheTest,
frees 17 entries through reset() and through the distressed clear(),
then asserts through weak references and full collections that none
stays reachable. The QWP test also fails every writer close and asserts
that the closed writers do not stay reachable either.
testInterleavedTablesAllocateNothingOnceWarm measured 3,000 frames
against a 64 KiB bound, which leaves 21.8 bytes per frame. The
smallest Java object takes 16 bytes, so a frame path that allocated
one object per frame still passed: with a new Object() stored on every
joinCommitBatch() call, the window allocated 55,392 bytes.

The bound has to absorb one-off JIT work that HotSpot charges to the
test thread when it queues a method for C2, 5 to 21 KB in the runs
measured, and that work does not grow with the frame count. A
per-frame allocation does, so the test now measures 30,000 frames
against the same bound: one 16-byte object per frame adds 480,000
bytes. The warm-up grows to 3,000 frames and the batch commits every
300 frames instead of every 30, so the window keeps its 100 batch
commits and the test makes no more WAL commits than before. Under the
ADAPTIVE commit mode each WAL commit fdatasyncs, so the commit count
sets the test's duration: about 2.0 s, against 1.6 s before. Each
table now ends with 11,000 rows.

Mutations show the test fails: a new Object() on every
joinCommitBatch() call fails it at 487,344 bytes, on every
prepareMessage() call at 486,504 bytes, and a 1,000-byte array on
every commitBatch() call at 107,616 bytes. Unmutated, the window
allocates 5,680 bytes under ADAPTIVE and 8,344 bytes under NOSYNC,
and 19,192 and 21,080 bytes under the JaCoCo agent.
Two earlier commits on this branch each added a test, with its
Javadoc, between an existing comment and the test that comment
describes, which left each comment above the wrong test.

In QwpIngressCommitBatchTest the Javadoc of
testRedefiningSymbolDictionaryEndsCommitBatch sat stacked on top of
the Javadoc of testRecreatedTableAfterFailedEvictionSharesOneBatch.
It now sits directly above its own test, which had no comment.

In QwpIngressProcessorStateTest the comment of
testZeroRowDeferredFrameIsNeverAckCovered sat above the Javadoc of
testWideSchemaFramesJoinCommitBatch. It now sits directly above its
own test again, as on the base branch.

The move keeps the wording of both comments and changes no code.
When a frame failed or could not join after a run of batched frames,
the server rolled the batch back and redid its frames one at a time,
committing each as its own WAL transaction. A pass of K one-row frames
followed by one bad frame turned one commit into K: at K=5,000 the I/O
worker stalled for 31 s under the sync commit mode (base: 0.06 s). A
first block that failed without writing a row took the same redo, and
an unexpected error looking up a table new to the batch distressed the
table cache and refused the whole batch at its first, good frame.

joinCommitBatch() now answers BOUNDARY exactly when the batch can
commit as it stands: no row of the frame sits in a writer, the table
cache is healthy, no batch writer is left mid columnar write, and the
frame's first block is still prepared. Every other frame that fails or
cannot join is a REDO. A frame that failed in its preparation is
classified the same way, and the redefined-symbol-dictionary check runs
first, so a redo never re-decodes frames against redefined ids.
redoCommitBatch() now joins the batch's frames again and commits each
table once; a frame that no longer joins ends that batch, commits what
joined, and goes on its own, and the frames after it start a new batch.
prepareMessage() gains a documented test hook, onSymbolDictApplied().

Existing expectations that change, because the redo now commits each
table once instead of once per frame: testRedoAnswersFrameThatFailed-
WhileJoining and testMissingTableAfterAppendRedoesBatch expect a at
aBefore + 1 instead of aBefore + 2, and tests that expected the batch's
first frame to be NACKed after an unexpected lookup error now expect
the batch to commit and the failing frame to be NACKed, as on base.

Tests: QwpIngressCommitBatchTest covers failing first and later blocks
of every stage (lookup, Phase 1, append) for LONG, timestamp, VARCHAR,
STRING, BINARY, DOUBLE[] and SYMBOL columns; distress and columnar-
write faults; a real ALTER, DROP and demote during the redo; durable
ack seqTxns; sync and nosync; the commit-batch ack-contract fuzz gains
an invariant that bounds each table's txn count. QwpIngressProcessor-
StateTest, QwpAckSeqTxnCoverageBlackBoxTest, QwpIngressUpgradeProcessor-
ResumeRecvTest and QwpIngressOracleFuzzTest are adjusted to match.
QuestDB commits each table separately, so a commit batch that held rows
of several tables could commit some of them and then fail on another.
The ack then stopped before the failed table's first frame, and the
client's replay stored the committed tables' later rows a second time:
for 1,000 alternating frame pairs of a and b, 999 duplicate rows in a,
against none on base. A concurrent DROP TABLE of a batch table, a
demote landing between two tables' commits, or a refused insert
permission all produced it.

commitBatch() now takes the role-switch read lock once and holds it
across every table's commit, so a demote can no longer fall between two
tables. Before the first commit it checks every table the way a commit
would refuse it: the node is writable, no table was dropped (the
table's sequencer txn tracker), and the table's insert permission holds
(TableUpdateDetails.authorizeCommit(), now public). Any refusal there
refuses the whole batch with nothing committed, and the NACK names the
batch's first frame, so the replay sends no row twice. What can still
fail between two tables is a commit that hits a storage fault; that
case keeps the existing partial-commit handling.

Changed expectations, because these triggers now refuse the whole
batch: testDroppedTableStopsBatchCommitAtIt becomes
testDroppedTableRefusesWholeBatch (a commits nothing, NACK at frame 0),
and testDemoteAtSecondTableCommitAcksFirstTableBeforeClose becomes
testDemoteAtBatchCommitCommitsNoTable (only the CLOSE, no table
commits). The tests of the remaining partial-commit path
(testCommitFailureOnLaterTableAcksFramesBeforeItsFirstFrame,
assertCommitFailureAtTable, QwpIngressUpgradeProcessorResumeRecvTest's
testBoundaryFailureResumesPrefixAckBeforeErrorWithoutAppendingTail) now
fail the table's own commit after the batch's checks passed, standing
in for a storage fault, and keep their expectations. denyCommitsAfter-
First now denies once the table has committed a transaction, instead of
counting insert checks, which a batch now runs twice per table.

New: testPermissionRefusalOfAnyTableCommitsNoTable refuses each of
three tables in turn. Without the checks, it and
testDroppedTableRefusesWholeBatch fail; without the lock held across
the loop, testDemoteAtBatchCommitCommitsNoTable fails.
A commit batch keeps three longs per frame so that a redo can find the
batch's frames in the receive buffer again. The record grows with the
largest receive pass a connection processes: a 2 MiB pass of one-row
frames from the Java client takes 1 MiB of heap, of 18-byte frames
4 MiB. LongList.clear() keeps the capacity, and the record lives on the
connection's QwpIngressProcessorState, which the pooled HTTP connection
context keeps after the connection ends. The upgrade processor's copy
for a redo kept its size on the worker's processor the same way.

onDisconnected() now restores the record's initial capacity, three
longs for 64 frames, when it grew past it, and redoCommitBatch()
restores its copy once the redo ends. A live connection still holds the
record at the size of its largest pass, as it holds its receive buffer.

Tests: testFrameRecordShrinksWhenConnectionEnds grows the record with a
300-frame batch, checks onDisconnected() shrinks it, and that the
pooled state then takes a new connection; testFullReceivePassRedo-
CommitsEachTableOnce now checks the processor's copy shrinks after a
24,000-frame redo. Both fail without the trims (1,536 and 72,000 longs).
commitBatch() skips the role-switch lock and the batch checks when the
batch holds no table: a run of frames that name no table. No test sent
only such frames, so that branch never ran. The new test sends three
zero-table frames in one receive pass and checks that a single ack
covers all three and no frame gets an error response.
When a frame cannot join the commit batch, handleBinaryMessage commits
the batch first. If a table that joined after the batch's first frame
fails its commit there, the NACK must name that table's first frame,
where the client's replay starts, and the ack must stop just before it.
No deterministic test reached that line with the failed table being
other than the batch's first: the existing partial-failure tests commit
at the end of the receive pass, through commitPendingBatch.

testBoundaryCommitFailureOnLaterTableNacksItsFirstFrame sends a, b, a
and a frame that needs a new column on a, and makes b's commit fail
after a's. It fails when the NACK names the batch's first frame.

The commit also moves the javadoc of
testClientClosePipelinedBeforeRoleChangeCloseIsNotTheEcho back onto that
test; it had ended up above testCaseVariantBlocksWithDifferentColumns-
JoinCommitBatch.
The javadoc of assertBatchLostToReplacement had ended up on the new
assertLookupEvictsDistressedWriter, and the SameParameterValue
suppression of assertEmptySymbolDiff on the new
assertCommitRefusalCheckMatchesCommit, when the new helpers were
inserted above them.
A commit batch held the role-switch read lock across every table's
commit, and checked every table's sequencer before the first commit.
A demote then waited for the whole batch, 2.7 s for 1,000 tables
under adaptive commit, while every other write on the node waited
behind it. Every batch also paid two sequencer lookups and lock
round trips per table, for refusals that are rare.

commitBatch() now takes the lock once and reads the node's role once
under it. Between two tables' commits it hands the lock to a pending
role switch and reads the role again after it. A demote now waits
for one table's commit. One that lands mid-batch stops the commits
as a storage fault does: the replay can duplicate rows of the tables
that committed, bounded by one receive pass.

The tables commit through the new
TableUpdateDetails.commitUnderRoleSwitchFence(), which neither
re-enters the lock nor reads the role again. The pre-commit refusal
loop is gone: a DROP, RENAME, SUSPEND WAL or REVOKE that lands while
a table waits in the batch fails that table's commit instead. The
check as a table joins reads the table's cached sequencer tracker
and a rename flag that the cache lookup sets, instead of opening the
sequencer, so TableSequencerAPI.throwIfCommitRefused() goes away.
The batch's integrity assert checks the entry's slot, which closing
an entry clears, instead of a hash lookup per table.
A commit batch handed the role-switch read lock to a pending role
switch between two tables' commits. A demote that landed there left
the earlier tables committed, while frames past the next table's first
frame still carried their rows: alternating and multi-table frames put
a table's rows on both sides of that frame. The ack stopped before it,
and the client's replay stored those rows a second time. With N
alternating one-row frames of two tables, N - 1 rows of the first
table came back twice, where master stores none twice.

commitBatchEntries() now holds the read lock across every table's
commit and reads the node's role once, so a role switch lands before
the batch or after it, never between two tables. A read-only node
still refuses the whole batch with nothing committed. A demote now
waits for each connection's batch in flight instead of one table's
commit.

CairoEngine.hasPendingRoleSwitchWriter() and
RoleSwitchReadWriteLock.hasPendingWriter() lose their only caller and
go away. The authorizeCommit() Javadoc no longer mentions the
pre-commit check of every table, which an earlier commit removed.

RoleSwitchingEngine in QwpIngressCommitBatchTest queues a real role
switch on the write lock as a batch reads the node's role, before any
table commits. Three tests check, for interleaved and multi-table
frames, that the switch gets the lock only after the last table, that
one ack covers the batch, and that after a demote the replay of every
frame above the ack stores each row once. They replace the tests of
the hand-over.
@jerrinot

jerrinot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Reviewing PR #8195 at level 3 with anthropic/claude-opus-5-5, thinking xhigh, review-pr 4.0.
Revisions: head cfb010fb48, base 2bb61a061b, java-questdb-client pointer 38087d56.

Verdict: approve. Nothing blocks. The body needs one bullet shortened.

Title and description

  • Title: OK.
  • Labels: OK.
  • Body: bullet 6 has 18 words; the limit is 15. Replacement bullets (approach, conditions and table unchanged):
    - One-row frames spread across tables ingest up to 7x faster.
    - Java client from its main branch: one-row frames across tables up to 5x faster.
    - Interleaved frames commit once per table per receive pass, cutting WAL transactions.
    - Acks larger than `http.send.buffer.size` no longer drop the connection and duplicate rows.
    - A failed table commit can duplicate other non-DEDUP tables' rows, up to 2 MiB.
    - QWP over UDP resumes a table's ingestion after a failed WAL segment roll.
    

Blocking

None.

Moderate

None.

Minor

None.

Adjacent findings

None.

Coverage map

  • Test gate: passes. No coverage gaps admitted.
  • Measured at cfb010fb48: 454/454 changed lines and 205/208 branches, from 16 test classes (685 tests, all passing). The 16 are the 15 PR-related classes plus QwpSenderSuspendedTableAlterE2ETest.
  • Branches that never ran, none of them a finding:
    • TableUpdateDetails.java:193, ownSecurityContext == null. The PR only made the method public.
    • QwpIngressProcessorState.java:2264, the error text for a rewind that fails with a non-Cairo exception. It only affects the message.
    • QwpTudCache.java:931, a writer that is not a WalWriter. Unreachable, because the lookup refuses non-WAL tables.
  • QwpWalAppender.java:1144, the per-table mapping when the metadata version changed, is covered. QwpSenderSuspendedTableAlterE2ETest#testRenameColumnOnSuspendedTableBetweenFrames fails if the version term is dropped.

Summary

  • Verdict: approve. Both gates pass.
  • Earlier findings, rechecked at this head:
    • The boundary NACK test now pins its line. Swapping in getFirstCommitBatchSequence() at QwpIngressUpgradeProcessor.java:1812 fails testBoundaryCommitFailureOnLaterTableNacksItsFirstFrame.
    • The design doc was removed and the misplaced Javadocs were moved back.
    • The tests that pin schema-parse retention pass.
    • The double append was settled by the author's reply and is not raised again.
  • Tradeoffs the PR makes on purpose:
    • A partial batch commit can duplicate rows of non-DEDUP tables, up to one receive pass. The body states this.
    • A demote waits for each in-flight batch: 2.4–25 ms per 1,000 tables under the default commit mode.
  • Not re-measured: the speedups were reproduced at 6399ec82fe. The commits since then only remove per-batch sequencer checks and lock round trips, so I expect the numbers to hold or improve, but I have not measured it.
  • Submodule: java-questdb-client is an upstream sync (already on the client's main branch), so it is out of scope.
  • Enterprise: an enterprise checkout (e8438cf6b, not matched to this PR) uses none of the APIs this PR changes or removes.
  • Counts: Blocking 0, Moderate 0, Minor 0.

@jerrinot jerrinot added the QUEUED FOR MERGE Approved PR in the merge queue. Do not merge master into this PR. label Nov 9, 2026
The PR's tests had grown to about 7,200 net lines, much of it fixture
boilerplate repeated per test and multi-paragraph Javadoc that
restated each scenario.

- Each test Javadoc the PR added becomes one line, or goes where the
  test name says it all. The fuzz test keeps its Javadoc: it defines
  the invariants I1-I4 that its assertion messages name.
- QwpIngressCommitBatchTest gains an httpConfig field, createTables()
  and ingestInOnePass() overloads that build the processor. They
  replace 79 local default configs, 54 CREATE TABLE lines and 36
  processor arguments.
- Four pairs of near-identical tests share a helper each: symbol
  redefinition, the every-column-type rewind, the later table's
  commit failure, and a table change between passes that still joins.

No test and no assertion goes. Two merged tests gain the checks of
their twin. The 685 tests of the 16 affected classes pass, and the
PR's production lines keep their coverage: 454/454 lines and 205/208
branches.
@jerrinot

jerrinot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Reviewing PR #8195 at level 3 with anthropic/claude-opus-5-5, thinking xhigh, review-pr 4.0.
Revisions: head d504f33944, base 2bb61a061b, java-questdb-client pointer 38087d56.

Verdict: approve. Nothing blocks. One body bullet is too long, and there are three Minor items.

Title and description

  • Title: OK.
  • Labels: OK.
  • Body: bullet 6 still has 18 words; the limit is 15. Nobody replied when the last review raised this. Replacement bullets (keep the approach, the conditions and the table as they are):
    - One-row frames spread across tables ingest up to 7x faster.
    - Java client from its main branch: one-row frames across tables up to 5x faster.
    - Interleaved frames commit once per table per receive pass, cutting WAL transactions.
    - Acks larger than `http.send.buffer.size` no longer drop the connection and duplicate rows.
    - A failed table commit can duplicate other non-DEDUP tables' rows, up to 2 MiB.
    - QWP over UDP resumes a table's ingestion after a failed WAL segment roll.
    

Blocking

None.

Moderate

None.

Minor

  • QwpIngressCommitBatchTest.java:4027: commit d504f33944 put the Javadoc "{@code ddl} changes b between passes…" above assertSymbolRedefinitionCommitsOnce. It describes assertTableChangeBetweenPassesNacksItsFirstFrame (:4089), which now has no Javadoc.
  • QwpIngressCommitBatchTest.java:196: the static field V_100_TO_109 is declared between instance test methods.
  • QwpIngressUpgradeProcessor.java:1781: state.getErrorText() allocates a String for every frame that cannot join a pending batch, even when that frame did not fail. Base read the error text only for failed frames. This happens once per batch boundary, so it is negligible. Reading it only when isFrameFailed is set would avoid it. Optional.

Adjacent findings

None.

Coverage map

  • Test gate: passes, with 0 admitted coverage gaps.
  • Measured at d504f33944: 454/454 changed lines and 205/208 branches ran, over 16 test classes and 685 tests (0 failures, 2 skipped).
  • Mutation spot-check: the tests caught each of these 7 mutations, applied one at a time:
    • the NACK sequence at :1812;
    • an allocation per joinCommitBatch;
    • the cache reset moved out of the finally;
    • the watermark at onCommitBatchFailed;
    • the < cap check changed to <=;
    • the hasDirtyColumns term in the savepoint rollback;
    • the entry.rows restore in rewindFrame.
  • Branches that never ran, none of them a finding:
    • TableUpdateDetails.java:193, ownSecurityContext == null: low exposure, because the PR only made the method public.
    • QwpIngressProcessorState.java:2264: only the error text for a rewind that fails with a non-Cairo exception.
    • QwpTudCache.java:931, a writer that is not a WalWriter: unreachable, because the lookup refuses non-WAL tables.

Summary

  • Verdict: approve. Both gates pass.
  • Earlier findings, re-run at this head; none reproduce:
    • Schema parses after disconnect: 0 of the 40, 80 and 120 parses stay reachable, counted by heap histogram and weak references, including on a real server with the real client.
    • Fuzz seed 4551/36039382: passed 3 of 3 runs on ext4 NVMe, in 12-15 s each.
    • The tests still catch both the zero-allocation bound and the boundary-NACK mutation after the test refactor.
    • The misplaced Javadoc and annotation from the last review are fixed. The first Minor item is a new one from d504f33944.
    • The double append was settled by the author's reply, so I did not raise it again.
  • Test refactor (d504f33944): every class has the same number of @Test methods as before. I found no lost assertion in the four merged test pairs.
  • Tradeoffs the PR makes on purpose:
    • A partial batch commit can duplicate rows of non-DEDUP tables, up to one receive pass. The body states this.
    • Under the default nosync mode, a demote waits for the in-flight batch: 6 ms at 1,000 tables and 63 ms at 5,000 tables. Base waits for one commit.
  • Submodule: java-questdb-client is an upstream sync (28c84830 -> 38087d56, on main), so it is out of scope.
  • Enterprise: the enterprise checkout (e8438cf6b) uses none of the APIs this PR changes, outside the questdb/ submodule.
  • Findings: all in the diff. Blocking 0, Moderate 0, Minor 3.

…commit_batch

# Conflicts:
#	java-questdb-client
@jerrinot
jerrinot marked this pull request as ready for review November 12, 2026 17:16
handleBinaryMessage read the error text of every frame that left a
pending commit batch, allocating a String even for a frame that did not
fail and so has no error. It now reads the text only for a failed
frame, the only case that uses it.

QwpIngressCommitBatchTest moves V_100_TO_109 into the static field
block and puts the Javadoc of assertTableChangeBetweenPassesNacks-
ItsFirstFrame back on that method, from assertSymbolRedefinition-
CommitsOnce.
CI failed two checks on this branch.

WalSequencingCallSiteTest scans the main sources for ".nextTxn(" and
allows it only in TableSequencerAPI and WalWriterBase. A comment in
QwpIngressProcessorState.throwIfJoinRefused() named
TableSequencerImpl.nextTxn() that way; it now reads "TableSequencerImpl's
nextTxn()".

The formatting check runs the IntelliJ formatter over the changed files
and fails on any change. This commit applies its output to the test
files: one-line Javadocs become three lines, and one long case arm
wraps. Nothing else changes.
testCommitBatchAckContractFuzz failed in CI under nosync: a frame that
closes a deferred group commits its tables on the normal path, and a
demote that lands at its second table's commit leaves the first table's
rows above the ack. A durable-ack connection then defers the
role-change CLOSE until that work is durable, at most for the upload
grace, so the pass sends no NACK and no CLOSE, and the fuzz oracle's I2
check reported that the client is never told to replay. Under adaptive
the work is durable at once and the CLOSE goes out in the pass.

The server refuses every frame while the close is deferred, so the
deferred close tells the client too. ingestInOnePass now reports
whether the pass left the close deferred, and I2 counts that.

testDemoteBetweenTablesOfGroupClosingFrame* pin the shape: the first
table commits, the second does not, and the server sends only the
CLOSE, or, on a durable-ack connection, nothing yet with the close
deferred.
@jerrinot

Copy link
Copy Markdown
Contributor Author

Reviewing PR #8195 at level 3 with anthropic/claude-opus-5-5, thinking xhigh, review-pr 4.0.
Revisions: head 3c6ffa29ed, base 352cc99204. No submodule pointer moved: java-questdb-client is at 9bc01254 (1.3.12) on both.

Verdict: approve with comments. Nothing blocks. One Moderate item: a role switch now waits for the whole batch commit, and the body doesn't say so or how long it takes.

Title and description

  • Title: OK.

  • Labels: OK.

  • Body: two problems.

    • Missing tradeoff (M1). No bullet says that a role switch now waits for every table commit in a batch. The approach describes the lock design but gives no size.
    • Approach too long. The commit batch takes two paragraphs of about ten sentences. The convention is one to three per feature.

    Replacement bullets and approach. Keep the tandem line, the conditions line and the table as they are:

    - One-row frames spread across tables ingest up to 7x faster.
    - Java client 1.3.12: one-row frames across tables up to 5x faster.
    - Interleaved frames commit once per table per receive pass, cutting WAL transactions.
    - Acks larger than `http.send.buffer.size` no longer drop the connection and duplicate rows.
    - A failed table commit can duplicate other non-DEDUP tables' rows, up to 2 MiB.
    - Role switches wait for whole batch commits: 1,000 tables, 4 ms nosync, 36 s sync.
    - QWP over UDP resumes a table's ingestion after a failed WAL segment roll.
    
    The server's commit batch now spans every table of a receive pass: each table commits once, when the pass ends or at the first frame that cannot join, and a WAL writer savepoint lets a frame that fails or cannot join roll back its own rows while the batch keeps its rows. A table is refused as it joins when its commit would be refused (insert permission, dropped, renamed or suspended), and the batch commits all its tables under one hold of the role-switch lock, so a role switch never splits a batch and waits for it instead. A storage fault, or a DROP, RENAME, SUSPEND WAL or REVOKE that lands while a table waits in the batch, can still fail a table after others committed; the ack then stops before that table's first frame, and the client's replay sends the committed tables' rows from the rest of the pass again.
    
    An ack names every table it covers and could outgrow the send buffer, which dropped the connection; the server now stages such a frame in connection memory and sends it in send-buffer-sized pieces.
    
    The table lookup now evicts a WAL writer that failed while it held no uncommitted rows, for example on a segment roll, so the next frame or datagram gets a fresh writer.
    

    Add the conditions of the role-switch figures to the conditions line: 1,000 one-row tables in one pass, ext4 on NVMe with about 17 ms per fsync.

Blocking

None.

Moderate

M1. A role switch waits for every table commit of a QWP batch

  • Problem: The batch commit holds the role-switch read lock across all its tables.

  • Net impact: A role switch waits for the whole batch: 36 s for 1,000 tables under sync.

  • Evidence: I ran a probe at 3c6ffa29ed and at base 352cc99204. It starts a role-switch write-lock waiter during the first table commit of a pass of 1,000 or 5,000 one-row frames, each for a different table. Results:

    Tables, commit mode Writer wait at head Writer wait at base
    1,000, nosync 3.6 ms 0.39 ms
    5,000, nosync 49 ms 0.22 ms
    1,000, adaptive 27.2 s 6.4 ms
    1,000, sync 35.7 s 0.44 ms
    1,000, sync, 5 s tryLock not acquired after 207 of 1,000 commits 0.47 ms

    A reader on another thread waited just as long in every run.

  • Where: in the diff, QwpIngressProcessorState.java:2123-2160. commitBatchEntries takes the lock once, then commits every table through TableUpdateDetails.commitUnderRoleSwitchFence (TableUpdateDetails.java:308). At base, commit(false) took and released the lock around each table.

  • Trigger and symptom:

    • An Enterprise operator demotes a primary while a QWP client spreads one-row frames over a few hundred tables per receive pass, with cairo.commit.mode set to sync or adaptive.
    • EntCairoEngine.prepareForRoleSwitch calls tryLock with its 5 s default budget, so the demote is refused with RoleSwitchDrainBudgetExpiredException and has to be retried.
    • While the writer waits, RoleSwitchReadWriteLock parks every new reader on the node: ILP, SQL INSERT and DDL, PG wire, view refresh and other QWP connections. That stall lasts up to the budget on each attempt.
    • On this host the 5 s budget is crossed at about 140-200 tables per pass, at about 25-36 ms per sync commit.
  • Net determination:

    • Population: Enterprise replication with an explicit sync or adaptive commit mode, many tables per pass, and a role switch in progress.
    • Delta vs base: the wait grows from one table commit to N table commits.
    • Magnitude: bounded by the tables in one receive pass and by the switch budget. It happens once per role-switch attempt.
    • Offsets: the refusal is clean and can be retried with a larger TIMEOUT. Enterprise defaults to nosync when replication is configured, where the wait is milliseconds.
    • Net: negative, but bounded and off the data path, so Moderate.
  • Strongest reply: the design is deliberate. A demote landing between two tables would leave committed rows above the ack, and the client's replay would duplicate them. The reply holds for nosync. Under sync or adaptive the cost is not stated anywhere and reaches seconds.

  • Suggested fix:

    • At minimum, state the tradeoff in the body with its size (the bullet above).
    • Optionally, bound the hold. For example, stop at a table boundary when a role-switch writer is queued, and handle the rest as a failed tail: the watermark stays below the next table's first frame and the client gets a role-change close. That accepts the replay duplication the PR already accepts for partial failures, but only while a switch is waiting. I have not prototyped this.

Minor

None.

Adjacent findings

None.

Coverage map

  • Test gate: passes, with 0 admitted coverage gaps.
  • Measured at 3c6ffa29ed: 455 of 455 changed lines and 207 of 210 branches ran. That run covered 16 test classes (the 14 changed ones, QwpSenderSuspendedTableAlterE2ETest and QwpWalAppenderTest): 692 tests, 0 failures, 2 skipped.
  • Branches that never ran, none of them a finding:
    • TableUpdateDetails.java:193, where ownSecurityContext == null. The PR only made the method public, so exposure is low.
    • QwpIngressProcessorState.java:2264. It only formats the error text when a rewind fails with something other than a Cairo exception.
    • QwpTudCache.java:931, a writer that is not a WalWriter. This can't happen, because the lookup refuses non-WAL tables.

Summary

  • Verdict: approve with comments. Add the role-switch bullet (M1) and shorten the approach.
  • Earlier findings re-run at this head, none reproduces:
    • Schema parses retained after disconnect: 0 of 40/80/120 reachable.
    • The same, when closing the writer throws: 0 of 40 reachable.
    • The weak zero-allocation bound: a 16-byte allocation per frame now fails the test.
    • The unpinned boundary NACK sequence: the mutation fails testBoundaryCommitFailureOnLaterTableNacksItsFirstFrame.
    • The fuzz timeout seed 4551/36039382: 3 of 3 runs pass in about 8 s.
    • The double append was settled by the author's reply, so I didn't raise it again.
  • Checked and holding up:
    • Staged acks reach the client whole before a fatal or role-change CLOSE (616 probe runs).
    • Reverting the staging makes testOkAckForManyFramesLargerThanSendBufferReachesClient fail 13 of 13 times.
    • The Enterprise engine does not override getRoleSwitchReadLock, so the new lock-held assert holds there.
  • Tradeoffs the PR makes on purpose:
    • Cross-table duplication on a partial commit (the body states it).
    • The role-switch wait (M1).
  • Submodules: none moved.
  • Findings: 1, in the diff. Blocking 0, Moderate 1, Minor 0. The callsite analysis outside the diff found every caller safe. No Critical finding needed a risk rating.

@jerrinot

Copy link
Copy Markdown
Contributor Author

Correction to the review above: M1's sync and adaptive numbers were inflated. The mechanism and the verdict stand.

The falsifier measured them with load average 7-11 from parallel builds, and on the first synced pass after a no-fsync setup, which pays a one-time flush. Re-measured on an idle machine (Samsung 990 PRO, consumer NVMe, ext4): one-row WAL commits to 300 distinct tables, one per table, which is what the batch commits under the role-switch lock.

Commit mode Per-table commit 1,000 tables Tables until the 5 s demote budget
nosync 8-11 µs about 10 ms over 400,000 (the 10,000-table cap binds first)
sync 6.0-6.6 ms about 6 s about 750-830
adaptive 2.6-3.9 ms about 2.6-3.9 s about 1,300-1,900

What changes in M1:

  • Net impact: a role switch waits for the whole batch, about 6 s for 1,000 tables under sync, not 36 s.
  • Budget crossing: the 5 s budget is crossed at about 800 tables per pass under sync and about 1,300-1,900 under adaptive on this disk, not 140-200.
  • Expectation, not measured: drives with power-loss protection fsync in tens of µs, so I expect the 10,000-table cap to bind before the budget there.
  • Suggested body bullet: Role switches wait for whole batch commits: 1,000 tables, 10 ms nosync, 6 s sync.

M1 stays Moderate. It needs an explicit sync or adaptive commit mode with replication (Enterprise defaults to nosync when replication is configured), hundreds of distinct tables per receive pass, and a demote at that moment. The refusal is clean and can be retried with a larger TIMEOUT.

WalWriter.prepareCommit() runs the part of a plain commit that comes
before sequencing: it appends the DATA event record and flushes the
writer's column, event and segment metadata files at the commit mode's
grade, including what nextSeqTxn() would flush first. The commit() that
follows only sequences the record, so a caller can do the file syncs
outside a lock and hold the lock only for the sequencing. A QWP commit
batch uses it in the next commit.

commit0() is split into appendAndFlushCommit() and sequenceCommit();
the plain path keeps head's behaviour and catch blocks. A prepared
commit sequences through the same helper, with a row guard first, and
any failure once the sequencer is asked distresses the writer. Under a
group window the due group flush is left to flushDueGroupCommit(),
which the caller runs once it holds no lock; cleanupBeforeClose()
clears that flag on every path.

A prepared record is never withdrawn. rollback() and close abandon it:
they distress the writer and decrement lastSegmentTxn once, on the
transition of isCommitPrepared, so the segment's closure reports only
sequenced records. A pool tenant's close runs cleanupBeforeClose()
twice and the REVOKE path runs rollback() and then a close; neither
decrements twice. A prepare that fails after its append restores
lastSegmentTxn for the same reason.

DeferredCommitInterceptor gets a default onSequencedBeforeDurablePublish
hook for the W=0 publish re-check, read from the same once-per-commit
local.

Methods of WalWriter audited for the prepared guard, which refuse a
prepared writer with a critical error and leave it prepared:
- guarded: prepareCommit, markSavepoint, rollbackToSavepoint,
  addColumn (8-arg), apply(AlterOperation, boolean),
  apply(UpdateOperation), truncateSoft, rollSegment, commitMatView,
  commitLiveView, commitLiveViewWithReplaceRange, commitWithParams,
  commitWithParamsIfSeqTxn, appendCustomEvent, resetMatViewState,
  commitRebaseSeed, goActive(long), and
  WalColumnarRowAppender.beginColumnarWrite through the writer;
- reached only through a guarded one: addColumn (3-arg) and
  renameTable (apply), goActive() (goActive(long)), the columnar
  finish, cancel and server-timestamp methods (beginColumnarWrite),
  newRow's segment roll (rollSegment; never pending while rows are
  uncommitted);
- dispatch to the prepared sequencing: commit, ic, ic(long);
- abandon the prepared commit: rollback, close and cleanupBeforeClose;
- left unguarded on purpose: the inner MetadataWriterService and
  MetadataValidatorService and applyMetadataChangeLog, which the
  NO_TXN replay of a concurrent DDL runs on a prepared writer;
  truncate() always throws; commit0 asserts it never sees a prepared
  writer.

WalWriterPreparedCommitTest covers the lifecycle under SYNC, ADAPTIVE
with and without a group window, NOSYNC and ASYNC: equality with a
plain commit, abandon by rollback and by close, segment closure
accounting, refusals, the row guard, faults before and after the
assignment, concurrent DDL replay, segment rollover, purge and the
deferred group flush. WalFlushCountingFacade counts WAL file flushes
and whether they ran under a role-switch lock.
Under SYNC and ADAPTIVE, each table of a QWP commit batch fsynced its
private WAL files under the role-switch read lock, so a demote waited
for every table's file syncs: seconds for 1,000 tables. The batch now
prepares every table first (WalWriter.prepareCommit), outside the
lock, and holds the lock only for the role check and the sequencing.
The batch reads the configured commit mode, not the resolved one;
NOSYNC and ASYNC keep the single-phase commit.

A prepare failure refuses the whole batch with nothing committed, at
the batch's first frame, with the text "commit batch prepare failed
[table=<name>]: <message>". Head committed the tables before the
failed one and the replay stored their rows twice. After a demote has
published REPLICA the refusal is the role-change close, read from the
live role on this branch only; a wrong answer there yields a
reconnect-eligible close, never a terminal error.

A group flush that falls due while the tables sequence runs after the
unlock. Its failure never fails the batch: the rows are sequenced, the
writer is distressed, and the table's next frame is refused once.

The abandoned paths need no cleanup under the lock: the prepared
writers are distressed, and the clear() that follows every failure
exit discards them. QwpTudCache.clear() asserts that no prepared
writer survives a healthy clear.
resumeSend drains buffered frames through the frame loop, which can
commit a batch. An unexpected throwable there escaped to the HTTP
dispatcher and left the connection's WAL writers checked out, among
them the prepared writers of an abandoned batch. resumeSend now logs
it and disconnects, as resumeRecv does, so onConnectionClosed frees
the writers. The three checked exceptions it declares pass through
unchanged.
QwpIngressCommitBatchTest pins the commit mode and group window of its
hook engines (pinnedConfiguration, DemotableEngine(configuration)),
which DefaultTestCairoConfiguration otherwise takes from the CI shard,
and adds tests of the prepare phase: it runs only under SYNC and
ADAPTIVE; a demote after it or during it commits nothing and discards
the prepared writers; a REVOKE abandons its table under the lock with
no file work; a prepare failure at the first, middle or last table
refuses the whole batch at its first frame with the truncated text,
and the replay stores every row once; after a demote it is the
role-change close; a fatal one poisons the engine; the due group flush
runs after the unlock, and its failure keeps the ack and refuses the
table's next frame once; an unexpected error after the prepare phase
loses the batch at the boundary and disconnects on the receive and the
send path. A flush count test shows that a batch flushes no
writer-private file under the lock, and as many files per pass as
plain commits of the same rows. The ack-contract fuzz runs per commit
mode and draws prepare faults from a Rnd of its own.

QwpCommitBatchPrepareRealClientTest drives the real client through a
demote after the prepare phase, with and without durable ack, and a
prepare fault: the client replays and every row is stored once.

QwpCommitBatchPrepareCrashTest takes a power loss at a table's
prepare, between the prepare phase and the lock, after the first table
sequenced and inside the deferred group flush, under SYNC and ADAPTIVE
with and without a window. AdaptiveQwpCommitBatchCrashSweepTest sweeps
every durability op of two batches in the nightly job.

FuzzRunner can send a share of the WAL commits and rollbacks through
prepareCommit(); WalWriterPreparedCommitFuzzTest runs the WAL fuzz so
and pins that it sequences and abandons prepared commits.

QwpCommitBatchLockHoldMeasurementTest, gated by
-Dquestdb.measure.e1=true, measures the lock hold, the refusal path
and the discard cost, with nothing the base lacks, so it measures the
base too.
commitPendingBatch's Javadoc names every exit through which a failed
commit batch frees its prepared WAL writers, none of them under the
role-switch lock: the abandon and clear() of a failure return, the
frame loop's abandon and the disconnect of resumeRecv or resumeSend
for an unexpected throw, and handleBinaryMessage's catch and finally
for its boundary commit.

testFatalDueGroupFlushFailurePoisonsEngineAndKeepsTheAck: a
data-sync failure of the group flush that runs after the unlock stops
the engine, the sequenced rows keep their ack, and the durable
frontier stays behind them.
QwpIngressMultiTableBenchmark connected to fixed ports 9000 and 8812,
so it could only measure a server on those ports.
-Dqdb.bench.http.port and -Dqdb.bench.pg.port now name others, with
the old ports as defaults.

The Javadoc now says how to measure each commit mode: one server per
mode, configured with cairo.commit.mode and
cairo.adaptive.commit.group.window, since the benchmark cannot set
them; and a root on a real disk for SYNC and ADAPTIVE, whose cost is
the device flush that tmpfs skips.
WalWriterPreparedCommitTest covers what the coverage run left out: a
prepare that fails flushing the new segment's _meta, which sequencing
would otherwise flush, with and without a data-sync failure, does not
count its record at close; a data-sync failure of the sequencer-log
flush while a prepared commit sequences stops the engine and
distresses the writer; and the plain path keeps its behaviour through
the helpers commit0 is now split into: the fence rejection, replace
ranges with and without rows, and the commits without rows.
Five cases of three repeats each overran the 20-minute test timeout in
one method. Each case now has its own test method, so a run of one
fits it and can be repeated alone when the host gets busy.
AdaptiveQwpCommitBatchCrashSweepTest checked only upper bounds on the
rows each crash point recovers. It stayed green when the prepare path
let a sequencer record become durable ahead of its WAL column data:
recovery suspended the tables and they read back empty, which every
upper bound accepts.

The oracle now requires each table to come back unsuspended, holding
exactly the rows of the transactions the sequencer recorded. The sweep
also requires that the recovered rows never drop as the crash point
moves later, that the first crash point loses rows and that the last
recovers every row, as the sibling sweeps do.

Dropping the column sync from WalWriter.prepareCommit() now fails both
methods at k=16, where the table is suspended. Skipping the sequencer
log fdatasync in the deferred group flush fails the group-window method
at the last crash point, which recovers no rows. Both regressions left
the old sweep green.
The ack-contract fuzz draws prepare faults that often never fire, yet
it treated every iteration with a drawn fault as commit-faulty. That
switched off the I2 check that no row above the ack is committed. NOSYNC
and ASYNC never prepare, so every fault drawn there was inert. Nothing
checked that a fault ever fired: breaking the trigger of
PrepareFaultFacade left every test green.

PrepareFaultFacade now counts the faults it throws, and those it throws
inside WalWriter.prepareCommit(). A prepare fault makes an iteration
commit-faulty only once it fired, and only SYNC and ADAPTIVE draw
prepare faults. Each leg logs its drawn, fired and in-prepare counts.

testCommitBatchFuzzPrepareFaultsFireSync and its ADAPTIVE twin replace
the generator-only test. They script a fault into the iterations of
fixed seeds and require transient and persistent faults to fail a
prepare in a multi-table batch. A dead trigger, or a prepare that no
longer flushes the column file, fails both.
@jerrinot jerrinot removed the QUEUED FOR MERGE Approved PR in the merge queue. Do not merge master into this PR. label Nov 14, 2026
QwpIngressProcessorState read cairo.commit.mode itself to decide whether
a commit batch prepares its tables before the role-switch lock. That
duplicated WalWriter.walCommitMode(), the grade every WAL writer commits
at, and added a configured-mode read that DataSyncFailureModePolicyTest
had not classified, which failed the test in CI.

WalWriter.isCommitSyncing() now answers whether its commits wait for WAL
file syncs, under SYNC and ADAPTIVE, and the batch asks the writer of its
first entry. The decision stays once per batch and the modes that
prepare are unchanged: testCommitBatchPreparesOnlyUnderSyncAndAdaptive
pins them in all five modes.
Enterprise main now overrides SqlCompilerImpl.getShowExecutionRequirements(), which master added in #8118, so its CI no longer compiles against this branch without master.
@ideoma

ideoma commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator

[PR Coverage check]

😍 pass : 680 / 684 (99.42%)

file detail

path covered line new line coverage
🔵 io/questdb/cairo/wal/WalWriter.java 190 194 97.94%
🔵 io/questdb/cutlass/qwp/protocol/QwpMessageCursor.java 7 7 100.00%
🔵 io/questdb/cutlass/qwp/server/QwpIngressProcessorState.java 263 263 100.00%
🔵 io/questdb/cutlass/line/tcp/TableUpdateDetails.java 16 16 100.00%
🔵 io/questdb/cutlass/qwp/protocol/QwpTableBlockCursor.java 19 19 100.00%
🔵 io/questdb/cutlass/qwp/server/QwpIngressUpgradeProcessor.java 74 74 100.00%
🔵 io/questdb/cutlass/line/tcp/WalTableUpdateDetails.java 23 23 100.00%
🔵 io/questdb/cutlass/qwp/server/QwpStreamingDecoder.java 2 2 100.00%
🔵 io/questdb/cutlass/line/tcp/QwpWalAppender.java 39 39 100.00%
🔵 io/questdb/cutlass/qwp/server/QwpUdpReceiver.java 1 1 100.00%
🔵 io/questdb/std/LowerCaseUtf8SequenceObjHashMap.java 1 1 100.00%
🔵 io/questdb/cutlass/qwp/protocol/QwpSchemaCacheEntry.java 5 5 100.00%
🔵 io/questdb/cutlass/qwp/server/QwpTudCache.java 39 39 100.00%
🔵 io/questdb/cairo/wal/WalColumnarRowAppender.java 1 1 100.00%

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Core Related to storage, data type, etc. Performance Performance improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants