Repository navigation
feat(qwp): add paging, row counts, timings and error positions to the query client - #129
Open
glasstiger wants to merge 5 commits into
Open
glasstiger wants to merge 5 commits into
glasstiger wants to merge 5 commits into
Conversation
QwpQueryClient and the Query facade can now use the query-details flags of QUERY_REQUEST, which the server advertises with the CAP_QUERY_DETAILS bit of SERVER_INFO (questdb PR #8356): - QwpQueryOptions sets a row window (skipRows, maxRows) and asks for the row count of the whole result set (countRows), execution statistics (stats) and the error position (errorPosition). It also carries resetSymbolDict. QwpQueryClient.execute() and Query.options() take it; the client reads it once, when the query starts, so a failover replay sends the same options. - The handler receives the answers through new default overloads: onEnd and onExecDone with a QwpQueryDetails (the row count, the compile, execute, count and client-wait times, and the JIT flag), and onError with the position. Each delegates to the next simpler overload, so existing handlers keep working. The client now calls the detailed overloads for every terminal event, client-side failures included, so one override sees them all. - QueryException.getPosition() carries the position to callers of the facade's Completion.await(). A query without options is byte-identical on the wire. Against a server without CAP_QUERY_DETAILS, the client leaves out the count, the statistics and the position, and fails a query that sets a row window before sending it, since that server would stream every row. After a failover reconnect the client checks the new server again. The I/O thread decodes the trailers the request asked for: row_count and the statistics block on RESULT_END, the statistics block on EXEC_DONE, and the position on QUERY_ERROR. A RESULT_END or EXEC_DONE without a requested trailer is a terminal transport error, like any other truncated terminator. A QUERY_ERROR without the position is the baseline frame the server sends for a request whose flags it could not read; it reports no position. RESULT_END and EXEC_DONE now share one bounds-checked varint reader, which also rejects data bits above bit 63 on a tenth byte, as the result batch decoder already does.
The query-details change routes every failure that execute() detects itself to the position-aware onError overload, but the tests written for it reached none of those paths. QwpQueryClientFailurePathsTest now drives each one with a handler that fails on any other overload: - a failover reconnect that fails, and the reconnect of the next query that fails after it; - a stop request between two attempts, from onFailoverReset; - a stop request while the failover reconnect waits for SERVER_INFO; - a stop request right after a transport failure; - a close() between two attempts, from onFailoverReset. A stop request can reach the point right after a transport failure only through a race with another thread, so QwpQueryClient gains setAfterTransportFailureHookForTest(), a barrier that runs there, like setBeforeCloseHookForTest(). TestWebSocketServer gains a hook that runs before SERVER_INFO, and its rejected egress upgrades now drop the connection, as its documentation says, instead of leaving the client to wait out its auth timeout. The default branch of the event switch in executeOnce() is unreachable, as the I/O thread emits only the kinds it handles; an assertion now says so. A comment explains the null guard on serverInfo, which only a connect() racing the query from another thread can trip.
glasstiger
added a commit
to questdb/questdb
that referenced
this pull request
Nov 12, 2026
QwpEgressQueryDetailsClientTest sends the new query_flags through the real java-questdb-client, with QwpQueryOptions on both QwpQueryClient.execute() and the facade's Query.options(). It checks the error positions that onError() and QueryException receive, the row window, the full row count, the timings and the JIT flag against /exec on the same server and against SQL LIMIT, and asserts that the client's wire constants match the server's. The released client 1.3.12 cannot send these flags, so the submodule moves to the head of questdb/java-questdb-client#129 and the poms pin questdb.client.version to 1.3.13-SNAPSHOT. CI then builds the client from the submodule with the local-client profile. The Javadoc of QwpEgressQueryDetailsWireTest no longer claims that the pinned client cannot send the flags.
The javadoc of QwpColumnBatchHandler.onError(long, byte, String, int) and of QwpEgressMsgKind.NO_ERROR_POSITION said that a cancellation reports no error position. Only a CANCEL this client sends does: the server reports position 0 for a query timeout and for a cancellation on the server, such as CANCEL QUERY, as the /exec HTTP endpoint does. The javadoc now says so.
The query details took SERVER_INFO capability 0x08 and query flag 0x02, which per-query timeouts already use in released clients: c-questdb-client 7.2.0 and later, and the Python client 6.0.0 and later, read capability 0x08 as "the server honours a per-query timeout" and then send flag 0x02 followed by timeout_ms. A server with the query details would have read that flag as an error-position request, ignored the timeout and answered with a QUERY_ERROR those clients cannot decode. CAP_QUERY_DETAILS moves to 0x10, and the flags to 0x04 (ERROR_POSITION), 0x08 (STATS), 0x10 (COUNT_ROWS), 0x20 (SKIP_ROWS) and 0x40 (MAX_ROWS), as on the server. CAP_QUERY_TIMEOUT (0x08) and QUERY_FLAG_TIMEOUT (0x02) now name the reserved bits, with the values the timeout change gives them, and the docs say a flag's field goes after query_flags in ascending order of the flag bits, so timeout_ms comes ahead of skip_rows and max_rows. New tests pin the allocation: every capability and flag is a bit of its own and none takes the timeout bits. A server that advertises per-query timeouts without the query details gets none of the details, and a row window fails before it is sent.
QwpQueryClient reported the failures execute() detects itself through eleven separate handler.onError() calls. Five of them -- an exhausted failover (two sites), an interrupted backoff, and a rejected or unavailable credential during a failover reconnect -- had no test that would notice a call to the 3-arg overload. A regression there would hide the error from a handler that overrides only onError(long, byte, String, int), as QwpColumnBatchHandler tells users to do. deliverClientError() now makes that call for every client-detected failure, deliverFinal() included, so any test that reaches one of them pins all of them. QwpQueryClientFailurePathsTest adds the exhausted failover, the rejected credential and the unavailable credential; QwpEgressMockServer gains setAuthorizationValidator() for the rejected credential. QuestDBQueryOptionsTest now checks that options() on a closed handle throws and leaves the next borrower's query alone. No test covered the stale-lease guard of the new Query.options() before. Test helper cleanup: - RecordingQueryHandler drops the unread batchCount and hosts the assertNoDetails(QwpQueryDetails) that QueryEventTest and QwpEgressTerminatorDecodeTest each carried a copy of. - QwpEgressFrames.QueryRequest drops the unread initialCredit. - testEachDetailOptionSendsItsFlag becomes testEachBooleanOptionSendsItsFlag: resetSymbolDict is not a query-details option. - The QueryException test moves from QwpQueryOptionsTest to its own QueryExceptionTest. - QwpEgressMockServer drops a connection on its read thread. The comment that sent the close to another thread claimed close() joins the read thread, which it skips when called on that thread.
Contributor
[PR Coverage check]😍 pass : 270 / 274 (98.54%) file detail
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
skipRowsandmaxRows./execdoes.QueryException.Tandem: questdb/questdb#8356.
QwpQueryOptionsholds the options, andQwpQueryClient.execute()andQuery.options()take it. The client sets the matchingquery_flagsbits, withskip_rowsandmax_rowsafter them, only for a server that advertisesCAP_QUERY_DETAILS, so a query without options stays byte-identical on the wire. The new bits leave capability0x08and flag0x02to per-query timeouts, which the released C and Python clients already use.The I/O thread decodes the trailers the request asked for and hands them to new default handler overloads:
onEndandonExecDonewith aQwpQueryDetails, andonErrorwith the position. Each overload delegates to the next simpler one, so existing handlers keep working.RESULT_ENDandEXEC_DONEnow share one bounds-checked varint reader, which also rejects data bits above bit 63, as the batch decoder does.