Repository navigation
Conversation
The first setter of a row waited at most one second for the initial WebSocket handshake and the schema DESCRIBE round trip, then threw SCHEMA_UNAVAILABLE and dropped the row. One second is a per-row bound on what is really a connection-setup cost, so a cold TLS handshake, a GC pause, or an I/O thread busy draining a backlog could push an ordinary lookup over the line. Every reconnect clears the schema cache, so a flaky link turned into a stream of lost rows. QwpWebSocketSender now reads the budget from a single field defaulting to DEFAULT_SCHEMA_WAIT_MILLIS (30 s). Both the per-row deadline in bindingForEffectiveWrite() and the refresh in refreshAfterSchemaRejection() use it. A healthy lookup still completes in milliseconds; the constant only bounds a dead or still-connecting link. setSchemaWaitMillisForTesting() lets the one test that asserts the deadline fires against an unreachable port keep its short budget. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BDP5ZVN8xijRp9LhQoreMf
Parse textual decimals through the existing Decimal256 scratch and reuse the established decimal and text conversion paths. Preserve row rollback, schema refresh, duplicate ordering, and missing-column scale semantics without adding retained converter state. Cover exact wire values, every decimal width, text targets, nulls, invalid rows, inference, and real-server ingestion. Record the design decisions, validation evidence, and remaining branch-level blockers.
The client carried two double-to-text implementations that disagreed on about 0.3% of values. Numbers.append() used the pre-Ryu algorithm and served the legacy ILP path, while QwpSchemaDoubleFormatter wrapped a private copy of the server's Ryu for schema-aware QWP writes. The same double therefore reached the server as two different strings, depending only on which transport carried it. Numbers.append() now formats through Ryu, copied from the server so that client and server produce identical text. RyuDouble moves to io.questdb.client.std as the single copy, and QwpSchemaBinding calls Numbers.append() directly. QwpSchemaRyuDouble and QwpSchemaDoubleFormatter are gone. RyuDouble calls Math.multiplyHigh, which Java 8 lacks, so Compat gains a multiplyHigh shim: the intrinsic on Java 9+, and a hand-rolled equivalent on Java 8. A differential over 20 million random pairs plus edge cases shows the fallback matches the intrinsic exactly. Retiring the old formatter also retires the FdBig bridge and the jdk.internal.math export flags it needed. The multi-release jar stays, because Compat still supplies currentPid() and onSpinWait() to five other callers. JarPackagingIT now guards the Compat variants rather than the FdBig ones, and still executes the formatter out of the packaged jar on both JDKs. This also fixes two Java 8 breaks that predate the change and that a modern JDK hides. QwpSchemaRyuDouble called Math.multiplyHigh from a shared source root, and QwpSchemaFeedbackResponseTest chained putShort() off position(), which returns Buffer rather than ByteBuffer on Java 8. Both would have failed the Java 8 CI job. The legacy ILP path changes its wire text for roughly 0.3% of doubles. Ryu emits the shortest form that still reads back to the same value, so the output is never longer and never loses precision; over 3 million random values both formatters round-trip to identical doubles. One NumbersTest expectation encoded the old extra digit and is updated. Verified on JDK 8 and JDK 25: 3721 unit tests and the packaging ITs pass on both, and the shared floating-to-text corpus matches on all 40 rows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BDP5ZVN8xijRp9LhQoreMf
Every row-building setter in QwpWebSocketSender repeated the same two catch blocks: roll the partial row back, run the one-refresh rule for a LineSenderSchemaException, and rethrow everything else. The policy lived in 31 copies, and D045 already recorded one setter (CHAR) that shipped without the schema branch. onRowFailure() now owns that policy. It rolls the whole row back first, so a failed refresh cannot leave a half-written row, then hands a schema rejection to refreshAfterSchemaRejection() and lets every other failure propagate unchanged. All 36 row guards call it from a single catch. The two catches in installSchemaBinding() guard buffer cleanup, not rows, and stay as they are. Behaviour change: the four longArray() overloads had only the generic catch, so their UNSUPPORTED_FEATURE rejection skipped the refresh. They now follow the standard path: the first rejection costs one DESCRIBE round trip and reports SCHEMA_CHANGED when the target column changed. atNow() also routes through the helper, but its null column name keeps it on the no-refresh path. QwpSchemaSenderIntegrationTest covers both longArray() outcomes, asserts whole-row rollback and exactly one refresh, and extends the null no-op test with the four longArray() null forms. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The QWP longArray setters cannot become valid after a schema change. Preserve their UNSUPPORTED_FEATURE rejection without issuing DESCRIBE, while retaining the shared row rollback helper. Cover all four overloads, rollback, changed schemas and unavailable metadata. Keep double-array schema recovery and null no-ops unchanged.
A QWP frame has one wire family, so a flush that held both legacy and schema table blocks needed its own delivery pipeline. flushPendingRowsMixed() duplicated the split path's pre-flight sizing, dictionary-chunk fallback, publish loop, baseline advance and commit bookkeeping, and it encoded every block two to three times. Row-boundary adoption was the only source of such batches. A batch now has one wire contract, selected by its first committed row. bindingForEffectiveWrite() keeps returning the legacy contract while a legacy batch is pending, and the first row after the flush adopts schema mode. The upgrade stays one-way and is delayed by at most one batch. This deletes flushPendingRowsMixed(), mixedFramesFit(), encodeSingleTableFrame() and isMixedSchemaBatch(); the encoder is unchanged. Trade-off: rows written after the supporting handshake but inside the pending legacy batch keep legacy conversion, for example microsecond Instant truncation. Which rows preceded the handshake was already a timing race. The change also fixes an existing defect. A legacy row, a flush, a supporting reconnect and then a row on the same table threw IllegalStateException from the column setter: installSchemaBinding() bound in place whenever the buffer had no rows, but a flushed legacy buffer keeps its column layout. It now binds in place only a buffer with no columns and replaces the buffer otherwise. The two upgrade integration tests now assert the batch-boundary rule and cover the flushed-legacy-buffer path. The design document and the journal record the superseded row-boundary rule. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The ASYNC initial-connect javadoc in Sender still described the superseded rule: once schema support is confirmed, all future rows use strict schema lookup and only a row already in progress keeps its contract. The sender now selects one wire contract per batch, so rows added to a pending legacy batch stay legacy, and the first row after that batch flushes adopts schema mode. The javadoc states that rule. Two passages in the design document still named the row-boundary transition; they now name the batch boundary. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
QwpSchemaProtocol.decodePayload() built a case-insensitive column-name index to reject duplicate names and then discarded it. Every QwpSchemaBinding rebuilt the same index in its constructor, and hasSameRelevantTarget() resolved names in the refreshed schema with a third mechanism, a linear equalsIgnoreCase scan. The decoder now hands its index to QwpSchemaResponse, which exposes getColumnIndex() and owns hasSameRelevantTarget(). The binding drops its columns field and its constructor loop, and delegates the comparison to the snapshot. The index is never mutated after the decoder publishes it through a final field, so the I/O thread to user thread handoff stays safe. Each cached QwpSchemaResponse now retains its index for its lifetime. Before, only live bindings held one. QwpSchemaProtocolTest gains direct coverage for the case-insensitive lookup and for the target comparison, which only sender-level tests exercised before: absence on one or both sides, a retyped column, the extension-parameter flag, a moved or removed designated timestamp, and a schema-less RESULT_MISSING response. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The 21 QwpSchemaBinding*Test classes each carried a private wire Reader plus their own copies of the schema-response builders: 21 readers in 17 textual variants and two naming dialects (u8/i32/i64 vs byteValue/intValue/longValue), 8 variants of known(), 6 of column(), 4 of decode(). The copies were semantically identical, so a change to the schema envelope or the table header meant hand-editing up to 19 slightly different helpers. QwpSchemaTestFixtures now owns the schema-response builders (known, missing, schemaResult, column, parameterizedColumn, frame, decode), binding(), assertReason(), and the encoded-table header checks (tableHeader, skipTablePrefix, skipTableHeader). It writes the schema envelope by hand, independently of any production encoder. QwpTestWireReader is the single bounds-checked reader, with one naming dialect and a bounded varint; it never calls production decoders. The tests keep tableReader(), wireType(), targetType() and every conversion expectation, because those encode per-test intent. Two shared assertions are stricter than some of the copies they replace. assertReason() always asserts that the rejection is not retryable. tableHeader() always asserts the message magic, version, flags, and table count; six classes only skipped the header before. Four of the old readers read i32/i64 without a bounds check, and only one bounded its varint loop. The test count is unchanged at 139 across the binding classes. The change removes about 2,000 lines of test code. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The schema-aware sender no longer swaps a table's schema binding under pending rows. Once a table holds rows in the current batch, every later row of that batch validates against the snapshot the first row pinned; bindingForEffectiveWrite() consults the cache only when the table buffer is row-less, which is the first row after a flush or reset. installSchemaBinding() then clears the buffer and rebinds it in place instead of retiring the old buffer alongside a new one. A failed setter reports its local INVALID_VALUE or UNSUPPORTED_FEATURE directly. onRowFailure() rolls the row back and rethrows; it never performs a network lookup, so a rejection costs no round trip and cannot fail for a transport reason. The cache stays current through the paths that already exist: the server attaches the table's current schema to the ACK of a frame that carries a stale identity, the send loop applies that feedback before it advances the ack watermark, INVALIDATE_ALL empties the cache so the next batch looks the table up again, and a reconnect clears the cache. An explicit refresh API was considered and dropped: a caller cannot tell a stale-snapshot rejection from invalid input, so there is no sound rule for when to call it. This removes retiredTableBuffers and its handling on flush collection, post-flush reset, reset(), close() and byte accounting; the schemaResolutionFresh out-parameter threaded through resolveSchema() and QwpSchemaCoordinator.resolve(); refreshAfterSchemaRejection(); QwpSchemaResponse .hasSameRelevantTarget() and its binding delegate; and the SCHEMA_CHANGED reason, which nothing raises any more. The integration test fixtures now acknowledge like the real server: SchemaHandler and LongTextSchemaHandler attach the current schema to the ACK when the snapshot they last delivered is stale, or answer INVALIDATE_ALL when they cannot describe. The generation rebind tests drive adoption through flushAndGetSequence() and awaitAckedFsn() and read the two generations from two single-block frames via a multi-frame FrameReader. New coverage: an INVALIDATE_ALL ACK makes the next batch surface the typed lookup failure from the setter, a version change does not split the pending batch, and reset() keeps the cached snapshot because it ships no frame. The design doc's "Local errors and stale schemas" and "One schema snapshot per table per batch" sections describe the new contract, and the contract-simplification proposal records proposal 1 as implemented without an explicit refresh. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Server PR #7531 (browser negotiation) already claimed header flag bit 0x02 for FLAG_DURABLE_ACK_POLL, a zero-table control frame. Merging server master into the schema-aware branch produced a clean textual merge, but the ingress dispatch then read a poll frame as a schema-flagged frame on a connection that had not negotiated schema and rejected it. The poll bit is already on server master, while FLAG_SCHEMA is unreleased, so FLAG_SCHEMA moves to the free bit 0x40. The server constant changes in lockstep. The design doc and the journal record the new value and the reason. Store-and-forward frames persisted by an earlier build of this branch carry 0x02 and no longer replay as schema frames. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Retain pending schema lookups when the transport disconnects, including requests that have not reached the wire. Resend on the replacement connection with a fresh request ID while preserving the original lookup deadline, so a brief server outage does not abort row construction. Keep explicit schema errors and malformed-response cancellation intact. Add lifecycle coverage for repeated disconnects, stale replies, unsent requests, timeout, interruption and close, and update the design contract. Validate 296 client tests on Java 8 and Java 25, the original failing fuzz seeds, and 18 server integration tests. All pass.
QwpSchemaBinding re-validated and re-hashed the column name on every value: isValidColumnName, a schema hash lookup, and the buffer lookup, plus the designated, wire-type and parameterized checks. All of these depend only on the column, so every row paid for work whose answer never changes while the binding is attached. QwpTableBuffer.ColumnBuffer now records the schema index the binding resolved it to (-1 when the schema lacks the column). Each write first finds the column through the buffer's sequential cursor; a resolved column skips name validation, the schema lookup and the per-column checks, leaving one name compare plus the value conversion, the same lookup cost as the legacy path. A binding only attaches to an empty buffer and unbinding closes its columns, so a recorded resolution can never outlive its schema. Columns without one fall back to the full path. The parameterized-type check now runs before column creation, so a rejected write no longer leaves an empty column behind. The three copies of the inferred-column logic share isInferredWritable and claimInferred. QwpSchemaBindingBenchmark (10 columns, 5 LONG + 5 DOUBLE) measures the schema path at 42 ns/row against 31 ns/row for legacy, down from 141 ns (2-char names) and 282 ns (14-char names). The remaining ~1 ns per value is the target-type dispatch.
QwpSchemaTimestampParser did work whose outcome was already decided. Every ISO timestamp first ran the PG parser to completion of the day and failed there. parse() fetched an exception on the success path. Each fraction attempt copied a Parsed object, the greedy fraction threw to signal a miss, numeric zones allocated a Zone, and named zones were found by a linear scan over ~4400 names with an iterator per value. The first timestamp with any zone, even 'Z', loaded every zone's rules on the producer thread. appendStringTimestamp also UTF-8 encoded each value into a sink before parsing it back through a byte view. The character after the day now selects the parser: only the PG layout has a space there, and a space fails every ISO layout. DATE dispatches the same way between PG, UTC and a raw number. Fields live in locals, the greedy fraction returns a sentinel, numeric zones return an offset, and named zones sit in a hash table inside a holder class that 'Z' and numeric offsets never initialize. TIMESTAMP text now parses the caller's CharSequence directly. The server parses TIMESTAMP text as UTF-8 bytes, where no non-ASCII byte matches anything, so named-zone lookup rejects non-ASCII for TIMESTAMP; DATE keeps its UTF-16 matching, as the server's fallback does. Zone names are upper-cased with Locale.ROOT instead of the default locale. Under a Turkish default locale about 2240 names containing 'i' could never match; results elsewhere are unchanged. The client corpora, the OSS e2e suites against a real server (22 tests, including named-zone gap, overlap, alias and history cases) and 2471 QWP client tests pass; the binding tests also pass on JDK 8.
QwpSchemaTimestampParser carried a micros copy (TIMESTAMP) and a millis copy (DATE) of most of its arithmetic: local time, year and month starts, day of week, next/previous weekday, zone offsets and recurring zone transitions. The copies differed only in their unit and in three behaviours of the server's two parser families. A Target now holds the unit sizes plus those three differences as named flags: micros year starts saturate on negative overflow, TIMESTAMP zone names must be ASCII (the server matches them as UTF-8 bytes), and DATE recurring zone rules reject negative years. Each function pair becomes one function over a Target. Zone stores its cutoff and offsets in seconds and scales them per unit; the scaling reproduces the old per-unit values exactly, including Instant.ofEpochMilli for historic DATE offsets. Month tables hold day counts instead of per-unit values. The PG DATE tail fallback and the UTC DATE layout stay DATE-specific. The file shrinks from 889 to 752 lines. Flipping any one Target flag fails a pinned corpus case (micro_parsed_long_min, min_int_year, negative_year_named_zone_wrap, micro_nonascii_zone_lookalike). The client corpora, the OSS e2e suites against a real server (22 tests) and 2471 QWP client tests pass; the binding tests also pass on JDK 8.
QwpSchemaCoordinator.resolve() unparked the I/O thread after queueing a DESCRIBE. While the wire was down, that unpark cut connectLoop's backoff park short, so a producer retrying a STRICT lookup drove one reconnect attempt per lookup instead of one per backoff interval: 940 lookups in one second produced 940 connect attempts. The unpark bought nothing. The I/O loop polls hasPendingRequest on every pass, parking at most DEFAULT_PARK_NANOS (50 us) between passes, and connectLoop never services the request anyway. Removing it restores the base invariant that only close() unparks the I/O thread. A lookup now waits up to one idle park before it goes out, which is negligible next to its network round trip. testLookupsDuringOutageDoNotShortenReconnectBackoff drops the wire with a 200 ms backoff and resolves in a tight loop for one second; attempts stay within the backoff budget. 2472 QWP client tests pass.
schema_wait_millis, drain(timeoutMillis) and close_flush_timeout_millis each built an absolute deadline as nanoTime() + millis * 1_000_000. A large value such as Long.MAX_VALUE, the usual "no limit" convention, overflowed that sum into the past, so the wait ended at once: STRICT failed the row with SCHEMA_UNAVAILABLE, AUTO silently fell back to the legacy contract, drain() returned false, and close() skipped its flush wait, which loses unacked rows in memory mode. Each wait now records its start and a saturated budget from TimeUnit.MILLISECONDS.toNanos and compares elapsed time against it, which cannot wrap. QwpSchemaCoordinator and awaitInitialSchemaMode drop their hand-rolled saturating multiplies for the same call. The Sender javadoc states that Long.MAX_VALUE waits without a limit; the builder is unchanged. testUnboundedSchemaWaitStillResolves binds the schema in STRICT and AUTO with an unlimited wait, and QwpWebSocketSenderUnboundedWaitTest holds the server ACK to prove drain() and close() keep waiting. All three fail against the previous code. 2514 client tests pass; the new class also passes on JDK 8.
QwpSchemaCoordinator evicted a TOO_LARGE DESCRIBE answer instead of caching it, so AUTO paid one blocking round trip on the producer thread for every batch of a table too wide to describe (more than 2048 columns, or a schema larger than the server's send buffer), and a pending schema-framed batch was flushed early, splitting the frame. TOO_LARGE now caches like MISSING through one isCacheable rule shared by completeResponse and applyFeedback; DENIED and UNAVAILABLE may clear at any time and are still evicted. The coordinator returns the cached TOO_LARGE response and the sender maps it in one place: AUTO writes the legacy contract, STRICT fails with UNSUPPORTED_FEATURE. An AUTO batch for an oversized table costs a map lookup, with no round trip and no exception. The negative entry stays correct because the server reports each legacy-framed table once per schema version: when the table shrinks, the next legacy ACK carries the describable schema and replaces it. STRICT writes no rows to such a table and so earns no feedback; it asks again when it finds a cached TOO_LARGE, as it did before. Coordinator tests cover caching from a response and from feedback, and that UNAVAILABLE is not cached. testTooLargeSchemaIsDescribedOnce sends five AUTO batches with one DESCRIBE (the old code sent ten) and keeps STRICT failing. 2517 client tests pass; also JDK 8.
QWP WebSocket senders resolve the table's schema eagerly in table(), which widened its contract beyond the old "select the table" javadoc. It does not connect: build() always runs ensureConnected(), so the call inside table() is a no-op for any built sender. It can, however, wait up to schema_wait_millis (with an asynchronous initial connect that includes waiting for the first connection), fail with a schema error in STRICT mode, and publish the pending batch when the table's wire contract differs, which can raise flush-timeout and backpressure errors. The javadoc now states that surface, and that a failure leaves the table selected with no row started, so calling table() again retries. No behaviour changes.
The client's DecimalParser required NaN and Infinity to be the whole token, while the server's DecimalParser matches them as prefixes. So "NaN " or "NaNjunk" became a NULL decimal on every server ingest path, including legacy QWP text into a DECIMAL column, but failed the row in schema mode, where the client converts the text itself. isNanOrInfinite is back to the prefix match it had before this branch, so both paths store NULL. The client keeps its long exponent arithmetic and range guards: they only reject inputs such as "1e2147483647" that the server does not parse into a valid value either. DecimalParserServerCompatibilityTest now pins the prefix behaviour and that truncated tokens are still rejected, and the string-to-decimal corpus expects NULL for its NaN and Infinity junk rows. Both fail against the previous parser. 3205 client tests pass.
testUuidConformanceCorpus parsed every row but never checked what the valid rows encoded, so 8 of its 20 rows, including the case-folding row, asserted nothing. It now compares each valid row's limbs with UUID.fromString(expected); a NULL row accepts a null-bitmap entry or the both-Long.MIN_VALUE UUID null sentinel, which the server reads as NULL. Swapping the limbs in the binding now fails it. The timestamp-inputs, timestamp-units and long-to-numeric loaders had no row-count guard, so an emptied corpus or a filter that skipped every row would pass with zero cases. Each now asserts its exact count. QwpSchemaProtocolTest drops its private frame and column copies for the shared QwpSchemaTestFixtures helpers, so the frame version byte comes from QwpConstants.VERSION instead of a hardcoded 1.
The schema cache was a LinkedHashMap capped at 1_000_000 entries with an eviction loop. The map has no access order, so the eviction was FIFO and would have dropped the hottest tables first; in practice it never fired, because the cache is bounded by the tables the sender writes, like its per-table buffers, each far heavier than an entry. It is now a final HashMap without the cap; lookups allocate nothing. With the map always present, the four null checks, the lazy creation and the put/remove helpers go away, and completeResponse evicts once before its failure switch instead of in each branch. Request ids are now assigned only in requestToSend. resolve() used to check id exhaustion and encode the DESCRIBE itself, duplicating the path requestToSend already ran after a reconnect cleared the id. Every send attempt, first or after a reconnect, now takes a fresh id there. An exhausted id space (2^63 lookups) now fails the lookup at send time with the same error. 2517 QWP client tests, the coordinator tests on JDK 8 and the OSS QWP suite (1745 tests) pass.
Brings in the JDK 26 compatibility fix (#99) and the token-store rename retry fix (#97). Upstream #99 rewrote the Java 11 FdBig bridge around method handles because JDK 26 made jdk.internal.math.FDBigInteger package-private. This branch already retired FdBig entirely: Numbers.append() formats through Ryu over the public Compat.multiplyHigh shim, so no JDK-internal class is needed on any JDK. The merge therefore keeps both FdBig variants deleted, keeps JarPackagingIT asserting the Compat variants instead of FdBig, and takes upstream's removal of the now-unused compiler/javadoc arg properties from the pom. DoubleFormatBenchmark from #99 still benchmarks Numbers.append(); its Javadoc now describes the Ryu formatter instead of the FdBig bridge.
The send loop cleared the schema cache on every reconnect, so the next batch for each table paid a blocking DESCRIBE round trip, one table at a time. The server ingests by column name and uses the frame's schema identity only to decide on feedback, and a new connection reports feedback for every stale table on its first ACK. The cache now survives the reconnect. The clear was also the only way out of one stuck state. When a column changes type, a row that the stale snapshot rejects never reaches the server, so no feedback refreshes the snapshot. QwpSchemaBinding now records when the snapshot type of a known column rejects a value: an unsupported conversion, an out-of-range value, excess decimal precision or an array rank mismatch. The sender's row-failure path then evicts that snapshot if the cache still holds the same identity, so the next row looks the table up again. Inputs that no column type can take, invalid names, nulls and malformed text never evict. Tests cover a reconnect that reuses the cache without a DESCRIBE, and the lookup that follows a type rejection.
A store-and-forward slot recovered with only a deferred (uncommitted) tail of schema-framed frames made requiresSchema() true. The foreground sender set that requirement on its reconnect factory before any send loop existed, so build() failed with QwpSchemaCapabilityMismatchException against every server without schema support, even with schema_mode=off. The tail belongs to an aborted transaction and never reaches the wire. ensureConnected() now calls retireRecoveredOrphanTailIfReady() before it sets the schema requirement, matching BackgroundDrainer. The sender starts against a legacy server and sends new rows as legacy frames. QwpSchemaReplayNetworkTest now asserts that outcome instead of the startup failure.
QwpSchemaCoordinator.resolve() lower-cased the table name into a new String before every cache lookup, hit or miss. In AUTO and STRICT modes the sender resolves twice on the first row of each table in each batch, so a schema-aware sender produced two garbage Strings per table per batch on the producer thread, and one pair per row for callers that flush every row. QwpTableBuffer now computes the lower-cased schema key once, reusing the name itself when it is already lower case. The sender passes that key to resolveSchema(), peekSchema() and refreshSchema(), and normalize() returns an already lower-case String unchanged, so a cache hit no longer allocates. DESCRIBE requests now carry the lower-cased name; the server resolves table names case-insensitively, so the answer is unchanged.
Move FLAG_SCHEMA into its alphabetical slot after FLAG_GORILLA in QwpConstants. Reword the Unsafe comment on the override field offset: it named the FdBig double-formatting bridge, which this branch deleted, so it now names the makeAccessible() callers that the offset protects. No behavior change.
Schema-validated rows now go out as ordinary QWP frames. The encoder no longer writes FLAG_SCHEMA or the per-table (tableId, metadataVersion) identity; the snapshot stays in client memory, where the binding validates and converts rows against it. The server ingested both frame families identically and used the identity only to decide schema feedback, which it decides from the version it last reported on the connection instead. With one frame format every persisted frame is sendable to any peer, so the store-and-forward replay path needs no capability gating. This removes maxSchemaFsn, requiresSchema(), setSchemaRequired(), QwpSchemaCapabilityMismatchException, the recovery scan for schema frames and the drainer checks. The sender drops its one-wire-family- per-batch rule: a legacy row joins a pending schema batch instead of forcing a flush. The encoder keeps one path; a schema-bound table differs only by always shipping the GEOHASH null bitmap. The cost is that the first ACK per table on each connection carries the table's schema. Senders above roughly 400 wide tables exceed the 1 MiB feedback limit after a reconnect, receive invalidate-all and send one DESCRIBE per table in the next batch. The design doc records the measurements and marks the head-frame gating proposal superseded: a prototype of it livelocked at the format boundary and its transaction gate failed against 9.4.x servers.
Replace per-digit division by variable powers of ten with a two-digit lookup and constant division by 100. Reuse the existing thread-local scratch array and decode only the digits that fixed-point scale emits. Keep Ryu conversion, notation boundaries and scale behavior unchanged. Add character and UTF-8 sink coverage for scale truncation, scratch reuse, special values and randomized raw-bit round trips. JDK 25 measurements show 28-30% faster full-precision and wide-range formatting and about 17% faster eight-field text-row construction for those workloads. Short decimals remain slower than the pre-Ryu formatter; this change does not close that regression. Selected low-scale cases cost about 0.6 ns/value more than the previous head. Validate 138 focused tests on Java 8 and Java 25 and compare more than 11 million formatting cases with the unchanged head.
Clarify that AUTO avoids schema waits after a detected disconnect, but an open unresponsive connection can consume a fresh timeout per lookup. Document the intentional repeated-wait trade-off and keep the timeout API description concise. This changes comments only.
Brings in upstream through 1.3.13-SNAPSHOT: the event-driven I/O loop, batched WebSocket sends, symbol dictionary recycling, durable-ack tiers, refreshing token providers and the azure module. Conflict resolutions that go beyond picking both sides: - CursorWebSocketSendLoop: upstream's loop now blocks in poll(2) when idle instead of parking 50 us, so a queued DESCRIBE would wait until the next PING. QwpSchemaCoordinator now rings the loop's idle doorbell (wakeIoIfParked) after it queues a request, and the loop does not park while hasUnsentRequest() is true. The doorbell only ends the idle poll, so lookups during an outage still cannot cut the reconnect backoff short. - Symbol dictionary recycle: the rebuilt loop starts without a client, which made STRICT treat the sender as never having negotiated schema and let legacy rows through, and dropped the schema cache. The sender now carries hasNegotiatedSchema and the cache across the rebuild (seedSchemaState), the schema counterpart of markEverConnected(). The coordinator keeps its cache after close so retireStoppedLoop() can export it. - connectWithCredentialSupplier: upstream's 27-argument master stays as an overload; a new master adds schemaMode and schemaWaitMillis. The obsolete boolean-durable-ack overload is gone, as upstream removed the boolean variants. - awaitAckedFsn() and the close drain keep the branch's elapsed-versus-budget timeout, so Long.MAX_VALUE still waits without a limit under upstream's park-based waits. - QwpColumnWriter uses upstream's pre-encoded table header; the schema-bound GEOHASH null bitmap stays. - SymbolDictRecycleTest: a row's first setter now resolves the schema and so completes a pending REBUILD before the row takes a value. The test asserts that instead of atNow() refusing the row. QwpSchemaSenderIntegrationTest#testSymbolDictRecycleKeepsSchemaContractAndCache covers the recycle fix; with the seed removed, STRICT ships the UUID text as VARCHAR. Without the doorbell, 18 schema coordinator and lifecycle tests time out.
Remove the four design notes the branch added under design/: the implementation journal, the 47-test failure plan, the superseded head-frame gating proposal and the contract simplification proposals. They recorded work in progress, including local paths and log files, and parts of them contradicted the shipped code. Fix documentation that still described the removed schema frame family: - Sender.SchemaMode: OFF never requests schema support, and a table, not a batch, keeps one contract while it holds pending rows. - Sender.table(): only STRICT waits for the first connection, and the first setter after a flush resolves the schema again. - WebSocketClient: the Sender does request schema support. - README: drop the link to docs/schema-aware-ingestion.md, which never existed. Document the four LineSenderSchemaException.Reason values, merge the two summaries in the RyuDouble class comment, and record why QwpSchemaBinding.longColumn() does not delegate to integerColumn(): delegating measured about 4% slower per row in QwpSchemaBindingBenchmark.schemaRow (38.2 to 39.8 ns, three forks each). Tests: drop the unchecked tableId and version parameters from the QwpSchemaSenderIntegrationTest frame readers and from QwpSchemaTestFixtures, and rename the STRICT recovery test after what it checks now, the wait for the first connection.
A sender converts values against a cached table schema. After ALTER COLUMN TYPE widened a column (FLOAT to DOUBLE, TIMESTAMP to TIMESTAMP_NS, INT to LONG), it kept converting to the old type until the server's reply to a frame for that table reported the change. An idle sender never got such a reply, so its next batch lost precision (0.1 stored as 0.10000000149011612, nanoseconds truncated) or rejected values the column now takes, however long ago the ALTER ran. Reconnects kept the stale cache too. The sender now tracks when the server last confirmed each table's cached schema. A DESCRIBE confirms it as of the moment before the request. An OK reply to a frame that writes the table confirms it as of that frame's flush: the I/O thread applies the reply's schema feedback before it publishes the OK watermark. At the first row of a batch, a schema unconfirmed for more than SCHEMA_MAX_AGE_MILLIS (1 s) is looked up again while the wire is up. Busy senders keep getting confirmations from their replies and never pay the lookup. A re-check that fails as unavailable keeps the cached entry and defers the next re-check by the same limit; a denial still fails the row. While the wire is down the sender keeps using the cache, as before. - QwpSchemaFreshness holds the per-table confirmation state. - QwpSchemaCoordinator.revalidate() asks even on a cache hit and keeps the entry when the lookup fails. - CursorWebSocketSendLoop publishes its OK-level watermark to the producer; in durable-ack mode the trim watermark lags it. Rows converted between a change and the reply that reports it still use the old type: frames in flight, rows written within the age limit, and rows written while the wire is down. Sender.SchemaMode documents this. Tests: QwpSchemaFreshnessTest pins the confirmation rules and checks random sequences against a log of confirmations. Coordinator tests pin revalidation on a hit, on a timeout, on UNAVAILABLE and on DENIED. Sender tests against a fake server pin expiry at the limit and that only acknowledged frames confirm, including in durable-ack mode, split flushes, frames written while the wire is down and across a symbol-dictionary recycle, plus failed and denied re-checks. Test helpers that count DESCRIBE requests across outages now freeze the sender clock, so the age limit cannot fire under them.
The comments of the schema age limit explained the same mechanism in five places. QwpSchemaFreshness and Sender.SchemaMode now carry it; the constant, the sender's helpers and the send loop's accessors point there and keep only what is specific to them. Also corrects two docs: QwpTableBuffer.getSchemaFreshness() returns a tracker, not a time, and a frame published while another waits for its reply is never recorded, not recorded later. No code changes.
Ryu finds the shortest decimal by removing one digit per pass from a 17-digit interval, so it is slowest when the answer is short: 0.1 or 3.25 took about 36 ns against 21 ns with the pre-Ryu formatter of 1.3.12, which made legacy ILP text rows with such values up to 1.5x more expensive to build. Numbers.append() now tries an exact path first. For a value below 10^7 it computes m = round(abs * 10^8) and checks m / 10^8 == abs. The check is exact, because both operands are exact doubles and IEEE division rounds correctly with ties to even. Below 10^7 the rounding interval holds at most one decimal with 8 fractional digits, so a passing check gives the shortest decimal, the one Ryu returns, once the trailing zeros of m are stripped. Every other value goes to Ryu as before, so the output does not change. JMH on JDK 25, ns, mean of two interleaved passes, 1.3.12 / before / after: - 0.1, 3.25, 42.5, 101.75: 21.3 / 36.1 / 12.7 - random two-decimal values: 27.8 / 38.2 / 15.1 - full-mantissa values: 92.6 / 31.1 / 32.6 - integers below 10^7 (one pass): 17.5 / 12.5 / 14.2 - ILP text row with five short values: 155 / 233 / 129 JDK 8 and JDK 11 show the same order. Nothing allocates. The tests check the output against the shortest nearest decimal, computed with BigDecimal from its definition, on every JDK, and against Double.toString on JDK 19 and later. Also remove the helpers and tables of the pre-Ryu formatter, which nothing calls any more.
QwpSchemaSenderIntegrationTest called Thread.onSpinWait(), which exists only from Java 9 on, so the test tree did not compile on JDK 8 and the JDK 8 build-and-test CI jobs failed. Call Compat.onSpinWait(), as the other tests do.
After ALTER COLUMN TYPE widened a column, a sender whose every row needed the new type kept rejecting rows against its cached schema until the application flushed or restarted. The batch pins its schema while it holds pending rows, and a row that fails at a setter never reaches the at() that runs auto-flush, so no reply could report the change. Client 1.3.12 stored those rows. Now two consecutive rows the cached schema rejects for a column it knows, as an invalid value or one its type cannot take, make the next row of that table look the schema up again. The lookup waits at most one second, runs at most once a second per table, and only while the wire is up; a failed lookup keeps the cached schema. When the server reports a change, the sender flushes its pending rows, whose old types the server converts, and converts the row to the new types. A single bad value in an otherwise accepted batch still costs no lookup, and inputs no column type takes, such as LONG_ARRAY, never count. testIpv4TargetsUsePinnedTargetWireAndRollback now expects the one lookup its two consecutive rejected rows cause: the schema has not changed, so the row keeps the pinned wire.
Self-review of the re-check after a run of rejected rows found: - A re-check that got no answer timed the next one from its start. The wait lasts up to the interval, so a server that never answers lookups could hold the producer almost all the time on a table with bad data. A failed re-check now defers the next one from the end of its wait, through the same deferCheck() the age-based lookup uses, and either failure defers both. - The re-check skipped the lookup on a wire without schema support and first peeked at the cache. A lookup on such a wire fails at once, and the peek saved a round trip only when an ACK raced the pending row, a case no test can force. Both branches are gone; the loop that ensureConnected() installs is now asserted rather than null-checked. New tests cover a transactional sender, whose re-check flush keeps the transaction open; a recreated table, which changes the table id; a re-check with no pending rows; a lookup that gets no answer, which waits at most one second although schema_wait_millis is 60 s; rejections no type change explains, an invalid column name and an out-of-range designated timestamp, which look nothing up; and a run longer than the int range. Renames the test whose name said the opposite of what it checked: the sender does flush the pending rows itself.
Contributor
[PR Coverage check]😍 pass : 2836 / 3006 (94.34%) file detail
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.