Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions documentation/changelog.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ This page tracks significant updates to the QuestDB documentation.

### New

- [OIDC device flow](/docs/security/oidc/#device-authorization-flow) and [rotating bearer tokens](/docs/security/oidc/#rotating-bearer-tokens) - Sign in interactively or use expiring application credentials with the C, C++, and Rust 7.3.0+ clients and Python 6.1.0+. Covers discovery, TLS trust, token storage, refresh, recovery, and the [Python PGWire adapters](/docs/connect/clients/python/#pgwire-adapters)
- [Passthrough materialized views](/docs/concepts/materialized-views/#passthrough-views) - A materialized view no longer has to aggregate: a query without `SAMPLE BY` copies base rows one-for-one, so the view is a live copy of some columns or rows of one table. Covers what it inherits from the base table (designated timestamp, partitioning, symbol indexes), why its columns are fixed at creation, [which queries qualify](/docs/concepts/materialized-views/#which-queries-are-passthrough), and that `REFRESH PERIOD` is rejected
- [dbt connector](/docs/connect/connectors/dbt/) - New page for the QuestDB Enterprise dbt adapter, which builds, tests, and documents SQL models over QWP. Covers installation from the release archive, a TLS profile for the bundled trades project, SQL transformations from staging to joined and aggregated models, views, tables, incremental upserts, materialized views with timer and period refresh, snapshots, reading ingested tables such as the Databento connector's as dbt sources, connection and model settings, and limitations
- [Expiring rows (`EXPIRE ROWS`)](/docs/concepts/expire-rows/) and [ALTER MATERIALIZED VIEW SET EXPIRE](/docs/query/sql/alter-mat-view-set-expire/) - Keep only some rows in a materialized view. Set it when you create the view (see [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/)) or add it later. Queries hide expired rows right away, and a background job frees their disk space for the policies where that is safe.
Expand All @@ -38,6 +39,7 @@ This page tracks significant updates to the QuestDB documentation.

### Updated

- [Native and Python client startup](/docs/connect/clients/connect-string/#reconnect-keys) and [authentication failures](/docs/high-availability/client-failover/concepts/#authentication-is-cluster-wide) - Pools require an explicit initial-connect retry mode rather than inferring it from reconnect settings. Documents eager reader prewarming, established senders retrying rejected credentials, and terminal protocol-version failures in native 7.3.0+/Python 6.1.0+
- [REINDEX](/docs/query/sql/reindex/) - Explains that `LOCK EXCLUSIVE` is required and takes the table's writer lock, and that `REINDEX` rebuilds native partitions only: it skips Parquet partitions and rejects a Parquet partition named in `PARTITION`
- [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/#limit-and-window-function-restrictions) - Breaking change in QuestDB 10.2.0 and QuestDB Enterprise 4.2.0: the part of a materialized view's query that reads its base table (this includes a subquery on the base table) can no longer use `LIMIT`, or a window function nested inside an expression, such as `row_number() OVER (ORDER BY timestamp) + 1`. A window function that makes up a whole column was already rejected. A `LIMIT` in a subquery over a *different* table is still allowed
- [ALTER TABLE SET TYPE](/docs/query/sql/alter-table-set-type/) - A conversion to non-WAL applies the WAL first, and is refused while it cannot
Expand Down
379 changes: 349 additions & 30 deletions documentation/connect/clients/c-and-cpp.md

Large diffs are not rendered by default.

114 changes: 77 additions & 37 deletions documentation/connect/clients/connect-string.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,12 +149,20 @@ wss::addr=node-a:443,node-b:443;target=replica;zone=eu-west-1a;
### Tolerate a slow or restarting server at startup

```
ws::addr=node-a:9000;reconnect_max_duration_millis=120000;
ws::addr=node-a:9000;initial_connect_retry=on;reconnect_max_duration_millis=120000;
```

The 2-minute reconnect budget covers both the *first* connect and any
subsequent reconnect: setting any explicit `reconnect_*` key implicitly
turns on `initial_connect_retry`. See
The 2-minute budget bounds the blocking initial **ingestion** connection.
Once connected, the sender retries transport outages indefinitely. Set
`initial_connect_retry` explicitly: `reconnect_*` keys alone do not enable
startup retry for native 7.3.0 and Python 6.1.0 or later pools.

Pools also prewarm readers when `query_pool_min` is nonzero; their initial
connections do not use this ingestion retry policy. Set `query_pool_min=0`
to defer reader connections until first use. For background ingestion startup
instead of blocking retry, use `lazy_connect=true`, which also defaults
`query_pool_min` to `0` in native/Python pools; do not combine it with an
explicit blocking initial-connect mode. See
[Ingress reconnect](#reconnect-keys).

## Recipes {#recipes}
Expand Down Expand Up @@ -182,8 +190,8 @@ caveats), follow the section links from the [Key index](#key-index).
| Orphan recovery for crashed senders | ingress | `drain_orphans=on` | `max_background_drainers` |
| Confirm rows are on the server's disk | ingress | `request_durable_ack=local` | `durable_ack_keepalive_interval_millis`; the server needs adaptive durability mode |
| Confirm rows reached replication storage | ingress | `request_durable_ack=replicated` | `on` for QuestDB Enterprise 4.1.0 and earlier; `durable_ack_keepalive_interval_millis` |
| Tune ingress reconnect backoff | ingress | — | `reconnect_initial_backoff_millis`, `reconnect_max_backoff_millis`, `reconnect_max_duration_millis` (any of these also implies `initial_connect_retry=on`) |
| Force fail-fast on initial connect | ingress | `initial_connect_retry=off` | overrides the implicit promotion from any explicit `reconnect_*` key |
| Tune ingress reconnect backoff | ingress | — | `reconnect_initial_backoff_millis`, `reconnect_max_backoff_millis`, `reconnect_max_duration_millis`; set `initial_connect_retry` explicitly for portable startup behavior |
| Force fail-fast on initial connect | ingress | `initial_connect_retry=off` | prevents implicit promotion in clients that support it; native 7.3.0+/Python 6.1.0+ pools already default to fail-fast |
| Retry initial connect in background | ingress | `initial_connect_retry=async` | `reconnect_*` |
| Fast `close()` without drain | ingress | `close_flush_timeout_millis=0` | — |
| Disable per-query egress failover | egress | `failover=off` | — |
Expand Down Expand Up @@ -242,12 +250,33 @@ the TLS handshake is server-authenticated only. `tls_roots` /

### Token providers {#token-providers}

*Applies to: ingress and egress. Java client 1.3.11 or later only.*
*Applies to: ingress and egress.*

A fixed `token` does not follow credential rotation. A token provider supplies
a current bearer token for each connection, reconnect, and failover without
rebuilding the client. The C, C++, Rust, and Python clients accept providers
through their in-code APIs, not this configuration string:

- [C and C++](/docs/connect/clients/c-and-cpp/#rotating-bearer-tokens): native
client 7.3.0 or later.
- [Rust](/docs/connect/clients/rust/#rotating-bearer-tokens): `questdb-rs` 7.3.0
or later.
- [Python](/docs/connect/clients/python/#rotating-bearer-tokens): client 6.1.0
or later.

These clients also support interactive
[OIDC device sign-in](/docs/security/oidc/#device-authorization-flow). A
provider cannot be combined with fixed `token`, `username`, or `password`
credentials, or with a second provider. The `token_provider`, `azure_resource`,
`azure_client_id`, and `azure_credential` keys below are rejected by these
clients on every connection-string scheme. Use `wss` for QWP or `https` for
ILP so the bearer token is encrypted in transit.

#### Java connection-string providers

A fixed `token` stops working when it expires. A token provider requests
bearer tokens from an identity provider instead, and renews them in the
background before they expire, so every connection, reconnect, and failover
presents a valid token:
The following provider keys are available in **Java client 1.3.11 or later
only**. Java can select an Azure provider from the connection string and
refresh its tokens in the background:

```
wss::addr=node-a:9000,node-b:9000;token_provider=azure;azure_resource=api://<application ID>;azure_credential=managed_identity;
Expand Down Expand Up @@ -700,16 +729,13 @@ architecture is that a producer survives an arbitrarily long outage.

- `reconnect_initial_backoff_millis` — initial wait between reconnect
attempts. Backoff grows exponentially up to `reconnect_max_backoff_millis`.
Default: `100`. Setting this enables `initial_connect_retry=on` implicitly;
see below.
Default: `100`.
- `reconnect_max_backoff_millis` — cap on per-attempt backoff.
Default: `5000` (5 s). Setting this enables
`initial_connect_retry=on` implicitly; see below.
Default: `5000` (5 s).
- `reconnect_max_duration_millis` — time budget for the **blocking sync
initial connect** (`initial_connect_retry=on`): once exceeded, the
constructor gives up and returns the error. The running loop and the
`async` initial connect never consult it. Default: `300000` (5 min).
Setting this enables `initial_connect_retry=on` implicitly; see below.
- `initial_connect_retry` — whether the client retries the initial connect
attempt on failure.
- `off` (default, alias `false`) — fail fast on initial connect failure.
Expand All @@ -719,13 +745,16 @@ architecture is that a producer survives an arbitrarily long outage.
the background indefinitely, surfacing only genuine terminal failures
(auth reject, durable-ack mismatch) via the error inbox.

**Implicit promotion.** Setting any explicit `reconnect_*` key without
also choosing an `initial_connect_retry` mode promotes
`initial_connect_retry` to `on` automatically, so the budget also covers
the *first* connect attempt — without the promotion,
`reconnect_max_duration_millis` would be inert. To keep fail-fast
behaviour on the first connect while still tuning the backoff, set
`initial_connect_retry=off` explicitly; the explicit setting wins.
**Implicit promotion is client-specific.** Some clients, including Java
and standalone native senders, promote the mode to `on` when an explicit
`reconnect_*` key is set without an `initial_connect_retry` mode. Pools in
C, C++, and Rust 7.3.0 or later, and Python 6.1.0 or later, do **not** apply
that promotion: they honor only an explicitly set mode and otherwise fail
fast. Set `initial_connect_retry=on` explicitly for blocking ingestion
startup retry, or `off` to retain fail-fast behavior while tuning backoff.
Neither `on` nor `async` changes reader prewarming: a nonzero
`query_pool_min` still creates readers synchronously and can fail pool
construction.
- `close_flush_timeout_millis` — `close()` blocks up to this many
milliseconds waiting for buffered rows to drain. Set to `0` or `-1` for
fast close (skip the drain). **The default differs by client**: `60000`
Expand All @@ -736,20 +765,31 @@ architecture is that a producer survives an arbitrarily long outage.
entirely: in memory-only mode, un-ACKed batches are lost on every clean
shutdown; in SF mode they stay in the sender directory for the next sender.

In most clients, rejections of the upgrade request during reconnect
(authentication rejected, version mismatch, durable-ack mismatch, non-101
upgrade without a role hint) are immediately terminal: the loop does not
retry them. The Java client treats a version mismatch like a transport
failure, to ride out a rolling upgrade, and the other rejections as terminal
only before the sender first connects. A Java sender that has connected keeps
retrying all of them, and reports each failed attempt in its connection
health. It also reports the rejections other than a version mismatch to its
error handler. A Node.js sender that uses store-and-forward or
`initial_connect_retry=async` also keeps retrying all of them once it has
connected, and reports each failed attempt as a connection event. See
[How the client handles token failures](/docs/connect/clients/java/#how-the-client-handles-token-failures)
and, for orphan drainers,
Upgrade rejections have client- and phase-specific retry policies:

- In C, C++, and Rust **7.3.0 or later**, and Python **6.1.0 or later**, a
caller-awaited initial connection fails on HTTP `401` / `403`. A sender
that has connected keeps retrying those rejections with capped backoff,
even with fixed credentials, retaining unacknowledged rows. Each rejection
ends the current endpoint walk, not the sender's reconnect loop, and is
reported as `auth_failed` / `AuthFailed`. This exception does **not** cover
protocol-version failures, which remain terminal in these clients.
- Java treats a version mismatch like a transport failure to ride out a
rolling upgrade. Other rejections (authentication, durable-ack mismatch,
non-101 upgrade without a role hint) are terminal only before the sender
first connects. Once connected, it retries all of them and reports each
failed attempt in its connection health; rejections other than a version
mismatch also reach its error handler.
- A Node.js sender using store-and-forward or `initial_connect_retry=async`
also retries these rejections once it has connected, reporting each failed
attempt as a connection event.

Do not assume that an authentication exception makes all upgrade failures
retryable. For the full phase table, browser behavior, and orphan drainers,
see
[Authentication failures and token providers](/docs/connect/wire-protocols/qwp-client-behavior/#authentication-failures-and-token-providers).
For Java, see also
[How the client handles token failures](/docs/connect/clients/java/#how-the-client-handles-token-failures).

### Egress failover {#egress-failover}

Expand Down Expand Up @@ -1019,7 +1059,7 @@ description and behaviour notes.
| `failover_max_attempts` | int | `8` | [Egress failover](#reconnect-keys) |
| `failover_max_duration_ms` | int (ms) | `30000` | [Egress failover](#reconnect-keys) |
| `init_buf_size` | size | `65536` (64 KiB) | [Buffer sizing](#buffer) |
| `initial_connect_retry` | enum (`off` / `on` / `async`) | `off` (auto-promoted to `on` when any explicit `reconnect_*` key is set) | [Ingress reconnect](#reconnect-keys) |
| `initial_connect_retry` | enum (`off` / `on` / `async`) | `off`; implicit promotion is client-specific, not applied by native 7.3.0+/Python 6.1.0+ pools | [Ingress reconnect](#reconnect-keys) |
| `initial_credit` | int (bytes) | `0` (unbounded) | [Query client keys](#egress-keys) |
| `housekeeper_interval_ms` | int (ms) | `5000` | [Connection pool](#pool-keys) |
| `idle_timeout_ms` | int (ms) | `60000` (`0` ⇒ infinite) | [Connection pool](#pool-keys) |
Expand Down
Loading
Loading