Skip to content

Reuse one HTTP client for Solr requests instead of opening one per query - #310

Draft
gaurav wants to merge 8 commits into
mainfrom
reuse-solr-client
Draft

gaurav wants to merge 8 commits into
mainfrom
reuse-solr-client

Conversation

@gaurav

@gaurav gaurav commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

NameRes built a new httpx.AsyncClient for every Solr request: every /lookup, every /synonyms lookup and every /status call. Each one built an SSL context and opened a new TCP connection. It showed up in #231's /status as a recent_queries.solr_round_trip_ms many times solr.queries.handler_ms. This PR shares one client, whose connections stay open between requests, across NameRes's Solr lookups. Against a local Solr with the test data, mean /lookup time over 220 sequential lookups fell from 6.4 ms on main (39b0217) to 2.5 ms on 6a77ca4.

The calls worth overruling

  • The app's lifespan owns the shared client. It opens the client at startup and closes it on shutdown, and async with solr_client() hands it out only on the event loop the lifespan ran on. Uvicorn serves every request from that loop, so production shares one client. Anything else gets its own client, closed when the request finishes: a TestClient used without with (which starts a loop per request), or a script calling lookup() through asyncio.run(). Caching one client per loop in a WeakKeyDictionary doesn't work. A kept-alive connection refers back to its loop, so no finished loop, its client or its open socket was ever freed. The CLAUDE.md gotcha records this.
  • A broken connection is retried once; a timeout is not. Kept-alive connections bring a new failure: Solr restarts or its pod moves while a pooled connection to it is in use. httpx doesn't retry a POST, so solr_request() retries once on RemoteProtocolError, ReadError or WriteError, all of which are safe to resend because every request NameRes sends Solr only reads. It doesn't retry timeouts, which would each cost another SOLR_TIMEOUT_SECONDS, or ConnectError. httpcore already discards idle connections that Solr has closed, and httpx closes connections idle for more than 5 s, so the retry only covers a close that lands mid-request.
  • /status opens its own connection. It is what the Kubernetes probes call, so it should report whether Solr accepts a connection now, not fail on one a lookup left broken and get a healthy pod restarted. One connection per probe costs nothing worth saving.
  • No cap on the connection pool (max_connections=None), keeping up to SOLR_MAX_CONCURRENT_LOOKUPS connections open while idle. This matches the old behaviour, which had no limit either. A cap below what /bulk-lookup can have in flight would make queries wait for a connection inside SOLR_TIMEOUT_SECONDS, so a busy pool would look like Solr timeouts.

What's here

  • solr_client(), new_solr_client(), solr_request() and the app lifespan in api/server.py, replacing the four async with httpx.AsyncClient(...) blocks. The timeout is passed with each request, so changing config after startup still takes effect.
  • Tests that fake Solr, as the rest of tests/test_status.py does, and pin the following: lookups share the client, /status gets its own, every client is closed, the timeout follows config, and which errors are retried and how often.
  • Performance.md no longer describes the per-query connection as current behaviour. Deployment.md notes that a Kubernetes Service balances connections, not requests, so with several Solr replicas a long-lived connection stays on the replica it reached first. A CLAUDE.md gotcha says to send Solr requests through solr_client() and solr_request().

What it deliberately does not do

It doesn't cap the total connections across concurrent bulk requests; that is a separate decision. It doesn't balance requests across Solr replicas, which NameRes doesn't run today; Deployment.md says what to do if it ever does.

Verification

All 100-odd tests pass on 6a77ca4 against Solr 9.10. Five asyncio.run() lookups now leave no event loop alive after gc.collect(). The retry is tested only with faked errors; no one has restarted Solr under a running server to watch it happen.

Before merging

Nothing blocking.

🤖 Generated with Claude Code

Every Solr request (lookup(), the /synonyms lookup, and both /status calls) built its own
httpx.AsyncClient. That meant constructing an SSL context and opening a new TCP connection
each time. On a local Solr it took a 1 ms query to about 8 ms, which accounted for most
of the gap #231's /status showed between recent_queries.solr_round_trip_ms and
solr.queries.handler_ms. With the client shared, lookups on the same instance went from
a mean of 12.4 ms to 3.5 ms.

solr_client() hands out one client per running event loop, because an httpx client's
connections belong to the loop that opened them. Uvicorn has one loop, so production gets
one client. TestClient and asyncio.run() each start a fresh loop, and without the key
they would be handed connections from a loop that has already closed. The app's lifespan
closes the client on shutdown.

The pool is uncapped (max_connections=None), as it effectively was with a client per
query, and keeps up to SOLR_MAX_CONCURRENT_LOOKUPS idle connections alive. A cap below
what /bulk-lookup can have in flight would make queries wait for a connection inside
SOLR_TIMEOUT_SECONDS, so a busy pool would be reported as Solr timeouts. Bounding the
total across requests is a separate decision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Base automatically changed from track-recent-times to main November 10, 2026 23:56
gaurav added a commit that referenced this pull request Nov 10, 2026
…nce-diagnostics guide (#231)

This PR makes `/status` report how hard a NameRes instance and its Solr
backend are working. It adds latency and failure counts over a rolling
window of recent `/lookup` calls, and logs slow and failed lookups at
WARNING. Behind `?full=true`, it adds Solr's own query, cache, JVM and
OS metrics to the `solr` object that #304 introduced. That gives
monitoring something to alert on and us the inputs for sizing the Solr
pod, and a new guide, `documentation/Performance.md`, explains how to
read them. This addresses the `/status` part of #228 and supplies the
inputs for tuning issues #265 and #267.

## The calls worth overruling

- **Solr's metrics go under `solr`**, following #304's rule that
everything read from Solr goes there: `solr.queries`, `solr.cache`,
`solr.jvm` and `solr.os`. Only `recent_queries`, which NameRes times
itself, stays at the top level. They come from the core `/status`
reports on, cloud-mode cores included; if it has none, `metrics_error`
says so rather than borrowing another core's.
- **`os`, not `host` or `node`:** the figures are Solr's `os.*` gauges,
and in a container they are its limits (`--cpus 2 --memory 1g` on a
12-CPU machine reports 2 processors and 1024 MB).
- **Solr metrics only with `?full=true`** (the name v1.5.2 and #267
use), so the probes' plain `/status` makes no extra Solr round-trip. A
failed fetch is reported as `solr.metrics_error`, not omitted.
- **Latencies are named for the span they time.**
`recent_queries.lookup_ms` (all of `lookup()`),
`recent_queries.solr_round_trip_ms` (the Solr request as NameRes sees
it) and `solr.queries.handler_ms` (Solr's handler alone) each nest
inside the previous one. Matching flat names made a 12 ms lookup beside
a 1.35 ms Solr mean look like a contradiction; most of that gap is the
per-query connection #310 removes.
- **Latencies stay on the default path, cached for 10 seconds.** The
counts are running totals and always current. A series' mean, min, max
and percentiles need a pass over the window (about 4 ms at 50000
entries), so they are recomputed together at most every 10 s, and its
mean can never fall outside its min and max.
- **Failed lookups count in the latency window**, since their callers
waited too; otherwise `/status` would look healthiest exactly when Solr
is timing out. `errors`, `timeouts` and `cancelled` are also counted on
their own, because a fast failure such as a refused connection pulls the
percentiles *down*.
- **`/bulk-lookup` stays out of `recent_queries` and out of the WARNING
log**, since one request can hold thousands of strings. Its strings log
at INFO, including failures. When one string fails, the request is a 500
regardless, so the strings not yet done are now cancelled rather than
each sent to a Solr that is probably down, and the request logs one
`FAILED BULK QUERY` warning.
- **`gc_count`/`gc_time_ms` are pause totals.** They leave out
collectors that time concurrent work (JDK 20+'s `G1 Concurrent GC`,
ZGC's and Shenandoah's `… Cycles`), which would otherwise read as heap
pressure. Solr 9.10's image runs JDK 17, where nothing is excluded.
- **Deliberately left out:** OS page cache (Solr exposes only `MemFree`,
so #267 needs a node-level exporter), file descriptors, and #253's
per-second rates (sample the counters twice instead).

## What's here

- **`recent_queries`, on every `/status`:** the failure counts, plus
`lookup_ms` and `solr_round_trip_ms` with mean, min, max and
p50/p95/p99, over the last `RECENT_TIMES_COUNT` lookups (default 50000).
- **Logging:** a `/lookup` slower than `SLOW_QUERY_THRESHOLD_MS`
(default 500) logs `SLOW QUERY: …`, and a failed one `FAILED QUERY
(timeout|error|cancelled: …)`. The Solr figure now reads `(with Yms for
the Solr round trip)` where v1.7.1 wrote `waiting for Solr`;
TranslatorSRI/babel-validation#107's log parser accepts both. A
malformed setting logs a warning and falls back to its default.
- **With `?full=true`, from one `/admin/metrics` request:** `/select`
counts and `handler_ms`, `filterCache` and `queryResultCache`,
heap/CPU/GC, and processors/load/memory. Each value is checked on its
own, so one odd gauge (or a heap maximum of -1) nulls only its own
field.
- **Docs:**
[`API.md`](https://github.com/NCATSTranslator/NameResolution/blob/fc8c493/documentation/API.md#status)
gives the full response, and
[`Performance.md`](https://github.com/NCATSTranslator/NameResolution/blob/fc8c493/documentation/Performance.md)
is the diagnostics guide, adapted from #253, which went to `ci` and
won't reach `main`. A `CLAUDE.md` gotcha records how `/admin/metrics`
encodes and names things, GC beans included.

## Verification

All 90-odd tests pass on fc8c493 against Solr 9.10.
`tests/test_status.py` checks each metrics block for real values against
the test Solr, and fakes Solr for the failure paths and odd metrics. The
cloud registry name was read from a cloud-mode Solr 9.10, and the GC
bean names from Temurin 21 JVMs running G1, ZGC and Shenandoah.

## Before merging

Nothing blocking. `Performance.md`'s GC-log section describes the chart
as helxplatform/translator-devops#1044 left it: GC lines go to stdout,
which was confirmed in ITRB CI's Solr logs on 5 November.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
gaurav and others added 3 commits November 10, 2026 18:57
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tions once

The per-loop WeakKeyDictionary never let a finished loop's client go: a
kept-alive connection's transport refers back to its loop, so every loop
that made a request (each TestClient request without `with`, each
asyncio.run()) stayed alive with its client and an open socket to Solr,
and nothing could close them. Five asyncio.run() lookups left five loops
alive after gc.collect(); now none.

Instead, the lifespan opens one client and closes it on shutdown, and
`async with solr_client()` hands it out only on the loop the lifespan ran
on. Anywhere else gets a fresh client closed on exit.

Also from the review of #310:
- solr_request() passes the timeout per request, so a change to `config`
  after startup is obeyed, and retries once on RemoteProtocolError,
  ReadError or WriteError -- a pooled connection broken under the request,
  which reusing connections makes possible after a Solr restart. Timeouts
  and ConnectError are not retried.
- /status opens its own connection rather than using the lookups' pool, so
  a probe can't fail on a connection a lookup left broken.
- The client tests fake Solr like the rest of test_status.py, instead of
  needing a live one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Kubernetes Service balances connections, not requests, so with more than
one Solr replica a long-lived connection keeps going to the replica it
first reached. Deployment.md now says so. Performance.md had "kept-" and
"alive" on separate lines, which Markdown renders as "kept- alive".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

1 participant