Repository navigation
Conversation
Every Solr request (lookup(), the /synonyms lookup, and both /status calls) built its own httpx.AsyncClient. That meant constructing an SSL context and opening a new TCP connection each time. On a local Solr it took a 1 ms query to about 8 ms, which accounted for most of the gap #231's /status showed between recent_queries.solr_round_trip_ms and solr.queries.handler_ms. With the client shared, lookups on the same instance went from a mean of 12.4 ms to 3.5 ms. solr_client() hands out one client per running event loop, because an httpx client's connections belong to the loop that opened them. Uvicorn has one loop, so production gets one client. TestClient and asyncio.run() each start a fresh loop, and without the key they would be handed connections from a loop that has already closed. The app's lifespan closes the client on shutdown. The pool is uncapped (max_connections=None), as it effectively was with a client per query, and keeps up to SOLR_MAX_CONCURRENT_LOOKUPS idle connections alive. A cap below what /bulk-lookup can have in flight would make queries wait for a connection inside SOLR_TIMEOUT_SECONDS, so a busy pool would be reported as Solr timeouts. Bounding the total across requests is a separate decision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # tests/test_status.py
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gaurav
added a commit
that referenced
this pull request
Nov 10, 2026
…nce-diagnostics guide (#231) This PR makes `/status` report how hard a NameRes instance and its Solr backend are working. It adds latency and failure counts over a rolling window of recent `/lookup` calls, and logs slow and failed lookups at WARNING. Behind `?full=true`, it adds Solr's own query, cache, JVM and OS metrics to the `solr` object that #304 introduced. That gives monitoring something to alert on and us the inputs for sizing the Solr pod, and a new guide, `documentation/Performance.md`, explains how to read them. This addresses the `/status` part of #228 and supplies the inputs for tuning issues #265 and #267. ## The calls worth overruling - **Solr's metrics go under `solr`**, following #304's rule that everything read from Solr goes there: `solr.queries`, `solr.cache`, `solr.jvm` and `solr.os`. Only `recent_queries`, which NameRes times itself, stays at the top level. They come from the core `/status` reports on, cloud-mode cores included; if it has none, `metrics_error` says so rather than borrowing another core's. - **`os`, not `host` or `node`:** the figures are Solr's `os.*` gauges, and in a container they are its limits (`--cpus 2 --memory 1g` on a 12-CPU machine reports 2 processors and 1024 MB). - **Solr metrics only with `?full=true`** (the name v1.5.2 and #267 use), so the probes' plain `/status` makes no extra Solr round-trip. A failed fetch is reported as `solr.metrics_error`, not omitted. - **Latencies are named for the span they time.** `recent_queries.lookup_ms` (all of `lookup()`), `recent_queries.solr_round_trip_ms` (the Solr request as NameRes sees it) and `solr.queries.handler_ms` (Solr's handler alone) each nest inside the previous one. Matching flat names made a 12 ms lookup beside a 1.35 ms Solr mean look like a contradiction; most of that gap is the per-query connection #310 removes. - **Latencies stay on the default path, cached for 10 seconds.** The counts are running totals and always current. A series' mean, min, max and percentiles need a pass over the window (about 4 ms at 50000 entries), so they are recomputed together at most every 10 s, and its mean can never fall outside its min and max. - **Failed lookups count in the latency window**, since their callers waited too; otherwise `/status` would look healthiest exactly when Solr is timing out. `errors`, `timeouts` and `cancelled` are also counted on their own, because a fast failure such as a refused connection pulls the percentiles *down*. - **`/bulk-lookup` stays out of `recent_queries` and out of the WARNING log**, since one request can hold thousands of strings. Its strings log at INFO, including failures. When one string fails, the request is a 500 regardless, so the strings not yet done are now cancelled rather than each sent to a Solr that is probably down, and the request logs one `FAILED BULK QUERY` warning. - **`gc_count`/`gc_time_ms` are pause totals.** They leave out collectors that time concurrent work (JDK 20+'s `G1 Concurrent GC`, ZGC's and Shenandoah's `… Cycles`), which would otherwise read as heap pressure. Solr 9.10's image runs JDK 17, where nothing is excluded. - **Deliberately left out:** OS page cache (Solr exposes only `MemFree`, so #267 needs a node-level exporter), file descriptors, and #253's per-second rates (sample the counters twice instead). ## What's here - **`recent_queries`, on every `/status`:** the failure counts, plus `lookup_ms` and `solr_round_trip_ms` with mean, min, max and p50/p95/p99, over the last `RECENT_TIMES_COUNT` lookups (default 50000). - **Logging:** a `/lookup` slower than `SLOW_QUERY_THRESHOLD_MS` (default 500) logs `SLOW QUERY: …`, and a failed one `FAILED QUERY (timeout|error|cancelled: …)`. The Solr figure now reads `(with Yms for the Solr round trip)` where v1.7.1 wrote `waiting for Solr`; TranslatorSRI/babel-validation#107's log parser accepts both. A malformed setting logs a warning and falls back to its default. - **With `?full=true`, from one `/admin/metrics` request:** `/select` counts and `handler_ms`, `filterCache` and `queryResultCache`, heap/CPU/GC, and processors/load/memory. Each value is checked on its own, so one odd gauge (or a heap maximum of -1) nulls only its own field. - **Docs:** [`API.md`](https://github.com/NCATSTranslator/NameResolution/blob/fc8c493/documentation/API.md#status) gives the full response, and [`Performance.md`](https://github.com/NCATSTranslator/NameResolution/blob/fc8c493/documentation/Performance.md) is the diagnostics guide, adapted from #253, which went to `ci` and won't reach `main`. A `CLAUDE.md` gotcha records how `/admin/metrics` encodes and names things, GC beans included. ## Verification All 90-odd tests pass on fc8c493 against Solr 9.10. `tests/test_status.py` checks each metrics block for real values against the test Solr, and fakes Solr for the failure paths and odd metrics. The cloud registry name was read from a cloud-mode Solr 9.10, and the GC bean names from Temurin 21 JVMs running G1, ZGC and Shenandoah. ## Before merging Nothing blocking. `Performance.md`'s GC-log section describes the chart as helxplatform/translator-devops#1044 left it: GC lines go to stdout, which was confirmed in ITRB CI's Solr logs on 5 November. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tions once The per-loop WeakKeyDictionary never let a finished loop's client go: a kept-alive connection's transport refers back to its loop, so every loop that made a request (each TestClient request without `with`, each asyncio.run()) stayed alive with its client and an open socket to Solr, and nothing could close them. Five asyncio.run() lookups left five loops alive after gc.collect(); now none. Instead, the lifespan opens one client and closes it on shutdown, and `async with solr_client()` hands it out only on the loop the lifespan ran on. Anywhere else gets a fresh client closed on exit. Also from the review of #310: - solr_request() passes the timeout per request, so a change to `config` after startup is obeyed, and retries once on RemoteProtocolError, ReadError or WriteError -- a pooled connection broken under the request, which reusing connections makes possible after a Solr restart. Timeouts and ConnectError are not retried. - /status opens its own connection rather than using the lookups' pool, so a probe can't fail on a connection a lookup left broken. - The client tests fake Solr like the rest of test_status.py, instead of needing a live one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Kubernetes Service balances connections, not requests, so with more than one Solr replica a long-lived connection keeps going to the replica it first reached. Deployment.md now says so. Performance.md had "kept-" and "alive" on separate lines, which Markdown renders as "kept- alive". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NameRes built a new
httpx.AsyncClientfor every Solr request: every/lookup, every/synonymslookup and every/statuscall. Each one built an SSL context and opened a new TCP connection. It showed up in #231's/statusas arecent_queries.solr_round_trip_msmany timessolr.queries.handler_ms. This PR shares one client, whose connections stay open between requests, across NameRes's Solr lookups. Against a local Solr with the test data, mean/lookuptime over 220 sequential lookups fell from 6.4 ms onmain(39b0217) to 2.5 ms on 6a77ca4.The calls worth overruling
async with solr_client()hands it out only on the event loop the lifespan ran on. Uvicorn serves every request from that loop, so production shares one client. Anything else gets its own client, closed when the request finishes: aTestClientused withoutwith(which starts a loop per request), or a script callinglookup()throughasyncio.run(). Caching one client per loop in aWeakKeyDictionarydoesn't work. A kept-alive connection refers back to its loop, so no finished loop, its client or its open socket was ever freed. TheCLAUDE.mdgotcha records this.solr_request()retries once onRemoteProtocolError,ReadErrororWriteError, all of which are safe to resend because every request NameRes sends Solr only reads. It doesn't retry timeouts, which would each cost anotherSOLR_TIMEOUT_SECONDS, orConnectError. httpcore already discards idle connections that Solr has closed, and httpx closes connections idle for more than 5 s, so the retry only covers a close that lands mid-request./statusopens its own connection. It is what the Kubernetes probes call, so it should report whether Solr accepts a connection now, not fail on one a lookup left broken and get a healthy pod restarted. One connection per probe costs nothing worth saving.max_connections=None), keeping up toSOLR_MAX_CONCURRENT_LOOKUPSconnections open while idle. This matches the old behaviour, which had no limit either. A cap below what/bulk-lookupcan have in flight would make queries wait for a connection insideSOLR_TIMEOUT_SECONDS, so a busy pool would look like Solr timeouts.What's here
solr_client(),new_solr_client(),solr_request()and the app lifespan inapi/server.py, replacing the fourasync with httpx.AsyncClient(...)blocks. The timeout is passed with each request, so changingconfigafter startup still takes effect.tests/test_status.pydoes, and pin the following: lookups share the client,/statusgets its own, every client is closed, the timeout followsconfig, and which errors are retried and how often.Performance.mdno longer describes the per-query connection as current behaviour.Deployment.mdnotes that a Kubernetes Service balances connections, not requests, so with several Solr replicas a long-lived connection stays on the replica it reached first. ACLAUDE.mdgotcha says to send Solr requests throughsolr_client()andsolr_request().What it deliberately does not do
It doesn't cap the total connections across concurrent bulk requests; that is a separate decision. It doesn't balance requests across Solr replicas, which NameRes doesn't run today;
Deployment.mdsays what to do if it ever does.Verification
All 100-odd tests pass on 6a77ca4 against Solr 9.10. Five
asyncio.run()lookups now leave no event loop alive aftergc.collect(). The retry is tested only with faked errors; no one has restarted Solr under a running server to watch it happen.Before merging
Nothing blocking.
🤖 Generated with Claude Code