From 0981983ce91cf2cf0e4d0ee47e941923f00f052a Mon Sep 17 00:00:00 2001 From: gaurav Date: Tue, 10 Nov 2026 17:24:12 -0500 Subject: [PATCH 1/3] Reuse one HTTP client for Solr requests instead of one per query Every Solr request (lookup(), the /synonyms lookup, and both /status calls) built its own httpx.AsyncClient. That meant constructing an SSL context and opening a new TCP connection each time. On a local Solr it took a 1 ms query to about 8 ms, which accounted for most of the gap #231's /status showed between recent_queries.solr_round_trip_ms and solr.queries.handler_ms. With the client shared, lookups on the same instance went from a mean of 12.4 ms to 3.5 ms. solr_client() hands out one client per running event loop, because an httpx client's connections belong to the loop that opened them. Uvicorn has one loop, so production gets one client. TestClient and asyncio.run() each start a fresh loop, and without the key they would be handed connections from a loop that has already closed. The app's lifespan closes the client on shutdown. The pool is uncapped (max_connections=None), as it effectively was with a client per query, and keeps up to SOLR_MAX_CONCURRENT_LOOKUPS idle connections alive. A cap below what /bulk-lookup can have in flight would make queries wait for a connection inside SOLR_TIMEOUT_SECONDS, so a busy pool would be reported as Solr timeouts. Bounding the total across requests is a separate decision. Co-Authored-By: Claude Opus 5.5 --- CLAUDE.md | 2 + api/server.py | 99 ++++++++++++++++++++++++++---------- documentation/Performance.md | 9 ++-- tests/test_status.py | 36 +++++++++++++ 4 files changed, 115 insertions(+), 31 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 27e8cdd8..699dd678 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -105,6 +105,8 @@ Solr documents contain: `curie`, `preferred_name`, `names` (synonym list), and b - **Declaring metadata in `openapi.yml` is not enough to serve it.** `construct_open_api_schema()` copies an explicit allowlist of `info` keys into the document (and `get_app_info()` a narrower one for the `FastAPI()` constructor); anything not named there is dropped without a word. That is how `info.contact` and `info.license` sat declared-but-unserved for years. Adding a key means adding it to the copy list *and* asserting it in `tests/test_openapi.py`. - **Check a new Solr `/admin/metrics` field against a live Solr before trusting it.** The encoding varies by metric type, and a wrong guess silently yields `null` rather than an error. Counters (`QUERY./select.requests`) are scalars, timers (`requestTimes`) and meters (`errors`/`timeouts`) are nested objects, and JVM gauges (`memory.heap.used`) come back as *flat dotted keys*, not a nested `memory.heap` map. `status()` reads them accordingly. Names matter as much as encodings: Solr 9 registers filterCache and queryResultCache as `CACHE.searcher.*`, and reading `CACHE.core.*` returned `null` for every cache field while a key-presence test kept passing -- so test that a metric has a value, not just that its key exists. Registry names vary too: a standalone core's figures are under `solr.core.`, but a SolrCloud core's are under `solr.core...` (`name_lookup_shard1_replica_n1` reports as `solr.core.name_lookup.shard1.replica_n1`); `core_metrics_registry()` resolves both, and reports a `metrics_error` rather than nulls when neither is there. +- **Send Solr requests through `solr_client()`, not a new `httpx.AsyncClient`.** A client per query builds an SSL context and opens a TCP connection every time, which took a 1 ms Solr query to about 8 ms as NameRes measured it. `solr_client()` hands out one client per event loop (an httpx client's connections belong to the loop that opened them, and `TestClient` and `asyncio.run()` start fresh loops), with no connection cap so that a busy pool can't surface as Solr timeouts; the app's lifespan closes it. Tests that fake Solr patch `httpx.AsyncClient.get`/`post` on the class, which still reaches the shared instance. + ## Documentation - `documentation/API.md` - Endpoint reference - `documentation/Performance.md` - Reading the `/status` metrics; diagnosing Solr CPU/memory/load diff --git a/api/server.py b/api/server.py index 8645e778..2773e7b1 100755 --- a/api/server.py +++ b/api/server.py @@ -9,10 +9,12 @@ import logging import statistics import warnings +import weakref import time import os import re from collections import Counter, deque +from contextlib import asynccontextmanager from dataclasses import dataclass from enum import Enum from typing import Dict, List, Union, Annotated, Optional @@ -76,7 +78,50 @@ def public(self) -> dict: config = Config() -app = FastAPI(**get_app_info()) + +# One httpx client per event loop, shared by every Solr request, so that requests reuse +# kept-alive connections. A new client per query (as lookup() used to make) builds an SSL +# context and opens a TCP connection each time: on a local Solr that took a 1 ms query to about +# 8 ms, most of what /status reported as solr_round_trip_ms. +# +# Keyed by loop because an httpx client's connections belong to the loop that opened them. +# Uvicorn serves everything from one loop, so in production this is a single client; the +# key matters for TestClient and asyncio.run(), which start a fresh loop per call, and +# would otherwise be handed connections from a loop that has gone. Weak keys let a dead +# loop's client go with it. +_solr_clients: "weakref.WeakKeyDictionary[asyncio.AbstractEventLoop, httpx.AsyncClient]" = \ + weakref.WeakKeyDictionary() + + +def solr_client() -> httpx.AsyncClient: + """ The shared Solr client for the running event loop, created on first use. """ + loop = asyncio.get_running_loop() + client = _solr_clients.get(loop) + if client is None or client.is_closed: + client = httpx.AsyncClient( + timeout=config.solr_timeout, + # No cap on connections, as when each query had its own client: the in-flight + # bound is SOLR_MAX_CONCURRENT_LOOKUPS per /bulk-lookup request, and a pool cap + # below that would make queries wait for a connection -- inside + # SOLR_TIMEOUT_SECONDS, so a busy pool would show up as Solr timeouts. Keep up + # to that many idle connections alive for reuse. + limits=httpx.Limits(max_connections=None, + max_keepalive_connections=config.solr_max_concurrent_lookups), + ) + _solr_clients[loop] = client + return client + + +@asynccontextmanager +async def lifespan(app: FastAPI): + """ Close the shared Solr client when the server shuts down. """ + yield + client = _solr_clients.pop(asyncio.get_running_loop(), None) + if client is not None: + await client.aclose() + + +app = FastAPI(**get_app_info(), lifespan=lifespan) logger = logging.getLogger(__name__) logging.basicConfig(level=os.getenv("LOGLEVEL", logging.INFO)) @@ -297,21 +342,21 @@ async def fetch_solr_metrics(core_name: Optional[str], core: Optional[Dict] = No """ metrics_url = f"http://{config.solr_host}:{config.solr_port}/solr/admin/metrics" try: - async with httpx.AsyncClient(timeout=config.solr_timeout) as client: - # A single call with group=core&group=jvm retrieves both in one round-trip. - metrics_resp = await client.get(metrics_url, params=[ - ('group', 'core'), - ('group', 'jvm'), - ('prefix', 'QUERY./select'), - # Solr 9 registers the searcher caches as CACHE.searcher.*, not - # CACHE.core.* -- a wrong name here yields nulls, not an error. - ('prefix', 'CACHE.searcher.filterCache'), - ('prefix', 'CACHE.searcher.queryResultCache'), - ('prefix', 'memory.heap'), - ('prefix', 'gc.'), - ('prefix', 'os.'), - ('wt', 'json'), - ]) + client = solr_client() + # A single call with group=core&group=jvm retrieves both in one round-trip. + metrics_resp = await client.get(metrics_url, params=[ + ('group', 'core'), + ('group', 'jvm'), + ('prefix', 'QUERY./select'), + # Solr 9 registers the searcher caches as CACHE.searcher.*, not + # CACHE.core.* -- a wrong name here yields nulls, not an error. + ('prefix', 'CACHE.searcher.filterCache'), + ('prefix', 'CACHE.searcher.queryResultCache'), + ('prefix', 'memory.heap'), + ('prefix', 'gc.'), + ('prefix', 'os.'), + ('wt', 'json'), + ]) if metrics_resp.status_code >= 300: logger.warning("Solr error on accessing /solr/admin/metrics: HTTP %d: %s", metrics_resp.status_code, metrics_resp.text) @@ -435,13 +480,13 @@ async def status_get( async def status(full: bool = False) -> Dict: """ Return a dictionary containing status and count information for the underlying Solr instance. """ query_url = f"http://{config.solr_host}:{config.solr_port}/solr/admin/cores" - async with httpx.AsyncClient(timeout=config.solr_timeout) as client: - response = await client.get(query_url, params={ - 'action': 'STATUS' - }) - if response.status_code >= 300: - logger.error("Solr error on accessing /solr/admin/cores?action=STATUS: %s", response.text) - response.raise_for_status() + client = solr_client() + response = await client.get(query_url, params={ + 'action': 'STATUS' + }) + if response.status_code >= 300: + logger.error("Solr error on accessing /solr/admin/cores?action=STATUS: %s", response.text) + response.raise_for_status() result = response.json() @@ -659,8 +704,8 @@ async def name_lookup(curies) -> Dict[str, Dict]: "query": curie_filter, "limit": 1000000, } - async with httpx.AsyncClient(timeout=config.solr_timeout) as client: - response = await client.post(query, json=params) + client = solr_client() + response = await client.post(query, json=params) response.raise_for_status() response_json = response.json() output = { @@ -1088,8 +1133,8 @@ async def lookup(string: str, time_solr_start = time.time_ns() query_url = f"http://{config.solr_host}:{config.solr_port}/solr/{config.solr_core}/select" try: - async with httpx.AsyncClient(timeout=config.solr_timeout) as client: - response = await client.post(query_url, json=params) + client = solr_client() + response = await client.post(query_url, json=params) if response.status_code >= 300: logger.error("Solr REST error: %s", response.text) response.raise_for_status() diff --git a/documentation/Performance.md b/documentation/Performance.md index 206ff107..9eb613eb 100644 --- a/documentation/Performance.md +++ b/documentation/Performance.md @@ -32,9 +32,10 @@ Three latencies nest inside one another, and the fields are named for which one | Solr handler | `solr.queries.handler_ms` (with `?full=true`) | Only the time inside Solr's `/select` handler, as Solr's own timer measures it. | So `lookup_ms` − `solr_round_trip_ms` is NameRes's result processing, and `solr_round_trip_ms` − -`handler_ms` is connection, transfer and JSON overhead. That second gap is large: `lookup()` -currently opens a new HTTP client and connection per query, which on a local Solr made a 1 ms -query take about 8 ms. +`handler_ms` is connection, transfer and JSON overhead. NameRes shares one HTTP client with kept- +alive connections across its Solr requests, so that second gap should be small; a large one points +at the network between the API and Solr, or at large responses. (When `lookup()` opened a new +client and connection per query, that overhead took a 1 ms query on a local Solr to about 8 ms.) | Field | What it means | |---|---| @@ -207,7 +208,7 @@ Solr seems slow or the service is unresponsive │ ├─ solr_round_trip_ms is high but handler_ms is small │ │ → the time is between NameRes and Solr: connection setup, network, transfer, │ │ JSON decoding. Check: network between the API and Solr pods; large responses -│ │ (`limit`, highlighting). lookup() opens a new connection per query today. +│ │ (`limit`, highlighting). │ │ │ └─ handler_ms is most of solr_round_trip_ms → problem is INSIDE Solr → continue │ diff --git a/tests/test_status.py b/tests/test_status.py index 58132d1e..3d905403 100644 --- a/tests/test_status.py +++ b/tests/test_status.py @@ -527,3 +527,39 @@ def test_diagnostic_setting_reads_valid_values(monkeypatch): assert diagnostic_setting("NAMERES_TEST_SETTING", 7, int) == 0 monkeypatch.setenv("NAMERES_TEST_SETTING", "250.5") assert math.isclose(diagnostic_setting("NAMERES_TEST_SETTING", 7.0, float), 250.5) + + +def test_solr_client_is_shared_within_an_event_loop(): + """Solr requests reuse one client, and so its kept-alive connections, rather than + paying for a new client and connection per query. A new loop gets its own client, + since connections belong to the loop that opened them.""" + import api.server + + async def two(): + return api.server.solr_client(), api.server.solr_client() + + first, again = asyncio.run(two()) + assert first is again + other, _ = asyncio.run(two()) + assert other is not first + + +def test_shared_solr_client_is_reused_and_closed_on_shutdown(monkeypatch): + import api.server + handed_out = [] + real_solr_client = api.server.solr_client + + def spy(): + handed_out.append(real_solr_client()) + return handed_out[-1] + monkeypatch.setattr(api.server, "solr_client", spy) + + # As a context manager, TestClient runs the app's lifespan and serves every request + # from one event loop, as uvicorn does. + with TestClient(app) as client: + client.get("/lookup", params={'string': 'alzheimer'}) + client.get("/lookup", params={'string': 'diabetes'}) + client.get("/status") + assert not handed_out[0].is_closed + assert len(handed_out) == 3 and all(c is handed_out[0] for c in handed_out) + assert handed_out[0].is_closed From facb0e0ebb139a5eafa886bacb8eb8bf2ed522fe Mon Sep 17 00:00:00 2001 From: gaurav Date: Wed, 11 Nov 2026 01:18:16 -0500 Subject: [PATCH 2/3] Own the shared Solr client from the lifespan, and retry broken connections once The per-loop WeakKeyDictionary never let a finished loop's client go: a kept-alive connection's transport refers back to its loop, so every loop that made a request (each TestClient request without `with`, each asyncio.run()) stayed alive with its client and an open socket to Solr, and nothing could close them. Five asyncio.run() lookups left five loops alive after gc.collect(); now none. Instead, the lifespan opens one client and closes it on shutdown, and `async with solr_client()` hands it out only on the loop the lifespan ran on. Anywhere else gets a fresh client closed on exit. Also from the review of #310: - solr_request() passes the timeout per request, so a change to `config` after startup is obeyed, and retries once on RemoteProtocolError, ReadError or WriteError -- a pooled connection broken under the request, which reusing connections makes possible after a Solr restart. Timeouts and ConnectError are not retried. - /status opens its own connection rather than using the lookups' pool, so a probe can't fail on a connection a lookup left broken. - The client tests fake Solr like the rest of test_status.py, instead of needing a live one. Co-Authored-By: Claude Opus 5.5 --- CLAUDE.md | 2 +- api/server.py | 184 ++++++++++++++++++++++++++++--------------- tests/test_status.py | 132 +++++++++++++++++++++++++------ 3 files changed, 229 insertions(+), 89 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index fee25956..60273b02 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -105,7 +105,7 @@ Solr documents contain: `curie`, `preferred_name`, `names` (synonym list), and b - **Declaring metadata in `openapi.yml` is not enough to serve it.** `construct_open_api_schema()` copies an explicit allowlist of `info` keys into the document (and `get_app_info()` a narrower one for the `FastAPI()` constructor); anything not named there is dropped without a word. That is how `info.contact` and `info.license` sat declared-but-unserved for years. Adding a key means adding it to the copy list *and* asserting it in `tests/test_openapi.py`. - **Check a new Solr `/admin/metrics` field against a live Solr before trusting it.** The encoding varies by metric type, and a wrong guess silently yields `null` rather than an error. Counters (`QUERY./select.requests`) are scalars, timers (`requestTimes`) and meters (`errors`/`timeouts`) are nested objects, and JVM gauges (`memory.heap.used`) come back as *flat dotted keys*, not a nested `memory.heap` map. `status()` reads them accordingly. The `gc.*` gauges are named after the JVM's collector beans, and which beans exist depends on the JDK and collector: JDK 20+ G1 adds a `G1 Concurrent GC` bean, and ZGC and Shenandoah report `… Cycles` beside `… Pauses`, so `_gc_total()` leaves the concurrent ones out to keep `gc_time_ms` a pause total. Solr 9.10's image runs JDK 17; to see another JDK's bean names, run a one-line `ManagementFactory.getGarbageCollectorMXBeans()` loop in `eclipse-temurin:-jdk`. Names matter as much as encodings: Solr 9 registers filterCache and queryResultCache as `CACHE.searcher.*`, and reading `CACHE.core.*` returned `null` for every cache field while a key-presence test kept passing -- so test that a metric has a value, not just that its key exists. Registry names vary too: a standalone core's figures are under `solr.core.`, but a SolrCloud core's are under `solr.core...` (`name_lookup_shard1_replica_n1` reports as `solr.core.name_lookup.shard1.replica_n1`); `core_metrics_registry()` resolves both, and reports a `metrics_error` rather than nulls when neither is there. To check a cloud-mode name, a throwaway cloud Solr is two commands: `docker run -d --name nr-cloud-test -p 8984:8983 solr:9.10 solr-foreground -c`, then `docker exec nr-cloud-test solr create -c name_lookup`. -- **Send Solr requests through `solr_client()`, not a new `httpx.AsyncClient`.** A client per query builds an SSL context and opens a TCP connection every time, which took a 1 ms Solr query to about 8 ms as NameRes measured it. `solr_client()` hands out one client per event loop (an httpx client's connections belong to the loop that opened them, and `TestClient` and `asyncio.run()` start fresh loops), with no connection cap so that a busy pool can't surface as Solr timeouts; the app's lifespan closes it. Tests that fake Solr patch `httpx.AsyncClient.get`/`post` on the class, which still reaches the shared instance. +- **Send Solr requests through `solr_client()` and `solr_request()`, not a new `httpx.AsyncClient`.** A client per query builds an SSL context and opens a TCP connection every time, which took a 1 ms Solr query to about 8 ms as NameRes measured it. The app's lifespan opens one shared client and closes it on shutdown; `async with solr_client()` hands it out on the loop the lifespan ran on (uvicorn's, and `with TestClient(app)`'s), and a fresh client closed on exit anywhere else (a `TestClient` used without `with` starts a loop per request, as does `asyncio.run()`). Don't go back to caching a client per loop in a `WeakKeyDictionary`: a kept-alive connection's transport refers back to its loop, so finished loops, their clients and open sockets were never collected. `solr_request()` passes the timeout per request (so a test that replaces `config` is obeyed) and retries once when a pooled connection broke under the request, which kept-alive connections make possible after a Solr restart; it doesn't retry timeouts. `/status` deliberately uses `new_solr_client()`, not the shared pool, so that a probe never fails on a connection a lookup left broken. Tests that fake Solr patch `httpx.AsyncClient.get`/`post` on the class, which reaches every client, and `solr_request()` calls those methods by name so they still do. ## Documentation - `documentation/API.md` - Endpoint reference diff --git a/api/server.py b/api/server.py index 820b8d24..51e374d2 100755 --- a/api/server.py +++ b/api/server.py @@ -9,7 +9,6 @@ import logging import statistics import warnings -import weakref import time import os import re @@ -79,45 +78,96 @@ def public(self) -> dict: config = Config() -# One httpx client per event loop, shared by every Solr request, so that requests reuse -# kept-alive connections. A new client per query (as lookup() used to make) builds an SSL -# context and opens a TCP connection each time: on a local Solr that took a 1 ms query to about -# 8 ms, most of what /status reported as solr_round_trip_ms. +# One httpx client shared by every Solr lookup, so that lookups reuse kept-alive +# connections. A new client per query (as lookup() used to make) builds an SSL context and +# opens a TCP connection each time: on a local Solr that took a 1 ms query to about 8 ms, +# most of what /status reported as solr_round_trip_ms. # -# Keyed by loop because an httpx client's connections belong to the loop that opened them. -# Uvicorn serves everything from one loop, so in production this is a single client; the -# key matters for TestClient and asyncio.run(), which start a fresh loop per call, and -# would otherwise be handed connections from a loop that has gone. Weak keys let a dead -# loop's client go with it. -_solr_clients: "weakref.WeakKeyDictionary[asyncio.AbstractEventLoop, httpx.AsyncClient]" = \ - weakref.WeakKeyDictionary() - - -def solr_client() -> httpx.AsyncClient: - """ The shared Solr client for the running event loop, created on first use. """ - loop = asyncio.get_running_loop() - client = _solr_clients.get(loop) - if client is None or client.is_closed: - client = httpx.AsyncClient( - timeout=config.solr_timeout, - # No cap on connections, as when each query had its own client: the in-flight - # bound is SOLR_MAX_CONCURRENT_LOOKUPS per /bulk-lookup request, and a pool cap - # below that would make queries wait for a connection -- inside - # SOLR_TIMEOUT_SECONDS, so a busy pool would show up as Solr timeouts. Keep up - # to that many idle connections alive for reuse. - limits=httpx.Limits(max_connections=None, - max_keepalive_connections=config.solr_max_concurrent_lookups), - ) - _solr_clients[loop] = client - return client +# The app's lifespan creates it and closes it, and it is only handed out on the event loop +# that lifespan ran on, because an httpx client's connections belong to the loop that +# opened them. Uvicorn runs the lifespan and serves every request on one loop, so in +# production every lookup shares it. Anything else -- a TestClient used without `with`, +# which starts a fresh loop per request, or a script calling lookup() through +# asyncio.run() -- gets a client of its own that is closed when the request is done. +# +# An earlier version kept one client per loop in a WeakKeyDictionary, expecting a finished +# loop's client to be collected with it. It never was: a kept-alive connection's transport +# refers back to its loop, so the dictionary kept every finished loop, its client and an +# open socket to Solr alive, and nothing could close them from another loop. +_shared_solr_client: Optional[tuple[asyncio.AbstractEventLoop, httpx.AsyncClient]] = None + + +def new_solr_client() -> httpx.AsyncClient: + """ A fresh httpx client for Solr requests; the caller must close it. """ + return httpx.AsyncClient( + timeout=config.solr_timeout, + # No cap on connections, as when each query had its own client: the in-flight bound + # is SOLR_MAX_CONCURRENT_LOOKUPS per /bulk-lookup request, and a pool cap below that + # would make queries wait for a connection -- inside SOLR_TIMEOUT_SECONDS, so a busy + # pool would show up as Solr timeouts. Keep up to that many idle connections alive + # for reuse. This is read once, when the client is made; the timeout is not, since + # solr_request() passes it with every request. + # + # httpx's default keepalive_expiry closes a connection after 5 s idle, well inside + # Jetty's idle timeout, so Solr rarely closes one we are about to reuse. + limits=httpx.Limits(max_connections=None, + max_keepalive_connections=config.solr_max_concurrent_lookups), + ) + + +@asynccontextmanager +async def solr_client(): + """ + The Solr client to use for one request: the shared one when running on the loop the + app's lifespan opened it on, otherwise a fresh client closed on the way out. + """ + shared = _shared_solr_client + if shared is not None and shared[0] is asyncio.get_running_loop() and not shared[1].is_closed: + yield shared[1] + else: + async with new_solr_client() as client: + yield client + + +# The errors that mean a connection broke under a request rather than that Solr answered +# badly or slowly: Solr closed it as we sent (RemoteProtocolError, "Server disconnected +# without sending a response"), or reset it (ReadError, WriteError). With kept-alive +# connections, a Solr that restarted or a pod that moved can leave one of these in the +# pool. httpcore already drops idle connections the server closed cleanly, so this is +# the race where the close lands while the request is in flight. httpx doesn't retry it +# for a POST, so solr_request() does, once; every request NameRes sends Solr is a read. +# Timeouts are not retried: they cost SOLR_TIMEOUT_SECONDS each, and a slow Solr is not +# helped by a second query. +_RETRYABLE_SOLR_ERRORS = (httpx.RemoteProtocolError, httpx.ReadError, httpx.WriteError) + + +async def solr_request(client: httpx.AsyncClient, method: str, url: str, **kwargs) -> httpx.Response: + """ + Send a request to Solr through `client`, retrying once if the connection broke under it. + + `method` is "get" or "post", called by name so that tests patching + httpx.AsyncClient.get/post still see the request. The timeout is read from `config` + on every call rather than fixed when the client was made. + """ + send = getattr(client, method) + try: + return await send(url, timeout=config.solr_timeout, **kwargs) + except _RETRYABLE_SOLR_ERRORS as e: + logger.info("Solr connection broke during a request to %s (%s: %s); retrying once.", + url, type(e).__name__, e) + return await send(url, timeout=config.solr_timeout, **kwargs) @asynccontextmanager async def lifespan(app: FastAPI): - """ Close the shared Solr client when the server shuts down. """ - yield - client = _solr_clients.pop(asyncio.get_running_loop(), None) - if client is not None: + """ Open the shared Solr client for the server's lifetime, and close it on shutdown. """ + global _shared_solr_client + client = new_solr_client() + _shared_solr_client = (asyncio.get_running_loop(), client) + try: + yield + finally: + _shared_solr_client = None await client.aclose() @@ -335,7 +385,8 @@ def _dict(value) -> Dict: return value if isinstance(value, dict) else {} -async def fetch_solr_metrics(core_name: Optional[str], core: Optional[Dict] = None) -> Dict: +async def fetch_solr_metrics(client: httpx.AsyncClient, core_name: Optional[str], + core: Optional[Dict] = None) -> Dict: """ Read Solr's own health metrics from /admin/metrics, for /status?full=true. @@ -353,9 +404,8 @@ async def fetch_solr_metrics(core_name: Optional[str], core: Optional[Dict] = No """ metrics_url = f"http://{config.solr_host}:{config.solr_port}/solr/admin/metrics" try: - client = solr_client() # A single call with group=core&group=jvm retrieves both in one round-trip. - metrics_resp = await client.get(metrics_url, params=[ + metrics_resp = await solr_request(client, "get", metrics_url, params=[ ('group', 'core'), ('group', 'jvm'), ('prefix', 'QUERY./select'), @@ -510,15 +560,35 @@ async def status_get( async def status(full: bool = False) -> Dict: """ Return a dictionary containing status and count information for the underlying Solr instance. """ query_url = f"http://{config.solr_host}:{config.solr_port}/solr/admin/cores" - client = solr_client() - response = await client.get(query_url, params={ - 'action': 'STATUS' - }) - if response.status_code >= 300: - logger.error("Solr error on accessing /solr/admin/cores?action=STATUS: %s", response.text) - response.raise_for_status() + # /status opens a connection of its own rather than taking one from the pool the lookups + # share. It is what the Kubernetes probes call, so it should answer whether Solr takes a + # new connection now, not depend on the state lookups left a pooled one in -- a probe + # failing on a connection a cancelled bulk lookup broke would restart a pod whose Solr is + # fine. One connection setup every few seconds costs nothing worth saving. + async with new_solr_client() as client: + response = await solr_request(client, "get", query_url, params={ + 'action': 'STATUS' + }) + if response.status_code >= 300: + logger.error("Solr error on accessing /solr/admin/cores?action=STATUS: %s", response.text) + response.raise_for_status() + + result = response.json() - result = response.json() + # We should have a status for our core. Standalone Solr calls it $SOLR_CORE + # (name_lookup); the older cloud-mode backups called it + # name_lookup_shard1_replica_n1. A NameRes Solr only ever has one core, so if the + # expected name isn't there but there is exactly one core, report on that one. + cores = result.get('status', {}) + core_name = config.solr_core + core = cores.get(core_name) + if core is None and len(cores) == 1: + core_name, core = next(iter(cores.items())) + + # Solr's own health metrics are an extra round-trip, so only on ?full=true. Without a + # core there are no query or cache figures to read, only the JVM's. + metrics = await fetch_solr_metrics(client, core_name if core is not None else None, + core) if full else {} # Do we know the Babel version and version URL? It will be stored in an environmental variable if we do. babel_version = os.environ.get("BABEL_VERSION", "unknown") @@ -541,16 +611,6 @@ async def status(full: bool = False) -> Dict: # the default /status path. recent_queries = recent_lookups.summary() - # We should have a status for our core. Standalone Solr calls it $SOLR_CORE - # (name_lookup); the older cloud-mode backups called it - # name_lookup_shard1_replica_n1. A NameRes Solr only ever has one core, so if the - # expected name isn't there but there is exactly one core, report on that one. - cores = result.get('status', {}) - core_name = config.solr_core - core = cores.get(core_name) - if core is None and len(cores) == 1: - core_name, core = next(iter(cores.items())) - # The fields both branches below report. Built once and spread into each, so that the # two can't drift: the error branch is what a mis-pointed or still-loading deployment # returns, and that is exactly the case in which a caller most needs to know which @@ -574,10 +634,6 @@ async def status(full: bool = False) -> Dict: 'config': config.public(), } - # Solr's own health metrics are an extra round-trip, so only on ?full=true. Without a - # core there are no query or cache figures to read, only the JVM's. - metrics = await fetch_solr_metrics(core_name if core is not None else None, core) if full else {} - if core is not None: index = {} if 'index' in core: @@ -734,8 +790,8 @@ async def name_lookup(curies) -> Dict[str, Dict]: "query": curie_filter, "limit": 1000000, } - client = solr_client() - response = await client.post(query, json=params) + async with solr_client() as client: + response = await solr_request(client, "post", query, json=params) response.raise_for_status() response_json = response.json() output = { @@ -1169,8 +1225,8 @@ async def lookup(string: str, time_solr_end = None # Set once Solr has answered; see the except below. query_url = f"http://{config.solr_host}:{config.solr_port}/solr/{config.solr_core}/select" try: - client = solr_client() - response = await client.post(query_url, json=params) + async with solr_client() as client: + response = await solr_request(client, "post", query_url, json=params) if response.status_code >= 300: logger.error("Solr REST error: %s", response.text) response.raise_for_status() diff --git a/tests/test_status.py b/tests/test_status.py index fd1e3926..5353c745 100644 --- a/tests/test_status.py +++ b/tests/test_status.py @@ -631,40 +631,124 @@ def test_diagnostic_setting_reads_valid_values(monkeypatch): assert math.isclose(diagnostic_setting("NAMERES_TEST_SETTING", 7.0, float), 250.5) -def test_solr_client_is_shared_within_an_event_loop(): - """Solr requests reuse one client, and so its kept-alive connections, rather than - paying for a new client and connection per query. A new loop gets its own client, - since connections belong to the loop that opened them.""" - import api.server +# The tests below pin how NameRes holds its httpx clients. They fake Solr, like the tests +# above, and record which client instance each request went through (`self`). - async def two(): - return api.server.solr_client(), api.server.solr_client() +EMPTY_SOLR_RESULTS = {'response': {'docs': []}} - first, again = asyncio.run(two()) - assert first is again - other, _ = asyncio.run(two()) - assert other is not first +def _recording_solr(monkeypatch, post_result=None): + """Patch httpx.AsyncClient.get/post with fakes that record (client, url, kwargs) for each + call. The POST returns post_result(call_number), or an empty result set by default.""" + gets, posts = [], [] + fake_get = _fake_solr_get() -def test_shared_solr_client_is_reused_and_closed_on_shutdown(monkeypatch): + async def get(self, url, params=None, **kwargs): + gets.append((self, url, kwargs)) + return await fake_get(self, url, params=params, **kwargs) + + async def post(self, url, **kwargs): + posts.append((self, url, kwargs)) + if post_result is not None: + result = post_result(len(posts)) + if isinstance(result, BaseException): + raise result + return result + return httpx.Response(200, request=httpx.Request("POST", url), json=EMPTY_SOLR_RESULTS) + + monkeypatch.setattr(httpx.AsyncClient, "get", get) + monkeypatch.setattr(httpx.AsyncClient, "post", post) + return gets, posts + + +def test_lookups_share_one_client_closed_on_shutdown(monkeypatch): + """Under a running lifespan (as under uvicorn), lookups reuse one client, and so its + kept-alive connections; /status uses its own; shutdown closes the shared one.""" import api.server - handed_out = [] - real_solr_client = api.server.solr_client - - def spy(): - handed_out.append(real_solr_client()) - return handed_out[-1] - monkeypatch.setattr(api.server, "solr_client", spy) + gets, posts = _recording_solr(monkeypatch) # As a context manager, TestClient runs the app's lifespan and serves every request # from one event loop, as uvicorn does. with TestClient(app) as client: + assert client.get("/lookup", params={'string': 'alzheimer'}).status_code == 200 + assert client.get("/lookup", params={'string': 'diabetes'}).status_code == 200 + assert client.post("/synonyms", json={'preferred_curies': ['MONDO:0004975']}).status_code == 200 + assert client.get("/status").status_code == 200 + + shared = posts[0][0] + assert all(c is shared for c, _, _ in posts) and len(posts) == 3 + assert not shared.is_closed + # The probe's client is not the lookups' pool, and is closed once /status answers. + assert gets[0][0] is not shared and gets[0][0].is_closed + + assert shared.is_closed + assert api.server._shared_solr_client is None + + +def test_solr_clients_outside_the_lifespan_are_closed(monkeypatch): + """Without a lifespan -- a TestClient used without `with`, which starts a loop per + request, or asyncio.run() -- each request gets a client of its own and closes it, + rather than leaving one per finished loop holding an open socket.""" + import api.server + _, posts = _recording_solr(monkeypatch) + + client = TestClient(app) + client.get("/lookup", params={'string': 'alzheimer'}) + client.get("/lookup", params={'string': 'diabetes'}) + assert posts[0][0] is not posts[1][0] + assert all(c.is_closed for c, _, _ in posts) + + async def borrow(): + async with api.server.solr_client() as c: + assert not c.is_closed + return c + assert asyncio.run(borrow()).is_closed + assert api.server._shared_solr_client is None + + +def test_solr_timeout_is_read_per_request(monkeypatch): + """The shared client is made once, at startup, but a later change to `config` (as tests + make with dataclasses.replace) still sets the timeout of the next request.""" + import dataclasses + import api.server + _, posts = _recording_solr(monkeypatch) + + with TestClient(app) as client: + monkeypatch.setattr(api.server, "config", + dataclasses.replace(api.server.config, solr_timeout_seconds=7)) client.get("/lookup", params={'string': 'alzheimer'}) - client.get("/lookup", params={'string': 'diabetes'}) - client.get("/status") - assert not handed_out[0].is_closed - assert len(handed_out) == 3 and all(c is handed_out[0] for c in handed_out) - assert handed_out[0].is_closed + assert posts[0][2]['timeout'] == 7 + + +@pytest.mark.parametrize("failure", [ + httpx.RemoteProtocolError("Server disconnected without sending a response."), + httpx.ReadError("Connection reset by peer"), + httpx.WriteError("Broken pipe"), +]) +def test_lookup_retries_once_on_a_broken_connection(monkeypatch, failure): + """A pooled connection Solr closed under the request (after a Solr restart, say) is + retried once, on a new connection, instead of becoming a 500.""" + _, posts = _recording_solr(monkeypatch, lambda n: failure if n == 1 else + httpx.Response(200, request=httpx.Request("POST", "http://solr"), + json=EMPTY_SOLR_RESULTS)) + with TestClient(app) as client: + assert client.get("/lookup", params={'string': 'alzheimer'}).status_code == 200 + assert len(posts) == 2 + + +@pytest.mark.parametrize("failure, attempts", [ + # Retried once, then reported: a Solr that keeps dropping connections is down. + (httpx.RemoteProtocolError("Server disconnected without sending a response."), 2), + # Not retried: each attempt would cost another SOLR_TIMEOUT_SECONDS. + (httpx.ReadTimeout("timed out"), 1), + # Not retried: a refused connection was never a pooled one going stale. + (httpx.ConnectError("connection refused"), 1), +]) +def test_lookup_retry_is_limited(monkeypatch, failure, attempts): + _, posts = _recording_solr(monkeypatch, lambda n: failure) + with TestClient(app, raise_server_exceptions=False) as client: + assert client.get("/lookup", params={'string': 'alzheimer'}).status_code == 500 + assert len(posts) == attempts # The lookup log line is parsed downstream (TranslatorSRI/babel-validation's NameRes log From 6a77ca4eac3a1eb14d26b2212c0a780c419b368c Mon Sep 17 00:00:00 2001 From: gaurav Date: Wed, 11 Nov 2026 01:18:16 -0500 Subject: [PATCH 3/3] Note that kept-alive Solr connections pin to replicas; fix a split word A Kubernetes Service balances connections, not requests, so with more than one Solr replica a long-lived connection keeps going to the replica it first reached. Deployment.md now says so. Performance.md had "kept-" and "alive" on separate lines, which Markdown renders as "kept- alive". Co-Authored-By: Claude Opus 5.5 --- documentation/Deployment.md | 2 ++ documentation/Performance.md | 8 ++++---- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/documentation/Deployment.md b/documentation/Deployment.md index b0b728d2..257f0141 100644 --- a/documentation/Deployment.md +++ b/documentation/Deployment.md @@ -107,6 +107,8 @@ NameRes can be configured by setting environmental variables: * `SOLR_CORE`: The Solr core to query (defaults to `name_lookup`). * `SOLR_MAX_CONCURRENT_LOOKUPS`: The greatest number of Solr queries a single `/bulk-lookup` request may have in flight at once (defaults to `100`, and is clamped to at least `1`). The bound is per request, so the queries this service has in flight against Solr is this multiplied by the number of bulk lookups being served at once. * `SOLR_TIMEOUT_SECONDS`: How long to wait for any one Solr query before giving up (defaults to `60`). Set it to `0` to wait indefinitely. + + NameRes keeps its connections to Solr open between lookups, closing one after it has sat idle for 5 seconds. If Solr ever runs as more than one replica behind a Kubernetes Service, note that a Service balances *connections*, not requests: each API process keeps sending to the replicas its open connections reached, and a newly added replica gets traffic from it only as those connections are closed and reopened, which under steady load may be never. Restart the API pods after scaling Solr up, or put a request-level balancer (an Ingress or a service mesh) in front of Solr. * `NAMERES_MINIMUM_QUERY_LENGTH`: The shortest query, in characters after leading/trailing whitespace is stripped, that `/lookup` and `/bulk-lookup` will search for (defaults to `2`). Shorter queries are rejected with an HTTP 422 from `/lookup` and return no results from `/bulk-lookup`, since they are slow in Solr and never useful. This applies only to the default tokenized search: exact matching accepts any non-empty string, and an empty string is rejected whatever this is set to. The value in force is reported as `minimum_query_length` by `/status`. * `SERVER_NAME`: The name of this server (defaults to `infores:sri-name-resolver`) * `SERVER_ROOT`: The server root (defaults to `/`) diff --git a/documentation/Performance.md b/documentation/Performance.md index 6a482ce5..e3860a0a 100644 --- a/documentation/Performance.md +++ b/documentation/Performance.md @@ -32,10 +32,10 @@ Three latencies nest inside one another, and the fields are named for which one | Solr handler | `solr.queries.handler_ms` (with `?full=true`) | Only the time inside Solr's `/select` handler, as Solr's own timer measures it. | So `lookup_ms` − `solr_round_trip_ms` is NameRes's result processing, and `solr_round_trip_ms` − -`handler_ms` is connection, transfer and JSON overhead. NameRes shares one HTTP client with kept- -alive connections across its Solr requests, so that second gap should be small; a large one points -at the network between the API and Solr, or at large responses. (When `lookup()` opened a new -client and connection per query, that overhead took a 1 ms query on a local Solr to about 8 ms.) +`handler_ms` is connection, transfer and JSON overhead. NameRes's lookups share one HTTP client +whose connections are kept alive between requests, so that second gap should be small; a large one +points at the network between the API and Solr, or at large responses. (When `lookup()` opened a +new client and connection per query, that overhead took a 1 ms query on a local Solr to about 8 ms.) | Field | What it means | |---|---|