Skip to content
2 changes: 2 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,8 @@ Solr documents contain: `curie`, `preferred_name`, `names` (synonym list), and b
- **Declaring metadata in `openapi.yml` is not enough to serve it.** `construct_open_api_schema()` copies an explicit allowlist of `info` keys into the document (and `get_app_info()` a narrower one for the `FastAPI()` constructor); anything not named there is dropped without a word. That is how `info.contact` and `info.license` sat declared-but-unserved for years. Adding a key means adding it to the copy list *and* asserting it in `tests/test_openapi.py`.
- **Check a new Solr `/admin/metrics` field against a live Solr before trusting it.** The encoding varies by metric type, and a wrong guess silently yields `null` rather than an error. Counters (`QUERY./select.requests`) are scalars, timers (`requestTimes`) and meters (`errors`/`timeouts`) are nested objects, and JVM gauges (`memory.heap.used`) come back as *flat dotted keys*, not a nested `memory.heap` map. `status()` reads them accordingly. The `gc.*` gauges are named after the JVM's collector beans, and which beans exist depends on the JDK and collector: JDK 20+ G1 adds a `G1 Concurrent GC` bean, and ZGC and Shenandoah report `… Cycles` beside `… Pauses`, so `_gc_total()` leaves the concurrent ones out to keep `gc_time_ms` a pause total. Solr 9.10's image runs JDK 17; to see another JDK's bean names, run a one-line `ManagementFactory.getGarbageCollectorMXBeans()` loop in `eclipse-temurin:<n>-jdk`. Names matter as much as encodings: Solr 9 registers filterCache and queryResultCache as `CACHE.searcher.*`, and reading `CACHE.core.*` returned `null` for every cache field while a key-presence test kept passing -- so test that a metric has a value, not just that its key exists. Registry names vary too: a standalone core's figures are under `solr.core.<core>`, but a SolrCloud core's are under `solr.core.<collection>.<shard>.<replica>` (`name_lookup_shard1_replica_n1` reports as `solr.core.name_lookup.shard1.replica_n1`); `core_metrics_registry()` resolves both, and reports a `metrics_error` rather than nulls when neither is there. To check a cloud-mode name, a throwaway cloud Solr is two commands: `docker run -d --name nr-cloud-test -p 8984:8983 solr:9.10 solr-foreground -c`, then `docker exec nr-cloud-test solr create -c name_lookup`.

- **Send Solr requests through `solr_client()` and `solr_request()`, not a new `httpx.AsyncClient`.** A client per query builds an SSL context and opens a TCP connection every time, which took a 1 ms Solr query to about 8 ms as NameRes measured it. The app's lifespan opens one shared client and closes it on shutdown; `async with solr_client()` hands it out on the loop the lifespan ran on (uvicorn's, and `with TestClient(app)`'s), and a fresh client closed on exit anywhere else (a `TestClient` used without `with` starts a loop per request, as does `asyncio.run()`). Don't go back to caching a client per loop in a `WeakKeyDictionary`: a kept-alive connection's transport refers back to its loop, so finished loops, their clients and open sockets were never collected. `solr_request()` passes the timeout per request (so a test that replaces `config` is obeyed) and retries once when a pooled connection broke under the request, which kept-alive connections make possible after a Solr restart; it doesn't retry timeouts. `/status` deliberately uses `new_solr_client()`, not the shared pool, so that a probe never fails on a connection a lookup left broken. Tests that fake Solr patch `httpx.AsyncClient.get`/`post` on the class, which reaches every client, and `solr_request()` calls those methods by name so they still do.

## Documentation
- `documentation/API.md` - Endpoint reference
- `documentation/Performance.md` - Reading the `/status` metrics; diagnosing Solr CPU/memory/load
Expand Down
177 changes: 139 additions & 38 deletions api/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
import os
import re
from collections import Counter, deque
from contextlib import asynccontextmanager
from dataclasses import dataclass
from enum import Enum
from typing import Dict, List, Union, Annotated, Optional
Expand Down Expand Up @@ -76,7 +77,101 @@ def public(self) -> dict:

config = Config()

app = FastAPI(**get_app_info())

# One httpx client shared by every Solr lookup, so that lookups reuse kept-alive
# connections. A new client per query (as lookup() used to make) builds an SSL context and
# opens a TCP connection each time: on a local Solr that took a 1 ms query to about 8 ms,
# most of what /status reported as solr_round_trip_ms.
#
# The app's lifespan creates it and closes it, and it is only handed out on the event loop
# that lifespan ran on, because an httpx client's connections belong to the loop that
# opened them. Uvicorn runs the lifespan and serves every request on one loop, so in
# production every lookup shares it. Anything else -- a TestClient used without `with`,
# which starts a fresh loop per request, or a script calling lookup() through
# asyncio.run() -- gets a client of its own that is closed when the request is done.
#
# An earlier version kept one client per loop in a WeakKeyDictionary, expecting a finished
# loop's client to be collected with it. It never was: a kept-alive connection's transport
# refers back to its loop, so the dictionary kept every finished loop, its client and an
# open socket to Solr alive, and nothing could close them from another loop.
_shared_solr_client: Optional[tuple[asyncio.AbstractEventLoop, httpx.AsyncClient]] = None


def new_solr_client() -> httpx.AsyncClient:
""" A fresh httpx client for Solr requests; the caller must close it. """
return httpx.AsyncClient(
timeout=config.solr_timeout,
# No cap on connections, as when each query had its own client: the in-flight bound
# is SOLR_MAX_CONCURRENT_LOOKUPS per /bulk-lookup request, and a pool cap below that
# would make queries wait for a connection -- inside SOLR_TIMEOUT_SECONDS, so a busy
# pool would show up as Solr timeouts. Keep up to that many idle connections alive
# for reuse. This is read once, when the client is made; the timeout is not, since
# solr_request() passes it with every request.
#
# httpx's default keepalive_expiry closes a connection after 5 s idle, well inside
# Jetty's idle timeout, so Solr rarely closes one we are about to reuse.
limits=httpx.Limits(max_connections=None,
max_keepalive_connections=config.solr_max_concurrent_lookups),
)


@asynccontextmanager
async def solr_client():
"""
The Solr client to use for one request: the shared one when running on the loop the
app's lifespan opened it on, otherwise a fresh client closed on the way out.
"""
shared = _shared_solr_client
if shared is not None and shared[0] is asyncio.get_running_loop() and not shared[1].is_closed:
yield shared[1]
else:
async with new_solr_client() as client:
yield client


# The errors that mean a connection broke under a request rather than that Solr answered
# badly or slowly: Solr closed it as we sent (RemoteProtocolError, "Server disconnected
# without sending a response"), or reset it (ReadError, WriteError). With kept-alive
# connections, a Solr that restarted or a pod that moved can leave one of these in the
# pool. httpcore already drops idle connections the server closed cleanly, so this is
# the race where the close lands while the request is in flight. httpx doesn't retry it
# for a POST, so solr_request() does, once; every request NameRes sends Solr is a read.
# Timeouts are not retried: they cost SOLR_TIMEOUT_SECONDS each, and a slow Solr is not
# helped by a second query.
_RETRYABLE_SOLR_ERRORS = (httpx.RemoteProtocolError, httpx.ReadError, httpx.WriteError)


async def solr_request(client: httpx.AsyncClient, method: str, url: str, **kwargs) -> httpx.Response:
"""
Send a request to Solr through `client`, retrying once if the connection broke under it.

`method` is "get" or "post", called by name so that tests patching
httpx.AsyncClient.get/post still see the request. The timeout is read from `config`
on every call rather than fixed when the client was made.
"""
send = getattr(client, method)
try:
return await send(url, timeout=config.solr_timeout, **kwargs)
except _RETRYABLE_SOLR_ERRORS as e:
logger.info("Solr connection broke during a request to %s (%s: %s); retrying once.",
url, type(e).__name__, e)
return await send(url, timeout=config.solr_timeout, **kwargs)


@asynccontextmanager
async def lifespan(app: FastAPI):
""" Open the shared Solr client for the server's lifetime, and close it on shutdown. """
global _shared_solr_client
client = new_solr_client()
_shared_solr_client = (asyncio.get_running_loop(), client)
try:
yield
finally:
_shared_solr_client = None
await client.aclose()


app = FastAPI(**get_app_info(), lifespan=lifespan)
logger = logging.getLogger(__name__)
logging.basicConfig(level=os.getenv("LOGLEVEL", logging.INFO))

Expand Down Expand Up @@ -290,7 +385,8 @@ def _dict(value) -> Dict:
return value if isinstance(value, dict) else {}


async def fetch_solr_metrics(core_name: Optional[str], core: Optional[Dict] = None) -> Dict:
async def fetch_solr_metrics(client: httpx.AsyncClient, core_name: Optional[str],
core: Optional[Dict] = None) -> Dict:
"""
Read Solr's own health metrics from /admin/metrics, for /status?full=true.

Expand All @@ -308,21 +404,20 @@ async def fetch_solr_metrics(core_name: Optional[str], core: Optional[Dict] = No
"""
metrics_url = f"http://{config.solr_host}:{config.solr_port}/solr/admin/metrics"
try:
async with httpx.AsyncClient(timeout=config.solr_timeout) as client:
# A single call with group=core&group=jvm retrieves both in one round-trip.
metrics_resp = await client.get(metrics_url, params=[
('group', 'core'),
('group', 'jvm'),
('prefix', 'QUERY./select'),
# Solr 9 registers the searcher caches as CACHE.searcher.*, not
# CACHE.core.* -- a wrong name here yields nulls, not an error.
('prefix', 'CACHE.searcher.filterCache'),
('prefix', 'CACHE.searcher.queryResultCache'),
('prefix', 'memory.heap'),
('prefix', 'gc.'),
('prefix', 'os.'),
('wt', 'json'),
])
# A single call with group=core&group=jvm retrieves both in one round-trip.
metrics_resp = await solr_request(client, "get", metrics_url, params=[
('group', 'core'),
('group', 'jvm'),
('prefix', 'QUERY./select'),
# Solr 9 registers the searcher caches as CACHE.searcher.*, not
# CACHE.core.* -- a wrong name here yields nulls, not an error.
('prefix', 'CACHE.searcher.filterCache'),
('prefix', 'CACHE.searcher.queryResultCache'),
('prefix', 'memory.heap'),
('prefix', 'gc.'),
('prefix', 'os.'),
('wt', 'json'),
])
if metrics_resp.status_code >= 300:
logger.warning("Solr error on accessing /solr/admin/metrics: HTTP %d: %s",
metrics_resp.status_code, metrics_resp.text)
Expand Down Expand Up @@ -465,15 +560,35 @@ async def status_get(
async def status(full: bool = False) -> Dict:
""" Return a dictionary containing status and count information for the underlying Solr instance. """
query_url = f"http://{config.solr_host}:{config.solr_port}/solr/admin/cores"
async with httpx.AsyncClient(timeout=config.solr_timeout) as client:
response = await client.get(query_url, params={
# /status opens a connection of its own rather than taking one from the pool the lookups
# share. It is what the Kubernetes probes call, so it should answer whether Solr takes a
# new connection now, not depend on the state lookups left a pooled one in -- a probe
# failing on a connection a cancelled bulk lookup broke would restart a pod whose Solr is
# fine. One connection setup every few seconds costs nothing worth saving.
async with new_solr_client() as client:
response = await solr_request(client, "get", query_url, params={
'action': 'STATUS'
})
if response.status_code >= 300:
logger.error("Solr error on accessing /solr/admin/cores?action=STATUS: %s", response.text)
response.raise_for_status()

result = response.json()
result = response.json()

# We should have a status for our core. Standalone Solr calls it $SOLR_CORE
# (name_lookup); the older cloud-mode backups called it
# name_lookup_shard1_replica_n1. A NameRes Solr only ever has one core, so if the
# expected name isn't there but there is exactly one core, report on that one.
cores = result.get('status', {})
core_name = config.solr_core
core = cores.get(core_name)
if core is None and len(cores) == 1:
core_name, core = next(iter(cores.items()))

# Solr's own health metrics are an extra round-trip, so only on ?full=true. Without a
# core there are no query or cache figures to read, only the JVM's.
metrics = await fetch_solr_metrics(client, core_name if core is not None else None,
core) if full else {}

# Do we know the Babel version and version URL? It will be stored in an environmental variable if we do.
babel_version = os.environ.get("BABEL_VERSION", "unknown")
Expand All @@ -496,16 +611,6 @@ async def status(full: bool = False) -> Dict:
# the default /status path.
recent_queries = recent_lookups.summary()

# We should have a status for our core. Standalone Solr calls it $SOLR_CORE
# (name_lookup); the older cloud-mode backups called it
# name_lookup_shard1_replica_n1. A NameRes Solr only ever has one core, so if the
# expected name isn't there but there is exactly one core, report on that one.
cores = result.get('status', {})
core_name = config.solr_core
core = cores.get(core_name)
if core is None and len(cores) == 1:
core_name, core = next(iter(cores.items()))

# The fields both branches below report. Built once and spread into each, so that the
# two can't drift: the error branch is what a mis-pointed or still-loading deployment
# returns, and that is exactly the case in which a caller most needs to know which
Expand All @@ -529,10 +634,6 @@ async def status(full: bool = False) -> Dict:
'config': config.public(),
}

# Solr's own health metrics are an extra round-trip, so only on ?full=true. Without a
# core there are no query or cache figures to read, only the JVM's.
metrics = await fetch_solr_metrics(core_name if core is not None else None, core) if full else {}

if core is not None:
index = {}
if 'index' in core:
Expand Down Expand Up @@ -689,8 +790,8 @@ async def name_lookup(curies) -> Dict[str, Dict]:
"query": curie_filter,
"limit": 1000000,
}
async with httpx.AsyncClient(timeout=config.solr_timeout) as client:
response = await client.post(query, json=params)
async with solr_client() as client:
response = await solr_request(client, "post", query, json=params)
response.raise_for_status()
response_json = response.json()
output = {
Expand Down Expand Up @@ -1124,8 +1225,8 @@ async def lookup(string: str,
time_solr_end = None # Set once Solr has answered; see the except below.
query_url = f"http://{config.solr_host}:{config.solr_port}/solr/{config.solr_core}/select"
try:
async with httpx.AsyncClient(timeout=config.solr_timeout) as client:
response = await client.post(query_url, json=params)
async with solr_client() as client:
response = await solr_request(client, "post", query_url, json=params)
if response.status_code >= 300:
logger.error("Solr REST error: %s", response.text)
response.raise_for_status()
Expand Down
2 changes: 2 additions & 0 deletions documentation/Deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,8 @@ NameRes can be configured by setting environmental variables:
* `SOLR_CORE`: The Solr core to query (defaults to `name_lookup`).
* `SOLR_MAX_CONCURRENT_LOOKUPS`: The greatest number of Solr queries a single `/bulk-lookup` request may have in flight at once (defaults to `100`, and is clamped to at least `1`). The bound is per request, so the queries this service has in flight against Solr is this multiplied by the number of bulk lookups being served at once.
* `SOLR_TIMEOUT_SECONDS`: How long to wait for any one Solr query before giving up (defaults to `60`). Set it to `0` to wait indefinitely.

NameRes keeps its connections to Solr open between lookups, closing one after it has sat idle for 5 seconds. If Solr ever runs as more than one replica behind a Kubernetes Service, note that a Service balances *connections*, not requests: each API process keeps sending to the replicas its open connections reached, and a newly added replica gets traffic from it only as those connections are closed and reopened, which under steady load may be never. Restart the API pods after scaling Solr up, or put a request-level balancer (an Ingress or a service mesh) in front of Solr.
* `NAMERES_MINIMUM_QUERY_LENGTH`: The shortest query, in characters after leading/trailing whitespace is stripped, that `/lookup` and `/bulk-lookup` will search for (defaults to `2`). Shorter queries are rejected with an HTTP 422 from `/lookup` and return no results from `/bulk-lookup`, since they are slow in Solr and never useful. This applies only to the default tokenized search: exact matching accepts any non-empty string, and an empty string is rejected whatever this is set to. The value in force is reported as `minimum_query_length` by `/status`.
* `SERVER_NAME`: The name of this server (defaults to `infores:sri-name-resolver`)
* `SERVER_ROOT`: The server root (defaults to `/`)
Expand Down
9 changes: 5 additions & 4 deletions documentation/Performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,10 @@ Three latencies nest inside one another, and the fields are named for which one
| Solr handler | `solr.queries.handler_ms` (with `?full=true`) | Only the time inside Solr's `/select` handler, as Solr's own timer measures it. |

So `lookup_ms` − `solr_round_trip_ms` is NameRes's result processing, and `solr_round_trip_ms` −
`handler_ms` is connection, transfer and JSON overhead. That second gap is large: `lookup()`
currently opens a new HTTP client and connection per query, which on a local Solr made a 1 ms
query take about 8 ms.
`handler_ms` is connection, transfer and JSON overhead. NameRes's lookups share one HTTP client
whose connections are kept alive between requests, so that second gap should be small; a large one
points at the network between the API and Solr, or at large responses. (When `lookup()` opened a
new client and connection per query, that overhead took a 1 ms query on a local Solr to about 8 ms.)

| Field | What it means |
|---|---|
Expand Down Expand Up @@ -222,7 +223,7 @@ Solr seems slow or the service is unresponsive
│ ├─ solr_round_trip_ms is high but handler_ms is small
│ │ → the time is between NameRes and Solr: connection setup, network, transfer,
│ │ JSON decoding. Check: network between the API and Solr pods; large responses
│ │ (`limit`, highlighting). lookup() opens a new connection per query today.
│ │ (`limit`, highlighting).
│ │
│ └─ handler_ms is most of solr_round_trip_ms → problem is INSIDE Solr → continue
│
Expand Down
Loading
Loading