Repository navigation
Report the backend and API version from /status, and group its Solr stats under solr - #304
Merged
Merged
Conversation
Adds two keys to /status, matching NodeNorm (NCATSTranslator/NodeNormalization#392): - backend: always "solr" here. The Elasticsearch-backed NameRes at biothings/NameResolutionAPI reports "elasticsearch". Until now the two differed only by a word of prose in `message`, so nothing machine-readable said which implementation answered -- which is what blocked TranslatorSRI/babel-validation from pinning a NameRes target to its expected backend the way it already does for NodeNorm. - version: the OpenAPI info.version verbatim, so a caller can tell when the API changed without fetching /openapi.json, and so NameRes and NodeNorm spell it the same way. `version` was already taken by the Solr index version, which is renamed to `index_version`. This is a breaking change for anyone reading the Solr index version out of /status. The keys both branches share are now built once as `common` and spread into each return, so the error branch ("Expected core not found.") cannot drift from the ok branch. That branch is what a mis-pointed or still-loading deployment returns, and is exactly when a caller most needs to know which NameRes it reached. test_status.py pins `version` against /openapi.json's info.version rather than a literal, so the identity between the two is actually tested and the assertion survives version bumps. Closes #296. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
API.md describes `backend`, `version` and `index_version`, and calls out that `index_version` was `version` before v1.8.0. Also refreshes the stale nameres_version in the example, which still said v1.5.1. The CLAUDE.md gotcha records why `version` means the API version here and not the Solr index version, and why both branches of status() have to report `backend`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit moved the Solr index version to a top-level `index_version` to
free `version` for the API version. Instead, every field read from Solr's core
STATUS (startTime, numDocs, maxDoc, deletedDocs, version, segmentCount,
lastModified, size) now sits in a `solr` object, and the index version keeps
Solr's own name there as `solr.version`.
Why `solr` rather than `index_version`:
- /status is already changing incompatibly in this PR, so consumers adjust once
rather than again when more Solr stats arrive.
- It is the layout the improve-performance branch already uses (deployed as
release-v1.5.2-with-metrics), whose JVM, OS and cache stats belong in the same
object; babel-validation's trim_status() already reads raw["solr"] first.
- The key is named after `backend`'s value, so backend-specific detail can't
collide with the service's own keys again -- which is how the index version
came to squat on `version` in the first place.
Also adds `solr.core`, the name of the core actually reported on: with the
single-core fallback it is not always SOLR_CORE (older cloud-mode backups call it
name_lookup_shard1_replica_n1). The error branch is unchanged and has no `solr`.
The test asserts the old flat keys are absent, so a partial revert fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
API.md describes the `solr` object and notes that its fields were at the top level before v1.8.0, with `solr.version` formerly the top-level `version`. The CLAUDE.md gotcha now says Solr-sourced fields go under `solr` and not back at the top level. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/status, renaming the Solr index version to index_version/status, and group its Solr stats under solr
gaurav
added a commit
that referenced
this pull request
Nov 10, 2026
Brings in #304, which adds `backend` and the API `version` to /status and moves the Solr core's stats under a `solr` object. Conflicts in status(), tests/test_status.py and the API.md example were resolved by taking main's `common` keys and `solr` object, and keeping this branch's `recent_queries` and `solr_metrics` at the top level as they were. Both now appear on the error branch too. test_status_shape looks for numDocs under `solr`, and Performance.md's "Index health" section now points at `solr` instead of the top level. `solr_metrics` is still top-level, which conflicts with the CLAUDE.md gotcha merged with #304 (Solr-sourced fields go under `solr`). Moving it is a design change for this PR, not something for a merge to decide. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gaurav
added a commit
that referenced
this pull request
Nov 10, 2026
Everything /status reads from Solr goes under `solr` (the rule #304 set), so the ?full=true metrics now join the core's stats there instead of sitting in a top-level solr_metrics: solr_metrics.query_handler -> solr.queries solr_metrics.cache -> solr.cache solr_metrics.jvm -> solr.jvm solr_metrics.host -> solr.os `recent_queries` stays at the top level: NameRes times it end to end, so it is not a Solr reading. `os` rather than `host` or `node`: the figures are the JVM's operating-system bean (Solr's own os.* gauges), and in a container that bean reports the container's limits -- checked on solr:9.10 with --cpus 2 --memory 1g, which reported 2 processors and 1 GiB on a 12-CPU machine -- so "host" overstates them, and "node" would be read as the Kubernetes node. Without ?full=true the metrics blocks are simply absent; the old {"message": "Use /status?full=true..."} placeholder had no single place to live once the blocks were spread across `solr`, so the hint moved to the `full` parameter's OpenAPI description. A failed fetch is `solr.metrics_error`, so not asking and failing still look different. The fetch moves into fetch_solr_metrics(), called after the core is resolved, so that it reads solr.core.<the resolved core> instead of whichever solr.core.* came first. On the error branch (no core) ?full=true still reports `solr.jvm` and `solr.os`, but no query or cache figures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gaurav
added a commit
that referenced
this pull request
Nov 10, 2026
…nce-diagnostics guide (#231) This PR makes `/status` report how hard a NameRes instance and its Solr backend are working. It adds latency and failure counts over a rolling window of recent `/lookup` calls, and logs slow and failed lookups at WARNING. Behind `?full=true`, it adds Solr's own query, cache, JVM and OS metrics to the `solr` object that #304 introduced. That gives monitoring something to alert on and us the inputs for sizing the Solr pod, and a new guide, `documentation/Performance.md`, explains how to read them. This addresses the `/status` part of #228 and supplies the inputs for tuning issues #265 and #267. ## The calls worth overruling - **Solr's metrics go under `solr`**, following #304's rule that everything read from Solr goes there: `solr.queries`, `solr.cache`, `solr.jvm` and `solr.os`. Only `recent_queries`, which NameRes times itself, stays at the top level. They come from the core `/status` reports on, cloud-mode cores included; if it has none, `metrics_error` says so rather than borrowing another core's. - **`os`, not `host` or `node`:** the figures are Solr's `os.*` gauges, and in a container they are its limits (`--cpus 2 --memory 1g` on a 12-CPU machine reports 2 processors and 1024 MB). - **Solr metrics only with `?full=true`** (the name v1.5.2 and #267 use), so the probes' plain `/status` makes no extra Solr round-trip. A failed fetch is reported as `solr.metrics_error`, not omitted. - **Latencies are named for the span they time.** `recent_queries.lookup_ms` (all of `lookup()`), `recent_queries.solr_round_trip_ms` (the Solr request as NameRes sees it) and `solr.queries.handler_ms` (Solr's handler alone) each nest inside the previous one. Matching flat names made a 12 ms lookup beside a 1.35 ms Solr mean look like a contradiction; most of that gap is the per-query connection #310 removes. - **Latencies stay on the default path, cached for 10 seconds.** The counts are running totals and always current. A series' mean, min, max and percentiles need a pass over the window (about 4 ms at 50000 entries), so they are recomputed together at most every 10 s, and its mean can never fall outside its min and max. - **Failed lookups count in the latency window**, since their callers waited too; otherwise `/status` would look healthiest exactly when Solr is timing out. `errors`, `timeouts` and `cancelled` are also counted on their own, because a fast failure such as a refused connection pulls the percentiles *down*. - **`/bulk-lookup` stays out of `recent_queries` and out of the WARNING log**, since one request can hold thousands of strings. Its strings log at INFO, including failures. When one string fails, the request is a 500 regardless, so the strings not yet done are now cancelled rather than each sent to a Solr that is probably down, and the request logs one `FAILED BULK QUERY` warning. - **`gc_count`/`gc_time_ms` are pause totals.** They leave out collectors that time concurrent work (JDK 20+'s `G1 Concurrent GC`, ZGC's and Shenandoah's `… Cycles`), which would otherwise read as heap pressure. Solr 9.10's image runs JDK 17, where nothing is excluded. - **Deliberately left out:** OS page cache (Solr exposes only `MemFree`, so #267 needs a node-level exporter), file descriptors, and #253's per-second rates (sample the counters twice instead). ## What's here - **`recent_queries`, on every `/status`:** the failure counts, plus `lookup_ms` and `solr_round_trip_ms` with mean, min, max and p50/p95/p99, over the last `RECENT_TIMES_COUNT` lookups (default 50000). - **Logging:** a `/lookup` slower than `SLOW_QUERY_THRESHOLD_MS` (default 500) logs `SLOW QUERY: …`, and a failed one `FAILED QUERY (timeout|error|cancelled: …)`. The Solr figure now reads `(with Yms for the Solr round trip)` where v1.7.1 wrote `waiting for Solr`; TranslatorSRI/babel-validation#107's log parser accepts both. A malformed setting logs a warning and falls back to its default. - **With `?full=true`, from one `/admin/metrics` request:** `/select` counts and `handler_ms`, `filterCache` and `queryResultCache`, heap/CPU/GC, and processors/load/memory. Each value is checked on its own, so one odd gauge (or a heap maximum of -1) nulls only its own field. - **Docs:** [`API.md`](https://github.com/NCATSTranslator/NameResolution/blob/fc8c493/documentation/API.md#status) gives the full response, and [`Performance.md`](https://github.com/NCATSTranslator/NameResolution/blob/fc8c493/documentation/Performance.md) is the diagnostics guide, adapted from #253, which went to `ci` and won't reach `main`. A `CLAUDE.md` gotcha records how `/admin/metrics` encodes and names things, GC beans included. ## Verification All 90-odd tests pass on fc8c493 against Solr 9.10. `tests/test_status.py` checks each metrics block for real values against the test Solr, and fakes Solr for the failure paths and odd metrics. The cloud registry name was read from a cloud-mode Solr 9.10, and the GC bean names from Temurin 21 JVMs running G1, ZGC and Shenandoah. ## Before merging Nothing blocking. `Performance.md`'s GC-log section describes the chart as helxplatform/translator-devops#1044 left it: GC lines go to stdout, which was confirmed in ITRB CI's Solr logs on 5 November. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds
backendand an APIversionto/status, matching NodeNorm's/status(NCATSTranslator/NodeNormalization#392). Until now, nothing machine-readable said whether a NameRes/statuscame from this Solr implementation or from the Elasticsearch-backed one at biothings/NameResolutionAPI. The two differed only by a word of prose inmessage. That gap is what stopped TranslatorSRI/babel-validation from pinning a NameRes target to its expected backend.Breaking change: everything
/statusreads from Solr now sits in asolrobject rather than at the top level. That coversstartTime,numDocs,maxDoc,deletedDocs,segmentCount,lastModifiedandsize, plus the Solr index version, which was the top-levelversionand is nowsolr.version. The freed top-levelversionis the API version.Closes #296.
The calls worth overruling
solrobject, not a renamedindex_version. The API version needed the top-levelversion, so the Solr index version had to move. Renaming it in place would have fixed only that one key. With asolrobject, consumers adjust once, the key is named afterbackend's value, and backend detail can't collide with the service's own keys again. It is also the layout theimprove-performancebranch already uses (deployed asrelease-v1.5.2-with-metrics). That branch's JVM, OS and cache stats can therefore land in the same object without a second break. babel-validation'strim_status()already readsraw["solr"]first.solr.versionkeeps Solr's own name for the index version counter, as onimprove-performance, rather thansolr.index_version.solr.coreis new: the name of the core actually reported on. Because of the single-core fallback, it is not alwaysSOLR_CORE. Older cloud-mode backups call itname_lookup_shard1_replica_n1.versioncomes fromget_app_info(), notapp.openapi_schema. NodeNorm reads the latter, but here it isNoneuntil the first request to/openapi.json(Restore the x-translator, contact and termsOfService OpenAPI metadata dropped by FastAPI 0.137.0 #294), so a fresh process would report"unknown".backendandversion. The error branch (Expected core not found.) is what a mis-pointed deployment returns, which is exactly when a caller needs to know which NameRes it reached. The shared keys are built once ascommonand spread into each return, so the two can't drift. The error branch has nosolrobject.What's here
backend"solr"; the ES NameRes reports"elasticsearch"versioninfo.versionverbatim ("1.7.0")solrcore,startTime,numDocs,maxDoc,deletedDocs,version,segmentCount,lastModified,sizenameres_version("v1.7.0") is unchanged. API.md documents the new shape, with an example. A CLAUDE.md gotcha records why Solr fields must not go back to the top level.tests/test_status.pychecksversionagainst/openapi.json'sinfo.versionrather than a literal, so the two are tested to be identical. It also asserts that none of the old top-level Solr keys remain, so a partial revert fails.What it deliberately does not do
It doesn't fetch any more from Solr. JVM, OS and cache stats need extra Solr calls and a
?full=trueswitch; that isimprove-performance's work, and this PR only gives it a place to land.Verification
python -m pytest tests/passes in full (50-odd tests) on 639a7a2 against a local Solr 9.10 with the 89-document test index. I also checked three mocked Solr responses by hand:backendandversionand nosolrobjectname_lookup_shard1_replica_n1core:solr.corenames itBefore merging
api/resources/openapi.ymlis still at1.7.0, and API.md dates the move to v1.8.0. Bump it when the release is cut.version, now that it is a release rather than the index counter.🤖 Generated with Claude Code