Skip to content

Report the backend and API version from /status, and group its Solr stats under solr - #304

Merged
gaurav merged 4 commits into
mainfrom
add-backend-to-status
Nov 10, 2026
Merged

gaurav merged 4 commits into
mainfrom
add-backend-to-status

Conversation

@gaurav

@gaurav gaurav commented Sep 1, 2026 •

Copy link
Copy Markdown
Collaborator

This PR adds backend and an API version to /status, matching NodeNorm's /status (NCATSTranslator/NodeNormalization#392). Until now, nothing machine-readable said whether a NameRes /status came from this Solr implementation or from the Elasticsearch-backed one at biothings/NameResolutionAPI. The two differed only by a word of prose in message. That gap is what stopped TranslatorSRI/babel-validation from pinning a NameRes target to its expected backend.

Breaking change: everything /status reads from Solr now sits in a solr object rather than at the top level. That covers startTime, numDocs, maxDoc, deletedDocs, segmentCount, lastModified and size, plus the Solr index version, which was the top-level version and is now solr.version. The freed top-level version is the API version.

Closes #296.

The calls worth overruling

  • A solr object, not a renamed index_version. The API version needed the top-level version, so the Solr index version had to move. Renaming it in place would have fixed only that one key. With a solr object, consumers adjust once, the key is named after backend's value, and backend detail can't collide with the service's own keys again. It is also the layout the improve-performance branch already uses (deployed as release-v1.5.2-with-metrics). That branch's JVM, OS and cache stats can therefore land in the same object without a second break. babel-validation's trim_status() already reads raw["solr"] first.
  • solr.version keeps Solr's own name for the index version counter, as on improve-performance, rather than solr.index_version.
  • solr.core is new: the name of the core actually reported on. Because of the single-core fallback, it is not always SOLR_CORE. Older cloud-mode backups call it name_lookup_shard1_replica_n1.
  • version comes from get_app_info(), not app.openapi_schema. NodeNorm reads the latter, but here it is None until the first request to /openapi.json (Restore the x-translator, contact and termsOfService OpenAPI metadata dropped by FastAPI 0.137.0 #294), so a fresh process would report "unknown".
  • Both branches report backend and version. The error branch (Expected core not found.) is what a mis-pointed deployment returns, which is exactly when a caller needs to know which NameRes it reached. The shared keys are built once as common and spread into each return, so the two can't drift. The error branch has no solr object.

What's here

key value
backend always "solr"; the ES NameRes reports "elasticsearch"
version the OpenAPI info.version verbatim ("1.7.0")
solr core, startTime, numDocs, maxDoc, deletedDocs, version, segmentCount, lastModified, size

nameres_version ("v1.7.0") is unchanged. API.md documents the new shape, with an example. A CLAUDE.md gotcha records why Solr fields must not go back to the top level.

tests/test_status.py checks version against /openapi.json's info.version rather than a literal, so the two are tested to be identical. It also asserts that none of the old top-level Solr keys remain, so a partial revert fails.

What it deliberately does not do

It doesn't fetch any more from Solr. JVM, OS and cache stats need extra Solr calls and a ?full=true switch; that is improve-performance's work, and this PR only gives it a place to land.

Verification

python -m pytest tests/ passes in full (50-odd tests) on 639a7a2 against a local Solr 9.10 with the 89-document test index. I also checked three mocked Solr responses by hand:

  • two cores: the error branch, with backend and version and no solr object
  • a single name_lookup_shard1_replica_n1 core: solr.core names it
  • a sparse, still-initializing core: blank counts, not a 500

Before merging

  • api/resources/openapi.yml is still at 1.7.0, and API.md dates the move to v1.8.0. Bump it when the release is cut.
  • Follow-up in TranslatorSRI/babel-validation, not blocking this PR: its comment "under solr before v1.7, at the top level from v1.7 on" goes stale. It could also start reading NameRes's top-level version, now that it is a release rather than the index counter.

🤖 Generated with Claude Code

gaurav and others added 2 commits September 1, 2026 01:51
Adds two keys to /status, matching NodeNorm (NCATSTranslator/NodeNormalization#392):

  - backend: always "solr" here. The Elasticsearch-backed NameRes at
    biothings/NameResolutionAPI reports "elasticsearch". Until now the two
    differed only by a word of prose in `message`, so nothing machine-readable
    said which implementation answered -- which is what blocked
    TranslatorSRI/babel-validation from pinning a NameRes target to its expected
    backend the way it already does for NodeNorm.
  - version: the OpenAPI info.version verbatim, so a caller can tell when the API
    changed without fetching /openapi.json, and so NameRes and NodeNorm spell it
    the same way.

`version` was already taken by the Solr index version, which is renamed to
`index_version`. This is a breaking change for anyone reading the Solr index
version out of /status.

The keys both branches share are now built once as `common` and spread into each
return, so the error branch ("Expected core not found.") cannot drift from the ok
branch. That branch is what a mis-pointed or still-loading deployment returns, and
is exactly when a caller most needs to know which NameRes it reached.

test_status.py pins `version` against /openapi.json's info.version rather than a
literal, so the identity between the two is actually tested and the assertion
survives version bumps.

Closes #296.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
API.md describes `backend`, `version` and `index_version`, and calls out that
`index_version` was `version` before v1.8.0. Also refreshes the stale
nameres_version in the example, which still said v1.5.1.

The CLAUDE.md gotcha records why `version` means the API version here and not the
Solr index version, and why both branches of status() have to report `backend`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
gaurav and others added 2 commits November 10, 2026 14:51
The previous commit moved the Solr index version to a top-level `index_version` to
free `version` for the API version. Instead, every field read from Solr's core
STATUS (startTime, numDocs, maxDoc, deletedDocs, version, segmentCount,
lastModified, size) now sits in a `solr` object, and the index version keeps
Solr's own name there as `solr.version`.

Why `solr` rather than `index_version`:

  - /status is already changing incompatibly in this PR, so consumers adjust once
    rather than again when more Solr stats arrive.
  - It is the layout the improve-performance branch already uses (deployed as
    release-v1.5.2-with-metrics), whose JVM, OS and cache stats belong in the same
    object; babel-validation's trim_status() already reads raw["solr"] first.
  - The key is named after `backend`'s value, so backend-specific detail can't
    collide with the service's own keys again -- which is how the index version
    came to squat on `version` in the first place.

Also adds `solr.core`, the name of the core actually reported on: with the
single-core fallback it is not always SOLR_CORE (older cloud-mode backups call it
name_lookup_shard1_replica_n1). The error branch is unchanged and has no `solr`.

The test asserts the old flat keys are absent, so a partial revert fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
API.md describes the `solr` object and notes that its fields were at the top
level before v1.8.0, with `solr.version` formerly the top-level `version`. The
CLAUDE.md gotcha now says Solr-sourced fields go under `solr` and not back at the
top level.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gaurav gaurav changed the title Report the backend and API version from /status, renaming the Solr index version to index_version Report the backend and API version from /status, and group its Solr stats under solr Nov 10, 2026
@gaurav
gaurav merged commit a1011e1 into main Nov 10, 2026
1 check passed
@gaurav
gaurav deleted the add-backend-to-status branch November 10, 2026 19:57
@github-project-automation github-project-automation Bot moved this from Backlog to Done in NameRes sprints Nov 10, 2026
gaurav added a commit that referenced this pull request Nov 10, 2026
Brings in #304, which adds `backend` and the API `version` to /status and moves
the Solr core's stats under a `solr` object.

Conflicts in status(), tests/test_status.py and the API.md example were resolved
by taking main's `common` keys and `solr` object, and keeping this branch's
`recent_queries` and `solr_metrics` at the top level as they were. Both now
appear on the error branch too. test_status_shape looks for numDocs under
`solr`, and Performance.md's "Index health" section now points at `solr`
instead of the top level.

`solr_metrics` is still top-level, which conflicts with the CLAUDE.md gotcha
merged with #304 (Solr-sourced fields go under `solr`). Moving it is a design
change for this PR, not something for a merge to decide.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gaurav added a commit that referenced this pull request Nov 10, 2026
Everything /status reads from Solr goes under `solr` (the rule #304 set), so
the ?full=true metrics now join the core's stats there instead of sitting in a
top-level solr_metrics:

  solr_metrics.query_handler -> solr.queries
  solr_metrics.cache         -> solr.cache
  solr_metrics.jvm           -> solr.jvm
  solr_metrics.host          -> solr.os

`recent_queries` stays at the top level: NameRes times it end to end, so it is
not a Solr reading.

`os` rather than `host` or `node`: the figures are the JVM's operating-system
bean (Solr's own os.* gauges), and in a container that bean reports the
container's limits -- checked on solr:9.10 with --cpus 2 --memory 1g, which
reported 2 processors and 1 GiB on a 12-CPU machine -- so "host" overstates
them, and "node" would be read as the Kubernetes node.

Without ?full=true the metrics blocks are simply absent; the old
{"message": "Use /status?full=true..."} placeholder had no single place to live
once the blocks were spread across `solr`, so the hint moved to the `full`
parameter's OpenAPI description. A failed fetch is `solr.metrics_error`, so not
asking and failing still look different.

The fetch moves into fetch_solr_metrics(), called after the core is resolved,
so that it reads solr.core.<the resolved core> instead of whichever solr.core.*
came first. On the error branch (no core) ?full=true still reports `solr.jvm`
and `solr.os`, but no query or cache figures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gaurav added a commit that referenced this pull request Nov 10, 2026
…nce-diagnostics guide (#231)

This PR makes `/status` report how hard a NameRes instance and its Solr
backend are working. It adds latency and failure counts over a rolling
window of recent `/lookup` calls, and logs slow and failed lookups at
WARNING. Behind `?full=true`, it adds Solr's own query, cache, JVM and
OS metrics to the `solr` object that #304 introduced. That gives
monitoring something to alert on and us the inputs for sizing the Solr
pod, and a new guide, `documentation/Performance.md`, explains how to
read them. This addresses the `/status` part of #228 and supplies the
inputs for tuning issues #265 and #267.

## The calls worth overruling

- **Solr's metrics go under `solr`**, following #304's rule that
everything read from Solr goes there: `solr.queries`, `solr.cache`,
`solr.jvm` and `solr.os`. Only `recent_queries`, which NameRes times
itself, stays at the top level. They come from the core `/status`
reports on, cloud-mode cores included; if it has none, `metrics_error`
says so rather than borrowing another core's.
- **`os`, not `host` or `node`:** the figures are Solr's `os.*` gauges,
and in a container they are its limits (`--cpus 2 --memory 1g` on a
12-CPU machine reports 2 processors and 1024 MB).
- **Solr metrics only with `?full=true`** (the name v1.5.2 and #267
use), so the probes' plain `/status` makes no extra Solr round-trip. A
failed fetch is reported as `solr.metrics_error`, not omitted.
- **Latencies are named for the span they time.**
`recent_queries.lookup_ms` (all of `lookup()`),
`recent_queries.solr_round_trip_ms` (the Solr request as NameRes sees
it) and `solr.queries.handler_ms` (Solr's handler alone) each nest
inside the previous one. Matching flat names made a 12 ms lookup beside
a 1.35 ms Solr mean look like a contradiction; most of that gap is the
per-query connection #310 removes.
- **Latencies stay on the default path, cached for 10 seconds.** The
counts are running totals and always current. A series' mean, min, max
and percentiles need a pass over the window (about 4 ms at 50000
entries), so they are recomputed together at most every 10 s, and its
mean can never fall outside its min and max.
- **Failed lookups count in the latency window**, since their callers
waited too; otherwise `/status` would look healthiest exactly when Solr
is timing out. `errors`, `timeouts` and `cancelled` are also counted on
their own, because a fast failure such as a refused connection pulls the
percentiles *down*.
- **`/bulk-lookup` stays out of `recent_queries` and out of the WARNING
log**, since one request can hold thousands of strings. Its strings log
at INFO, including failures. When one string fails, the request is a 500
regardless, so the strings not yet done are now cancelled rather than
each sent to a Solr that is probably down, and the request logs one
`FAILED BULK QUERY` warning.
- **`gc_count`/`gc_time_ms` are pause totals.** They leave out
collectors that time concurrent work (JDK 20+'s `G1 Concurrent GC`,
ZGC's and Shenandoah's `… Cycles`), which would otherwise read as heap
pressure. Solr 9.10's image runs JDK 17, where nothing is excluded.
- **Deliberately left out:** OS page cache (Solr exposes only `MemFree`,
so #267 needs a node-level exporter), file descriptors, and #253's
per-second rates (sample the counters twice instead).

## What's here

- **`recent_queries`, on every `/status`:** the failure counts, plus
`lookup_ms` and `solr_round_trip_ms` with mean, min, max and
p50/p95/p99, over the last `RECENT_TIMES_COUNT` lookups (default 50000).
- **Logging:** a `/lookup` slower than `SLOW_QUERY_THRESHOLD_MS`
(default 500) logs `SLOW QUERY: …`, and a failed one `FAILED QUERY
(timeout|error|cancelled: …)`. The Solr figure now reads `(with Yms for
the Solr round trip)` where v1.7.1 wrote `waiting for Solr`;
TranslatorSRI/babel-validation#107's log parser accepts both. A
malformed setting logs a warning and falls back to its default.
- **With `?full=true`, from one `/admin/metrics` request:** `/select`
counts and `handler_ms`, `filterCache` and `queryResultCache`,
heap/CPU/GC, and processors/load/memory. Each value is checked on its
own, so one odd gauge (or a heap maximum of -1) nulls only its own
field.
- **Docs:**
[`API.md`](https://github.com/NCATSTranslator/NameResolution/blob/fc8c493/documentation/API.md#status)
gives the full response, and
[`Performance.md`](https://github.com/NCATSTranslator/NameResolution/blob/fc8c493/documentation/Performance.md)
is the diagnostics guide, adapted from #253, which went to `ci` and
won't reach `main`. A `CLAUDE.md` gotcha records how `/admin/metrics`
encodes and names things, GC beans included.

## Verification

All 90-odd tests pass on fc8c493 against Solr 9.10.
`tests/test_status.py` checks each metrics block for real values against
the test Solr, and fakes Solr for the failure paths and odd metrics. The
cloud registry name was read from a cloud-mode Solr 9.10, and the GC
bean names from Temurin 21 JVMs running G1, ZGC and Shenandoah.

## Before merging

Nothing blocking. `Performance.md`'s GC-log section describes the chart
as helxplatform/translator-devops#1044 left it: GC lines go to stdout,
which was confirmed in ITRB CI's Solr logs on 5 November.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Report the backend (Solr or Elasticsearch) in /status, as NodeNorm does

1 participant