Repository navigation
Add query-latency and Solr health metrics to /status, with a performance-diagnostics guide - #231
Conversation
There was a problem hiding this comment.
Pull request overview
Adds lightweight in-process tracking of recent lookup durations to help estimate service strain, exposing this data via /status to support the logging/monitoring goals described in #228.
Changes:
- Track recent lookup timings in a bounded in-memory deque.
- Extend
/statusresponse to include recent query timing statistics and samples. - Refactor lookup logging to reuse computed timing values.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Track Solr-only wait time in a separate deque so mean_solr_time_ms can be distinguished from total API time in /status - Pull query handler (requests/errors/timeouts/p75/p95/p99), cache (hitratio/evictions), and JVM (heap %, CPU load) from Solr's /admin/metrics API; fails gracefully with solr_metrics: null - Fix RECENT_TIMES_COUNT env var type cast (int) to prevent deque crash - Replace -1 sentinel with None for empty mean; remove verbose recent_times_ms list from response Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Combine two /admin/metrics calls into one (group=core&group=jvm), halving the round-trip overhead per /status request - Pin core key to name_lookup_shard1_replica_n1 instead of non- deterministic next() iteration over the metrics dict - Move raise_for_status() inside the async with block so all Solr I/O is co-located - Add test_status_shape and test_status_recent_queries_populated tests - Update API.md with recent_queries and solr_metrics response fields Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…atus The Solr metrics round-trip is skipped unless the caller explicitly passes ?metrics=true. The solr_metrics key is omitted from the response entirely when not requested. Tests and API.md updated accordingly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Adds some metrics for tracking performance on the Solr database: * Adds a `?full=true` mode to the /status endpoint which provides more detailed information from Solr, including memory/CPU information and cache information. * Maintains a "recent times" deque which is used to continuously report on recent queries and to inform people when query time is going up (reimplements part of PR #231). * Logs queries along with time taken for tracking performance in the long term (incorporates PR #230, which has been merged into branch `master` but not into branch `ci`). Also some unrelated changes: * Renames `reverse_lookup()` to `curie_lookup()` and `lookup_names_(get|post)` with `synonyms_(get|post)` for clarity. * Fixed a typo in data-loading/README.md.
# Conflicts: # api/server.py
- Use the single solr.core.* metrics registry instead of the hardcoded
name_lookup_shard1_replica_n1, so metrics populate under the standalone
name_lookup core (matches the SOLR_CORE handling already on main).
- API.md: document the recent_queries.max field and correct the solr_metrics
description (it holds a {message: ...} placeholder, not null, when metrics
aren't requested or Solr's metrics API is unavailable).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the Solr host resource and GC signals needed to right-size the Solr
pod's CPU/memory, all from the existing single /admin/metrics round-trip:
- host: available_processors, system/process CPU load, total/free physical
memory, file-descriptor counts. Solr mmaps its read-only index, so physical
RAM beyond the heap is OS page cache — this is what sizing decisions hinge on.
- jvm: gc_count and gc_time_ms (summed across whichever collectors run), a
direct heap-pressure signal.
Also fixes two latent parse bugs in the metrics block:
- heap gauge is returned by Solr as flat dotted keys (memory.heap.used), not a
nested object, so heap_used_mb/heap_max_mb/heap_used_pct were always null.
Now handles both shapes.
- errors/timeouts are meters ({count, meanRate, ...}), not scalars; report the
cumulative count instead of dumping the whole meter dict.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Performance.md said the name-lookup chart did not yet route Solr's GC log to stdout and that helxplatform/translator-devops#1044 would. #1044 merged on 2 November, and ITRB CI's Solr pod was logging GC lines to stdout by 5 November (seen in Grafana, with the 16G heap the chart sets). The section now leads with stdout as the default, shows a real ITRB CI line and how to read it, says how to check another deployment, and keeps the file-based solr_gc.log as the fallback when solr.gc_log is cleared. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A person reading /status mostly wants the versions, config and recent_queries; `solr` -- the core's stats and, with ?full=true, its query, cache, JVM and OS metrics -- is a long block of diagnostics, mainly useful for tracking and tuning performance. It was between config and recent_queries on the ok branch, pushing recent_queries below it; it is now the last key on both branches. JSON objects are unordered, so this only decides what a reader sees first; the comment in status() says nothing may depend on it. A test pins the order so it doesn't drift back. API.md's example and its paragraph order follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
recent_queries and solr.queries are NameRes's and Solr's views of the same traffic, meant to be read side by side, but named differently. Renamed while recent_queries is still unreleased (no tag contains it): count -> requests mean_time_ms -> mean_ms mean_solr_time_ms -> mean_solr_wait_ms (as the log line's "waiting for Solr") failed, timed_out -> errors, timeouts max -> window_size `max` read as a maximum latency next to the latency fields; it is the window's capacity. errors and timeouts are now disjoint, as in solr.queries, where `failed` used to include `timed_out`; their sum is the old `failed`. The two `timeouts` still mean different things, and the docs now say so. Solr's counts only requests Solr itself cut short and answered with partial results (as timeAllowed would; NameRes doesn't set it), so it normally stays at 0 -- Performance.md had said a non-zero value means Solr is dropping queries under load. NameRes's counts lookups that hit SOLR_TIMEOUT_SECONDS. The release-v1.5.2-with-metrics deployment reported mean_time_ms, mean_solr_time_ms and count, and babel-validation's trim_status() reads mean_time_ms; API.md notes the old names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…xist API.md said the Solr fields were at the top level "before NameRes v1.8.0". The release carrying #296 may well be v1.7.2, so the note now names the last release that had the old layout, which stays true whatever the next one is called. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng the rest /status?full=true read a core's metrics from solr.core.<coreName>. That is right for a standalone core, but the single-core fallback in status() exists for the old cloud-mode backups, whose core name_lookup_shard1_replica_n1 reports under solr.core.name_lookup.shard1.replica_n1 (checked against Solr 9.10 in cloud mode). The lookup missed, and every queries and cache field came back null with no metrics_error. core_metrics_registry() now builds the dotted name from the core's cloud descriptor. When neither name is present it reports a metrics_error alongside jvm and os, rather than falling back to whichever solr.core.* registry exists, which could report another core's figures as this one's. Parsing now checks each value before using it, so one unexpected value nulls only its own field. Before, a non-numeric heap maximum raised a TypeError in heap_used_pct, and the broad except threw away the query, cache and OS figures along with it. The outer except is still there as a backstop for a response that isn't a metrics response at all. Drops the nested 'memory.heap' branch: Solr 9 sends the JVM gauges as flat dotted keys (CLAUDE.md records this), so the branch never ran and had no test. test_status_metrics_param now checks that the query, JVM and GC figures have values, not only that their keys exist, as CLAUDE.md asks. A wrong metric name gives a null rather than an error, so the old key checks would have passed regardless. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd /status cheap recent_queries is documented as timing /lookup, but the recording sat in lookup(), which bulk_lookup() calls once per string. So one 1000-string request added 1000 entries to the window, and with every slow string logged as a SLOW QUERY warning, one bulk request against a struggling Solr could fill the log. lookup() now takes bulk=True from bulk_lookup(): its strings are left out of the window and logged at INFO however slow. Failures still log at WARNING. The failure timing caught Exception, which lets asyncio.CancelledError (a BaseException) through. A lookup cancelled while it waited on Solr was then neither timed nor logged, which is the gap the failure timing was added to close. It now catches BaseException, re-raises as before, and counts these as a new `cancelled` outcome. They aren't counted in `errors`, because a cancelled lookup says nothing about Solr. /status is what the Kubernetes probes call, and on every request it used to make several full passes over the 50000-entry window, on the event loop: Counter(), two sums and a percentile sort. The three index-aligned deques are now one RecentLookups. It keeps running totals in integer nanoseconds, which can't drift as entries are evicted, and reuses its percentiles for up to 10 s. The sort is the only part that has to look at the whole window (about 4 ms at 50000 entries on a laptop). mean_ms and mean_solr_wait_ms are now rounded to 2 places like the percentiles. A malformed or negative RECENT_TIMES_COUNT or SLOW_QUERY_THRESHOLD_MS used to raise at import and keep the API from starting. These settings only tune diagnostics, so they now log a warning and fall back to the default. The Config settings still fail loudly, since they change what the service returns. Docs: the means are null only until one query has run (the percentiles need two), and recent_queries' counts cover the window, while solr.queries' count from Solr's start. The two share names but can't be compared. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Latency percentiles can exceed observed values, unavailable GC telemetry is misreported as zero, and the GC test can fail on healthy instances.
7 open issues
Use inclusive quantiles to avoid extrapolated latency percentiles · New Avoid requiring a GC during the metrics test · New Avoid extrapolated percentiles for small timing windows Report unavailable GC metrics as null, not zero Distinguish heap pressure from page-cache pressure · New Qualify frontend-versus-API latency comparisons · New Use the cache settings that exist in repository config
1 resolved since last review
Review effort: Balanced
_gc_total() summed whatever numeric gc.* gauges it found, so finding none -- a misread metric name, or a Solr that stopped sending them -- read as a JVM that had never collected garbage. That contradicts API.md's rule that a field is null when its Solr metric is unavailable. It now returns None when no gauge matches. That gives the live-Solr test a better check than the one it had. It required gc_count > 0 to catch a misread name, but a healthy Solr with a large heap may not have collected yet. A misread name now shows up as None, so the test accepts any non-negative int. Copilot review on #231 (discussion_r4139025739, discussion_r4531100522). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
statistics.quantiles() defaults to the 'exclusive' method, which extrapolates past the observed values when the window is small. A freshly started instance with 1 ms and 100 ms lookups reported a p99 of 196 ms, which no caller experienced and which could set off a latency alert. It now uses 'inclusive'. In a full window the two methods agree to within rounding. Copilot review on #231 (discussion_r4139025781, discussion_r4531100400). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Performance.md said to raise `<maxSize>` for an evicting cache, but neither checked-in cache has that setting. filterCache is bounded by `size` (an entry count), and queryResultCache by `maxRamMB`, which makes Solr ignore `size` for it. Copilot review on #231 (discussion_r4139025816). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 3 called a heap under 50% "not a memory issue" and sent low-CPU slow queries on to GC. But Solr reads its index through mmap, so the memory it most needs is page cache, outside the heap (#265). A small heap on a pod with too little RAM for its index fits exactly that description and has nothing to do with GC. The step now checks spare RAM against the index size, and disk I/O on the Solr host, before moving on. Copilot review on #231 (discussion_r4531100594). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The docs said an end-to-end recent_queries.p99_ms well above solr.queries.p99_ms puts the tail in NameRes. The two aren't like for like, though. recent_queries covers /lookup over the last RECENT_TIMES_COUNT lookups, which can be hours, and its times include failures and the network. solr.queries covers every /select over a few minutes. A run of timeouts, or a Solr slowdown that has since passed, can open the gap with Python doing nothing wrong. Step 5 of the decision tree now lists what to rule out first, and the field descriptions in Performance.md and API.md call the comparison a pointer, not proof. Copilot review on #231 (discussion_r4531100673). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
recent_queries.mean_ms and solr.queries.mean_ms had matching names, which f36ae97 chose so the two blocks could be read side by side. They time different spans, though. On a local Solr the first was 12 ms and the second 1.35 ms, which reads as a discrepancy. In fact NameRes was adding the difference itself: lookup() builds a new httpx client and connection per query, and that took a 1 ms Solr query to about 8 ms. The latencies are now grouped by what they time: - recent_queries.lookup_ms: all of lookup(). - recent_queries.solr_round_trip_ms: the Solr request as NameRes sees it, including the client and connection setup and the JSON decoding. This was mean_solr_wait_ms. - solr.queries.handler_ms: only Solr's /select handler. Each latency block holds mean, min, max and p50/p95/p99, except solr_round_trip_ms, which has only the mean. recent_queries takes min and max from the sort its percentiles already do, so they share the 10-second reuse. Solr's come from its request timer, so its max is the slowest request still in Solr's recent sample. Solr's figures are rounded to 2 places, like NameRes's. Performance.md's Step 2 used to read "solr wait is most of the lookup" as "the problem is inside Solr". With the connection overhead counted in the round trip, that misdiagnosed it, so the step now compares all three spans. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CLAUDE.md says to check a new /admin/metrics field against a live Solr, but the cloud-mode registry name (solr.core.<collection>.<shard>.<replica>) can only be seen on a cloud-mode Solr, which the test setup doesn't run. This is the recipe that confirmed it for #231. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… too solr_round_trip_ms had only a mean, so a long tail in the connection, network or transfer between NameRes and Solr had nowhere to show. It now has the same six values as lookup_ms. A p99 or max well above solr.queries.handler_ms's points at that tail and away from Solr itself. The window already holds each lookup's round-trip time, so this is a second sort beside the one lookup_ms does. It is reused for the same 10 seconds, so a recomputation costs two sorts of the window rather than one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lookup log line said "(with Yms waiting for Solr)", and /status calls the same figure recent_queries.solr_round_trip_ms. "Waiting for Solr" also undersold what it covers, which includes the client and connection setup and the JSON decoding. The line now reads "(with Yms for the Solr round trip)". The line is an interface. TranslatorSRI/babel-validation#107's log-analysis parser matches its text, so that parser is being updated to accept both wordings: logs from v1.7.1 and earlier keep the old one. A code comment and a new test pin the format. Performance.md documents both, and its "≈ total" reading no longer sends a slow round trip straight to Solr: it says to compare against solr.queries.handler_ms first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
time.time_ns() can be stepped by NTP or a VM resuming while a lookup is in flight, which would put a negative or inflated duration into recent_queries' running totals, min, max and percentiles (and might log a false SLOW QUERY). perf_counter_ns() is monotonic, so every duration in server.py now uses it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The comment said babel-validation's log analysis parses this line but not where. The parser is LOOKUP_LINE_RE in babel-validation PR #107, which already accepts both 'waiting for Solr' and 'for the Solr round trip', and doesn't key on the log level. Naming it lets the next person check it before changing the wording. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both are documented as pause totals, but _gc_total() summed every gc.* gauge. On JDK 20+ G1 also registers a 'G1 Concurrent GC' bean, and ZGC and Shenandoah report '... Cycles' beside '... Pauses', so concurrent marking would have been counted as time Solr spent paused. An operator following Performance.md would then grow a heap that wasn't causing pauses. The bean names were read from Temurin 21 JVMs running G1, ZGC, generational ZGC and Shenandoah. The local solr:9.10 image runs JDK 17, which has only the two G1 generation beans, so nothing changes there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The JVM's MemoryUsage.getMax() is -1 when no maximum is defined. heap_used_pct only checked heap_max for truthiness, so -1 produced a negative percentage and heap_max_mb came out as -0.0. A non-positive maximum is now unknown, and _mb() treats any negative size the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only the Solr request was inside lookup()'s timing try/except, so a lookup that raised while building its results (a document LookupResult can't take, say) was a 500 to its caller but wasn't logged as a FAILED QUERY or counted in recent_queries.errors. /status could then report no failures while /lookup was returning them. The try now covers building the results too. Such a failure's solr_round_trip_ms ends when Solr answered, not when the lookup failed. Most of the diff is reindentation; git diff -w shows the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each failed bulk string logged its own FAILED QUERY warning, so one /bulk-lookup of 5000 strings against an unreachable Solr logged 5000 warnings: the flood the bulk flag exists to prevent. gather() also left every remaining string running after the first failure, each sent to a Solr that was probably down, though the request had already failed. bulk_lookup() now cancels the strings not yet done when one fails, waits for them, and logs a single FAILED BULK QUERY warning. The strings' own FAILED QUERY lines are logged at INFO. babel-validation's parser (PR #107) ignores log levels and skips lines without 'Lookup query to Solr', so it reads the per-string lines as before and skips the new one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The means were running totals, always current, while min, max and the percentiles were reused for up to 10 s. After a burst of slow lookups one /status response could report a mean above the max (the test showed mean 50.5 beside max 1.0). That reads as broken data just when latency is rising. Each series' mean is now computed in the same pass as its order statistics, so all of a series' figures describe one moment. The counts stay running totals and current. The latency running totals are gone; summing the window at recomputation time is cheap next to the sort it accompanies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


This PR makes
/statusreport how hard a NameRes instance and its Solr backend are working. It adds latency and failure counts over a rolling window of recent/lookupcalls, and logs slow and failed lookups at WARNING. Behind?full=true, it adds Solr's own query, cache, JVM and OS metrics to thesolrobject that #304 introduced. That gives monitoring something to alert on and us the inputs for sizing the Solr pod, and a new guide,documentation/Performance.md, explains how to read them. This addresses the/statuspart of #228 and supplies the inputs for tuning issues #265 and #267.The calls worth overruling
solr, following Report the backend and API version from/status, and group its Solr stats undersolr#304's rule that everything read from Solr goes there:solr.queries,solr.cache,solr.jvmandsolr.os. Onlyrecent_queries, which NameRes times itself, stays at the top level. They come from the core/statusreports on, cloud-mode cores included; if it has none,metrics_errorsays so rather than borrowing another core's.os, nothostornode: the figures are Solr'sos.*gauges, and in a container they are its limits (--cpus 2 --memory 1gon a 12-CPU machine reports 2 processors and 1024 MB).?full=true(the name v1.5.2 and Surface OS page-cache stats (Buffers/Cached) in /status?full=true #267 use), so the probes' plain/statusmakes no extra Solr round-trip. A failed fetch is reported assolr.metrics_error, not omitted.recent_queries.lookup_ms(all oflookup()),recent_queries.solr_round_trip_ms(the Solr request as NameRes sees it) andsolr.queries.handler_ms(Solr's handler alone) each nest inside the previous one. Matching flat names made a 12 ms lookup beside a 1.35 ms Solr mean look like a contradiction; most of that gap is the per-query connection Reuse one HTTP client for Solr requests instead of opening one per query #310 removes./statuswould look healthiest exactly when Solr is timing out.errors,timeoutsandcancelledare also counted on their own, because a fast failure such as a refused connection pulls the percentiles down./bulk-lookupstays out ofrecent_queriesand out of the WARNING log, since one request can hold thousands of strings. Its strings log at INFO, including failures. When one string fails, the request is a 500 regardless, so the strings not yet done are now cancelled rather than each sent to a Solr that is probably down, and the request logs oneFAILED BULK QUERYwarning.gc_count/gc_time_msare pause totals. They leave out collectors that time concurrent work (JDK 20+'sG1 Concurrent GC, ZGC's and Shenandoah's… Cycles), which would otherwise read as heap pressure. Solr 9.10's image runs JDK 17, where nothing is excluded.MemFree, so Surface OS page-cache stats (Buffers/Cached) in /status?full=true #267 needs a node-level exporter), file descriptors, and Release v1.5.2 with metrics #253's per-second rates (sample the counters twice instead).What's here
recent_queries, on every/status: the failure counts, pluslookup_msandsolr_round_trip_mswith mean, min, max and p50/p95/p99, over the lastRECENT_TIMES_COUNTlookups (default 50000)./lookupslower thanSLOW_QUERY_THRESHOLD_MS(default 500) logsSLOW QUERY: …, and a failed oneFAILED QUERY (timeout|error|cancelled: …). The Solr figure now reads(with Yms for the Solr round trip)where v1.7.1 wrotewaiting for Solr; Add a NameRes autocomplete log-analysis notebook and replayable Solr benchmark TranslatorSRI/babel-validation#107's log parser accepts both. A malformed setting logs a warning and falls back to its default.?full=true, from one/admin/metricsrequest:/selectcounts andhandler_ms,filterCacheandqueryResultCache, heap/CPU/GC, and processors/load/memory. Each value is checked on its own, so one odd gauge (or a heap maximum of -1) nulls only its own field.API.mdgives the full response, andPerformance.mdis the diagnostics guide, adapted from Release v1.5.2 with metrics #253, which went tociand won't reachmain. ACLAUDE.mdgotcha records how/admin/metricsencodes and names things, GC beans included.Verification
All 90-odd tests pass on fc8c493 against Solr 9.10.
tests/test_status.pychecks each metrics block for real values against the test Solr, and fakes Solr for the failure paths and odd metrics. The cloud registry name was read from a cloud-mode Solr 9.10, and the GC bean names from Temurin 21 JVMs running G1, ZGC and Shenandoah.Before merging
Nothing blocking.
Performance.md's GC-log section describes the chart as helxplatform/translator-devops#1044 left it: GC lines go to stdout, which was confirmed in ITRB CI's Solr logs on 5 November.🤖 Generated with Claude Code