Repository navigation
One shared lookup submission; plain lookups fail loudly (#223 C6-3, C9-5) - #248
Merged
Merged
Conversation
…9-5) C9-5: aragorn_lookup, bte_lookup and aragorn_pathfinder each carried their own copy of the submit / triage / wait code, and the copies had drifted. Only aragorn bounded its in-flight submissions per process, and pathfinder's wait swallowed CancelledError. They now share shepherd_utils.lookups: - run_async_lookup registers the callback with the submission's trace context, then POSTs; - submit_lookups bounds submissions per process (MAX_LOOKUPS_IN_FLIGHT), drops the callback ids of submissions that failed outright and keeps the unacknowledged ones; - wait_for_callbacks waits, then clears whatever is still outstanding. bte's template groups, template index and template texts are now read once per process, not on every inferred query. C6-3: the plain (non-inferred) lookup in aragorn_lookup and bte_lookup registered its callback and POSTed with no error handling. A rejected or failed POST left the query waiting out its whole lookup timeout and then finishing as an empty but successful answer. It now goes through submit_lookups like the expanded ones. When every submission failed outright, the query now fails with a 502 QueryError naming the first error, instead of waiting on nothing. This also applies to inferred and pathfinder queries whose lookups were all rejected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QGsF4qJ2EoSTDy2M2RRTVm
Codecov Report✅ All modified and coverable lines are covered by tests.
... and 1 file with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
maximusunc
pushed a commit
that referenced
this pull request
Nov 11, 2026
#248 moved MAX_LOOKUPS_IN_FLIGHT out of aragorn_lookup and aragorn_pathfinder into shepherd_utils/lookups.py. This takes devel's workers and derives the bound there from POOL_MAX_SIZE, as this branch did in the workers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QGsF4qJ2EoSTDy2M2RRTVm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 2 of 4 for #223. It is independent of #247 (the poll loop), and both branch from devel.
C6-3: the plain lookup had no error handling
For non-inferred queries,
aragorn_lookupandbte_lookupregistered a callback and then made a bareclient.post(...). Nothing looked at the result:Either way, the query waited out its whole lookup timeout for a callback that was never coming, then finished as an empty but successful answer. The plain lookup now goes through the same submission as the expanded ones.
C9-5: one shared lookup helper
aragorn_lookup,bte_lookupandaragorn_pathfindereach had their own copy ofAsyncResponse,run_async_lookup, the gather-and-triage block and the wait-and-cleanup block. The copies had drifted:CancelledError, so a cancelled query went on to build paths.They now share the new
shepherd_utils/lookups.py:run_async_lookup(client, message, query_id, ara, logger, ...)arapicks the/<ara>/callback/<id>route.submit_lookups(messages, query_id, ara, logger, ...)MAX_LOOKUPS_IN_FLIGHTper process across every task. Drops the callback ids of submissions that failed outright and keeps the unacknowledged (ReadTimeout) ones.wait_for_callbacks(query_id, response_id, max_query_time, logger)bte's
template_groups.json, the template index and each template file are now read once per process (functools.cache). Previously every inferred query re-read all of them.Behaviour change to review
When every submission to the retrieval service failed outright, the query now fails with a 502 (
QueryError) naming the first error, instead of waiting on nothing and returning an empty answer as success. This follows the "fail loudly" call from #243. It applies to inferred and pathfinder queries too, but only when none of their lookups were accepted. If a single lookup was accepted or is unacknowledged, the query waits as before.Out of scope / notes
examine_queryis also duplicated, but the lookup workers' copy andshared.examine_querydisagree on what a pathfinder query is (three inferred edges vspaths), so I left both alone.example_lookup(the template worker) keeps its own copies.aragorn_lookupsets an unuseduse_gandalfvariable. That was already the case on devel, and I left it.Tests
shepherd_utils.lookups.New
tests/unit/test_lookups.py, run for both aragorn and bte where it applies:Validation
test_query_parityfails locally only because of Python 3.13 caret formatting; CI runs 3.12.black --check .is clean.pyflakesreports no unused names in the touched workers.Generated by Claude Code