Skip to content

One shared lookup submission; plain lookups fail loudly (#223 C6-3, C9-5) - #248

Merged
maximusunc merged 1 commit into
develfrom
audit/223-lookup
Nov 10, 2026
Merged

maximusunc merged 1 commit into
develfrom
audit/223-lookup

Conversation

@maximusunc

Copy link
Copy Markdown
Collaborator

Part 2 of 4 for #223. It is independent of #247 (the poll loop), and both branch from devel.

C6-3: the plain lookup had no error handling

For non-inferred queries, aragorn_lookup and bte_lookup registered a callback and then made a bare client.post(...). Nothing looked at the result:

  • a 4xx/5xx response was ignored;
  • a connection error escaped with the callback still registered.

Either way, the query waited out its whole lookup timeout for a callback that was never coming, then finished as an empty but successful answer. The plain lookup now goes through the same submission as the expanded ones.

C9-5: one shared lookup helper

aragorn_lookup, bte_lookup and aragorn_pathfinder each had their own copy of AsyncResponse, run_async_lookup, the gather-and-triage block and the wait-and-cleanup block. The copies had drifted:

  • only aragorn bounded in-flight submissions per process;
  • pathfinder's wait swallowed CancelledError, so a cancelled query went on to build paths.

They now share the new shepherd_utils/lookups.py:

Function What it does
run_async_lookup(client, message, query_id, ara, logger, ...) Registers the callback with the submission's trace context, then POSTs. ara picks the /<ara>/callback/<id> route.
submit_lookups(messages, query_id, ara, logger, ...) Runs at most MAX_LOOKUPS_IN_FLIGHT per process across every task. Drops the callback ids of submissions that failed outright and keeps the unacknowledged (ReadTimeout) ones.
wait_for_callbacks(query_id, response_id, max_query_time, logger) Waits, then clears whatever is still outstanding. Cancellation propagates.

bte's template_groups.json, the template index and each template file are now read once per process (functools.cache). Previously every inferred query re-read all of them.

Behaviour change to review

When every submission to the retrieval service failed outright, the query now fails with a 502 (QueryError) naming the first error, instead of waiting on nothing and returning an empty answer as success. This follows the "fail loudly" call from #243. It applies to inferred and pathfinder queries too, but only when none of their lookups were accepted. If a single lookup was accepted or is unacknowledged, the query waits as before.

Out of scope / notes

  • examine_query is also duplicated, but the lookup workers' copy and shared.examine_query disagree on what a pathfinder query is (three inferred edges vs paths), so I left both alone.
  • example_lookup (the template worker) keeps its own copies.
  • aragorn_lookup sets an unused use_gandalf variable. That was already the case on devel, and I left it.

Tests

  • Tests that patched these names on the worker modules now patch shepherd_utils.lookups.
  • Two bte tests where the only lookup was rejected now expect the 502.

New tests/unit/test_lookups.py, run for both aragorn and bte where it applies:

  • a rejected plain lookup raises a 502, drops its callback and does not wait;
  • an unreachable service fails the plain lookup;
  • an unacknowledged plain lookup is still waited for;
  • the plain lookup uses its own trace carrier and the right callback route;
  • one accepted lookup out of three is enough to wait on;
  • an empty submission is not a failure;
  • the callback wait can be cancelled;
  • bte templates are read once over repeated expansions.

Validation

  • Full unit suite against real Postgres: 2206 passed. The deselected test_query_parity fails locally only because of Python 3.13 caret formatting; CI runs 3.12.
  • black --check . is clean.
  • pyflakes reports no unused names in the touched workers.

Generated by Claude Code

…9-5)

C9-5: aragorn_lookup, bte_lookup and aragorn_pathfinder each carried
their own copy of the submit / triage / wait code, and the copies had
drifted. Only aragorn bounded its in-flight submissions per process,
and pathfinder's wait swallowed CancelledError. They now share
shepherd_utils.lookups:
- run_async_lookup registers the callback with the submission's trace
  context, then POSTs;
- submit_lookups bounds submissions per process
  (MAX_LOOKUPS_IN_FLIGHT), drops the callback ids of submissions that
  failed outright and keeps the unacknowledged ones;
- wait_for_callbacks waits, then clears whatever is still outstanding.

bte's template groups, template index and template texts are now read
once per process, not on every inferred query.

C6-3: the plain (non-inferred) lookup in aragorn_lookup and
bte_lookup registered its callback and POSTed with no error handling.
A rejected or failed POST left the query waiting out its whole lookup
timeout and then finishing as an empty but successful answer. It now
goes through submit_lookups like the expanded ones.

When every submission failed outright, the query now fails with a 502
QueryError naming the first error, instead of waiting on nothing. This
also applies to inferred and pathfinder queries whose lookups were all
rejected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QGsF4qJ2EoSTDy2M2RRTVm
@codecov

codecov Bot commented Nov 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 55.61%. Comparing base (e495dd9) to head (1e08532).
⚠️ Report is 1 commits behind head on devel.

Files with missing lines Coverage Δ
shepherd_utils/lookups.py 100.00% <100.00%> (ø)
workers/aragorn_lookup/worker.py 85.41% <100.00%> (-1.06%) ⬇️
workers/aragorn_pathfinder/worker.py 84.58% <100.00%> (+3.19%) ⬆️
workers/bte_lookup/worker.py 84.21% <100.00%> (-2.43%) ⬇️

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f633000...1e08532. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@maximusunc
maximusunc merged commit 8f53952 into devel Nov 10, 2026
2 checks passed
@maximusunc
maximusunc deleted the audit/223-lookup branch November 10, 2026 23:46
maximusunc pushed a commit that referenced this pull request Nov 11, 2026
#248 moved MAX_LOOKUPS_IN_FLIGHT out of aragorn_lookup and
aragorn_pathfinder into shepherd_utils/lookups.py. This takes devel's
workers and derives the bound there from POOL_MAX_SIZE, as this branch
did in the workers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QGsF4qJ2EoSTDy2M2RRTVm
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants