Skip to content

Fix pool workers failing on unpicklable loggers; nest operations under their ARA's span - #254

Merged
maximusunc merged 3 commits into
develfrom
claude/dazzling-ride-skev3p
Nov 12, 2026
Merged

maximusunc merged 3 commits into
develfrom
claude/dazzling-ride-skev3p

Conversation

@maximusunc

Copy link
Copy Markdown
Collaborator

Two cleanups from the recent run of PRs.

1. Pool workers failed every task with "logger cannot be pickled"

Per-task loggers became unregistered in "Free per-query loggers with their task or request" (d4728d5), and an unregistered logger can't be pickled. Four workers still passed their task logger into the process-pool child, so every task died at dispatch and went straight to finish_query:

  • aragorn.omnicorp (where it was reported)
  • aragorn.score
  • arax.rank
  • arax.pathfinder

Their pool entrypoints now take the task's log level instead of the logger, log through a call-scoped logger in the child, and return the formatted records for the parent to fold into the query's logs. This is the pattern score_paths already used; it is now shared as pool_task_logger / ingest_query_logs in shepherd_utils/logger.py, and score_paths uses the helpers too.

Behaviour changes:

  • Log lines from these children now appear in the query's TRAPI logs. Before, they only reached the console.
  • If a child raises, its records are not returned (same as score_paths today).

2. Trace: a query's operations now nest under its ARA's span

Every operation in a workflow is handed the same trace carrier, and that carrier was the one the query arrived with. Under the ARS that is the ars.fanout span, so every ARA's lookup, omnicorp, score, sort and finish_query landed under ars.fanout, interleaved, while the aragorn / bte spans sat beside them as empty leaves. After a lookup's nested callback and merge_message spans, the next operation appeared to jump back to the top of the trace.

The entry worker (aragorn, bte, arax) now hands on its own span's context in wrap_up_task, giving ars.fanout → aragorn → lookup, omnicorp, score, sort, finish_query. Operations within an ARA remain siblings, so a slow aragorn.score shows under the ARA whose query it was. A direct /{ara}/query gets the same grouping.

Testing

  • Full unit suite passes locally (2363 passed, 246 skipped at the first commit; the second commit's suite run had one over-strict assertion in a new test, fixed and re-run with the related files).
  • tests/unit/test_pool_budgets.py now dispatches each pool worker with a real task logger and checks that everything bound for the child pickles. That check is what surfaced arax.pathfinder.
  • New tests in tests/unit/test_logger.py (pool logger helpers) and tests/unit/test_shared_utils.py (carrier hand-off).
  • Not run in the compose stack, and the trace shape has not been checked against a live trace.

Generated by Claude Code

Per-task loggers are unregistered since they were made to be freed with
their task, so they can no longer be pickled. aragorn.omnicorp,
aragorn.score, arax.rank and arax.pathfinder still passed theirs to the
pool child, so every task failed with "logger cannot be pickled" and
went straight to finish_query.

These entrypoints now take the task's log level, log through a
call-scoped logger in the child, and return the formatted records for
the parent to fold into the query's logs -- the pattern score_paths
already used, now shared as pool_task_logger / ingest_query_logs. As a
side effect the children's records reach the query's logs rather than
only the console.

The pool budget test now dispatches with a real task logger and checks
that every argument bound for the child pickles.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRDtq1MR3H5JiFLxVMv
Every operation in a workflow is handed the same trace carrier, so they
are siblings under one span. That carrier was the one the query arrived
with: the intake span. Under the ARS one intake span fans out to several
ARAs, so every ARA's lookup, overlay, scoring and finish_query landed
under ars.fanout, interleaved, while each ARA's own span sat beside them
as an empty leaf. After a lookup's deeply nested callback and
merge_message spans, the next operation appeared to jump back out to the
top of the trace.

The entry worker (aragorn, bte, ...) now hands on its own span's
context, so an ARA's operations nest under that ARA. Operations still
pass the carrier along unchanged and stay siblings of one another.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRDtq1MR3H5JiFLxVMv
@codecov

codecov Bot commented Nov 11, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.00000% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.07%. Comparing base (a137231) to head (de67685).
⚠️ Report is 3 commits behind head on devel.

Files with missing lines Patch % Lines
workers/arax_rank/worker.py 36.36% 7 Missing ⚠️
workers/aragorn_omnicorp/worker.py 86.66% 0 Missing and 2 partials ⚠️
workers/aragorn_score/worker.py 91.66% 2 Missing ⚠️
workers/score_paths/worker.py 66.66% 1 Missing ⚠️
Files with missing lines Coverage Δ
shepherd_utils/logger.py 100.00% <100.00%> (ø)
shepherd_utils/shared.py 83.04% <100.00%> (+0.16%) ⬆️
workers/arax_pathfinder/worker.py 70.44% <100.00%> (+0.44%) ⬆️
workers/score_paths/worker.py 76.23% <66.66%> (+4.94%) ⬆️
workers/aragorn_omnicorp/worker.py 81.64% <86.66%> (+0.21%) ⬆️
workers/aragorn_score/worker.py 93.60% <91.66%> (+0.04%) ⬆️
workers/arax_rank/worker.py 66.66% <36.36%> (-2.39%) ⬇️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 942b8c1...de67685. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…tion

The entry worker was recognised as the worker that isn't the head of the
workflow. ARAX runs the whole query itself and names itself as the one
operation, so it never matched and its finish_query stayed under the
intake span.

The first worker to hand a query on is now the one that hands on its own
span, and it marks the task (otel_rooted) so later operations pass that
carrier along. The failure hand-off to finish_query does the same, so a
query that fails in its entry worker is also nested under it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRDtq1MR3H5JiFLxVMv
@maximusunc
maximusunc merged commit 5c640a8 into devel Nov 12, 2026
@maximusunc
maximusunc deleted the claude/dazzling-ride-skev3p branch November 12, 2026 01:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants