Repository navigation
Fix pool workers failing on unpicklable loggers; nest operations under their ARA's span - #254
Merged
Merged
Conversation
Per-task loggers are unregistered since they were made to be freed with their task, so they can no longer be pickled. aragorn.omnicorp, aragorn.score, arax.rank and arax.pathfinder still passed theirs to the pool child, so every task failed with "logger cannot be pickled" and went straight to finish_query. These entrypoints now take the task's log level, log through a call-scoped logger in the child, and return the formatted records for the parent to fold into the query's logs -- the pattern score_paths already used, now shared as pool_task_logger / ingest_query_logs. As a side effect the children's records reach the query's logs rather than only the console. The pool budget test now dispatches with a real task logger and checks that every argument bound for the child pickles. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRDtq1MR3H5JiFLxVMv
Every operation in a workflow is handed the same trace carrier, so they are siblings under one span. That carrier was the one the query arrived with: the intake span. Under the ARS one intake span fans out to several ARAs, so every ARA's lookup, overlay, scoring and finish_query landed under ars.fanout, interleaved, while each ARA's own span sat beside them as an empty leaf. After a lookup's deeply nested callback and merge_message spans, the next operation appeared to jump back out to the top of the trace. The entry worker (aragorn, bte, ...) now hands on its own span's context, so an ARA's operations nest under that ARA. Operations still pass the carrier along unchanged and stay siblings of one another. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRDtq1MR3H5JiFLxVMv
Codecov Report❌ Patch coverage is
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
…tion The entry worker was recognised as the worker that isn't the head of the workflow. ARAX runs the whole query itself and names itself as the one operation, so it never matched and its finish_query stayed under the intake span. The first worker to hand a query on is now the one that hands on its own span, and it marks the task (otel_rooted) so later operations pass that carrier along. The failure hand-off to finish_query does the same, so a query that fails in its entry worker is also nested under it. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRDtq1MR3H5JiFLxVMv
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two cleanups from the recent run of PRs.
1. Pool workers failed every task with "logger cannot be pickled"
Per-task loggers became unregistered in "Free per-query loggers with their task or request" (
d4728d5), and an unregistered logger can't be pickled. Four workers still passed their task logger into the process-pool child, so every task died at dispatch and went straight tofinish_query:aragorn.omnicorp(where it was reported)aragorn.scorearax.rankarax.pathfinderTheir pool entrypoints now take the task's log level instead of the logger, log through a call-scoped logger in the child, and return the formatted records for the parent to fold into the query's logs. This is the pattern
score_pathsalready used; it is now shared aspool_task_logger/ingest_query_logsinshepherd_utils/logger.py, andscore_pathsuses the helpers too.Behaviour changes:
score_pathstoday).2. Trace: a query's operations now nest under its ARA's span
Every operation in a workflow is handed the same trace carrier, and that carrier was the one the query arrived with. Under the ARS that is the
ars.fanoutspan, so every ARA's lookup, omnicorp, score, sort andfinish_querylanded underars.fanout, interleaved, while thearagorn/btespans sat beside them as empty leaves. After a lookup's nested callback andmerge_messagespans, the next operation appeared to jump back to the top of the trace.The entry worker (
aragorn,bte,arax) now hands on its own span's context inwrap_up_task, givingars.fanout→aragorn→ lookup, omnicorp, score, sort,finish_query. Operations within an ARA remain siblings, so a slowaragorn.scoreshows under the ARA whose query it was. A direct/{ara}/querygets the same grouping.Testing
tests/unit/test_pool_budgets.pynow dispatches each pool worker with a real task logger and checks that everything bound for the child pickles. That check is what surfacedarax.pathfinder.tests/unit/test_logger.py(pool logger helpers) andtests/unit/test_shared_utils.py(carrier hand-off).Generated by Claude Code