Repository navigation
One shared worker poll loop (#223 C5-4) - #247
Merged
Merged
Conversation
Every worker carried its own copy of the poll loop, and the copies had drifted: - most fired asyncio.create_task without keeping a reference, so a running task could be garbage-collected mid-flight; - their except arms swallowed CancelledError and went back to polling, so cancelling a worker never stopped it, and in-flight tasks and pools were never cleaned up. shared.run_poll_loop is the single loop now. It keeps a reference to every task it starts. On cancellation it cancels those tasks, runs the worker's on_stop (pool shutdown, ars_notify's delivery drain) and returns. Any other polling error is logged and retried after a pause, as before. All 25 workers call it, and merge_message's inflight loop folds into it. Tests that patched a worker's get_tasks now patch shared's, which is where the loop reads it. The static dispatch guard now requires run_poll_loop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QGsF4qJ2EoSTDy2M2RRTVm
This was referenced Nov 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 1 of 4 for #223 (C5-4). The C6-3/C9-5 lookup PR, the C11-3 pool budget PR and the C11-5 base image PR follow; the last of them closes the issue.
Problem
Each of the 25 workers had its own copy of the poll loop, and the copies had drifted:
asyncio.create_task(...)and kept no reference to the task. The event loop holds only a weak one, so a running task could be garbage-collected mid-flight.except Exception/except BaseExceptionarms caughtCancelledErrorand went back to polling. Cancelling a worker never stopped it, and neither in-flight tasks nor process pools were cleaned up on that path.Change
shepherd_utils.shared.run_poll_loop(stream, group, consumer, task_limit, handler, on_stop=None)is now the single loop:on_stop(sync or async) and returns;on_stop=pool.shutdown;ars_notifypasses its delivery drain;merge_message's own inflight-set loop folds into it.get_tasksnow patchshared.get_tasks, which is where the loop reads it.test_worker_dispatch_concurrency.pynow requiresrun_poll_loop(, and its docstring no longer says the loop can't return.Tests
New
tests/unit/test_poll_loop.py:on_stop.on_stopis awaited.gc.collect().These are new tests of a new function, so on devel they fail only with a missing attribute. The old behaviour is visible in the removed loop bodies in this diff.
Validation
test_query_parityfails locally only because of Python 3.13 caret formatting; CI runs 3.12.black --check .is clean.Generated by Claude Code