Skip to content

One shared worker poll loop (#223 C5-4) - #247

Merged
maximusunc merged 1 commit into
develfrom
audit/223-poll-loop
Nov 10, 2026
Merged

maximusunc merged 1 commit into
develfrom
audit/223-poll-loop

Conversation

@maximusunc

Copy link
Copy Markdown
Collaborator

Part 1 of 4 for #223 (C5-4). The C6-3/C9-5 lookup PR, the C11-3 pool budget PR and the C11-5 base image PR follow; the last of them closes the issue.

Problem

Each of the 25 workers had its own copy of the poll loop, and the copies had drifted:

  • Tasks without a reference. Most loops called asyncio.create_task(...) and kept no reference to the task. The event loop holds only a weak one, so a running task could be garbage-collected mid-flight.
  • Cancellation swallowed. The except Exception / except BaseException arms caught CancelledError and went back to polling. Cancelling a worker never stopped it, and neither in-flight tasks nor process pools were cleaned up on that path.

Change

  • shepherd_utils.shared.run_poll_loop(stream, group, consumer, task_limit, handler, on_stop=None) is now the single loop:
    • it keeps a strong reference to every task it starts;
    • on cancellation it cancels those tasks, runs on_stop (sync or async) and returns;
    • any other polling error is logged and polling resumes after 5 s, as before.
  • All 25 workers call it:
    • pool workers pass on_stop=pool.shutdown;
    • ars_notify passes its delivery drain;
    • merge_message's own inflight-set loop folds into it.
  • Tests:
    • Tests that patched a worker module's get_tasks now patch shared.get_tasks, which is where the loop reads it.
    • The static guard in test_worker_dispatch_concurrency.py now requires run_poll_loop(, and its docstring no longer says the loop can't return.

Tests

New tests/unit/test_poll_loop.py:

  • Cancelling the loop makes it return, cancels its in-flight tasks and runs on_stop.
  • An async on_stop is awaited.
  • A running task survives gc.collect().
  • A polling error is logged and polling resumes.

These are new tests of a new function, so on devel they fail only with a missing attribute. The old behaviour is visible in the removed loop bodies in this diff.

Validation

  • Full unit suite against real Postgres: 2198 passed. The deselected test_query_parity fails locally only because of Python 3.13 caret formatting; CI runs 3.12.
  • black --check . is clean.
  • Every worker module imports cleanly.

Generated by Claude Code

Every worker carried its own copy of the poll loop, and the copies had
drifted:
- most fired asyncio.create_task without keeping a reference, so a
  running task could be garbage-collected mid-flight;
- their except arms swallowed CancelledError and went back to polling,
  so cancelling a worker never stopped it, and in-flight tasks and pools
  were never cleaned up.

shared.run_poll_loop is the single loop now. It keeps a reference to
every task it starts. On cancellation it cancels those tasks, runs the
worker's on_stop (pool shutdown, ars_notify's delivery drain) and
returns. Any other polling error is logged and retried after a pause,
as before. All 25 workers call it, and merge_message's inflight loop
folds into it.

Tests that patched a worker's get_tasks now patch shared's, which is
where the loop reads it. The static dispatch guard now requires
run_poll_loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QGsF4qJ2EoSTDy2M2RRTVm
@codecov

codecov Bot commented Nov 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 64.70588% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 55.97%. Comparing base (e495dd9) to head (4e3dd02).
⚠️ Report is 1 commits behind head on devel.

Files with missing lines Patch % Lines
shepherd_utils/shared.py 95.45% 0 Missing and 1 partial ⚠️
workers/aragorn/worker.py 0.00% 1 Missing ⚠️
workers/aragorn_lookup/worker.py 50.00% 1 Missing ⚠️
workers/aragorn_omnicorp/worker.py 50.00% 1 Missing ⚠️
workers/aragorn_pathfinder/worker.py 0.00% 1 Missing ⚠️
workers/aragorn_score/worker.py 50.00% 1 Missing ⚠️
workers/arax/worker.py 50.00% 1 Missing ⚠️
workers/arax_pathfinder/worker.py 50.00% 1 Missing ⚠️
workers/arax_rank/worker.py 50.00% 1 Missing ⚠️
workers/ars_fanout/worker.py 50.00% 1 Missing ⚠️
... and 14 more
Files with missing lines Coverage Δ
workers/ars_notify/worker.py 76.85% <100.00%> (+3.01%) ⬆️
workers/merge_message/worker.py 91.59% <100.00%> (+0.40%) ⬆️
shepherd_utils/shared.py 82.87% <95.45%> (+0.46%) ⬆️
workers/aragorn/worker.py 90.62% <0.00%> (+18.12%) ⬆️
workers/aragorn_lookup/worker.py 89.94% <50.00%> (+3.47%) ⬆️
workers/aragorn_omnicorp/worker.py 81.42% <50.00%> (+2.49%) ⬆️
workers/aragorn_pathfinder/worker.py 83.33% <0.00%> (+1.93%) ⬆️
workers/aragorn_score/worker.py 93.56% <50.00%> (+1.68%) ⬆️
workers/arax/worker.py 86.07% <50.00%> (+1.87%) ⬆️
workers/arax_pathfinder/worker.py 70.00% <50.00%> (+2.69%) ⬆️
... and 16 more

... and 2 files with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f633000...4e3dd02. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@maximusunc
maximusunc merged commit 9b7e02e into devel Nov 10, 2026
2 checks passed
@maximusunc
maximusunc deleted the audit/223-poll-loop branch November 10, 2026 23:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants