Skip to content

Keep merge_message busy whenever there is work - #255

Merged
maximusunc merged 5 commits into
develfrom
claude/compassionate-thompson-omadnz
Nov 12, 2026
Merged

maximusunc merged 5 commits into
develfrom
claude/compassionate-thompson-omadnz

Conversation

@maximusunc

@maximusunc maximusunc commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Under heavy load, merge_message on a 2-CPU pod fell far behind, with more than 300 messages backed up in its queue. It used to keep up.

Two changes combined to cause it:

Together, both slots could sit in the debounce while the pool was idle. Lock losers, which only need to ack, waited behind them. Throughput dropped to about 2 queries a second.

Changes

  1. The in-flight task limit is higher than the pool size. The pool already waits for a free child, so the pool alone limits CPU and memory use. The extra in-flight tasks are coroutines in the parent process that hold only callback IDs.
  2. The debounce is removed (merge_debounce_sec is gone). The drain loop already re-reads the ready set after every pass, so callbacks that arrive during a pass go into the next one.
  3. A pass reads its batch only after it has claimed a pool child. ProcessPoolManager gains slot(), an async context manager that yields a PoolSlot whose run() executes on the claimed child. run() is now slot() plus slot.run(), so other workers are unchanged. merge_message claims a slot first and then reads and sizes its batch (_next_batch). Callbacks that arrive while a task waits for a CPU join that pass. Batching now comes from time the worker spends busy, never from time it sits idle.
  4. The in-flight limit is 2× the pool size (TASKS_PER_POOL_CHILD), so 4 on a 2-CPU pod. That keeps each child's next pass queued and lets lock losers ack without waiting behind merges. It also stays small, so that once this worker scales out, one busy pod doesn't hold backlog another idle pod could take. The TASK_LIMIT env var still overrides it.

Deploy notes

  • MERGE_DEBOUNCE_SEC is no longer read; remove it if it's set.
  • Don't set TASK_LIMIT unless you mean to override the 2× default.

Tests

  • New: test_merge_dispatch_limit.py checks the task limit is 2× the pool size. test_merge_passes.py checks that a pass starts immediately, that a callback arriving during a pass joins the next pass, and that callbacks arriving while a task waits for a child join that task's pass. test_process_pool.py checks that a slot stays claimed for its block and is released when the block raises.
  • The fake pools in the existing merge tests gain a slot() method.
  • The full unit suite has the same set of failures before and after this change. All of them come from packages missing in the environment where it was run, such as lmdb, biothings_annotator and biolink_helper_pkg.

🤖 Generated with Claude Code

https://claude.ai/code/session_01S2yuRAbvfoHwTUCX9uNik6

The poll loop's in-flight limit was the pool size (the pod's CPU count),
so on a 2-CPU pod only two wake tasks could run at once. Since the
merge debounce (#218) every lock holder sleeps up to merge_debounce_sec
before its first pass, unless merge_max_fold callbacks are already
ready. Both slots could sit in that sleep with the pool idle, and lock
losers that only need to ack queued behind them. Throughput fell to
about two queries a second and the stream backed up.

The process pool already waits for a free child, so it alone bounds the
CPU and memory heavy work. Run TASK_LIMIT tasks in flight and keep the
pool at the CPU count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRAbvfoHwTUCX9uNik6
After taking a query's lock, the holder waited up to merge_debounce_sec
(1s) for more callbacks before its first pass, and in practice almost
always waited the full second. That is CPU left idle while work is
queued, and it cost the most under heavy load, when the backlog needs
the worker busiest.

The drain loop already batches without waiting: it re-reads the ready
set after every pass, so callbacks that land during a pass are folded
by the next one, and batches grow on their own as passes slow down.
Start the first pass at once and remove the setting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRAbvfoHwTUCX9uNik6
merge_message read a query's ready callbacks, then waited for a free
pool child to merge them. Callbacks that landed during that wait were
left for a later pass, which reloads and re-saves the whole response
again. Under load, when tasks queue for the pool, that wait can be long.

ProcessPoolManager gains slot(): an async context manager that claims a
free child for the block and yields a PoolSlot whose run() executes in
it. run() is now slot() plus slot.run(), so other workers are unchanged.
merge_message claims a slot first and only then reads and sizes its
batch (_next_batch), so time spent queued for a CPU batches callbacks
the way the removed debounce meant to, without leaving a CPU idle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRAbvfoHwTUCX9uNik6
A fixed TASK_LIMIT of 10 let a 2-CPU pod pull five times the backlog it
can merge at once. With one replica that is harmless, but once the
worker scales out, tasks held by a busy pod wait on its CPUs while
another pod sits idle. Two tasks per pool child still keeps the next
pass queued for each child and lets lock losers ack without waiting
behind merges. The TASK_LIMIT env var still overrides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRAbvfoHwTUCX9uNik6
@codecov

codecov Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.56098% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 56.07%. Comparing base (de67685) to head (d40627e).
⚠️ Report is 4 commits behind head on devel.

Files with missing lines Patch % Lines
workers/merge_message/worker.py 96.29% 0 Missing and 1 partial ⚠️
Files with missing lines Coverage Δ
shepherd_utils/config.py 100.00% <ø> (ø)
shepherd_utils/process_pool.py 79.72% <100.00%> (+1.62%) ⬆️
workers/merge_message/worker.py 92.22% <96.29%> (+0.63%) ⬆️

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 20e0480...d40627e. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@maximusunc maximusunc changed the title Let merge_message keep more tasks in flight than its pool has children Keep merge_message busy whenever there is work Nov 12, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRAbvfoHwTUCX9uNik6
@maximusunc
maximusunc merged commit 25c8fb6 into devel Nov 12, 2026
2 checks passed
@maximusunc
maximusunc deleted the claude/compassionate-thompson-omadnz branch November 12, 2026 12:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants