Repository navigation
Keep merge_message busy whenever there is work - #255
Merged
Merged
Conversation
The poll loop's in-flight limit was the pool size (the pod's CPU count), so on a 2-CPU pod only two wake tasks could run at once. Since the merge debounce (#218) every lock holder sleeps up to merge_debounce_sec before its first pass, unless merge_max_fold callbacks are already ready. Both slots could sit in that sleep with the pool idle, and lock losers that only need to ack queued behind them. Throughput fell to about two queries a second and the stream backed up. The process pool already waits for a free child, so it alone bounds the CPU and memory heavy work. Run TASK_LIMIT tasks in flight and keep the pool at the CPU count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRAbvfoHwTUCX9uNik6
After taking a query's lock, the holder waited up to merge_debounce_sec (1s) for more callbacks before its first pass, and in practice almost always waited the full second. That is CPU left idle while work is queued, and it cost the most under heavy load, when the backlog needs the worker busiest. The drain loop already batches without waiting: it re-reads the ready set after every pass, so callbacks that land during a pass are folded by the next one, and batches grow on their own as passes slow down. Start the first pass at once and remove the setting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRAbvfoHwTUCX9uNik6
merge_message read a query's ready callbacks, then waited for a free pool child to merge them. Callbacks that landed during that wait were left for a later pass, which reloads and re-saves the whole response again. Under load, when tasks queue for the pool, that wait can be long. ProcessPoolManager gains slot(): an async context manager that claims a free child for the block and yields a PoolSlot whose run() executes in it. run() is now slot() plus slot.run(), so other workers are unchanged. merge_message claims a slot first and only then reads and sizes its batch (_next_batch), so time spent queued for a CPU batches callbacks the way the removed debounce meant to, without leaving a CPU idle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRAbvfoHwTUCX9uNik6
A fixed TASK_LIMIT of 10 let a 2-CPU pod pull five times the backlog it can merge at once. With one replica that is harmless, but once the worker scales out, tasks held by a busy pod wait on its CPUs while another pod sits idle. Two tasks per pool child still keeps the next pass queued for each child and lets lock losers ack without waiting behind merges. The TASK_LIMIT env var still overrides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRAbvfoHwTUCX9uNik6
Codecov Report❌ Patch coverage is
... and 1 file with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRAbvfoHwTUCX9uNik6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Under heavy load, merge_message on a 2-CPU pod fell far behind, with more than 300 messages backed up in its queue. It used to keep up.
Two changes combined to cause it:
c16aae5) made every query's lock holder wait up to 1s before its first merge. In practice it waited the full second almost every time, because it only stopped early once 25 callbacks were ready.Together, both slots could sit in the debounce while the pool was idle. Lock losers, which only need to ack, waited behind them. Throughput dropped to about 2 queries a second.
Changes
merge_debounce_secis gone). The drain loop already re-reads the ready set after every pass, so callbacks that arrive during a pass go into the next one.ProcessPoolManagergainsslot(), an async context manager that yields aPoolSlotwhoserun()executes on the claimed child.run()is nowslot()plusslot.run(), so other workers are unchanged. merge_message claims a slot first and then reads and sizes its batch (_next_batch). Callbacks that arrive while a task waits for a CPU join that pass. Batching now comes from time the worker spends busy, never from time it sits idle.TASKS_PER_POOL_CHILD), so 4 on a 2-CPU pod. That keeps each child's next pass queued and lets lock losers ack without waiting behind merges. It also stays small, so that once this worker scales out, one busy pod doesn't hold backlog another idle pod could take. TheTASK_LIMITenv var still overrides it.Deploy notes
MERGE_DEBOUNCE_SECis no longer read; remove it if it's set.TASK_LIMITunless you mean to override the 2× default.Tests
test_merge_dispatch_limit.pychecks the task limit is 2× the pool size.test_merge_passes.pychecks that a pass starts immediately, that a callback arriving during a pass joins the next pass, and that callbacks arriving while a task waits for a child join that task's pass.test_process_pool.pychecks that a slot stays claimed for its block and is released when the block raises.slot()method.lmdb,biothings_annotatorandbiolink_helper_pkg.🤖 Generated with Claude Code
https://claude.ai/code/session_01S2yuRAbvfoHwTUCX9uNik6