Repository navigation
Kill a pool child's whole process group; run compose workers under an init - #262
Merged
Merged
Conversation
catrax-pathfinder runs each BFS leg in a nested ProcessPoolExecutor. When arax.pathfinder's slot timed out (routinely under heavy load), only the slot's child was killed; its two BFS processes were orphaned and kept querying the retriever until their searches ran out, long after the task had been failed and the load had stopped. Each pool child now leads its own process group, and _kill_children SIGKILLs the group after the SIGABRT grace -- also when the child is already dead (OOM), since what it started may not be. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRAdXfZ78XX9HD87VWM
worker.py was PID 1, so processes orphaned in the container were re-parented to it and never reaped -- arax_pathfinder collected <defunct> entries from its orphaned BFS processes. init: true makes Docker's tini PID 1. A test keeps new workers on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRAdXfZ78XX9HD87VWM
Codecov Report❌ Patch coverage is
... and 1 file with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
After a heavy load,
arax_pathfinderkept sending queries to the retriever for a long time even though the monitor showed it working on nothing. Its memory also stayed at ~21.4 GB of its 22 GB limit after every query had finished.catrax-pathfinder's
BidirectionalPathFinderruns each BFS leg in a nested 2-processProcessPoolExecutor. When a task ran out of time (routinely under load),ProcessPoolManager._kill_childrenkilled only the slot's child, so its two BFS processes were orphaned. They:worker.pyis PID 1 in the container and never reaps orphans it didn't start, so the dead ones also piled up as<defunct>entries.Changes
shepherd_utils/process_pool.py: each pool child callsos.setpgrp()in_init_pool_worker, so it leads its own process group and anything it starts joins that group._kill_childrenstill sends the child SIGABRT first (to get the faulthandler stack dump), then after the grace period SIGKILLs the child's whole group. It does this also when the child is already dead (an OOM kill), because what the child started may still be running.compose.yml:init: trueon all 27 worker services, so Docker's tini is PID 1 and reaps orphans. A header comment explains why.test_process_pool.py: a task that starts a grandchild and then times out, or dies abruptly, must take the grandchild with it; and each pool child must lead its own process group. All three tests fail without the fix.test_dockerfiles.py: every compose worker must run under an init.Notes
init: true. The process-group fix stops orphans in production too. To reap zombies there as well, installtiniin the base image and make it the entrypoint (not done here).test_process_pool.pyandtest_dockerfiles.pypass.test_pool_budgets.pypasses apart from 2 tests that couldn't import worker dependencies in the dev sandbox, which is unrelated to this change.Generated by Claude Code