Skip to content

Kill a pool child's whole process group; run compose workers under an init - #262

Merged
maximusunc merged 3 commits into
develfrom
claude/pensive-ramanujan-ta16e6
Nov 13, 2026
Merged

maximusunc merged 3 commits into
develfrom
claude/pensive-ramanujan-ta16e6

Conversation

@maximusunc

Copy link
Copy Markdown
Collaborator

Problem

After a heavy load, arax_pathfinder kept sending queries to the retriever for a long time even though the monitor showed it working on nothing. Its memory also stayed at ~21.4 GB of its 22 GB limit after every query had finished.

catrax-pathfinder's BidirectionalPathFinder runs each BFS leg in a nested 2-process ProcessPoolExecutor. When a task ran out of time (routinely under load), ProcessPoolManager._kill_children killed only the slot's child, so its two BFS processes were orphaned. They:

  • kept querying the retriever until their searches ran out, with no task left to show in the monitor;
  • then never exited. In the observed container, 43 of them were still alive about 50 minutes later, idle with every thread waiting on a lock, holding 21.4 GiB between them. That accounts for all of the memory.

worker.py is PID 1 in the container and never reaps orphans it didn't start, so the dead ones also piled up as <defunct> entries.

Changes

  • shepherd_utils/process_pool.py: each pool child calls os.setpgrp() in _init_pool_worker, so it leads its own process group and anything it starts joins that group. _kill_children still sends the child SIGABRT first (to get the faulthandler stack dump), then after the grace period SIGKILLs the child's whole group. It does this also when the child is already dead (an OOM kill), because what the child started may still be running.
  • compose.yml: init: true on all 27 worker services, so Docker's tini is PID 1 and reaps orphans. A header comment explains why.
  • Tests:
    • test_process_pool.py: a task that starts a grandchild and then times out, or dies abruptly, must take the grandchild with it; and each pool child must lead its own process group. All three tests fail without the fix.
    • test_dockerfiles.py: every compose worker must run under an init.

Notes

  • The Helm chart doesn't get init: true. The process-group fix stops orphans in production too. To reap zombies there as well, install tini in the base image and make it the entrypoint (not done here).
  • Already-orphaned processes are only cleared by restarting the container.
  • test_process_pool.py and test_dockerfiles.py pass. test_pool_budgets.py passes apart from 2 tests that couldn't import worker dependencies in the dev sandbox, which is unrelated to this change.

Generated by Claude Code

catrax-pathfinder runs each BFS leg in a nested ProcessPoolExecutor. When
arax.pathfinder's slot timed out (routinely under heavy load), only the
slot's child was killed; its two BFS processes were orphaned and kept
querying the retriever until their searches ran out, long after the task
had been failed and the load had stopped.

Each pool child now leads its own process group, and _kill_children
SIGKILLs the group after the SIGABRT grace -- also when the child is
already dead (OOM), since what it started may not be.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRAdXfZ78XX9HD87VWM
worker.py was PID 1, so processes orphaned in the container were
re-parented to it and never reaped -- arax_pathfinder collected <defunct>
entries from its orphaned BFS processes. init: true makes Docker's tini
PID 1. A test keeps new workers on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRAdXfZ78XX9HD87VWM
@codecov

codecov Bot commented Nov 13, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 55.55556% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.68%. Comparing base (08952ef) to head (8346e35).
⚠️ Report is 1 commits behind head on devel.

Files with missing lines Patch % Lines
shepherd_utils/process_pool.py 55.55% 7 Missing and 1 partial ⚠️
Files with missing lines Coverage Δ
shepherd_utils/process_pool.py 79.61% <55.55%> (-0.12%) ⬇️

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0235f95...8346e35. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@maximusunc
maximusunc merged commit fa5087e into devel Nov 13, 2026
2 checks passed
@maximusunc
maximusunc deleted the claude/pensive-ramanujan-ta16e6 branch November 13, 2026 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants