Repository navigation
Fleet: leased+cancel-requested job with dead owner never resolves; stop grace too short for guest teardown #165
Description
Found live during #39 GPU stop testing (2026-10-14, queue rsi-train-20261013, job 6684468d).
Repro: POST {action:stop} to a worker's emergency-control socket while it holds a training lease. The rsi-job cleanup waits for terminal job status within the 5s grace period, but OpenShell guest teardown (sandbox delete over the gateway) takes longer. Cleanup reports cleanup exceeded 5000ms grace period, control enters failed-to-stop (409), the worker exits with RSI worker is exiting with unconfirmed guest teardown.
Wedge: the job row stays leased with cancel_requested_at set and a dead owner forever:
acknowledgeCancellationrequires the dead worker's lease token;- the expiry reaper (
postgres-queue.ts~L475) only reaps leased jobs withcancel_requested_at IS NULL; - capacity counting keeps the slot reserved indefinitely.
The guest itself was torn down (sandbox gone from the gateway) but teardown was never verified, so nothing can confirm it.
Expected: either the stop grace accommodates real teardown time (or teardown continues asynchronously with later verification), or the supervisor can resolve a leased+cancel-requested job whose owner is dead/expired (reap to cancelled with an owner-lost note, releasing capacity). At minimum the coordinator should not wait out the full lease timeout with no owner progress.
Evidence: job 6684468d-ffe6-498f-9302-f29884486deb left wedged on the dev database for inspection; worker log at time of exit; control record showed failed-to-stop with per-cleanup results (failed teardown WAS reported honestly — that part works).
Activity
w4ffl35 commentedon Oct 14, 2026
Second repro, same wedge class (2026-10-14): restarted a private OpenShell gateway (
hcls-rsi-39test) mid-training-lease. Guest died with the gateway; workerrsi-39-gpu-1exited withRSI worker is exiting with unconfirmed guest teardown; job6937e5b4is nowleased+ cancel-requested with a dead owner — equally unresolvable (reaper skips cancel-requested; ack needs the dead lease token). GPU returned to idle (guest compute stopped with the gateway). Two wedged rows now occupy both max_in_flight slots of dev queuersi-train-20261013, leaving that queue unable to schedule — the wedge also degrades fleet capacity, not just the one job. Left in place as evidence; further test runs will use a fresh queue.w4ffl35 commentedon Oct 14, 2026
Fix in PR #166, live-verified: job 6684468d (stop-test wedge) was reaped to cancelled by the fixed claim path with error "supervisor emergency stop (lease expired before teardown was acknowledged)" — only the lease clock was advanced manually; the state transition went through the new code. Job 6937e5b4 left untouched for natural expiry at 03:18 UTC as a control.
w4ffl35 commentedon Oct 14, 2026
Control confirmation: job 6937e5b4 (gateway-restart wedge, left untouched) resolved itself to cancelled at natural lease expiry 03:18:28 UTC — reaped within ~20s by the fixed claim path with zero manual intervention. Error preserves the original cancel reason plus the expiry note. The wedge class is closed live as well as in tests.