Skip to content

Fix fleet queue wedge: reap expired cancel-requested leases - #166

Merged
w4ffl35 merged 1 commit into
masterfrom
fix/fleet-cancel-reaper
Oct 14, 2026
Merged

w4ffl35 merged 1 commit into
masterfrom
fix/fleet-cancel-reaper

Conversation

@w4ffl35

@w4ffl35 w4ffl35 commented Oct 14, 2026

Copy link
Copy Markdown
Member

Closes #165.

Root cause: cancel-requested leases were excluded from the expiry reaper in claim() but fenced out of complete()/fail()/heartbeat(), so a dead owner left the job leased+cancel-requested forever, holding capacity.

Fix: claim() now resolves cancel-requested leases to cancelled once the lease expires (error preserves the cancel reason plus an expiry note). Valid cancel-requested leases still hold capacity; acknowledgeCancellation stays the fast path for live owners.

Evidence:

  • New PG integration test (failed pre-fix, 18/18 pass post-fix)
  • Live on rsi-train-20261013: job 6684468d reaped to cancelled with provenance note; second wedged job 6937e5b4 left for natural expiry as a control
  • tsc clean, check-standards 0 new, check-architecture 0 new

claim() now resolves cancel-requested leases whose owners never
acknowledged teardown to cancelled once the lease expires, freeing
capacity. Valid cancel-requested leases still hold capacity.

Regression test: expired cancel-requested lease resolves without its
owner; live-verified on rsi-train-20261013 (job 6684468d).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fleet: leased+cancel-requested job with dead owner never resolves; stop grace too short for guest teardown

1 participant