Skip to content

devkit's CLI exits by draining the event loop, so Node 24's process.exit deadlock no longer hangs a finished command (sc-8775) - #282

Merged
norvalbv merged 2 commits into
mainfrom
sc-8775-prove-regression-post-output-hang
Nov 20, 2026
Merged

norvalbv merged 2 commits into
mainfrom
sc-8775-prove-regression-post-output-hang

Conversation

@norvalbv

@norvalbv norvalbv commented Nov 20, 2026 •

Copy link
Copy Markdown
Owner

Why

prove-regression.test.mts cases sometimes failed with "testSpawnSync: 90s harness deadline fired". The story put it down to missing headroom, but each case takes 1–7s. The deadline was catching a real hang.

I captured a stalled case under load with a ps snapshot and /usr/bin/sample of the CLI. The CLI had printed its result and reached its process.exit exit event 1.2s in. It then hung for 88s inside Node itself:

  • main thread: Environment::Exit → DisposePlatform → WorkerThreadsTaskRunner::Shutdown → pthread_join
  • V8Worker: concurrent Sparkplug BaselineCompiler::Build → CollectionBarrier::AwaitCollectionBackground, waiting for a GC that the exiting main thread never runs

This is a known Node 24 bug, nodejs/node#64274. The fix is still pending upstream in nodejs/node#66171, and the repro is nodejs/node#67596. devkit's dispatcher ended every command with process.exit, so any devkit <cmd> could hang the same way, inside a consumer hook or a ship as well as in our own suite.

What

  • exitWhenDrained(code, graceMs = 1000) in gate-engine/deterministic/reason.mts sets process.exitCode and lets the event loop drain. A normal exit disposes the isolate before the platform, so it never deadlocks. An unref'd timer forces process.exit only when a handle is still holding the loop, by which point the loop has serviced any pending GC request.
  • cli/index.mts ends both the returned path and the thrown path with it. stdoutDrained still runs first, to bound a slow pipe reader on the forced path.
  • New decision Target cli-exit-drains-event-loop. Its Ruling states the one carve-out: a throw during a gate run (GATE_REASON_ENV) keeps crashExit's immediate exit, which other gates' crash preload shares.
  • The forced fallback calls process.exit() with no argument, so it ends with the current process.exitCode just as a natural drain does. A late exitCode write wins on both paths (correctness-reviewer advisory, with a test).
  • cli/lib/ship/run-packaged-script.mts: a comment that said the CLI calls process.exit() now says exitWhenDrained. The behaviour it describes is unchanged: the signal handlers stay registered through the drain.

The 90s deadline and the vitest project placement are unchanged. test-deadlines-are-hang-detectors holds, and this deadline did its job.

Proof

Arm process.exit (before) drain (after)
prove-regression.test.mts, 20 file runs (5 rounds × 4 concurrent), ~560 CLI calls 1 stall (90.4s, 124), 1 file failed 0 stalls, 20/20 passed
nodejs/node#67596 repro, 8-way parallel, Node v24.19.0 2/400 hung 0/400 hung
  • File duration is unchanged, 42–51s per run in the after arm. Spot checks of base-status, guard-branch, doctor and ci-verdict never reached the 1s forced path.
  • Revert check: with process.exit restored, the 3 new tests fail. Two cli-exit.test.mts cases fail on "process.exit called". The reason.test.mts case fails with stdout cut at 65536 of 5000000 bytes. All of them pass with the change.
  • ship --dry-gates passed, including related-tests (155 files, 4092 tests).

Follow-ups (not in this PR)

  • The process.exit calls before dispatch in cli/index.mts (--version, help, unknown command, <cmd> --help): bug, sev 3, traced (sc-8955).
  • crashExit still calls process.exit on the gate crash path: bug, sev 3, traced. Folded into sc-8956.
  • About 220 other process.exit( calls in 79 gate and command files (exitGate, guard-* CLIs): bug, sev 2, traced (sc-8956).
  • sc-6032 (existing): have gate-supervisor report its own deadline instead of inferring it.
  • A harness deadline captures no snapshot of the hung child (autonomous becb7c5d-1a4c-4e76-8d0e-4bc9f8926b16).

Not filed

  • Check every command for open handles that pay the 1s fallback. This cost is deliberate and recorded in the cli-exit-drains-event-loop Negative, and the fallback caps it at 1s.
  • Async work or a late promise rejection running during the drain window. Unverified: no command doing this could be traced.
  • dist/ is rebuilt at the next devkit release. This is the intended state: tracked dist lags its source until release.

Reviewer notes declined:

  • commit-guard, cli-exit.test.mts not in GIT_INTEGRATION_TESTS: that list is for files that spawn real repositories or worktrees. This file spawns only the CLI's usage and missing-git paths, the same shape as help-cli.test.mts, which is also not listed.
  • api-security-reviewer, exitWhenDrained returns normally: both call sites are the last statement of their block (end of main() and the main().catch handler), and its void return type, unlike never, tells a caller it returns.

Fixes [sc-8775]

🤖 Generated with Claude Code

Complete against acceptance criteria: yes

Automated evidence

Merge verdict: 🟡 human-review · advisory, at 720d019bb42e

  • decision-record
✅ Regression evidence · captured: 5 of 5 red-failing tests passed on green (setup 12s, red 4s, green 3s, cleanup 3s)

Head 720d019bb42e · base e7bfe421f0fb

run what ran exit passed failed skipped
red PR test files on the PR base 1 39 5 0
green PR head 0 44 0 0
Failed on red (5 tests)
  • devkit exit ends a returning command with its code without calling process.exit: AssertionError: expected '🚫 prove-regression usage: separate t…' not to contain 'process.exit called'
  • devkit exit ends a command that throws with 1 without calling process.exit: AssertionError: expected 'devkit: git is not installed or not o…' not to contain 'process.exit called'
  • exitWhenDrained ends with its code once every queued stdout byte reached the pipe: AssertionError: expected 1 to be 3 // Object.is equality
  • exitWhenDrained forces the exit with its code when a handle keeps the event loop alive: AssertionError: expected 1 to be 4 // Object.is equality
  • exitWhenDrained honours a later exitCode write on the forced path, as a natural drain does: AssertionError: expected 1 to be 5 // Object.is equality
Test command
[
  "node_modules/.bin/vitest",
  "run",
  "--reporter=json",
  "--outputFile.json=.devkit-evidence-report.json",
  "cli/__tests__/cli-exit.test.mts",
  "gate-engine/deterministic/__tests__/reason.test.mts"
]

Captured runs, not proof of causality. Dependencies are the shipping checkout's installed packages. Stale unless the PR head is 720d019bb42e377891f3805a8111abcf5cc9b7fd.

💬 Reviewer notes · 3 notes at 7e002128d509

cli/__tests__/cli-exit.test.mts

  • L25 — This test spawns the devkit CLI through testSpawnSync, but the file is not listed in GIT_INTEGRATION_TESTS in vitest.config.mjs. I did not check whether it needs to be. · commit-guard

gate-engine/deterministic/reason.mts

  • L107 — The forced path exits with the argument code, while a natural drain honors any later process.exitCode write. A dispatched command that returns 0 and has a ref'd callback setting process.exitCode = 1 after 1s ends with status 0 instead of 1, depending on the timer race. I did not verify that any current command does this. · correctness
  • L108 — exitWhenDrained returns normally, so code after the call in main() still runs. A caller that does not return immediately would keep executing instead of stopping, as process.exit did. · api-security

…xit deadlock no longer hangs a finished command (sc-8775)

## Why

`prove-regression.test.mts` cases sometimes failed with "testSpawnSync: 90s harness deadline fired". The story put it down to missing headroom, but each case takes 1–7s. The deadline was catching a real hang.

I captured a stalled case under load with a `ps` snapshot and `/usr/bin/sample` of the CLI. The CLI had printed its result and reached its `process.exit` exit event 1.2s in. It then hung for 88s inside Node itself:

- main thread: `Environment::Exit` → `DisposePlatform` → `WorkerThreadsTaskRunner::Shutdown` → `pthread_join`
- V8Worker: concurrent Sparkplug `BaselineCompiler::Build` → `CollectionBarrier::AwaitCollectionBackground`, waiting for a GC that the exiting main thread never runs

This is a known Node 24 bug, [nodejs/node#64274](nodejs/node#64274). The fix is still pending upstream in nodejs/node#66171, and the repro is nodejs/node#67596. devkit's dispatcher ended every command with `process.exit`, so any `devkit <cmd>` could hang the same way, inside a consumer hook or a ship as well as in our own suite.

## What

- `exitWhenDrained(code, graceMs = 1000)` in `gate-engine/deterministic/reason.mts` sets `process.exitCode` and lets the event loop drain. A normal exit disposes the isolate before the platform, so it never deadlocks. An unref'd timer forces `process.exit` only when a handle is still holding the loop, by which point the loop has serviced any pending GC request.
- `cli/index.mts` ends both the returned path and the thrown path with it. `stdoutDrained` still runs first, to bound a slow pipe reader on the forced path.
- New decision Target `cli-exit-drains-event-loop`. Its Ruling states the one carve-out: a throw during a gate run (`GATE_REASON_ENV`) keeps `crashExit`'s immediate exit, which other gates' crash preload shares.
- `cli/lib/ship/run-packaged-script.mts`: a comment that said the CLI calls `process.exit()` now says `exitWhenDrained`. The behaviour it describes is unchanged: the signal handlers stay registered through the drain.

The 90s deadline and the vitest project placement are unchanged. `test-deadlines-are-hang-detectors` holds, and this deadline did its job.

## Proof

| Arm | `process.exit` (before) | drain (after) |
|---|---|---|
| prove-regression.test.mts, 20 file runs (5 rounds × 4 concurrent), ~560 CLI calls | 1 stall (90.4s, 124), 1 file failed | 0 stalls, 20/20 passed |
| nodejs/node#67596 repro, 8-way parallel, Node v24.19.0 | 2/400 hung | 0/400 hung |

- File duration is unchanged, 42–51s per run in the after arm. Spot checks of `base-status`, `guard-branch`, `doctor` and `ci-verdict` never reached the 1s forced path.
- Revert check: with `process.exit` restored, the 3 new tests fail. Two `cli-exit.test.mts` cases fail on "process.exit called". The `reason.test.mts` case fails with stdout cut at 65536 of 5000000 bytes. All of them pass with the change.
- `ship --dry-gates` passed, including related-tests (155 files, 4092 tests).

## Follow-ups (not in this PR)

- The `process.exit` calls before dispatch in `cli/index.mts` (`--version`, `help`, unknown command, `<cmd> --help`): bug, sev 3, traced.
- `crashExit` still calls `process.exit` on the gate crash path: bug, sev 3, traced.
- About 220 other `process.exit(` calls in 79 gate and command files (`exitGate`, guard-* CLIs): bug, sev 2, claimed.
- sc-6032 (existing): have gate-supervisor report its own deadline instead of inferring it.
- Check every command for open handles that pay the 1s fallback: chore.
- Async work or a late promise rejection running during the drain window: bug, sev 3, claimed.
- `dist/` is rebuilt at the next `devkit release`: chore.

Fixes [sc-8775]

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Complete against acceptance criteria: yes
@coderabbitai

coderabbitai Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 14 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 5187956b-3d59-4872-b96a-53b317f8b504
Commits

Reviewing files that changed from the base of the PR and between e7bfe42 and 720d019.

Files selected for processing: 7

  • cli/__tests__/cli-exit.test.mts
  • cli/index.mts
  • cli/lib/ship/run-packaged-script.mts
  • docs/decisions/INDEX.md
  • docs/decisions/cli-exit-drains-event-loop.md
  • gate-engine/deterministic/__tests__/reason.test.mts
  • gate-engine/deterministic/reason.mts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

…xit deadlock no longer hangs a finished command (sc-8775)

The forced exit uses the current process.exitCode, as a natural drain does (sc-8775)

exitWhenDrained's 1s fallback now calls process.exit() with no argument, so a process.exitCode written after the command returned wins on both the natural-drain and the forced path. A reason.test.mts case holds a handle open, writes exitCode 5 after exitWhenDrained(0), and expects status 5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@norvalbv
norvalbv merged commit 1701252 into main Nov 20, 2026
8 checks passed
@norvalbv
norvalbv deleted the sc-8775-prove-regression-post-output-hang branch November 20, 2026 08:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant