Repository navigation
devkit's CLI exits by draining the event loop, so Node 24's process.exit deadlock no longer hangs a finished command (sc-8775) - #282
Merged
Conversation
…xit deadlock no longer hangs a finished command (sc-8775) ## Why `prove-regression.test.mts` cases sometimes failed with "testSpawnSync: 90s harness deadline fired". The story put it down to missing headroom, but each case takes 1–7s. The deadline was catching a real hang. I captured a stalled case under load with a `ps` snapshot and `/usr/bin/sample` of the CLI. The CLI had printed its result and reached its `process.exit` exit event 1.2s in. It then hung for 88s inside Node itself: - main thread: `Environment::Exit` → `DisposePlatform` → `WorkerThreadsTaskRunner::Shutdown` → `pthread_join` - V8Worker: concurrent Sparkplug `BaselineCompiler::Build` → `CollectionBarrier::AwaitCollectionBackground`, waiting for a GC that the exiting main thread never runs This is a known Node 24 bug, [nodejs/node#64274](nodejs/node#64274). The fix is still pending upstream in nodejs/node#66171, and the repro is nodejs/node#67596. devkit's dispatcher ended every command with `process.exit`, so any `devkit <cmd>` could hang the same way, inside a consumer hook or a ship as well as in our own suite. ## What - `exitWhenDrained(code, graceMs = 1000)` in `gate-engine/deterministic/reason.mts` sets `process.exitCode` and lets the event loop drain. A normal exit disposes the isolate before the platform, so it never deadlocks. An unref'd timer forces `process.exit` only when a handle is still holding the loop, by which point the loop has serviced any pending GC request. - `cli/index.mts` ends both the returned path and the thrown path with it. `stdoutDrained` still runs first, to bound a slow pipe reader on the forced path. - New decision Target `cli-exit-drains-event-loop`. Its Ruling states the one carve-out: a throw during a gate run (`GATE_REASON_ENV`) keeps `crashExit`'s immediate exit, which other gates' crash preload shares. - `cli/lib/ship/run-packaged-script.mts`: a comment that said the CLI calls `process.exit()` now says `exitWhenDrained`. The behaviour it describes is unchanged: the signal handlers stay registered through the drain. The 90s deadline and the vitest project placement are unchanged. `test-deadlines-are-hang-detectors` holds, and this deadline did its job. ## Proof | Arm | `process.exit` (before) | drain (after) | |---|---|---| | prove-regression.test.mts, 20 file runs (5 rounds × 4 concurrent), ~560 CLI calls | 1 stall (90.4s, 124), 1 file failed | 0 stalls, 20/20 passed | | nodejs/node#67596 repro, 8-way parallel, Node v24.19.0 | 2/400 hung | 0/400 hung | - File duration is unchanged, 42–51s per run in the after arm. Spot checks of `base-status`, `guard-branch`, `doctor` and `ci-verdict` never reached the 1s forced path. - Revert check: with `process.exit` restored, the 3 new tests fail. Two `cli-exit.test.mts` cases fail on "process.exit called". The `reason.test.mts` case fails with stdout cut at 65536 of 5000000 bytes. All of them pass with the change. - `ship --dry-gates` passed, including related-tests (155 files, 4092 tests). ## Follow-ups (not in this PR) - The `process.exit` calls before dispatch in `cli/index.mts` (`--version`, `help`, unknown command, `<cmd> --help`): bug, sev 3, traced. - `crashExit` still calls `process.exit` on the gate crash path: bug, sev 3, traced. - About 220 other `process.exit(` calls in 79 gate and command files (`exitGate`, guard-* CLIs): bug, sev 2, claimed. - sc-6032 (existing): have gate-supervisor report its own deadline instead of inferring it. - Check every command for open handles that pay the 1s fallback: chore. - Async work or a late promise rejection running during the drain window: bug, sev 3, claimed. - `dist/` is rebuilt at the next `devkit release`: chore. Fixes [sc-8775] 🤖 Generated with [Claude Code](https://claude.com/claude-code) Complete against acceptance criteria: yes
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 14 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
…xit deadlock no longer hangs a finished command (sc-8775) The forced exit uses the current process.exitCode, as a natural drain does (sc-8775) exitWhenDrained's 1s fallback now calls process.exit() with no argument, so a process.exitCode written after the command returned wins on both the natural-drain and the forced path. A reason.test.mts case holds a handle open, writes exitCode 5 after exitWhenDrained(0), and expects status 5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
prove-regression.test.mtscases sometimes failed with "testSpawnSync: 90s harness deadline fired". The story put it down to missing headroom, but each case takes 1–7s. The deadline was catching a real hang.I captured a stalled case under load with a
pssnapshot and/usr/bin/sampleof the CLI. The CLI had printed its result and reached itsprocess.exitexit event 1.2s in. It then hung for 88s inside Node itself:Environment::Exit→DisposePlatform→WorkerThreadsTaskRunner::Shutdown→pthread_joinBaselineCompiler::Build→CollectionBarrier::AwaitCollectionBackground, waiting for a GC that the exiting main thread never runsThis is a known Node 24 bug, nodejs/node#64274. The fix is still pending upstream in nodejs/node#66171, and the repro is nodejs/node#67596. devkit's dispatcher ended every command with
process.exit, so anydevkit <cmd>could hang the same way, inside a consumer hook or a ship as well as in our own suite.What
exitWhenDrained(code, graceMs = 1000)ingate-engine/deterministic/reason.mtssetsprocess.exitCodeand lets the event loop drain. A normal exit disposes the isolate before the platform, so it never deadlocks. An unref'd timer forcesprocess.exitonly when a handle is still holding the loop, by which point the loop has serviced any pending GC request.cli/index.mtsends both the returned path and the thrown path with it.stdoutDrainedstill runs first, to bound a slow pipe reader on the forced path.cli-exit-drains-event-loop. Its Ruling states the one carve-out: a throw during a gate run (GATE_REASON_ENV) keepscrashExit's immediate exit, which other gates' crash preload shares.process.exit()with no argument, so it ends with the currentprocess.exitCodejust as a natural drain does. A lateexitCodewrite wins on both paths (correctness-reviewer advisory, with a test).cli/lib/ship/run-packaged-script.mts: a comment that said the CLI callsprocess.exit()now saysexitWhenDrained. The behaviour it describes is unchanged: the signal handlers stay registered through the drain.The 90s deadline and the vitest project placement are unchanged.
test-deadlines-are-hang-detectorsholds, and this deadline did its job.Proof
process.exit(before)base-status,guard-branch,doctorandci-verdictnever reached the 1s forced path.process.exitrestored, the 3 new tests fail. Twocli-exit.test.mtscases fail on "process.exit called". Thereason.test.mtscase fails with stdout cut at 65536 of 5000000 bytes. All of them pass with the change.ship --dry-gatespassed, including related-tests (155 files, 4092 tests).Follow-ups (not in this PR)
process.exitcalls before dispatch incli/index.mts(--version,help, unknown command,<cmd> --help): bug, sev 3, traced (sc-8955).crashExitstill callsprocess.exiton the gate crash path: bug, sev 3, traced. Folded into sc-8956.process.exit(calls in 79 gate and command files (exitGate, guard-* CLIs): bug, sev 2, traced (sc-8956).Not filed
cli-exit-drains-event-loopNegative, and the fallback caps it at 1s.dist/is rebuilt at the nextdevkit release. This is the intended state: tracked dist lags its source until release.Reviewer notes declined:
cli-exit.test.mtsnot inGIT_INTEGRATION_TESTS: that list is for files that spawn real repositories or worktrees. This file spawns only the CLI's usage and missing-git paths, the same shape ashelp-cli.test.mts, which is also not listed.exitWhenDrainedreturns normally: both call sites are the last statement of their block (end ofmain()and themain().catchhandler), and itsvoidreturn type, unlikenever, tells a caller it returns.Fixes [sc-8775]
🤖 Generated with Claude Code
Complete against acceptance criteria: yes
Automated evidence
Merge verdict: 🟡 human-review · advisory, at
720d019bb42e✅ Regression evidence · captured: 5 of 5 red-failing tests passed on green (setup 12s, red 4s, green 3s, cleanup 3s)
Head
720d019bb42e· basee7bfe421f0fbFailed on red (5 tests)
devkit exit ends a returning command with its code without calling process.exit:AssertionError: expected '🚫 prove-regression usage: separate t…' not to contain 'process.exit called'devkit exit ends a command that throws with 1 without calling process.exit:AssertionError: expected 'devkit: git is not installed or not o…' not to contain 'process.exit called'exitWhenDrained ends with its code once every queued stdout byte reached the pipe:AssertionError: expected 1 to be 3 // Object.is equalityexitWhenDrained forces the exit with its code when a handle keeps the event loop alive:AssertionError: expected 1 to be 4 // Object.is equalityexitWhenDrained honours a later exitCode write on the forced path, as a natural drain does:AssertionError: expected 1 to be 5 // Object.is equalityTest command
Captured runs, not proof of causality. Dependencies are the shipping checkout's installed packages. Stale unless the PR head is
720d019bb42e377891f3805a8111abcf5cc9b7fd.💬 Reviewer notes · 3 notes at
7e002128d509cli/__tests__/cli-exit.test.mtstestSpawnSync, but the file is not listed inGIT_INTEGRATION_TESTSinvitest.config.mjs. I did not check whether it needs to be. · commit-guardgate-engine/deterministic/reason.mtscode, while a natural drain honors any laterprocess.exitCodewrite. A dispatched command that returns 0 and has a ref'd callback settingprocess.exitCode = 1after 1s ends with status 0 instead of 1, depending on the timer race. I did not verify that any current command does this. · correctnessexitWhenDrainedreturns normally, so code after the call inmain()still runs. A caller that does not return immediately would keep executing instead of stopping, asprocess.exitdid. · api-security