Repository navigation
Conversation
A `hack up` or `hack restart` could fail with `provider_busy` while `hack ps` or `hack logs` polled, because the reads hold the exclusive provider lease briefly and several mutating steps took it without waiting. When the restore of a retained run hit that refusal, its pre-admission cleanup also took the lease without waiting and exited with the foreground publication still bound. From then on `up` refused with `graph_retained_startup`, `down` reported the project stopped, `doctor --native-cleanup inspect` saw nothing pending and `down --recover` was refused. Mutating steps now wait out a brief holder within a bound, and only acquisition is retried: - retained-start eligibility (5 s), as runtime startup already did; - restore admission, within its restore budget and cancellation; - the foreground's pre-admission cleanup and owned cleanup (30 s); - inspections inside up, restart, down and recovery (15 s). ps and logs keep their 2 s budget and still report a busy runtime. A remaining foreground publication now names the next step instead of the generic refusal: a running owner (stop it), an owner that exited while its services ran (`hack down --recover`), or an owner that exited before its restore was admitted (reported; no supported retirement). `hack down --recover` now works for native image projects. It selects the run's exact receipt with the new read-only `graph foreground-owner --run-id RUN --json` and stops an exited ready owner's services through the existing same-boot `recover-live-owner`, retaining data. If that run's frontend also exited, it prints the exact `hack restart --recover-frontend --expect-finalization-attempt` step. Frontend recovery now accepts a project that never recorded lifecycle processes (no lifecycle state file), as the lifecycle runtime does. Refs HACK-1954 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@codex review |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6c672b4e1f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
…ck helper `hack down --recover` refused a retry after an interruption past the runtime's cleanup intent: the receipt was no longer `ready-observed`, so the guard reported an owner that exited before its start was confirmed and stranded the partially stopped graph and its stale publication. `graph foreground-owner` now also returns `live_owner_recovery`, the original receipt digest of a same-boot dead-owner recovery whose intent exists but whose cleanup or publisher retirement is unfinished. `down --recover` resumes that recovery by its own selection, then takes the stopped path (host processes, owed journaled hooks, finalization) without replaying a stop. An exit before restore admission, with no recovery intent, still refuses. `Lock::acquire_existing_waiting` is used only by macOS code and is now gated like its callers, fixing the Linux clippy dead-code error. Refs HACK-1954 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c46d34e0dd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
An interrupted `hack down --recover` could stop after both foreground publisher paths were retired but before `finish_retirement` released the run's dependency reservation. The resume selection then treated the recovery as finished because the publication was absent, so a retry took the already-stopped path and left the reservation behind; the next start of a run with dependency bindings refused because a reservation for the same run already existed. The selection now stays resumable while either retirement effect remains: the stale publication or the run's dependency reservation record. A retry therefore re-enters `recover-live-owner`, whose completed-intent path finishes retirement and releases the reservation. Refs HACK-1954 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@codex review |
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
Summary
A native
hack uporhack restartcould fail withprovider_busywhilehack psorhack logspolled the project. A failed restore could then leave a state that no public command recovered from. This PR fixes both.Root cause (reproduced with a synthetic project)
hack psandhack logsrungraph inspect, which holds the exclusive provider lease for a short time. Several mutating steps took that lease without waiting:graph retained-preflight) failedupbefore anything started.restore_inputs) failed the foreground restore of a retained run.serve_inputthen also took the lease without waiting. Onprovider_busyit returned early, so the process exited with its foreground publication (owner.json,control.sock) still bound.After step 3,
uprefused withgraph_retained_startupbecause the publication was not retired.downreported the project stopped, since the receipt wasstopped-data-retained.doctor --native-cleanup inspectfound nothing pending, anddown --recoverwas refused for native image projects. A trace of the failing process confirmed this exact chain.The repro: a stopped retained run, then
hack upwhile a lock holder keeps the provider lease (standing in for a pollingps). On 5.0.0-next.11 this leaves exactly the stuck state described above.Changes
Wait for the lease with a bound. Only acquisition is retried. Nothing is admitted while a step waits, and when the bound expires the step returns the usual
provider_busyrefusal.Lock::acquire_existing_waiting), the same as runtime startup.Engine::connect_until), the same as graph startup.connect_cleanup_wait.connect_cleanup_wait) now wait up to 30 s instead of 5 s. A foreground owner that gives up on its cleanup strands its publication and services.up,restart,downand recovery retry busy for 15 s.psandlogskeep their 2 s budget and still report a busy runtime.The refusal names the next step. When a foreground publication remains, retained startup now reports one of these:
graph_foreground_owner_running: stop the owner with Ctrl-C orhack down.graph_foreground_owner_exited: the owner exited while its services were running; runhack down --recover.graph_foreground_owner_exited_unadmitted: the owner exited before its restore was admitted. This is reported, but nothing retires it (see Gaps).The classification grants no authority.
hack down --recoverworks for native image projects.graph foreground-owner --run-id RUN --json, returns the receipt phase, the SHA-256 of the exact receipt and the publication state. It takes no lease and contacts no guest.down --recovercalls the existing same-bootgraph recover-live-owner --expect-receipt <sha>. That command revalidates owner death, endpoints and inventories, stops the services with data retained and retires the publication. Down hooks run as for a normaldown.down --recoverwas interrupted after the runtime committed its cleanup intent,graph foreground-owneralso returnslive_owner_recovery: the original receipt digest of that unfinished recovery. A retry resumes the recovery by that digest, then takes the stopped path (host processes, owed journaled hooks, finalization) without replaying a stop.hack uporhack restart) that owned the run also exited,down --recoverprints the exacthack restart --recover-frontend --expect-finalization-attempt <attempt>step and does not wait 150 s for a dead frontend.hack downnow nameshack down --recoverwhen the foreground owner cannot be reached.Validation
Native, on this host. These runs used a candidate built from this branch (debug
hack-nativewith the bundle's features, plus this CLI) and synthetic single-service projects, each in its own private native home:provider_busy, publication left,uprefusesgraph_retained_startupupagain worksup,downanddoctorrefuse; no recoveryupanddownnamehack down --recover;down --recoverstops services and retires the publicationdown --recoverstops services, prints the exactrestart --recover-frontendstep; that step and thenupboth reach readyupanddown --recovername the state; no supported recovery (gap)Rust.
acquire_existing_waiting; preflight waits out a brief holder; each remaining-publication state is classified with no state effect; an interrupted recovery is selected by its original receipt until its publication is retired.RUSTUP_TOOLCHAIN=1.97.1 bun run check:local: clean.bun run test:local: 1380 lib tests passed, 0 failed (110 ignored), and every integration target passed. A clippy run for the Linux target (aarch64-unknown-linux-musl,--all-targets -D warnings) is clean; before the cfg gate onacquire_existing_waiting, the same run reproduced CI's dead-code error.TypeScript.
down --recoverselection and refusals; resuming an interrupted recovery by its original receipt; the exact frontend step when the frontend has also exited (real exited child process);downnames--recover; inspect busy budgets and abort; frontend recovery with absent and unsafe lifecycle state.bun run --cwd packages/cli typecheck: only the pre-existingtests/fixtures/atomic-write-foreign-owner-worker.ts(39,25)error, which also fails onorigin/next.origin/nextworktree in this environment. I reran each failing file there. They are innative-compose-*/native-vm-file-*,git-info-exclude,pre-commit-hook,runtime-turbo-inputs,test-config-compiler,native-ca-doctor-cliandbun-version-guard.docs/reference/cli.mdregenerated.docs/guides/native-candidate.mddocuments the recovery step and the lease waits.Gaps
ensure-imagestill takes the lease without waiting. fix(runtime-core): wait briefly for a held provider lease at native startup #100 left it that way because its refusal may follow an image-cache effect.hack-nextcheck remain for the merge train.Refs HACK-1954
🤖 Generated with Claude Code