[roodboi]

Problem

When something outside Hack changes the prepared SmolVM provider tree (the real-world trigger was a cleanup tool that pruned every empty directory, such as agent-rootfs/dev, proc, tmp, etc/init.d), every runtime command failed with artifact_digest_mismatch, including runtime down on an already stopped runtime. The candidate installer's quiescence check (status, down, status) then failed and upgrade was blocked, and there was no supported way to re-prepare the tree from the archive preparation had already kept.

Final behavior

• runtime down on a stopped pool runs nothing from the provider tree, so it no longer verifies it. "Stopped" is the existing condition: a phase written only before a start or after proven provider absence (stopped, stopped-before-engine, stopped-after-engine-failure, recovered-unclean) and no live provider. For any other pool, down still verifies the tree before the hostname-authority and publication steps, exactly as before. runtime status already read no tree; a regression test now covers it.
• Execution is still refused. up and every other path that runs the provider still call verify. Its fixed message now names the repair: "With the runtime stopped, run runtime repair-provider-tree to re-prepare it from the verified retained archive; runtime state and volumes are kept." The Docker engine tree gets the same message.
• New runtime repair-provider-tree [--json]. It holds the provider lock and requires a stopped (or uninitialized) pool with no active image build. For each prepared tree (SmolVM, and the Docker engine, which has the same shape):
  • the preparation record must match this executor's pins;
  • an intact tree is reported as repaired: false and left alone;
  • the kept archive (providers/smolvm-installing/release.tar.gz, providers/docker-installing/release.tgz) must be an owned regular file, with one link, in an owned directory, with the pinned SHA-256;
  • it is copied into a private -repairing- staging directory and hashed again, extracted, signature-checked (SmolVM, as preparation does), and the fresh tree must hash to the digest recorded at preparation;
  • only then is the changed tree renamed aside as <tree>-changed-<id> (kept, never deleted) and the fresh tree renamed into place, then verified again.
    Runtime state, disks and volumes under run/ are not touched. A repair that fails a check changes no tree and keeps its staging. Without a kept archive it refuses with provider_repair_unavailable.
• If nothing was prepared, the command creates no state.

Validation

• artifact::tests::a_tree_pruned_of_empty_directories_is_repaired_from_its_retained_archive: a pinned fixture archive with nested empty directories, a symlink and an executable is prepared the same way preparation lays out state. Pruning its empty directories makes verify refuse. The repair restores the recorded digest, keeps the changed tree and the kept archive, leaves no staging, and an intact tree is not replaced (same inode).
• artifact::tests::repair_refuses_without_a_verified_archive_or_a_matching_extraction: covers an unprepared tree, a wrong archive digest, a missing archive, a symlinked archive, a record no extraction can match (the tree is unchanged and the staging is kept), and a record for another version.
• lifecycle::provider_repair_tests (macOS): on a stopped pool with a pruned SmolVM tree under the real pins, status, down and status all succeed (the installer's sequence) while verify still refuses and names the repair. The repair then fixes the fixture tree, and the owner, machine and phase are unchanged. On a pool that is not proven stopped, down still refuses with artifact_digest_mismatch and the repair refuses with provider_repair_requires_stopped_runtime. A home with nothing prepared creates no state.
• Gates, at head (base origin/next e19e0b1):
  • Focused Rust: 12 pass, 0 fail (artifact::tests, provider_repair_tests).
  • RUSTUP_TOOLCHAIN=1.97.1 bun run check:local (cargo fmt --check plus clippy --all-targets -D warnings): pass.
  • RUSTUP_TOOLCHAIN=1.97.1 bun run test:local: 1559 passed, 0 failed, 115 ignored across 23 test binaries. This ran from a short-path worktree because node_contract needs socket paths under SUN_LEN. From the long scratch path, only node_contract failed, with the known SUN_LEN error.
  • There are no TypeScript changes. bun run --cwd packages/cli typecheck shows only the existing tests/fixtures/atomic-write-foreign-owner-worker.ts(39,25) error, which is identical on origin/next.
  • gitleaks protect --staged and gitleaks detect --log-opts=origin/next..HEAD: no leaks.

Remaining gaps

• Not yet verified with an installed candidate: repair, then the installer's upgrade. That acceptance item stays open.
• The installer runs the old version's executor for its quiescence check. A home stranded by a version that predates this change (for example the next.10 candidate) still needs this command from an executor with the same SmolVM pin, or a fresh candidate home. This change prevents the block for later upgrades.
• A pool whose provider is still live with a changed tree can neither be stopped (down must verify before it runs the provider) nor repaired (repair requires a stopped pool). That case is unchanged.
• The single-file BuildKit client is not covered: its preparation does not keep a source archive.
• The changed tree that is moved aside is kept for inspection and is not cleaned up automatically.

Refs HACK-1950

🤖 Generated with Claude Code

---

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

[linear-code]

HACK-1950

[roodboi]

@codex review

[chatgpt-codex-connector]

Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-07T16:59:24.926439Z	7ddc193	Manual request
🔒 Security Review	✅ Completed 2026-11-07T17:01:37.520055Z	7ddc193	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 7ddc193de7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 7ddc193de7

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.