bug(cli): forward start reports 'ssh exited before local forward listener opened' while the listener is live — and leaks an orphaned ssh mux holding the port

[dlynch90]

Environment

OpenShell 0.1.1 (Homebrew tap), macOS 15 Apple Silicon, local Docker driver, gateway via brew services.

Reproduction

openshell sandbox create --name web --from python:3.13-alpine -- python3 -m http.server 8080 --bind 0.0.0.0
openshell forward start 18080 web
# × ssh exited before local forward listener opened on 127.0.0.1:18080

Observed

1. The CLI exits with the readiness error above.
2. Despite the error, the listener is live: lsof -nP -iTCP:18080 -sTCP:LISTEN shows an ssh process (started with ~/.ssh/control/<hash> [mux]), PPID 1, holding the port.
3. openshell forward list shows "No active forwards" — the leaked mux is invisible to the registry, so forward stop cannot reclaim it; only a manual kill <pid> cleans up. Retrying forward start on the same port then fails with the (accurate) "Port 18080 is already in use" diagnostic naming the orphan.

Expected

• The readiness race should either be resolved (wait for the listener before reporting failure) or the tracked forward should be torn down before exiting with the error.
• On failure, no orphaned ssh mux should survive detached at PPID 1; if one does, it should remain in the forward registry so forward stop (or the next forward start with a prompt) can reclaim it.

Related but distinct: #2999 (SSH-session cleanup ordering), #3051 (structured errors).

[kvnloo]

I think the key invariant here should be stronger than fixing the readiness timing:

forward start returning an error must leave no listener, child process, mux, or registry entry behind

and:

forward start returning success must leave exactly one tracked forward

That gives us a clean lifecycle contract regardless of why readiness fails.

For the regression I would force the race rather than rely on normal SSH timing:

1. launch the SSH child;
2. make listener detection miss/delay once;
3. take the failure path;
4. assert the child/mux is terminated and reaped;
5. assert the port can immediately be rebound;
6. assert forward list has no entry.

Then separately test a slow-but-successful listener startup and make sure we don't kill a forward that became ready just before the deadline.

That should prevent both false failure and invisible orphan variants.

[drew]

🏗️ build-plan

Implementation Plan

Issue type: fix
Complexity: Medium
Confidence: High — reproduced against a local gateway with real OpenSSH

Summary

Make CLI-owned SSH forwards ignore user multiplexing and automatic backgrounding settings, so the spawned PID remains the listener owner. Reap the child on startup/tracking failures and ensure a background forward is still alive before writing its PID record.

Scope

• crates/openshell-cli/src/ssh.rs: add forward-only SSH options, tighten startup ownership checks, and await failed-child cleanup.
• crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs: cover the affected forward startup and cleanup paths.
• architecture/sandbox.md, docs/how-it-works/sandboxes/overview.mdx, and skills/openshell-cli/SKILL.md: document the forward lifecycle and SSH config boundary.

Implementation Steps

1. Build forward SSH commands with explicit multiplexing and forking overrides.
2. Ensure startup errors and PID-record failures terminate and reap the owned SSH process; avoid tracking an exited process.
3. Add regression coverage, then verify with the same real SSH configuration used to reproduce bug(cli): forward start reports 'ssh exited before local forward listener opened' while the listener is live — and leaks an orphaned ssh mux holding the port #3738.

Test Plan

• Unit tests: verify forward-only SSH options and child cleanup/ownership decisions.
• Integration tests: exercise successful background tracking and startup failure cleanup using the CLI test gateway.
• E2E tests: manually rerun the reproduced local Podman gateway workflow with real OpenSSH; no e2e/ fixture changes planned.

Risks & Open Questions

• A user SSH config can alter other SSH behaviors; this fix targets process ownership for port forwarding without changing normal connect/exec/editor behavior. A port probe alone does not prove socket ownership, so the spawned child's liveness must be checked before tracking.
• Foreground forwards are intentionally not in the local registry. The comment's “exactly one tracked forward” success contract applies to background mode.

Documentation Impact

Update the existing architecture overview, sandbox port-forwarding guide, and CLI skill. No gateway TOML or driver configuration changes.

---

Revision 1 — initial plan

[drew]

🏗️ build-from-issue-agent

Implemented the SSH forward ownership fix in #3759. CLI forwards now override user SSH multiplexing and automatic forking settings, clean up failed child processes, and only accept a listener while the spawned SSH process is still running. A regression test checks the effective OpenSSH configuration. Linux gateway smoke testing confirmed foreground and background forwards stay live without leaving a mux socket.

mise run pre-commit, mise run test, and the CLI forward integration tests passed. The full mise run ci was attempted, but its Go gateway tests see this host's existing /etc/openshell/gateways/default and fail their isolated fixture expectations; mise run go:lint passed after clearing a stale cache.