Repository navigation
sandbox: keep background jobs supervised after main exit - #299
Closed
congwang-mk wants to merge 23 commits into
Closed
congwang-mk wants to merge 23 commits into
congwang-mk wants to merge 23 commits into
Conversation
Waiting for the main process also tore down supervision and killed background jobs. Embedders could not observe command completion while keeping a service started by that command alive. Make wait return the main ExitStatus and retain the session. Add shutdown to terminate remaining jobs, reap children, collect output and settle COW changes. Expose a snapshot of tracked process groups, and have run wait and shut down for callers needing a complete result. Update the internal callers and keep complete-result FFI adapters working until the bindings expose the new lifecycle. Use a private subreaper to retain ancestry for supervised syscalls and reap orphaned descendants without changing the embedder's subreaper setting. Pin the main PID until the supervisor obtains its pidfd, and use a shutdown channel to clean up descendants outside tracked groups. Cover repeated and cancelled waits, background output and COW writes, embedder child ownership, descriptor exhaustion and descendant cleanup. The 124 affected integration tests, the reaper regression and workspace checks pass. Fixes #292 Signed-off-by: Cong Wang <cwang@multikernel.io>
The C wait adapters still collected a complete result and released supervision, preventing bindings from retaining background jobs after the main process exited. Return status-only results from handle waits and add explicit shutdown to collect output and filesystem changes. Leave the session alive when a wait times out so callers can retry or shut it down. Expose tracked process groups and pause/resume operations through the C API. Update complete-result test callers and regenerate the C header. Add coverage for retained background jobs, repeated waits and recovery after a wait timeout. The complete FFI test suite passes. Signed-off-by: Cong Wang <cwang@multikernel.io>
Python wait freed the native handle and returned the complete result, so background jobs could not survive the main process's exit. Return ExitStatus from Sandbox and Process waits, retaining the session until shutdown collects output and filesystem changes. Expose process group inspection. Raise TimeoutError without terminating jobs when a wait expires, while keeping run timeout cleanup and result reporting. Serialize native handle access and use bounded waits so another thread can signal jobs without aliasing the native wait. Coordinate context exit with an active wait, clean up abandoned sessions, and clear cached status when restoring a new process. Keep deferred branches available for commit or abort after shutdown. Update the Python API documentation and cover session reuse, timeout recovery, concurrent cleanup and restored status. All 139 lifecycle, sandbox and checkpoint regression tests pass. Signed-off-by: Cong Wang <cwang@multikernel.io>
Process.Wait released the native handle after collecting a result, ending background supervision when the main command finished. Return a cached ExitStatus from Wait and add Shutdown for the complete result. Expose ProcessGroups and apply signals to tracked groups through the native API. Have Run wait and shut down, preserving timeout results and deferred branch ownership. Serialize bounded native waits with signals and handle cleanup so Kill and Close can interrupt a blocked Wait. Let native handle cleanup reap abandoned sessions instead of signalling a cached numeric process group that could have been reused after the main process exited. Update the Go API documentation and add session lifetime, concurrent close, run timeout and finalizer regressions. All 55 Go tests pass with the race detector enabled. Signed-off-by: Cong Wang <cwang@multikernel.io>
Main-process completion and session completion are now separate events. The sandbox reference needs to explain when background supervision, captured output and staged filesystem changes remain active. Document the shared create, wait, signal, shutdown and run semantics without binding-specific examples. Explain timeout behavior, tracked group snapshots, output ownership and COW settlement at session completion. Clarify that shutdown uses SIGKILL and that releasing the owning handle also cleans up its jobs. Signed-off-by: Cong Wang <cwang@multikernel.io>
Shutdown asked the private reaper to terminate descendants, then also sent pidfd group signals. Kernels such as Linux 6.8 reject that flag with EINVAL, causing even a completed echo command to fail under a degraded policy. Use the reaper's shutdown channel for sessions it owns and wait for it to finish reaping before releasing supervision or settling COW. Keep policy callback cancellation and the legacy cleanup path for sessions without a reaper. Skip group signalling on repeated shutdown as well. Add a subprocess regression that rejects pidfd group signals through seccomp. Cover complete runs, repeated shutdown and cleanup of a child that outlives the main process in a separate session. All 87 selected protection, process-group, policy, streaming and COW tests pass. Signed-off-by: Cong Wang <cwang@multikernel.io>
The lifecycle changes routed kill through sandlock_handle_kill without registering its ctypes signature. The default integer conversion can truncate a native handle to 32 bits, crashing checkpoint and lifecycle tests on hosts where allocations use higher addresses. Declare the handle argument as c_void_p alongside pause and resume. Add a regression that checks pointer-width argument handling and kills a live sandbox from a fresh interpreter. The full Python suite passes: 659 tests passed and 15 skipped. Signed-off-by: Cong Wang <cwang@multikernel.io>
The session reaper killed leftover descendants by reading /proc/thread-self/children. That needs CONFIG_PROC_CHILDREN and a usable procfs, and when the read failed the reaper waited forever, hanging shutdown and Drop. Every supervised process is in a tracked process group, since setsid and setpgid are trapped, so the leader pidfds already reach all of them. Let the reaper only reap: it reports the main status along with whether anything else remains, and exits once it has no children. Its exit is a kernel-stamped sign that the session is empty, and shutdown awaits it through a pidfd instead of polling waitpid. Jobs outside tracked groups (new groups under no_supervisor, or any job on kernels without pidfd group signals) cannot be killed without procfs. Shutdown asks the reaper to finish, releasing them still confined, and stops the capture drains so a released writer cannot hold the result hostage. Finish is a byte, not EOF, because children parked before exec may hold a copy of the pipe. Signed-off-by: Cong Wang <cwang@multikernel.io>
Kill reaches the sandbox only through recorded process groups. When pidfd_open failed for a new leader, or the parent check could not confirm the target, setsid and setpgid were let through unrecorded, so a group could form that no kill or shutdown would ever reach. Fail closed instead: return the pidfd_open error, and ESRCH for a target that is not the caller or a live child of it, matching what the kernel itself would report for a non-child. Signed-off-by: Cong Wang <cwang@multikernel.io>
PDEATHSIG fires when the parent thread exits, not the parent process. The FFI forks the reaper on the caller's thread, so a binding that created a sandbox from a short-lived thread lost the session when that thread ended: the reaper was killed, the main process followed, and background jobs were released unsupervised. Watch a pidfd of the owning process instead, opened before checking that the owner is still the parent. When the owner dies the reaper kills the main process, still its unreaped child, and finishes. The main process keeps PDEATHSIG, since the reaper is single-threaded. Signed-off-by: Cong Wang <cwang@multikernel.io>
Bindings could only wait through the handle, which a blocking wait borrows mutably. To let another thread signal the session meanwhile, they waited in 50ms slices under a lock, adding latency and spinning the runtime for every slice. Add sandlock_handle_main_exit_fd: it prepares the wait (closing untaken stdin and starting the output drains, so a chatty child cannot fill its pipe) and returns a pidfd duplicate that turns readable once the main process exits. A binding blocks on it without the handle and then calls sandlock_handle_wait, which no longer blocks. Signed-off-by: Cong Wang <cwang@multikernel.io>
Waits polled the native handle in 50ms slices so kill could interleave. Poll the main-exit descriptor without holding the lock instead, then collect the status, which returns at once. Since no native call now holds the handle across a wait, process_groups and ports no longer refuse while another thread waits. Signed-off-by: Cong Wang <cwang@multikernel.io>
Process.Wait looped over 50ms native waits so Kill and Close could take the handle in between. Block in poll on the main-exit descriptor without the mutex, retrying on EINTR from Go's runtime signals, then collect the status. Ports no longer reports empty during a Wait. Signed-off-by: Cong Wang <cwang@multikernel.io>
Shutdown and Drop re-sent SIGKILL every 10ms until no tracked group had members left. One sweep is enough: the kernel restarts a fork that races a group kill, and a process killed while parked in a trapped setsid never completes it. With every supervised process in a swept group, the reaper exits on its own once the last one is reaped, and its pidfd reports that. Only without a supervisor, or when the sweep itself failed, can jobs outside the sweep remain; finish the reaper right away then. That mode has no COW branch that must outwait them. Clones still poll, since no reaper reports the end of their descendants. Signed-off-by: Cong Wang <cwang@multikernel.io>
server_closes_on_another_uid_without_answering flaked with EPIPE: the server checks SO_PEERCRED on accept and closes a foreign peer at once, which can land before the client finishes writing its request, and the helper unwrapped the write. Ignore write errors in the helper. A server that answers still has to produce the response the own-uid test checks, so nothing is masked. Signed-off-by: Cong Wang <cwang@multikernel.io>
Sandbox.run creates its handle with sandlock_create_for_run, whose current_thread runtime advances the supervisor only inside a native call. Polling the main-exit descriptor from Python starved it: trapped syscalls were never answered, the child never exited, and the Python suite hung in CI. Wait natively in run, as the Go binding does; nothing else shares that handle. Make sandlock_handle_main_exit_fd refuse current_thread handles so the same mistake fails instead of hanging. Signed-off-by: Cong Wang <cwang@multikernel.io>
A wait that timed out returned a result whose reason was TIMEOUT, the same reason a run reports after killing the process for exceeding its timeout. One reason then meant both "still running" and "killed", and a caller had to know which call produced it. Return 1 from sandlock_handle_wait_timeout when the timeout elapses and hand the status back through an out-pointer, so TIMEOUT only ever describes a process that was killed. Signed-off-by: Cong Wang <cwang@multikernel.io>
The sandbox child is now forked by the session reaper, which sets PR_SET_PDEATHSIG and checks getppid() itself before returning into do_create. confine_child then captured getppid() right after that fork and compared it again, a check that could never fail, and set the same death signal a second time. Signed-off-by: Cong Wang <cwang@multikernel.io>
Sandbox and Process each carried their own lock, idle condition, waiting flag, cached status and copies of the same wait, shutdown and close logic. Move that into _Session, which both now own. shutdown also both reserved the handle and held the lock across the native call. The FFI forbids concurrent handle calls, so the lock is the part that matters; the reservation only made a concurrent wait fail. Keep the lock, and reserve only around wait, the one call that blocks outside it. run() reuses the shared result decoding instead of a copy of it. Signed-off-by: Cong Wang <cwang@multikernel.io>
Sandbox.__del__ ran the whole of __exit__, which can block on an idle condition for a wait in progress and also shuts down a popen Process that has its own finalizer. Neither applies when the sandbox is collected: a running wait() keeps it reachable. Free the session handle and any pending branch directly; the native drop kills what is left. Cover the case the finalizer exists for: a sandbox dropped after wait() must not leave its background jobs running. Signed-off-by: Cong Wang <cwang@multikernel.io>
Kernels before 6.9 reject PIDFD_SIGNAL_PROCESS_GROUP with EINVAL, so every group sweep failed there. shutdown() could then end no session that had a job left, and owns() refused job-control moves into the sandbox's own groups. Fall back to kill(-pgid) while the group's leader has not exited: its pid is still held, so no other group can have the number. Once the leader is gone the number may be reused, but a probe that finds no group at all is still conclusive, because an empty group can never regain members. Only a group whose leader is gone and that still has members stays out of reach, and shutdown reports it. Signed-off-by: Cong Wang <cwang@multikernel.io>
sandlock kill sent the control request, then also called killpg(child_pid) and SIGKILLed the supervisor. The child pid was recorded at startup. Now that a session outlives its main process, that pid can be reaped and reused long before the name is released, so the killpg could hit an unrelated group. Killing the supervisor unconditionally also took down an embedding host on success, and on a kernel where some groups were out of reach it released the remaining jobs while the CLI still reported success. Kill only through the tracked groups, which the control loop now holds with or without a seccomp supervisor. Report a refusal instead of claiming success. Only a supervisor that accepts the connection but never answers is killed, through a pidfd opened while SO_PEERCRED vouches for it, and the session reaper then kills the main process. Signed-off-by: Cong Wang <cwang@multikernel.io>
ps showed the main pid from the registry and read its uptime and command line from /proc. Background jobs now keep a session alive after main exits, and its pid is then free for reuse, so ps could describe an unrelated process. The info verb now reports whether main has exited, from the main pidfd the control loop holds. ps reads /proc for the pid only while the supervisor vouches that main is alive, and otherwise shows the session as running with "(main exited)" in place of a pid. Signed-off-by: Cong Wang <cwang@multikernel.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Waiting for the main process used to tear down supervision and terminate background jobs. Separate main-process completion from session shutdown so a caller can wait for a launcher to exit while continuing to use the service it started.
This changes the wait API: complete output and filesystem changes are collected at shutdown. Python wait timeouts raise TimeoutError. Shutdown is immediate termination, not a graceful stop.
Limits: terminating jobs requires pidfd process-group signals (Linux 6.9). On older kernels shutdown fails if jobs outlive the main process. Under no_supervisor, a job that moves to a new group or session is not tracked; shutdown releases it, still confined, instead of terminating it.
Validated by CI across Rust (x86_64, arm64, low-ABI 22.04, feature matrix, riscv64 cross-build), Python 3.10 to 3.14, and the Go SDK.
Fixes #292