Skip to content

Keep background jobs supervised after the main process exits - #301

Merged
congwang-mk merged 18 commits into
mainfrom
issue-292-retain-jobs
Oct 16, 2026
Merged

Keep background jobs supervised after the main process exits#301
congwang-mk merged 18 commits into
mainfrom
issue-292-retain-jobs

Conversation

@congwang-mk

@congwang-mk congwang-mk commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Fixes #292. Replaces #299 with a simpler design.

Problem

wait() killed every remaining process as soon as the main process exited, so a command that started background work lost it, and an embedder had no way to let that work finish under supervision.

Design

  • Session reaper. Each sandbox runs under a small reaper process outside the sandbox. It is a subreaper, so orphaned jobs stay below the supervisor (needed under Yama) and get reaped without the embedder's help. It relays main's exit status, and kills main's group if its owner dies.
  • Main's group is named after the reaper. The reaper creates process group R from its own pid, forks main into it and steps back out. While the reaper lives no other process can hold that number, so kill(-R) reaches main and its cmd & jobs exactly on every kernel, including after main is reaped.
  • wait() / shutdown(). wait() returns main's exit status and keeps the session. shutdown() kills what is left, waits until no session process remains, then settles COW and collects output.
  • Controller. Kill, pause, resume and process-group listing work from another task or thread while wait() holds the sandbox. The FFI stores it beside the sandbox in the handle, so the bindings need no locking around a blocked wait.
  • Older kernels. Before 6.9 there is no group signal by pidfd. Groups the sandbox creates are signalled by number while their leader lives, and a probe proves an empty group gone. Only a double-forked setsid daemon whose leader exited is unreachable there; shutdown() reports it.
  • CLI. sandlock kill goes only through the supervisor's group table and kills the supervisor only if it accepts a connection and never answers. sandlock ps hides main's pid once main has exited, since that pid can be reused while the session lives on.

API changes

  • Rust: Sandbox::wait() returns ExitStatus; new shutdown() returns RunResult; new controller(). Process gains shutdown() and controller().
  • C: sandlock_handle_wait(h, timeout_ms, &status) returns 0 / 1 (timeout) / -1; new sandlock_handle_shutdown, _pause, _resume, _process_groups; sandlock_handle_wait_timeout removed.
  • Python and Go: wait() returns an ExitStatus (timeouts raise TimeoutError / return ErrTimeout without killing); new shutdown() and process_groups().

Testing

Rust core (919 unit, 542 integration), CLI, FFI, Python (443) and Go suites pass locally. New tests cover jobs surviving wait(), shutdown() with a job holding captured stdout, a paused session, killing during a blocked wait, a cancelled wait, Drop after wait(), owner death, an emulated pre-6.9 kernel, sandlock kill and ps after main exits, and learn --timeout.

🤖 Generated with Claude Code

@congwang-mk
pgroup: signal groups by number on kernels without pidfd group signals
99bda46 
Kernels before 6.9 reject PIDFD_SIGNAL_PROCESS_GROUP with EINVAL, so
every group sweep failed there and a session could not be killed.

Fall back to kill(-pgid) while the group's key process has not exited:
that process still holds the number, so no other group can have it.
Once it has exited the number may be reused, but a probe that finds no
group at all is still conclusive, because an empty group can never
regain members. Only a populated group whose key process is gone stays
out of reach, and it is reported. Add live() to list the groups that
still have members.

Cover each case in a child whose seccomp filter rejects group pidfd
signals, so the tests run on any kernel.

Signed-off-by: Cong Wang <cwang@multikernel.io>
@congwang-mk
reaper: add a session reaper that names main's process group
ec70b38 
Background jobs that outlive the main process need two things the
supervisor cannot give them. Under Yama, the supervisor can only read
memory and fds of its own descendants, but an orphaned job is
reparented to init. And once main is reaped, the group named after
main's pid can be reused, so nothing can signal it safely by number on
kernels without group pidfd signals.

Fork a reaper between the supervisor and main. It is a subreaper, so
orphans stay below the supervisor, and it reaps them so the embedder
never has to. It creates process group R from its own pid, forks main
into R and moves itself back to the supervisor's group: while the
reaper lives no other process can hold that number, so R stays exact
to signal for the whole session, and a sweep of R never kills the
reaper itself. It relays main's wait status over a pipe, exits once it
has no children, kills R if its owning process dies, and on request
exits early to release jobs nobody can track.

Test the status relay and group naming, outliving main and the
creating thread, releasing jobs on request, and killing R when the
owner dies.

Signed-off-by: Cong Wang <cwang@multikernel.io>
@congwang-mk
sandbox: run every sandbox under a session reaper
21c25ba 
The main process led its own process group, so once main was reaped
its group number was free for reuse and nothing could signal the jobs
left in it by number. Orphaned jobs were reparented to init, out of
the supervisor's reach under Yama.

Fork each sandbox through the session reaper. Main now runs in the
reaper's group R, recorded with the reaper's pidfd, so the group stays
signallable after main exits on every kernel. Main's exit status comes
from the reaper through a watch slot that a background task fills as
soon as main is reaped. The reaper also sets main's parent-death signal
and checks its parent, so confine_child no longer repeats that. The
terminal foreground is handed back by comparing against R. wait() still
ends the session with main: it sweeps the groups and waits for the
reaper to exit, releasing untracked jobs when there is no supervisor.

The tty handover test now expects the foreground to go to the child's
group rather than its pid, and the no-supervisor control test expects
main to run in the reaper's group. All core, CLI and FFI tests pass.

Signed-off-by: Cong Wang <cwang@multikernel.io>
@congwang-mk
reaper: kill the session when its owner goes away
2f8aee6 
When the owning process died, its end of the finish pipe closed too, so
the reaper saw the finish descriptor readable and exited before it
looked at the owner pidfd. Group R was never killed: main died of its
parent-death signal, but every other job in the group kept running
with nobody supervising it.

Check the owner pidfd first, and tell the finish byte apart from EOF.
A byte still releases the remaining jobs; EOF means the handle is gone
without a finish or a join, so the session is killed like on owner
death.

The owner-death test now re-executes itself as the owner instead of
running the reaper in a child forked from the multithreaded test
binary, and reads only up to the reported group, since concurrent
forks in the test binary can hold its stdout pipe open.

Signed-off-by: Cong Wang <cwang@multikernel.io>
@congwang-mk
sandbox: keep background jobs supervised after main exits
2464823 
wait() killed every remaining process as soon as the main process
exited, so a command that started background work lost it, and an
embedder had no way to let that work finish under supervision.

Split the lifetime in two. wait() now returns main's exit status and
leaves the session running. shutdown() kills what is left, waits until
the session has no process, then settles the COW branch and collects
the captured output. A Controller, obtained from the sandbox, signals
and lists the session's process groups from another task or thread
while wait() holds the sandbox, and cancels a held policy_fn verdict
on kill. Capture drains are cut once jobs are released without a
supervisor, since those may hold the write ends forever. Runs, popen
processes, pipelines, transactions, the OCI supervisor and sandlock
learn call shutdown() where they need the full result.

Add tests for jobs surviving wait(), shutdown() with a job holding the
captured stdout, shutdown() of a paused session, killing through the
Controller during a blocked wait(), a cancelled wait(), and dropping
the sandbox after wait(). An emulation of kernels without group pidfd
signals checks that a backgrounded job is killed, a setsid job with a
live leader is killed, and a double-forked daemon is reported as
unreachable. All core, CLI and FFI tests pass.

Fixes #292

Signed-off-by: Cong Wang <cwang@multikernel.io>
@congwang-mk
ffi: separate main-process waits from session shutdown
b9d0191 
sandlock_handle_wait returned only when the whole session was done and
killed everything left at main's exit. Killing a session from another
thread meant racing the handle that the blocked wait held.

sandlock_handle_wait now takes a timeout and an out-pointer: it returns
0 with main's status once main exits, or 1 if the timeout elapsed, and
leaves the session running either way. sandlock_handle_shutdown ends
the session and returns the full result, replacing
sandlock_handle_wait_timeout. Kill, pause, resume and the new
process-group listing go through a Controller stored beside the
Sandbox in the handle, so they are safe while another thread is
blocked in sandlock_handle_wait.

Existing tests wait, then shut down. New tests kill a session from
another thread during a blocked wait, and check that a timed-out wait
leaves the session running. They live in their own test binary, since
the popen fd-leak test counts the whole process's descriptors.

Signed-off-by: Cong Wang <cwang@multikernel.io>
@congwang-mk
python: wait for main, shut down the session explicitly
95f0b98 
Sandbox.wait() and Process.wait() freed the handle and returned the full
Result, so background jobs died with the main process. Kill, pause and
resume signalled main's pid by number, which after this change can be
reused while the session lives on.

wait() now returns an ExitStatus when main exits, keeps the session, and
raises TimeoutError on a timeout without killing anything. shutdown()
ends the session and returns the Result, and process_groups() lists
what is left. Signals go through the native controller instead of
os.killpg. Every native call except a wait runs under a lock, and waits
only count themselves in and out, so a signal from another thread
reaches a blocked wait while nothing can use a handle that shutdown()
is freeing. Leaving the context or dropping a Process shuts it down.

Existing tests now wait and then shut down where they read output, and
expect a wait timeout to raise. New session tests cover jobs surviving
wait(), a timed-out wait, killing from another thread, shutdown() ending
a blocked wait, context exit ending leftover jobs, and run() timeouts.
All 443 Python tests pass.

Signed-off-by: Cong Wang <cwang@multikernel.io>
@congwang-mk
go: wait for main, shut down the session explicitly
087cbb1 
Process.Wait returned only when the whole session was done, and Kill,
Pause and Resume signalled main's pid by number, which can be reused
once main is reaped while the session lives on.

Wait now returns main's ExitStatus and keeps background jobs
supervised; WaitTimeout returns ErrTimeout and leaves the session
untouched. Shutdown ends the session and returns the Result, and
ProcessGroups lists what is left. Signals go through the native
controller, which is safe beside a blocked native wait, so the wait
runs without the mutex; Shutdown and Close kill an in-flight Wait and
let it return before they touch the handle. That hand-off uses a
channel rather than a sync.Cond on the mutex, since a Cond would point
back into the Process and keep Go from running its finalizer. Run
waits with the context deadline, then shuts down.

Existing tests shut down where they need changes or a deferred branch.
New tests cover jobs surviving Wait, a timed-out wait, and Shutdown
ending a blocked Wait. All Go tests pass.

Signed-off-by: Cong Wang <cwang@multikernel.io>
@congwang-mk
cli: kill and list sessions through the supervisor
9018336 
sandlock kill sent the kill request, then also called killpg on main's
recorded pid and SIGKILLed the supervisor. A session now outlives its
main process, so that pid can be reaped and reused while the name is
still registered, and the killpg could hit an unrelated group. Killing
the supervisor on every call also took down a Python or Go host that
embeds the sandbox. sandlock ps had the same stale pid problem when it
read main's uptime and command line from /proc.

The control loop now holds the session's Controller, with or without a
seccomp supervisor. sandlock kill goes only through it and reports a
refusal instead of claiming success. Only a supervisor that accepts the
connection and then never answers is killed, through a pidfd opened
while SO_PEERCRED vouches for it, and the reaper then kills main's
group; a closed connection means the host is tearing down and is left
alone. The kill request waits longer than the server's per-client cap,
so queueing behind another client is not mistaken for a wedge. The info
verb reports whether main has exited, and ps shows main's pid, uptime
and command only while it is alive.

Add tests for killing a sandbox without a supervisor, for ps after main
exits, and for kill racing a host that is shutting down. All core and
CLI tests pass.

Signed-off-by: Cong Wang <cwang@multikernel.io>
@congwang-mk
sandbox: kill remaining jobs when a sandbox is dropped after wait()
ac96a51 
Drop swept the process groups only while the main process was created,
running or paused. After wait() the state is MainExited, so a sandbox
dropped between wait() and shutdown() skipped the sweep. Dropping the
reaper handle still made the reaper kill main's group, but jobs that
had moved to groups of their own, such as a setsid daemon, kept running
with nobody supervising them.

Sweep in MainExited too, so Drop kills every tracked group and waits for
the reaper, as shutdown() does.

Add a test that drops the sandbox after wait() while a setsid job runs
in its own group. All Rust, Python and Go tests pass.

Signed-off-by: Cong Wang <cwang@multikernel.io>
@congwang-mk
learn: end the whole session on timeout
6349967 
When the observed command ran past --timeout, sandlock learn killed the
main process by its recorded pid and waited again. Background jobs it
had started were only reached later by shutdown, and the pid was used
as an identity at all, which the session lifetime change avoids
everywhere else: once main is reaped its pid can be reused.

Drop the pid kill. shutdown() already kills every process group of the
session and waits for all of them before the partial profile is
written.

Add a test that runs learn with a one-second timeout over a command
that backgrounds a job, and checks that the partial profile is written
and the job is gone. All CLI tests pass.

Signed-off-by: Cong Wang <cwang@multikernel.io>
@congwang-mk
sandbox: detect exec completion through a child-owned exec pipe
1891e59 
spawn() decided that main had exec'd by polling /proc/<pid>/exe every
1ms until it stopped matching the supervisor's binary. The session
reaper collects main the moment it exits, so a main like
`sh -c "sleep 60 & exit 0"` could exec and vanish between two polls and
be reported as "child exited before exec() completed". The comparison
also could never succeed for a workload that execs the supervisor's own
binary.

The child now opens a CLOEXEC pipe, hands the read end to the parent
alongside the notif fd, marks it right before execve and again if
execve returns. A successful exec closes the write end, so spawn() reads
to EOF and judges the marks. The pipe is created after fork so that a
sibling forked by the parent, such as another sandbox parked before
start(), cannot inherit the write end and hold back the EOF.

Signed-off-by: Cong Wang <cwang@multikernel.io>
@congwang-mk
sandbox: refuse to checkpoint once main has exited
10479d0 
checkpoint() captured main by its recorded pid. The session reaper
reaps main as soon as it exits, so that number can be reused while the
session lives on, and a checkpoint could freeze the session and capture
an unrelated process, such as one in another sandbox of the same host.
Return NotRunning once main has exited.

Signed-off-by: Cong Wang <cwang@multikernel.io>
@congwang-mk
sandbox: report main's pid only until it exits
793fec7 
The session reaper reaps main as soon as it exits, so its pid can be
reused while the session lives on. pid() kept returning it until the
Sandbox was dropped, and Python and Go cached it at creation, so a
caller reading /proc or signalling that number could reach an
unrelated process.

pid() now returns None once main has exited, in Rust, through the
Controller for the FFI, and in Python and Go, which query it on each
call instead of caching it. Main's exit is published on one watch
owned by the runtime: the reaper's status relay fills it, and wait()
fills it on paths without a reaper, such as fork() clones, so
checkpoint() relies on pid() as well.

Signed-off-by: Cong Wang <cwang@multikernel.io>
@congwang-mk
sandbox: read main's exit status without a background task
36f06d4 
Main's status was read by a task spawned on the runtime that created
the sandbox and published on a watch. If that runtime went away, the
task died with it, and since the runtime also held a sender, wait()
never returned. The reaper also reaped main before sending its status,
so for a moment main's pid was free while pid() still reported it.

The reaper now reads main's status with WNOWAIT, sends it, and only
then reaps. A shared MainExit cell reads the status pipe without
blocking for pid(), checkpoint() and the controller, and wait() polls
it through an AsyncFd on whichever runtime calls it. Paths without a
reaper still publish through wait().

Signed-off-by: Cong Wang <cwang@multikernel.io>
@congwang-mk
oci: drop the killpg fallback on the container's init pid
309b58c 
kill --all and delete --force fell back to killpg(init pid) when the
daemon could not be reached. sandlock-init now runs in its reaper's
process group and never leads one, so that number names no group of
the session: the call either failed or, once the number was reused as
a group id, signalled unrelated processes. The fallback is not needed
either: the daemon owns the reaper, which kills the session when its
owner dies.

Signed-off-by: Cong Wang <cwang@multikernel.io>
@congwang-mk
sandbox: return the terminal from any sandbox group on shutdown
5d04f4e 
wait() handed the terminal back only while main's own group held it,
and only once. A job that outlives main can block SIGTTOU and take the
terminal back afterwards; shutdown() then killed it and left the
caller's group in the background, so its next read from the terminal
stopped it. The same happened when main moved itself into another
group before taking the terminal.

Restore the caller's foreground whenever a group the sandbox tracked
holds the terminal, on wait() and again in shutdown() and Drop after
the session is gone.

Signed-off-by: Cong Wang <cwang@multikernel.io>
@congwang-mk
cli: make ps exact about main and show what a session has left
52e0734 
ps asked the supervisor whether main was alive, then read /proc/<pid>.
If main exited in between and its pid was reused, the row showed
another process's uptime and command. It now opens a pidfd for main
before asking, so a positive answer proves the pidfd names main, and
keeps what it read from /proc only if main is still alive afterwards.

After main exits, a session stays listed until its owner shuts it down,
and ps showed "(main exited)" whether jobs were still running or none
were. The info verb now reports how many process groups still have
members, and ps shows "(main exited, N groups)" or "(main exited,
idle)".

Signed-off-by: Cong Wang <cwang@multikernel.io>
@congwang-mk
congwang-mk merged commit 378c5d4 into main Oct 16, 2026
23 checks passed
@congwang-mk
congwang-mk deleted the issue-292-retain-jobs branch October 16, 2026 02:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

sandbox: keep background jobs supervised after the main process exits

1 participant

@congwang-mk