Repository navigation
test(ci): isolate the unit and guard suites from Docker - #43
Draft
knittingcode wants to merge 32 commits into
Draft
knittingcode wants to merge 32 commits into
knittingcode wants to merge 32 commits into
Conversation
tests/devenv/unit.sh sources and runs bin/devenv, and the guard regression suites run the guards; both accept a modified or mutated copy (DEVENV=, GUARD=, ADAPTED= or an edited file). Nothing kept that code away from the Docker daemon of the machine, so a mutant with a destructive Docker command ran against the developer's real images. scripts/ci/no-docker.sh runs a command with no route to a container engine: - macOS: sandbox-exec denies Unix socket connects, TCP beyond loopback and to loopback ports 22, 2375 and 2376, writes under $HOME, open, osascript, launchctl, ssh, Launch Services and Apple Events; ps is an ad-hoc signed copy because setuid programs cannot run inside. - Linux: sudo unshare with a network namespace that has only loopback, a tmpfs over /run, then the calling user without supplementary groups and with no_new_privs. - DOCKER_* and similar variables are removed; DOCKER_CONFIG is empty. scripts/ci/no-docker-probe.py proves from inside that the mechanism is in effect and that no engine answers on the socket paths, contexts, DOCKER_HOST, sockets in the usual directories and listening loopback ports. Otherwise the command does not run (exit 125). unit.sh and the four guard suites re-execute themselves under it before they source or run anything. integration.sh, which uses the real daemon, refuses a DEVENV that differs from bin/devenv. Inside the isolation sudo is unavailable, so the U-16 cross-user read check reports INFO and is not counted (287 instead of 288 in CI).
scripts/ci/tests/no-docker.test.sh inserts a read-only payload into a copy of bin/devenv, passed as DEVENV and edited into the file. The payload reaches Docker in 22 ways: inherited DOCKER_HOST, PATH lookup, absolute paths, command, env, --context, nested bash and sh, command substitution, backticks, a variable, a quoted name, exec, xargs, a background subshell, sudo, curl --unix-socket, a raw Python socket and the real socket paths. Its target is a canary engine that answers like Docker and records every request. - T-00: the read-only lint rejects non-read-only payloads; the payload itself is pinned by sha256, as the macOS cases run on the host. - T-01/T-02: --check fails outside the isolation; a claimed but unproven isolation is refused. - T-03/T-04: on macOS the mutant runs to its end with no request reaching the canary and the real socket paths refused; on Linux the harness refuses because the canary is a reachable engine. - T-05: the same mutant under unit.sh without the re-execution reaches the canary, behind an outer guard that leaves only the canary reachable. Run with HARNESS=<unit.sh of main> the suite fails T-03 and T-04. CI runs it in Repository guard; ShellCheck covers the new scripts.
Documents how the unit and guard suites isolate themselves from Docker on macOS and Linux, what the probe checks, the U-16 check that no longer runs inside the isolation, the regression suite, and the rules for mutation testing: read-only payloads only, never through the real-Docker suites.
U-04 points RUNTIME_DIR, STATE_FILE and LOCK_DIR at the suite's temporary directory but left GATE_DIR, GATE_FILE and SECRETS_DIR at the checkout's .runtime, so write_state in U-05 created (and chmodded) .runtime/gate in the checkout under test. Inside the macOS isolation, which denies writes under $HOME, a fresh checkout therefore failed at U-05. All runtime paths now point at the temporary directory.
This was referenced Oct 29, 2026
The macOS profile of scripts/ci/no-docker.sh left routes by which a mutated script could reach the Docker daemon outside the sandbox: - loopback TCP to local services that were already listening (agents, IDE servers, Docker Desktop's published-port forwarders, a socket proxy); the probe only recognised a Docker API answering /_ping; - writes outside $HOME, e.g. into Homebrew's prefix or /Applications (the Docker CLI under Docker.app is user-writable), whose programs run later outside the sandbox; - the exec denials of open, osascript and launchctl match a literal path, and a re-signed copy runs. Deny connects to every loopback port listening before the sandbox starts (listed by the probe's new --listening mode, refused if it cannot be listed), allow writes only under $TMPDIR, /tmp and /dev (refuse if $TMPDIR is under $HOME), and deny job-creation, lsopen and appleevent-send so the routes are closed whatever binary is used. The probe now also proves inside the sandbox that the held ports cannot be connected to and that nothing can be created under $HOME or next to $TMPDIR.
…tion T-06 (macOS) runs a plain command inside no-docker.sh that tries to create a file under $HOME and next to $TMPDIR and one in the suite's own directory: only the last may exist. T-07 starts a loopback listener before the isolation and requires that a command inside it cannot reach it while a port it opens itself stays reachable. Neither case uses the mutant; both fail against the previous profile.
…nels Follow-ups from the safety review of the previous isolation change: - The held-port list is accepted only if netstat succeeded, printed its column header and every LISTEN row parsed; otherwise the command is refused (125). An empty list is thus provably empty. - /tmp is no longer writable, except bash 3.2's sh-thd* here-document files, so agent scratchpads and other tools' files there cannot be planted. $TMPDIR must be a directory of its own: /, /tmp, under /tmp or under $HOME is refused, which also removes the vacuous TMPDIR=/tmp write check. The probe now also proves /tmp is write-denied. - Deny user-preference-write: defaults write reached ~/Library/Preferences through cfprefsd although $HOME is write-denied (measured). - Deny the TIOCSTI ioctl: a command run from an interactive terminal could inject keystrokes into the shell that started it (measured). Hard links from $TMPDIR to a write-denied file were measured to be refused already. The docs list the measured and the residual routes.
…imits Plain commands inside no-docker.sh, no mutant (macOS): - T-06 now also requires that /tmp is write-denied while a bash 3.2 here-document still works; - T-08: a fake netstat without its header or with an unparsable LISTEN row is refused (125, command not run); header and no LISTEN row is an accepted empty list; - T-09: TMPDIR=/tmp is refused; - T-10: defaults write changes no preference, and TIOCSTI on the controlling terminal of a pseudo-terminal is denied. Against the previous profile, T-06, T-08 (two cases), T-09 and both T-10 cases fail.
… neutrally Reword the comments of scripts/ci/no-docker.sh and the docs that named the local services and the other programs' files in /tmp the isolation protects, using generic examples.
…ces and narrow the macOS sandbox Linux is supported on GitHub-hosted runners only (GITHUB_ACTIONS, RUNNER_ENVIRONMENT, a Runner.Worker ancestor, workspace and RUNNER_TEMP under $HOME) and uid 0 is refused everywhere. The namespace adds --pid --fork --kill-child --mount-proc --ipc under a root pid 1 that waits and reaps; tmpfs over /run, /var/run, /tmp, /var/tmp and /dev/shm, a fresh /dev/mqueue, a masked /sys/fs/cgroup, every other mount remounted read-only; setpriv drops groups and all capabilities with no_new_privs; setsid --wait; an allowlisted environment; workflow commands are stopped around the run. Unix sockets are listed outside within a time budget and none may accept a connection inside. On macOS $TMPDIR must be the per-user directory under /private/var/folders, writes are limited to a fresh run directory (the command's $TMPDIR) and a measured set of /dev nodes, signals stay within the sandbox, and the login item, background task and Docker Desktop services cannot be looked up. --work DIR makes a fresh, empty directory of the caller the run directory and keeps it; unit.sh passes NO_DOCKER_WORK on as --work. The probe checks each property from inside and fails closed; every listening port number is held; a truncated socket walk fails on Linux.
…s and CI order T-11 to T-15 (macOS): writes outside the run directory, the TMPDIR allowlist, signals to an outside canary, integrity of the helpers run outside the isolation (kept read-only to the code under test and checked by sha256 before every bare run and every mutant), and --work. T-16 to T-18 (Linux): a canary inside checks the pid view, sockets, read-only mounts, private /tmp, masked cgroups and IPC; no daemon survives the command or a killed unshare; uid 0 and non-GitHub-hosted use are refused. T-19 checks statically that isolated steps come last in each CI job. The mutant's log moves into the run directory, the only place it may write; the payload text and its hash are unchanged.
Code under test runs only in the last steps of Repository guard (the guard regression suites and the isolation tests, after ShellCheck and actionlint) and of Dev environment (the allocator unit tests, after the container scan), so no step that is not isolated runs after it. Job names, the job set, triggers and permissions are unchanged.
… and residual risks Supported platforms (macOS developer machines, Linux on GitHub-hosted runners only), each enforced property and how the probe verifies it, --work, the CI order, the regression cases, the accepted residual risks and the interim mutation-testing policy.
unshare runs as root and, with --fork, blocks SIGINT and SIGTERM, and pid 1 ignores signals from outside its namespace, so a cancelled or timed-out step left the namespace running. no-docker.sh now runs sudo in the background and, on SIGINT, SIGTERM or SIGHUP, sends SIGKILL through sudo to the unshare it started (a process named unshare below its sudo); --kill-child then kills pid 1 and everything inside. The command keeps the caller's stdin, which an asynchronous command would otherwise lose. The stop-commands start line also goes to stderr, and the duration of the socket listing is printed on stderr and as a notice in the CI log.
T-17's canary is a shell kept alive ("sleep 60; :") so the process with
the marker is the one that must die, and the command confirms it runs
before returning; the cancellation case sends TERM to no-docker.sh, as a
cancelled step does. The macOS guard used for T-05 and HARNESS= gets the
write policy and denies of no-docker.sh. Files that code under test could
write are read only when they are regular files. T-19 counts a step as
isolated only when all its commands are no-docker.sh, its test harness or
scripts with the re-exec stanza, and fails a step that mixes them.
…ual and the stricter CI check The Linux isolation is killed when no-docker.sh is cancelled; SIGKILL of no-docker.sh itself remains a residual until the VM is discarded. bash 3.2 here-documents in /tmp, measured to ignore $TMPDIR, stay allowed and are a documented macOS residual. Also the stop-commands framing on both streams, the listing duration in the CI log, T-17, T-05's guard and the definition of an isolated step.
…the isolation's trap The INT/TERM/HUP trap of no-docker.sh handles signals delivered to the script itself, such as Ctrl-C or a caller forwarding the signal. A cancelled or timed-out GitHub step signals only the step's shell and later SIGKILLs the process tree, so it does not reach the trap; the remaining namespace keeps read-only mounts and no network until the VM is discarded, an accepted residual. Comments, documentation and the T-17 description only; no logic changes.
…ents bash 3.2 keeps here-documents and here-strings in a temporary file in /var/tmp, or in /tmp where /var/tmp is not writable, as inside the macOS sandbox of scripts/ci/no-docker.sh, which no longer allows either. usage prints its lines with printf (same output), require_owned_project reads the container list through a process substitution (the loop still runs in the current shell, so its refusals still exit) and proc_start splits /proc/<pid>/stat the same way without a here-string.
The suite runs inside the isolation, where bash 3.2 here-documents fail on macOS. The nine helper and fake files are written with printf; each file is byte for byte what the here-document produced.
…emporary files and debugging The macOS profile no longer allows writes in /tmp: the sh-thd allowance let code in one sandbox rewrite the here-documents of bash 3.2 in another sandbox that cannot use /var/tmp. It also denies ptrace (system-debug), task ports of processes outside the sandbox (as used by debuggers, vmmap and sample) and reading other terminals' devices. no-docker.sh now hands its process to a supervisor in no-docker-probe.py. The sandbox carries a marker of the run and one common to every run (a denied read of a file in a marker directory outside the run's directory); from outside, sandbox_check() tells which processes are in it. When the command ends, or SIGINT, SIGTERM, SIGHUP or SIGQUIT arrives, the supervisor SIGKILLs every member as soon as it finds it, rechecking its start time, membership and the markers first, until none is left, without starting a process. A run refuses to start while a process left by an earlier run runs on its own (after SIGKILL of no-docker.sh). Status 125 if the processes cannot be identified or ended. Inside, the probe also checks that no here-document file can be created in /tmp and that the sandbox carries both markers. On Linux it prints the uid and gid, refuses a reachable second procfs or nsfs mount and counts a listener with a full backlog as reachable; /proc/<pid>/stat is parsed without here-strings.
…imits canary.py, drive.sh and the mutant payload are tracked files whose sha256 values are constants of the suite, checked on every copy before it runs outside the isolation, instead of hashes taken after the suite wrote them. The suite's own guard sandbox loses the /tmp allowance, gets the same markers, rules and supervisor as no-docker.sh. New cases: T-06 now expects a bash 3.2 here-document to fail inside; T-20 (no here-document in code run inside), T-21 (here-document files in /tmp and /var/tmp cannot be changed or created), T-22 (no change through links, rename or exchange), T-23 (pinned helpers), T-24 and T-25 (no process outlives a normal end, a failure, SIGKILL of the command, or SIGINT, SIGTERM and SIGHUP to no-docker.sh), T-26 (after SIGKILL of no-docker.sh new runs refuse until the leftovers end), T-27 (refusal when the probe cannot find or end the processes), T-28 (no debugger attach, task port or memory map of an outside process) and T-29 (no reading of another terminal).
A job or step with continue-on-error would let a failing step, a security gate included, leave its job green. T-19 now fails on any such key.
…nts and debugging limits Where bash 3.2 keeps here-documents, why code run inside uses none, the supervisor and its markers, the measured launchd, terminal, ptrace and task-port limits, the residual risks after SIGKILL of no-docker.sh, the pinned helpers and the new regression cases; the probe's checks are described per platform.
…ep their marker on failure On macOS the marker directory and the check for processes of an earlier run now come before anything is written into the run's directory, so a leftover cannot prepare files there (such as a link where the ps copy is made). When the supervisor cannot end every process, it keeps the run's marker directory and names the command that ends them. A signal that arrives while the wait for the command is being left can no longer skip ending the processes.
…nding and the Linux probe additions A run fails with 125 when processes remain after 10 seconds, for example under a fork storm; after SIGKILL of no-docker.sh or a failed ending the command and its leftovers keep their remaining channels until they are ended. The Linux list names the proc and nsfs check, the full-backlog rule and the uid and gid line, and says the path has not yet run in CI.
… its processes runs One process table without a member of the run ended the run as a success, so a process that keeps forking and ending could survive it (B-1). The supervisor now takes one copy of the kernel's process list at a time and needs a copy in which every process is shown, by its unique id read before and after the check, not to be one of the run's, or was dead before that copy; a process that ended before it could be checked leaves the copy open. Members found are killed at once. Within 10 s otherwise the run fails with 125 and keeps its marker directory. The marker directory records the run's owner; before a run, directories whose owner is gone are checked the same way without killing, removed once shown empty, and otherwise refuse the run. --end refuses a run whose owner still runs. A process that ends while it is checked is no longer a false 125, and the supervisor's own child is killed on any failure. The marker common to every run is no longer needed.
…e macOS isolation T-30 runs a bounded chain of forks that end at once and checks that the run is never reported done while it hops, that new runs refuse meanwhile and start after it ended. T-31 runs two isolations at the same time, T-32 keeps unrelated processes starting and ending during the ending, T-33 makes the ending impossible to show (125, own child ended, marker kept) and T-34 sends TERM at several moments while processes are started. T-21 creates its canaries in the shared temporary directories with random names, exclusively and without following links, and reads and removes them only while they are the files it made.
The supervisor's proof over copies of the process list, the 125 when it cannot be given (measured for a process that keeps forking and ending, which is contained but not ended), the owner record of each run and the new regression cases.
…nding a macOS run A zombie is found only with an argument of 1 to proc_pidinfo; read as vanished, one that lives long kept every copy of the process table open and failed every run with 125. It now counts as dead from the next copy on. When a run whose owner is gone is checked (before a run, or --end), a process that owner started and that is not yet a member keeps the copy open, as it may be the run's sandbox-exec before it applied the profile. Waiting for the supervisor's own child after killing it is bounded.
…bie outside T-34 now waits until the command recorded its first process, sends TERM 0, 0.15 and 0.4 s later, requires processes to have been recorded and accepts 125 only with its message. T-35 keeps a zombie of the suite's own outside while a run with leftovers must end with status 0. T-30 and T-34 print what they observed.
… end relies on The run's own sandbox-exec before it applies the profile, zombies as dead from the next copy on, pids not reused within one copy, and how marker directories of earlier versions are handled.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Runs the code under test of
tests/devenv/unit.shand of the guard regression suites only where no container engine can be reached, and adds a regression suite that proves it with a mutatedbin/devenv.Motivation
tests/devenv/unit.shsources and executesbin/devenv, and the guard suites execute the guards. Both are used for mutation testing: withDEVENV=,GUARD=orADAPTED=, or by editing the file. Nothing kept that code away from the Docker daemon of the machine. On 2026-10-28, a review run edited a destructive Docker command into a copy ofbin/devenvand then ranunit.sh. The command ran against the developer's real Docker daemon and removed 17 unused images, including images of unrelated projects. They have since been restored.Implementation
scripts/ci/no-docker.shruns a command with no route to a container engine.sandbox-execdenies Unix socket connects, TCP beyond loopback and to loopback ports 22, 2375 and 2376, and writes under$HOME. It also blocksopen,osascript,launchctlandssh, and access to Launch Services and Apple Events. Setuid programs cannot run inside the sandbox, sopsis an ad-hoc signed copy.sudo unsharecreates a network namespace with only loopback and puts a tmpfs over/run. The command then runs as the calling user without supplementary groups and withno_new_privs.DOCKER_*and similar variables are removed, andDOCKER_CONFIGpoints to an empty directory.scripts/ci/no-docker-probe.pyproves from inside the isolation that it is in effect, and that no engine answers on any of these:DOCKER_HOST;/run,/var/run,/tmp,$TMPDIRand~/.docker;If isolation cannot be proven, the command does not run and the exit status is 125.
unit.shand the four guard suites re-execute themselves underno-docker.shbefore they source or run anything.tests/devenv/integration.shuses the real daemon on purpose, so it now refuses aDEVENVthat differs frombin/devenv.scripts/ci/tests/no-docker.test.shuses a read-only mutant whose payload is pinned by sha256, so any change to it needs a new review. The mutant reaches Docker in 22 ways. Its target is a canary engine of the test that answers like Docker and records every request.--checkfails outside the isolation, and an isolation that is claimed but not proven is refused.DEVENVand edited intobin/devenv. On macOS it must run to its end with no request reaching the canary and the real socket paths refused. On Linux the harness must refuse, because the canary is a reachable engine.unit.shwithout the re-execution must reach the canary, behind a guard that leaves only the canary reachable.CI: new step "Run test isolation regression tests" in Repository guard, and ShellCheck now covers the new scripts.
Docs:
docs/dev-environment.md, "Isolation from Docker".U-04 fix: U-04 now points
SECRETS_DIR,GATE_DIRandGATE_FILEat the suite's temporary directory. Before,write_statecreated.runtime/gatein the checkout under test, which the macOS isolation correctly refuses.Testing
All runs were local on macOS 26 with Docker Desktop running.
unit.sh: 287/287 with bash 5 and with/bin/bash3.2, inside the isolation. The probe reported every check ok.no-docker.test.sh: 14/14 passed.HARNESS=set to theunit.shof main (1badf76), T-03 and T-04 fail: the mutant reached the canary 34 times.docker eventsshowed no image, volume, network or container lifecycle event, and the image list was identical before and after.no_new_privs, no groups, loopback only and a private/run. Dev environment 14/14 steps; Repository guard 15/15 steps. Unit tests 287/287 with U-16 cross-user INFO; integration 140, backend 18, frontend 5, application stack 124, production-like stack 39, container scan 20; guard suites 36, 73, 87 and 221; no-docker.test.sh 14/14.Security considerations
The macOS profile filters the listed routes; it is not a complete jail. On Linux, an engine socket owned by the calling user is found only in the directories listed above. Payloads used for mutation testing must only read or print, and the docs say so.
Known limitations
sudois unavailable. The U-16 check that another local user cannot read a secret therefore prints INFO in CI as well, so CI counts 287 unit checks instead of 288. The U-16 mode checks still run.integration.sh,tests/app/) must not be used for mutation testing.