Repository navigation
Conversation
Add a Codable description of a VZVirtualMachineConfiguration's board (platform, boot loader, storage, network, virtio-fs, socket, entropy and serial devices) for VM save/restore. The snapshot is built by walking the final, validated configuration, and is used only to check that a newly built configuration matches the one a saved machine state depends on. It is not used to rebuild a configuration; toVZ() remains the only builder. Building a snapshot also sanitizes the configuration: any device or attachment outside the allowlist in docs/vm-save-restore.md throws .unsupported. This includes nested virtualization, virtio-fs devices with shares, storage devices with a non-empty blockDeviceIdentifier, and any graphics, audio, input, USB, console or balloon device. Identity is established by content where the bits live in guest memory (kernel, initial ramdisk, initial filesystem, by SHA-256) and by value elsewhere (machine identifier, effective MAC addresses, cpu and memory size, host OS version and model). Disk image paths are deliberately not compared so a restore from a cloned image is accepted. Network attachments are limited to NAT and vmnet.
Write a paused virtual machine's state to a new directory bundle at `url`: - `vm.state`: the machine state written by VZVirtualMachine's saveMachineStateTo, a dump of guest memory and device state. - `configuration.json`: a VZVirtualMachineConfigurationSnapshot of the board the state depends on, so a later restore can verify that the configuration it is given matches. save does not change the run state. The caller owns the pause/resume lifecycle: pause, preserve the backing peripheral state (e.g. clone disk images) while the guest is quiescent, then resume or stop. Pausing inside save would leave no window to capture disks that match the memory dump. Everything that can fail without touching the bundle runs first (exists check, paused check, configuration validation, snapshot and encoding), so an unsupported configuration never leaves a directory behind. The bundle is created with 0700 permissions since the state file contains guest memory, and is removed again if writing it fails.
Restore a stopped virtual machine from a state bundle written by save(to:) and run it. This is the counterpart of start() for a saved machine: the guest does not boot, it continues from where it was saved. Before asking VZ to restore, the bundle's configuration snapshot is loaded and compared against a snapshot of the instance's own configuration. Any difference in the board (kernel, initial filesystem, device layout, machine identifier, ...) fails with .invalidArgument and lists every mismatch, rather than letting VZ restore a machine state into an incompatible board. The caller remains responsible for the backing peripheral state (disk image contents, network identity). VZ leaves a restored machine paused, so restore resumes it, reconnects the agent, sets the guest clock (which is as stale as the save) and starts the time syncer. Connections the host held across the save do not survive and must be re-established by the caller. Add save(to:) and restore(from:) to the VirtualMachineInstance protocol, with .unsupported defaults for backends that cannot save and restore, and a restoreMachineState helper on VZVirtualMachine.
Checkpoint a running container to a new directory at `url`, then keep it running. The directory holds: - `container.json`: a LinuxContainer.Checkpoint manifest (format version and container id), with Checkpoint.save(to:) and Checkpoint.load(from:). - `state/`: the virtual machine's saved state bundle. The container is paused for the duration of the checkpoint and resumed afterwards, whether or not the checkpoint succeeded. A failure removes the partially written directory. Containers with exec'd processes are rejected, since those would have to be reconstructed on restore. Only memory and board are captured, not disks, and the saved state is only consistent with the disks as they were while the container was paused. The optional `whilePaused` closure runs after the state is saved and before the container resumes, so the caller can preserve the backing disk images (e.g. clone them) while the guest is quiescent. If it throws, the checkpoint is removed and the error is rethrown.
Let LinuxContainer.create(from:) take a directory written by checkpoint(to:), and restore the virtual machine from its saved state instead of booting it. - VirtualMachineManager.create gains a `from savedState: URL?` parameter, with an extension overload that keeps the old call shape for booting normally. A manager that cannot restore must throw .unsupported for a non-nil value; CHVirtualMachineManager does. - VZVirtualMachineManager passes the saved state to VZVirtualMachineInstance, which reads the machine identifier from the saved configuration snapshot. VZ requires a restored machine to carry the saved machine's identifier, and it is otherwise generated afresh per configuration. - LinuxContainer.create(from:) calls restore(from:) on the new instance and skips the guest setup that follows a normal boot (mounts, networking, and so on), since all of that is already part of the saved state. If the restore fails, the instance is stopped and the container is marked errored.
IOPair closes both of its ends when its relay finishes, including when the vsock peer goes away (read EOF/error, or a failed write). For the process-side end of a relay, a pipe or the pty master, that hands the process an EOF, SIGPIPE or SIGHUP just because the host connection died. Wrap the process-side ends in UnownedIOCloser so a relay only ever closes its socket, and have StandardIO and TerminalIO keep the real handles and close them in close(), when the process is done: - StandardIO: stdout/stderr pipe read ends and the stdin pipe write end. closeStdin() now closes the write end itself. The relay keeps its own duplicate descriptor until it has flushed, so the child still sees EOF after the last buffered byte. - TerminalIO: the stdout relay no longer owns the pty master (stdin already didn't); close() closes it. Normal behavior is unchanged: the host still delivers stdin EOF through the CloseProcessStdin RPC, and output still flushes after the process exits. Two edge cases differ. A host that closes its stdin socket without the RPC no longer gives the child EOF, and one that closes a stdout or stderr socket early no longer kills the process with SIGPIPE (it blocks once the pipe fills). This is groundwork for detaching a process's stdio from the host before a VM is paused, and reattaching afterwards. The runc path (RuncIO) is not changed.
…ocess Add IOPair.detach(readUntilEOF:), which stops a relay and returns the bytes that were taken from the source but never reached the sink, so a replacement relay can be built over the same process-side pipe or pty and write them first. This is for detaching a process's stdio from the host before the VM is paused, and reattaching it afterwards. detach unregisters the pair's descriptors from epoll first, so a socket that goes away can no longer trigger a drain or a close. It then closes the pair, which closes the socket end and frees the buffer; an end wrapped in UnownedIOCloser is left open for whoever holds it. Nothing is ever written to the process's pipes: - For a socket carrying stdin (readUntilEOF: true), it reads the socket to a real EOF and returns everything, including what the pair had buffered. The host is expected to have stopped sending. If the source does not reach EOF, errors, or sends more than 16 MiB, detach restores the pair: the bytes go into a new overflow queue, epoll is re-armed, and the error is thrown. drain() writes the overflow before reading anything else, so nothing is lost and ordering is kept. - For a process's output (readUntilEOF: false), it does not read the pipe. Data the process has written stays in the pipe for the next relay; only the pair's own unsent buffer is returned. - A failed detach cannot reopen a stream the host already closed. If the host had closed (or half-closed) its stdin socket, as it must for detach to reach EOF, every byte is still delivered, but the relay then reads EOF and closes the process's stdin, so the process sees end-of- input even though the detach failed. If the host had only stopped sending, the socket is still open and the relay carries on normally. To support re-arming, relay() is split into descriptor duplication and a private arm(), and ignoreHup is kept in the pair's state. With an empty overflow, which is the normal case, drain behaves as before. Nothing calls detach yet. StandardIO/TerminalIO wiring, the vminitd RPCs and the host side come separately.
Take a process's stdio relays down without disturbing the process, so its virtual machine can be paused and the relays rebuilt afterwards. Each class detaches its IOPairs and keeps what they could not deliver in its state (stdinPending, stdoutPending, stderrPending), oldest first. The process side of every relay is untouched: StandardIO keeps its pipe ends, TerminalIO keeps the pty master open, so the process is never given an EOF, SIGPIPE or SIGHUP. Output it writes waits in the pipe or pty, and the process blocks only if it fills one. stdin is detached first, because it is the one that can fail: it has to read the host's socket to EOF. If it throws, no other relay has been touched and all of them carry on as before. TerminalIO also drops its references to the raw sockets that the pairs close, so close() does not close them a second time. The host is expected to close its stdin socket before asking for a detach, and to keep reading stdout and stderr until EOF, since each relay closes its own socket here. IOPair.detach now returns no bytes for a relay that already ended (its socket went away, or the process closed its output) instead of throwing, so one finished stream does not fail the whole detach. A relay whose relay() was never called still throws. Nothing calls these yet. The reattach side, the vminitd RPCs and the host side come separately.
Connect new relays to the host for a process whose relays were taken down by detach(), so a process's stdio can be reattached after its VM has been paused or restored. reattach(stdio:) takes the new vsock ports the host is listening on, connects a socket to each, and builds new IOPairs over the process-side pipe ends (StandardIO) or pty master (TerminalIO) that detach() left open. Everything that can fail on the host's side, the connects, happens before any relay moves bytes. It refuses to run while relays are still attached, and for a stream the process was not started with. TerminalIO also rejects a stderr port, since a terminal has none. If it fails, nothing is attached and the pending bytes from detach() are kept, except if a later relay fails after an earlier one has taken its bytes. What detach() could not deliver is written first. IOPair takes the pending bytes at construction and puts them in its overflow queue, which drain() writes to the sink before reading anything new: stdin bytes go to the process's pipe or pty, output bytes to the new socket. relay() now drains once after arming when there is anything queued, because epoll is edge-triggered and the "writable" edge may have come and gone. Nothing calls detach() or reattach() yet. The vminitd RPCs and the host side come separately.
Add the guest and agent-client sides of taking a process's stdio relays down and bringing them back, so a VM can be paused with its processes running and their stdio reconnected afterwards. Protocol: DetachProcessIO(id, containerID) and AttachProcessIO(id, containerID, stdin?, stdout?, stderr?). Attach carries the new vsock ports the host is listening on, optional per stream like CreateProcess. The generated SandboxContext sources are regenerated with make protos. vminitd: Server handlers follow closeProcessStdin and go through ManagedContainer to the process. ContainerProcess gains detachIO() and attachIO(stdio:); ManagedProcess forwards to the new detach() and reattach(stdio:) on its IO (StandardIO, TerminalIO), using its own terminal flag instead of the request's, since that was decided when the process was created. The runc-backed processes (RuncProcess, RuncExecProcess) throw .unsupported. Host: Vminitd.detachProcessIO and attachProcessIO, added to the VirtualMachineAgent protocol with .unsupported defaults like closeProcessStdin. For a detach the host must have closed its stdin connection first, and must read stdout and stderr until EOF, since each relay closes its own socket. Nothing on the host calls these yet; LinuxProcess and LinuxContainer come separately.
Use the new DetachProcessIO and AttachProcessIO RPCs from the host so a checkpoint no longer catches the process's stdio connections in use, and the container carries on with them afterwards. LinuxProcess gains detachIO() and attachIO(): - detachIO() cancels the stdin relay, which a cancelled relay does without sending CloseProcessStdin, so the process gets no EOF. It closes the host's stdin connection, which is the EOF the guest reads stdin up to, sends the RPC, then reads stdout and stderr to EOF before closing them. The guest closes its own end of those, so everything it sent is read first. Unlike waitIoComplete, running out of time there is an error, because it means output may be unread. - attachIO() listens on the process's existing ports and, as start() does, runs the guest's dial-back wait next to the RPC request. It then restarts the stdin relay and stores the new handles. Reusing the ports is fine: the listeners were finished when the guest connected. LinuxContainer.checkpoint() now detaches the init process's stdio before pausing and attaches it again after resuming, whether or not the save worked, so a failed checkpoint still leaves the container with its stdio. If the detach fails, the process's stdin is already gone, so checkpoint gives up and throws. Known gaps: stdin typed between the detach and the attach is lost, since Terminal's stream keeps yielding to a stream nobody reads once its relay is cancelled. create(from:) still ends in the created state with no LinuxProcess for the running init process, so reattaching after a restore is not wired up yet.
LinuxContainer.create(from:) restored the virtual machine but ended in the created state, with no handle on the init process that is already running in the restored guest. Finish the restore so the container is usable like a started one. After vm.restore it now dials a fresh agent connection, builds a LinuxProcess for the init process the way start() does (same stdio setup from config.process and the same generated spec, which is only stored, since the process already exists), connects its stdio with attachIO(), and lands in the started state. start() must not be called afterwards. wait(), resize(), kill() and stop() then work as usual. A failure closes the agent, stops the relays and the VM, and marks the container errored, as the other create paths do. Not covered yet: the root filesystem and network state a checkpoint depends on are still the caller's to preserve, host unix socket relays (config.sockets) are not re-established, LinuxProcess.pid stays -1 for a restored process, and exec'd processes still block a checkpoint.
Cover Checkpoint save/load and VZVirtualMachineConfigurationSnapshot round-tripping, difference detection, and rejection of unsupported devices.
VZVirtualMachineInstance.init(checkpoint:) passed the bundle directory to VZVirtualMachineConfigurationSnapshot.load, so every restore failed with 'Is a directory'. Load configuration.json inside it, as restore(from:) does.
Cover stdio detach/reattach across repeated checkpoints, the checkpoint bundle contents and whilePaused behavior, a real save and restore into a new container, and failure on a missing checkpoint.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds checkpointing for a LinuxContainer on the Virtualization.framework backend. A running container can be paused, written to a directory, and later restored into a new VM with its init process still running.
Usage
After create(from:) the container is already started, so don't call start().
What's in it
Limitations