Skip to content

Add a memory balloon to VZ virtual machines - #1000

Open
JaewonHur wants to merge 3 commits into
apple:mainfrom
JaewonHur:virtio-balloon
Open

JaewonHur wants to merge 3 commits into
apple:mainfrom
JaewonHur:virtio-balloon

Conversation

@JaewonHur

Copy link
Copy Markdown
Contributor

Attach a virtio memory balloon device to every VZVirtualMachineInstance and add setTargetVirtualMachineMemory(size:) so a host can ask a running guest to give memory back, or take it back.

  • The device is added by default. It is inert until the target is lowered, and a guest without the virtio-balloon driver ignores it.
  • The method requires a running VM and throws .invalidArgument rather than relying on Virtualization.framework's silent clamping. The size must be a multiple of 1 MiB, at least minimumAllowedMemorySize, and at most the configured memory.
  • It is reached through withVirtualMachineInstance, not LinuxContainer or LinuxPod, since it is a low-level VMM operation. The library does no estimation or policy.
  • The target is a request. The guest may honor it partly or never, and the actual balloon size can't be read back.

Attach a VZVirtioTraditionalMemoryBalloonDeviceConfiguration to every
VZVirtualMachineConfiguration built by VZVirtualMachineInstance. The
device is inert until the host lowers its targetVirtualMachineMemorySize,
which starts at the configured memory size, so existing VMs behave the
same. A guest kernel without the virtio-balloon driver ignores it; the
repo's arm64 and x86_64 kernels build it in.

This only makes the device available. A follow-up adds an API to set the
target size. Also add docs/memory-balloon.md describing the design:
balloon semantics, how Virtualization.framework exposes it, and why the
target is a request rather than a guarantee.

VZ requires the configuration to match when restoring saved machine
state, so state saved before this change will not restore.
Expose the virtio memory balloon's target memory size so callers can ask
a running guest to give memory back to the host, or take it back. It is
reached through withVirtualMachineInstance rather than LinuxContainer or
LinuxPod, since it is a low-level VMM operation.

The method requires a running VM and validates the size instead of
relying on Virtualization.framework, which silently rounds and clamps:
the size must be a multiple of 1 MiB, at least
VZVirtualMachineConfiguration.minimumAllowedMemorySize, and at most the
configured memory. It throws if the VM has no balloon device. The target
is a request; the guest may honor it partly or never, and the actual
balloon size cannot be read back.

The configured memory rounding is factored into
Configuration.vzMemorySize so toVZ and the validation share it. Add unit
tests for the range checks and the rounding.
Boot a VM, check that the guest bound a driver to the default balloon
device, and fill the page cache with a 128 MB file so the container's
memory usage rises. Then lower the balloon's target memory size to an
eighth of the VM's memory and check, through container.statistics, that
usage falls below 3/4 of its earlier value as the guest reclaims the
cache to feed the balloon. Without the balloon call, usage stays flat.

Also check that setTargetVirtualMachineMemory(size:) rejects a size above
the configured memory, a size that is not a multiple of 1 MiB, and zero.

The test is macOS-only, as the balloon is a VZ feature.
@zackarychapple

Copy link
Copy Markdown

Host-side measurement of this PR on macOS 26.6.2

I built this branch (5ed4f6e) and measured whether an inflate returns memory to macOS. The PR's integration test checks the guest's cgroup usage, so I measured the host side of the same operation.

Setup: Apple M5 Max, macOS 26.6.2 (Xcode 27.2, Swift 6.4.2). vminit:0.53.0 served as the init image (the branch is 2 commits past 0.53.0, with no vminitd or proto changes), plus the 6.18.35 kernel that container 1.6.1 installs. A local change to cctl run called setTargetVirtualMachineMemory(size:) on a timer. cctl run --memory 8000 --cpus 4 ran debian:stable-slim. The guest held about 2 GiB of anonymous memory and exited, then wrote and deleted a 3 GiB file. A host script read the VM process's phys_footprint (footprint -p) every 5 s.

Event Guest MemAvailable Host phys_footprint
Boot — 876 MB
Guest memory and file cache freed 7.6 GB 3737 MB
Target 1024 MiB 658 MB (within 5 s) 3740 MB
Target 8128 MiB 7.9 GB (within 5 s) 3742 MB

A second run compared vmmap -summary of the VM process before and 30 s after an inflate to 1024 MiB. Physical footprint stayed at 2.6 GB, RSS at 4.93 GB, and dirty memory at 2.6 GB.

Result: the guest side works as the PR describes. The driver inflates and deflates within seconds. On macOS 26.6.2, though, the inflate does not lower the VM process's footprint, RSS, or dirty memory. This agrees with the note in #65 that ballooning only partially works on macOS. I did not test whether macOS reclaims ballooned pages faster under host memory pressure.

Two small notes:

  • cctl run --memory 8000 creates a VM of 8128 MiB (with guestMemoryOverhead). A target of 8192 MiB is then rejected with invalidArgument, which is correct, but callers need the VM size, not the container limit.
  • cctl run needs a pty. script -q /dev/null cctl run … < /dev/null worked for unattended runs.

@JaewonHur

Copy link
Copy Markdown
Contributor Author

Yeah. Seems we have some issue in the host side VM memory management. Will figure out!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants