Skip to content

fix(cloudhypervisor): balloon settle counts progress only past 1% of guest memory per stall window - #330

Merged
CMGS merged 1 commit into
masterfrom
fix/balloon-settle-min-step
Nov 9, 2026
Merged

CMGS merged 1 commit into
masterfrom
fix/balloon-settle-min-step

Conversation

@CMGS

@CMGS CMGS commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

waitBalloonSettled (added in #276) reset its 2 s stall window on any drop in memory_actual_size, even a single page. A guest whose anon sits above the balloon target can only trickle pages to the balloon, so the pause ran on until the one-minute cap.

Change

  • Within a stall window, the balloon must free at least 1% of guest memory, counted from the last progress mark, to count as progress. Otherwise the pause goes ahead with the existing warning.
  • The minute cap stays.
  • docs/snapshots.md describes the new stall rule.
  • TestSnapshotPauseWaitsForBalloonToSettle changes:
    • New case "trickling inflation counts as stalled": 4 KiB per poll now waits the 2 s stall window. Against master it waits 1m0s.
    • The cap case now inflates steadily above the step (1/1000 of memory per poll) toward a balloon of 3/4 of memory, so it still hits the minute cap.

Evidence (hardware, through sandboxd's idle hibernate, cocoon 2afbc27, CH 750aa2b)

Guest: 8G, holding 5000 MiB of anon, balloon target 50%.

Arm Hibernate Exec on that VM
Balloon on 14.7 s, 59.7 s (cap) 15.8 s, 60.3 s
Balloon pre-inflated by the idle sweep 29.6 s 31.2 s
Balloon off 3.9 s, 3.8 s

The host RSS of that guest did not fall in any balloon arm, so the wait gained nothing. #276's own case, a golden settling right after boot, freed about 280 MB in under 2 s on an 8 GiB guest. That is well above the 82 MiB step, so the step does not cut it short.

Hardware run on this branch: pending. It will cover the anon-heavy hibernate time with master vs this branch, and a golden saved right after boot still reaching the balloon target. Results will be posted here.

Gates

  • make lint: 0 issues. linux + darwin.
  • make fmt-check: rc 0.
  • asl -forwarder=false ./...: clean on both GOOS.
  • go test -race ./hypervisor/...: ok.

…guest memory per stall window

Any drop in memory_actual_size reset the 2 s stall window, so a guest whose
anon sat above the balloon target trickled pages and held the pause for up
to the one-minute cap. A hibernate of such a guest took 14.7-59.7 s against
3.8-3.9 s without the balloon, and every exec on that VM waited it out. A
settling golden frees far more than 1% per window and still reaches its
target.
@CMGS

CMGS commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Scope note: migrateSpec (hypervisor/cloudhypervisor/migrate.go:32-47) reuses snapshotSpec's Pause, so every vm migrate capture round also runs waitBalloonSettled. Before this change, a live move of an anon-heavy guest whose balloon was raised paid up to 60 s per round. The final round is the one sandboxd holds the claim across. This PR bounds that path the same way. The hardware run on this PR covers hibernate and snapshot; the migrate rounds share the same function, and no separate move run is planned.

@CMGS

CMGS commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Hardware verification: pass

Setup. Testbed host, kernel 6.18 guest, CH 750aa2b, sandboxd e38c762a. The arms swap only the cocoon binary: A = 2afbc27, B = 7b71b0f. Each run logged the /proc/exe of the hibernate child, and it matched the arm every time.

1. Anon-heavy idle hibernate. 8G guest holding 5000 MiB of anon, balloon at 50%. Runs in A B B A order, twice, so n=4 per arm. Times are the hibernate child / an exec fired on the claim at hibernate start.

Arm ms
A 37735 / 39488, 13248 / 14229, 23531 / 25270, 12559 / 14162
B 4408 / 5274, 4526 / 5492, 4918 / 7846, 4586 / 8062
Balloon off (base) A 2456 / 2586, B 2395 / 2555
  • B is the base plus 2.0–2.5 s.
  • An exec on another claim took 1–3 ms in every run.
  • memory_actual_size on arm A, sampled every 100 ms:
    • In the first 0.2 s it falls from 6144 to about 5430 MiB.
    • After that it sawtooths: it climbs in 4–5 MiB steps, then falls back 20–30 MiB. That gave 156 changes in 23 s, and the drop per 2 s window has a median of 0.
    • Every fall reset the old stall window.
  • B never reaches the 81 MiB step after the first drop, so it stops 2 s later. It captures the same final level as A: 5435/5427 against 5429/5421 MiB.

2. #276 guard. Snapshot taken the moment vm exec first answers, A B B A in both variants.

Variant Target Result on both arms
Boot balloon 25% 6144 Two runs were still inflating at exec. All paused at 6144, no "with balloon at" warning, and the restored clone read 6144.
vm balloon 75% immediately before the save 2048 Inflation ran from about 6000 to 2048 during the save, and every run paused at 2048 with no warning.

3. Page-cache guest. Hibernate took 2145 ms on A and 2146 ms on B, and both reached the target in 0.3–0.4 s.

@CMGS
CMGS merged commit bdf84b5 into master Nov 9, 2026
4 checks passed
@CMGS
CMGS deleted the fix/balloon-settle-min-step branch November 9, 2026 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant