Repository navigation
Continuously synchronize Linux KVM guest time with the host #378
Description
Problem
Long-running Linux/libkrun sandboxes can reject fresh signed execution manifests with not_yet_valid even while HTTP health remains healthy. Recycling the runner resets the clock but does not correct its ongoing rate error; the detection/recycling improvements in #39 and #81 do not provide guest time discipline.
Impact
- Ordinary code execution and code-based tool workflows stop after the guest clock falls outside the existing 30-second admission window.
- Recovery by restart is disruptive and temporary.
Root cause
Read-only diagnostics in a private, disposable microVM using runner sha-68c9f931b67fea8d8d96eb64b869dc6d70d04466 / libkrun1.19.0 / guest6.12.91 found an undisciplined guest clock (TSC; tick10000us, frequency0, STA_UNSYNC) while the immediate host applies approximately 88.240ppm combined tick/frequency correction. Guest lag measured against native KVM virtual PTP grew 88.237ppm over57seconds, approximately7.6seconds/day. /dev/ptp0 is already available and readable; no new time-transport or public NTP endpoint is required. This does not identify a particular hardware/libkrun defect.
Fix
Use standard chrony inside the trusted Linux KVM guest against its native host-referenced PTP clock. Synchronize before accepting work and supervise clock-daemon lifetime together with the guest API. Keep host capabilities, user-job clock syscall restrictions, network confinement and the manifest window unchanged. Leave direct NsJail/Lambda startup untouched.
Verification
The unsynchronized rate and native PTP source have been measured in an actual isolated KVM microVM. Implementation, bounded startup/failure regression coverage, actual synchronization/confinement smoke, and image build proof are in progress. Refs #37; existing detection is still useful defense in depth.
Activity
duynguyen158 commentedon Nov 6, 2026
A second disposable guest ruled out a clocksource-only fix: switching its active clocksource from
tscto the already availablekvm-clockstill produced 88.204 ppm lag growth against native KVM PTP over57seconds (versus88.237ppm with TSC). Continuous host-reference discipline is necessary on this measured setup; a clocksource selector alone does not remove the rate error. The isolated guest was removed; no running service clocksource was changed.duynguyen158 commentedon Nov 6, 2026
Implemented in #379 and verified with an actual amd64, read-only baked KVM runner, not only lifecycle mocks.
adjtimexquery remained blocked with SIGSYS/159.api/Dockerfilebaked runner; Docker checks for all three guest constructors reported no warnings. Clock lifecycle, DNS handoff, skew healthcheck and block-root package delivery suites passed together in Linux. The corrected lifecycle harness also passed on Debian Bash5.2.Operator note: libkrun reported container exit0 when stopping the VM after guest clock loss; keep the existing Compose
restart: alwaysbehavior rather than assumingon-failurealone will recover it. CI and Images currently require an upstream maintainer to approve the fork workflow runs. No permanent deployment or published-image promotion is claimed.