Skip to content

Keep KVM sandbox clocks synchronized with the host - #379

Open
duynguyen158 wants to merge 1 commit into
LibreChat-AI:mainfrom
duynguyen158:fix/kvm-guest-clock-sync
Open

duynguyen158 wants to merge 1 commit into
LibreChat-AI:mainfrom
duynguyen158:fix/kvm-guest-clock-sync

Conversation

@duynguyen158

@duynguyen158 duynguyen158 commented Nov 6, 2026 •

Copy link
Copy Markdown

Summary

Fixes #378. Long-running KVM guests can fall behind the signing worker and reject fresh execution manifests as not_yet_valid. Restarting resets the difference, but does not correct the clock's ongoing rate. This adds continuous guest time discipline using the existing KVM virtual PTP host-clock reference, rather than widening the manifest tolerance or scheduling restarts.

  • Start chrony in the trusted KVM bootstrap, wait for synchronization, then supervise the clock daemon and selected workload together.
  • Keep clock control in a root-only guest-private directory; disable network NTP listeners and leave host capabilities and user-job clock restrictions unchanged.
  • Install the dependency and helper consistently in all three existing guest image constructors.
  • Add focused lifecycle regression coverage, retain the DNS handoff checks, and document the KVM requirement.

The helper, configuration and regression test are added under the existing api/src, api/config and tests directories. The bootstrap/image/CI layout follows the guest resolver work in #152; no new directories, deployment infrastructure, release workflow or version changes are introduced.

Change Type

  • Bug fix
  • Breaking startup requirement for KVM: the guest must expose a usable UTC host-referenced /dev/ptp0. Startup fails closed if synchronization is unavailable. Direct NsJail, local OCI/macOS and Lambda startup remain unchanged.

Testing

  • Passed shell syntax checks for the clock helper, guest DNS bootstrap and their shell tests.
  • Passed bash tests/guest_clock_lifecycle.sh, bash tests/kvm_guest_dns.sh, bash tests/sandbox_runner_healthcheck.sh and bash tests/block_root_package_delivery.sh together in Linux; the lifecycle suite also passed on Debian Bash 5.2.
  • Passed docker buildx build --check for api/Dockerfile, docker/Dockerfile.worker-sandbox and launcher/Dockerfile, with no warnings.
  • Built the actual amd64 baked runner and exercised a read-only-root KVM guest: signed Bash succeeded, an unsigned request remained HTTP 401, and a user-job clock syscall remained blocked with SIGSYS.
  • Verified outer-workload TERM cleanup and workload shutdown when chronyd exits, including exit 0. In a 126-second clock observation, the final minute's maximum absolute host-reference offset was 26.604 microseconds. This is a bounded smoke test, not a multi-day soak claim.

Test Configuration

  • Linux/amd64 KVM with the baked, read-only ext4 guest root.
  • Linux shell regression environments with Bash 5.2 and 5.3.
  • No production image promotion is claimed. The current CI and Images runs require maintainer approval for this fork contribution.

Checklist

  • Changes follow the existing guest-bootstrap, configuration and test locations
  • I have performed a self-review and independent adversarial reviews
  • Complex lifecycle and security-sensitive behavior is commented
  • Regression tests cover startup gating, failures, exit status and bounded cleanup
  • Local tests pass with these changes
  • Upstream CI and Images have passed (awaiting maintainer approval)

Maintainer integration note: per CONTRIBUTING.md, this targets public main and leaves the internal bridge/review process to maintainers. The new public CI step also needs its internal ci-codeapi.yml counterpart kept in sync; that internal file is not present here and has not been invented or modified by this contribution.

@duynguyen158 duynguyen158 changed the title Keep sandbox clocks synchronized so code execution stays available Keep KVM sandbox clocks synchronized with the host Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Continuously synchronize Linux KVM guest time with the host

1 participant