Skip to content

CI qemu lane boots stale fixtures, not Linux 6.18 #491

Open

Description

@xalestar

The qemu-aarch64 lane in CI appears to boot a kernel older than Linux 6.17, although tests/fetch-fixtures.sh has pinned 6.18 since 1442b3c and elfuse reports 6.18 to the guest. Every oracle comparison in that lane is then made against the older kernel.

How the fixtures go stale

  • build.yml restores externals/test-fixtures from $HOME/.cache/elfuse-ci/test-fixtures on the runner, and saves it back after the Release job.
  • ensure_fixtures in tests/test-matrix.sh and qemu_ensure_fixtures in tests/qemu-runner.sh run fetch-fixtures.sh only when the kernel, initramfs, busybox or key file is missing.
  • With the cache restored, those files exist, so fetch-fixtures.sh never runs and its versions.lock check, which would rebuild the fixtures for the new pin, never fires. The cache is saved back unchanged, so it stays at whatever kernel it held when it was first built.

Evidence

On #485, commit 7d91b6c checked the MREMAP_FIXED order that Linux 6.17 introduced (a85dc37186a5, growth checked before the destination is unmapped). Steps 8 and 12 of test-rlimit-mem passed on a local VM booting 6.18.60, and failed in the CI qemu lane with FAIL: step 8 and FAIL: step 12 (run 42371740357, Runtime (Release)). A local VM booting 6.12.115 fails the same two steps and no others. The CI log does not print the VM's uname -r, so the exact version there is not confirmed.

Possible fixes

  • Run fetch-fixtures.sh unconditionally after the restore step in build.yml, so the lock check decides whether to rebuild (it fetches the APKINDEX each run and downloads only apks it does not already cache, but it also picks up any new patch release on the mirror, so the oracle would move with Alpine).
  • Or key the cache on the pinned versions, so a pin change starts from an empty cache.
  • Either way, printing uname -r from the VM at the start of the qemu lane would make the oracle version visible in the log.

Clearing $HOME/.cache/elfuse-ci/test-fixtures on the runner once would fix the current state but not the next pin change.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions