Skip to content

Investigate Linux x64 Vulkan LoRA mismatches after Ubuntu 26.04 toolchain drift #37

Open

Description

@leehack

Linux x64 Vulkan qualification began failing after the floating runner moved from Ubuntu 24.04 to 26.04. PR #36 pins qualification and release builders to Ubuntu 24.04 and passes all 17 checks; this issue retains the unresolved numerical/toolchain investigation.

Evidence

  • Passing baseline: run 40550578566, Ubuntu 24.04 image 20261012.334.1, Mesa 25.2.8/LLVM 20.1.2, Vulkan 1.3.275, glslc 2023.8.
  • Failing PR Reject unsafe VRAM-budget loads before context creation #35: job 125305159695, Ubuntu 26.04 image 20261025.179.1, Mesa 26.0.8/LLVM 21.1.8, Vulkan 1.4.341, glslc 2026.1. The software GPU newly reports KHR_coopmat.
  • Twelve F16/BF16 linear and linear-FP8 LoRA immediate/auto comparisons against at_runtime fail with MAD around 1.37 against the unchanged 0.5 threshold. Build/export/device checks pass. These probes do not set max_vram, so PR Reject unsafe VRAM-budget loads before context creation #35's new rejection branch is bypassed.
  • Controlled Linux arm64 baseline/candidate builds on Ubuntu 24.04 produce identical images. The same candidate binary on Ubuntu 26.04 also matches, MAD 0.11637. Those controls lack cooperative matrices and do not clear the failing x64 path.
  • PR #36 run passes the pinned Ubuntu 24.04 x64 Vulkan comparisons. No threshold or cooperative-matrix setting was disabled.

Acceptance

Separate source, compiler/SPIR-V and runtime/driver effects with exact same-source x64 controls; isolate the failing operation/path before assigning upstream or driver causality. Keep the numerical threshold and positive controls. Qualify an eventual Ubuntu/toolchain upgrade before changing the pinned producer and qualification environments together.

No PR-caused runtime regression or hardware-GPU failure is established. Failed-run artifacts were not uploaded, so no GLIBC import-floor claim is made. Related owner qualification umbrella: #20; release tracking: leehack/llamadart#1703.

Activity

  1. leehack commented on Nov 3, 2026

    @leehack
    OwnerAuthor

    Triage: bug / P2 — bounded software-Vulkan numerical/toolchain qualification failure. The recorded Ubuntu 24.04 builder pin is mitigation, not root-cause resolution. Preserve thresholds and separate same-source compiler/SPIR-V/runtime controls before assigning causality. This is not established as a PR-caused runtime regression or hardware-GPU failure; no new numerical run was performed by this scan.

  2. added
    bugSomething isn't working
    priority:P2Important improvement or regression; schedule next
    on Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority:P2Important improvement or regression; schedule next

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions