Skip to content

Support vLLM 0.31 and 0.32, with per-line GPU evidence - #151

Open
ItzikVa wants to merge 36 commits into
mainfrom
feature/vllm-0.26-to-0.32
Open

Support vLLM 0.31 and 0.32, with per-line GPU evidence#151
ItzikVa wants to merge 36 commits into
mainfrom
feature/vllm-0.26-to-0.32

Conversation

@ItzikVa

@ItzikVa ItzikVa commented Oct 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Ready. Every gate below has run and passed on each added line. Each cell is
written from an actual run, with its run link, or left as — with the reason;
nothing is ticked by inference.

This PR targets main directly, and the upgrade-vllm skill is not in it.
The skill is internal tooling and now lives in
generative-computing/granite-switch-internal; the final commit here removes it
from the public tree, along with the docs/UPGRADE_VLLM_PIPELINE.html page that
documented it.

What is in this PR is the port itself plus the test infrastructure it needed:
tests/eval/ (the drift gate, its three committed baselines and its prompt set)
and the eval / eval-record arms of gpu-tests.yaml. Those are repository test
infrastructure — a contributor needs them to reproduce or re-record a baseline —
and they do not depend on the skill, which only dispatched them. Every drift-eval
result in the tables below came from that workflow, so no evidence here rests on
code the last commit removed.

Supported vLLM range

before after
[vllm] extra >=0.26.0,<0.31.0 >=0.26.0,<0.33.0
lines claimed 0.26 … 0.30 0.26 … 0.32
CI GPU legs vllm26 / vllm30 vllm26 / vllm32

New lines in this PR: 0.31 and 0.32.

Support dropped

None — the range only widened. Every line previously claimed is still claimed
and still resolvable. The floor stays at 0.26: nothing the port now requires is
missing there, so no [vllm] lower bound moved and no downstream pin has to.

No dependency group was removed either. vllm27 / dev-vllm27 and
vllm31 / dev-vllm31 are all still declared even though dev-latest does not
point at them — a declared group is a GPU leg some commit can name, and
derive_legs.py resolves groups against the tested commit, so deleting one
makes that leg's uv sync --group X fail before pytest starts (REFERENCE.md §4).

Dependency ceilings touched

  • transformers: unchanged at >=5.16,<=5.17.0. PyPI metadata asks for
    >=5.10.4,<5.18.0 (0.31) and >=5.16.1,<5.20.0 (0.32); our existing window
    sits inside both, and uv resolved 5.17.0 for every leg.
  • torch: unchanged at >=2.11.0. All of 0.30 / 0.31 / 0.32 pin
    torch==2.13.0, which satisfies it.
  • tokenizers: not declared here; 0.32 raises its own floor to >=0.23.2,
    resolved transitively.

So this upgrade is a pyproject-and-one-source-file change, which is the intended
shape — but only because the one source file was found. See below.

What upstream changed, and how it is handled

BaseMultiModalProcessor._call_hf_processor — reintroduced with a new contract

Upstream: 0.29 deleted the hook (REFERENCE.md §3 records this as the canonical
worst case for this repo). 0.31 brought it back — with a different signature:
(hf_data, hf_kwargs) returning the modality tensors only, where the 0.26-0.28
hook took (prompt, mm_data, mm_kwargs, tokenization_kwargs) and returned
input_ids as well. 0.31 also replaced _get_hf_mm_data with the
_get_hf_mm_inputs / _finalize_hf_mm_data pair, and renamed
_apply_hf_processor_main's hf_processor_mm_kwargs kwarg to hf_kwargs.

There are therefore three eras of this surface, not two:

era lines shape
A 0.26-0.28 _call_hf_processor(prompt, mm_data, mm_kwargs, tok_kwargs) → ids + tensors
B 0.29-0.30 hook absent; _apply_hf_processor_main is the only seam
C 0.31-0.32 _call_hf_processor(hf_data, hf_kwargs) → tensors only

Here: our gate was hasattr(super(), "_call_hf_processor") with a comment
reading # 0.26 - 0.28. That predicate is true again on 0.31+, so eras A and
C collapsed onto one branch — the branch written for A's signature. The era-B
path reads kwargs["hf_processor_mm_kwargs"], which 0.32 no longer passes, so a
misread era is a KeyError on the audio path at best. At worst it is the 0.29
failure all over again: an override that is never called, no transcript, and
every test still green.

Fix: _base_hook_era() gates on the hook's shape rather than its
existence — it inspects the first non-self parameter (prompt ⇒ era A,
anything else ⇒ era C) and falls back to probing for _get_hf_mm_inputs when the
hook is not introspectable. Still a capability check: no version number appears
anywhere in src/, and a line shipping a fourth shape gets classified by what it
looks like rather than by what it is numbered. hasattr structurally cannot do
this, because A and C both have the attribute.

Test: tests/unit/test_audio_processor_hook_era.py, 6 cases. It asserts the
effect — which era self-extracts — not that a call happened, because a
never-called override is precisely the failure being guarded. Two cases exist
only because hasattr passes them: test_era_a_and_c_are_distinguished and
test_only_era_b_self_extracts. Negative control: reverting _base_hook_era to
hasattr semantics fails 3 of the 6.

Bounding the port: the processor's other overrides
(_get_mm_fields_config, _get_prompt_updates, _postprocess_prompt,
_apply_prompt_updates) are unchanged between 0.30 and 0.32, and the other
load-bearing surfaces from REFERENCE.md §3 — the MoE topk is_padding mask,
AttentionBackend KV-cache shape helpers, vllm.Attention, LLM/EngineArgs
kwargs, the vllm.general_plugins entry point — show no deletion, rename or
signature narrowing across 0.30 → 0.32.

That bound held for src/, but not for the test harnesses, and the next
finding is the one the static pass could not have produced.

FlashAttentionMetadata — cascade attention removed at 0.32

Upstream: e1a508625 ("[Attention] Remove cascade attention", vllm#61126)
removes cascade attention outright at 0.32, and with it five required,
no-default fields of FlashAttentionMetadata: use_cascade,
common_prefix_len, cu_prefix_query_lens, prefix_kv_lens, suffix_kv_lens.

Here: five GPU harnesses build that dataclass by hand — they drive a real
FlashAttention kernel rather than a stub — and each passed all five fields to
mean "no cascade, one plain prefill". On 0.32 every construction raises
TypeError: FlashAttentionMetadata.__init__() got an unexpected keyword argument 'use_cascade'. The worker reports that as vLLM multi worker cannot start, so
it lands as 67 failing tests across tests/vllm/ and tests/integration/
rather than as one error — worth stating, because the symptom reads like a broad
model regression and is in fact one deleted kwarg.

Dropping the five fields is equally wrong in the other direction: on 0.26-0.31
they have no defaults, so omitting them is the same TypeError.

Fix: cascade_kwargs() in tests/shared/vllm_kv_cache.py — already the home
for "build real FlashAttention inputs on any supported line" — returns the five
fields when the dataclass still declares them and {} when it does not; the five
sites splat it. The gate is FlashAttentionMetadata.__dataclass_fields__, i.e.
the shape of the API, not vllm.__version__. No behaviour changes on any
line: the values passed were exactly "cascade off", which is the only mode 0.32
has, so a harness that omits them there gets the attention it always asked for.

How it was attributed: the vllm31 leg of the same dispatch is unaffected,
and the 0.26 floor leg shows none of these 67 failures — which is what identifies
this as the 0.32 removal rather than a change of ours.

Why Phase 3 did not predict it, and what that says about the detector.
upstream_api_diff.py reports symbols that were deleted, renamed, or had their
signature narrowed. FlashAttentionMetadata suffered none of those: the class is
still there, still constructible, and its __init__ is generated — the breaking
change is the field list of a dataclass, which no symbol-level diff sees.
This is a known gap rather than a bug to fix in passing, and it is why the GPU
legs are the gate and the static pass is only a map.

Not a vLLM finding: the CTC checkpoint was re-uploaded mid-flight

Disclosed because it is a change in this PR that a reviewer would otherwise read
as a test being weakened.

ibm-granite/granite-speech-5.0-470m-turboctc was re-uploaded on 2026-10-21
("Upload 13 files", "Remove legacy custom-code files") and now truecases and
punctuates: the committed clip returns "What is the capital of Israel?" where
it used to return "what is the capital of israel".
ASRTranscriber.transcribe() pins no revision, so every runner picked the new
weights up at once and all four transcript assertions in
tests/integration/test_asr_ctc_default_gpu.py failed together.

This is not the port. The four failures are identical on the vllm26 floor
leg and on vllm31 and vllm32; uv.lock resolves the same transformers 5.17.0
and torch 2.11.0 as the base branch; and neither asr.py nor that test is
touched by this branch. The cross-leg identity is the tell — a version-specific
break cannot be identical on the floor.

Fix: a _words() normalizer (lowercase, strip non-alphanumerics), with the
assertions comparing word lists. Casing and terminal punctuation are the ASR
model's editorial choice, not a contract this repo owns, and were never what
these tests were for: they exist to show that audio routes through the CTC
backend (no generate(), decode kwargs dropped), that the dtype override still
transcribes, and that the chunked path neither drops nor duplicates words at the
seams. The word sequence is still exact, so a transcript of
"what is the capital of france" fails exactly as before, and the chunked case
still requires the final word to survive the merge — the assertion that actually
catches a seam bug. Pinning a revision was rejected: it would freeze the test
onto weights a served deployment would not be using.

The detector that was supposed to find all of this reported nothing

Worth its own section, because it is the reason the above was nearly missed and
it ships as its own commit.

Here: upstream_api_diff.py builds regexes and hands them to git grep -E,
which is POSIX ERE. POSIX ERE has no \s / \w / \d, and git grep does
not error on them — it silently matches nothing. Both patterns used \s, so
every symbol reported zero definitions at every tag, every comparison was 0-to-0,
and the report concluded "nothing moved" for any pair of tags. That is worse than
useless: a clean report is exactly what Phase 3 reads as "no port work needed".
Any previous upgrade this script cleared was never actually checked.

Fix: [[:space:]] for \s, with a NOTE above each pattern so the next edit
does not reintroduce a Perl-ism.

How it was caught: by refusing to accept an empty result as evidence and
running a positive control with a documented answer. REFERENCE.md §3 records
that 0.29 deleted _call_hf_processor, so v0.28.0 → v0.29.0 must be non-empty.
The script said nothing had changed. After the fix the same invocation reports
_call_hf_processor DELETED (74 → 0).

Per-version coverage

One column per line this PR adds, plus the floor and the previous ceiling as
controls. ✅ means that gate ran and passed on that line; — means it did not
run or does not apply. No cell is ticked by inference from its neighbours.

gate 0.26 (floor) 0.30 (prev ceiling) 0.31 (new) 0.32 (new)
Pre-commit ✅¹ ✅¹ ✅¹ ✅¹
CPU tests (3.11) ✅ 314P/1449S ³ ✅ ³ ✅ ³ ✅ ³
CPU tests (3.12) ✅ 314P/1449S ³ ✅ ³ ✅ ³ ✅ ³
GPU full (unit, composer, hf, vllm, integration) ✅ 2868P/74S ⁴ — ² ✅ 2867P/75S ⁵ ✅ 2867P/75S ⁶
GPU audio ✅ 204P ⁴ — ² ✅ 203P/1S ⁵ ✅ 203P/1S ⁶
Drift eval granite42_vllm ✅ ⁴ — ² ✅ ⁵ ✅ ⁶
Drift eval granite5_vllm ✅ ⁴ — ² ✅ ⁵ ✅ ⁶
Drift eval sr_vllm —⁷ — ² —⁷ —⁷
Drift eval granite42_hf ✅ ⁴ — ² ✅ ⁵ ✅ ⁶

Every leg's own log header states which line it installed — Deps: dev (vllm 0.26.x) on the floor leg, Deps: dev-latest (vllm 0.31.x) / (vllm 0.32.x) on
the new ones — and the vLLM banner inside confirms it (V1 LLM engine (v0.31.0),
(v0.32.0)). That header is the receipt that a column is the line it claims;
without it, two cells could be the same leg reported twice.

On the one-test count difference, stated rather than smoothed over. The new
lines report 2867 passed / 75 skipped where the floor reports 2868 / 74, and 203
passed / 1 skipped where the floor reports 204. It is the same single test in
both suites —
tests/vllm/test_audio_processor.py::TestPromptUpdatesAreApplied::test_uncached_path_reports_updates_not_applied
— which pytest.skips when vLLM has no _apply_hf_processor_text_mm, deleted
upstream at 0.29. Its guard comes from 7da02b0, an ancestor of origin/main,
so the previous ceiling 0.30 skipped it too: this is pre-existing behaviour,
not coverage lost by this port. Mentioned because a differing pass count between
columns is exactly the kind of thing that should be explained rather than left
for a reviewer to wonder about.

¹ Pre-commit ran both locally over all files (uvx pre-commit run --all-files)
and in CI, run
40297204607
(49s, pass). Like the CPU matrix it installs no vLLM, so it does not vary by line
and the same ✅ fills every column.

² 0.30 gets no GPU leg on this branch: derive_legs.py emits at most two legs
per commit, and both commits here spend theirs on the floor and on a new line.
0.30 remains a declared, supported line and is bracketed by two tested lines, but
this PR does not re-prove it — hence —, not ✅.

³ One run covers all four columns, because the CPU matrix installs no vLLM at
all. ci.yaml syncs a CPU-only environment, so there is no per-line CPU leg to
run and the same ✅ is the honest entry in every column — it is not four
independent measurements. Run
40297204607:
314 passed / 1449 skipped on Python 3.11 and 3.12, read from the Run CPU tests step (not the job's step list — "Upload coverage" is
continue-on-error: true and can be red under a green job). Non-vacuous: 314 is
a real pass count, not an all-skip.

This row read — until the skill-removal commit, because the PR previously
targeted feature/vllm-upgrade-skill and both ci.yaml and dco.yaml filter
on: pull_request: branches: [main] — a filter GitHub evaluates against the
base branch, so neither was eligible and the PR showed zero checks. Now that
it targets main the matrix actually ran, and DCO passes with all 13 commits
signed.

Per-line CPU coverage still comes from the GPU legs, which do install vLLM: the
full suite runs tests/unit/ as part of "everything" (1854 unit test lines in
each leg's log), so the unit suite is exercised on each of 0.26 / 0.31 / 0.32
under its own wheel.

⁴ The floor is the lower leg of every dispatch, so it is re-proven six times
over. Cited above from the runs at 806aced (full
40177367456,
audio
40177402752,
eval
40177436576)
and c841131 (full
40177881390,
audio
40177916472,
eval
40177952586).

⁵ 0.31 at sha 806aced, leg vllm31: full
40177367456
(3:43:08), audio
40177402752,
eval
40177436576.

⁶ 0.32 at sha c841131, leg vllm32: full
40177881390
(3:48:11), audio
40177916472,
eval
40177952586.

⁷ sr_vllm is skipped by design on every line, including the floor and the
previous ceiling — not a hole this PR opened. SR adapters are not public, so
there is no compose recipe and no committed baseline (tests/eval/baselines/
holds granite42_hf, granite42_vllm, granite5_vllm only). The suite's own
docstring defines the split: no checkpoint and no recipe ⇒ SKIP, whereas a
checkpoint with no baseline ⇒ FAIL. So 3 passed, 1 skipped is this suite's
fully green state, and the drift rows above are complete rather than partial. The
SR code path is not left unmeasured — the scored granite5_sr / granite5_sr_hf
pair below covers it on both lines.

GPU legs are dispatched with no_cache=true — the test pod pins
vllm/vllm-openai:latest, so a cached pass would answer "what did this commit
do" rather than "is this still true today".

A green run is not proof a gate ran. pytest exits 0 when every test skips,
and the pod then prints ALL GPU TESTS PASSED. gpu_dispatch.sh fails a green
run whose log has no N passed, so the only ✅ in this table is one a real test
earned; a vacuous leg gets — plus a note.

How each column got its own evidence. The GPU legs are not written in
gpu-tests.yaml: /opt/gsw/derive_legs.py (baked into the runner image, not
editable from this repo) reads the tested commit's pyproject.toml and emits
at most two — one from dev (the floor) and one from dev-latest. So rather
than widening a matrix, which this repo cannot do, the PR walks dev-latest up
one line per commit and dispatches at each sha:

commit dev-latest legs produced status
d08fa78 0.31 floor (0.26) + 0.31 superseded — predates the audio fixes
eb83c33 0.32 floor (0.26) + 0.32 superseded — predates the audio fixes
10ba7ae 0.31 floor (0.26) + 0.31 superseded — predates the cascade fix
2040e81 0.32 floor (0.26) + 0.32 superseded — predates the cascade fix
806aced 0.31 floor (0.26) + 0.31 all three suites green — fills the 0.31 column
c841131 0.32 floor (0.26) + 0.32 all three suites green — fills the 0.32 column

The alias is walked three times because each round of fixes landed after the
preceding walk, and derive_legs.py reads the tested commit: a sha can only
evidence the tree as it stood at that sha. Amending an already-dispatched,
already-linked commit is not an option, so the alias walks again and the earlier
shas are retained as history rather than cited as proof. Only 806aced and
c841131 contain every fix in this PR, so only their runs fill the table.

Workflow concurrency is keyed on inputs.sha, so those runs are parallel rather
than sequential, and the floor is re-tested by every one of them. The leg label
in each result is the receipt: two runs reporting the same label would mean the
shas were wrong, not that the lines agree.

The final commit leaves dev-latest on the new ceiling, which is the right
steady state and the only mergeable one — breakage lands at the ends of the range
(an API deleted upstream fails at the ceiling, one that does not exist yet fails
at the floor), and plumb_pyproject.py --check --final refuses the branch while
the alias is parked lower. CI re-checks the ends on every future run; the table
above is what proves the middle.

Scored accuracy evals

3565 answerability rows per config on the Vela cluster, against fixed per-config
accuracy bars and tolerances (held with the job definitions in the internal
repo's upgrade-vllm skill; each config's bar and tolerance are quoted in the
table below, so the numbers here are self-contained). These answer a question the
drift evals cannot: did the model get worse. A model that answers confidently
and wrongly has a perfectly normal top-k distribution.

Reported per added line, against the same unchanged bars — so a reviewer can
see that 0.32 specifically did not regress, rather than that "the evals passed".

config backend bar 0.31 accuracy Δ 0.32 accuracy Δ verdict
granite42_dense vLLM 0.9052 0.9064 +0.0012 0.9064 +0.0012 MATCH
granite5_lora vLLM 0.9196 0.9172 −0.0024 0.9172 −0.0024 MATCH
granite5_sr vLLM 0.8948 0.8960 +0.0012 0.8960 +0.0012 MATCH
granite5_sr_hf HF 0.8948 0.8940 −0.0008 0.8940 −0.0008 MATCH

All eight jobs returned MATCH, so there is no drift for the author to accept.
Largest deviation anywhere is 0.0024, within tolerance on every row.

The two columns being identical is the result, not a copy-paste. Each line's
four jobs ran in their own pods (itzikv-eval-v031-… / itzikv-eval-v032-…,
with paths.results carrying the line so neither set can overwrite the other),
and each pod logs a vLLM line gate that was read as the receipt: the v031 pods
report vllm-0.31.0 under === vLLM line gate: expect 0.31.x ===, the v032
pods vllm-0.32.0 under expect 0.32.x. Two different wheels scoring the same
3565 rows identically is the expected outcome when neither line changes the
numerics — and worth saying out loud, because identical figures across columns
are otherwise indistinguishable from one line's results reported twice.

Tolerance is ±0.006 for the vLLM configs, which absorbs the HF→vLLM gap for bars
measured on the HF path, and ±0.002 for granite5_sr_hf, whose bar is the HF
number so there is no gap to allow for.

granite5_lora has a second bar at 0.9168 (the switch-HF number). A move toward
it is convergence on HF, not a regression — compare against both before calling
drift.

granite5_sr and granite5_sr_hf run the same checkpoint through the two
backends, so the pair isolates the backend as the only variable; granite5_sr_hf
is also the only scored cover for hf/modeling_granite_switch.py, which a break
confined to the HF path could otherwise hide behind three green vLLM legs.

Drift eval baselines

Baselines in tests/eval/baselines/ were recorded on vLLM 0.30 (the previously
supported ceiling), on the GPU runner itself so the environment that checks
them can reproduce them, and are unchanged by this PR. Re-recording them to
make a line pass would delete the gate, so the numbers below are measured against
the 0.30 baselines exactly as committed.

config 0.31 max mean JSD 0.32 max mean JSD adapter effect (vs base)
granite42_vllm 0.000436 0.000424 ≈0.1940
granite5_vllm 0.002138 0.002112 ≈0.2215
granite42_hf 0.000000 0.000000 0.145573

The right-hand column is why these pass comfortably rather than narrowly. The
adapter effect is the distribution movement the adapter itself produces — the
signal this gate exists to protect — and the observed cross-line movement is
two to three orders of magnitude smaller. granite42_hf is bit-identical on
both lines, which is the expected result for a path that does not go through
vLLM at all and is a useful control on the harness itself.

The measurement is teacher-forced on the baseline's recorded continuation and
sweeps top-k agreement only at k < TOP_K, so a reported mean JSD is distribution
movement at fixed contexts — not a greedy trajectory that desynchronized, and not
churn at the edge of the capture window. Both of those produced false positives
before the gate was corrected; see tests/eval/_eval_core.py.

Summary of evidence

  • 12 GPU legs green: full / audio / eval × (floor + new line) × two
    per-line commits. Both full legs were red before the cascade fix and are
    green after it, with the floor leg green throughout — which is what shows the
    capability gate reads the API shape in both directions rather than just
    silencing 0.32.
  • 8 scored eval jobs green, all MATCH, worst deviation 0.0024.
  • 6 drift eval configs green against unchanged 0.30 baselines, worst movement
    0.002138 against an adapter effect of ≈0.22.
  • Zero version-number branching added to src/. Both fixes gate on shape:
    the processor hook's first parameter name, and
    FlashAttentionMetadata.__dataclass_fields__.
  • plumb_pyproject.py --check --final passes, i.e. dev-latest is on 0.32, the
    true ceiling — the only state in which merging does not leave the top
    supported line untested by CI.

Review notes

Two rows are deliberately not ticked and are worth attention rather than a
silent pass:

  • The 0.30 GPU rows (—²). derive_legs.py emits at most two legs per
    commit, and every commit here spends them on the floor and on a new line. 0.30
    stays declared and supported, and is bracketed by two tested lines, but this PR
    does not re-prove it.
  • sr_vllm drift (—⁷). No compose recipe, since the SR adapters are not
    public. This is the one genuine coverage hole on the vLLM side of the drift
    tier — though note the scored eval granite5_sr does cover SR on
    switch-vllm and returns MATCH, so the SR path is not unevidenced overall.

And two rows are ticked identically across all four columns on purpose, not by
inference: pre-commit (¹) and the CPU matrix (³) both install no vLLM, so there
is one measurement and it applies to every line equally. Per-line CPU evidence
comes from the GPU full legs, which do install a wheel each.

Prepared with AI assistance (Claude Code). Reviewed by the PR author before
submission.

ItzikVa added 27 commits October 19, 2026 11:04
@ItzikVa
Automate the vLLM port so the recurring cost is review, not labour
7e872cc 
vLLM ships a minor line roughly every two weeks and each one costs a day of
the same work: find what moved on our API surface, add the dependency plumbing,
chase two GPU legs green, write up what changed. The steps are mechanical; the
judgement is in the port itself and in reviewing the result.

The skill drives the whole sequence -- detect the gap, port, plumb, run the
gates, open a draft PR -- and leaves a human exactly one decision: merge or
don't.

Four things it knows that are easy to get wrong:

The gap is every unsupported line, not the newest one. The `vllm` extra is a
contiguous range claim, so going from <0.31 straight to 0.32 would claim 0.31
without ever testing it.

Gate on capability, not on version. `hasattr` states the actual requirement and
survives the next upstream rename; a version comparison needs editing every
line and silently misfires on backports. This is the repo's existing rule and
the skill does not get to relax it.

Dispatch GPU runs at the PR branch ref. The `/gpu-test` comment path hardcodes
the default branch, so a new matrix leg added in a PR would never run from a
comment -- the skill would be driving legs green that it had not actually
executed.

Never delete a dependency group. Lines leave the supported range but main's GPU
matrix may still name their groups, and `uv sync --group X` for a group that no
longer exists fails before pytest starts. That shows up as a log under 1.1 KB,
which reads like a quiet pass rather than a hard failure; both this and the
"coverage step red, job green" case are written down so the skill does not
misread its own results.

The scripts carry the parts that are tedious or error-prone by hand:

  vllm_versions.py     the gap, from PyPI vs the extra's claim
  upstream_api_diff.py the few dozen symbols we import or override, classified
                       DELETED / THINNED / SIGNATURE between two tags. A
                       deleted hook we override is a correctness incident, not
                       a refactor -- that is what 0.29 did to ASR.
  plumb_pyproject.py   groups, the extra, and the conflicts table, which has
                       N*(N-1)/2*4 entries and grows quadratically. A missing
                       pair resolves two vLLM wheels into one venv instead of
                       erroring.
  gpu_dispatch.sh      dispatch at a ref, wait, triage
  record_eval_baseline.py  baselines, via the same module the check uses
  preflight.sh         fail before starting, not halfway through

Run state lives in a gitignored file so a session that outlives a 420-minute
GPU leg can resume instead of restarting.

.claude/skills/ is now tracked while the rest of .claude/ stays local -- a
skill only on one laptop is not automation. The pattern is `.claude/*` rather
than `.claude/`, because git does not descend into an ignored directory and the
negation would never be consulted.

Written with Claude Code.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Check the uv pin resolves, not which uv happens to be installed
0c7b55f 
Preflight failed an unattended run whenever the local uv differed from the
uv-lock hook's rev, which it does on any Homebrew box. The failure was wrong on
its own terms: the comment two lines up already said "Not fatal: `uvx uv@<rev>`
can relock with the right version", and then called bad() anyway. Nothing in
the skill relocks with the local uv.

What actually breaks a run is the PIN being unresolvable -- offline or behind a
proxy -- because then the skill falls back to the local uv, writes a different
lockfile revision, and the uv-lock hook rejects the commit at the very end of a
multi-hour upgrade. So check that `uvx uv@<rev> --version` works and report the
local version as information.

SKILL.md told the skill to run bare `uv lock` with the comment "must be uv
0.8.4 - preflight verified this". Preflight verified no such thing once the
above is true, and bare `uv lock` would have run 0.7.19 here. Now the pinned
form, with the instruction to read the rev from .pre-commit-config.yaml so a
hook bump cannot leave the skill relocking against a stale version.

Written with Claude Code.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Make the four scored evals data instead of four 34 KB YAMLs
acdfb67 
The scored accuracy evals existed only as four hand-maintained AppWrapper
YAMLs on one laptop. Their bodies were effectively identical -- the same six
gates, the same harness invocation, the same verdict block -- and only the
checkpoint, the accuracy bar and the eval mode differed. Four copies of a
34 KB file drift, and the drift is invisible: a fix applied to one job's
render gate and not the other three leaves three jobs that cannot detect a
dropped document.

So the differences move into `evals.yaml` and the body is generated.
Changing an adapter, a checkpoint, a bar or the cluster is now an edit to
one short file rather than a four-way copy-paste. `render_eval_jobs.py`
emits one AppWrapper per config for whichever vLLM line is under test.

Secrets stay out. `HF_TOKEN` and `GITHUB_COM_TOKEN` are read from the
environment at render time and land only in the rendered output, which goes
to the gitignored `scratch/eval-jobs/`; the renderer warns and writes
placeholders when they are unset rather than silently producing jobs that
die at the clone.

Two properties of the generated job are worth naming because they are the
reason it can run unattended:

Each job asserts the checkpoint it actually loaded has (or has not) routed
experts. Mounting the G5 path for the 4.2 job produces a plausible score for
the wrong model -- a duplicate run masquerading as independent coverage --
and nothing else in the pipeline would notice.

The render gate runs before any model loads and asserts the document text is
present in the rendered prompt and that the longest prompt fits
MAX_MODEL_LEN. Both failures are silent: documents dropped from the prompt
land the score at the ~0.40 unanswerable base rate, which looks exactly like
a bad adapter, and vLLM truncates an over-long prompt without complaint.
Catching either costs seconds instead of hours of GPU.

SKILL.md now states why there are two eval tiers rather than one. The
`tests/eval/` suite measures distribution drift on 4 prompts in minutes and
catches "the adapter stopped firing"; these jobs measure accuracy on 3565
rows against a fixed bar in hours and catch "the model got worse". A model
that answers confidently and wrongly has a perfectly normal top-k
distribution, so the cheap check cannot see the second failure. The PR-body
template gains the matching scored-accuracy table.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
State the skill's contract, and prove each added version separately
129862f 
The skill's promise was spread across nine phases, so the thing it actually
guarantees was not written down anywhere: compare released vLLM against the
supported ceiling, port every line in the gap, verify each line, and hand back
a reviewable PR. SKILL.md now opens with that contract, and the autonomy
boundary is explicit -- branching, editing, relocking, pushing, dispatching and
opening the PR are pre-authorized by invoking the skill. There are four reasons
to stop and ask, and merging is never the skill's to do.

The substantive change is per-version evidence. The GPU matrix carries two legs
at the ends of the supported range, which is the right standing configuration
but the wrong one during an upgrade: adding 0.31 and 0.32 under a two-leg
matrix tests 0.32 and infers 0.31. So the matrix is widened to one leg per
added line for the duration of the PR, the per-leg results are recorded as they
land, and a final commit narrows it back to the ends. That last commit is also
what makes pyproject's existing "verified out of band" comment true rather than
aspirational.

The scored evals get the same treatment -- rendered and submitted once per added
line rather than once per PR, each with its own --out directory. Without the
per-line directory the loop's `kubectl apply -f <dir>` resubmits the previous
line's four jobs, since job names are unique but the directory is not.

The PR body templates follow: a coverage table with one column per added line
where a tick means that gate ran on that line, and a regression table carrying
each line's accuracy and delta against the unchanged bars. A reviewer should not
have to take "the suite is green" on faith for a version that never had a leg.

Three things in here would have failed at runtime:

The eval jobs installed `.[hf,vllm,compose]` and let pip choose from the `vllm`
extra's RANGE, so a job rendered for 0.31 would install whatever the newest
in-range line was and then die at its own version gate. The pin now travels in
the same pip invocation as the extra.

SKILL.md told the operator to run `plumb_pyproject.py --ceiling 0.32`; there is
no such flag, as the ceiling is derived from the highest --add. And Phase 3
invoked `upstream_api_diff.sh` when the file is `.py`.

REFERENCE.md gains the matrix-leg mechanics it was missing: entries are
{label, dep_group}, the dep_group is the dev flavour because the bare groups
omit pytest, and the floor's dev flavour is bare `dev` rather than `dev-vllm26`.
A leg whose group is absent from the dispatched ref dies before pytest with a
sub-1.1-KB log that reads like a test failure.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Make the tier-1 adapter evals runnable on the GPU cluster
e7b0f6d 
The drift evals existed as pytest files but nothing could reach them. They
carry an `eval` marker that pyproject never declared, so pytest warned on it,
and the GPU workflow had no `eval` scope, so there was no way to ask a pod to
run them. Both halves are needed: the marker makes `-m eval` meaningful and
the workflow arm makes it dispatchable, including via /gpu-test-eval (the
slash-command handler derives families from the `options:` line, so it needs
no edit of its own).

The arm pins `-m "eval and not deep"` rather than a bare `-m eval` for the
same reason the `audio` arm does: pytest's command-line -m REPLACES addopts'
`-m "not deep"` instead of intersecting with it, so the short form would
quietly start running the expensive deep cases every other scope excludes.

Baselines are not recorded yet -- tests/eval/baselines/ holds only its README
-- so the suite is wired but not yet green; recording them needs the adapter
checkpoints staged on a pod.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Get per-line vLLM evidence out of a two-leg GPU workflow
888947a 
The skill told itself to "widen the GPU matrix" for each newly supported
vLLM line. There is no matrix to widen. The legs are derived at run time by
/opt/gsw/derive_legs.py, which lives in the runner image, reads the tested
commit's own pyproject.toml, and emits at most two entries -- one from `dev`
(the floor) and one from `dev-latest` (the ceiling alias). No pull request
edit can produce a third leg, so the old instruction could never have run.

Per-line coverage instead comes from one commit per added line: each commit
repoints `dev-latest` at the next line and is dispatched at its own sha. The
workflow's concurrency group is keyed on sha, so those dispatches run in
parallel rather than cancelling one another, and the floor is re-tested on
every round because it is always the lower leg. The last commit lands the
alias on the true ceiling, which is also the state that should merge.

plumb_pyproject.py grows --point-latest to walk the alias forward and
--check --final to refuse an alias parked mid-range, which would otherwise
leave the top supported line permanently untested by CI. --floor now moves
all three names that identify the floor: the vllm extra's lower bound,
`dev`'s include-group, and default-groups. Missing the last one is not
untidy but fatal -- a default group below the extra's lower bound makes
`uv sync` unsatisfiable, the failure pyproject's own comment records.

The conflicts table keeps its dev-latest pairs even though uv infers them
transitively (verified: `uv sync --no-default-groups --group vllm31 --group
dev-latest` still refuses without them). They stay because the written
table should be truthful on its face and because --check compares against
main's committed form.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Stop the skill from checking out a main that would delete it
33c7813 
Phase 2 opened with `git checkout main`. On any clone where the skill is not
yet committed to main -- which is every clone today -- that removes
.claude/skills/upgrade-vllm/ from the working tree, and an unattended run has
no Phase 3 onward left to read. The branch is created and then the run ends
mid-flight.

Branch at origin/main instead, which never puts main in the tree, and assert
the skill file still exists afterwards so the failure is reported rather than
silently truncating the run.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Make preflight check the workflow that actually runs, and the eval ba…
384b4d8 
…selines

Two false greens, both of which would have let an unattended run start and
then fail deep into the GPU phase.

The suite check grepped the LOCAL gpu-tests.yaml. workflow_dispatch always
executes the workflow file from the DEFAULT BRANCH, whatever ref or sha is
being tested -- the property pyproject's `vllm27` comment already records for
the leg matrix. So a feature branch that adds a suite passes the check while a
dispatch of that suite is rejected with `unknown suite`. That is the current
state exactly: `eval` is on this branch and not on main. Read the file from
origin/<default-branch> instead, and say which branch the verdict is about.

Second, a dispatchable eval suite is still useless with no recorded baselines:
tier-1 compares against committed numbers, so with none the legs cannot be
green and the PR body's drift column has no source. Count the files, excluding
the directory's own README.md, which is documentation and not a baseline.

Preflight now fails honestly on this tree, which is the point -- both gaps are
real and block a trial run.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Refuse a green GPU run that skipped every test
90a84e8 
Two corrections, both from dispatching the eval suite for real (run
39299824463) instead of reasoning about it.

First, revert the preflight change that read gpu-tests.yaml from the default
branch. The premise was that workflow_dispatch always executes main's copy of
the workflow, so a branch-only suite could not be dispatched. That is false.
The run dispatched suite=eval -- an option present only on this branch -- with
--ref on this branch, and it was accepted, derived its legs, and resolved
TEST_PATHS to `-m "eval and not deep" tests/eval/`. GitHub validates the choice
list against the dispatched ref. What is actually true about main is narrower:
gpu_test_command.sh chooses --ref <default branch>, so the /gpu-test COMMENT
path uses main's file. That is a deliberate property of the comment path, not a
platform limit, and this skill does not use it. The comment now records the run
id so the next reader does not re-derive the wrong conclusion.

Second, the thing that run did expose. It came back GREEN having collected 4
eval tests and skipped all 4 for want of staged checkpoints -- pytest exits 0
on an all-skipped suite, the pod prints ALL GPU TESTS PASSED, and the workflow
posts success. gpu_dispatch.sh took the green conclusion and exited 0 without
reading the log, so the skill would have stamped a per-version checkmark for a
suite that evaluated nothing. That is precisely the false coverage claim the
per-line design exists to prevent.

On success the log is now read and a non-zero `N passed` is required; a green
run that only skipped, or that published no artifact at all, is reported as
VACUOUS/UNPROVEN and exits 1. Verified both ways against the real log from that
run and against a genuine "12 passed, 1 skipped" summary.

Preflight additionally counts tier-1 baselines, excluding the directory's own
README.md, since a dispatchable eval suite with nothing to compare against
cannot be green for the right reason.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Let the eval suite obtain its own checkpoint instead of skipping forever
47c5899 
The eval gate could not run anywhere. Every config required its checkpoint to
be pre-staged under GSW_EVAL_*_DIR, and nothing sets those variables on the GPU
pod -- so all four tests skipped, pytest exited 0, the pod printed ALL GPU TESTS
PASSED and the workflow posted success. Run 39299824463 is the evidence: 4
collected, 4 skipped, 0.03s, green. An eval gate that cannot run is worse than
no gate, because a reviewer reads it as coverage.

Staged paths still win when present, for a pod holding a large or non-public
checkpoint. Otherwise the config is composed from public ibm-granite repos into
the same cache root tests/hf/test_multi_switch_e2e.py already uses, so a
checkpoint built by either side is warm-reused rather than built twice in one
leg, and no new artifact is introduced -- these are the base and adapters the
e2e suites already download.

sr_vllm intentionally gets no recipe. Its dual-stream adapters are not public,
and composing a non-SR checkpoint for it would evaluate the wrong thing while
reporting SR coverage.

There is no second env var guarding the compose. The e2e suites need their
GRANITE_SWITCH_E2E_MODELS opt-in because a plain `pytest tests/` reaches them;
these evals are excluded from the default selection by the `eval` marker, so
asking for the suite IS the opt-in. It is also the only workable design: the pod
is launched by /opt/gsw/submit_and_poll.sh from the runner image, which cannot
be handed a new flag from this repo, and a gate nothing can satisfy is exactly
how the suite came to skip permanently. GSW_EVAL_NO_COMPOSE restores the old
staged-only behaviour for anyone who wants it.

Compose failure raises rather than skipping: a recipe that stopped working is a
real break and must not read as an absent checkpoint.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Record baselines with the same sampler the check uses
edaae7d 
record_eval_baseline.py runs standalone, so it never loads tests/conftest.py and
never got VLLM_USE_FLASHINFER_SAMPLER=0. On a GPU pod that meant the recorder
booted vLLM with the FlashInfer sampler, which JIT-compiles through ninja; the
build failed and the engine core never started, so no baseline could be
recorded at all.

The eval CHECK runs under pytest and therefore had the sampler OFF. Recording
and checking differing in engine configuration is exactly what sharing
_eval_core.py is supposed to prevent, so the default belongs here, where both
sides pick it up. Still setdefault, so an image with a working toolchain can
opt back in.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Commit the tier-1 eval baselines, recorded on vLLM 0.30
8dbf058 
Recorded on an A100 from a known-good line (vLLM 0.30.0, torch 2.13.0,
transformers 5.17.0) with checkpoints composed from public ibm-granite repos.
Three of the four configs: granite42_vllm and granite42_hf against
'factuality-correction', granite5_vllm against 'answerability'. sr_vllm is
absent because its dual-stream adapters are not public, so it keeps skipping
rather than being compared to a non-SR model.

Each file carries the checkpoint fingerprint and the versions it was recorded
against, so a baseline silently describing a different checkpoint is caught
instead of quietly passing.

These are what makes the eval gate real: until now it had nothing to compare
against, so the suite failed with "there is nothing to compare against" even
once it could obtain a checkpoint.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Measure the eval gate teacher-forced instead of over a greedy rollout
f22db19 
The granite42_vllm eval failed against its OWN recording line (vLLM 0.30)
with mean JSD 0.95 on prompt 3, and failed byte-identically on 0.26 -- which
ruled out both version drift and non-determinism.

The cause was the measurement, not the model. The gate compared position-
aligned distributions across a self-conditioning greedy rollout, so one
flipped near-tie at position 1 (`527` vs `16`) put the two runs on different
sentences and every later position compared unrelated contexts. The numbers
proved it: mean JSD 0.953644 and mean(1-Jaccard) 0.9583 are both exactly
23/24, the fraction of positions at or after the flip, while every position
before it agreed to JSD < 0.01.

Tolerating that tie is the entire reason this metric is distributional rather
than token-exact -- the fused SWITCH kernel's float-reduction order differs
from vLLM's native linear -- so a rollout gate cannot do its job: it cannot
separate "this line computes different logits" from "this line broke a tie
the other way".

Both the recording and the check now force the baseline's recorded
continuation through the model and read prompt_logprobs at each position, so
every compared position has a byte-identical prefix. Only recording
generates. Verified against the two committed captures of the same
checkpoint: forcing takes prompt 3 from mean JSD 0.9536 to 0.0028. A real
logit or weight regression still moves the distribution at the first position
it touches.

Forcing the same continuation through the base pass also aligns the liveness
comparison by construction: it now asks what the un-adapted model predicts at
the adapter's own contexts.

compare() rejects a pre-forcing baseline rather than comparing against it,
and a short prompt_logprobs raises instead of being averaged over whatever
positions survived. The existing baselines must be re-recorded; the format
changed, not the verdict.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Re-record the tier-1 eval baselines teacher-forced
b8b9c00 
Same checkpoints, same adapters, same fingerprints (89d9b2f9da91e834,
da2c066319fb00d3) and the same known-good line as before -- vLLM 0.30.0 /
torch 2.13.0 / transformers 5.17.0 on an A100. Only the measurement changed,
so these carry a `continuations` key and the distributions are read at those
forced positions rather than over a self-conditioning rollout.

The old files could not be kept: compare() rejects a baseline without
`continuations` rather than comparing a forced capture against a rollout one.

Validated before committing: 4 prompts x 24 positions per pass, every
captured position exactly 20 wide, and 91-97% of forced adapter argmaxes
reproduce the greedily generated continuation token. That last number is the
offset check -- a wrong index would score near zero -- and the few misses are
the near-tie flips between batched prefill and incremental decode that the
distribution metric is designed to tolerate.

Adapter effect is 0.167 / 0.225 / 0.165, well clear of the 0.01 liveness
floor, so the adapters are measurably firing in all three.

sr_vllm still has no baseline by design: its dual-stream adapters are not
public, so it has no compose recipe and keeps skipping.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Disable prefix caching while capturing eval distributions
5e2ee66 
Teacher forcing fixed the trajectory cascade -- granite42_vllm's prompt 3 went
from mean JSD 0.9536 to 0.0000, and granite42_hf now reproduces its baseline
at exactly 0.000000 on every prompt (run 39556314476). What remained was a
small residual that threshold-watching would have misread as drift.

It is not drift: granite5_vllm's prompts 1-3 are byte-identical on the 0.26
and 0.30 legs (0.000642 / 0.002751 / 0.003302), so nothing version-dependent
is moving. But prompt 0 -- the longest, and the only one that moved -- differs
BETWEEN those two legs in both vLLM configs (granite42_vllm 0.000000 ->
0.001857, granite5_vllm 0.001914 -> 0.003221). Same GPU model on both legs and
on the recorder, so that is not hardware class either.

Prefix caching explains exactly that shape. The adapter and base passes differ
only by a leading control token, so they share nearly every block, and whether
a position is recomputed or served from a cached block depends on what already
ran in that engine. The capture therefore depended on pass order, which is
precisely what a baseline must not do.

Turning it off makes the measurement a function of the checkpoint and the
forced tokens alone. The bar for accepting the re-recorded baselines that
follow is that the check reproduces them at 0.000000 -- stronger than the
thresholds, which are deliberately left untouched.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Re-record the vLLM eval baselines with prefix caching off
0cc6cbf 
Same checkpoints and fingerprints, same known-good line (vLLM 0.30.0 / torch
2.13.0 / transformers 5.17.0, A100). granite42_hf is byte-identical to the
previous recording, which is the expected result and a useful check: the HF
backend never had a prefix cache, so only the two vLLM configs move.

granite5_vllm's prompt 3 now ends after 18 tokens instead of 24 -- it reaches
EOS on its own once cached blocks are not in play. The capture handles a short
continuation already (every position is driven by len(continuation)).

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Pin the eval's adapter by name: compose order is filesystem order
7224bbe 
Run 39569446055 failed both granite42 configs on one GPU leg with "baseline
evaluates adapter 'factuality-correction' but the checkpoint now offers
'factuality-detection' first", while the other leg of the same commit composed
the same recipe in the opposite order. The structural guard did its job -- it
refused to compare two different adapters and call the difference a regression
-- but the eval should not have been order-dependent to begin with.

The order is genuinely unstable: `discover_adapters` accumulates into a dict
keyed by name while walking `root_path.rglob("*/*/*/io.yaml")` and returns
`list(...values())` with no sort, and `compose_granite_switch` extends its
adapter list with that result as-is. Its sibling `list_available_adapters`
sorts the same walk, so this looks like an oversight rather than a deliberate
ordering. Consequence beyond the evals: adapter indices and control-token
assignments for a given recipe are not reproducible across machines. Worth
fixing in the composer separately -- not done here, because changing the order
changes every existing checkpoint's token mapping.

The eval now selects its adapter by NAME via EVAL_ADAPTERS, so it measures the
same adapter wherever the checkpoint was composed, and raises if the named
adapter is absent instead of falling back to index 0 and reporting the
substitution as drift.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Record eval baselines on the runner that checks them
06e2140 
A baseline is only a reference if the environment doing the checking can
reproduce it, and a Vela-recorded one could not: run 39569446055 missed it by a
systematic ~0.004 mean JSD on both vLLM configs, identically on the 0.26 and
0.30 legs.

Identical across versions rules out version drift. It is also not torch, which
I assumed first and was wrong about -- granite42_hf reproduces its Vela
baseline at exactly 0.000000 on the torch-2.11 leg against a torch-2.13
recording, so the toolchain gap is demonstrably not what moves these numbers.
Something else in the vLLM capture path differs between the two machines.
Rather than keep guessing at which, record where the check runs: the baseline
is then reproducible by construction and later movement is attributable to the
vLLM line, which is the only comparison this gate is for.

Adds a `suite=eval-record` scope and tests/eval/test_record_baselines.py. It is
a recording job reachable through pytest because pytest is the only thing
submit_and_poll.sh runs on the pod and that script is baked into the runner
image. It prints each baseline between --8<-- / -->8-- markers for a human to
extract and commit, and writes nothing into the repo: a job that could rewrite
a baseline in place is one bad dispatch away from erasing the thing being
gated. It also asserts the capture is well-formed, so a broken recording fails
instead of printing something unusable.

Two selection details that would otherwise bite:

* the cases are marked `deep` as well as `eval_record`, so pyproject's default
  `-m "not deep"` cannot reach them -- which is why this is the one scope whose
  -m does NOT intersect "and not deep". Doing so would select nothing and the
  leg would pass vacuously.
* eval-record bypasses the result cache. Its entire output is the printed JSON,
  and a cache hit skips the pod, so a repeat dispatch would report success with
  nothing to extract.

Verified locally: `-m eval_record` collects exactly the 4 recorders and the
default selection still collects only the 4 checks; the scope arm emits the
right PATHS/LEG for eval-record and still rejects an unknown suite.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Re-record the eval baselines from the GPU runner's own 0.30 leg
6277cd4 
Extracted from run 39578693370 (suite=eval-record), which recorded 3 configs
and skipped sr_vllm on both legs. These come from the vllm30 leg -- the
known-good ceiling -- so the baseline and the check now share one environment.

granite42_hf is unchanged from the Vela recording, byte-for-byte: the only
difference is the dropped trailing newline. That is a useful control. It says
the HF capture path is fully deterministic across both machines AND across the
two legs (0.000000 on all four prompts, 0.26 vs 0.30), which is what rules
torch out as the source of the vLLM offset -- the 0.26 leg carries torch 2.11
against these torch 2.13 recordings and still reproduces exactly.

Validated before committing: 4 prompts x 24 positions, every captured position
20 wide, and 92-94 of 96 forced argmaxes reproduce the generated continuation.

Note on reading cross-leg numbers: each recording leg generates its OWN
continuation, so diffing two recordings prompt-by-prompt re-creates the
trajectory cascade and is meaningless (it showed 0.95 on prompts whose
continuations diverged). Cross-version drift is only measurable the way the
gate measures it -- one committed baseline, both legs forced through it.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Stop the eval metric from measuring its own capture window
4b60198 
The eval gate failed `granite5_vllm` at mean_jsd 0.003472 > 0.003 on the vLLM
0.26 leg of run 39587132597 while passing on 0.30. That looked like a version
regression and was not one. `granite42_vllm` read exactly 0.000000 on the same
0.26 leg and 0.001857 on 0.30 -- the line the baseline was recorded on -- which
is backwards for version drift, and run 39569446055 produced byte-identical
granite5 numbers on both legs. A difference that is the same on both lines
cannot be a property of either line.

Cause: `_pairwise` swept k over (1, 5, 10, 20) with TOP_K == 20, so one of the
four sweeps compared the captures over the full capture WINDOW, and the worst k
is the one reported. The window's edge churns between runs. A token ranked just
inside one capture's top-20 and just outside the other's enters the compared
union present on one side and absent on the other, and `jsd_bits` renormalizes
over the union, so that token contributes roughly its entire probability mass in
bits no matter how small the logit difference that moved it across the boundary
was. It is an artifact of truncating at 20, not a distribution difference.

Measured on the two `eval-record` recordings of run 39578693370, which capture
the same checkpoint on 0.26 and 0.30 and so isolate the line. On granite5_vllm
prompt 0, ~3 asymmetric tokens per position holding 0.0157 of the mass produced
mean JSD 0.016248 at k=20 against 0.000378 at k<=10 over the same positions --
43x, none of it distribution movement. Restricting the support to the two
captures' intersection instead of capping k closes almost none of the gap
(0.000664), so the k cap is the load-bearing change and the restriction would be
redundant; it is not added.

Fix: sweep only k < TOP_K, as the new `COMPARE_KS`. Every compared token is then
ranked well inside both captures, so its presence is not itself a measurement.
Thresholds are unchanged and no baseline is re-recorded -- the committed
0.30-recorded baselines now read mean_jsd 0.000423 / 0.002121 / 0.000000 for the
three configs against the 0.26 capture, inside the 0.003 gate on the metric's own
merits.

Sensitivity is retained, which is the thing worth checking before believing a fix
that makes a failure go away. Against the calibrated corruptions a 30% logit
scaling still reads mean JSD 0.011 and a single-position top-2 swap reads max JSD
0.72, versus gates of 0.003 and 0.05. `granite42_hf` remains exactly 0.000000 on
all four prompts across 0.26/torch 2.11 and 0.30/torch 2.13, so the gate has not
been loosened into blindness.

tests/unit/test_eval_metric_truncation.py builds the pathological pair by hand on
CPU -- two captures agreeing on every retained logprob and disagreeing only on
which near-tie token fell outside the window -- and asserts the configured sweep
reports exactly zero for it while still tripping on both corruptions. Three of
its six cases fail against the previous (1, 5, 10, 20) sweep, so they discriminate
rather than merely pass. It also pins k < TOP_K directly and rejects a
GSW_EVAL_TOP_K set below the whole sweep, which would have dropped every k and
made the eval pass without measuring anything.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Teach the skill the eval gate's two invariants and where to record
06f387e 
The skill's tier-1 section still told a future run to record baselines locally
with `record_eval_baseline.py`, which is no longer the sanctioned path, and said
nothing about the two properties of the measurement that took four GPU runs to
establish. A skill that documents the gate incorrectly is worse than one that is
silent about it: it instructs the next run to reintroduce the bugs.

Document, with the reasoning rather than just the rule, so a future run can judge
an edge case instead of pattern-matching:

- The measurement is teacher-forced, and must not be "simplified" into a greedy
  rollout. Includes the (N - p)/N arithmetic signature that identifies a
  rollout-cascade false positive, so it is recognized rather than re-derived.
- The comparison sweeps only k < TOP_K, because at k == TOP_K it measures the
  capture window's own edge churn.
- The adapter is pinned by name, never by index, because the composer walks rglob
  unsorted.
- Prefix caching stays off in the capture path, for reproducibility not speed.
- Never widen a threshold, alongside the existing never-re-record rule. Plus the
  tell that distinguishes the two cases: a failure that reproduces identically
  across vLLM lines, INCLUDING the line the baseline was recorded on, is a defect
  in the measurement, not stale data. That is exactly how both false positives
  were caught, and it is the cheapest available check.

Baselines are now recorded on the GPU runner via `suite=eval-record`, so the
baseline is reproducible by construction and later movement is attributable to
the one thing that changed. The old script keeps working for a dev box with
checkpoints staged, and its docstring now says it is not the sanctioned path and
why -- a Vela-recorded baseline missed the CI runner by a systematic ~0.004 mean
JSD on both lines, which ruled out both version drift and torch.

gpu_dispatch.sh's usage grows the `eval-record` option with a note that it RECORDS
rather than CHECKS, so nobody reaches for it to clear a failing eval.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
State the drift metric's semantics in the PR body
764e2cd 
A reviewer reading "mean JSD 0.002" has no way to know whether that number
describes distribution movement or an artifact, and the two false positives this
gate produced were both artifacts that looked exactly like regressions. Say in the
body what the number means: teacher-forced on the baseline's continuation, swept
at k < TOP_K, and recorded on the GPU runner rather than a dev pod.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Explain the upgrade-vllm pipeline in one page
72edb07 
The skill is ~400 lines of procedure across SKILL.md and REFERENCE.md, which
is the right form for a run to follow and the wrong form for a human deciding
whether to trust it. This page covers the ten phases, why per-line GPU evidence
needs one commit per line, and the two eval tiers.

It leads with the non-obvious parts rather than the happy path: that a green
run is not proof a gate ran, that --ref and --sha mean different things, and
that an eval failure reproducing identically across lines is a defect in the
metric. Those are the three things that produced wrong results before they were
fixed, so they are what a reviewer should know the pipeline handles.

The footer names SKILL.md, REFERENCE.md and _eval_core.py as the sources of
record and says they win on disagreement -- a summary that drifts is worse than
no summary, and this one will drift.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Fix the upstream API diff reporting every symbol as unchanged
5201e38 
`upstream_api_diff.py` builds regexes and hands them to `git grep -E`, which
is POSIX ERE. POSIX ERE has no `\s`, `\w` or `\d` shorthands -- and `git grep`
does not error on them, it just matches nothing. Both patterns in the script
used `\s`, so `defs_at()` returned 0 definitions for every symbol at every
tag, every comparison was 0-to-0, and the report concluded "nothing moved" no
matter which pair of tags it was given.

That makes the script worse than useless: a clean report is exactly what Phase
3 of the skill reads as "no port work needed", so a silent-breakage upgrade
would have been blessed. Any prior upgrade this script cleared was never
actually checked.

Caught with a positive control rather than by inspection: REFERENCE.md §3
documents that vLLM 0.29 deleted `BaseMultiModalProcessor._call_hf_processor`,
so v0.28.0 -> v0.29.0 has a known non-empty answer. The script reported
nothing. With `[[:space:]]` substituted for `\s` the same invocation reports
`_call_hf_processor DELETED (74 -> 0)`, which is the documented truth.

Replaces `\s` with `[[:space:]]` in `defs_at()` and `symbol_exists()`, and
records the trap in a NOTE above each pattern so the next edit does not
reintroduce a Perl-ism.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Gate the ASR processor on the multimodal hook's shape, not its existence
23d2c8d 
vLLM 0.31 reintroduced `BaseMultiModalProcessor._call_hf_processor`, which 0.29
had deleted -- but with a different contract. There are now three eras of this
surface, not two:

  A (0.26-0.28)  _call_hf_processor(prompt, mm_data, mm_kwargs, tok_kwargs)
                 returns input_ids AND the audio tensors
  B (0.29-0.30)  hook absent; _apply_hf_processor_main(mm_items,
                 hf_processor_mm_kwargs) is the only seam
  C (0.31-0.32)  _call_hf_processor(hf_data, hf_kwargs) returns the audio
                 tensors ONLY; _get_hf_mm_data replaced by the
                 _get_hf_mm_inputs / _finalize_hf_mm_data pair

The existing gate was `hasattr(super(), "_call_hf_processor")` with a comment
reading `# 0.26 - 0.28`. That predicate is true again on 0.31+, so eras A and C
collapsed onto one branch -- the branch written for A's signature. Era C passes
`hf_kwargs=`, while our era-B path reads `kwargs["hf_processor_mm_kwargs"]`, so
a misread era surfaces as a KeyError on the audio path at best, and at worst as
the 0.29 failure mode all over again: an override that is never called, no
transcript, and every test still green. REFERENCE.md §3 calls that the canonical
worst case for a reason -- it is how this exact hook broke last time.

`hasattr` cannot separate A from C because both HAVE the attribute. So the gate
now reads the hook's *signature*: `_base_hook_era()` inspects the first
non-self parameter -- `prompt` means era A, anything else means era C -- and
falls back to probing for `_get_hf_mm_inputs` when the hook is not
introspectable. Still a capability check (operating rule 1): no version number
appears anywhere, and a line that ships a fourth shape will be classified by
what it looks like rather than by what it is numbered.

`_call_hf_processor` is correspondingly rewritten as `(self, *args, **kwargs)`
handling both A and C, and `_apply_hf_processor_main` self-extracts only in era
B, where there is no hook to re-enter.

The test asserts the *effect*, not the call, since a never-called override is
the failure being guarded against. It defines stub parents matching each
released signature and checks that exactly one era self-extracts. Two of the six
cases exist specifically because `hasattr` passes them:
`test_era_a_and_c_are_distinguished` and `test_only_era_b_self_extracts`.
Negative control: reverting `_base_hook_era` to `hasattr` semantics fails 3 of
the 6.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Declare vLLM 0.31 and 0.32, with dev-latest parked on 0.31
d08fa78 
Widens the supported range from 0.26-0.30 to 0.26-0.32. The `vllm` extra's
upper bound moves to <0.33.0, and each new line gets a `vllmNN` pin plus a
`dev-vllmNN` flavour. No group is removed -- `vllm27` and `dev-vllm27` stay,
because a group is a leg that some commit's `dev-latest` can name and
`derive_legs.py` resolves groups against the *tested* commit, so deleting one
makes that leg's `uv sync --group X` fail before pytest starts (REFERENCE.md
§4). The floor is unchanged at 0.26: nothing the port now requires is missing
there, so no supported line is dropped.

`[tool.uv] conflicts` is regenerated rather than extended, by
`plumb_pyproject.py`. The table needs an entry for every pair of groups that
could pull two vLLM lines into one venv; five declared lines with a bare and a
dev flavour, plus `dev-latest` as a third name for whichever line it aliases,
comes to 48 entries where the previous three lines needed 16. A *missing* pair
does not fail loudly -- it lets `uv sync` resolve two vLLM wheels into one
environment -- which is why the table is generated and `--check` verifies it.

`dev-latest` points at **0.31, deliberately not the ceiling**, because this is
the first commit of the per-line evidence walk. The GPU workflow has no matrix
to widen: `/opt/gsw/derive_legs.py` is baked into the runner image and emits at
most two legs, read from the tested commit's `dev` and `dev-latest`. So the only
way each added line gets its own green run is for each added line to BE
`dev-latest` at some commit, dispatched at that commit's sha -- workflow
concurrency is sha-keyed, so those dispatches run in parallel instead of
cancelling one another. This commit buys the floor + 0.31 legs; the next walks
the alias to 0.32. `plumb_pyproject.py --check --final` refuses to pass while
the alias is parked here, so the branch cannot merge in this state.

No dependency ceiling had to rise. PyPI metadata for the new lines asks for
transformers >=5.10.4,<5.18.0 (0.31) and >=5.16.1,<5.20.0 (0.32); our existing
`transformers>=5.16,<=5.17.0` sits inside both, and all three lines pin
torch==2.13.0, which satisfies `torch>=2.11.0`. Relocked with uv 0.8.4, the rev
of the `uv-lock` pre-commit hook.

Prose ranges updated wherever they are stated: README's install note,
tutorials/PREREQUISITES.md, docs/AUDIO.md (plus a `dev-vllm32` sync example),
and the stale "claims 0.26-0.30" comments in pyproject itself. The whitespace
in docs/SWITCH_KERNELS.md is the trailing-whitespace hook, not an edit.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Walk dev-latest to 0.32, the merge-ready state
eb83c33 
Second and last step of the per-line evidence walk. The previous commit parked
`dev-latest` on 0.31 so that the GPU run dispatched at *its* sha would derive a
0.31 upper leg; this commit moves the alias to 0.32 so a run dispatched here
derives a 0.32 upper leg. Both runs exist simultaneously because the workflow's
concurrency group is keyed on `inputs.sha`, so the second dispatch does not
cancel the first. The floor (0.26) is the lower leg on both, which is why it
needs no commit of its own.

This is also the state the branch has to merge in. CI's upper leg is whatever
`dev-latest` names, so merging with the alias on an intermediate line would
leave 0.32 -- the top line the `vllm` extra claims -- permanently untested.
`plumb_pyproject.py --check --final` now passes, and it is the gate that says
so.

Nothing is deleted: `dev-vllm31` stays a declared group even though the alias
has moved off it, both because rule 2 forbids removing a leg some commit still
names and because 0.31 remains inside the supported range.

The conflicts table is regenerated rather than hand-edited, which is the real
reason this is a script and not a one-line edit. `dev-latest` is a *third*
group name for whichever line it aliases, so its six pairs have to move from
0.31's block to 0.32's; leaving them behind would both state a falsehood (that
`vllm31` conflicts with `dev-latest`) and drop the pairs that stop
`uv sync --group vllm31 --group dev-latest` resolving two vLLM wheels into one
venv. Relocked with uv 0.8.4. The alias's prose comment is updated to name 0.32
and to record that the 0.31 parking spot was the walk, not an oversight.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@github-actions

github-actions Bot commented Oct 22, 2026 •
edited
Loading

Copy link
Copy Markdown

✅ GPU tests passed — vllm26-audio

204 passed, 2747 deselected, 20 warnings in 433.56s (0:07:13)

Commit: c841131715bf42eac87d2bc69e175f6d824b2a92
Full run & artifact log

Last 40 log lines 
tests/vllm/test_audio_processor.py::TestMarkerSpoofingRejected::test_adapter_control_token_via_transcript_is_rejected PASSED
tests/vllm/test_audio_processor.py::TestMarkerSpoofingRejected::test_clean_transcript_is_unaffected PASSED

=============================== warnings summary ===============================
<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute

<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute

tests/composer/test_compose_e2e.py:83
  /tmp/granite-switch/tests/composer/test_compose_e2e.py:83: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("compose_e2e")

tests/composer/test_multi_audio_compose_e2e.py:124
  /tmp/granite-switch/tests/composer/test_multi_audio_compose_e2e.py:124: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("multi_audio_compose_e2e")

tests/composer/test_multi_audio_compose_e2e.py:130
  /tmp/granite-switch/tests/composer/test_multi_audio_compose_e2e.py:130: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("multi_audio_compose_e2e")

tests/composer/test_upstream_files.py:218
  /tmp/granite-switch/tests/composer/test_upstream_files.py:218: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("upstream_build_e2e")

.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: 14 warnings
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======== 204 passed, 2747 deselected, 20 warnings in 433.56s (0:07:13) =========
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
===== ALL GPU TESTS PASSED =====
[16:51:24] <job> Succeeded
[16:51:26] verified: found success sentinel in pod log
[16:51:26] cleanup: deleting <job> (exit=0)
<job> "<job>" deleted

Verdict: PASSED

@github-actions

github-actions Bot commented Oct 22, 2026 •
edited
Loading

Copy link
Copy Markdown

✅ GPU tests passed — vllm31-audio

203 passed, 1 skipped, 2747 deselected, 22 warnings in 429.53s (0:07:09)

Commit: 806acedcd95e47a078367471d18bba94d9f45615
Full run & artifact log

Last 40 log lines 
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute

<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute

tests/composer/test_compose_e2e.py:83
  /tmp/granite-switch/tests/composer/test_compose_e2e.py:83: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("compose_e2e")

tests/composer/test_multi_audio_compose_e2e.py:124
  /tmp/granite-switch/tests/composer/test_multi_audio_compose_e2e.py:124: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("multi_audio_compose_e2e")

tests/composer/test_multi_audio_compose_e2e.py:130
  /tmp/granite-switch/tests/composer/test_multi_audio_compose_e2e.py:130: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("multi_audio_compose_e2e")

tests/composer/test_upstream_files.py:218
  /tmp/granite-switch/tests/composer/test_upstream_files.py:218: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("upstream_build_e2e")

.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: 14 warnings
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`.
    warnings.warn(

tests/unit/test_asr.py::TestMonoAndResample::test_resample_real_vllm_backend[8000-16000]
tests/unit/test_asr.py::TestMonoAndResample::test_resample_real_vllm_backend[44100-16000]
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/torch/backends/mkldnn/__init__.py:96: UserWarning: TF32 acceleration on top of oneDNN is available for Intel GPUs. The current Torch version does not have Intel GPU Support. (Triggered internally at /__w/pytorch/pytorch/aten/src/ATen/Context.cpp:265.)
    torch._C._set_onednn_allow_tf32(_allow_tf32)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=== 203 passed, 1 skipped, 2747 deselected, 22 warnings in 429.53s (0:07:09) ===
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
===== ALL GPU TESTS PASSED =====
[13:38:39] <job> Succeeded
[13:38:41] verified: found success sentinel in pod log
[13:38:41] cleanup: deleting <job> (exit=0)
<job> "<job>" deleted

Verdict: PASSED

@github-actions

github-actions Bot commented Oct 23, 2026 •
edited
Loading

Copy link
Copy Markdown

✅ GPU tests passed — vllm31

2867 passed, 75 skipped, 23 warnings in 13388.68s (3:43:08)

Commit: 806acedcd95e47a078367471d18bba94d9f45615
Full run & artifact log

Last 40 log lines 
.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: 14 warnings
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`.
    warnings.warn(

tests/composer/test_compose_e2e.py:83
  /tmp/granite-switch/tests/composer/test_compose_e2e.py:83: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("compose_e2e")

tests/composer/test_multi_audio_compose_e2e.py:124
  /tmp/granite-switch/tests/composer/test_multi_audio_compose_e2e.py:124: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("multi_audio_compose_e2e")

tests/composer/test_multi_audio_compose_e2e.py:130
  /tmp/granite-switch/tests/composer/test_multi_audio_compose_e2e.py:130: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("multi_audio_compose_e2e")

tests/composer/test_upstream_files.py:218
  /tmp/granite-switch/tests/composer/test_upstream_files.py:218: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("upstream_build_e2e")

tests/unit/test_asr.py::TestMonoAndResample::test_resample_real_vllm_backend[8000-16000]
tests/unit/test_asr.py::TestMonoAndResample::test_resample_real_vllm_backend[44100-16000]
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/torch/backends/mkldnn/__init__.py:96: UserWarning: TF32 acceleration on top of oneDNN is available for Intel GPUs. The current Torch version does not have Intel GPU Support. (Triggered internally at /__w/pytorch/pytorch/aten/src/ATen/Context.cpp:265.)
    torch._C._set_onednn_allow_tf32(_allow_tf32)

tests/integration/test_hf_to_vllm_weights.py::TestMultiSwitchForwardEquivalence::test_forward_logit_equivalence
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/flashinfer/cute_dsl/availability.py:136: DeprecationWarning: Arch from arch.py is deprecated. Use Arch from enums.py instead.
    from cutlass.base_dsl.arch import Arch

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========= 2867 passed, 75 skipped, 23 warnings in 13388.68s (3:43:08) ==========
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
[rank0]:[W1025 15:37:39.445745806 ProcessGroupNCCL.cpp:1624] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
===== ALL GPU TESTS PASSED =====
[15:37:54] <job> Succeeded
[15:37:57] verified: found success sentinel in pod log
[15:37:57] cleanup: deleting <job> (exit=0)
<job> "<job>" deleted

Verdict: PASSED

@github-actions

github-actions Bot commented Oct 23, 2026 •
edited
Loading

Copy link
Copy Markdown

✅ GPU tests passed — vllm32-audio

203 passed, 1 skipped, 2747 deselected, 22 warnings in 426.54s (0:07:06)

Commit: c841131715bf42eac87d2bc69e175f6d824b2a92
Full run & artifact log

Last 40 log lines 
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute

<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute

tests/composer/test_compose_e2e.py:83
  /tmp/granite-switch/tests/composer/test_compose_e2e.py:83: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("compose_e2e")

tests/composer/test_multi_audio_compose_e2e.py:124
  /tmp/granite-switch/tests/composer/test_multi_audio_compose_e2e.py:124: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("multi_audio_compose_e2e")

tests/composer/test_multi_audio_compose_e2e.py:130
  /tmp/granite-switch/tests/composer/test_multi_audio_compose_e2e.py:130: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("multi_audio_compose_e2e")

tests/composer/test_upstream_files.py:218
  /tmp/granite-switch/tests/composer/test_upstream_files.py:218: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("upstream_build_e2e")

.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: 14 warnings
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`.
    warnings.warn(

tests/unit/test_asr.py::TestMonoAndResample::test_resample_real_vllm_backend[8000-16000]
tests/unit/test_asr.py::TestMonoAndResample::test_resample_real_vllm_backend[44100-16000]
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/torch/backends/mkldnn/__init__.py:96: UserWarning: TF32 acceleration on top of oneDNN is available for Intel GPUs. The current Torch version does not have Intel GPU Support. (Triggered internally at /__w/pytorch/pytorch/aten/src/ATen/Context.cpp:265.)
    torch._C._set_onednn_allow_tf32(_allow_tf32)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=== 203 passed, 1 skipped, 2747 deselected, 22 warnings in 426.54s (0:07:06) ===
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
===== ALL GPU TESTS PASSED =====
[17:00:02] <job> Succeeded
[17:00:04] verified: found success sentinel in pod log
[17:00:04] cleanup: deleting <job> (exit=0)
<job> "<job>" deleted

Verdict: PASSED

@github-actions

github-actions Bot commented Oct 23, 2026 •
edited
Loading

Copy link
Copy Markdown

✅ GPU tests passed — vllm32-eval

3 passed, 1 skipped, 4 deselected, 17 warnings in 419.89s (0:06:59)

Commit: c841131715bf42eac87d2bc69e175f6d824b2a92
Full run & artifact log

Last 40 log lines 
PASSED
tests/eval/test_adapter_evals.py::test_adapter_eval[sr_vllm] SKIPPED
tests/eval/test_adapter_evals.py::test_adapter_eval[granite42_hf] 
Loading weights:   0%|          | 0/804 [00:00<?, ?it/s]
Loading weights:  12%|█▏        | 94/804 [00:00<00:00, 860.80it/s]
Loading weights:  23%|██▎       | 181/804 [00:00<00:00, 823.65it/s]
Loading weights:  33%|███▎      | 264/804 [00:00<00:00, 787.50it/s]
Loading weights:  43%|████▎     | 347/804 [00:00<00:00, 801.70it/s]
Loading weights:  54%|█████▍    | 434/804 [00:00<00:00, 803.58it/s]
Loading weights:  65%|██████▍   | 519/804 [00:00<00:00, 808.65it/s]
Loading weights:  75%|███████▍  | 600/804 [00:00<00:00, 808.15it/s]
Loading weights:  85%|████████▍ | 681/804 [00:00<00:00, 802.69it/s]
Loading weights:  95%|█████████▍| 762/804 [00:00<00:00, 797.95it/s]
Loading weights: 100%|██████████| 804/804 [00:00<00:00, 807.41it/s]
[transformers] `use_return_dict` is deprecated! Use `return_dict` instead!

[granite42_hf] One adapter on the HuggingFace backend
baseline recorded on vllm 0.30.0
current run     on vllm 0.32.0
  prompt 0: mean JSD 0.000000  max 0.000000 @pos -1  mean(1-Jacc) 0.1111
  prompt 1: mean JSD 0.000000  max 0.000000 @pos -1  mean(1-Jacc) 0.0379
  prompt 2: mean JSD 0.000000  max 0.000000 @pos -1  mean(1-Jacc) 0.0833
  prompt 3: mean JSD 0.000000  max 0.000000 @pos -1  mean(1-Jacc) 0.0303
  adapter effect (vs base): mean JSD 0.145573
PASSED

=============================== warnings summary ===============================
tests/eval/test_adapter_evals.py::test_adapter_eval[granite42_vllm]
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute

tests/eval/test_adapter_evals.py::test_adapter_eval[granite42_vllm]
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute

tests/eval/test_adapter_evals.py: 14 warnings
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`.
    warnings.warn(

tests/eval/test_adapter_evals.py::test_adapter_eval[granite42_vllm]
  /usr/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=328) is multi-threaded, use of fork() may lead to deadlocks in the child.
    self.pid = os.fork()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===== 3 passed, 1 skipped, 4 deselected, 17 warnings in 419.89s (0:06:59) ======
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
===== ALL GPU TESTS PASSED =====
[17:17:27] <job> Succeeded
[17:17:30] verified: found success sentinel in pod log
[17:17:30] cleanup: deleting <job> (exit=0)
<job> "<job>" deleted

Verdict: PASSED

@github-actions

github-actions Bot commented Oct 23, 2026 •
edited
Loading

Copy link
Copy Markdown

✅ GPU tests passed — vllm26-eval

3 passed, 1 skipped, 4 deselected, 17 warnings in 455.29s (0:07:35)

Commit: c841131715bf42eac87d2bc69e175f6d824b2a92
Full run & artifact log

Last 40 log lines 
PASSED
tests/eval/test_adapter_evals.py::test_adapter_eval[sr_vllm] SKIPPED
tests/eval/test_adapter_evals.py::test_adapter_eval[granite42_hf] 
Loading weights:   0%|          | 0/804 [00:00<?, ?it/s]
Loading weights:  11%|█         | 86/804 [00:00<00:00, 856.26it/s]
Loading weights:  21%|██▏       | 172/804 [00:00<00:00, 832.32it/s]
Loading weights:  32%|███▏      | 256/804 [00:00<00:00, 780.54it/s]
Loading weights:  42%|████▏     | 339/804 [00:00<00:00, 788.06it/s]
Loading weights:  53%|█████▎    | 427/804 [00:00<00:00, 814.50it/s]
Loading weights:  63%|██████▎   | 509/804 [00:00<00:00, 797.97it/s]
Loading weights:  73%|███████▎  | 589/804 [00:00<00:00, 796.98it/s]
Loading weights:  84%|████████▍ | 674/804 [00:00<00:00, 794.63it/s]
Loading weights:  94%|█████████▍| 759/804 [00:00<00:00, 801.77it/s]
Loading weights: 100%|██████████| 804/804 [00:00<00:00, 807.31it/s]
[transformers] `use_return_dict` is deprecated! Use `return_dict` instead!

[granite42_hf] One adapter on the HuggingFace backend
baseline recorded on vllm 0.30.0
current run     on vllm 0.26.0
  prompt 0: mean JSD 0.000000  max 0.000000 @pos -1  mean(1-Jacc) 0.1111
  prompt 1: mean JSD 0.000000  max 0.000000 @pos -1  mean(1-Jacc) 0.0379
  prompt 2: mean JSD 0.000000  max 0.000000 @pos -1  mean(1-Jacc) 0.0833
  prompt 3: mean JSD 0.000000  max 0.000000 @pos -1  mean(1-Jacc) 0.0303
  adapter effect (vs base): mean JSD 0.145573
PASSED

=============================== warnings summary ===============================
tests/eval/test_adapter_evals.py::test_adapter_eval[granite42_vllm]
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute

tests/eval/test_adapter_evals.py::test_adapter_eval[granite42_vllm]
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute

tests/eval/test_adapter_evals.py: 14 warnings
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`.
    warnings.warn(

tests/eval/test_adapter_evals.py::test_adapter_eval[granite42_vllm]
  /usr/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=329) is multi-threaded, use of fork() may lead to deadlocks in the child.
    self.pid = os.fork()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===== 3 passed, 1 skipped, 4 deselected, 17 warnings in 455.29s (0:07:35) ======
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
===== ALL GPU TESTS PASSED =====
[17:09:02] <job> Succeeded
[17:09:05] verified: found success sentinel in pod log
[17:09:05] cleanup: deleting <job> (exit=0)
<job> "<job>" deleted

Verdict: PASSED

@github-actions

github-actions Bot commented Oct 23, 2026 •
edited
Loading

Copy link
Copy Markdown

✅ GPU tests passed — vllm31-eval

3 passed, 1 skipped, 4 deselected, 17 warnings in 421.27s (0:07:01)

Commit: 806acedcd95e47a078367471d18bba94d9f45615
Full run & artifact log

Last 40 log lines 
tests/eval/test_adapter_evals.py::test_adapter_eval[sr_vllm] SKIPPED
tests/eval/test_adapter_evals.py::test_adapter_eval[granite42_hf] 
Loading weights:   0%|          | 0/804 [00:00<?, ?it/s]
Loading weights:  11%|█         | 89/804 [00:00<00:00, 889.08it/s]
Loading weights:  22%|██▏       | 178/804 [00:00<00:00, 832.17it/s]
Loading weights:  33%|███▎      | 262/804 [00:00<00:00, 816.16it/s]
Loading weights:  44%|████▎     | 351/804 [00:00<00:00, 843.71it/s]
Loading weights:  54%|█████▍    | 436/804 [00:00<00:00, 797.55it/s]
Loading weights:  64%|██████▍   | 517/804 [00:00<00:00, 799.79it/s]
Loading weights:  75%|███████▍  | 599/804 [00:00<00:00, 800.95it/s]
Loading weights:  85%|████████▌ | 685/804 [00:00<00:00, 815.39it/s]
Loading weights:  96%|█████████▋| 774/804 [00:00<00:00, 813.42it/s]
Loading weights: 100%|██████████| 804/804 [00:00<00:00, 815.72it/s]
[transformers] `use_return_dict` is deprecated! Use `return_dict` instead!

[granite42_hf] One adapter on the HuggingFace backend
baseline recorded on vllm 0.30.0
current run     on vllm 0.31.0
  prompt 0: mean JSD 0.000000  max 0.000000 @pos -1  mean(1-Jacc) 0.1111
  prompt 1: mean JSD 0.000000  max 0.000000 @pos -1  mean(1-Jacc) 0.0379
  prompt 2: mean JSD 0.000000  max 0.000000 @pos -1  mean(1-Jacc) 0.0833
  prompt 3: mean JSD 0.000000  max 0.000000 @pos -1  mean(1-Jacc) 0.0303
  adapter effect (vs base): mean JSD 0.145573
PASSED

=============================== warnings summary ===============================
tests/eval/test_adapter_evals.py::test_adapter_eval[granite42_vllm]
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute

tests/eval/test_adapter_evals.py::test_adapter_eval[granite42_vllm]
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute

tests/eval/test_adapter_evals.py: 14 warnings
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`.
    warnings.warn(

tests/eval/test_adapter_evals.py::test_adapter_eval[granite42_vllm]
  /usr/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=332) is multi-threaded, use of fork() may lead to deadlocks in the child.
    self.pid = os.fork()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===== 3 passed, 1 skipped, 4 deselected, 17 warnings in 421.27s (0:07:01) ======
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
===== ALL GPU TESTS PASSED =====
[13:55:58] <job> Succeeded
[13:56:00] verified: found success sentinel in pod log
[13:56:00] cleanup: deleting <job> (exit=0)
<job> "<job>" deleted

Verdict: PASSED

@github-actions

github-actions Bot commented Oct 23, 2026 •
edited
Loading

Copy link
Copy Markdown

✅ GPU tests passed — vllm26

2868 passed, 74 skipped, 20 warnings in 9350.80s (2:35:50)

Commit: c841131715bf42eac87d2bc69e175f6d824b2a92
Full run & artifact log

Last 40 log lines 
tests/integration/test_multi_switch_vllm_generate.py::test_different_adapters_produce_different_text SKIPPED

=============================== warnings summary ===============================
<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute

<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute

.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: 14 warnings
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`.
    warnings.warn(

tests/composer/test_compose_e2e.py:83
  /tmp/granite-switch/tests/composer/test_compose_e2e.py:83: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("compose_e2e")

tests/composer/test_multi_audio_compose_e2e.py:124
  /tmp/granite-switch/tests/composer/test_multi_audio_compose_e2e.py:124: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("multi_audio_compose_e2e")

tests/composer/test_multi_audio_compose_e2e.py:130
  /tmp/granite-switch/tests/composer/test_multi_audio_compose_e2e.py:130: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("multi_audio_compose_e2e")

tests/composer/test_upstream_files.py:218
  /tmp/granite-switch/tests/composer/test_upstream_files.py:218: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("upstream_build_e2e")

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========== 2868 passed, 74 skipped, 20 warnings in 9350.80s (2:35:50) ==========
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
[rank0]:[W1025 16:42:18.406769716 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
===== ALL GPU TESTS PASSED =====
[16:42:43] <job> Succeeded
[16:42:46] verified: found success sentinel in pod log
[16:42:46] cleanup: deleting <job> (exit=0)
<job> "<job>" deleted

Verdict: PASSED

@github-actions

github-actions Bot commented Oct 23, 2026 •
edited
Loading

Copy link
Copy Markdown

✅ GPU tests passed — vllm32

2867 passed, 75 skipped, 39 warnings in 13691.96s (3:48:11)

Commit: c841131715bf42eac87d2bc69e175f6d824b2a92
Full run & artifact log

Last 40 log lines 
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/flashinfer/gemm/kernels/grouped_gemm_masked_blackwell.py:2391: DeprecationWarning: cutlass.utils.LayoutEnum is deprecated; use cutlass.tensor_utils.LayoutEnum instead. It will be removed in a future release.
    c_layout: utils.LayoutEnum,

tests/integration/test_hf_to_vllm_weights.py::TestMultiSwitchForwardEquivalence::test_forward_logit_equivalence
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/flashinfer/gemm/kernels/cute_dsl/low_latency_blockscaled_gemm.py:102: DeprecationWarning: cutlass.utils.smem_allocator.get_smem_capacity_in_bytes is deprecated; use cutlass.memory.get_smem_capacity_in_bytes instead. It will be removed in a future release.
    _SMEM_CAPACITY_BYTES = cutlass.utils.get_smem_capacity_in_bytes("sm_100")

tests/integration/test_hf_to_vllm_weights.py::TestMultiSwitchForwardEquivalence::test_forward_logit_equivalence
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/flashinfer/gemm/kernels/grouped_gemm_masked_rubin.py:215: DeprecationWarning: cutlass.utils.LayoutEnum is deprecated; use cutlass.tensor_utils.LayoutEnum instead. It will be removed in a future release.
    c_layout: utils.LayoutEnum,

tests/integration/test_hf_to_vllm_weights.py::TestMultiSwitchForwardEquivalence::test_forward_logit_equivalence
tests/integration/test_hf_to_vllm_weights.py::TestMultiSwitchForwardEquivalence::test_forward_logit_equivalence
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/flashinfer/gdn_kernels/blackwell/gated_delta_net_chunked.py:95: DeprecationWarning: cutlass.utils.TensorMapUpdateMode is deprecated; use cutlass.tensor_utils.TensorMapUpdateMode instead. It will be removed in a future release.
    from cutlass.utils import TensorMapManager, TensorMapUpdateMode

tests/integration/test_hf_to_vllm_weights.py::TestMultiSwitchForwardEquivalence::test_forward_logit_equivalence
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/flashinfer/gdn_kernels/delta_rule_dsl/collective_store_tma.py:6: DeprecationWarning: cutlass.utils.tensormap_manager.TensorMapUpdateMode is deprecated; use cutlass.tensor_utils.TensorMapUpdateMode instead. It will be removed in a future release.
    from cutlass.utils.tensormap_manager import TensorMapManager, TensorMapUpdateMode

tests/integration/test_hf_to_vllm_weights.py::TestMultiSwitchForwardEquivalence::test_forward_logit_equivalence
tests/integration/test_hf_to_vllm_weights.py::TestMultiSwitchForwardEquivalence::test_forward_logit_equivalence
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/flashinfer/gdn_kernels/blackwell/gated_delta_net_cp_prefill.py:55: DeprecationWarning: cutlass.utils.TensorMapUpdateMode is deprecated; use cutlass.tensor_utils.TensorMapUpdateMode instead. It will be removed in a future release.
    from cutlass.utils import TensorMapManager, TensorMapUpdateMode

tests/integration/test_hf_to_vllm_weights.py::TestMultiSwitchForwardEquivalence::test_forward_logit_equivalence
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/flashinfer/cute_dsl/availability.py:136: DeprecationWarning: Arch from arch.py is deprecated. Use Arch from enums.py instead.
    from cutlass.base_dsl.arch import Arch

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========= 2867 passed, 75 skipped, 39 warnings in 13691.96s (3:48:11) ==========
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
[rank0]:[W1025 19:27:58.931116296 ProcessGroupNCCL.cpp:1624] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
===== ALL GPU TESTS PASSED =====
[19:28:04] <job> Succeeded
[19:28:07] verified: found success sentinel in pod log
[19:28:07] cleanup: deleting <job> (exit=0)
<job> "<job>" deleted

Verdict: PASSED

@ItzikVa
Read the 0.31+ multimodal data under the key upstream actually passes
d3ecc6b 
vLLM 0.31 normalizes the modality key before calling `_call_hf_processor`:
`_get_hf_mm_inputs` ends with

    # HF processors accept (images, text, videos, audio)
    # so we need to remap vLLM's "audios" -> HF "audio"
    if "audios" in hf_data:
        hf_data["audio"] = hf_data.pop("audios")

(vllm/multimodal/processing/processor.py, v0.31.0 and v0.32.0; absent at
v0.30.0). Our era-C branch read `hf_data.get("audios")`, which by that point
no longer exists, so `_audio_features` was handed an empty clip list on every
request and returned `{}` -- no transcription at all, on exactly the lines
this port exists to support.

This is the 0.29 failure mode wearing a different hat. Reading the wrong key
does not raise: `.get` returns the default, the hook hands back an empty
BatchFeature, and the request dies much later and somewhere else -- in
`_get_prompt_updates`, as

    IndexError: index 1 is out of bounds for dimension 0 with size 1

a cumsum over an empty `audio_num_tokens` indexed for an audio item vLLM
knows exists. Signature on the 0.31 GPU audio leg: 13 integration tests
ERRORing at engine init during `profile_run`, with the real cause three
frames up and no mention of audio anywhere in the message. The floor leg
(0.26) was green in the same run, which is the tell that the fault is
era-specific rather than in the transcriber.

Both keys are accepted rather than just the new one, so that if upstream
moves or drops the normalization the failure cannot come back as silence.
Era A is unaffected: it is handed the raw mm_data, where the key is still
plural.

Found by materializing the v0.31.0 tag and reading `_get_hf_mm_inputs`
directly, rather than by trusting that an empty clip list meant no audio.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Call the processor hooks the way each supported vLLM line calls them
1d86469 
The audio tests invoked `_call_hf_processor` and `_apply_hf_processor_main`
with one era's parameter names hardcoded, so they passed on the lines whose
shape they happened to match and raised on the others. On the 0.31 GPU audio
leg that was 11 failures against a green 0.26 floor leg in the same run:

    KeyError: 'hf_data'                                            (x6)
    TypeError: BaseMultiModalProcessor._apply_hf_processor_main()
      got an unexpected keyword argument 'hf_processor_mm_kwargs'  (x5)

Neither is a product bug -- the src port handles all three eras -- but the
suite cannot report on a line it cannot run on, and "the tests error" is not
distinguishable from "the feature is broken" without reading each traceback.

Two seams now dispatch on era:

* `_call_hook` builds the call for whichever shape is live: era A gets
  `(prompt, mm_data={"audios": ...}, mm_kwargs, tok_kwargs)`, era C gets
  `(hf_data={"audio": ...}, hf_kwargs)` -- singular, because
  `_get_hf_mm_inputs` renames the key before the hook sees it, so passing
  the plural form would exercise a path production never takes. Era B has
  no such hook and skips with a reason naming what covers it instead.
* `_apply_uncached` grew a third branch. 0.31 renamed
  `hf_processor_mm_kwargs` -> `hf_kwargs` AND collapsed
  `_maybe_apply_prompt_updates`'s four loose arguments into a single
  `MultiModalProcessingResult(prompt_ids, kwargs, hashes, prompt_updates)`.
  Both are reshapes of the same chain, so both live in the same helper
  rather than being spread across the six tests that use it.

Dispatch reuses the module's own `_base_hook_era` rather than
reimplementing the discriminator. A second copy would be free to drift from
the first, and then these tests would agree with themselves while
disagreeing with production -- which is the specific thing a capability
check is supposed to rule out.

The previous gate here was `hasattr(proc, "_apply_hf_processor_text_mm")`,
which is still correct for the A/B split it was written for; it is replaced
only because it cannot express a third era.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Re-park dev-latest on 0.31 to re-evidence it with the audio fixes
10ba7ae 
The two preceding commits fix an era-C read of the multimodal data and the
era-shaped test call sites, but they land AFTER the per-line walk, so neither
of the walk's shas contains them: d08fa78 (alias on 0.31) and eb83c33 (alias
on 0.32) both predate the fix. derive_legs.py reads the tested commit, so
re-dispatching at those shas would re-run the broken tree and report the same
11 failures.

Rather than amend the walk -- which would rewrite two shas already dispatched
and already linked from the PR body -- the alias is walked forward again from
here. The history reads as a second lap, which is what actually happened, and
each lap's evidence stays attached to the sha that produced it.

The alias comment is rewritten with it. It claimed 0.32 while the alias names
0.31, and a comment asserting the wrong line is worse here than no comment:
this is the one place a reader checks to find out which line CI's upper leg
will install.

Not a mergeable state -- `--check --final` fails while the alias is parked
below the ceiling. The next commit lands it back on 0.32.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Walk dev-latest back to 0.32, the merge-ready state
2040e81 
Second lap of the per-line walk, carrying the audio fixes that the first lap's
shas predate. This is the state `--check --final` accepts: the alias is on the
ceiling of the `vllm` extra's range, so CI's upper leg installs the top
supported line on every future run.

The alias comment moves with it, for the same reason it moved in the previous
commit -- it names the line, and a reader checking which line the upper leg
installs has nowhere else to look.

Verified:
  declared lines : 0.26, 0.27, 0.30, 0.31, 0.32 (floor 0.26)
  GPU legs       : dev -> 0.26, dev-latest -> 0.32
  conflicts      : 48 entries, 48 required
Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Pass the cascade-attention metadata fields only where they exist
bc16035 
vLLM 0.32 removed cascade attention (upstream e1a508625, "[Attention]
Remove cascade attention", #61126). That deleted five fields from
`FlashAttentionMetadata`, all of them required positional fields with no
defaults: `use_cascade`, `common_prefix_len`, `cu_prefix_query_lens`,
`prefix_kv_lens` and `suffix_kv_lens`.

Five GPU harnesses build that dataclass by hand -- they drive a real
FlashAttention kernel rather than a stub -- and each passed all five to
say "no cascade, this is one plain prefill". On 0.32 every one of those
constructions raises

    TypeError: FlashAttentionMetadata.__init__() got an unexpected
    keyword argument 'use_cascade'

which the worker reports as `vLLM multi worker cannot start`, so the
failure arrives as 71 failed tests across tests/vllm/ and
tests/integration/ rather than as one import error. The vllm31 leg of the
same dispatch is unaffected, which is what identifies this as the 0.32
removal and not a change of ours.

The fields cannot simply be dropped either: on 0.26-0.31 they have no
defaults, so omitting them is the same TypeError in the other direction.
So `tests/shared/vllm_kv_cache.py` -- already the home for "build real
FlashAttention inputs on any supported line" -- gains `cascade_kwargs()`,
which returns the five fields when the dataclass still declares them and
`{}` when it does not, and the five sites splat it. The gate is on
`FlashAttentionMetadata.__dataclass_fields__`, i.e. the shape of the API,
not on `vllm.__version__`.

No behaviour changes on any line. The values passed were exactly "cascade
off", which is the only thing 0.32 can do, so a harness that omits them
there gets the attention it has always asked for.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Assert the CTC transcript on its words, not on its orthography
7312f4f 
`granite-speech-5.0-470m-turboctc` was re-uploaded on 2026-10-21
("Upload 13 files", "Remove legacy custom-code files") and now truecases
and punctuates its output: the committed test clip comes back as "What is
the capital of Israel?" where it used to be "what is the capital of
israel". `ASRTranscriber.transcribe()` pins no revision, so every runner
picked the new weights up at once and all four transcript assertions
started failing together.

This is not the vLLM port. The four failures are identical on the vllm26
floor leg and on vllm31 and vllm32, the lockfile resolves the same
transformers 5.17.0 / torch 2.11.0 as the base branch, and neither
`asr.py` nor this test is touched by this branch -- the checkpoint changed
underneath an unpinned reference.

Casing and terminal punctuation are the ASR model's editorial choice, not
a contract we own, so comparing them was never what these tests were for:
they exist to show that our pipeline routes audio through the CTC backend
(no generate(), decode kwargs dropped), that the dtype override still
transcribes, and that the chunked path neither drops nor duplicates words
at the seams. A `_words()` helper lowercases and strips non-alphanumerics,
and the assertions compare word lists.

This keeps the tests strict about content while no longer breaking when
upstream re-renders the same sentence. The word SEQUENCE is still exact --
a transcript of "what is the capital of france" fails as before -- and the
chunked case still requires the final word to survive the merge, which is
the assertion that actually catches a seam bug. Pinning a revision instead
would freeze the test onto weights a served deployment would not be using.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Re-park dev-latest on 0.31 to re-evidence it with the cascade fix
806aced 
The two fixes in the preceding commits -- the 0.32 cascade-attention
metadata removal and the re-uploaded CTC checkpoint's new orthography --
landed after the previous per-line walk, so neither of that walk's shas
contains them. `derive_legs.py` reads the TESTED commit, so evidence for a
line only exists at a commit whose `dev-latest` points at it; the earlier
shas can only ever prove what the tree looked like before the fixes.

Amending them is not an option: they are already dispatched and already
linked from PR #151, and the skill's rule is to add a commit rather than
rewrite one. So `dev-latest` walks 0.32 -> 0.31 -> 0.32 once more, giving
each line a fresh sha that contains both fixes. The next commit lands the
alias back on 0.32, the merge-ready state.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
Walk dev-latest back to 0.32, the merge-ready state
c841131 
Lands the alias on the true ceiling of the `vllm` extra's range, which is
the only state that may merge: CI's upper GPU leg is derived from
`dev-latest`, so merging with it parked on an intermediate line would
leave 0.32 -- the top line the extra claims -- permanently untested.
`plumb_pyproject.py --check --final` enforces this and now passes.

This sha carries both of this lap's fixes (the 0.32 cascade-attention
metadata removal and the CTC checkpoint's new orthography), so the 0.32
leg dispatched here is evidence for the merged tree rather than for a
tree that predates them.

Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
ItzikVa marked this pull request as ready for review October 25, 2026 21:36
@ItzikVa
Move the upgrade-vllm skill out of the public repo
2baba46 
The skill is internal tooling, not part of what granite-switch publishes, so
it moves to generative-computing/granite-switch-internal and is removed here.
This drops the eleven files under .claude/skills/upgrade-vllm/ and the
docs/UPGRADE_VLLM_PIPELINE.html page that documents them, and reverts the
`!.claude/skills/` negation in .gitignore that existed only to track them --
with no committed skills left, `.claude/*` plus the settings.json negation is
again the whole rule.

What stays public is deliberate: tests/eval/ (the drift gate, its three
committed baselines and its prompt set) and the `eval` / `eval-record` arms of
.github/workflows/gpu-tests.yaml. Those are repository test infrastructure that
any contributor needs in order to reproduce or re-record a baseline, and they do
not depend on the skill -- the skill merely dispatched them. All six drift-eval
results in this PR were produced by that workflow, so removing the skill does
not weaken any evidence here.

Five places referenced the deleted path and would otherwise have become dead
pointers. Three are pyproject.toml comments explaining the two-leg GPU walk and
the generated `[tool.uv] conflicts` table; they now name the internal repo, so a
reader can still find the generator. The other two mattered more, because they
told a human how to record a baseline by running a script this commit deletes:
tests/eval/test_adapter_evals.py's assertion message and
tests/eval/baselines/README.md now give the `-f suite=eval-record` workflow
dispatch instead. That is a better instruction regardless of where the skill
lives -- a baseline has to be captured on the same runner that checks it, since
one recorded elsewhere missed by a systematic ~0.004 mean JSD on both vLLM
configs, identically on the 0.26 and 0.30 legs.

No dependency, source or test behaviour changes, so uv.lock is untouched and the
twelve GPU legs and eight scored evals already collected remain valid.

Assisted-by: Claude Code
Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
@ItzikVa
ItzikVa changed the base branch from feature/vllm-upgrade-skill to main October 26, 2026 08:54
@ItzikVa ItzikVa closed this Oct 26, 2026
@ItzikVa ItzikVa reopened this Oct 26, 2026
@codecov-commenter

codecov-commenter commented Oct 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@antonpibm antonpibm Awaiting requested review from antonpibm antonpibm is a code owner
@aviv1ron1 aviv1ron1 Awaiting requested review from aviv1ron1 aviv1ron1 is a code owner
@freunda freunda Awaiting requested review from freunda freunda is a code owner
@yairallouche yairallouche Awaiting requested review from yairallouche yairallouche is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ItzikVa @codecov-commenter