Audit the suite's skips and fix what un-gating exposed by ItzikVa · Pull Request #152 · generative-computing/granite-switch

ItzikVa · 2026-10-25T07:11:29Z

tests/hf/test_quantization.py — GRANITE_SWITCH_E2E_MODELS was doing two
unrelated jobs. Gating cost (composing a real ~3B checkpoint) is legitimate and
stays. Gating these 16 tests was not: what constrains them is capability, not
cost, and the flag skipped them before importorskip could ever run. Split into
gates that each report their own reason — CUDA at module level, backends via
importorskip in the owning fixture, and a real probe for Quanto float8. Lifting
the flag alone would have swapped one silent skip for another since neither
backend was installed, so a pinned quant dependency group comes with it.
test_skinning_equivalence_hf — carried an unconditional skip. Its stated
reason was sound (fused QKV/gate-up projections sum in a different order than
upstream's separate ones, so max_diff == 0.0 is unreachable) but the conclusion
was too strong: the sibling vLLM worker had already answered the same arithmetic
with a tolerance. This is the only CPU-runnable check that the composer's weight
transfer reproduces the original model.
test_granite_4_2_chatml — gated on GRANITE_4_2_BASE pointing at a local
path because "4.2 ships on a local path, not the Hub." No longer true;
ibm-granite/granite-4.2-3b is public and ungated, and the test needs only its
tokenizer. Now defaults to the Hub with the variable kept as an override. Also
adds the assertion the docstring always promised: id 100256, not merely
something that stringifies to <|im_start|>.
make test-e2e — the knowledge of which variable to export lived only in
shell history. The 11 gated modules split 4/5/1 across tests/hf, tests/vllm
and tests/integration, so the target runs all three; an integration-only target
would leave six still skipping.

test_granite_4_2_chatml was gated on GRANITE_4_2_BASE pointing at an existing local path, on the stated grounds that "the 4.2 base ships on a local path, not the Hub". That is no longer true: ibm-granite/granite-4.2-3b is public and ungated, and this test needs only its tokenizer -- no weights and no GPU -- so there was nothing left for the gate to protect and it skipped on every machine that had not set the variable by hand. It now defaults to the Hub id with GRANITE_4_2_BASE still honoured as an override for a local or pre-release 4.2, and _tok keeps skipping on a fetch failure so an offline run degrades to a skip rather than a red test. While the assertions were being touched, add the one the docstring always promised and never checked: that the probe returns id 100256, not merely something that stringifies to <|im_start|>. The file goes to 9 passed / 0 skipped on CPU. Found while auditing every skip the GPU suite reports; that inventory is kept as a local working note rather than committed, since it is a point-in-time snapshot that goes stale as soon as a gate changes. Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>

The skipped-test audit found that GRANITE_SWITCH_E2E_MODELS was doing two unrelated jobs, and doing one of them wrong. Where it gates cost, keep it -- composing a real ~3B checkpoint is genuinely expensive and should stay opt-in. But the knowledge of which variable to export lived only in shell history, so add `make test-e2e`. It runs tests/hf, tests/vllm and tests/integration, not just integration: the eleven gated modules split 4/5/1 across those directories, so an integration-only target would leave six still skipping and quietly under-deliver. Separate pytest invocations per directory so one leaking CUDA context cannot poison the next. Where the same flag gated tests/hf/test_quantization.py it was wrong -- what constrains those sixteen tests is capability, not cost. The flag skipped them before importorskip ever ran, so the quantization path had zero coverage on every CI run. Split it into gates that each report honestly: CUDA at module level, bitsandbytes/optimum-quanto via importorskip in the owning fixture, and a real probe for Quanto float8. Lifting the flag alone would only have swapped one silent skip for another, since neither backend was installed, so add a `quant` dependency group to dev, dev-vllm27 and dev-vllm30. Pinned, unlike most of this file: these JIT-compile CUDA extensions, so a floating version could turn the suite red from a background upgrade rather than from a commit. The four float8 tests needed a probe rather than an import check because they ERROR instead of skipping. The visible exception is a transformers "issues during automatic conversion of the weights" naming base_layer.weight, which reads like a granite-switch LoRA fault and is not one: ptxas aborts with "Feature 'cp.async' requires .target sm_80 or higher" while building marlin_cuda.o -- an INT4 kernel float8 never uses, pulled in because quanto_cuda builds all-or-nothing. Reproduced on an A100 with a bare nn.Linear and no granite-switch code, which is also what the probe runs, in a subprocess so a hard abort cannot take the session down. A skip now carries the real ptxas error instead of a misattributed LoRA one. Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>

test_two_adapters_split and test_partial_tile_padding each carried a pytest.skip for block_m=1, which reported two tests as skipped on every run. The skips were correct but mislabelled what they were doing, and the two were not the same thing: - split[1] genuinely fails without its skip. M // 2 is 0 at M=1, so adapter_indices is [2] and the reference returns bitmask 2 where the test asserts 3. One token cannot be split between two adapters. - padding[1] passes. pad_m = (-2) % 1 == 0, so no padding exists and the reference already satisfies both assertions. Neither is a case that merely went unchecked, so a skip overstates it: a tile of one token cannot hold a split, and M % 1 is always 0 so there is never a partial tile. Narrow the parametrization instead -- a second fixture supplies the same sweep without 1 -- and the configurations are simply not generated. Nothing is reported as skipped and nothing asserts the degenerate answer, which would have been worse than either: a green cell pinning arithmetic no caller relies on, at a tiling production never uses (BLOCK_M is 32). BLOCK_M=1 keeps its coverage from the other five tests, including test_different_patterns_per_tile[1], which already exercises the tile boundary padding[1] was nominally about. 36 collected / 2 skipped becomes 34 / 0. Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>

test_skinning_equivalence_hf carried an unconditional @pytest.mark.skip whose stated reason was sound -- the skin's fused QKV/gate-up projections sum in a different order than upstream HuggingFace's separate ones, so the worker's `max_diff == 0.0` was unreachable -- but the conclusion drawn was too strong. The sibling vLLM worker had already faced the same arithmetic and answered it with a tolerance (LOGPROB_ATOL) rather than a skip. Do the same here, because both vLLM skinning tests need a GPU and this is the only end-to-end check that the composer's weight transfer reproduces the original model on CPU. The worker now applies three checks, ordered most to least meaningful: 1. argmax agreement, exact -- what greedy decoding actually emits. 2. top-100 token set agreement, exact, as sets so benign tie ordering does not count as a regression. 3. magnitude agreement within ULP_SLACK ULP. Ranking leads because magnitude is the weaker signal: log_softmax is shift-invariant, so adding a constant to every logit leaves the distribution identical while moving max|diff| arbitrarily. Verified -- a uniform +1.5 shift changes log-probabilities by 3.8e-06, i.e. nothing. Check 3 still earns its place by catching the order-preserving faults ranking cannot see, such as a wrong logits_scaling. The bound is derived from the dtype rather than fixed. A flat atol does not transfer from the vLLM worker, which compares logprobs: one ULP of a raw logit measures ~5e-06 in fp32 but ~0.34 in bf16, so any constant tight enough to mean something in fp32 would reject plain arithmetic in bf16. ULP_SLACK is 8 against a measured noise floor near zero -- matmul accumulates in fp32 and rounds once, so two reduction orders give the same value or adjacent ones. Exercised on synthetic logits: identical, fp32 ULP noise, bf16 round-trip and a uniform shift all pass; per-token shifts, swapped positions, a 1.05x scale and a reorder inside the top-100 all fail. The real download-compose-forward path is unverified until a GPU run; if granite-4.0-micro's true noise exceeds 8 ULP the printed max|diff| and bound make that a one-look adjustment. Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>

The three files changed by the skip audit cannot be verified on a CPU box, and `full` is not currently a usable way to reach them: every full-suite dispatch since 2026-10-18 has been cancelled from outside the pod at ~95-120 min, which stops a 3h46m suite around 40% and reports no summary. The limit is none of the three this workflow configures (360/390/420 min). So give them a narrow scope that finishes well inside that window. What each path is here to prove: tests/hf/test_quantization.py 16 tests that used to skip on an env flag before importorskip could run; 4 now gate on a Quanto float8 probe whose answer only exists on a GPU. tests/composer/test_skinning_... was @pytest.mark.skip outright; now compares within a dtype-derived ULP bound chosen against simulated noise, not the real model's. tests/unit/test_bitmask_computation cheap, and not collectable off-GPU at all: it imports lora_kernel_meta, which imports vllm.distributed. Deliberately not wired into the /gpu-test comment path's defaults, and not a substitute for `full`. Remove this arm once the ~95 min cancellation is diagnosed and `full` can be trusted again. Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>

The previous commit un-skipped this test by comparing logit magnitudes within an 8-ULP bound plus an exact top-100 token-set match. The GPU leg showed the set check was unreachable: max |diff| = 7.5e-01 tolerance = 9.1e-01 (8 ULP) <- passed argmax mismatches: 0 / 12 <- passed top-100 set mismatches: 11 / 12 <- FAILED bf16 resolution is ~0.11 at these logits, so the rank-100 boundary is a pile of exact ties -- measured 4, 7 and 7 tokens sharing one bf16 value at the cut on the first three positions. Which tie-mate lands inside the cut is arbitrary, not a regression. Comparing sets rather than sorted order was an attempt at tie-robustness, but membership *at the boundary* is exactly as tie-sensitive as order. Replace both with the gate tests/vllm/_generation_equivalence_worker.py already calibrated -- same question, so a second set of thresholds would just be a second thing to keep calibrated: top-k JSD + Jaccard swept over k=1..100, plus the exact argmax check (measured a strict maximum at all 12 positions, so safe to demand exactly). The magnitude bound was the weaker problem -- not wrong, just near-vacuous, since a uniform shift moves max|diff| arbitrarily while leaving the distribution identical. Those three are all post-log_softmax, which leaves them blind to a uniform logit SCALING: a 5% error measures mean JSD 7.2e-04, inside the benign floor, while being the signature of a mistransferred logits_scaling (typically 8.0, so one step is ~12%) or attention_multiplier. Add a least-squares slope fit with free intercept, which separates it because a scale error is systematic across all 1.2M elements where reduction-order noise is random: 3.2e-05 for the real skin versus 1.0e-02 for a 1% scaling. Verified against 11 cases through the real comparator: the actual skin, an identical copy, a +1.5 uniform shift and a bf16 round-trip pass; a per-token shift, 1%/5% scaling, logits_scaling 8.0->8.1, two swapped positions, one corrupted position, a 100-token weight permutation and a subtle +0.9 all fail. Real skin passes with 27x margin (max JSD 1.9e-03 against a 0.05 gate). Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>

The slope check added in the previous commit read 3.2e-05 on arm64 but 4.0e-03 on CI's x86 leg, failing its 1e-3 gate on a skin that was otherwise clean: argmax 0/12, JSD at the benign floor, nothing else moved. The platform was not giving the wrong answer -- OLS is the wrong estimator. cov(x,y)/var(x) assumes a noise-free predictor. Here x = t + e_x and y = t + e_y (t the exact logit, e each backend's reduction error), so cov = var(t) while var(x) = var(t) + var(e_x), and the fit converges to 1 - var(e_x)/var(t). The reference is itself bf16, so there is always some e_x, and the bias goes as the SQUARE of the residual -- a platform with 2x the reduction noise reports 4x the scale error for an equally correct skin. Confirmed by injecting noise into both sides of the measured logit pair: extra noise residual OLS |a-1| std ratio |a-1| none 2.21e-02 3.2e-05 2.8e-04 1x 3.13e-02 2.7e-04 2.2e-04 2x 4.94e-02 9.0e-04 3.2e-04 4x 9.11e-02 3.9e-03 2.6e-04 <- the CI reading 6x 1.34e-01 8.7e-03 3.2e-04 Use std(sw)/std(ref) instead: symmetric in the two noises so they cancel rather than bias it, and still exactly `a` under a true uniform scaling. Mean-centred via std, so a benign uniform shift still reads exactly 1. SLOPE_TOL goes to 3e-3, ~10x over the measured 2.8e-04 floor and still ~35x under the 1.03e-02 a 1% scaling produces. Also print the reduction residual, ungated. It is the quantity that explains a scale reading, so a future platform difference is one run to diagnose instead of another round of bisecting CI. Re-verified through the real comparator: real skin, identical, uniform shifts (+1.5, +0.9) and a bf16 round-trip pass; 1%/5% scalings, logits_scaling 8.0->8.1, a per-token shift, swapped positions, a corrupted position and a 100-token weight permutation all fail. Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>

launch_vllm spawned the API server as `python -m vllm.entrypoints.openai.api_server`, letting PATH decide which interpreter ran it. In a container with both a system Python and a project venv those are different interpreters, and the one first on PATH need not have vLLM installed. That is not hypothetical: five tests/integration serving modules failed this way under `make test-e2e PYTHON=.../.venv/bin/python` on a GPU pod. The venv ran the tests, but the child got /usr/local/bin/python and died on "No module named 'vllm'" before binding its port, so every test in those modules timed out waiting for a server that was already dead. Use sys.executable so the server always runs in the same environment as its caller. Verified on the pod: test_conversation_prefix_cache now reaches SERVE_OK where it previously could not start the server. Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>

The HF skinning check gated a numeric estimate of the skin's logit scale to catch a mistransferred Granite multiplier, which the three distribution checks cannot see (they run after log_softmax, and a uniform scaling barely dents a normalized shape -- a 5% error measures inside the benign JSD floor). That gate could not be made portable. As an OLS slope it read 3.2e-05 on arm64 and 4.0e-03 on CI's x86, and the previous commit swapped in std(sw)/std(ref) on the theory that OLS was diluted by noise in the predictor. The theory was wrong: the ratio still read 3.7e-03 on x86 against 2.8e-04 on arm64. Both platforms report the same reduction residual (2.318e-02 vs 2.213e-02, within 5%), so noise is not the difference. Independent per-side noise bounds |a-1| by residual^2/2; arm64 sits at 1.1x that ceiling while x86 sits at 14x it, so x86 really does contract these logits ~0.37% on a skin that is otherwise clean (argmax 0/12, JSD at the floor). No estimator fixes a real platform difference, and a gate loose enough for x86 would sit 2.7x under what a 1% scaling reads. Check the config instead, where the bug actually lives and needs no tolerance. compose_utils.py propagates these with getattr(base_config, field, 1.0), so a field upstream renames or drops silently becomes 1.0 -- confirmed for all four, including logits_scaling, which is 10 on granite-4.0-micro. Comparing against the base catches that exactly, on every platform, and covers residual_multiplier and embedding_multiplier too, which no logit-scale reading could isolate. test_save_load_compose only checks these survive a round-trip (loaded == built), never that built matches the base. Scale and residual are still printed, ungated, since they are what made the platform gap diagnosable in one run. Validated by injection: all four silent 1.0 defaults convict and a clean config passes; the logit gates still convict a vocab tilt, a top-100 permutation, a corrupted position and swapped positions, while passing the real skin, a bf16 round-trip and uniform/per-position shifts. Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>

_ADAPTER_MAP_RE spelled out the exact key order of an adapter_map entry and required the closing brace right after 'invocation_text'. The composer's aLoRA branch now emits a fourth key as well (invocation_drop_chars, tokenizer_setup.py:986-992), so every aLoRA entry stopped matching while LoRA entries -- still three keys -- kept parsing. The failure that produced was badly misleading. On an all-aLoRA checkpoint the map came back EMPTY, and both workers that share this parser reported "need >=2 alora adapters, got []" against a sound 3-adapter model, pointing at the adapters instead of at the reader. Match an entry by its braces and read keys out of the body instead, so the map stays the composer's to extend. Verified against all four entry shapes the composer emits (4-key aLoRA, 3-key LoRA, SR, and the legacy 3-key aLoRA form), and it ignores unrelated Jinja dicts. Confirmed on a GPU pod: test_conversation_prefix_cache now resolves all three aLoRA adapters and passes, measuring the routing the empty map had been hiding (PRESERVE reuses 80 cached positions against RE_PREFILL's 64, with distinct decode write addresses). Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>

The skinning vLLM tests failed on every `skips` run at engine init: vLLM's default gpu_memory_utilization (0.92 in newer releases) is a fraction of total device memory, and earlier in-process HF tests leave ~9 GiB held by the parent pytest, so only 70.19/79.25 GiB is free against the 72.91 GiB requested. Pass gpu_memory_utilization=0.8 and max_model_len=2048 (inputs top out at 256 tokens; this stops vLLM sizing KV cache for the full 131k context). The original and the skin both run through cmd_run, so they keep identical settings. Signed-off-by: aviv ron <rona@il.ibm.com>

Four lines in docs/SWITCH_KERNELS.md carry trailing whitespace, which the `trailing-whitespace` pre-commit hook strips. The file arrived in 7e72456 (#148) and is already on main, so the hook fails for every branch cut from current main rather than for anything the branch changed -- this PR included, where it also held back the CPU test job that gates on pre-commit passing. Whitespace only: `git diff -w` on the file is empty, so no prose is altered and the rendered Markdown is unchanged. Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>

codecov-commenter · 2026-10-25T07:38:18Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

ItzikVa · 2026-10-25T07:42:29Z

/gpu-test-multi

github-actions · 2026-10-25T10:40:02Z

❌ GPU tests failed — `vllm26-multi`

1 failed, 2875 passed, 52 skipped, 20 warnings in 10546.23s (2:55:46) — plus 4 nested failure(s) in child pytest processes

Commit: 4e4ff82083966f80a73813dba3e72b0e7247b8c2
Full run & artifact log

Last 40 log lines

    @pytest.mark.xdist_group("multi_audio_compose_e2e")

tests/composer/test_upstream_files.py:218
  /tmp/granite-switch/tests/composer/test_upstream_files.py:218: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("upstream_build_e2e")

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED ::test_asr_ctc_default_gpu.py::<isolated-subprocess> - Isolated module...
==== 1 failed, 2875 passed, 52 skipped, 20 warnings in 10546.23s (2:55:46) =====
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
[rank0]:[W1025 10:39:30.892281122 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
FATAL pytest failed
[10:39:49] <job> Failed
[10:39:51] --- post-mortem: why the job ended (status and events only) ---
[10:39:51] <job> status:
  QuotaReserved=True Resuming: Suspend is false
  ResourcesDeployed=True Resuming: Suspend is false
  PodsReady=True SufficientPodsReady: 1 pods running; 0 pods succeeded
  Unhealthy=False Resuming: Suspend is false
[10:39:53] pod <job> phase and conditions:
  phase=Failed
  Initialized=True : 
  Ready=False PodFailed: 
  ContainersReady=False PodFailed: 
  PodScheduled=True : 
[10:39:53] pod <job> container states:
  pytorch: ready=false restarts=0 state={"terminated":{"containerID":"cri-o://4942ffbb1473297c8d708d517ff0495323db3d6781dd5eba846fc53de9bd6467","exitCode":1,"finishedAt":"2026-10-25T10:39:32Z","reason":"Error","startedAt":"2026-10-25T07:43:02Z"}}
[10:39:54] kubelet events for <job>:
TIME                   TYPE     REASON           MESSAGE
<nil>                  Normal   Scheduled        Successfully assigned security/<job> to dmf-nnnqh-gpu-worker-3-il-pcqjz
2026-10-25T07:43:02Z   Normal   AddedInterface   Add eth0 [10.134.7.184/23] from ovn-kubernetes
2026-10-25T07:43:02Z   Normal   Pulled           Container image "vllm/vllm-openai:latest" already present on machine
2026-10-25T07:43:02Z   Normal   Created          Created container pytorch
2026-10-25T07:43:02Z   Normal   Started          Started container pytorch
[10:39:55] --- end post-mortem ---
[10:39:55] cleanup: deleting <job> (exit=1)
<job> "<job>" deleted

Verdict: FAILED — 1 failed, 2875 passed, 52 skipped, 20 warnings in 10546.23s (2:55:46) — plus 4 nested failure(s) in child pytest processes

github-actions · 2026-10-25T11:51:23Z

❌ GPU tests failed — `vllm30-multi`

1 failed, 2874 passed, 53 skipped, 23 warnings in 14794.59s (4:06:34) — plus 4 nested failure(s) in child pytest processes

Commit: 4e4ff82083966f80a73813dba3e72b0e7247b8c2
Full run & artifact log

Last 40 log lines

    @pytest.mark.xdist_group("upstream_build_e2e")

tests/unit/test_asr.py::TestMonoAndResample::test_resample_real_vllm_backend[8000-16000]
tests/unit/test_asr.py::TestMonoAndResample::test_resample_real_vllm_backend[44100-16000]
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/torch/backends/mkldnn/__init__.py:96: UserWarning: TF32 acceleration on top of oneDNN is available for Intel GPUs. The current Torch version does not have Intel GPU Support. (Triggered internally at /__w/pytorch/pytorch/aten/src/ATen/Context.cpp:265.)
    torch._C._set_onednn_allow_tf32(_allow_tf32)

tests/integration/test_hf_to_vllm_weights.py::TestMultiSwitchForwardEquivalence::test_forward_logit_equivalence
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/flashinfer/cute_dsl/availability.py:135: DeprecationWarning: Arch from arch.py is deprecated. Use Arch from enums.py instead.
    from cutlass.base_dsl.arch import Arch

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED ::test_asr_ctc_default_gpu.py::<isolated-subprocess> - Isolated module...
==== 1 failed, 2874 passed, 53 skipped, 23 warnings in 14794.59s (4:06:34) =====
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
[rank0]:[W1025 11:50:55.979348768 ProcessGroupNCCL.cpp:1624] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
FATAL pytest failed
[11:51:07] <job> Failed
[11:51:09] --- post-mortem: why the job ended (status and events only) ---
[11:51:09] <job> status:
  QuotaReserved=True Resuming: Suspend is false
  ResourcesDeployed=True Resuming: Suspend is false
  PodsReady=True SufficientPodsReady: 1 pods running; 0 pods succeeded
  Unhealthy=False Resuming: Suspend is false
[11:51:10] pod <job> phase and conditions:
  phase=Failed
  Initialized=True : 
  Ready=False PodFailed: 
  ContainersReady=False PodFailed: 
  PodScheduled=True : 
[11:51:11] pod <job> container states:
[11:51:12]   (unavailable)
[11:51:12] kubelet events for <job>:
TIME   TYPE   REASON   MESSAGE
[11:51:14] --- end post-mortem ---
[11:51:14] cleanup: deleting <job> (exit=1)
<job> "<job>" deleted

Verdict: FAILED — 1 failed, 2874 passed, 53 skipped, 23 warnings in 14794.59s (4:06:34) — plus 4 nested failure(s) in child pytest processes

test_asr_ctc_default_gpu.py asserted against a single literal _EXPECTED string, so any change in how the CTC head punctuates or capitalises the same words failed the test. Four of its six tests failed that way on an A100 while the transcript was, as text, correct. What the test exists to prove is that the default CTC path transcribes the clip and that the transcript reaches the request -- not that the decoder reproduces one byte sequence. Normalise both sides (casefold, drop punctuation, collapse whitespace) before comparing, so a wrong transcription still fails and a re-punctuated correct one does not. Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>

Three GPU failures shared one root cause, and none of them was the bug the failing test's own message named: a prompt of random filler token ids, compared token-exactly under greedy decoding. Measured on an A100 (vLLM 0.26.0, bf16): _pp_equivalence_worker.py built 16 ids from torch.randint(1, 1000). PP=1 and PP=2 agreed for four tokens, then the PP=1 arm collapsed into a repeating two-token loop. The message blamed the PP=2 per-rank metadata recompute, but PP=2 was byte-identical to an earlier green run; PP=1 was the arm that moved. A 2x2 probe cleared async_scheduling (no effect either way) and implicated CUDA-graph capture mode, which vLLM downgrades above PP=1. A negative control then cleared capture too: at PP=1 on real-language prompts, cudagraph_mode=PIECEWISE is byte-identical to enforce_eager with routing intact -- three adapters, three distinct continuations. The prompt was the whole problem. test_batched_generation_matches_solo diverged on the third request at step 8, where the solo run's top two logprobs were the same bf16 value (-1.07067, -1.07067; gap exactly 0.000000). A coin flip, not mis-routing. Out-of-distribution prompts leave the argmax exactly tied, so anything that legitimately reorders a float reduction -- capture mode, co-batching, a version bump -- lands the tie on the other side, and greedy amplifies the one flipped token across every remaining step. A sub-ULP difference presents as total divergence. Fix both halves, because neither alone suffices. Real-language prompts (appended control token rather than spliced, which would break the text back out of distribution) made PP=1 and PP=2 agree on all 32 tokens -- but PP=1 still reported min_top2_gap=0.000000, so no prompt guarantees a separated argmax everywhere. So the tolerance is measured rather than assumed: request per-step logprobs, and fail a divergence only where the top-2 gap is strictly above zero. That still catches what the tests exist for. A rank that rebuilt its bitmasks wrongly, or a request anchored to the wrong place in the flat batch, applies a different adapter delta; that moves the whole distribution and separates the argmax, so it fails with a nonzero gap. Only coin flips pass. This is the reasoning CLAUDE.md section 9 already applies to the skinning and generation-equivalence tests. Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>

test_correctness_survives_cache_eviction never ran. It died at engine init: ValueError: To serve at least one request with the model's max seq len (4096), 0.31 GiB KV cache is needed, which is larger than the available KV cache memory (0.02 GiB). Based on the available memory, the estimated maximum model length is 256. _launch hardcoded max_model_len=4096 while the test passed --blocks 16, a 256-token cache. Threading max_model_len through so it can be pinned to capacity is necessary but not sufficient -- the test as written had an empty solution set. The arithmetic: vLLM will not start unless the cache holds one whole max_model_len request, so every servable turn satisfies capacity >= max_model_len >= prompt + max_tokens. Under PRESERVE, turn N's prompt *contains* turn N-1's transcript as its own prefix, i.e. the same blocks. So a single conversation the server can serve is one whose history still fits and nothing is ever evicted, while the first turn that would overflow is the turn the server refuses. There is no --blocks value in between. A second conversation with a different filler has a disjoint prefix and therefore its own blocks, so it applies pressure the first conversation did not create. Measured at turn 3 the two prompts are ~173 and ~148 tokens: each alone needs 189 of the 256 available and stays servable, while together they need ~321 and evict each other. The run now reports "eviction bit on ['turn 1/A']" with a 16-block shortfall against the block-math ceiling, and every turn's decode still matches its expected adapter index -- which is the claim: pressure costs recomputation, not correctness. Per-turn records carry a conv label so the printed table and the assertion messages say which conversation a turn belongs to. Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>

ItzikVa and others added 11 commits October 25, 2026 09:22

ItzikVa force-pushed the docs/skipped-test-inventory branch from 4317eba to cd71021 Compare October 25, 2026 07:22

AlonMalach mentioned this pull request Oct 25, 2026

Trim trailing whitespace in docs/SWITCH_KERNELS.md #153

Merged

ItzikVa added 3 commits October 26, 2026 10:26

ItzikVa marked this pull request as ready for review October 26, 2026 12:11

ItzikVa requested review from antonpibm, aviv1ron1, freunda and yairallouche as code owners October 26, 2026 12:11

Navigation Menu

Repository navigation

Audit the suite's skips and fix what un-gating exposed - #152

Audit the suite's skips and fix what un-gating exposed#152
ItzikVa wants to merge 15 commits into
mainfrom
docs/skipped-test-inventory

ItzikVa commented Oct 25, 2026

Uh oh!

codecov-commenter commented Oct 25, 2026

Uh oh!

ItzikVa commented Oct 25, 2026

Uh oh!

github-actions Bot commented Oct 25, 2026

Uh oh!

github-actions Bot commented Oct 25, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ItzikVa commented Oct 25, 2026

Uh oh!

codecov-commenter commented Oct 25, 2026

Codecov Report

Uh oh!

ItzikVa commented Oct 25, 2026

Uh oh!

github-actions Bot commented Oct 25, 2026

❌ GPU tests failed — vllm26-multi

Uh oh!

github-actions Bot commented Oct 25, 2026

❌ GPU tests failed — vllm30-multi

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

❌ GPU tests failed — `vllm26-multi`

❌ GPU tests failed — `vllm30-multi`