Skip to content

fix: run paged decode on torch 2.10 CUDA with pages under 64 tokens (#294) - #297

Merged
gspark-etri merged 1 commit into
optimus-rmfrom
fix/294-flex-attention-cuda
Nov 4, 2026
Merged

gspark-etri merged 1 commit into
optimus-rmfrom
fix/294-flex-attention-cuda

Conversation

@gspark-etri

Copy link
Copy Markdown
Collaborator

Background

With a GPU visible, the three [cuda] cases of opt_prime/tests/test_pd_stream_admission.py failed on torch 2.10.0 with an inductor flex_attention LoweringException: NoValidChoicesError (#294). CI is CPU only, so it never ran them.

Root cause: a torch 2.10 inductor limitation that also hits real serving, not the test's tiny geometry. The opt_paged decode builds its block mask with BLOCK_SIZE=(128, page_size), and a one-token query lowers to inductor's flex decode kernel. In torch 2.10 that kernel's only default config (no max-autotune) is block_n=64, and create_flex_decoding_kernel drops every config where SPARSE_KV_BLOCK_SIZE % conf.block_n != 0. So any page size that is not a multiple of 64 leaves no choices. The test uses page_size=16 on CUDA. torch 2.11 clamps conf.block_n to SPARSE_KV_BLOCK_SIZE (checked in torch/_inductor/kernel/flex/flex_decoding.py at the v2.10.0, v2.11.0 and v2.12.0 tags), which is why torch 2.13 passes.

Standalone repro (bf16, GQA 4/2, flex_attention compiled with dynamic=False and the same block mask construction), on the 2x A100 development host:

page size torch 2.10.0+cu128 torch 2.13.0+cu130
8 NoValidChoicesError Triton CompilationError (tl.dot needs >= 16)
16 NoValidChoicesError (head_dim 16 and 128) OK
32 NoValidChoicesError OK
64, 128 OK OK

Production serving is affected on torch 2.10 whenever --page-size / page_size is not a multiple of 64 (for example 16, vLLM's default block size, which serve_batched.py also reports as the KV-event block_size). Without this fix a tiny Llama in ContinuousBatcher(kv="paged", page_size=16) on CUDA fails at its first decode step on torch 2.10. The default page_size=64 was never affected.

Changes

opt_prime/opt_prime/accel/pd/paged.py: on CUDA with torch < 2.11, if the block mask's KV block (the page size) is not a multiple of 64, _paged_attention passes kernel_options={"FORCE_USE_FLEX_ATTENTION": True, "BLOCK_N": page_size}. That makes inductor use the regular flex kernel, whose config filter respects a user BLOCK_N. Every other case (torch >= 2.11, page sizes that are multiples of 64, CPU eager) passes kernel_options=None as before. No test changes.

Verification

All runs on the 2x A100 development host with one GPU visible (CUDA_VISIBLE_DEVICES=0). Base is optimus-rm 9d70fe5, fix is this branch.

  1. Issue test: CUDA_VISIBLE_DEVICES=0 PYTHONPATH=opt_prime <py> -m pytest -q opt_prime/tests/test_pd_stream_admission.py
environment before after
torch 2.10.0+cu128 / transformers 5.1.0 (system python3) 3 failed, 5 passed 8 passed
torch 2.10 / transformers 5.8.1 3 failed, 5 passed 8 passed
torch 2.13 / transformers 5.15 8 passed 8 passed
  1. Serving path on CUDA (ad-hoc script, not committed): ContinuousBatcher(model.cuda(), kv="paged", slots=3, max_context=128, prefill_chunk=8, page_size=P), 5 prompts x 24 tokens, compared token for token with HF greedy full recompute on the same GPU. Families: Llama (dense GQA), Gemma 3 (sliding + full groups), Gemma 4 (K=V globals, MoE) and Qwen3.5 (Gated DeltaNet hybrid). Gemma 4 and Qwen3.5 need transformers >= 5.8.

    • After: 5/5 match for every family at P = 16, 32 and 64, on torch 2.10 (transformers 5.1.0 and 5.8.1) and on torch 2.13.
    • Before, on torch 2.10: NoValidChoicesError at P=16.
  2. Kernel cost of the fallback, from a microbenchmark of one decode attention call. Shapes: 8 slots x 2048 context, 32 q / 8 kv heads, head_dim 128, bf16, pages randomly scattered.

torch page size kernel time per call
2.10 64 decode kernel 141 us
2.10 32 regular kernel (this fix) 392 us
2.10 16 regular kernel (this fix) 512 us
2.13 16 decode kernel 142 us
  1. Full suites:
    • CPU suite as in .github/workflows/cpu-tests.yml: cd opt_prime && CUDA_VISIBLE_DEVICES= python3 -m pytest -q tests gives 576 passed, 34 skipped. CUDA_VISIBLE_DEVICES= python3 -m unittest discover -s opt_prime/examples/pd/native_kv -p 'test_*.py' gives 51 tests, OK (17 skipped).
    • Full suite with a GPU, system python3: cd opt_prime && CUDA_VISIBLE_DEVICES=0 python3 -m pytest -q tests. Before: 3 failed (the three tests: flex_attention NoValidChoicesError in test_pd_stream_admission CUDA cases on torch 2.10 #294 cases), 579 passed, 28 skipped. After: 582 passed, 28 skipped. There are no other GPU-only failures. The native_kv unittest suite with the GPU visible is also OK.

Compatibility and limits

  • Only CUDA runs on torch < 2.11 with page sizes that are not a multiple of 64 change behavior. Before this fix they could not compile at all. Torch >= 2.11, the default page size of 64 and CPU are unchanged.
  • On torch 2.10 the fallback kernel is about 2.8-3.6x slower per attention call than the 64-token decode kernel (table above). On torch 2.10, keep page_size=64 for best decode speed, or use torch >= 2.11.
  • Page sizes below 16 still cannot run on CUDA with any torch version tested: the Triton tl.dot minimum block is 16. That case is unchanged and out of scope.

Related Issue

Closes #294

🤖 Generated with Claude Code

…294)

torch 2.10's inductor flex decode kernel only has 64-wide KV block configs and skips any config that does not
divide the block mask's KV block (the page size), so `opt_paged` decode with page_size 16/32 failed to compile
with NoValidChoicesError (torch 2.11 clamps the block to the page). On torch < 2.11 such pages now run the
regular flex kernel with BLOCK_N = page size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gspark-etri
gspark-etri merged commit fff45de into optimus-rm Nov 4, 2026
1 check passed
@gspark-etri
gspark-etri deleted the fix/294-flex-attention-cuda branch November 4, 2026 04:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant