Repository navigation
fix: run paged decode on torch 2.10 CUDA with pages under 64 tokens (#294) - #297
Merged
Merged
Conversation
…294) torch 2.10's inductor flex decode kernel only has 64-wide KV block configs and skips any config that does not divide the block mask's KV block (the page size), so `opt_paged` decode with page_size 16/32 failed to compile with NoValidChoicesError (torch 2.11 clamps the block to the page). On torch < 2.11 such pages now run the regular flex kernel with BLOCK_N = page size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
With a GPU visible, the three
[cuda]cases ofopt_prime/tests/test_pd_stream_admission.pyfailed on torch 2.10.0 with an inductorflex_attentionLoweringException: NoValidChoicesError(#294). CI is CPU only, so it never ran them.Root cause: a torch 2.10 inductor limitation that also hits real serving, not the test's tiny geometry. The
opt_pageddecode builds its block mask withBLOCK_SIZE=(128, page_size), and a one-token query lowers to inductor's flex decode kernel. In torch 2.10 that kernel's only default config (no max-autotune) isblock_n=64, andcreate_flex_decoding_kerneldrops every config whereSPARSE_KV_BLOCK_SIZE % conf.block_n != 0. So any page size that is not a multiple of 64 leaves no choices. The test usespage_size=16on CUDA. torch 2.11 clampsconf.block_ntoSPARSE_KV_BLOCK_SIZE(checked intorch/_inductor/kernel/flex/flex_decoding.pyat the v2.10.0, v2.11.0 and v2.12.0 tags), which is why torch 2.13 passes.Standalone repro (bf16, GQA 4/2, flex_attention compiled with
dynamic=Falseand the same block mask construction), on the 2x A100 development host:Production serving is affected on torch 2.10 whenever
--page-size/page_sizeis not a multiple of 64 (for example 16, vLLM's default block size, whichserve_batched.pyalso reports as the KV-eventblock_size). Without this fix a tiny Llama inContinuousBatcher(kv="paged", page_size=16)on CUDA fails at its first decode step on torch 2.10. The defaultpage_size=64was never affected.Changes
opt_prime/opt_prime/accel/pd/paged.py: on CUDA with torch < 2.11, if the block mask's KV block (the page size) is not a multiple of 64,_paged_attentionpasseskernel_options={"FORCE_USE_FLEX_ATTENTION": True, "BLOCK_N": page_size}. That makes inductor use the regular flex kernel, whose config filter respects a userBLOCK_N. Every other case (torch >= 2.11, page sizes that are multiples of 64, CPU eager) passeskernel_options=Noneas before. No test changes.Verification
All runs on the 2x A100 development host with one GPU visible (
CUDA_VISIBLE_DEVICES=0). Base isoptimus-rm9d70fe5, fix is this branch.CUDA_VISIBLE_DEVICES=0 PYTHONPATH=opt_prime <py> -m pytest -q opt_prime/tests/test_pd_stream_admission.pyServing path on CUDA (ad-hoc script, not committed):
ContinuousBatcher(model.cuda(), kv="paged", slots=3, max_context=128, prefill_chunk=8, page_size=P), 5 prompts x 24 tokens, compared token for token with HF greedy full recompute on the same GPU. Families: Llama (dense GQA), Gemma 3 (sliding + full groups), Gemma 4 (K=V globals, MoE) and Qwen3.5 (Gated DeltaNet hybrid). Gemma 4 and Qwen3.5 need transformers >= 5.8.Kernel cost of the fallback, from a microbenchmark of one decode attention call. Shapes: 8 slots x 2048 context, 32 q / 8 kv heads, head_dim 128, bf16, pages randomly scattered.
.github/workflows/cpu-tests.yml:cd opt_prime && CUDA_VISIBLE_DEVICES= python3 -m pytest -q testsgives 576 passed, 34 skipped.CUDA_VISIBLE_DEVICES= python3 -m unittest discover -s opt_prime/examples/pd/native_kv -p 'test_*.py'gives 51 tests, OK (17 skipped).cd opt_prime && CUDA_VISIBLE_DEVICES=0 python3 -m pytest -q tests. Before: 3 failed (the three tests: flex_attention NoValidChoicesError in test_pd_stream_admission CUDA cases on torch 2.10 #294 cases), 579 passed, 28 skipped. After: 582 passed, 28 skipped. There are no other GPU-only failures. The native_kv unittest suite with the GPU visible is also OK.Compatibility and limits
page_size=64for best decode speed, or use torch >= 2.11.tl.dotminimum block is 16. That case is unchanged and out of scope.Related Issue
Closes #294
🤖 Generated with Claude Code