Repository navigation
indexer: half-run work split in the four-GPU decode scorer - #6
Merged
Merged
Conversation
The four-GPU decode scorer gave a workgroup one whole 16-block run per rank, or one 4-block unit. With 53 to 200 runs per rank that left most CUs idle (whole runs) or paid for 16-byte peer stores (units). A new half-run mode (8 blocks per item, two pages per wave, 32-byte peer stores) doubles the busy workgroups. The mode is chosen on the device from the sequence lengths. Scores are bitwise identical to main. - Three modes per call, chosen on the device from the rank's run count: units up to 52 runs, halves up to 200, whole runs above. Block placement, scores and slots are unchanged, so the output does not depend on the mode. - Half-run items stage through LDS with 8-byte writes and a rank stores 32 B per (head, row). Signed-off-by: chao-xu-spec <chaox@amd.com>
Member
|
Cool. Let me review and trigger a ci. |
Member
|
Thanks, I reran this on MI355X (4 GPUs, a ROCm 7.2 image with FlyDSL 0.5.0, the SDK the repo asks for) and the claims hold. LGTM from my side. Nothing below blocks the merge; all of it can go in a follow-up PR. Reproduced
Further gain (non-blocking, follow-up PR is fine)
Minor (non-blocking)
|
Member
|
LGTM. |
benenzhu
approved these changes
Nov 19, 2026
Member
|
wait for vllm-project/vllm#67850 to merge to vllm. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The four-GPU decode scorer (scores and top-16) is up to 1.20x faster (12 requests at 64K) where a rank has about 53-200 runs, and unchanged elsewhere; 5 of the 20 benchmark shapes read 0.4-0.7 % slower (listed below). Scores and ids are bitwise identical to main.
Changes
Validation
Setup: MI355X (gfx950), image
vllm/vllm-openai-rocm:minimax-m31-flash-75bc8cf1(vLLM 0.33.0rc1.dev910+g4fab5a074, amd-aiter 0.1.28.post1, flydsl 0.5.0, torch 2.13.0+rocm10.0.0, ROCm SDK 10.0.0 with HIP 7.15.26333; the repo asks for ROCm 7.2 and was not run on it). Model MiniMax-M3.1-Flash-Quark-MXFP4, vLLM TP4, NVFP4 KV and index cache.Microbench:
benchmark_rank_owned.py --decode 40,64,96,128 --ctx 32768,65536,131072,262144,524288 --ql 8(decode unit:score_topk, graph replay of at least 100 calls with their own pages, cold; time = the slowest of the four ranks), MI355X, 3 fresh processes per arm (arms alternated), median (min-max) of the processes' times in us. The benchmark's--refside could not be used (two copies of the package register the same warm-up entry), so each arm is a separate checkout.Slower than main: 5 requests at 32K 0.993x; 8 requests at 32K 0.995x; 12 requests at 32K 0.995x; 5 requests at 64K 0.994x; 12 requests at 512K 0.995x (0.04-0.14 us: the first four run in unit mode and the last in whole-run mode on both sides, so the new mode does not run there; the kernel with three modes compiled in is a little slower in the other two). On request counts and contexts of the AgentX c32 decode-time distribution (
--reqs 5,6,8,12,16 --lens-json) the gain is 1.032x at 5 requests and 1.023x at 6; 8 requests and more read 1.000-1.003x because their runs per rank are above 200.E2E: MiniMax-M3.1-Flash-Quark-MXFP4, TP4, NVFP4 KV, InferenceX AgentX trace replay (CI recipe, 3600 s client; mean ISL / OSL 167K / 1.17K at c8 and 182K / 1.08K at c32), 2 runs per arm, arms alternated over two lanes, noise floor = the spread of the two stock runs. The numbers are for a whole tree, not for this PR: the tree measured earlier is this series plus four further changes that are not in it (an 8-split table for the fused attention, a bf16 per-wave tail, an LDS stash in the four-GPU selector and a mixed split). Per-PR end-to-end runs were not done: the op share bounds the effect of each PR below the noise floor (sparse decode attention 9.04 % / 6.05 % and the decode scorer 5.21 % / 7.41 % of device time at c8 / c32).
Series tree, c8, 2 runs, against 2 stock runs of the same session:
Series tree, c8, against the two stock runs of the earlier session (different day, same recipe):
TPOT is reported in steps of 0.01 ms, so the p50 of both runs of an arm can be identical (stock 1.66 ms, ours 1.64 ms in all four runs of this session); the same-session runs of the two arms were interleaved (ours, stock, ours, stock) on the same four GPUs, except that the second stock run followed a host change.
Per-rank evidence (series arm, MI355X, TP4): the profiler windows of all four ranks show the edited kernels with equal call counts on every rank: split attention (MXFP8) 7260 calls per rank and four-GPU scorer + top-16 9000 per rank in the GSM8K windows; fused attention 14220 and tq512 prefill scorer 1980 per rank in three windows of the agentic replay (stock windows: 13980 and 1560). The overlay's per-process counters are equal on the four workers (index_sparse_decode 8,280 and index_prefill_top16 36,302 calls each in the agentic run) and every loaded kernel module comes from the patched tree. Kernel names are the same as on main by design, so the names alone do not tell the arms apart; durations from windows with different shape mixes are not a comparison, except the GSM8K windows (same prompts), where the split attention reads 9.5-9.6 us on stock and 8.5-8.8 us on the series tree on every rank.
Accuracy: GSM8K, vLLM
tests/evals/gsm8k/gsm8k_eval.py(chat endpoint, 5-shot, 1319 questions, up to 1024 new tokens, thinking off, concurrency 8, seed 42), against the TP4 server of the recipe with native block rejection, two server launches per arm. The first launch of the patched arm had a short profiler window during the eval. The eval's plain-completion mode (no chat template) scores about 13 % on this model (stock, one launch) and was not used.Run-to-run difference within an arm: stock 0.8 points, ours 0.2 points; stock mean 95.3 %, ours mean 95.5 %.
Tests (MI355X, run on this branch alone):
test_split4_rank_owned.py,test_topk.py,test_decode_top16.py(one GPU): 757 passed in 72.68s (0:01:12);test_split4_rank_owned_ranks4.py(four GPUs): 3 passed in 20.66s. The new tests put the per-rank run totals on both sides of the 52 / 53 and 200 / 201 boundaries, with rank 0 one run past the others (so ranks take different modes), on one GPU and on four GPUs with a graph replay.The new tests also pass on main (main + the series' tests only: fused / decode block / sparse decode: 560 passed in 52.95s; split4 + prefill scorer: 144 passed in 15.88s; four-GPU: 3 passed in 21.08s), so they check the existing contract, not the change.
Details