Skip to content

indexer: half-run work split in the four-GPU decode scorer - #6

Merged
benenzhu merged 1 commit into
Inferact:mainfrom
chao-xu-spec:pr/03-split4-half-runs
Nov 19, 2026
Merged

benenzhu merged 1 commit into
Inferact:mainfrom
chao-xu-spec:pr/03-split4-half-runs

Conversation

@chao-xu-spec

Copy link
Copy Markdown
Contributor

Summary

The four-GPU decode scorer (scores and top-16) is up to 1.20x faster (12 requests at 64K) where a rank has about 53-200 runs, and unchanged elsewhere; 5 of the 20 benchmark shapes read 0.4-0.7 % slower (listed below). Scores and ids are bitwise identical to main.

Changes

  • Three modes per call, chosen on the device from the rank's run count: units up to 52 runs, halves up to 200, whole runs above. Block placement, scores and slots are unchanged, so the output does not depend on the mode.
  • Half-run items stage through LDS with 8-byte writes and a rank stores 32 B per (head, row).

Validation

Setup: MI355X (gfx950), image vllm/vllm-openai-rocm:minimax-m31-flash-75bc8cf1 (vLLM 0.33.0rc1.dev910+g4fab5a074, amd-aiter 0.1.28.post1, flydsl 0.5.0, torch 2.13.0+rocm10.0.0, ROCm SDK 10.0.0 with HIP 7.15.26333; the repo asks for ROCm 7.2 and was not run on it). Model MiniMax-M3.1-Flash-Quark-MXFP4, vLLM TP4, NVFP4 KV and index cache.

Microbench: benchmark_rank_owned.py --decode 40,64,96,128 --ctx 32768,65536,131072,262144,524288 --ql 8 (decode unit: score_topk, graph replay of at least 100 calls with their own pages, cold; time = the slowest of the four ranks), MI355X, 3 fresh processes per arm (arms alternated), median (min-max) of the processes' times in us. The benchmark's --ref side could not be used (two copies of the package register the same warm-up entry), so each arm is a separate checkout.

Requests (rows) Context Stock (us) Ours (us) Speedup
5 (40) 32K 10.00 (9.99-10.06) 10.07 (10.06-10.07) 0.993x
8 (64) 32K 10.47 (10.46-10.48) 10.52 (10.51-10.60) 0.995x
12 (96) 32K 11.28 (11.28-11.30) 11.34 (11.34-11.38) 0.995x
16 (128) 32K 12.43 (12.43-12.43) 11.99 (11.96-11.99) 1.037x
5 (40) 64K 10.82 (10.80-10.84) 10.89 (10.88-10.89) 0.994x
8 (64) 64K 12.37 (12.37-12.37) 11.54 (11.54-11.55) 1.072x
12 (96) 64K 14.51 (14.51-14.52) 12.14 (12.12-12.20) 1.195x
16 (128) 64K 14.77 (14.77-14.78) 13.11 (13.06-13.17) 1.127x
5 (40) 128K 14.10 (14.09-14.14) 11.82 (11.81-11.82) 1.193x
8 (64) 128K 14.38 (14.37-14.39) 13.00 (12.99-13.00) 1.106x
12 (96) 128K 15.21 (15.16-15.23) 14.85 (14.85-14.91) 1.024x
16 (128) 128K 16.27 (16.25-16.28) 16.22 (16.21-16.29) 1.003x
5 (40) 256K 15.94 (15.89-16.00) 15.89 (15.88-15.90) 1.003x
8 (64) 256K 17.26 (17.26-17.26) 17.25 (17.23-17.25) 1.001x
12 (96) 256K 20.74 (20.74-20.75) 20.68 (20.66-20.72) 1.003x
16 (128) 256K 23.02 (22.98-23.07) 22.97 (22.95-23.08) 1.002x
5 (40) 512K 20.88 (20.88-20.92) 20.81 (20.79-20.88) 1.003x
8 (64) 512K 23.54 (23.45-23.59) 23.42 (23.39-23.48) 1.005x
12 (96) 512K 30.22 (30.20-30.24) 30.36 (30.33-30.38) 0.995x
16 (128) 512K 37.57 (37.57-37.70) 37.57 (37.51-37.61) 1.000x

Slower than main: 5 requests at 32K 0.993x; 8 requests at 32K 0.995x; 12 requests at 32K 0.995x; 5 requests at 64K 0.994x; 12 requests at 512K 0.995x (0.04-0.14 us: the first four run in unit mode and the last in whole-run mode on both sides, so the new mode does not run there; the kernel with three modes compiled in is a little slower in the other two). On request counts and contexts of the AgentX c32 decode-time distribution (--reqs 5,6,8,12,16 --lens-json) the gain is 1.032x at 5 requests and 1.023x at 6; 8 requests and more read 1.000-1.003x because their runs per rank are above 200.

E2E: MiniMax-M3.1-Flash-Quark-MXFP4, TP4, NVFP4 KV, InferenceX AgentX trace replay (CI recipe, 3600 s client; mean ISL / OSL 167K / 1.17K at c8 and 182K / 1.08K at c32), 2 runs per arm, arms alternated over two lanes, noise floor = the spread of the two stock runs. The numbers are for a whole tree, not for this PR: the tree measured earlier is this series plus four further changes that are not in it (an 8-split table for the fused attention, a bf16 per-wave tail, an LDS stash in the four-GPU selector and a mixed split). Per-PR end-to-end runs were not done: the op share bounds the effect of each PR below the noise floor (sparse decode attention 9.04 % / 6.05 % and the decode scorer 5.21 % / 7.41 % of device time at c8 / c32).

Concurrency Metric Stock, 2 runs Whole tree, 2 runs Change Noise floor (stock vs stock)
c8 total tok/s/GPU 16845 / 17109 16978 / 16989 +0.04 % 1.56 %
c8 TPOT p50 (ms) 1.66 / 1.67 1.63 / 1.61 -2.70 % 0.60 %
c8 interactivity p50 (1/TPOT) 602.4 / 598.8 613.5 / 621.1 +2.78 % 0.60 %
c8 interactivity p90 (1/TPOT) 471.7 / 469.5 478.5 / 485.4 +2.41 % 0.47 %
c32 total tok/s/GPU 62704 / 62589 62679 / 62643 +0.02 % 0.18 %
c32 TPOT p50 (ms) 2.67 / 2.65 2.61 / 2.64 -1.32 % 0.75 %
c32 interactivity p50 (1/TPOT) 374.5 / 377.4 383.1 / 378.8 +1.34 % 0.75 %
c32 interactivity p90 (1/TPOT) 212.3 / 213.7 218.3 / 214.6 +1.63 % 0.64 %

Series tree, c8, 2 runs, against 2 stock runs of the same session:

Metric Stock Ours Change Noise floor (stock spread)
total tok/s/GPU 16983 / 17096 16961 / 17110 -0.02 % 0.67 %
TPOT p50 (ms) 1.66 / 1.66 1.64 / 1.64 -1.20 % 0.00 %
interactivity p50 (1/TPOT) 602.4 / 602.4 609.8 / 609.8 +1.22 % 0.00 %
interactivity p90 (1/TPOT) 471.7 / 465.1 480.8 / 485.4 +3.14 % 1.41 %

Series tree, c8, against the two stock runs of the earlier session (different day, same recipe):

Metric Stock Ours Change Noise floor (stock spread)
total tok/s/GPU 16845 / 17109 16961 / 17110 +0.35 % 1.56 %
TPOT p50 (ms) 1.66 / 1.67 1.64 / 1.64 -1.50 % 0.60 %
interactivity p50 (1/TPOT) 602.4 / 598.8 609.8 / 609.8 +1.52 % 0.60 %
interactivity p90 (1/TPOT) 471.7 / 469.5 480.8 / 485.4 +2.66 % 0.47 %

TPOT is reported in steps of 0.01 ms, so the p50 of both runs of an arm can be identical (stock 1.66 ms, ours 1.64 ms in all four runs of this session); the same-session runs of the two arms were interleaved (ours, stock, ours, stock) on the same four GPUs, except that the second stock run followed a host change.

Per-rank evidence (series arm, MI355X, TP4): the profiler windows of all four ranks show the edited kernels with equal call counts on every rank: split attention (MXFP8) 7260 calls per rank and four-GPU scorer + top-16 9000 per rank in the GSM8K windows; fused attention 14220 and tq512 prefill scorer 1980 per rank in three windows of the agentic replay (stock windows: 13980 and 1560). The overlay's per-process counters are equal on the four workers (index_sparse_decode 8,280 and index_prefill_top16 36,302 calls each in the agentic run) and every loaded kernel module comes from the patched tree. Kernel names are the same as on main by design, so the names alone do not tell the arms apart; durations from windows with different shape mixes are not a comparison, except the GSM8K windows (same prompts), where the split attention reads 9.5-9.6 us on stock and 8.5-8.8 us on the series tree on every rank.

Accuracy: GSM8K, vLLM tests/evals/gsm8k/gsm8k_eval.py (chat endpoint, 5-shot, 1319 questions, up to 1024 new tokens, thinking off, concurrency 8, seed 42), against the TP4 server of the recipe with native block rejection, two server launches per arm. The first launch of the patched arm had a short profiler window during the eval. The eval's plain-completion mode (no chat template) scores about 13 % on this model (stock, one launch) and was not used.

Stock, 2 launches Ours, 2 launches
Accuracy (1319 questions) 95.7 % / 94.9 % 95.4 % / 95.5 %
Invalid responses 0.8 % / 1.4 % 1.1 % / 0.9 %
Mean accept length (DSpark 7) 4.66 / 4.65 4.64 / 4.64

Run-to-run difference within an arm: stock 0.8 points, ours 0.2 points; stock mean 95.3 %, ours mean 95.5 %.

Tests (MI355X, run on this branch alone): test_split4_rank_owned.py, test_topk.py, test_decode_top16.py (one GPU): 757 passed in 72.68s (0:01:12); test_split4_rank_owned_ranks4.py (four GPUs): 3 passed in 20.66s. The new tests put the per-rank run totals on both sides of the 52 / 53 and 200 / 201 boundaries, with rank 0 one run past the others (so ranks take different modes), on one GPU and on four GPUs with a graph replay.
The new tests also pass on main (main + the series' tests only: fused / decode block / sparse decode: 560 passed in 52.95s; split4 + prefill scorer: 144 passed in 15.88s; four-GPU: 3 passed in 21.08s), so they check the existing contract, not the change.

# tests (the *_ranks4 files need four GPUs)
python3 -m pytest tests/indexer/indexer_nvfp4_d128_p128_top16 tests/attention/nvfp4_sparse_decode_gqa16_d128_p128_top16 -q

# microbench: run in a main checkout and in this branch, in fresh processes
python3 benchmarks/indexer/indexer_nvfp4_d128_p128_top16/benchmark_indexer.py --prefill
python3 benchmarks/attention/nvfp4_sparse_decode_gqa16_d128_p128_top16/benchmark_sparse_decode.py --configs kv1_f16_ql8 --requests 5 6 8 12 16 32 64 128 256 --outputs mxfp8 bf16
HIP_VISIBLE_DEVICES=<4 GPUs> python3 benchmarks/attention/nvfp4_sparse_decode_gqa16_d128_p128_top16/benchmark_index_decode_ranks4.py --ql 8 --reqs 1,2,3,4,5,6,8,12,16 [--lens-json <decode-time contexts>]
HIP_VISIBLE_DEVICES=<4 GPUs> python3 benchmarks/indexer/indexer_nvfp4_d128_p128_top16/benchmark_rank_owned.py --decode 40,64,96,128 --ctx 32768,65536,131072,262144,524288 --ql 8

# server: TP4 InferenceX recipe (c8), this branch first on PYTHONPATH for the patched arm
PYTHONPATH=<checkout> vllm serve <model path> --host 0.0.0.0 --port <port> --tensor-parallel-size 4 --attention-config '{"indexer_kv_dtype":"nvfp4","indexer_decode_cp":true}' \
  --block-size 128 --kv-cache-dtype nvfp4 --max-model-len 1048576 --max-num-batched-tokens 32768 --max-num-seqs 16 --gpu-memory-utilization 0.85 \
  --compilation-config '{"max_cudagraph_capture_size":2048}' --enable-dbo --dbo-decode-token-threshold 32768 --dbo-prefill-token-threshold 16384 \
  --enable-prefix-caching --enable-batch-sharded-sampling --moe-backend aiter --language-model-only --trust-remote-code --disable-hybrid-kv-cache-manager \
  --speculative-config '{"method": "dspark", "num_speculative_tokens": 7, "attention_backend": "TRITON_ATTN", "kv_cache_dtype": "fp8", "draft_sample_method": "probabilistic", "rejection_sample_method": "block"}'
# (the throughput runs use "rejection_sample_method": "synthetic", "synthetic_acceptance_length": 4.04, as the recipe does)

# accuracy: vLLM's tests/evals/gsm8k, chat endpoint, 5-shot, thinking off
python3 tests/evals/gsm8k/gsm8k_eval.py --port <port> --num-questions 1319 --num-shots 5 --max-tokens 1024 --use-chat-completions \
  --chat-template-kwargs '{"enable_thinking": false}' --max-concurrency 8 --seed 42 --save-results gsm8k.json

# throughput / latency: the InferenceX AgentX client of the same recipe, 3600 s at concurrency 8 and 32

Details

  • Guard: The mode is a function of the device-side lengths only; ranks may take different modes because each rank writes only its own slots. Tests put the run counts on both sides of both boundaries, on one GPU and on four.
  • Compile guard: the four-GPU scorer, its compile entry and the compile tests (76 passed on this branch) compile 60 kernel variants; 48 compile to the same cached module as on main (the cache key is the traced MLIR), and the 12 that differ are the four-GPU decode scorers (push and fused top-16, 16 / 24 / 32 rows). The one-GPU scorers and the top-k kernels are unchanged.
  • The gain is where a rank has 53-200 runs (for example 5 requests of 128K tokens or 12 requests of 64K); with more runs (longer contexts, more requests) whole runs are kept and nothing changes, and with 52 or fewer the unit mode is as on main.
  • The four-rank scorer spins until every rank has called; this change does not alter that contract.
  • Roofline: Latency-bound: a call is 10-40 us and moves a few MB per rank; the gain comes from filling more workgroups, not from bandwidth.

The four-GPU decode scorer gave a workgroup one whole 16-block run per rank, or one 4-block unit. With 53 to 200 runs
per rank that left most CUs idle (whole runs) or paid for 16-byte peer stores (units). A new half-run mode (8 blocks
per item, two pages per wave, 32-byte peer stores) doubles the busy workgroups. The mode is chosen on the device from
the sequence lengths. Scores are bitwise identical to main.

- Three modes per call, chosen on the device from the rank's run count: units up to 52 runs, halves up to 200, whole
  runs above. Block placement, scores and slots are unchanged, so the output does not depend on the mode.
- Half-run items stage through LDS with 8-byte writes and a rank stores 32 B per (head, row).

Signed-off-by: chao-xu-spec <chaox@amd.com>
@benenzhu

Copy link
Copy Markdown
Member

Cool. Let me review and trigger a ci.

@benenzhu

Copy link
Copy Markdown
Member

Thanks, I reran this on MI355X (4 GPUs, a ROCm 7.2 image with FlyDSL 0.5.0, the SDK the repo asks for) and the claims hold. LGTM from my side. Nothing below blocks the merge; all of it can go in a follow-up PR.

Reproduced

  • Tests: test_split4_rank_owned.py, test_topk.py, test_decode_top16.py: 757 passed. test_split4_rank_owned_ranks4.py: 3 passed.
  • The new tests on main: also pass (one GPU and four GPUs), so they check the existing contract, as you wrote.
  • Compile tests (test_compile*.py of the indexer): 31 passed, 1 xfailed (the existing strict xfail).
  • Microbench, your command, main and PR alternated, 3 fresh processes each, same 4 GPUs: main/PR is within 0.02 of your table on all 20 shapes. Examples: 12 req @ 64K 1.200x (yours 1.195x), 5 req @ 128K 1.207x (yours 1.193x). Largest gaps: 8 req @ 64K 1.052x (yours 1.072x), 16 req @ 32K 1.019x (yours 1.037x). The five slower shapes: 0.991-0.998x.
  • ISA: VGPRs +1..+2 per scorer (e.g. 185 -> 187), no occupancy change. Instruction count +21..+45% (fused top-16, 32 rows: 5688 -> 7003). This fits the small loss in the unchanged modes.

Further gain (non-blocking, follow-up PR is fine)

  • HALF_MAX can go up to about 432 (216 runs). I timed copies that change only the constant. Halves still beat whole runs by 1.3-1.8% at 204-210 runs per rank (e.g. 12 requests x 17 chunks: 15.55 -> 15.28 us). At 224 runs the two tie. The lower edge is right: halves win by 2.1% at 56 runs and 5.3% at 64, and lose 0.3% at 54.

Minor (non-blocking)

  1. The boundary tests use 6-13 requests, so they only reach the 32-request (_n32) build. Please add a half-mode case with more than 32 requests, for the 256-request build. Examples: 53 requests x 1 chunk; 100 x 2 chunks with request 0 one block longer (ranks at 200 / 201); 64 x 3 chunks at ql 4 (grid 768). I ran 22 such cases (these, plus random lengths with 33-48 requests). They pass on this branch and on main.
  2. Three comments describe only units and whole runs: the page mapping in scan (page s is item pid + (s / qdv) * grid's ...), the poll (lane i polls source i % 4 ...) and the LDS stage (a unit's 4 B or a run quarter's 16 B per wave ...).
  3. Nits: UNIT_MAX counts units and HALF_MAX counts halves. Both in runs (52, 200) would read more easily. k = it >> (2 - qsh) assumes QD == 4.

@benenzhu

Copy link
Copy Markdown
Member

LGTM.

@benenzhu
benenzhu merged commit 3d62453 into Inferact:main Nov 19, 2026
1 of 3 checks passed
@benenzhu

Copy link
Copy Markdown
Member

wait for vllm-project/vllm#67850 to merge to vllm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants