Skip to content

amd-kernels: tensor parallel 2 for MiniMax-M3.1-Flash on MI355X - #5

Open
Fangzhou-Ai wants to merge 1 commit into
Inferact:mainfrom
Fangzhou-Ai:tp2-minimax-m31
Open

Fangzhou-Ai wants to merge 1 commit into
Inferact:mainfrom
Fangzhou-Ai:tp2-minimax-m31

Conversation

@Fangzhou-Ai

Copy link
Copy Markdown
Collaborator

Summary

The kernels vLLM runs for MiniMax-M3.1-Flash on MI355X now also serve tensor parallel 2. Per GPU at TP2: 32 q / 2 KV / 2 index-q heads (+1 replicated index-K), QKV N 4992 (not 5120: index-K is one replicated head), o_proj K 4096, MoE I 1536, dense FFN I 6144, lm_head N 100032, and two-rank collectives over a single xGMI link.

  • New per-shape packages (kernel rule 5; TP4 packages untouched):
    • GEMM: gemm/mxfp8_n4992_k6144, gemm/mxfp8_n6144_k4096, gemm/mxfp8_gateup_swiglu_n12288_k6144_i6144, gemm/bf16_n100032_k6144, gemm/bf16_concat_linear_n3072_k30720
    • MoE and activation: moe/mxfp4_h6144_i1536_e128_top4, activation/swiglu_oai_mxfp8_i6144
    • Attention, decoder and norm: attention/mxfp8_qkv_norm_rope_nvfp4_insert_n4992_k6144_rot64_p128, decoder/dense_mxfp8_h6144_i6144_ranks2, norm/allreduce_gemma_bf16_mxfp8_h6144_ranks2
    • Sampling: sampling/batch_sharded_markov_v200064_h6144_r256_ranks2
    • Two-rank collectives: pull_collectives_ranks2, seq_parallel_handoffs_h6144_q4096_ranks2, sdma_push_ranks2, all_to_all_bf16_n100032_ranks2, all_gather_sampler_i64_c8_ranks2, all_gather_i64_c7_ranks2.
  • Head-count-generic packages gain tuned TP2 paths, with TP4 dispatch byte-identical:
    • sparse decode and prefill attention, the sparse layer, the NVFP4 and fp8 inserts, SWA decode;
    • the indexer: two-head scorers, plus a decode split over the two ranks;
    • the embedding gathers at two ranks;
    • the all-reduce + norm (allreduce_add_rmsnorm_ranks2);
    • gemm/mxfp8_common rows for every TP2 shape;
    • top-p above 64 rows (bit for bit with vLLM's kernel).
  • Mid-M decode GEMMs. A new decode_splitk kernel kind in gemm/mxfp8_common: split-K with a fixed-order in-kernel reduce. At decode M 64-256 it makes QKV 15-19% faster, o_proj 8-12% and the draft linears 4-13%. Rows switch only where interleaved A/B wins; QKV stays bit-identical to the deferred chain.
  • MoE shared expert at TP2. Each rank holds the full shared-expert copy.
    • Decode: reads its TP shard as a view of that copy, bit-identical to the sharded path.
    • Sequence-parallel prefill: each rank runs the shared expert on its own rows on a second stream; its output stays local and is added in the reduce-scatter epilogue. The all-gather carries only the routed FP4 payload, 3296 instead of 7328 B/token.
    • Effect: SP layer −7..−14%; memory cost +1.55 GiB per GPU.
  • Fix: sdma_push_ranks4 keyed HSA agents by bus/device only. On nodes with two PCI domains, GPU 0 and GPU 4 collided, and SDMA copies went to the wrong agent. Agents are now keyed by (domain, bus, device).
  • Infra: tests/multi_gpu.py runs 2- or 4-rank bodies (run_on_2_gpus, run_on_gpus(world=)); tools/ci/targets.tsv has rows for the 2-GPU tests; amd_kernels/README.md indexes the new packages.

Performance

AgentX, one hour per point, TP2 on 2 MI355X per instance. Throughput mode is DSpark 7 at the golden acceptance length 4.04; KV offload is SimpleCPUOffload, 600 GB per instance; admission is 2× concurrency. The metrics are InferenceX's metric code (p90 interactivity = 1 / p90 ITL). These runs predate the mid-M GEMM and shared-expert changes above.

concurrency total tok/s per GPU p90 interactivity (tok/s/user) TTFT p90
32 112,417 111.9 0.76 s
40 119,709 99.4 1.41 s
48 120,495 68.8 9.89 s

The image's stock TP2 path (most routes falling back to Triton/AITER) measured 58,972 tok/s per GPU and p90 34.9 at concurrency 32 (15-minute smoke). Per-kernel roofline tables are in each package README.

Tests

  • Full suite: 258 of 261 test files pass, TP4 and TP2 on 8 GPUs. The 3 failures are torch-profiler kernel-name checks (test_score_xcd, test_score_top16, test_shared_expert_rows) that return empty traces on the test host and fail the same way on unmodified main.
  • Accuracy: GSM8K, full 1319, 5-shot, real DSpark block rejection.
    • TP2: 0.9598-0.9666 strict across the final builds, against 0.9613 for the stock TP2 image.
    • TP4 on this tree: 0.985 on the first 400.

Not in this PR

The vLLM caller changes that dispatch to the TP2 packages live in the MiniMax-M3.1 model code in vLLM, not in this repo:

  • akr_moe, akr_arnorm, sample_akr, dspark_akr, dspark_akr_fused and sp_prefill_akr pick the TP2 package by world size;
  • the overlay holds the shared expert replicated at TP2.

🤖 Generated with Claude Code

The kernels vLLM runs for MiniMax-M3.1-Flash now also serve tensor parallel 2 (per GPU: 32 q / 2 KV / 2 index heads,
QKV N 4992, o_proj K 4096, MoE I 1536, dense I 6144, lm_head N 100032, two-rank collectives over one xGMI link).
New per-shape packages for the TP2 shapes and two-rank collectives; the head-count-generic packages (sparse
attention, indexer, inserts, SWA decode, embedding gathers) gain tuned two-KV-head / two-rank paths. TP4 dispatch
is unchanged except where a shared shape measured faster at every served row.

Also: a split-K decode kernel kind for mid-M MXFP8 GEMMs, the full MoE shared expert on every TP2 rank with an
FP4-only sequence-parallel gather and a second stream, and the SDMA transport's agents keyed by PCI domain.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@benenzhu

Copy link
Copy Markdown
Member

@Fangzhou-Ai Did the AgentX have cache hit rate?

@benenzhu

Copy link
Copy Markdown
Member

The TP2 moe kernels is duplicate a lot files.
May have a common on later.

@benenzhu

benenzhu commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

The TP2 moe kernels is duplicate a lot files. May have a common on later.

Their should have an entry file for moe so vLLM will not see the excat name and dimisions of the kernel. I will update that later.

@Fangzhou-Ai

Copy link
Copy Markdown
Collaborator Author

Yes I will release the cache hit rate table later, there are lots of duplicated files because I didn't add any abstractions here so kernel of a slightly different shape will copy most parts of what we already have feel free to trim it.

@benenzhu

Copy link
Copy Markdown
Member

Yes I will release the cache hit rate table later, there are lots of duplicated files because I didn't add any abstractions here so kernel of a slightly different shape will copy most parts of what we already have feel free to trim it.

Yeah, I think it's fine for no abstractions in kernels now.

@benenzhu

Copy link
Copy Markdown
Member

Wait for agent-v's review

@benenzhu

Copy link
Copy Markdown
Member

Thanks for the TP2 work. I ran the full suite of this branch (cbec2b0) and of main with the repo's tools/ci on 8 x MI355X (ROCm 7.2 image, FlyDSL 0.5.0) and reviewed the diff against the TP4 sources. Four points before merge:

1. test_handoffs_sdma_bitwise gave wrong rows once (please look).
tests/collectives/seq_parallel_handoffs_h6144_q4096_ranks2/test_handoffs.py::test_handoffs_sdma_bitwise[tokens0] failed in the full run:

rank 1 t 2048 owner slots reduce-scatter: rows [0, 1, 2, 3, 4, 5] differ
rank 1 t 4096 slots reduce-scatter: rows [0, 1, 2, 3, 4, 5] differ

Alone it passed 13 times out of 13 (two nodes). In the failing run the other GPUs of the node were running other multi-GPU targets, including the four-rank handoffs tests. Wrong rows rather than a hang look like an ordering race in the SDMA hand-off that only shows under load. (sdma_push_ranks2/hsa_runtime.py already keys agents by PCI domain, so it is not the agent mix-up sdma_push_ranks4 had.)

2. The shared MXFP8 tuned table changes four-rank rows. The body says the TP4 dispatch is byte-identical, but mxfp8_gemm_tuned.py changes two shapes TP4 runs:

  • 6144 x 6144 (TP4 dense gate_up; also the TP2 dense down): every bf16 row from 33 to 1280. 33-256 become decode_splitk with K split 2, so the TP4 gate_up's bits change there (still deterministic).
  • 2560 x 6144 (TP4 QKV, not used at TP2): the 2049-3072 row, tile 128 x 256 -> 256 x 128.

One table row per (M, N, K) cannot serve both callers. Fix in Fangzhou-Ai#1 (a PR to this branch): the table's rows for those two shapes are main's again, the TP2 down projection reads its own rows (this PR's) through a new scoped gemm.tuned_rows(...), and main's test_gateup_decode_rows.py is back. The 2560 x 6144 tuning can come back in its own PR with TP4 numbers.

3. shared_expert_full raises TypeError on a first call with keep=. _fork_shared (collectives/seq_parallel_handoffs_h6144_q4096_ranks2/handoffs.py:126) calls it with stream=side, keep=keep; the warm-up entry passes the call's kwargs to _full_inputs (moe/mxfp4_h6144_i1536_e128_top4/shared_expert.py:162), which takes stream only. Without an explicit warm-up first (SeqParallelHandoffs.warm_up / warm_up_mxfp4_moe(sp=True)), the first SP MoE call raises, and so does every later one (the mode never counts as warmed). The tests run with auto warm-up off, so they do not see it. Same PR: _full_inputs(..., keep=None).

4. FYI, not blocking: two public names changed. nvfp4_sparse_layer_gqa16_d128_p128_top16.layer.QUANT_OUT_MIN_ROWS is now a dict (still in __all__), and nvfp4_sparse_decode_gqa16_d128_p128_top16.index_decode.FUSED_MAX_ROWS is gone (now host.SELECT_MAX_ROWS). Fine if no caller outside the repo reads them.

Full-suite results: this branch 260 / 261 targets (13,571 tests, 1 failed: point 1); main 166 / 168. Of the three tests the body lists as failing on main too, test_score_xcd and test_score_top16 pass on both here, and test_shared_expert_rows failed on main and passed on this branch (a flaky profiler check).

A few test issues (warm-up checks that cannot fail, a skipped code comparison, a four-GPU file without a targets.tsv row, allocator-dependent pool checks) will come as a separate PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants