Repository navigation
amd-kernels: tensor parallel 2 for MiniMax-M3.1-Flash on MI355X - #5
Fangzhou-Ai wants to merge 1 commit into
Conversation
The kernels vLLM runs for MiniMax-M3.1-Flash now also serve tensor parallel 2 (per GPU: 32 q / 2 KV / 2 index heads, QKV N 4992, o_proj K 4096, MoE I 1536, dense I 6144, lm_head N 100032, two-rank collectives over one xGMI link). New per-shape packages for the TP2 shapes and two-rank collectives; the head-count-generic packages (sparse attention, indexer, inserts, SWA decode, embedding gathers) gain tuned two-KV-head / two-rank paths. TP4 dispatch is unchanged except where a shared shape measured faster at every served row. Also: a split-K decode kernel kind for mid-M MXFP8 GEMMs, the full MoE shared expert on every TP2 rank with an FP4-only sequence-parallel gather and a second stream, and the SDMA transport's agents keyed by PCI domain. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
@Fangzhou-Ai Did the AgentX have cache hit rate? |
|
The TP2 moe kernels is duplicate a lot files. |
Their should have an entry file for moe so vLLM will not see the excat name and dimisions of the kernel. I will update that later. |
|
Yes I will release the cache hit rate table later, there are lots of duplicated files because I didn't add any abstractions here so kernel of a slightly different shape will copy most parts of what we already have feel free to trim it. |
Yeah, I think it's fine for no abstractions in kernels now. |
|
Wait for agent-v's review |
|
Thanks for the TP2 work. I ran the full suite of this branch (cbec2b0) and of main with the repo's 1. Alone it passed 13 times out of 13 (two nodes). In the failing run the other GPUs of the node were running other multi-GPU targets, including the four-rank handoffs tests. Wrong rows rather than a hang look like an ordering race in the SDMA hand-off that only shows under load. ( 2. The shared MXFP8 tuned table changes four-rank rows. The body says the TP4 dispatch is byte-identical, but
One table row per (M, N, K) cannot serve both callers. Fix in Fangzhou-Ai#1 (a PR to this branch): the table's rows for those two shapes are main's again, the TP2 down projection reads its own rows (this PR's) through a new scoped 3. 4. FYI, not blocking: two public names changed. Full-suite results: this branch 260 / 261 targets (13,571 tests, 1 failed: point 1); main 166 / 168. Of the three tests the body lists as failing on main too, A few test issues (warm-up checks that cannot fail, a skipped code comparison, a four-GPU file without a |
Summary
The kernels vLLM runs for MiniMax-M3.1-Flash on MI355X now also serve tensor parallel 2. Per GPU at TP2: 32 q / 2 KV / 2 index-q heads (+1 replicated index-K), QKV N 4992 (not 5120: index-K is one replicated head), o_proj K 4096, MoE I 1536, dense FFN I 6144, lm_head N 100032, and two-rank collectives over a single xGMI link.
gemm/mxfp8_n4992_k6144,gemm/mxfp8_n6144_k4096,gemm/mxfp8_gateup_swiglu_n12288_k6144_i6144,gemm/bf16_n100032_k6144,gemm/bf16_concat_linear_n3072_k30720moe/mxfp4_h6144_i1536_e128_top4,activation/swiglu_oai_mxfp8_i6144attention/mxfp8_qkv_norm_rope_nvfp4_insert_n4992_k6144_rot64_p128,decoder/dense_mxfp8_h6144_i6144_ranks2,norm/allreduce_gemma_bf16_mxfp8_h6144_ranks2sampling/batch_sharded_markov_v200064_h6144_r256_ranks2pull_collectives_ranks2,seq_parallel_handoffs_h6144_q4096_ranks2,sdma_push_ranks2,all_to_all_bf16_n100032_ranks2,all_gather_sampler_i64_c8_ranks2,all_gather_i64_c7_ranks2.allreduce_add_rmsnorm_ranks2);gemm/mxfp8_commonrows for every TP2 shape;decode_splitkkernel kind ingemm/mxfp8_common: split-K with a fixed-order in-kernel reduce. At decode M 64-256 it makes QKV 15-19% faster, o_proj 8-12% and the draft linears 4-13%. Rows switch only where interleaved A/B wins; QKV stays bit-identical to the deferred chain.sdma_push_ranks4keyed HSA agents by bus/device only. On nodes with two PCI domains, GPU 0 and GPU 4 collided, and SDMA copies went to the wrong agent. Agents are now keyed by (domain, bus, device).tests/multi_gpu.pyruns 2- or 4-rank bodies (run_on_2_gpus,run_on_gpus(world=));tools/ci/targets.tsvhas rows for the 2-GPU tests;amd_kernels/README.mdindexes the new packages.Performance
AgentX, one hour per point, TP2 on 2 MI355X per instance. Throughput mode is DSpark 7 at the golden acceptance length 4.04; KV offload is SimpleCPUOffload, 600 GB per instance; admission is 2× concurrency. The metrics are InferenceX's metric code (p90 interactivity = 1 / p90 ITL). These runs predate the mid-M GEMM and shared-expert changes above.
The image's stock TP2 path (most routes falling back to Triton/AITER) measured 58,972 tok/s per GPU and p90 34.9 at concurrency 32 (15-minute smoke). Per-kernel roofline tables are in each package README.
Tests
test_score_xcd,test_score_top16,test_shared_expert_rows) that return empty traces on the test host and fail the same way on unmodifiedmain.Not in this PR
The vLLM caller changes that dispatch to the TP2 packages live in the MiniMax-M3.1 model code in vLLM, not in this repo:
akr_moe,akr_arnorm,sample_akr,dspark_akr,dspark_akr_fusedandsp_prefill_akrpick the TP2 package by world size;🤖 Generated with Claude Code