Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
89 changes: 66 additions & 23 deletions amd_kernels/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,60 +5,82 @@ One line per kernel package. Each package's README has what it does, its entry p
## GEMM

- [`gemm/mxfp8_common`](gemm/mxfp8_common/README.md): MXFP8 dense GEMM entries (`mxfp8_linear`, `mxfp8_gemm`, ...),
the shape -> kernel selection and the shared kernels.
- [`gemm/mxfp8_n2560_k6144`](gemm/mxfp8_n2560_k6144/README.md): the QKV projection.
- [`gemm/mxfp8_n6144_k2048`](gemm/mxfp8_n6144_k2048/README.md): the attention output projection.
the shape -> kernel selection (TP4 and TP2 shards) and the shared kernels.
- [`gemm/mxfp8_n2560_k6144`](gemm/mxfp8_n2560_k6144/README.md): the QKV projection at TP4.
- [`gemm/mxfp8_n4992_k6144`](gemm/mxfp8_n4992_k6144/README.md): the QKV projection at TP2.
- [`gemm/mxfp8_n6144_k2048`](gemm/mxfp8_n6144_k2048/README.md): the attention output projection at TP4.
- [`gemm/mxfp8_n6144_k4096`](gemm/mxfp8_n6144_k4096/README.md): the attention output projection at TP2 (decode and
every prefill chunk).
- [`gemm/mxfp8_n6144_k8192`](gemm/mxfp8_n6144_k8192/README.md): the full output projection of the sequence-parallel
prefill.
- [`gemm/mxfp8_gateup_swiglu_n6144_k6144_i3072`](gemm/mxfp8_gateup_swiglu_n6144_k6144_i3072/README.md): dense MLP
gate_up + SwiGLU at 1-8 rows.
- [`gemm/bf16_n50016_k6144`](gemm/bf16_n50016_k6144/README.md): the lm_head vocabulary shard, with an optional row map.
- [`gemm/bf16_n200064_k256`](gemm/bf16_n200064_k256/README.md): the draft's Markov projection over selected rows.
gate_up + SwiGLU at 1-8 rows (TP4).
- [`gemm/mxfp8_gateup_swiglu_n12288_k6144_i6144`](gemm/mxfp8_gateup_swiglu_n12288_k6144_i6144/README.md): dense MLP
gate_up + SwiGLU at 1-16 rows (TP2).
- [`gemm/bf16_n50016_k6144`](gemm/bf16_n50016_k6144/README.md): the TP4 lm_head vocabulary shard, with an optional row
map.
- [`gemm/bf16_n100032_k6144`](gemm/bf16_n100032_k6144/README.md): the TP2 lm_head vocabulary shard, with an optional
row map, a strided output and the exchange publish.
- [`gemm/bf16_n200064_k256`](gemm/bf16_n200064_k256/README.md): the draft's Markov projection over 0-128 selected rows.
- [`gemm/bf16_concat_linear_n1536_k30720`](gemm/bf16_concat_linear_n1536_k30720/README.md): the draft's input fc over
five taps, and the four-rank gather.
five taps at TP4, and the four-rank gather.
- [`gemm/bf16_concat_linear_n3072_k30720`](gemm/bf16_concat_linear_n3072_k30720/README.md): the draft's input fc over
five taps at TP2, and the two-rank exchange.

## MoE

- [`moe/mxfp4_h6144_i768_e128_top4`](moe/mxfp4_h6144_i768_e128_top4/README.md): routed MXFP4 experts (A4W4) with the
MXFP8 shared expert.
- [`moe/mxfp4_h6144_i768_e128_top4`](moe/mxfp4_h6144_i768_e128_top4/README.md): the TP4 MoE block (I 768 per rank):
routed MXFP4 experts (A4W4) with the MXFP8 shared expert.
- [`moe/mxfp4_h6144_i1536_e128_top4`](moe/mxfp4_h6144_i1536_e128_top4/README.md): the TP2 MoE block (I 1536 per rank):
routed MXFP4 experts (A4W4) with the MXFP8 shared expert (a TP shard, or the full copy: its shard read in place,
the whole expert on a sequence-parallel rank's own rows).

## Attention

- [`attention/nvfp4_sparse_prefill_gqa16_d128_p128_top16`](attention/nvfp4_sparse_prefill_gqa16_d128_p128_top16/README.md):
sparse prefill attention over the NVFP4 KV cache (bf16, fp16 or fp8 compute).
sparse prefill attention over the NVFP4 KV cache, 1 or 2 KV heads per GPU (TP4 / TP2; bf16, fp16 or fp8 compute).
- [`attention/nvfp4_sparse_decode_gqa16_d128_p128_top16`](attention/nvfp4_sparse_decode_gqa16_d128_p128_top16/README.md):
sparse decode attention over the NVFP4 KV cache (bf16, fp16 or fp8 compute).
sparse decode attention over the NVFP4 KV cache, 1 or 2 KV heads per GPU (TP4 / TP2; bf16, fp16 or fp8 compute).
- [`attention/norm_rope_nvfp4_insert_d128_rot64_p128`](attention/norm_rope_nvfp4_insert_d128_rot64_p128/README.md):
q / k norm + partial RoPE + NVFP4 K / V / index-K inserts in one launch.
q / k norm + partial RoPE + NVFP4 K / V / index-K inserts in one launch (q16 kv1 iq1 at TP4, q32 kv2 iq2 at TP2).
- [`attention/mxfp8_qkv_norm_rope_nvfp4_insert_n2560_k6144_rot64_p128`](attention/mxfp8_qkv_norm_rope_nvfp4_insert_n2560_k6144_rot64_p128/README.md):
the QKV GEMM, its reduce and the norm / RoPE / inserts, fused for small batches.
the TP4 QKV GEMM, its reduce and the norm / RoPE / inserts, fused for small batches.
- [`attention/mxfp8_qkv_norm_rope_nvfp4_insert_n4992_k6144_rot64_p128`](attention/mxfp8_qkv_norm_rope_nvfp4_insert_n4992_k6144_rot64_p128/README.md):
the TP2 QKV GEMM, its reduce and the norm / RoPE / inserts: one launch at 1-16 rows, the deferred chain to 768.
- [`attention/nvfp4_sparse_layer_gqa16_d128_p128_top16`](attention/nvfp4_sparse_layer_gqa16_d128_p128_top16/README.md):
one NVFP4 sparse attention layer at its served dtypes: the indexer, the warm-up, the decode / prefill calls.
one NVFP4 sparse attention layer at its served dtypes, TP4 and TP2: the indexer, the warm-up, the decode / prefill
calls.
- [`attention/swa_decode_fp8kv_gqa16_d128`](attention/swa_decode_fp8kv_gqa16_d128/README.md): the draft's
sliding-window decode attention over an fp8 KV cache.
sliding-window decode attention over an fp8 KV cache, 1 or 2 KV heads per GPU (TP4 / TP2).
- [`attention/norm_rope_fp8_insert_d128_rot64`](attention/norm_rope_fp8_insert_d128_rot64/README.md): the draft's
norm + RoPE + fp8 KV insert.
norm + RoPE + fp8 KV insert, 1 or 2 KV heads per GPU (TP4 / TP2).

## Indexer

- [`indexer/indexer_nvfp4_d128_p128_top16`](indexer/indexer_nvfp4_d128_p128_top16/README.md): the sparse-attention
indexer: block scores over NVFP4 index pages and the top-16 blocks, decode and prefill, one or four GPUs.
indexer: block scores over NVFP4 index pages and the top-16 blocks, decode and prefill, one GPU at 1 / 2 index heads
(TP4 / TP2), or two or four GPUs at decode.

## Embedding and activation

- [`embedding/vocab_gather_h6144`](embedding/vocab_gather_h6144/README.md): vocabulary lookup over four shards without
an all-reduce.
- [`embedding/vocab_gather_h6144`](embedding/vocab_gather_h6144/README.md): vocabulary lookup over two or four shards
without an all-reduce.
- [`embedding/vocab_gather_gemma_mxfp8_h6144`](embedding/vocab_gather_gemma_mxfp8_h6144/README.md): the same lookup +
Gemma norm + MXFP8 quant in one launch.
- [`activation/swiglu_oai_mxfp8_i3072`](activation/swiglu_oai_mxfp8_i3072/README.md): SwiGLU + MXFP8 quant for the
down GEMM.
TP4 down GEMM (I 3072).
- [`activation/swiglu_oai_mxfp8_i6144`](activation/swiglu_oai_mxfp8_i6144/README.md): SwiGLU + MXFP8 quant for the
TP2 down GEMM (I 6144).

## Norms

- [`norm/gemma_add_rmsnorm_h6144`](norm/gemma_add_rmsnorm_h6144/README.md): add + Gemma RMSNorm to bf16 or MXFP8, with
the four-rank all-reduce fused in front; a decoder layer's tail (norm, FFN, the next norm).
the four-rank (`allreduce_add_rmsnorm`) or two-rank (`allreduce_add_rmsnorm_ranks2`) all-reduce fused in front; a
decoder layer's tail (norm, FFN, the next norm).
- [`norm/allreduce_gemma_bf16_mxfp8_h6144`](norm/allreduce_gemma_bf16_mxfp8_h6144/README.md): four-rank all-reduce +
Gemma norm with bf16-rounded MXFP8.
- [`norm/allreduce_gemma_bf16_mxfp8_h6144_ranks2`](norm/allreduce_gemma_bf16_mxfp8_h6144_ranks2/README.md): two-rank
all-reduce + Gemma norm with bf16-rounded MXFP8.
- [`norm/gemma_rmsnorm_bf16_mxfp8_h6144`](norm/gemma_rmsnorm_bf16_mxfp8_h6144/README.md): Gemma RMSNorm to
bf16-rounded MXFP8.

Expand All @@ -77,21 +99,42 @@ One line per kernel package. Each package's README has what it does, its entry p
- [`collectives/all_gather_i64_c7_ranks4`](collectives/all_gather_i64_c7_ranks4/README.md): draft token ids into batch
order.

## Collectives (two ranks)

- [`collectives/pull_collectives_ranks2`](collectives/pull_collectives_ranks2/README.md): the unpacking all-to-all,
all-gather, fixed-order reduce-scatter and SP norm tails over the one xGMI link of a GPU pair.
- [`collectives/seq_parallel_handoffs_h6144_q4096_ranks2`](collectives/seq_parallel_handoffs_h6144_q4096_ranks2/README.md):
the sequence-parallel prefill's layer hand-offs (pull collectives, an SDMA form for DBO; the shared expert local to
its rows' owner on a side stream).
- [`collectives/sdma_push_ranks2`](collectives/sdma_push_ranks2/README.md): pushes on one SDMA engine, GPU-ordered,
without CUs.
- [`collectives/all_to_all_bf16_n100032_ranks2`](collectives/all_to_all_bf16_n100032_ranks2/README.md): logits rows
into the final vocabulary layout.
- [`collectives/all_gather_sampler_i64_c8_ranks2`](collectives/all_gather_sampler_i64_c8_ranks2/README.md): the
sampler output gather.
- [`collectives/all_gather_i64_c7_ranks2`](collectives/all_gather_i64_c7_ranks2/README.md): draft token ids into batch
order.

## Sampling

- [`sampling/batch_sharded_markov_v200064_h6144_r256_ranks4`](sampling/batch_sharded_markov_v200064_h6144_r256_ranks4/README.md):
batch-sharded sampling on four ranks: logits exchange, output gather and the Markov draft chain.
- [`sampling/batch_sharded_markov_v200064_h6144_r256_ranks2`](sampling/batch_sharded_markov_v200064_h6144_r256_ranks2/README.md):
batch-sharded sampling on two ranks: owned-row logits (exchange, or locally on a copy of the peer shard), output
gather and the Markov draft chain.
- [`sampling/gumbel_partials_bias_bf16_v200064`](sampling/gumbel_partials_bias_bf16_v200064/README.md): Gumbel noise
and block partials of one Markov draft step.
- [`sampling/gumbel_combine_markov_embed_v200064_r256`](sampling/gumbel_combine_markov_embed_v200064_r256/README.md):
the end of one Markov draft step: combine, draft id, next row.
- [`sampling/top_p_mask_f32_v200064`](sampling/top_p_mask_f32_v200064/README.md): top-p mask of <= 64 fp32 logits
rows in one launch (vLLM's split top-p chain, bit for bit).
- [`sampling/top_p_mask_f32_v200064`](sampling/top_p_mask_f32_v200064/README.md): vLLM's top-p without top-k on fp32
logits rows, bit for bit: the split chain (<= 64 rows) in one launch, the one-program-per-row kernel above.

## Decoder blocks

- [`decoder/dense_mxfp8_h6144_i3072_ranks4`](decoder/dense_mxfp8_h6144_i3072_ranks4/README.md): the draft's dense
decoder layers around their attention, four ranks.
- [`decoder/dense_mxfp8_h6144_i6144_ranks2`](decoder/dense_mxfp8_h6144_i6144_ranks2/README.md): the draft's dense
decoder layers around their attention, two ranks (and the TP2 target's dense MLP).

## Shared code

Expand Down
47 changes: 47 additions & 0 deletions amd_kernels/activation/swiglu_oai_mxfp8_i6144/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# SwiGLU-OAI + MXFP8 quant, I 6144 (gfx950)

## What it does

One launch for the SwiGLU-OAI activation of a split-layout gate_up row (`[m, 2 x 6144]` bf16, gate | up, unit column
stride, any row stride) and the MXFP8 quant of the result in the layout the down GEMM (N 6144, K 6144) reads: e4m3
codes row-major `[m, 6144]` or in the decode GEMM's (16, 16) tiles with rows padded to the tile, E8M0 scales
`[ceil(rows / 32) * 32, 192]` in AITER's shuffled layout (padding rows and scale padding unwritten; the GEMM masks
them). Caller: the M3.1 dense FFN at TP2 (I = 6144 per rank): the DSpark draft's 5 layers and the target's dense
layers 0-2, above 16 rows (1-16 rows run `gemm/mxfp8_gateup_swiglu_n12288_k6144_i6144`). Numerics (`reference.py`):
`form` is the caller's, never chosen by row count. `bf16_chain` = the draft's aten chain (`clamp`, `sigmoid`, `mul`,
`add`, each rounded to bf16); `fp32` = the target's `swiglu_oai_split` (fp32 math, one bf16 rounding); both use
`1 / (1 + exp(-x))` with the accurate exp and an IEEE divide. Then the Triton MXFP8 quantizer on the bf16 value: scale
`clamp(ceil(log2(amax / 448)) + 127, 0, 254)` per 32 columns, e4m3 codes of `act * 2^(127 - scale)`. Codes and scales
equal the reference bit for bit on finite inputs, for every workgroup shape.

## Entry points

- `swiglu_oai_mxfp8`: the down GEMM's codes and scales of a gate_up row block, row-major or in (16, 16) tiles; the
caller takes the tile from `gemm.mxfp8_common.gemm.prequant_code_tile_rows(m, 6144, 6144)` and runs the down GEMM
with `gemm.mxfp8_common.gemm.mxfp8_gemm_prequant`. The output buffers may be the caller's.
- `warmup_swiglu_oai_mxfp8`: compiles all 8 variants (2 workgroup shapes x 2 forms x 2 code layouts; the row count
is a runtime argument). Call it before serving or graph capture.

`host.SHAPES` picks the workgroup shape (pieces per row, threads) by row count.

## Performance

One MI355X, graph replay, 100 distinct inputs per graph, us per call, against what vLLM runs at TP2 without it: the
aten chain + quantizer (`bf16_chain`) or Triton `swiglu_oai_split` + quantizer (`fp32`). HBM floor at 6.5 TB/s.

| Rows | Code tile (0: row-major) | bf16_chain baseline | bf16_chain | fp32 baseline | fp32 | HBM floor | fraction (fp32) |
|---|---|---|---|---|---|---|---|
| 7 | 16 | 17.79 | 2.49 | 3.76 | 2.38 | 0.03 | 1% |
| 64 | 64 | 23.51 | 2.78 | 4.31 | 2.70 | 0.30 | 11% |
| 128 | 64 | 27.90 | 3.66 | 5.04 | 3.51 | 0.61 | 17% |
| 256 | 128 | 33.25 | 4.60 | 7.03 | 4.33 | 1.22 | 28% |
| 512 | 128 | 40.20 | 6.74 | 10.40 | 6.20 | 2.43 | 39% |
| 1024 | 0 | 65.17 | 10.64 | 15.40 | 9.71 | 4.87 | 50% |
| 1792 | 0 | 99.54 | 16.90 | 24.12 | 15.06 | 8.52 | 57% |

## Tests

```bash
python3 -m pytest tests/activation/swiglu_oai_mxfp8_i6144 -q
python3 benchmarks/activation/swiglu_oai_mxfp8_i6144/benchmark_swiglu_oai_mxfp8.py [--form fp32] [--sweep]
```
15 changes: 15 additions & 0 deletions amd_kernels/activation/swiglu_oai_mxfp8_i6144/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the amd-kernels project
"""SwiGLU-OAI of a split-layout gate_up + MXFP8 quant of the result, I = 6144 output columns, FlyDSL
(gfx950). The torch contract is ``reference.py``."""


def __getattr__(name):
if name in __all__:
from . import host

return getattr(host, name)
raise AttributeError(name)


__all__ = ["swiglu_oai_mxfp8", "warmup_swiglu_oai_mxfp8"]
101 changes: 101 additions & 0 deletions amd_kernels/activation/swiglu_oai_mxfp8_i6144/host.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the amd-kernels project
"""Compile cache + launch wrapper of the SwiGLU-OAI + MXFP8 quant kernel."""

import functools
import struct

import torch

from . import kernel as _kernel

I = _kernel.I

# (max rows, pieces, threads): 12 one-wave pieces per row up to 704 rows, then one 12-wave workgroup per row
# (benchmark_swiglu_oai_mxfp8.py --sweep, both forms)
SHAPES = ((704, 12, 64), (1 << 31, 1, 768))


def _run_compiled(exe, *args):
"""First call compiles and runs (``flyc.compile``); later calls dispatch the cached CompiledFunction."""
import flydsl.compiler as flyc

cf = getattr(exe, "_cf", None)
if cf is None:
exe._cf = flyc.compile(exe, *args)
else:
cf(*args)


@functools.cache
def _get_kernel(form: str, shuffled: bool, pieces: int, threads: int):
return _kernel.compile_swiglu_oai_mxfp8(form, shuffled, pieces, threads)


def _shape(m: int) -> tuple[int, int]:
return next((p, t) for top, p, t in SHAPES if m <= top)


def _f32_bits(v: float) -> int:
return struct.unpack("<i", struct.pack("<f", v))[0]


def swiglu_oai_mxfp8(
gate_up: torch.Tensor, # [m, 2 * 6144] bf16, unit column stride (split layout: gate | up)
alpha: float,
beta: float,
limit: float,
code_tile_rows: int = 0,
form: str = "bf16_chain",
q_out: torch.Tensor | None = None,
scale_out: torch.Tensor | None = None,
shape: tuple[int, int] | None = None,
) -> tuple[torch.Tensor, torch.Tensor]:
"""(codes, scales) of ``quantize(swiglu(gate_up))`` for the down GEMM: codes row-major [m, I]
(``code_tile_rows`` 0) or in the decode GEMM's (16, 16) layout, rows padded to the tile (padding rows
unwritten); e8m0 scales [ceil(rows / 32) * 32, I / 32] in AITER's shuffled layout (padding unwritten).
``form``: the SwiGLU numerics, the caller's (``bf16_chain`` = the draft's aten chain, ``fp32`` = the
target's ``swiglu_oai_split``). ``shape`` (pieces, threads) overrides the row-count choice (sweeps only)."""
m = gate_up.shape[0]
assert gate_up.dtype == torch.bfloat16 and gate_up.shape[1] == 2 * I and gate_up.stride(1) == 1
shuffled = code_tile_rows > 0
if shuffled:
assert code_tile_rows % 16 == 0
rows = (m + code_tile_rows - 1) // code_tile_rows * code_tile_rows
else:
rows = m
q = q_out if q_out is not None else torch.empty((rows, I), dtype=torch.float8_e4m3fn, device=gate_up.device)
s = (
scale_out
if scale_out is not None
else torch.empty(((rows + 31) // 32 * 32, I // 32), dtype=torch.uint8, device=gate_up.device)
)
assert q.shape[0] >= rows and q.shape[1] == I and q.is_contiguous()
assert s.shape[0] >= (rows + 31) // 32 * 32 and s.shape[1] == I // 32 and s.is_contiguous()
if m:
_run_compiled(
_get_kernel(form, shuffled, *(shape or _shape(m))),
gate_up.data_ptr(),
q.data_ptr(),
s.data_ptr(),
q.numel() * q.element_size(),
s.numel(),
m,
gate_up.stride(0),
_f32_bits(alpha),
_f32_bits(beta),
_f32_bits(limit),
torch.cuda.current_stream(),
)
return q, s


def warmup_swiglu_oai_mxfp8(device: torch.device | None = None, forms=("bf16_chain", "fp32")) -> None:
"""Compile every variant (each row-count shape x form x row-major / tiled codes; the row count is a runtime
argument) before serving or graph capture (README rule 4)."""
for top, _, _ in SHAPES:
m = min(top, 2048)
x = torch.zeros((m, 2 * I), dtype=torch.bfloat16, device=device or "cuda")
for form in forms:
for tile in (0, 16):
swiglu_oai_mxfp8(x, 1.0, 0.0, 1.0, tile, form)
Loading
Loading