Skip to content

feat(metax): add MoE, GDN, quantization and attention operators for Metax - #1033

Open
rubik-hua wants to merge 1 commit into
InfiniTensor:masterfrom
rubik-hua:metax-ops
Open

rubik-hua wants to merge 1 commit into
InfiniTensor:masterfrom
rubik-hua:metax-ops

Conversation

@rubik-hua

@rubik-hua rubik-hua commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Add native MetaX (MACA) device kernels and linked torch operator
providers so InfiniLM can run on MetaX C550 GPUs:

  • Native ops (src/native/cuda/metax): moe_fused_gate, moe_fused_dense, deepseek_moe, chunked/recurrent gated delta rule (linear attention), mamba_selective_scan, causal_conv1d, mrope, w4_marlin, per-tensor/per-channel int8 quant kernels
  • Linked torch ops (src/linked/torch/metax): vLLM providers for moe_align_block_size, topk_sigmoid, quick_gelu; ATen providers for layer_norm, pad; flash_attn_varlen_func, rms_norm and silu_and_mul updates
  • Shared CUDA reference kernels for moe_fused_dense (src/native/cuda/ops)

Validated end-to-end with qwen3.8-27B (TP2, batch size 1): TTFT 350.23 ms, decode 29.78 tok/s.

使用qwen3.8-27B模型验证
python examples/test_infer.py --device metax
--model=/mnt/infini/infrahi/models/qwen38_27B/
--enable-paged-attn --tp 2 --disable-prefix-caching

image image

bench测试
python examples/bench.py --device metax
--model=/mnt/infini/infrahi/models/qwen38_27B/
--enable-paged-attn --tp 2 --disable-prefix-caching
--warmup --batch-size 1 --input-len 128 --output-len 64

image

@rubik-hua
rubik-hua requested a review from a team November 10, 2026 09:44

@baominghelly baominghelly left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

请加上这些算子单元测试通过的截图。
请给这些算子的metax平台都加一下测试,后续我们会定期回归测试确保这些算子都可以正常运行。

@rubik-hua

Copy link
Copy Markdown
Contributor Author

请加上这些算子单元测试通过的截图。 请给这些算子的metax平台都加一下测试,后续我们会定期回归测试确保这些算子都可以正常运行。

python3 -m pytest
tests/test_chunk_gated_delta_rule.py
tests/test_recurrent_gated_delta_rule.py
tests/test_moe_fused_dense.py
tests/test_deepseek_moe.py
tests/test_per_tensor_quant_i8.py
tests/test_metax_stub_ops.py
tests/test_w4_marlin_stub.py
-v --devices metax -p no:cacheprovider

image image

…etaX GPUs

  Add native MetaX (MACA) device kernels and linked torch operator
  providers so InfiniLM can run on MetaX C550 GPUs:

  - Native ops (src/native/cuda/metax): moe_fused_gate,
    moe_fused_dense, deepseek_moe, chunked/recurrent gated delta
    rule (linear attention), mamba_selective_scan, causal_conv1d,
    w4_marlin, per-tensor/per-channel int8 quant kernels
  - Linked torch ops (src/linked/torch/metax): vLLM providers for
    moe_align_block_size, topk_sigmoid, quick_gelu; ATen provider
    for pad; flash_attn_varlen_func, rms_norm and silu_and_mul
    updates
  - Shared CUDA reference kernels for moe_fused_dense
    (src/native/cuda/ops)
  - Metax specializations of argmax/layer_norm/mrope include
    caster.cuh so TypeMap<kMetax, fp16/bf16> is defined and the
    kernels compile for MetaX

  Validated end-to-end with qwen3.8-27B (TP2, batch size 1) on
  C550 / MACA 3.5.3: TTFT 285.01 ms, decode 33.54 tok/s.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants