Skip to content

feat(kunlun): add XPU backend with 45 operators (22 implemented, 23 link stubs) - #1049

Closed
rubik-hua wants to merge 1 commit into
InfiniTensor:masterfrom
rubik-hua:feat/kunlun-backend
Closed

rubik-hua wants to merge 1 commit into
InfiniTensor:masterfrom
rubik-hua:feat/kunlun-backend

Conversation

@rubik-hua

Copy link
Copy Markdown
Contributor

Register Kunlun as device type 10 and implement the operator set over the vendor kernel libraries shipped in the Kunlun SDK / vLLM-XPU wheel:

  • src/native/kunlun: 22 implemented operators (Qwen2-7B default path and the paged-attention path fully verified). 23 additional stubs keep the InfiniLM link intact: InfiniLM's csrc references their factory symbols unconditionally (MoE, w4/int8 quantization, mamba, qwen3_next, vision), so a missing symbol breaks every model's import at dlopen. Stubs fail fast via supports() — link scaffolding, not delivery promises. Operators with no InfiniLM references (awq/gptq marlin repack, fused qk-norm, upsample) are not stubbed; they land with their implementations when a model needs them.
  • flash-attention family reuses vendor kernels instead of hand-written device code: reshape_and_cache (libapiinfer.so) for the KV scatter, paged_attention_xft (libxpu_flash_attention.so) for paged decode, flash_attention_context_vllm for cache-only paged prefill. The wrappers absorb the vendor contracts (head-major cache blocks, int32 slots narrowed from int64, alpha = softmax_scale * sqrt(head_dim), total-KV prefix sums for kvlod, non-dense fused-QKV views compacted via host staging) and bind the vendor context to the caller's InfiniRT stream (set_stream + xpu_wait before every D2H read)
  • GEMM derives op_a from the b operand's layout (fixes a transposition bug)
  • dispatcher: Kunlun exposes exactly one implementation slot; the device-generic ATen templates never instantiate for Kunlun
  • torch device facade: kKunlun maps onto the "cuda" torch device so the pybind bindings accept Kunlun tensors

实现的算子是DeepSeek-R1-Distill-Qwen-7B跑到的,其它为stub实现,避免缺符号。

XPU_VISIBLE_DEVICES=1 python3 test_infer.py --device=kunlun --model=/mnt/infini/infrahi/infinilm-models/DeepSeek-R1-Distill-Qwen-7B --prompt "你是谁"

image

XPU_VISIBLE_DEVICES=7 python3 examples/test_infer.py --device=kunlun --model=/mnt/infini/infrahi/infinilm-models/DeepSeek-R1-Distill-Qwen-7B --enable-paged-attn --attn flash-attn --prompt "你是谁"

image

…ink stubs)

Register Kunlun as device type 10 and implement the operator set over the
vendor kernel libraries shipped in the Kunlun SDK / vLLM-XPU wheel:

- src/native/kunlun: 22 implemented operators (Qwen2-7B default path and
  the paged-attention path fully verified). 23 additional stubs keep the
  InfiniLM link intact: InfiniLM's csrc references their factory symbols
  unconditionally (MoE, w4/int8 quantization, mamba, qwen3_next, vision),
  so a missing symbol breaks every model's import at dlopen. Stubs fail
  fast via supports() — link scaffolding, not delivery promises. Operators
  with no InfiniLM references (awq/gptq marlin repack, fused qk-norm,
  upsample) are not stubbed; they land with their implementations when a
  model needs them.
- flash-attention family reuses vendor kernels instead of hand-written
  device code: reshape_and_cache (libapiinfer.so) for the KV scatter,
  paged_attention_xft (libxpu_flash_attention.so) for paged decode,
  flash_attention_context_vllm for cache-only paged prefill. The wrappers
  absorb the vendor contracts (head-major cache blocks, int32 slots
  narrowed from int64, alpha = softmax_scale * sqrt(head_dim), total-KV
  prefix sums for kvlod, non-dense fused-QKV views compacted via host
  staging) and bind the vendor context to the caller's InfiniRT stream
  (set_stream + xpu_wait before every D2H read)
- GEMM derives op_a from the b operand's layout (fixes a transposition bug)
- dispatcher: Kunlun exposes exactly one implementation slot; the
  device-generic ATen templates never instantiate for Kunlun
- torch device facade: kKunlun maps onto the "cuda" torch device so the
  pybind bindings accept Kunlun tensors
@rubik-hua
rubik-hua requested a review from a team November 18, 2026 09:36
@baominghelly

Copy link
Copy Markdown
Collaborator

等RT代码合入后我再看一下吧,现在因为kunlun RT修改没合入的原因,ci unit全失败了

@rubik-hua rubik-hua closed this Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants