Repository navigation
Conversation
…ink stubs) Register Kunlun as device type 10 and implement the operator set over the vendor kernel libraries shipped in the Kunlun SDK / vLLM-XPU wheel: - src/native/kunlun: 22 implemented operators (Qwen2-7B default path and the paged-attention path fully verified). 23 additional stubs keep the InfiniLM link intact: InfiniLM's csrc references their factory symbols unconditionally (MoE, w4/int8 quantization, mamba, qwen3_next, vision), so a missing symbol breaks every model's import at dlopen. Stubs fail fast via supports() — link scaffolding, not delivery promises. Operators with no InfiniLM references (awq/gptq marlin repack, fused qk-norm, upsample) are not stubbed; they land with their implementations when a model needs them. - flash-attention family reuses vendor kernels instead of hand-written device code: reshape_and_cache (libapiinfer.so) for the KV scatter, paged_attention_xft (libxpu_flash_attention.so) for paged decode, flash_attention_context_vllm for cache-only paged prefill. The wrappers absorb the vendor contracts (head-major cache blocks, int32 slots narrowed from int64, alpha = softmax_scale * sqrt(head_dim), total-KV prefix sums for kvlod, non-dense fused-QKV views compacted via host staging) and bind the vendor context to the caller's InfiniRT stream (set_stream + xpu_wait before every D2H read) - GEMM derives op_a from the b operand's layout (fixes a transposition bug) - dispatcher: Kunlun exposes exactly one implementation slot; the device-generic ATen templates never instantiate for Kunlun - torch device facade: kKunlun maps onto the "cuda" torch device so the pybind bindings accept Kunlun tensors
Collaborator
|
等RT代码合入后我再看一下吧,现在因为kunlun RT修改没合入的原因,ci unit全失败了 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Register Kunlun as device type 10 and implement the operator set over the vendor kernel libraries shipped in the Kunlun SDK / vLLM-XPU wheel:
实现的算子是DeepSeek-R1-Distill-Qwen-7B跑到的,其它为stub实现,避免缺符号。
XPU_VISIBLE_DEVICES=1 python3 test_infer.py --device=kunlun --model=/mnt/infini/infrahi/infinilm-models/DeepSeek-R1-Distill-Qwen-7B --prompt "你是谁"
XPU_VISIBLE_DEVICES=7 python3 examples/test_infer.py --device=kunlun --model=/mnt/infini/infrahi/infinilm-models/DeepSeek-R1-Distill-Qwen-7B --enable-paged-attn --attn flash-attn --prompt "你是谁"