Repository navigation
Conversation
b956fdd to
f7f619c
Compare
Convert eligible weights with aclnnTransMatmulWeight and execute them through aclnnMatmulWeightNz.
Cache the repeatable MatmulWeightNz executor and qualify the hot Ascend decode kernels for device graph capture. This ports the capture behavior used by the FM9G70B Ascend stack to the current operator API.
56ce68e to
4d9a340
Compare
|
TransMatmulWeightNz里可能不需要aclrtSynchronizeStream |
4d9a340 to
6d11356
Compare
|
Thanks for the review. Addressed in
Validation: clean Ascend build/install passed, dependent InfiniLM regression passed 26/26, and c64/in32/out256 completed 256/256 requests with 453.52 output tok/s and 127.74 ms mean TPOT. The Python binding test itself is included for a runtime with compatible installed bindings; this container's binding build is blocked by the installed torch_npu 2.10.0.post2 stream API mismatch, so the end-to-end FM9G70 path was used for runtime verification. |
baominghelly
left a comment
There was a problem hiding this comment.
已线下同步过,关于matmul_weight_nz和trans_matmul_weight_nz只开放c++接口给InfiniLM用,这个PR是短期解决方案,长期方案是这两个算子修改需要等TensorView支持ND/NZ分布的区分后迁移到对应的matmul/trans算子.
请在这两个算子里面添加需要迁移的注释,并把legacy C和python接口去掉,之后可以合入代码
6d11356 to
17cfcb4
Compare
|
@baominghelly Thanks. Addressed in 385f1de:
Validation: wrapper generator tests 49 passed; Ascend build/install and InfiniLM link/import passed; scheduler regression 26 passed. c64/in32/out256 completed 256/256 with 0 failures, 453.38 output tok/s, 229.54 ms mean TTFT, and 127.77 ms mean TPOT. The head was amended once more to |
Port the CANN FAI decode path, parallelize and chunk paged cache writes, and retain host-side uniform prefill sequence metadata to reduce synchronous D2H copies.
17cfcb4 to
385f1de
Compare
Summary
This PR ports and productizes the FM9G70 Ascend serving stack in three ordered commits:
98b0638— add Ascend FRACTAL_NZ weight-layout conversion and matmul operators.ea74120— mark FM9G hot operators graph-capture safe and make the NZ matmul executor repeatable.385f1de— add the specialized FM9G70 paged-attention decode/cache-write path and accelerate uniform prefill metadata handling.The change is Ascend-only. Generic and non-Ascend builds remain unchanged.
Temporary NZ Operator Interface
MatmulWeightNzandTransMatmulWeightNzare short-term compatibility operators used by InfiniLM. They are intentionally C++-only:m.defregistrations for these operators.Operator<T>::Call.Official ops-transformer Dependency
The specialized FAI source now comes from the official open-source repository:
https://gitcode.com/cann/ops-transformer.gite1e99f4832cf10d9af870d29ee4135f2e749f898third_party/ops-transformerDependency behavior:
.gitmodulessetsupdate = nonefor this submodule.git clone --recurse-submodulesskips this submodule.scripts/fetch-ops-transformer.sh.-DINFINIOPS_FETCH_OPS_TRANSFORMER=OFFdisables the automatic fetch for pre-provisioned/offline builds.After checkout, CMake applies
third_party/ops-transformer-fm9g70.patch. The patch is intentionally minimal (51 lines) and changes only the regular-decode entry:The upstream ops-transformer CMake project is not built or installed. Only the FAI header dependency closure participates in compiling InfiniOps'
paged_attention_fai.cppthrough the custom AscendC target.Validation
Default dependency behavior
A real
git clone --recurse-submodulestest printed:The submodule directory remained absent. Running
scripts/fetch-ops-transformer.shthen fetched from the official GitCode repository and checked out:CPU-only configuration with
WITH_ASCEND=OFFcompleted without fetching or creating any ops-transformer source.Ascend build and regression
e1e99f483.paged_attention_fai.cppwas rebuilt through AscendC precompile, AIC, and AIV stages.libinfiniopsand the dependent InfiniLM extension installed successfully.tests/test_generate_wrappers.py: 49 passed.test_chunked_prefill_sched.py: 26 passed.git diff --check: passed.InfiniLM/scripts/format.py.Performance
FM9G70B BF16 MHA, TP=8, Ascend 910B3, CANN 9.0.1, c64 / input 32 / output 256:
The equivalent full 36-case matrix using the same specialized kernel previously recorded 33 valid Jingbo comparisons, with wins in output throughput, mean TPOT, and mean TTFT in all 33 cases. The three c64/in1024 cases had no Jingbo baseline.
Reviewer Notes
Review Follow-up
Feedback from @humannumber196 was addressed in
6d11356:TransMatmulWeightNz.tests/test_flash_attn_with_kvcache.py.The refreshed review-feedback run completed c64/in32/out256 with 256/256 successful requests, 453.52 output tok/s, 232.47 ms mean TTFT, and 127.74 ms mean TPOT.
Feedback from @baominghelly was addressed in
385f1de:MatmulWeightNzandTransMatmulWeightNz.The refreshed C++-only run completed c64/in32/out256 with 256/256 successful requests, 453.38 output tok/s, 229.54 ms mean TTFT, and 127.77 ms mean TPOT.