Skip to content

feat(ascend): accelerate FM9G70 paged serving - #1041

Open
wooway777 wants to merge 3 commits into
masterfrom
merge/fm9g70-ascend-stack
Open

wooway777 wants to merge 3 commits into
masterfrom
merge/fm9g70-ascend-stack

Conversation

@wooway777

@wooway777 wooway777 commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR ports and productizes the FM9G70 Ascend serving stack in three ordered commits:

  1. 98b0638 — add Ascend FRACTAL_NZ weight-layout conversion and matmul operators.
  2. ea74120 — mark FM9G hot operators graph-capture safe and make the NZ matmul executor repeatable.
  3. 385f1de — add the specialized FM9G70 paged-attention decode/cache-write path and accelerate uniform prefill metadata handling.

The change is Ascend-only. Generic and non-Ascend builds remain unchanged.

Temporary NZ Operator Interface

MatmulWeightNz and TransMatmulWeightNz are short-term compatibility operators used by InfiniLM. They are intentionally C++-only:

  • The public contracts document migration to the general matmul/transpose-matmul paths after TensorView distinguishes ND and NZ distributions.
  • Wrapper generation emits no legacy C headers or C wrapper sources for these operators.
  • Wrapper generation emits no Python binding headers, sources, or m.def registrations for these operators.
  • The generated C++ operator-call instantiations remain available so InfiniLM can continue using Operator<T>::Call.
  • The Python-only NZ operator test was removed with the removed Python API.

Official ops-transformer Dependency

The specialized FAI source now comes from the official open-source repository:

  • Repository: https://gitcode.com/cann/ops-transformer.git
  • Pinned official gitlink: e1e99f4832cf10d9af870d29ee4135f2e749f898
  • Location: third_party/ops-transformer
  • No personal fork is used.

Dependency behavior:

  • .gitmodules sets update = none for this submodule.
  • A normal git clone --recurse-submodules skips this submodule.
  • Manual command: scripts/fetch-ops-transformer.sh.
  • Ascend custom-kernel configuration runs that command by default.
  • -DINFINIOPS_FETCH_OPS_TRANSFORMER=OFF disables the automatic fetch for pre-provisioned/offline builds.
  • The script checks out exactly the gitlink commit recorded by InfiniOps.

After checkout, CMake applies third_party/ops-transformer-fm9g70.patch. The patch is intentionally minimal (51 lines) and changes only the regular-decode entry:

  • Use direct raw paged K/V pointers instead of ListTensorDesc.
  • Use int32 cache lengths.
  • Read explicit Q/O strides from the FM9G70 tiling data.
  • Preserve the small-batch KV-N tile policy.

The upstream ops-transformer CMake project is not built or installed. Only the FAI header dependency closure participates in compiling InfiniOps' paged_attention_fai.cpp through the custom AscendC target.

Validation

Default dependency behavior

A real git clone --recurse-submodules test printed:

Skipping submodule 'third_party/ops-transformer'

The submodule directory remained absent. Running scripts/fetch-ops-transformer.sh then fetched from the official GitCode repository and checked out:

e1e99f4832cf10d9af870d29ee4135f2e749f898

CPU-only configuration with WITH_ASCEND=OFF completed without fetching or creating any ops-transformer source.

Ascend build and regression

  • Started from a deinitialized ops-transformer submodule.
  • Ascend CMake automatically fetched official commit e1e99f483.
  • CMake applied the 51-line FM9G70 patch idempotently.
  • paged_attention_fai.cpp was rebuilt through AscendC precompile, AIC, and AIV stages.
  • libinfiniops and the dependent InfiniLM extension installed successfully.
  • tests/test_generate_wrappers.py: 49 passed.
  • test_chunked_prefill_sched.py: 26 passed.
  • git diff --check: passed.
  • Changed files were formatted with InfiniLM/scripts/format.py.

Performance

FM9G70B BF16 MHA, TP=8, Ascend 910B3, CANN 9.0.1, c64 / input 32 / output 256:

Metric Jingbo This PR Difference
Output throughput 426.98 tok/s 453.38 tok/s +6.18%
Input throughput 53.37 tok/s 56.67 tok/s +6.18%
Total throughput 480.35 tok/s 510.05 tok/s +6.18%
Mean TTFT 309.29 ms 229.54 ms -25.79%
Mean TPOT 149.20 ms 127.77 ms -14.37%
Successful requests 256 256 unchanged
Failed requests 0 0 unchanged

The equivalent full 36-case matrix using the same specialized kernel previously recorded 33 valid Jingbo comparisons, with wins in output throughput, mean TPOT, and mean TTFT in all 33 cases. The three c64/in1024 cases had no Jingbo baseline.

Reviewer Notes

  • The upstream gitlink remains an official commit; FM9G70-specific behavior is isolated in the reviewed patch file.
  • The FAI path is guarded to the validated BF16 MHA paged layout and falls back for unsupported inputs.
  • The submodule can remain uninitialized on CPU, NVIDIA, MetaX, Cambricon, Moore, and other non-Ascend builds.

Review Follow-up

Feedback from @humannumber196 was addressed in 6d11356:

  • Removed the unnecessary host-side stream synchronization from TransMatmulWeightNz.
  • Added explicit zero defaults to all FM9G70 FAI tiling fields.
  • Added BF16 MHA paged-attention coverage in tests/test_flash_attn_with_kvcache.py.

The refreshed review-feedback run completed c64/in32/out256 with 256/256 successful requests, 453.52 output tok/s, 232.47 ms mean TTFT, and 127.74 ms mean TPOT.

Feedback from @baominghelly was addressed in 385f1de:

  • Added migration notes to both temporary NZ operators and their Ascend implementations.
  • Removed legacy C and Python API generation for MatmulWeightNz and TransMatmulWeightNz.
  • Retained the generated C++ call instantiations required by InfiniLM.
  • Removed the Python-only test that exercised the removed API.

The refreshed C++-only run completed c64/in32/out256 with 256/256 successful requests, 453.38 output tok/s, 229.54 ms mean TTFT, and 127.77 ms mean TPOT.

@wooway777
wooway777 requested a review from a team November 14, 2026 03:35
@wooway777 wooway777 changed the title Merge/fm9g70 ascend stack feat(ascend): accelerate FM9G70 paged serving Nov 14, 2026
@wooway777
wooway777 force-pushed the merge/fm9g70-ascend-stack branch 2 times, most recently from b956fdd to f7f619c Compare November 16, 2026 08:02
Jingbo-gao and others added 2 commits November 17, 2026 11:36
Convert eligible weights with aclnnTransMatmulWeight and execute them through aclnnMatmulWeightNz.
Cache the repeatable MatmulWeightNz executor and qualify the hot Ascend decode kernels for device graph capture. This ports the capture behavior used by the FM9G70B Ascend stack to the current operator API.
@wooway777
wooway777 force-pushed the merge/fm9g70-ascend-stack branch 3 times, most recently from 56ce68e to 4d9a340 Compare November 19, 2026 05:10
@humannumber196

humannumber196 commented Nov 19, 2026 •

Copy link
Copy Markdown

TransMatmulWeightNz里可能不需要aclrtSynchronizeStream
测试脚本可能可以把paged attention算子也覆盖一下
tiling data可能设一下默认值风险会更低

@wooway777
wooway777 force-pushed the merge/fm9g70-ascend-stack branch from 4d9a340 to 6d11356 Compare November 20, 2026 03:10
@wooway777

Copy link
Copy Markdown
Contributor Author

Thanks for the review. Addressed in 6d11356:

  • Removed the explicit aclrtSynchronizeStream from TransMatmulWeightNz; the ACLNN copy and conversion work now remain ordered on the caller's stream.
  • Added explicit zero-default member initializers to every InfinilmFaiTiling field, while retaining the explicit hot-path assignments.
  • Added test_flash_attn_with_kvcache_ascend_paged_bf16_mha to cover the BF16 MHA paged shape used by the specialized FAI path.

Validation: clean Ascend build/install passed, dependent InfiniLM regression passed 26/26, and c64/in32/out256 completed 256/256 requests with 453.52 output tok/s and 127.74 ms mean TPOT. The Python binding test itself is included for a runtime with compatible installed bindings; this container's binding build is blocked by the installed torch_npu 2.10.0.post2 stream API mismatch, so the end-to-end FM9G70 path was used for runtime verification.

@baominghelly baominghelly left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已线下同步过,关于matmul_weight_nz和trans_matmul_weight_nz只开放c++接口给InfiniLM用,这个PR是短期解决方案,长期方案是这两个算子修改需要等TensorView支持ND/NZ分布的区分后迁移到对应的matmul/trans算子.
请在这两个算子里面添加需要迁移的注释,并把legacy C和python接口去掉,之后可以合入代码

@wooway777
wooway777 force-pushed the merge/fm9g70-ascend-stack branch from 6d11356 to 17cfcb4 Compare November 20, 2026 08:33
@wooway777

wooway777 commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

@baominghelly Thanks. Addressed in 385f1de:

  • Added explicit short-term/migration comments to both NZ operators and their Ascend implementations.
  • Removed legacy C and Python wrapper generation for these two operators; only the C++ operator classes and call instantiations remain for InfiniLM.
  • Removed the Python-only test that used the removed API.

Validation: wrapper generator tests 49 passed; Ascend build/install and InfiniLM link/import passed; scheduler regression 26 passed. c64/in32/out256 completed 256/256 with 0 failures, 453.38 output tok/s, 229.54 ms mean TTFT, and 127.77 ms mean TPOT.

The head was amended once more to 385f1de to satisfy the CI clang-format 21 and ruff formatters; the implementation and validation results are unchanged.

Port the CANN FAI decode path, parallelize and chunk paged cache writes, and retain host-side uniform prefill sequence metadata to reduce synchronous D2H copies.
@wooway777
wooway777 force-pushed the merge/fm9g70-ascend-stack branch from 17cfcb4 to 385f1de Compare November 20, 2026 08:51

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants