Skip to content

feat(hygon): add native Clover diffusion and sparse MoE operators - #1015

Open
Ringssss wants to merge 4 commits into
InfiniTensor:masterfrom
Ringssss:feat/hygon-clover-operators
Open

Ringssss wants to merge 4 commits into
InfiniTensor:masterfrom
Ringssss:feat/hygon-clover-operators

Conversation

@Ringssss

Copy link
Copy Markdown

Summary

  • Add Hygon native/Triton providers for Clover preview convolution, bidirectional GDN, FP32 causal state commit, block coreset, live-row routing/cache merge, and packed dense MoE.
  • Enable Hygon Triton JIT module loading using the InfiniRT driver interface; retain gfx architecture and wave-size metadata.
  • Add missing Hygon causal-convolution/gating, layout and rotary providers needed by the model.
  • Add explicit public-API declaration selection for downstream model registries whose unused operators have no provider on this backend.

Motivation

Clover's noisy-block previews must preserve committed recurrent state and read a bidirectional block. The existing Hygon providers cannot execute that model. The companion InfiniLM integration uses these kernels directly from C++, with optional expert budgeting/live-row reuse during preview and full causal replay for commit.

Dependency: InfiniTensor/InfiniRT#52. The build was also checked against the installed Hygon SDK.

Type of Change

  • feat — new operators / backend capabilities
  • build — installed API generation
  • test — numerical/state and JIT metadata checks

Platforms Affected

  • Hygon (WITH_HYGON, added to this template)
  • Build system / CMake
  • Python bindings / user-facing API
  • CPU and NVIDIA: shared JIT/header-generation tooling; their kernel implementations are unchanged

Smoke Test Result

Hygon BW/gfx936, DTK 26.04, Torch 2.10.0, Triton 3.5.0, BF16 data and FP32 recurrent state:

Preview convolution/GDN: widths 1/8/32, batches 1/3; reference checks passed;
clean state remained bitwise unchanged.
Fused MoE: 1/32/64 tokens; independent Torch reference checks passed.
Causal GDN: packed lengths 4/7 and separate slots; output/state checks passed.
Coreset: widths 1/8/32, batches 1/4; routing masks matched exactly.
Live mask, inactive-route omission and cached output merge: passed.

The rebased build passed 62 wrapper/JIT/public-header and Clover tests. The required CPU/Hygon smoke selection passed: 121 passed, 43 skipped (inactive providers), 9485 deselected. Build, tooling and checked-in tests are documented in docs/clover-hygon.md. tests/native/clover_workspace_smoke.cpp exercises workspace-bearing providers through the native public API. Missing workspaces are rejected before launch. The original unguarded Python functional invocation failed; that failed prototype is not included as a passing result.

Test Results on Supported Platforms

Platform Affected Build / Smoke Result Full Result / Notes
Hygon Yes Full library build and numerical/state smoke passed Full-model TP2 generation and HTTP campaign passed; not the entire unrelated operator matrix
CPU API generation Wrapper, public-header and JIT unit tests passed Hygon build also includes CPU providers
NVIDIA Shared JIT metadata handling Unit tests cover CUDA and HIP metadata No NVIDIA hardware available for GPU validation
Iluvatar / MetaX / Moore / Cambricon / Ascend Shared opt-in declaration generator Existing default behavior retained Hardware unavailable; platform CI remains for maintainers

Benchmark / Performance Impact

The companion native InfiniLM campaign uses real Clover-35B-A3B-Preview weights, BF16 TP2, 32 short prompts repeated three times, concurrency one and a common OpenAI chat client. All 96 requests per lane completed. Visible output tokens are retokenized consistently; padded diffusion usage counters are excluded.

Measured integration throughput after upstream sync: native baseline 28.57 tok/s; frontier stride 38.47; expert budget/live reuse 28.70; joint 39.92; local SGLang Clover port 5.88. These are end-to-end integration results, not isolated operator speedups. The measurement record retains all requests, failures, warmups and rejected candidates. Final smoke repairs also cover strided casts and preserving the vendor HIP driver during autotuning.

Separate traces contain GPU events on both TP devices. Live-row kernels occur only in the joint path; summed routed-MoE GPU duration decreases from 1339 ms to 721 ms in the representative profiled request. Profiling time is excluded from throughput and summed device durations are not wall time.

Notes for Reviewers

Preview state is read-only. Causal replay remains exact to the specified recurrent equations; live-row reuse is an opt-in approximation controlled by the LM. The H100 DP/EP sequence-parallel communication pipeline is not ported here.

INFINI_OPS_DECLARATIONS exposes unsupported APIs for compilation/capability queries; it does not create fallback providers. Preserve explicit workspace lifetime and same-stream ordering. The code-generation comment-default issue found during bring-up was already fixed upstream; the branch keeps the upstream implementation.

Source provenance, API alignment, supported layouts and licenses are recorded in the operator guide and THIRD_PARTY.txt. Cross-platform accelerator CI is still needed for the shared tooling changes.

The clover* Triton operators and MoeFusedDense were registered only for
a single device backend (Hygon), which made the whole Clover diffusion
path unbuildable and unrunnable on any other Triton-capable platform.

Refactor the seven clover operators following the established pattern
used by shared operators such as Add:

- Extract the device-specific inline Operator specializations into
  template <Device::Type kDev> implementation classes under
  infini::ops::triton::jit, with Launch<kDev> parameterization.
- Register thin Operator<Op, kNvidia/kHygon, 10> shells for BOTH
  Triton-capable backends. Hygon behavior is unchanged.
- moe_fused_dense already had a templated implementation but only a
  Hygon shell; add the kNvidia shell.
- block_conv1d / local_memory / recurrent_gdn already templated with a
  kNvidia shell; add the missing kHygon shell for symmetry.

Also fix a CMake glob bug in the WITH_NVIDIA branch: NVIDIA_PATTERNS
uses GLOB_RECURSE with a top-level 'native/cuda/*.cu' pattern that also
matches sibling vendor subtrees (hygon/iluvatar/metax/moore/...), so
their kernels were compiled twice and failed to link with multiple
definition errors when both backends' sources were present. Add the
symmetric exclusion that the WITH_HYGON branch already has.

Verified on NVIDIA A100 (CUDA 13.1, sm80) with Clover-35B-A3B-Preview
under InfiniLM PR InfiniTensor#604: TP2 block-diffusion generation produces
byte-identical outputs and per-block iteration counts before and after
this refactor.
Absorb 136 upstream files (cambricon topk provider, wrapper-generator
and conftest updates, pinned deps) so the branch merges cleanly into
master. Auto-merge; the only remaining deltas versus master are this
PR's clover operators, the multi-device Triton shells, and the NVIDIA
CMake vendor-exclusion fix.
@Ringssss
Ringssss marked this pull request as ready for review November 19, 2026 07:36
@Ringssss
Ringssss requested a review from a team November 19, 2026 07:36
namespace infini::ops {

template <>
class Operator<MoeAlignBlockSize, Device::Type::kHygon, 0>

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个算子是不是可以把具体实现放到cuda下面,然后这里继承啊?

Comment on lines +2301 to +2306
parser.add_argument(
"--declarations",
nargs="*",
default=[],
help="Also emit operator APIs with no active provider.",
)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

应该可以通过构建自己的ops.json实现类似的功能,举例:https://github.com/InfiniTensor/InfiniOps/blob/master/scripts/configs/infiniops_ops_nvidia_dense_sampling_torch_free_no_marlin.json#L52-L54
编译方式添加DINFINI_OPS_OPS参数

cmake -S submodules/InfiniOps -B build/InfiniOps \
  -DWITH_TORCH=ON \
  -DWITH_LINKED=ON \
  -DINFINI_OPS_OPS=/workspace/ops.json \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX="$INFINI_ROOT"

这个修改应该不是必要的

namespace infini::ops::triton::jit {

template <Device::Type kDev>
class CloverBlockConv1d : public ::infini::ops::CloverBlockConv1d {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

海光triton实现的算子能帮忙加个单测覆盖一下不?

Move the clover operator family from Triton-only to the repository's
standard native layout: SIMT kernels in native/cuda/ops/ (one source
compiled by every CUDA-compatible toolchain) with generic host wrappers
(CudaX<Runtime<kDev>>), plus thin registration shells for NVIDIA and
Hygon. The Triton providers stay registered at implementation index 10
and become automatic fallbacks.

Operators:
- clover_live_mask / clover_mask_routes / clover_merge_moe: elementwise
  metadata kernels mirroring the clover_live_rows Triton stages.
- clover_coreset: three-stage block-routing whitelist. The mass stage
  uses one thread per expert with a rank-count top-8 test (deterministic,
  no atomics); the allow stage keeps experts until the normalized
  cumulative probability reaches top_p, mirroring the Triton reference
  (the reference's keep-OR-expert-0 operates in sorted coordinates,
  i.e. it pins the highest-mass expert, not expert id 0).
- clover_block_conv1d: 4-tap history + 3-tap symmetric block convolution
  with silu, dead-slot guards and per-row validity.
- clover_local_memory: read-only BD-GDN preview attention with the
  gate-decayed clean-state history bucket (L2-normalized q/k, per-head
  scalars, validity-aware softmax).
- clover_recurrent_gdn: sequential GDN state update with delta rule,
  final-state writeback and dead-slot/empty-range guards.

All kernels dispatch on the runtime dtype (bf16/fp16 data, fp32 state
and gates, int32/int64 indices where the interface allows both) — the
routing logits in the real model path are bf16, so fp32-only kernels
would silently corrupt them.

Tests (29 cases, all green on NVIDIA A100): torch golden models plus
Triton cross-checks per operator, covering block sizes 1..32, dead
slots, scattered validity masks, head-count mappings and empty ranges.
clover_coreset is exercised through tests/native/clover_native_adapter
(workspace-taking providers need an explicit Handle workspace).

End-to-end: Clover-35B-A3B-Preview TP2 on A100 — 'What is the capital
of France?' reproduces the Triton baseline token-for-token; the frontier
+ expert-budget paragraph stays semantically identical with ulp-level
numeric-path divergence only (greedy decoding is chaotic at that scale).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants