Repository navigation
Conversation
This was referenced Oct 29, 2026
humannumber196
force-pushed
the
feat/hygon-clover-operators
branch
from
November 17, 2026 07:59
fb1e360 to
c26e298
Compare
The clover* Triton operators and MoeFusedDense were registered only for a single device backend (Hygon), which made the whole Clover diffusion path unbuildable and unrunnable on any other Triton-capable platform. Refactor the seven clover operators following the established pattern used by shared operators such as Add: - Extract the device-specific inline Operator specializations into template <Device::Type kDev> implementation classes under infini::ops::triton::jit, with Launch<kDev> parameterization. - Register thin Operator<Op, kNvidia/kHygon, 10> shells for BOTH Triton-capable backends. Hygon behavior is unchanged. - moe_fused_dense already had a templated implementation but only a Hygon shell; add the kNvidia shell. - block_conv1d / local_memory / recurrent_gdn already templated with a kNvidia shell; add the missing kHygon shell for symmetry. Also fix a CMake glob bug in the WITH_NVIDIA branch: NVIDIA_PATTERNS uses GLOB_RECURSE with a top-level 'native/cuda/*.cu' pattern that also matches sibling vendor subtrees (hygon/iluvatar/metax/moore/...), so their kernels were compiled twice and failed to link with multiple definition errors when both backends' sources were present. Add the symmetric exclusion that the WITH_HYGON branch already has. Verified on NVIDIA A100 (CUDA 13.1, sm80) with Clover-35B-A3B-Preview under InfiniLM PR InfiniTensor#604: TP2 block-diffusion generation produces byte-identical outputs and per-block iteration counts before and after this refactor.
Absorb 136 upstream files (cambricon topk provider, wrapper-generator and conftest updates, pinned deps) so the branch merges cleanly into master. Auto-merge; the only remaining deltas versus master are this PR's clover operators, the multi-device Triton shells, and the NVIDIA CMake vendor-exclusion fix.
Ringssss
marked this pull request as ready for review
November 19, 2026 07:36
| namespace infini::ops { | ||
|
|
||
| template <> | ||
| class Operator<MoeAlignBlockSize, Device::Type::kHygon, 0> |
Collaborator
There was a problem hiding this comment.
这个算子是不是可以把具体实现放到cuda下面,然后这里继承啊?
Comment on lines
+2301
to
+2306
| parser.add_argument( | ||
| "--declarations", | ||
| nargs="*", | ||
| default=[], | ||
| help="Also emit operator APIs with no active provider.", | ||
| ) |
Collaborator
There was a problem hiding this comment.
应该可以通过构建自己的ops.json实现类似的功能,举例:https://github.com/InfiniTensor/InfiniOps/blob/master/scripts/configs/infiniops_ops_nvidia_dense_sampling_torch_free_no_marlin.json#L52-L54
编译方式添加DINFINI_OPS_OPS参数
cmake -S submodules/InfiniOps -B build/InfiniOps \
-DWITH_TORCH=ON \
-DWITH_LINKED=ON \
-DINFINI_OPS_OPS=/workspace/ops.json \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX="$INFINI_ROOT"
这个修改应该不是必要的
| namespace infini::ops::triton::jit { | ||
|
|
||
| template <Device::Type kDev> | ||
| class CloverBlockConv1d : public ::infini::ops::CloverBlockConv1d { |
Collaborator
There was a problem hiding this comment.
海光triton实现的算子能帮忙加个单测覆盖一下不?
Move the clover operator family from Triton-only to the repository's standard native layout: SIMT kernels in native/cuda/ops/ (one source compiled by every CUDA-compatible toolchain) with generic host wrappers (CudaX<Runtime<kDev>>), plus thin registration shells for NVIDIA and Hygon. The Triton providers stay registered at implementation index 10 and become automatic fallbacks. Operators: - clover_live_mask / clover_mask_routes / clover_merge_moe: elementwise metadata kernels mirroring the clover_live_rows Triton stages. - clover_coreset: three-stage block-routing whitelist. The mass stage uses one thread per expert with a rank-count top-8 test (deterministic, no atomics); the allow stage keeps experts until the normalized cumulative probability reaches top_p, mirroring the Triton reference (the reference's keep-OR-expert-0 operates in sorted coordinates, i.e. it pins the highest-mass expert, not expert id 0). - clover_block_conv1d: 4-tap history + 3-tap symmetric block convolution with silu, dead-slot guards and per-row validity. - clover_local_memory: read-only BD-GDN preview attention with the gate-decayed clean-state history bucket (L2-normalized q/k, per-head scalars, validity-aware softmax). - clover_recurrent_gdn: sequential GDN state update with delta rule, final-state writeback and dead-slot/empty-range guards. All kernels dispatch on the runtime dtype (bf16/fp16 data, fp32 state and gates, int32/int64 indices where the interface allows both) — the routing logits in the real model path are bf16, so fp32-only kernels would silently corrupt them. Tests (29 cases, all green on NVIDIA A100): torch golden models plus Triton cross-checks per operator, covering block sizes 1..32, dead slots, scattered validity masks, head-count mappings and empty ranges. clover_coreset is exercised through tests/native/clover_native_adapter (workspace-taking providers need an explicit Handle workspace). End-to-end: Clover-35B-A3B-Preview TP2 on A100 — 'What is the capital of France?' reproduces the Triton baseline token-for-token; the frontier + expert-budget paragraph stays semantically identical with ulp-level numeric-path divergence only (greedy decoding is chaotic at that scale).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Motivation
Clover's noisy-block previews must preserve committed recurrent state and read a bidirectional block. The existing Hygon providers cannot execute that model. The companion InfiniLM integration uses these kernels directly from C++, with optional expert budgeting/live-row reuse during preview and full causal replay for commit.
Dependency: InfiniTensor/InfiniRT#52. The build was also checked against the installed Hygon SDK.
Type of Change
feat— new operators / backend capabilitiesbuild— installed API generationtest— numerical/state and JIT metadata checksPlatforms Affected
WITH_HYGON, added to this template)Smoke Test Result
Hygon BW/gfx936, DTK 26.04, Torch 2.10.0, Triton 3.5.0, BF16 data and FP32 recurrent state:
The rebased build passed 62 wrapper/JIT/public-header and Clover tests. The required CPU/Hygon smoke selection passed: 121 passed, 43 skipped (inactive providers), 9485 deselected. Build, tooling and checked-in tests are documented in
docs/clover-hygon.md.tests/native/clover_workspace_smoke.cppexercises workspace-bearing providers through the native public API. Missing workspaces are rejected before launch. The original unguarded Python functional invocation failed; that failed prototype is not included as a passing result.Test Results on Supported Platforms
Benchmark / Performance Impact
The companion native InfiniLM campaign uses real Clover-35B-A3B-Preview weights, BF16 TP2, 32 short prompts repeated three times, concurrency one and a common OpenAI chat client. All 96 requests per lane completed. Visible output tokens are retokenized consistently; padded diffusion usage counters are excluded.
Measured integration throughput after upstream sync: native baseline 28.57 tok/s; frontier stride 38.47; expert budget/live reuse 28.70; joint 39.92; local SGLang Clover port 5.88. These are end-to-end integration results, not isolated operator speedups. The measurement record retains all requests, failures, warmups and rejected candidates. Final smoke repairs also cover strided casts and preserving the vendor HIP driver during autotuning.
Separate traces contain GPU events on both TP devices. Live-row kernels occur only in the joint path; summed routed-MoE GPU duration decreases from 1339 ms to 721 ms in the representative profiled request. Profiling time is excluded from throughput and summed device durations are not wall time.
Notes for Reviewers
Preview state is read-only. Causal replay remains exact to the specified recurrent equations; live-row reuse is an opt-in approximation controlled by the LM. The H100 DP/EP sequence-parallel communication pipeline is not ported here.
INFINI_OPS_DECLARATIONSexposes unsupported APIs for compilation/capability queries; it does not create fallback providers. Preserve explicit workspace lifetime and same-stream ordering. The code-generation comment-default issue found during bring-up was already fixed upstream; the branch keeps the upstream implementation.Source provenance, API alignment, supported layouts and licenses are recorded in the operator guide and
THIRD_PARTY.txt. Cross-platform accelerator CI is still needed for the shared tooling changes.