Repository navigation
feat(hygon): add native Clover diffusion and sparse MoE operators #1015
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Ringssss
wants to merge
4
commits into
InfiniTensor:master
Choose a base branch
from
Ringssss:feat/hygon-clover-operators
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
2b0da3c
feat(hygon): add native Clover diffusion and sparse MoE operators
Ringssss c26e298
feat(triton): make clover and moe_fused_dense ops multi-device
humannumber196 6fb0716
Merge official master (f52af91a) into feat/hygon-clover-operators
humannumber196 934475d
feat(clover): add shared-layer native providers for all seven clover ops
humannumber196 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| # Hygon Clover operators | ||
|
|
||
| These operators support the companion InfiniLM Clover implementation. Model | ||
| forward stays in C++; Hygon driver modules launch warmed Triton binaries. | ||
| Compilation uses the installed DTK Triton HIP target (for example `gfx936`, | ||
| wave size 64), not CUDA compute-capability arithmetic. | ||
|
|
||
| ## Build | ||
|
|
||
| Install an InfiniRT revision containing the Hygon driver interface first | ||
| ([InfiniRT PR #52](https://github.com/InfiniTensor/InfiniRT/pull/52)). Then: | ||
|
|
||
| ```sh | ||
| cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \ | ||
| -DCMAKE_INSTALL_PREFIX="$INFINI_ROOT" -DINFINI_RT_ROOT="$INFINI_ROOT" \ | ||
| -DWITH_CPU=ON -DWITH_HYGON=ON -DHYGON_ARCH=gfx936 -DWITH_TRITON=ON \ | ||
| -DGENERATE_PYTHON_BINDINGS=ON -DAUTO_DETECT_DEVICES=OFF \ | ||
| -DAUTO_DETECT_BACKENDS=OFF | ||
| cmake --build build -j 12 | ||
| cmake --install build | ||
| ``` | ||
|
|
||
| The Python `infini.triton` package must be importable. CMake installs its package | ||
| contents at the prefix, while wheel installation supplies the `infini` namespace. | ||
| Include DTK's CUDA compatibility library directory in `LD_LIBRARY_PATH`. | ||
|
|
||
| `INFINI_OPS_DECLARATIONS` accepts a semicolon-separated list of additional public | ||
| operator APIs. These can have no provider in the selected build: capability | ||
| queries return unsupported, and execution is rejected. This allows a downstream | ||
| application to compile its complete model registry without claiming that every | ||
| registered model runs on Hygon. InfiniLM supplies a matching declaration preset. | ||
|
|
||
| ## Operator contracts and source alignment | ||
|
|
||
| The Clover-specific APIs target the released Clover model equations and state | ||
| layout, rather than claiming to be PyTorch standard operators. Preview kernel | ||
| provenance and licensing are recorded in `THIRD_PARTY.txt`. | ||
|
|
||
| | Operator | Inputs and behavior | | ||
| | --- | --- | | ||
| | `CloverBlockConv1d` | Packed `[tokens,channels]` BF16/FP16 input, `[slots,channels,3]` clean history, slot indices, left `[channels,1,4]` and learned right `[channels,1,3]` weights, int8 validity, block width up to 32. Produces SiLU output without updating history. | | ||
| | `CloverLocalMemory` | Q/K/V `[1,tokens,heads,128]`, FP32 gates and `[slots,value_heads,128,128]` state, slot indices, validity and three per-head scalars. Implements bidirectional block attention plus the decayed clean-history bucket. Does not update state. | | ||
| | `CloverRecurrentGdn` | Packed Q/K/V, gates/beta, sequence offsets and initial/final slot vectors. Implements normalized causal delta recurrence with FP32 persistent state; writes sequence output and final state. Input/final slot indices must be valid and final destinations must not race. | | ||
| | `CloverCoreset` | Contiguous `[tokens,256]` router logits, block width and routing mass. Keeps the stable top-p support of aggregated per-token top-8 scores and masks other logits. Requires an explicit workspace. | | ||
| | `CloverLiveMask` | Current/previous int64 token IDs and the mask ID. Marks rows live if either token is masked, or refresh is forced; updates previous IDs. | | ||
| | `CloverMaskRoutes` | Replaces inactive rows' int32 expert IDs with the out-of-range sentinel `num_experts`. | | ||
| | `CloverMergeMoe` | Selects fresh routed output for live rows and cached output for other rows; updates the cache. Output and fresh input may alias. | | ||
| | `MoeFusedDense` (Hygon) | Existing packed dense-MoE API, with BF16/FP16 weights, top-k weights/IDs and aligned expert metadata. Triton computes gate/up, fused SiLU/down, and weighted reduction. Invalid routes contribute zero; alignment block size 16 is supported and expert biases are unsupported. Requires an explicit workspace. | | ||
|
|
||
| The live-row protocol is an optional approximation for iterative diffusion, | ||
| adapted from the supplied dllm-h100 TV6 study. It is not a general guarantee that | ||
| fixed token IDs imply unchanged hidden states. InfiniLM refreshes every five | ||
| previews and always runs full causal replay before committing state. | ||
|
|
||
| The standard `MoeAlignBlockSize`, causal convolution, gating, cast, fill, | ||
| multiplication, triangular masks and rotary providers also gain Hygon coverage. | ||
| They reuse this repository's existing native CUDA-like implementations. | ||
|
|
||
| ## Validation | ||
|
|
||
| Actual BW/DTK runs compare preview operators to independent Torch equations and | ||
| verify bitwise state immutability, mixed-length causal state commits, coreset | ||
| selection, dense MoE, liveness and cache merging. Public-header and wrapper/JIT | ||
| unit tests cover the HIP architecture string and metadata without CUDA scratch | ||
| fields. The companion InfiniLM campaign runs full checkpoint generation. | ||
|
|
||
| Workspace-bearing operators are invoked through the C++ API with an owning | ||
| workspace. The native test adapter in `tests/native/clover_workspace_smoke.cpp` | ||
| exercises that path. A missing workspace is rejected before launching a kernel. | ||
|
|
||
| Build and run the workspace adapter against the installed SDK: | ||
|
|
||
| ```sh | ||
| python tests/native/build_clover_workspace.py | ||
| PYTHONPATH="build/clover-tests:$PYTHONPATH" python tests/native/check_clover_workspace.py | ||
| python -m pytest tests/test_clover_preview.py tests/test_clover_live_rows.py -q | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| #pragma once | ||
|
|
||
| #include "operator.h" | ||
|
|
||
| namespace infini::ops { | ||
|
|
||
| // Clover preview convolution reads a clean prefix without updating its state. | ||
| class CloverBlockConv1d : public Operator<CloverBlockConv1d> { | ||
| public: | ||
| CloverBlockConv1d(const Tensor input, const Tensor state, | ||
| const Tensor indices, const Tensor weight, | ||
| const Tensor right_weight, const Tensor valid, | ||
| size_t block_size, Tensor out) { | ||
| assert((input.ndim() == 2 && out.shape() == input.shape() && | ||
| state.ndim() == 3 && weight.ndim() == 3 && | ||
| weight.shape() == Tensor::Shape{input.size(1), 1, 4} && | ||
| right_weight.shape() == Tensor::Shape{input.size(1), 1, 3} && | ||
| state.size(1) == input.size(1) && state.size(2) == 3 && | ||
| indices.ndim() == 1 && block_size > 0 && block_size <= 32 && | ||
| input.size(0) == indices.numel() * block_size && | ||
| valid.shape() == Tensor::Shape{input.size(0)}) && | ||
| "CloverBlockConv1d: invalid block or state shape."); | ||
| assert((input.dtype() == DataType::kBFloat16 || | ||
| input.dtype() == DataType::kFloat16) && | ||
| "CloverBlockConv1d: expected BF16 or FP16 input."); | ||
| for (const auto& tensor : {state, weight, right_weight, out}) { | ||
| assert(tensor.dtype() == input.dtype() && | ||
| "CloverBlockConv1d: input and weight dtypes must match."); | ||
| } | ||
| assert((indices.dtype() == DataType::kInt32 || | ||
| indices.dtype() == DataType::kInt64) && | ||
| indices.IsContiguous() && valid.dtype() == DataType::kInt8 && | ||
| valid.IsContiguous() && | ||
| "CloverBlockConv1d: expected contiguous indices and int8 validity."); | ||
| for (const auto& tensor : | ||
| {state, indices, weight, right_weight, valid, out}) { | ||
| assert(tensor.device() == input.device() && | ||
| "CloverBlockConv1d: device mismatch."); | ||
| } | ||
| assert(!out.HasBroadcastDim() && "CloverBlockConv1d: overlapping output."); | ||
| } | ||
|
|
||
| virtual void operator()(const Tensor input, const Tensor state, | ||
| const Tensor indices, const Tensor weight, | ||
| const Tensor right_weight, const Tensor valid, | ||
| size_t block_size, Tensor out) const = 0; | ||
| }; | ||
|
|
||
| } // namespace infini::ops |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| #pragma once | ||
|
|
||
| #include "operator.h" | ||
|
|
||
| namespace infini::ops { | ||
|
|
||
| class CloverCoreset : public Operator<CloverCoreset> { | ||
| public: | ||
| CloverCoreset(const Tensor logits, size_t block_size, double top_p, | ||
| Tensor out) { | ||
| assert(logits.ndim() == 2 && logits.size(1) == 256 && block_size > 0 && | ||
| block_size <= 32 && logits.size(0) % block_size == 0 && | ||
| out.shape() == logits.shape() && out.dtype() == logits.dtype() && | ||
| logits.IsContiguous() && out.IsContiguous() && | ||
| logits.device() == out.device() && top_p > 0 && top_p <= 1 && | ||
| "CloverCoreset: invalid logits, block size or routing mass."); | ||
| } | ||
|
|
||
| virtual void operator()(const Tensor logits, size_t block_size, double top_p, | ||
| Tensor out) const = 0; | ||
| }; | ||
|
|
||
| } // namespace infini::ops |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| #pragma once | ||
| #include "operator.h" | ||
|
|
||
| namespace infini::ops { | ||
|
|
||
| class CloverLiveMask : public Operator<CloverLiveMask> { | ||
| public: | ||
| CloverLiveMask(const Tensor ids, Tensor previous, int64_t mask_id, | ||
| bool refresh, Tensor live) { | ||
| assert(ids.IsContiguous() && previous.IsContiguous() && | ||
| live.IsContiguous() && ids.numel() == previous.numel() && | ||
| ids.numel() == live.numel() && ids.dtype() == DataType::kInt64 && | ||
| previous.dtype() == ids.dtype() && live.dtype() == DataType::kInt8 && | ||
| ids.device() == previous.device() && ids.device() == live.device() && | ||
| "CloverLiveMask: invalid token state."); | ||
| } | ||
|
|
||
| virtual void operator()(const Tensor ids, Tensor previous, int64_t mask_id, | ||
| bool refresh, Tensor live) const = 0; | ||
| }; | ||
|
|
||
| } // namespace infini::ops |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| #pragma once | ||
|
|
||
| #include "operator.h" | ||
|
|
||
| namespace infini::ops { | ||
|
|
||
| class CloverLocalMemory : public Operator<CloverLocalMemory> { | ||
| public: | ||
| CloverLocalMemory(const Tensor q, const Tensor k, const Tensor v, | ||
| const Tensor g, const Tensor state, const Tensor indices, | ||
| const Tensor valid, const Tensor logit_scale, | ||
| const Tensor history_bias, const Tensor history_c, | ||
| size_t block_size, Tensor out) { | ||
| assert((q.ndim() == 4 && q.size(0) == 1 && k.shape() == q.shape() && | ||
| v.ndim() == 4 && v.size(0) == 1 && v.size(1) == q.size(1) && | ||
| q.size(2) > 0 && v.size(2) > 0 && v.size(2) % q.size(2) == 0 && | ||
| q.size(3) == 128 && v.size(3) == 128 && out.shape() == v.shape() && | ||
| state.ndim() == 4 && state.size(1) == v.size(2) && | ||
| state.size(2) == 128 && state.size(3) == 128 && | ||
| indices.ndim() == 1 && block_size > 0 && block_size <= 32 && | ||
| q.size(1) == indices.numel() * block_size && | ||
| g.shape() == Tensor::Shape{1, q.size(1), v.size(2)} && | ||
| valid.shape() == Tensor::Shape{q.size(1)}) && | ||
| "CloverLocalMemory: invalid sequence, head or state shape."); | ||
| assert( | ||
| (q.dtype() == DataType::kBFloat16 || q.dtype() == DataType::kFloat16) && | ||
| k.dtype() == q.dtype() && v.dtype() == q.dtype() && | ||
| out.dtype() == q.dtype() && state.dtype() == DataType::kFloat32 && | ||
| g.dtype() == DataType::kFloat32 && | ||
| "CloverLocalMemory: expected reduced data and FP32 state/gates."); | ||
| assert((indices.dtype() == DataType::kInt32 || | ||
| indices.dtype() == DataType::kInt64) && | ||
| indices.IsContiguous() && valid.dtype() == DataType::kInt8 && | ||
| valid.IsContiguous() && | ||
| "CloverLocalMemory: invalid index/validity dtype."); | ||
| for (const auto& scalar : {logit_scale, history_bias, history_c}) { | ||
| assert(scalar.ndim() == 1 && scalar.numel() == v.size(2) && | ||
| scalar.IsContiguous() && | ||
| (scalar.dtype() == q.dtype() || | ||
| scalar.dtype() == DataType::kFloat32) && | ||
| "CloverLocalMemory: invalid per-head scalars."); | ||
| } | ||
| for (const auto& tensor : {k, v, g, state, indices, valid, logit_scale, | ||
| history_bias, history_c, out}) { | ||
| assert(tensor.device() == q.device() && | ||
| "CloverLocalMemory: device mismatch."); | ||
| } | ||
| assert(!out.HasBroadcastDim() && "CloverLocalMemory: overlapping output."); | ||
| } | ||
|
|
||
| virtual void operator()(const Tensor q, const Tensor k, const Tensor v, | ||
| const Tensor g, const Tensor state, | ||
| const Tensor indices, const Tensor valid, | ||
| const Tensor logit_scale, const Tensor history_bias, | ||
| const Tensor history_c, size_t block_size, | ||
| Tensor out) const = 0; | ||
| }; | ||
|
|
||
| } // namespace infini::ops |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| #pragma once | ||
| #include "operator.h" | ||
|
|
||
| namespace infini::ops { | ||
|
|
||
| class CloverMaskRoutes : public Operator<CloverMaskRoutes> { | ||
| public: | ||
| CloverMaskRoutes(const Tensor ids, const Tensor live, int64_t num_experts, | ||
| Tensor out) { | ||
| assert(ids.ndim() == 2 && ids.IsContiguous() && live.IsContiguous() && | ||
| out.IsContiguous() && live.numel() == ids.size(0) && | ||
| out.shape() == ids.shape() && ids.dtype() == DataType::kInt32 && | ||
| out.dtype() == ids.dtype() && live.dtype() == DataType::kInt8 && | ||
| ids.device() == live.device() && ids.device() == out.device() && | ||
| "CloverMaskRoutes: invalid routes."); | ||
| } | ||
|
|
||
| virtual void operator()(const Tensor ids, const Tensor live, | ||
| int64_t num_experts, Tensor out) const = 0; | ||
| }; | ||
|
|
||
| } // namespace infini::ops |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| #pragma once | ||
| #include "operator.h" | ||
|
|
||
| namespace infini::ops { | ||
|
|
||
| class CloverMergeMoe : public Operator<CloverMergeMoe> { | ||
| public: | ||
| CloverMergeMoe(const Tensor fresh, const Tensor live, Tensor cache, | ||
| Tensor out) { | ||
| assert(fresh.ndim() == 2 && live.numel() == fresh.size(0) && | ||
| cache.shape() == fresh.shape() && out.shape() == fresh.shape() && | ||
| cache.dtype() == fresh.dtype() && out.dtype() == fresh.dtype() && | ||
| live.dtype() == DataType::kInt8 && fresh.IsContiguous() && | ||
| cache.IsContiguous() && out.IsContiguous() && live.IsContiguous() && | ||
| fresh.device() == live.device() && | ||
| fresh.device() == cache.device() && fresh.device() == out.device() && | ||
| "CloverMergeMoe: invalid cached rows."); | ||
| } | ||
|
|
||
| virtual void operator()(const Tensor fresh, const Tensor live, Tensor cache, | ||
| Tensor out) const = 0; | ||
| }; | ||
|
|
||
| } // namespace infini::ops |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
应该可以通过构建自己的ops.json实现类似的功能,举例:https://github.com/InfiniTensor/InfiniOps/blob/master/scripts/configs/infiniops_ops_nvidia_dense_sampling_torch_free_no_marlin.json#L52-L54
编译方式添加DINFINI_OPS_OPS参数
这个修改应该不是必要的