Skip to content

feat: support CCL Scatter - #89

Merged
Ziminli merged 3 commits into
InfiniTensor:masterfrom
xingjiangjun:feat/ccl-scatter-followup
Nov 12, 2026
Merged

Ziminli merged 3 commits into
InfiniTensor:masterfrom
xingjiangjun:feat/ccl-scatter-followup

Conversation

@xingjiangjun

@xingjiangjun xingjiangjun commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add native NCCL/MCCL Scatter support through the existing shared CCL framework.

This PR is a follow-up to #68, originally contributed by @GordonYang1.

Changes

  • CCL Backend Support
    • Call native ncclScatter() / mcclScatter() through a shared implementation.
    • Align parameter validation with existing collective operations.
  • Examples
    • Add Native and Hybrid Scatter examples following the MPI example conventions.

Platform and Backend Affected

Platform

  • CPU
  • NVIDIA GPU
  • Iluvatar GPU
  • MetaX GPU
  • Moore Threads GPU
  • Cambricon MLU
  • HYGON DCU
  • T-Head PPU
  • Ascend NPU

Backend

  • OpenMPI
  • MPICH
  • NCCL/RCCL
  • MCCL
  • CNCL
  • HCCL

Performance Impact

  • No performance impact
  • Performance improved
  • Performance regression possible

N/A — New operator support; no comparative performance claim.

Known Issues & Future Work

  • Native NVIDIA Scatter requires NCCL 2.28.3 or newer. Other compatible SDKs and RDMA/RoCE remain unverified.

Test Results

Test Involved Platform

  • CPU
  • NVIDIA GPU
  • Iluvatar GPU
  • MetaX GPU
  • Moore Threads GPU
  • Cambricon MLU
  • HYGON DCU
  • T-Head PPU
  • Ascend NPU

Test Involved Backend

  • OpenMPI
  • MPICH
  • NCCL/RCCL
  • MCCL
  • CNCL
  • HCCL

NVIDIA A100 — NCCL 2.30.4:

  • Native Scatter, 1/2/4/8 GPUs — 5/5 PASS. Logs.
  • Hybrid Scatter, 2/8 GPUs — 2/2 PASS. Logs.
  • Nonzero root, in-place, datatypes and parameter checks — 12/12 PASS. Logs.

MetaX C550 — MCCL 2.16.5, OpenMPI 4.1.6:

  • Native Scatter, 1/2/4/8 GPUs — 5/5 PASS. Logs.
  • Hybrid Scatter, 2/8 GPUs — 2/2 PASS. Logs.
  • Nonzero root and root in-place — 2/2 PASS. Logs.

NVIDIA + MetaX — OpenMPI 4.1.6, UCX/TCP:

  • MPI Scatter, 2+2/8+8 GPUs — 2/2 PASS; supplementary all-rank validation — 2/2 PASS. Logs.

Native 8-GPU coverage includes counts 16 and 4096.


Checklist

Every contributor must verify every item below before requesting
review. Tick each box only after the check has actually been performed —
do not tick speculatively. If an item truly does not apply, replace the
checkbox with N/A and briefly explain why in an inline comment.

Title, Branch, and Commits

  • PR title follows Conventional Commits (e.g. feat: …, fix(nccl): …).
  • Branch name follows <type>/xxx-yyyy-zzzz where <type> matches the PR title's Conventional Commits type and words are joined with hyphens (see CONTRIBUTING.md §Branches).
  • Each commit message follows Conventional Commits.
  • Small PR is a single squashable commit; or, for a large PR, every commit is meaningful, well-formed, and independently reviewable (see CONTRIBUTING.md §Pull Requests).
  • No stray merge commits from master — the branch is rebased cleanly on top of the current master.
  • No fixup! / squash! / wip commits remain.

Scope and Design

  • Changes are minimal — no unrelated modifications were introduced (CONTRIBUTING.md §Code/General).
  • No dead code, commented-out blocks, debug prints, printf/std::cout/print(...) left behind, or TODO without an owner and issue link.
  • No unrelated formatting churn that would obscure the diff.
  • Public API changes (if any) are intentional, documented, and reflected in affected callers/tests.

General Code Hygiene

  • The code is self-explanatory; comments were added only where the intent or rationale is non-obvious (CONTRIBUTING.md §Code/General).
  • Every modified or added file ends with a single trailing newline (CONTRIBUTING.md §Code/General).
  • No trailing whitespace, inconsistent indentation, or mixed formatting styles remain.
  • N/A: Identifiers referenced in comments or error messages are wrapped in Markdown backticks (e.g. the `AllReduce` implementation) (CONTRIBUTING.md §Code/General). — Existing example section-label conventions retained.
  • All comments and error messages are in English (CONTRIBUTING.md §Code/General).
  • Comments and error messages are complete sentences — capitalized first letter, terminal punctuation — unless the language/framework convention says otherwise (CONTRIBUTING.md §Code/General; §Python).

C++ Specific (if C++ files changed)

  • N/A: Code follows the Google C++ Style Guide strictly. — Follows existing CCL example conventions.
  • clang-format (version 16, per .github/workflows/clang-format.yml) has been run against all modified applicable files; the diff is clean.
  • N/A: No exceptions are thrown. Error paths use assert with messages that include at least __FILE__, __LINE__, and __func__ (CONTRIBUTING.md §C++). — Uses existing CHECK_RT/CHECK_INFINI, ReturnStatus, and example CLI parsing; no assert-only error handling.
  • Error and warning message wording follows the LLVM Coding Standards (CONTRIBUTING.md §C++).
  • N/A: Constructor initializer list order matches member declaration order (CONTRIBUTING.md §C++). — No constructors added.
  • Exactly one blank line between classes, between classes and functions, and between functions (CONTRIBUTING.md §C++).
  • N/A: Exactly one blank line between members (functions and variables) within a class (CONTRIBUTING.md §C++). — Follows the existing example ThreadArgs layout.
  • Exactly one blank line before and after the contents of a namespace (CONTRIBUTING.md §C++).

Python Specific (if Python files changed)

N/A — No Python files changed.

Testing

  • All applicable example programs have been built and tested successfully on at least one supported heterogeneous cluster setup. — Scope: Scatter; MPI on a 2+2/8+8 NVIDIA–MetaX cluster, Native and Hybrid separately on their supported platforms.

Build, CI, and Tooling

  • N/A: New backends or devices have been added to auto-detection in CMakeLists.txt under if(AUTO_DETECT_DEVICES) or to if(AUTO_DETECT_BACKENDS) if applicable. — No new backend or device.
  • Both CI workflows (clang-format.yml, ruff.yml) are green locally (or expected to be green on CI).

Documentation

  • README.md, CONTRIBUTING.md, or inline docs updated when behavior, build flags, or developer workflow changed.
  • N/A: Any user-visible breaking change is called out explicitly under "Summary" and in the commit/PR title with a ! or BREAKING CHANGE: footer. — No existing operation removed or changed incompatibly; native NVIDIA support requires the version stated above.

Security and Safety

  • No secrets, access tokens, internal URLs, customer data, or personal hardware identifiers have been committed.
  • N/A: Third-party code is license-compatible and attributed. — No third-party code added.
  • No unsafe pointer arithmetic, uninitialized reads, or missing bounds checks were introduced.

@xingjiangjun xingjiangjun changed the title feat: support CCL Scatter feat: support CCL ’Scatter‘ Nov 11, 2026
@xingjiangjun xingjiangjun changed the title feat: support CCL ’Scatter‘ feat: support CCL Scatter Nov 11, 2026
@xingjiangjun xingjiangjun changed the title feat: support CCL Scatter feat: support CCL Scatter Nov 11, 2026
@xingjiangjun
xingjiangjun requested a review from Ziminli November 11, 2026 04:15
@xingjiangjun
xingjiangjun force-pushed the feat/ccl-scatter-followup branch from 3517f46 to c4278e2 Compare November 12, 2026 07:41
@Ziminli
Ziminli merged commit 95baf09 into InfiniTensor:master Nov 12, 2026
2 checks passed
@Ziminli Ziminli mentioned this pull request Nov 12, 2026
41 of 52 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants