Skip to content

feat(ccl): auto-select AllToAll, Scatter and Gather native vs fallback implementations - #92

Merged
Ziminli merged 3 commits into
masterfrom
feat/ccl-auto-api-selection
Nov 20, 2026
Merged

Ziminli merged 3 commits into
masterfrom
feat/ccl-auto-api-selection

Conversation

@Ziminli

@Ziminli Ziminli commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add a configure-time capability-probing mechanism so each CCL backend automatically selects between a vendor-native collective API and a grouped Send/Recv fallback, and apply it to AllToAll, Scatter, and Gather.

The new infiniccl_configure_ccl_capabilities() in cmake/CCLCapabilities.cmake try-compiles every registered native call for each enabled CCL backend at configure time and emits a generated capabilities.h. The Api layer keeps AllToAll(), Scatter(), and Gather() available on every backend and dispatches with if constexpr on the generated constants: when the installed SDK provides the native call it is used directly, otherwise the operation transparently falls back to the grouped Send/Recv implementation in src/backends/ccl/common/impl/.

No public API change is needed, and SDKs without the native calls (for example NCCL 2.22.3, where none of the three probes link) keep working while newer SDKs take the native fast path.

Changes

  • Configure-time capability probing

    • Add cmake/CCLCapabilities.cmake defining infiniccl_configure_ccl_capabilities(), called once per enabled CCL backend from cmake/backends/NCCL.cmake and cmake/backends/MCCL.cmake, and wire the generated include directory into CMakeLists.txt.
    • Register the probed operations with the Op|call[|alternate-call] grammar; AllToAll registers both spellings (e.g., ncclAlltoAll/ncclAllToAll).
    • Probe each spelling with CheckCXXSourceCompiles and emit ${CMAKE_BINARY_DIR}/src/backends/ccl/<provider>/capabilities.h holding kHas<Op>, the selected <op>Alternate spelling, and the kHasNative<Op> alias; the header is rewritten only when its content changes.
    • Cache probe results in capabilities.stamp keyed by an MD5 over the SDK fingerprint (generator version, header path, registered ops, include directories, definitions, and library paths with modification times); reconfiguring with an unchanged SDK performs zero probes.
    • Print one configure summary line per operation, e.g. CCL[nccl] Scatter: native (spelling 1 of 1) or CCL[nccl] Gather: no native API in this SDK (fallback).
    • Reject malformed registry entries (duplicate operations or more than two spellings) with a FATAL_ERROR at configure time.
  • API layer

    • Keep Api::AllToAll(), Api::Scatter(), and Api::Gather() present for both CCL backends in src/backends/ccl/nccl/api.h and src/backends/ccl/mccl/api.h, dispatching with if constexpr on the generated constants.
    • Wrap the native call in Check(...) when available; otherwise return kNotSupported after a single LOG(INFO) and let the common implementation layer take the grouped path.
  • Common collective implementations

    • Restructure src/backends/ccl/common/impl/all_to_all.h, scatter.h, and gather.h around three steps: Prepare(...) validates arguments, ApplyGrouped(...) expresses the operation as per-peer Send/Recv pairs inside a single Group, and Apply(...) selects the native API or the grouped path via the generated constants.
    • The grouped paths latch the first error and always issue GroupEnd before returning, so a failed peer exchange cannot leave the communicator group open.
  • Examples

    • Update examples/ccl_mpi_hybrid/all_to_all.cc for the restructured AllToAll entry points (call-site adaptation only).

Platform and Backend Affected

Platform

  • CPU
  • NVIDIA GPU
  • Iluvatar GPU
  • MetaX GPU
  • Moore Threads GPU
  • Cambricon MLU
  • HYGON DCU
  • T-Head PPU
  • Ascend NPU

Backend

  • OpenMPI
  • MPICH
  • NCCL/RCCL
  • MCCL
  • CNCL
  • HCCL

Performance Impact

  • No performance impact
  • Performance improved
  • Performance regression possible

Backends whose SDK provides the native calls now dispatch to them directly instead of always building the collective from Send/Recv groups; on SDKs without the calls the previous grouped behavior is unchanged. Bandwidths below are bus_bw from two-GPU runs of the ccl_mpi_hybrid examples at the default payload (fp32); every run reported Correct: YES.

Stack Scatter (GB/s) Gather (GB/s) AllToAll (GB/s) Selected API
NCCL 2.30.4 (CUDA 13.2) 82.0 77.5 54.1 native ncclScatter/ncclGather/ncclAlltoAll
NCCL 2.22.3 (CUDA 12.6) 85.9 79.1 55.7 grouped ncclSend/ncclRecv
MCCL on MetaX C550 44.9 45.9 43.7 native mcclScatter/mcclGather/mcclAllToAll
MCCL on Moore S5000 22.8 19.2 14.7 grouped mcclSend/mcclRecv

The NCCL 2.22.3 fallback performs on par with the 2.30.4 native path at these payloads, so the fallback is not a cliff; Scatter on NCCL 2.30.4 reaches approximately 237 GB/s at 4 MB per rank. The Moore numbers were taken while the node was externally loaded and are not representative peaks.

Known Issues & Future Work

  • Probe links run sequentially at configure time, which adds some extra time for a first-time build, but the capabilities.stamp cache removes them entirely on reconfigures with an unchanged SDK.
  • The registry grammar supports at most two spellings per operation, which covers the current NCCL and MCCL headers; raising the cap is a small localized change in cmake/CCLCapabilities.cmake if a future SDK needs more.
  • Possible follow-up: expose the generated constants as a template<Device::Type device> struct Capabilities that the Api layer inherits, instead of per-operation aliases.

Test Results

Test Involved Platform

  • CPU
  • NVIDIA GPU
  • Iluvatar GPU
  • MetaX GPU
  • Moore Threads GPU
  • Cambricon MLU
  • HYGON DCU
  • T-Head PPU
  • Ascend NPU

Test Involved Backend

  • OpenMPI
  • MPICH
  • NCCL/RCCL
  • MCCL
  • CNCL
  • HCCL

NVIDIA (NCCL + OMPI):
ccl_mpi_hybrid_all_gather.log
ccl_mpi_hybrid_all_reduce.log
ccl_mpi_hybrid_all_to_all.log
ccl_mpi_hybrid_gather.log
ccl_mpi_hybrid_reduce.log
ccl_mpi_hybrid_scatter.log
ccl_mpi_hybrid_send_recv.log

MetaX (MCCL + OMPI):
ccl_mpi_hybrid_all_gather.log
ccl_mpi_hybrid_all_reduce.log
ccl_mpi_hybrid_all_to_all.log
ccl_mpi_hybrid_gather.log
ccl_mpi_hybrid_reduce.log
ccl_mpi_hybrid_scatter.log
ccl_mpi_hybrid_send_recv.log

Moore Threads (MCCL + OMPI):
ccl_mpi_hybrid_all_gather.log
ccl_mpi_hybrid_all_reduce.log
ccl_mpi_hybrid_all_to_all.log
ccl_mpi_hybrid_gather.log
ccl_mpi_hybrid_reduce.log
ccl_mpi_hybrid_scatter.log
ccl_mpi_hybrid_send_recv.log

HYGON (RCCL + OMPI):
ccl_mpi_hybrid_all_gather.log
ccl_mpi_hybrid_all_reduce.log
ccl_mpi_hybrid_all_to_all.log
ccl_mpi_hybrid_gather.log
ccl_mpi_hybrid_reduce.log
ccl_mpi_hybrid_scatter.log
ccl_mpi_hybrid_send_recv.log


Checklist

Every contributor must verify every item below before requesting
review. Tick each box only after the check has actually been performed —
do not tick speculatively. If an item truly does not apply, replace the
checkbox with N/A and briefly explain why in an inline comment.

Title, Branch, and Commits

  • PR title follows Conventional Commits (e.g. feat: …, fix(nccl): …).
  • Branch name follows <type>/xxx-yyyy-zzzz where <type> matches the PR title's Conventional Commits type and words are joined with hyphens (see CONTRIBUTING.md §Branches).
  • Each commit message follows Conventional Commits.
  • Small PR is a single squashable commit; or, for a large PR, every commit is meaningful, well-formed, and independently reviewable (see CONTRIBUTING.md §Pull Requests).
  • No stray merge commits from master — the branch is rebased cleanly on top of the current master.
  • No fixup! / squash! / wip commits remain.

Scope and Design

  • Changes are minimal — no unrelated modifications were introduced (CONTRIBUTING.md §Code/General).
  • No dead code, commented-out blocks, debug prints, printf/std::cout/print(...) left behind, or TODO without an owner and issue link.
  • No unrelated formatting churn that would obscure the diff.
  • N/A- Public API changes (if any) are intentional, documented, and reflected in affected callers/tests.

General Code Hygiene

  • The code is self-explanatory; comments were added only where the intent or rationale is non-obvious (CONTRIBUTING.md §Code/General).
  • Every modified or added file ends with a single trailing newline (CONTRIBUTING.md §Code/General).
  • No trailing whitespace, inconsistent indentation, or mixed formatting styles remain.
  • Identifiers referenced in comments or error messages are wrapped in Markdown backticks (e.g. the `AllReduce` implementation) (CONTRIBUTING.md §Code/General).
  • All comments and error messages are in English (CONTRIBUTING.md §Code/General).
  • Comments and error messages are complete sentences — capitalized first letter, terminal punctuation — unless the language/framework convention says otherwise (CONTRIBUTING.md §Code/General; §Python).

C++ Specific (if C++ files changed)

  • Code follows the Google C++ Style Guide strictly.
  • clang-format (version 16, per .github/workflows/clang-format.yml) has been run against all modified applicable files; the diff is clean.
  • No exceptions are thrown. Error paths use assert with messages that include at least __FILE__, __LINE__, and __func__ (CONTRIBUTING.md §C++).
  • Error and warning message wording follows the LLVM Coding Standards (CONTRIBUTING.md §C++).
  • Constructor initializer list order matches member declaration order (CONTRIBUTING.md §C++).
  • Exactly one blank line between classes, between classes and functions, and between functions (CONTRIBUTING.md §C++).
  • Exactly one blank line between members (functions and variables) within a class (CONTRIBUTING.md §C++).
  • Exactly one blank line before and after the contents of a namespace (CONTRIBUTING.md §C++).

Python Specific (if Python files changed)

N/A. No Python files were changed.

Testing

  • All applicable example programs have been built and tested successfully on at least one supported heterogeneous cluster setup.

Build, CI, and Tooling

  • N/A- New backends or devices have been added to auto-detection in CMakeLists.txt under if(AUTO_DETECT_DEVICES) or to if(AUTO_DETECT_BACKENDS) if applicable.
  • Both CI workflows (clang-format.yml, ruff.yml) are green locally (or expected to be green on CI).

Documentation

  • README.md, CONTRIBUTING.md, or inline docs updated when behavior, build flags, or developer workflow changed.
  • N/A- Any user-visible breaking change is called out explicitly under "Summary" and in the commit/PR title with a ! or BREAKING CHANGE: footer.

Security and Safety

  • No secrets, access tokens, internal URLs, customer data, or personal hardware identifiers have been committed.
  • N/A- Third-party code is license-compatible and attributed.
  • No unsafe pointer arithmetic, uninitialized reads, or missing bounds checks were introduced.

At configure time, `infiniccl_configure_ccl_capabilities()` probes each
registered operation spelling (`ncclAlltoAll` vs `ncclAllToAll`) and
generates `capabilities.h`; `api.h` then dispatches to the native API
when the SDK provides it, else falls back to grouped `Send`/`Recv`.
Probe results are cached by SDK fingerprint; registering another API
is one registry line per spelling plus its dispatch branch.

Also fix the hybrid `AllToAll` example timer to exclude the validation
copy.
Register both operations with the configure-time capability probe so
`capabilities.h` selects `ncclScatter`/`ncclGather` where the SDK
provides them and grouped `Send`/`Recv` fallback elsewhere (e.g. NCCL
2.22.3), mirroring `AllToAll`.
@Ziminli Ziminli self-assigned this Nov 20, 2026
@Ziminli
Ziminli merged commit 93ed9b4 into master Nov 20, 2026
2 checks passed
@Ziminli
Ziminli deleted the feat/ccl-auto-api-selection branch November 20, 2026 03:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant