Repository navigation
feat(ccl): auto-select AllToAll, Scatter and Gather native vs fallback implementations - #92
Merged
Merged
Conversation
At configure time, `infiniccl_configure_ccl_capabilities()` probes each registered operation spelling (`ncclAlltoAll` vs `ncclAllToAll`) and generates `capabilities.h`; `api.h` then dispatches to the native API when the SDK provides it, else falls back to grouped `Send`/`Recv`. Probe results are cached by SDK fingerprint; registering another API is one registry line per spelling plus its dispatch branch. Also fix the hybrid `AllToAll` example timer to exclude the validation copy.
Register both operations with the configure-time capability probe so `capabilities.h` selects `ncclScatter`/`ncclGather` where the SDK provides them and grouped `Send`/`Recv` fallback elsewhere (e.g. NCCL 2.22.3), mirroring `AllToAll`.
…for the generated capability headers
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a configure-time capability-probing mechanism so each CCL backend automatically selects between a vendor-native collective API and a grouped
Send/Recvfallback, and apply it toAllToAll,Scatter, andGather.The new
infiniccl_configure_ccl_capabilities()incmake/CCLCapabilities.cmaketry-compiles every registered native call for each enabled CCL backend at configure time and emits a generatedcapabilities.h. TheApilayer keepsAllToAll(),Scatter(), andGather()available on every backend and dispatches withif constexpron the generated constants: when the installed SDK provides the native call it is used directly, otherwise the operation transparently falls back to the groupedSend/Recvimplementation insrc/backends/ccl/common/impl/.No public API change is needed, and SDKs without the native calls (for example NCCL 2.22.3, where none of the three probes link) keep working while newer SDKs take the native fast path.
Changes
Configure-time capability probing
cmake/CCLCapabilities.cmakedefininginfiniccl_configure_ccl_capabilities(), called once per enabled CCL backend fromcmake/backends/NCCL.cmakeandcmake/backends/MCCL.cmake, and wire the generated include directory intoCMakeLists.txt.Op|call[|alternate-call]grammar;AllToAllregisters both spellings (e.g.,ncclAlltoAll/ncclAllToAll).CheckCXXSourceCompilesand emit${CMAKE_BINARY_DIR}/src/backends/ccl/<provider>/capabilities.hholdingkHas<Op>, the selected<op>Alternatespelling, and thekHasNative<Op>alias; the header is rewritten only when its content changes.capabilities.stampkeyed by an MD5 over the SDK fingerprint (generator version, header path, registered ops, include directories, definitions, and library paths with modification times); reconfiguring with an unchanged SDK performs zero probes.CCL[nccl] Scatter: native (spelling 1 of 1)orCCL[nccl] Gather: no native API in this SDK (fallback).FATAL_ERRORat configure time.API layer
Api::AllToAll(),Api::Scatter(), andApi::Gather()present for both CCL backends insrc/backends/ccl/nccl/api.handsrc/backends/ccl/mccl/api.h, dispatching withif constexpron the generated constants.Check(...)when available; otherwise returnkNotSupportedafter a singleLOG(INFO)and let the common implementation layer take the grouped path.Common collective implementations
src/backends/ccl/common/impl/all_to_all.h,scatter.h, andgather.haround three steps:Prepare(...)validates arguments,ApplyGrouped(...)expresses the operation as per-peerSend/Recvpairs inside a singleGroup, andApply(...)selects the native API or the grouped path via the generated constants.GroupEndbefore returning, so a failed peer exchange cannot leave the communicator group open.Examples
examples/ccl_mpi_hybrid/all_to_all.ccfor the restructuredAllToAllentry points (call-site adaptation only).Platform and Backend Affected
Platform
Backend
Performance Impact
Backends whose SDK provides the native calls now dispatch to them directly instead of always building the collective from
Send/Recvgroups; on SDKs without the calls the previous grouped behavior is unchanged. Bandwidths below arebus_bwfrom two-GPU runs of theccl_mpi_hybridexamples at the default payload (fp32); every run reportedCorrect: YES.ncclScatter/ncclGather/ncclAlltoAllncclSend/ncclRecvmcclScatter/mcclGather/mcclAllToAllmcclSend/mcclRecvThe NCCL 2.22.3 fallback performs on par with the 2.30.4 native path at these payloads, so the fallback is not a cliff;
Scatteron NCCL 2.30.4 reaches approximately 237 GB/s at 4 MB per rank. The Moore numbers were taken while the node was externally loaded and are not representative peaks.Known Issues & Future Work
capabilities.stampcache removes them entirely on reconfigures with an unchanged SDK.cmake/CCLCapabilities.cmakeif a future SDK needs more.template<Device::Type device> struct Capabilitiesthat theApilayer inherits, instead of per-operation aliases.Test Results
Test Involved Platform
Test Involved Backend
NVIDIA (NCCL + OMPI):
ccl_mpi_hybrid_all_gather.log
ccl_mpi_hybrid_all_reduce.log
ccl_mpi_hybrid_all_to_all.log
ccl_mpi_hybrid_gather.log
ccl_mpi_hybrid_reduce.log
ccl_mpi_hybrid_scatter.log
ccl_mpi_hybrid_send_recv.log
MetaX (MCCL + OMPI):
ccl_mpi_hybrid_all_gather.log
ccl_mpi_hybrid_all_reduce.log
ccl_mpi_hybrid_all_to_all.log
ccl_mpi_hybrid_gather.log
ccl_mpi_hybrid_reduce.log
ccl_mpi_hybrid_scatter.log
ccl_mpi_hybrid_send_recv.log
Moore Threads (MCCL + OMPI):
ccl_mpi_hybrid_all_gather.log
ccl_mpi_hybrid_all_reduce.log
ccl_mpi_hybrid_all_to_all.log
ccl_mpi_hybrid_gather.log
ccl_mpi_hybrid_reduce.log
ccl_mpi_hybrid_scatter.log
ccl_mpi_hybrid_send_recv.log
HYGON (RCCL + OMPI):
ccl_mpi_hybrid_all_gather.log
ccl_mpi_hybrid_all_reduce.log
ccl_mpi_hybrid_all_to_all.log
ccl_mpi_hybrid_gather.log
ccl_mpi_hybrid_reduce.log
ccl_mpi_hybrid_scatter.log
ccl_mpi_hybrid_send_recv.log
Checklist
Title, Branch, and Commits
feat: …,fix(nccl): …).<type>/xxx-yyyy-zzzzwhere<type>matches the PR title's Conventional Commits type and words are joined with hyphens (seeCONTRIBUTING.md§Branches).CONTRIBUTING.md§Pull Requests).master— the branch is rebased cleanly on top of the currentmaster.fixup!/squash!/wipcommits remain.Scope and Design
CONTRIBUTING.md§Code/General).printf/std::cout/print(...)left behind, orTODOwithout an owner and issue link.General Code Hygiene
CONTRIBUTING.md§Code/General).CONTRIBUTING.md§Code/General).the `AllReduce` implementation) (CONTRIBUTING.md§Code/General).CONTRIBUTING.md§Code/General).CONTRIBUTING.md§Code/General; §Python).C++ Specific (if C++ files changed)
clang-format(version 16, per.github/workflows/clang-format.yml) has been run against all modified applicable files; the diff is clean.assertwith messages that include at least__FILE__,__LINE__, and__func__(CONTRIBUTING.md§C++).CONTRIBUTING.md§C++).CONTRIBUTING.md§C++).CONTRIBUTING.md§C++).CONTRIBUTING.md§C++).CONTRIBUTING.md§C++).Python Specific (if Python files changed)
N/A. No Python files were changed.
Testing
Build, CI, and Tooling
CMakeLists.txtunderif(AUTO_DETECT_DEVICES)or toif(AUTO_DETECT_BACKENDS)if applicable.clang-format.yml,ruff.yml) are green locally (or expected to be green on CI).Documentation
README.md,CONTRIBUTING.md, or inline docs updated when behavior, build flags, or developer workflow changed.!orBREAKING CHANGE:footer.Security and Safety