You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
accel.moe: CUDA training with an rngd expert tier hits the autograd stream assert #295
One process cannot train on CUDA while using the rngd MoE expert tier. Importing furiosa.torch registers rngd as the PrivateUse1 accelerator; torch then reports it as the current accelerator, the autograd engine keeps no CUDA streams, and every CUDA backward fails with opt_ready_stream && opt_parent_stream INTERNAL ASSERT FAILED (torch/csrc/autograd/engine.cpp:1085).
Evidence
Root-caused in #114 (PR #293): furiosa-torch 2026.2.0 on torch 2.10 triggers it, even for a one-line x.cuda()...backward(). Registering a dummy second accelerator on torch 2.13 reproduces the same assert, so it is not specific to torch 2.10. #293 only turns autoload off for the test suite; accel/platforms/rngd/experts.py still imports furiosa.torch explicitly. Inference is unaffected (no backward).
Scope
Decide whether MoE offload training with an rngd tier is supported. Options: run the rngd tier in a separate process (as the serving PD path does), or reject training with an rngd tier at construction with a clear message. Document the result in the MoE offload docs.
Done when
Training with an rngd expert tier either works on hardware with a test, or fails at construction with a clear message and a CPU test covers that.
Activity