Skip to content

accel.moe: CUDA training with an rngd expert tier hits the autograd stream assert #295

Open

Description

@gspark-etri

Problem

One process cannot train on CUDA while using the rngd MoE expert tier. Importing furiosa.torch registers rngd as the PrivateUse1 accelerator; torch then reports it as the current accelerator, the autograd engine keeps no CUDA streams, and every CUDA backward fails with opt_ready_stream && opt_parent_stream INTERNAL ASSERT FAILED (torch/csrc/autograd/engine.cpp:1085).

Evidence

Root-caused in #114 (PR #293): furiosa-torch 2026.2.0 on torch 2.10 triggers it, even for a one-line x.cuda()...backward(). Registering a dummy second accelerator on torch 2.13 reproduces the same assert, so it is not specific to torch 2.10. #293 only turns autoload off for the test suite; accel/platforms/rngd/experts.py still imports furiosa.torch explicitly. Inference is unaffected (no backward).

Scope

Decide whether MoE offload training with an rngd tier is supported. Options: run the rngd tier in a separate process (as the serving PD path does), or reject training with an rngd tier at construction with a clear message. Document the result in the MoE offload docs.

Done when

Training with an rngd expert tier either works on hardware with a test, or fails at construction with a clear message and a CPU test covers that.

Owner

gspark-etri

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions