Skip to content

tests: disable torch device-backend autoload so CUDA backward works (#114) - #293

Merged
gspark-etri merged 1 commit into
optimus-rmfrom
fix/114-moe-backward-stream
Nov 4, 2026
Merged

gspark-etri merged 1 commit into
optimus-rmfrom
fix/114-moe-backward-stream

Conversation

@gspark-etri

Copy link
Copy Markdown
Collaborator

Background

tests/test_moe_offload.py::test_backward_through_offloaded_experts failed in all 6 parametrizations with
RuntimeError: opt_ready_stream && opt_parent_stream INTERNAL ASSERT FAILED at torch/csrc/autograd/engine.cpp:1085
on the torch 2.10 environments with a GPU visible (#114).

Root cause: nothing in accel.moe. The torch 2.10 environments have furiosa-torch installed, which registers
itself through the torch.backends entry point (rngd = furiosa.torch:_register). On import torch it becomes the
PrivateUse1 accelerator, and torch.accelerator.current_accelerator() then reports rngd even though the test runs
on CUDA. The autograd engine only tracks streams for the current accelerator's device type, so CUDA nodes have no
ready or parent stream and the engine hits the assert. Any CUDA backward fails this way, without repo code:

CUDA_VISIBLE_DEVICES=0 python3 -c "import torch; x=torch.randn(4,4,device='cuda',requires_grad=True); (x*2).sum().backward()"
-> RuntimeError: opt_ready_stream && opt_parent_stream INTERNAL ASSERT FAILED ... engine.cpp:1085

This is not specific to torch 2.10. The torch 2.13 environment passed only because it has no furiosa-torch. With any
PrivateUse1 backend registered (torch.utils.backend_registration._setup_privateuseone_for_python_backend("npu")),
the same one-line CUDA backward fails identically on torch 2.13.0+cu130.

The test only looked GPU-specific because the CUDA paths are the ones that run backward. Earlier validation runs passed
TORCH_DEVICE_BACKEND_AUTOLOAD=0 by hand, which hid the problem. The repo already handles this for CUDA training:
examples/train_probe.py and examples/train_auto.py set the same variable, and docs/HETERO_MODEL_FEATURE_PLAN_KO.md
lists it.

Why the fix is in the tests and not in library code: the assert comes from process-global torch state, a second
accelerator registered at import torch, and opt_prime cannot undo that once torch is imported. Setting the variable
from inside the library would only work if the library were imported before torch. It would also silently turn off
autoloading for users who rely on it. The test process is a CUDA/CPU process, so its configuration is the right place.

Changes

  • opt_prime/tests/conftest.py: os.environ.setdefault("TORCH_DEVICE_BACKEND_AUTOLOAD", "0") before anything imports
    torch, with a short note on why. setdefault keeps an explicit value from the caller. Code that needs the rngd
    device (accel/platforms/rngd/experts.py) does import furiosa.torch, which registers it anyway (checked:
    with autoload off, import furiosa.torch gives current_accelerator() == rngd).

Verification

Run on the RNGD+A100 host, 1x A100 80GB PCIe (CUDA_VISIBLE_DEVICES=0), at e02ed6a on top of optimus-rm 52ff463.
The before/after runs do not set TORCH_DEVICE_BACKEND_AUTOLOAD (env -u TORCH_DEVICE_BACKEND_AUTOLOAD), so they
match a plain pytest invocation.

Target test: env -u TORCH_DEVICE_BACKEND_AUTOLOAD CUDA_VISIBLE_DEVICES=0 PYTHONPATH=opt_prime <python> -m pytest -q "opt_prime/tests/test_moe_offload.py::test_backward_through_offloaded_experts"

Environment Before (52ff463) After (e02ed6a)
torch 2.10.0+cu128 / transformers 5.1.0 (furiosa-torch 2026.2.0 installed) 6 failed (engine.cpp:1085 assert) 6 passed
torch 2.10.0+cu128 / transformers 5.8.1 (furiosa-torch installed) 6 failed (engine.cpp:1085 assert) 6 passed
torch 2.13.0+cu130 / transformers 5.15 (no furiosa-torch) 6 passed 6 passed

Whole file with the GPU visible, after: ... -m pytest -q opt_prime/tests/test_moe_offload.py

  • torch 2.10 / tf 5.1: 102 passed, 1 skipped
  • torch 2.10 / tf 5.8.1: 102 passed, 1 skipped
  • torch 2.13 / tf 5.15: 101 passed, 2 skipped

Full CPU suite, matching .github/workflows/cpu-tests.yml, after:
env -u TORCH_DEVICE_BACKEND_AUTOLOAD CUDA_VISIBLE_DEVICES= PYTHONPATH=opt_prime python3 -m pytest -q opt_prime/tests
gives 575 passed, 31 skipped, 135 subtests passed.

Compatibility and limits

  • Test-only change. Library behavior is unchanged.
  • If something imports torch before pytest loads opt_prime/tests/conftest.py (for example a pytest plugin that imports
    torch), the setting comes too late. None of the environments above do this.
  • Out of scope, worth a separate issue: one process cannot run CUDA autograd and also use the rngd expert tier,
    because importing furiosa.torch registers the second accelerator. MoE offload training with an rngd tier would hit
    this assert. Inference (no backward) is not affected.
  • Unrelated and already failing on this host with the GPU visible: 3 [cuda] cases in tests/test_pd_stream_admission.py
    fail with an inductor flex_attention NoValidChoicesError on torch 2.10. They fail the same way with
    autoload off and are not touched here.

Related Issue

Closes #114

🤖 Generated with Claude Code

…114)

furiosa-torch registers itself through the torch.backends entry point as the
PrivateUse1 accelerator. torch then reports rngd as the current accelerator,
the autograd engine records no CUDA streams, and every CUDA backward fails the
engine.cpp "opt_ready_stream && opt_parent_stream" assert. The tests run on
CUDA or the CPU, so conftest turns autoloading off before torch is imported;
the rngd backend imports furiosa.torch explicitly and still registers it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gspark-etri
gspark-etri merged commit da88589 into optimus-rm Nov 4, 2026
1 check passed
@gspark-etri
gspark-etri deleted the fix/114-moe-backward-stream branch November 4, 2026 02:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant