Repository navigation
tests: disable torch device-backend autoload so CUDA backward works (#114) - #293
Merged
Merged
Conversation
…114) furiosa-torch registers itself through the torch.backends entry point as the PrivateUse1 accelerator. torch then reports rngd as the current accelerator, the autograd engine records no CUDA streams, and every CUDA backward fails the engine.cpp "opt_ready_stream && opt_parent_stream" assert. The tests run on CUDA or the CPU, so conftest turns autoloading off before torch is imported; the rngd backend imports furiosa.torch explicitly and still registers it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
tests/test_moe_offload.py::test_backward_through_offloaded_expertsfailed in all 6 parametrizations withRuntimeError: opt_ready_stream && opt_parent_stream INTERNAL ASSERT FAILED at torch/csrc/autograd/engine.cpp:1085on the torch 2.10 environments with a GPU visible (#114).
Root cause: nothing in
accel.moe. The torch 2.10 environments have furiosa-torch installed, which registersitself through the
torch.backendsentry point (rngd = furiosa.torch:_register). Onimport torchit becomes thePrivateUse1 accelerator, and
torch.accelerator.current_accelerator()then reportsrngdeven though the test runson CUDA. The autograd engine only tracks streams for the current accelerator's device type, so CUDA nodes have no
ready or parent stream and the engine hits the assert. Any CUDA backward fails this way, without repo code:
This is not specific to torch 2.10. The torch 2.13 environment passed only because it has no furiosa-torch. With any
PrivateUse1 backend registered (
torch.utils.backend_registration._setup_privateuseone_for_python_backend("npu")),the same one-line CUDA backward fails identically on torch 2.13.0+cu130.
The test only looked GPU-specific because the CUDA paths are the ones that run backward. Earlier validation runs passed
TORCH_DEVICE_BACKEND_AUTOLOAD=0by hand, which hid the problem. The repo already handles this for CUDA training:examples/train_probe.pyandexamples/train_auto.pyset the same variable, anddocs/HETERO_MODEL_FEATURE_PLAN_KO.mdlists it.
Why the fix is in the tests and not in library code: the assert comes from process-global torch state, a second
accelerator registered at
import torch, andopt_primecannot undo that once torch is imported. Setting the variablefrom inside the library would only work if the library were imported before torch. It would also silently turn off
autoloading for users who rely on it. The test process is a CUDA/CPU process, so its configuration is the right place.
Changes
opt_prime/tests/conftest.py:os.environ.setdefault("TORCH_DEVICE_BACKEND_AUTOLOAD", "0")before anything importstorch, with a short note on why.
setdefaultkeeps an explicit value from the caller. Code that needs the rngddevice (
accel/platforms/rngd/experts.py) doesimport furiosa.torch, which registers it anyway (checked:with autoload off,
import furiosa.torchgivescurrent_accelerator() == rngd).Verification
Run on the RNGD+A100 host, 1x A100 80GB PCIe (
CUDA_VISIBLE_DEVICES=0), at e02ed6a on top ofoptimus-rm52ff463.The before/after runs do not set
TORCH_DEVICE_BACKEND_AUTOLOAD(env -u TORCH_DEVICE_BACKEND_AUTOLOAD), so theymatch a plain
pytestinvocation.Target test:
env -u TORCH_DEVICE_BACKEND_AUTOLOAD CUDA_VISIBLE_DEVICES=0 PYTHONPATH=opt_prime <python> -m pytest -q "opt_prime/tests/test_moe_offload.py::test_backward_through_offloaded_experts"Whole file with the GPU visible, after:
... -m pytest -q opt_prime/tests/test_moe_offload.pyFull CPU suite, matching
.github/workflows/cpu-tests.yml, after:env -u TORCH_DEVICE_BACKEND_AUTOLOAD CUDA_VISIBLE_DEVICES= PYTHONPATH=opt_prime python3 -m pytest -q opt_prime/testsgives 575 passed, 31 skipped, 135 subtests passed.
Compatibility and limits
opt_prime/tests/conftest.py(for example a pytest plugin that importstorch), the setting comes too late. None of the environments above do this.
rngdexpert tier,because importing furiosa.torch registers the second accelerator. MoE offload training with an rngd tier would hit
this assert. Inference (no backward) is not affected.
[cuda]cases intests/test_pd_stream_admission.pyfail with an inductor
flex_attentionNoValidChoicesErroron torch 2.10. They fail the same way withautoload off and are not touched here.
Related Issue
Closes #114
🤖 Generated with Claude Code