Skip to content

fix: refresh output-spec hook signatures after torch.export (#292) - #296

Merged
gspark-etri merged 1 commit into
optimus-rmfrom
fix/292-export-kv-torch210
Nov 4, 2026
Merged

gspark-etri merged 1 commit into
optimus-rmfrom
fix/292-export-kv-torch210

Conversation

@gspark-etri

Copy link
Copy Markdown
Collaborator

Background

On optimus-rm with torch 2.10.0 + transformers 4.46.2, KV-cache generation on the torch.export path (Optimus_Inference(..., use_kv_cache=True, dynamo_capture=True)) failed in prefill: B=1 IndexError: index 1 is out of bounds for dimension 0 with size 1, B=2 RuntimeError: shape '[2, 6, 4, 16]' is invalid (#292). torch 2.5.1 + transformers 4.46.2 and torch 2.10 + transformers 5.1.0 passed.

Root cause. It is not the KV rewrite (_inject_kv_cache_into_graph); the module graph rebuilt from the export was already wrong. register_output_spec_hooks records each leaf module's output signature as str(tuple(t.shape)) while export is tracing. With torch 2.10 + transformers 4.46 LLaMA, position_ids gets its own batch symbol, which a guard merges with input_ids' batch symbol only after layer 0's input_layernorm has run. That norm's spec therefore read (s72, s70, 64) while its graph node carries (s44, s70, 64). _sigs_match compares those strings, so the no-op alias fold (fp32 LlamaRMSNorm's .to(float32), added for the residual reuse) did not apply to that one module, the region kept two "outputs", and the call was indexed as a 2-tuple: getitem(norm, 1) indexed the batch dim of the returned tensor (B=1 IndexError, B=2 a wrong slice into q_proj, hence the reshape error). The other three norms were recorded after the merge and folded correctly.

Changes

  • opt_prime/opt_prime/IR.py: after export, re-render the hook signatures from the recorded FakeTensors (_tensor_output_sig(s[3])), so the shape strings reflect the final symbol replacements. Identity (id) and the tensor are unchanged.
  • opt_prime/tests/test_kv_cache_batch.py: test_export_kv_cache_trace, a CPU test (mirrors the HFTracer one) that the exported KV graph computes the model's logits at B=2. It fails without the fix on torch 2.10 + transformers 4.46.2 with the same shape '[2, 6, 4, 16]' is invalid error.

Verification

2x A100 development host, one A100 visible (CUDA_VISIBLE_DEVICES=1), commit d48b95b (base b8d6b3b).

CUDA_VISIBLE_DEVICES=1 TORCH_DEVICE_BACKEND_AUTOLOAD=0 PYTHONPATH=opt_prime <python> -m pytest opt_prime/tests/test_kv_cache_batch.py -q -rA
environment before (b8d6b3b) after (d48b95b)
torch 2.10.0 + transformers 4.46.2 export B=1, B=2 fail 6 passed (export B=1/B=2, HFTracer trace + B=1/B=2, new CPU test)
torch 2.5.1 + transformers 4.46.2 pass 6 passed
torch 2.10.0 + transformers 5.1.0 pass 3 passed, 3 skipped (HFTracer needs transformers < 5)

New test without the fix (torch 2.10 + transformers 4.46.2, CUDA_VISIBLE_DEVICES=): fails with RuntimeError: shape '[2, 6, 4, 16]' is invalid for input of size 384; with the fix it passes in all three environments.

CPU suite as in .github/workflows/cpu-tests.yml, torch 2.10.0 + transformers 5.1.0, CUDA_VISIBLE_DEVICES=:

cd opt_prime && python3 -m pytest -q tests          # 576 passed, 34 skipped, 135 subtests passed
python3 -m unittest discover -s opt_prime/examples/pd/native_kv -p 'test_*.py'   # OK (skipped=17)

Compatibility and limits

  • Only the strings used to match region outputs to module returns change; when shapes were already final (torch 2.5, transformers 5.x here) the result is identical.
  • Applies to the torch.export path only; the HFTracer path does not use these hooks.

Related Issue

Closes #292

🤖 Generated with Claude Code

The forward hooks record each leaf module's output shape as a string while
export is tracing. A later guard can still merge symbols: with torch 2.10 +
transformers 4.46 LLaMA, position_ids gets its own batch symbol that is
unified with input_ids' only after layer 0's input_layernorm ran, so its
spec read (s72, s70, 64) while the node carried (s44, s70, 64). The no-op
alias fold then did not match and the norm was indexed as a 2-tuple
(B=1 IndexError, B=2 invalid reshape in KV-cache generation).

Re-render the signatures from the recorded FakeTensors after export, and
add a CPU test that the exported KV graph computes the model's logits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gspark-etri
gspark-etri merged commit 9d70fe5 into optimus-rm Nov 4, 2026
1 check passed
@gspark-etri
gspark-etri deleted the fix/292-export-kv-torch210 branch November 4, 2026 02:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant