Repository navigation
fix: refresh output-spec hook signatures after torch.export (#292) - #296
Merged
Merged
Conversation
The forward hooks record each leaf module's output shape as a string while export is tracing. A later guard can still merge symbols: with torch 2.10 + transformers 4.46 LLaMA, position_ids gets its own batch symbol that is unified with input_ids' only after layer 0's input_layernorm ran, so its spec read (s72, s70, 64) while the node carried (s44, s70, 64). The no-op alias fold then did not match and the norm was indexed as a 2-tuple (B=1 IndexError, B=2 invalid reshape in KV-cache generation). Re-render the signatures from the recorded FakeTensors after export, and add a CPU test that the exported KV graph computes the model's logits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
On
optimus-rmwith torch 2.10.0 + transformers 4.46.2, KV-cache generation on the torch.export path (Optimus_Inference(..., use_kv_cache=True, dynamo_capture=True)) failed in prefill: B=1IndexError: index 1 is out of bounds for dimension 0 with size 1, B=2RuntimeError: shape '[2, 6, 4, 16]' is invalid(#292). torch 2.5.1 + transformers 4.46.2 and torch 2.10 + transformers 5.1.0 passed.Root cause. It is not the KV rewrite (
_inject_kv_cache_into_graph); the module graph rebuilt from the export was already wrong.register_output_spec_hooksrecords each leaf module's output signature asstr(tuple(t.shape))while export is tracing. With torch 2.10 + transformers 4.46 LLaMA,position_idsgets its own batch symbol, which a guard merges withinput_ids' batch symbol only after layer 0'sinput_layernormhas run. That norm's spec therefore read(s72, s70, 64)while its graph node carries(s44, s70, 64)._sigs_matchcompares those strings, so the no-op alias fold (fp32LlamaRMSNorm's.to(float32), added for the residual reuse) did not apply to that one module, the region kept two "outputs", and the call was indexed as a 2-tuple:getitem(norm, 1)indexed the batch dim of the returned tensor (B=1 IndexError, B=2 a wrong slice into q_proj, hence the reshape error). The other three norms were recorded after the merge and folded correctly.Changes
opt_prime/opt_prime/IR.py: after export, re-render the hook signatures from the recorded FakeTensors (_tensor_output_sig(s[3])), so the shape strings reflect the final symbol replacements. Identity (id) and the tensor are unchanged.opt_prime/tests/test_kv_cache_batch.py:test_export_kv_cache_trace, a CPU test (mirrors the HFTracer one) that the exported KV graph computes the model's logits at B=2. It fails without the fix on torch 2.10 + transformers 4.46.2 with the sameshape '[2, 6, 4, 16]' is invaliderror.Verification
2x A100 development host, one A100 visible (
CUDA_VISIBLE_DEVICES=1), commitd48b95b(baseb8d6b3b).b8d6b3b)d48b95b)New test without the fix (torch 2.10 + transformers 4.46.2,
CUDA_VISIBLE_DEVICES=): fails withRuntimeError: shape '[2, 6, 4, 16]' is invalid for input of size 384; with the fix it passes in all three environments.CPU suite as in
.github/workflows/cpu-tests.yml, torch 2.10.0 + transformers 5.1.0,CUDA_VISIBLE_DEVICES=:Compatibility and limits
Related Issue
Closes #292
🤖 Generated with Claude Code