Repository navigation
fix: KV cache on the HFTracer path traces with use_cache=False (#191) - #291
Merged
Merged
Conversation
…he HFTracer path (#191) With use_cache left at its default, transformers 4.x returns a DynamicCache that HFTracer cannot emit as a graph output (ProxyableDynamicCache), so Optimus_Inference(use_kv_cache=True, dynamo_capture=False) failed while building the IR. KV-cache inference injects CachedSDPA modules afterwards and does not use that output, so trace with use_cache=False. Test: the HFTracer graph traces with use_kv_cache=True and matches the model's logits (CPU), and KV-cache greedy generation matches HF at B=1/B=2 on both capture paths (GPU). HFTracer cases skip on transformers 5.x. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
On transformers 4.x,
Optimus_Inference(model, use_kv_cache=True, dynamo_capture=False)(the default HFTracer capture path) failed while building the IR, at any batch size:Root cause (confirmed):
_retrieve_IR_tracebuildsconcrete_argsfrom the forward signature defaults, souse_cacheisNoneand the model falls back toconfig.use_cache=True. Llama then creates aDynamicCacheforpast_key_values, HFTracer wraps it asProxyableDynamicCache, and torch.fx cannot emit that object as a graph output. Tracing the tiny Llama directly with HFTracer:use_cachedefault fails (with and withoutposition_idsininput_names);use_cache=Falsetraces and outputs{'logits': ...}. KV-cache inference does not use the HF cache output:_inject_kv_cache_into_graphreplaces the SDPA calls withCachedScaledDotProductAttentionmodules after the IR is built.Changes
opt_prime/opt_prime/IR.py(_retrieve_IR_trace, HF branch): whenuse_kv_cacheis set, passuse_cache=Falseas a concrete arg (5 lines including a comment). The non-KV HFTracer path and the torch.export path are unchanged.opt_prime/tests/test_kv_cache_batch.py:test_hftracer_kv_cache_trace(CPU):IR.retrieve_IR(tiny, use_kv_cache=True, dynamo_capture=False)traces, and the graph's logits match the model's at B=2.test_kv_cache_generation_matches_hf(GPU) is now parametrized overdynamo_capturein {True, False}: greedy generation with the KV cache matches HFgenerateat B=1 and B=2.transformers.utils.fxis unavailable (transformers >= 5, which CI uses); the GPU skip moved from module level to the generation test so the CPU trace test runs without a GPU.Model in the tests: tiny Llama from config (2 layers, hidden 64, 4 heads, 2 KV heads, vocab 128,
attn_implementation="sdpa", float32).Verification
All commands run from
opt_prime/withTORCH_DEVICE_BACKEND_AUTOLOAD=0 PYTHONPATH=.on a 2x A100 development host (GPU cases on one A100;Optimus_Inferencedoes not support CPU mode).Commit:
2f00678on top ofoptimus-rm52ff463.CUDA_VISIBLE_DEVICES= python -m pytest -q -rs tests/test_kv_cache_batch.pyCUDA_VISIBLE_DEVICES=1 python -m pytest -q -rA tests/test_kv_cache_batch.pyCUDA_VISIBLE_DEVICES= python -m pytest -q tests/test_kv_cache_batch.pyNotImplementedError: ... ProxyableDynamicCache(reproduces the issue)CUDA_VISIBLE_DEVICES= python -m pytest -q -rs tests/test_kv_cache_batch.pyCUDA_VISIBLE_DEVICES=1 python -m pytest -q tests/test_kv_cache_batch.pyCUDA_VISIBLE_DEVICES= python -m pytest -q -rs tests/test_kv_cache_batch.pyCUDA_VISIBLE_DEVICES=1 python -m pytest -q -rs tests/test_kv_cache_batch.pycpu-tests.ymlCUDA_VISIBLE_DEVICES= python -m pytest -q testsCUDA_VISIBLE_DEVICES= python -m unittest discover -s opt_prime/examples/pd/native_kv -p 'test_*.py'(repo root)Correctness criterion: generated token IDs equal HF greedy
generate(max_new_tokens=5, do_sample=False)exactly (torch.equal), prompt length 6. A standalone script with one engine for B=1 then B=2 also gave identical tokens to HF in both 4.46.2 environments.Pre-existing, not caused by this PR: on torch 2.10 + transformers 4.46.2 the existing
dynamo_capture=Truegeneration test fails on unmodifiedoptimus-rmtoo (B=1IndexError: index 1 is out of bounds for dimension 0 with size 1, B=2RuntimeError: shape '[2, 6, 4, 16]' is invalid for input of size 384). It passes on torch 2.5.1 + 4.46.2 and on torch 2.10 + 5.1.0. Worth a separate issue.Compatibility and limits
optimus-rmis changed. main / v4.1.x (where the issue was first reported) is not changed by this PR; the same one-line change should apply there when integrating.use_kv_cache=Truewithdynamo_capture=Falseon transformers 4.x. On transformers 5.x the HFTracer path stays unavailable (_retrieve_IR_traceexits with the "use --dynamo-capture" message), and the new HFTracer tests skip.use_cachedefault; examples setuse_cache=Falsein the config for that path. Not changed here.Related Issue
Closes #191
🤖 Generated with Claude Code