Skip to content

fix: KV cache on the HFTracer path traces with use_cache=False (#191) - #291

Merged
gspark-etri merged 1 commit into
optimus-rmfrom
fix/191-hftracer-kv-cache
Nov 4, 2026
Merged

gspark-etri merged 1 commit into
optimus-rmfrom
fix/191-hftracer-kv-cache

Conversation

@gspark-etri

Copy link
Copy Markdown
Collaborator

Background

On transformers 4.x, Optimus_Inference(model, use_kv_cache=True, dynamo_capture=False) (the default HFTracer capture path) failed while building the IR, at any batch size:

IR._retrieve_IR_trace -> tracer.trace(model, concrete_args=...)
NotImplementedError: argument of type: <class 'transformers.utils.fx.ProxyableDynamicCache'>

Root cause (confirmed): _retrieve_IR_trace builds concrete_args from the forward signature defaults, so use_cache is None and the model falls back to config.use_cache=True. Llama then creates a DynamicCache for past_key_values, HFTracer wraps it as ProxyableDynamicCache, and torch.fx cannot emit that object as a graph output. Tracing the tiny Llama directly with HFTracer: use_cache default fails (with and without position_ids in input_names); use_cache=False traces and outputs {'logits': ...}. KV-cache inference does not use the HF cache output: _inject_kv_cache_into_graph replaces the SDPA calls with CachedScaledDotProductAttention modules after the IR is built.

Changes

  • opt_prime/opt_prime/IR.py (_retrieve_IR_trace, HF branch): when use_kv_cache is set, pass use_cache=False as a concrete arg (5 lines including a comment). The non-KV HFTracer path and the torch.export path are unchanged.
  • opt_prime/tests/test_kv_cache_batch.py:
    • new test_hftracer_kv_cache_trace (CPU): IR.retrieve_IR(tiny, use_kv_cache=True, dynamo_capture=False) traces, and the graph's logits match the model's at B=2.
    • test_kv_cache_generation_matches_hf (GPU) is now parametrized over dynamo_capture in {True, False}: greedy generation with the KV cache matches HF generate at B=1 and B=2.
    • The HFTracer cases skip when transformers.utils.fx is unavailable (transformers >= 5, which CI uses); the GPU skip moved from module level to the generation test so the CPU trace test runs without a GPU.

Model in the tests: tiny Llama from config (2 layers, hidden 64, 4 heads, 2 KV heads, vocab 128, attn_implementation="sdpa", float32).

Verification

All commands run from opt_prime/ with TORCH_DEVICE_BACKEND_AUTOLOAD=0 PYTHONPATH=. on a 2x A100 development host (GPU cases on one A100; Optimus_Inference does not support CPU mode).

Commit: 2f00678 on top of optimus-rm 52ff463.

environment command result
torch 2.10.0+cu128, transformers 4.46.2 (optimus-rm 4.46 config) CUDA_VISIBLE_DEVICES= python -m pytest -q -rs tests/test_kv_cache_batch.py 1 passed, 4 skipped (GPU)
same CUDA_VISIBLE_DEVICES=1 python -m pytest -q -rA tests/test_kv_cache_batch.py trace test and HFTracer generation B=1, B=2 pass; torch.export generation B=1, B=2 fail (pre-existing, see below)
same, without the IR.py change CUDA_VISIBLE_DEVICES= python -m pytest -q tests/test_kv_cache_batch.py trace test fails with NotImplementedError: ... ProxyableDynamicCache (reproduces the issue)
torch 2.5.1+cu124, transformers 4.46.2 CUDA_VISIBLE_DEVICES= python -m pytest -q -rs tests/test_kv_cache_batch.py 1 passed, 4 skipped (GPU)
same CUDA_VISIBLE_DEVICES=1 python -m pytest -q tests/test_kv_cache_batch.py 5 passed (both capture paths, B=1 and B=2)
torch 2.10.0+cu128, transformers 5.1.0 (CI equivalent) CUDA_VISIBLE_DEVICES= python -m pytest -q -rs tests/test_kv_cache_batch.py 5 skipped (HFTracer needs transformers < 5 / GPU)
same CUDA_VISIBLE_DEVICES=1 python -m pytest -q -rs tests/test_kv_cache_batch.py 2 passed (torch.export B=1, B=2), 3 skipped (HFTracer)
same, full CPU suite as in cpu-tests.yml CUDA_VISIBLE_DEVICES= python -m pytest -q tests 575 passed, 34 skipped, 135 subtests passed
same CUDA_VISIBLE_DEVICES= python -m unittest discover -s opt_prime/examples/pd/native_kv -p 'test_*.py' (repo root) OK (skipped=17)

Correctness criterion: generated token IDs equal HF greedy generate(max_new_tokens=5, do_sample=False) exactly (torch.equal), prompt length 6. A standalone script with one engine for B=1 then B=2 also gave identical tokens to HF in both 4.46.2 environments.

Pre-existing, not caused by this PR: on torch 2.10 + transformers 4.46.2 the existing dynamo_capture=True generation test fails on unmodified optimus-rm too (B=1 IndexError: index 1 is out of bounds for dimension 0 with size 1, B=2 RuntimeError: shape '[2, 6, 4, 16]' is invalid for input of size 384). It passes on torch 2.5.1 + 4.46.2 and on torch 2.10 + 5.1.0. Worth a separate issue.

Compatibility and limits

  • Only optimus-rm is changed. main / v4.1.x (where the issue was first reported) is not changed by this PR; the same one-line change should apply there when integrating.
  • Only affects use_kv_cache=True with dynamo_capture=False on transformers 4.x. On transformers 5.x the HFTracer path stays unavailable (_retrieve_IR_trace exits with the "use --dynamo-capture" message), and the new HFTracer tests skip.
  • The non-KV HFTracer path still traces with the model's use_cache default; examples set use_cache=False in the config for that path. Not changed here.
  • Verified with Llama only (the issue's reproduction); other HF families on the HFTracer KV path are not covered by the test.

Related Issue

Closes #191

🤖 Generated with Claude Code

…he HFTracer path (#191)

With use_cache left at its default, transformers 4.x returns a DynamicCache that HFTracer cannot emit
as a graph output (ProxyableDynamicCache), so Optimus_Inference(use_kv_cache=True, dynamo_capture=False)
failed while building the IR. KV-cache inference injects CachedSDPA modules afterwards and does not use
that output, so trace with use_cache=False.

Test: the HFTracer graph traces with use_kv_cache=True and matches the model's logits (CPU), and KV-cache
greedy generation matches HF at B=1/B=2 on both capture paths (GPU). HFTracer cases skip on transformers 5.x.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gspark-etri
gspark-etri merged commit b8d6b3b into optimus-rm Nov 4, 2026
1 check passed
@gspark-etri
gspark-etri deleted the fix/191-hftracer-kv-cache branch November 4, 2026 02:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant