You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
core: KV cache with the default HFTracer path fails on v4.1.1 (ProxyableDynamicCache) #191
On v4.1.1 (62a7831, main), Optimus_Inference(model, use_kv_cache=True) with the default capture path (HFTracer, dynamo_capture=False) fails while building the IR, for any batch size:
opt_prime/inference.py:263 in __init__
opt_prime/IR.py:1657 in retrieve_IR
opt_prime/IR.py:1706 in _retrieve_IR_trace (tracer.trace(model, concrete_args=...))
NotImplementedError: argument of type: <class 'transformers.utils.fx.ProxyableDynamicCache'>
So on the release, KV-cache generation only works with dynamo_capture=True (torch.export path), but nothing says so and the default path crashes.
crashed before #190 (KV cache at batch >= 2), matches after
The result is the same before and after the v4.1.2 hotfix PR #190, so this is a separate pre-existing problem.
Probable cause (not verified): _retrieve_IR_trace leaves use_cache out of input_names, so it is passed as a concrete arg with its default. The model then builds a DynamicCache for past_key_values, HFTracer wraps it as ProxyableDynamicCache, and torch.fx cannot emit it as a graph output. The KV path replaces attention with CachedScaledDotProductAttention anyway (_inject_kv_cache_into_graph), so tracing with use_cache=False may be enough. This needs checking.
Environment
The support matrix stated by v4.1.1 (opt_prime/README.md): PyTorch 2.5.0 / CUDA 12.4 (torch 2.5.0+cu124), transformers==4.46.2, Python 3.10, one A100 80GB.
On optimus-rm with transformers 5.x, the HFTracer path is intentionally unavailable: _retrieve_IR_trace exits with "HFTracer is unavailable ... use --dynamo-capture". So this affects only transformers 4.x setups: main/v4.1.x, and optimus-rm's transformers 4.46 configuration.
Scope
IR._retrieve_IR_trace for HF models with use_kv_cache=True. Alternatively, if the HFTracer + KV-cache combination is not meant to be supported, reject it up front with a clear message and document that KV cache needs --dynamo-capture.
Done when
On torch 2.5.0 + transformers 4.46.2, use_kv_cache=True with dynamo_capture=False either matches HF greedy at B=1 and B=2, or fails at construction with a clear "use dynamo_capture=True" error. A test covers whichever behaviour is chosen.
Fixed on optimus-rm by #291 (b8d6b3b): HFTracer now traces with use_cache=False when use_kv_cache=True; matches HF greedy at B=1/B=2 on torch 2.5.1 and 2.10 with transformers 4.46.2. Leaving this open because main / v4.1.x still has the bug; it closes when optimus-rm is integrated into main (or with a separate main hotfix). The torch.export failure found on torch 2.10 + transformers 4.46 is tracked in #292.
Activity
fix: trace HF models with use_cache=False for KV-cache inference on t…
gspark-etri commentedon Nov 4, 2026
Fixed on optimus-rm by #291 (b8d6b3b): HFTracer now traces with use_cache=False when use_kv_cache=True; matches HF greedy at B=1/B=2 on torch 2.5.1 and 2.10 with transformers 4.46.2. Leaving this open because main / v4.1.x still has the bug; it closes when optimus-rm is integrated into main (or with a separate main hotfix). The torch.export failure found on torch 2.10 + transformers 4.46 is tracked in #292.