Skip to content

core: KV cache with the default HFTracer path fails on v4.1.1 (ProxyableDynamicCache) #191

Open

Description

@gspark-etri

Problem

On v4.1.1 (62a7831, main), Optimus_Inference(model, use_kv_cache=True) with the default capture path (HFTracer, dynamo_capture=False) fails while building the IR, for any batch size:

opt_prime/inference.py:263  in __init__
opt_prime/IR.py:1657        in retrieve_IR
opt_prime/IR.py:1706        in _retrieve_IR_trace   (tracer.trace(model, concrete_args=...))
NotImplementedError: argument of type: <class 'transformers.utils.fx.ProxyableDynamicCache'>

So on the release, KV-cache generation only works with dynamo_capture=True (torch.export path), but nothing says so and the default path crashes.

Evidence and reproduction

Tiny Llama from config (2 layers, hidden 64, 4 heads, 2 KV heads, vocab 128, attn_implementation="sdpa"), then:

eng = Optimus_Inference(model, use_kv_cache=True, dynamo_capture=False, dtype=torch.float32)
capture path B=1 B=2
dynamo_capture=False (HFTracer, default) fails as above fails as above
dynamo_capture=True (torch.export) matches HF greedy crashed before #190 (KV cache at batch >= 2), matches after

The result is the same before and after the v4.1.2 hotfix PR #190, so this is a separate pre-existing problem.

Probable cause (not verified): _retrieve_IR_trace leaves use_cache out of input_names, so it is passed as a concrete arg with its default. The model then builds a DynamicCache for past_key_values, HFTracer wraps it as ProxyableDynamicCache, and torch.fx cannot emit it as a graph output. The KV path replaces attention with CachedScaledDotProductAttention anyway (_inject_kv_cache_into_graph), so tracing with use_cache=False may be enough. This needs checking.

Environment

The support matrix stated by v4.1.1 (opt_prime/README.md): PyTorch 2.5.0 / CUDA 12.4 (torch 2.5.0+cu124), transformers==4.46.2, Python 3.10, one A100 80GB.

On optimus-rm with transformers 5.x, the HFTracer path is intentionally unavailable: _retrieve_IR_trace exits with "HFTracer is unavailable ... use --dynamo-capture". So this affects only transformers 4.x setups: main/v4.1.x, and optimus-rm's transformers 4.46 configuration.

Scope

IR._retrieve_IR_trace for HF models with use_kv_cache=True. Alternatively, if the HFTracer + KV-cache combination is not meant to be supported, reject it up front with a clear message and document that KV cache needs --dynamo-capture.

Done when

On torch 2.5.0 + transformers 4.46.2, use_kv_cache=True with dynamo_capture=False either matches HF greedy at B=1 and B=2, or fails at construction with a clear "use dynamo_capture=True" error. A test covers whichever behaviour is chosen.

Owner

@gspark-etri

Activity

  1. gspark-etri commented on Nov 4, 2026

    @gspark-etri
    CollaboratorAuthor

    Fixed on optimus-rm by #291 (b8d6b3b): HFTracer now traces with use_cache=False when use_kv_cache=True; matches HF greedy at B=1/B=2 on torch 2.5.1 and 2.10 with transformers 4.46.2. Leaving this open because main / v4.1.x still has the bug; it closes when optimus-rm is integrated into main (or with a separate main hotfix). The torch.export failure found on torch 2.10 + transformers 4.46 is tracked in #292.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions