[mosafariuk]

💡 Motivation and Context

PostHogTracingProcessor never reports cached input tokens or reasoning tokens for real OpenAI Agents runs, on either model path:

• Generation spans (OpenAIChatCompletionsModel): the Agents SDK builds span_data.usage with the counts nested under input_tokens_details.cached_tokens and output_tokens_details.reasoning_tokens (see agents.usage.model_usage_to_span_usage and openai_chatcompletions.py). The processor reads flat usage["cache_read_input_tokens"] and usage["reasoning_tokens"], which the SDK never sets.
• Response spans (OpenAIResponsesModel, the default): the processor reads only input_tokens and output_tokens from response.usage, and ignores its input_tokens_details and output_tokens_details.

So $ai_cache_read_input_tokens and $ai_reasoning_tokens are always missing. Cached input is then costed at the full input rate, and reasoning usage is invisible.

This PR reads the nested fields on both paths, and keeps the flat keys for custom models that report them that way. The values are mapped the same way as the Python OpenAI wrapper's extract_openai_usage_from_response: cached_tokens → $ai_cache_read_input_tokens and reasoning_tokens → $ai_reasoning_tokens, each only when greater than zero. It's the Python counterpart of the cache-read part of PostHog/posthog-js#5056. Cache writes are left out, because the Python OpenAI wrapper doesn't report them either.

💚 How did you test it?

• End to end with the real Agents SDK (openai-agents 0.18.3): a script runs Runner.run with OpenAIChatCompletionsModel and with OpenAIResponsesModel, against an httpx.MockTransport returning usage with 100 input, 60 cached, 50 output and 40 reasoning tokens, and the PostHog processor registered. On main, both runs capture $ai_cache_read_input_tokens: null and $ai_reasoning_tokens: null. With this change, both capture 60 and 40.
• Regression tests in posthog/test/ai/openai_agents/test_processor.py:
  • test_generation_span_reads_nested_cache_and_reasoning_tokens builds the span usage with the SDK's own model_usage_to_span_usage(Usage(...)).
  • test_response_span_reads_cache_and_reasoning_tokens uses a real openai.types.responses.Response.
  • Both fail on main with KeyError: '$ai_cache_read_input_tokens' and pass with this change. The existing flat-key test (test_generation_span_with_reasoning_tokens) still passes.
• Full suite: 5133 passed, 16 skipped. ruff format --check, ruff check, mypy | mypy-baseline filter and make public_api_check are clean.

📝 Checklist

• I reviewed the submitted code.
• I added tests to verify the changes.
• I updated the docs if needed.
• No breaking change or entry added to the changelog.

If releasing new changes

• Ran sampo add to generate a changeset file

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

• An AI coding agent helped me find this by comparing recent @posthog/ai fixes with the Python SDK and reading the Agents SDK's span-usage serialization. It drafted the fix and tests. I reviewed and ran all of it.
• The existing tests passed because they used a flat usage dict, which isn't the shape the SDK produces. The new tests build usage with the SDK's own types and serializer.

[dustinbyrne]

Nested token extraction is verified on both Agents model paths. One cache-accounting issue remains for OpenAI-compatible Claude requests: the new cache count increases estimated input cost without an inclusive-reporting flag. See the inline reproduction.

AI-assisted review.

[dustinbyrne]

[P1]Declare inclusive accounting when forwarding compatible-provider cache usage

IMPORTANT · REQUIRED · Amplified pre-existing defect
Location: posthog/ai/openai_agents/processor.py:561–565, RIGHT.

A cached Claude request through an OpenAI-compatible endpoint can now make PostHog's estimated input cost increase instead of decrease. The newly emitted cached tokens are already included in the input total, but the backend treats Claude-model usage as exclusive unless $ai_cache_reporting_exclusive explicitly says otherwise.

An ordinary supported caller is:

import os
from agents import Agent, Runner
from agents.models.openai_chatcompletions import OpenAIChatCompletionsModel
from openai import AsyncOpenAI
from posthog.ai.openai_agents import instrument

instrument(posthog_client)
provider = AsyncOpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
)
agent = Agent(name="Assistant", model=OpenAIChatCompletionsModel(
    model="anthropic/claude-sonnet-4.6", openai_client=provider,
))
await Runner.run(agent, "...")

Agents documents compatible-provider configuration. OpenRouter documents inclusive prompt usage, with 10339 prompt tokens including 10318 cached tokens, 60 completion tokens and no cache writes. Agents normalizes these details without changing the total.

The pinned backend calculator infers exclusive accounting from the Anthropic/Claude model name despite $ai_provider: "openai". Explicit false overrides it. Model/provider lookup resolves this model to catalog rates of $3/M ordinary input and $0.30/M cached input.

Using the same documented-shaped synthetic response through real Runner.run, instrument, SDK queue and HTTP delivery on both revisions, then executing that backend calculator on the captured properties:

Snapshot / interpretation	Estimated input cost
Merge base: cache field absent	$0.031017
Head: cache field present, convention absent	$0.0341124
Correct inclusive interpretation	$0.0031584

The correct calculation is (10339 − 10318) × 0.000003 + 10318 × 0.0000003.

Base was already wrong. This patch amplifies that defect by another $0.0030954 per fixture request, approximately 10% above the previous estimate. It affects analytics estimates, not the provider's actual bill. The demonstrated trigger is cached, inclusive Chat Completions usage with a model matched by the backend's Claude heuristic and no explicit convention/cost override; ordinary GPT requests are unaffected.

Consider declaring $ai_cache_reporting_exclusive: false for recognized inclusive nested-usage paths, preserving custom flat-key conventions and caller overrides. The Node Agents processor already declares this convention for Chat Completions and Responses. Add a compatible-Claude regression alongside the GPT case; the corresponding Responses mapping should remain consistent where its provider contract is inclusive.

Executed regression:

from unittest.mock import MagicMock
from agents import set_trace_processors
from agents.tracing import generation_span, trace
from posthog.ai.openai_agents import instrument


def test_cached_compatible_claude_usage_is_declared_inclusive():
    client = MagicMock(privacy_mode=False, enable_full_ai_capture=False)
    set_trace_processors([])
    processor = instrument(client, distinct_id="review-user")
    try:
        with trace("compatible-model"):
            with generation_span(
                model="anthropic/claude-sonnet-4.6",
                usage={
                    "input_tokens": 10339,
                    "output_tokens": 60,
                    "input_tokens_details": {"cached_tokens": 10318},
                },
            ):
                pass
        properties = next(
            call.kwargs["properties"] for call in client.capture.call_args_list
            if call.kwargs["event"] == "$ai_generation"
        )
        assert properties["$ai_cache_read_input_tokens"] == 10318
        assert properties.get("$ai_cache_reporting_exclusive") is False
    finally:
        set_trace_processors([])
        processor.shutdown()

Executed under identical Python/provider dependencies with sandboxed PYTHONPATH=<snapshot> python -m pytest -p pytest_timeout -c /dev/null --rootdir=<runtime> --confcutdir=<runtime> test_cache_reporting.py --timeout=30 -q -p no:cacheprovider:

• Head: exit 1 at the convention assertion: None is False.
• Base: exit 1 earlier because the cache field is absent. This alone is not a passing-base/failing-head proof; the captured-payload/calculator comparison establishes amplification.
• Calculator: node cost-check.cjs exits 0 confirming amplification and the inclusive control; node cost-check.cjs --assert-correct exits 1 with head cost 0.0341124, expected 0.0031584. It executes pinned calculator source with TypeScript erasure/module-linkage substitution and real js-big-decimal; only logger/metrics are stubbed, not arithmetic. No full ingestion stack was run.

A verified supported mitigation is instrument(posthog_client, properties={"$ai_cache_reporting_exclusive": False}). It is processor-wide, not automatic per-model handling, and can be inappropriate for mixed custom-model conventions. No fix was applied or tested.

[mosafariuk]

Good catch, thanks. Fixed in 4de76ae.

• When the cache count comes from the nested OpenAI-shaped details, the processor now also sends $ai_cache_reporting_exclusive: False, on both generation spans (input_tokens_details / prompt_tokens_details) and response spans. This matches the Node processor.
• Flat cache_read_input_tokens from custom models gets no flag, so their own convention is kept.
• A caller-supplied $ai_cache_reporting_exclusive (via instrument(..., properties=...)) still wins, because processor-wide properties are merged last.

Tests:

• Added your test_cached_compatible_claude_usage_is_declared_inclusive as written. It fails on e424062 (None is False) and passes now.
• Added test_flat_cache_keys_keep_their_own_convention and test_caller_cache_reporting_convention_takes_precedence.
• The two GPT tests now also assert the flag.

Running real Runner.run with OpenAIChatCompletionsModel("anthropic/claude-sonnet-4.6") and your OpenRouter-shaped usage now captures $ai_input_tokens: 10339, $ai_cache_read_input_tokens: 10318 and $ai_cache_reporting_exclusive: false. That's the inclusive interpretation behind your $0.0031584.

Full suite: 5136 passed, 16 skipped. Ruff, mypy and the public API check are clean.

[dustinbyrne]

and thank you for the pull request @mosafariuk! 🙌