Skip to content

feat(report): keep each LLM request's response id in run.json - #1795

Open
ian-at-strix wants to merge 1 commit into
mainfrom
replay/llm-response-id
Open

ian-at-strix wants to merge 1 commit into
mainfrom
replay/llm-response-id

Conversation

@ian-at-strix

Copy link
Copy Markdown
Contributor

Adds response_id (the OpenRouter generation id) to each llm_requests entry in run.json, so a replay's calls can be looked up on OpenRouter (e.g. to see what the model actually returned).

@arcanist

arcanist Bot commented Nov 19, 2026

Copy link
Copy Markdown

Ready to merge Re-run Arcanist review

PR summary

Recorded LLM requests in run.json did not retain the provider response ID. Each request entry now includes response_id when request timing recording is enabled, so successful calls keep that identifier for later lookup.

🟢 Low blast radius.

Flow diagram
%%{init: {"theme": "neutral"}}%%
flowchart LR
  n0["Request log"]
  n1["Send request details to report"]
  n0 --> n1
  n2["Keep the response id"]
  n1 --> n2
  pad["#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;"]
  n2 ~~~ pad
  style pad fill:none,stroke:none
Loading
✅ Ruled out (2)
  • Could adding the response ID break run record writing or existing readers?
    The value is a string or null, the writer serializes it directly, and existing readers either accept additional keys or were updated with the new field, so record writing and reading remain intact.
  • Could failed requests lose their terminal failure signal?
    The change only copies an optional identifier into each request entry, and it leaves retries, error details, and terminal run status untouched, so failure detection is unchanged.

@greptile-apps

greptile-apps Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Low impact] This PR appears safe to merge.

Summary

Adds each LLM request's response_id to run.json so saved calls can be matched to provider records.

  • Run records keep each LLM request’s response ID.

No actionable issues found.

Reviews (2) · Last reviewed commit: "feat(report): keep each LLM request's re..." · Reviewed by Greptile

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant