Repository navigation
test: keep reasoning before gateway calls on previous_response_id continuations - #519
Conversation
…tinuations Signed-off-by: Than0316 <tassel0316@gmail.com>
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues block the test-only regression coverage for reasoning order in gateway-tool continuations.
No open issues 🎉
What changed in this PR
Adds regression coverage for #482, guarding the reasoning order preserved by #505 when previous_response_id continuations rebuild gateway-tool history. This PR changes tests only; file-search ordering remains out of scope.
Changes:
- Adds web-search and MCP tests comparing live-round and continuation item order and reasoning content.
- Extends hosted tool-search coverage to verify reasoning stays before the call across live execution, continuation, and
store: falsereplay.
| File | Description |
|---|---|
crates/agentic-server-core/tests/web_search_tool_test.rs |
Adds reasoning-order coverage for web-search continuations. |
crates/agentic-server-core/tests/mcp_tool_test.rs |
Adds reasoning-order coverage for MCP continuations. |
crates/agentic-server-core/tests/hosted_tool_search_projection_test.rs |
Includes reasoning in live, continuation, and replay prefix comparisons. |
Review effort: Balanced
There was a problem hiding this comment.
Thanks for the tests!
file_search still reorders on main, as you noted on #482: record_gateway_results in executor/engine/agent_turn.rs stores its model-facing call and output in new_input_items, which persistence writes before the turn's outputs, so on a previous_response_id continuation they land ahead of the reasoning. Could you open a separate PR for that? It needs OpenAI and gateway cassette recordings of the file search continuation, so please take it only if you have the resources to record them; otherwise Francisco or I will pick it up.
PS: history no longer stores model-facing items. We keep only client-facing items, and the model-facing shapes are built at the one lowering point when the request is sent; model-facing shapes in stored history have been removed from main and are not encouraged. The file search path above is a leftover of that older storage, so the fix should store the client-facing file_search_call and lower it like the other gateway tools, not keep model-facing calls in history.
| // live round's; the order must not. | ||
| let bodies = fixture.request_bodies().await; | ||
| assert_eq!(bodies.len(), 3); | ||
| let kinds = |body: &Value| -> Vec<Value> { |
There was a problem hiding this comment.
nit: the same kinds closure is written again in web_search_tool_test.rs. A small helper in tests/support that returns the input item types of a request body would let both tests share it.
|
@maralbahari thanks for the review and for explaining the storage direction. I can't record the gateway cassettes for the file_search continuation (I don't have a GPU to run a reasoning model), so I'll leave the file_search fix to you or Francisco. For reference, #542 was opened today for this case. I'll open a small follow-up for your nit (a shared Is there anything else you'd like help with that can be tested against the mock upstream? |
…sts (#545) ## Summary Follow-up to @maralbahari's nit on #519: the MCP and web search continuation tests each defined the same closure to list the `type` of every input item in a request body sent upstream. This moves it to `tests/support` as `input_item_types` and uses it in both tests. ## Test Plan - `cargo test -p agentic-server-core --test mcp_tool_test --test web_search_tool_test --test hosted_tool_search_projection_test` on main: 18, 36 and 8 passed. - `cargo fmt -- --check`; `cargo clippy --all-targets -- -D warnings` (whole workspace, since `tests/support` is compiled into every core test binary). - `pre-commit run` on the changed files. Signed-off-by: Than0316 <tassel0316@gmail.com>
Summary
Adds the regression tests #482 asked for: when the model reasons before a gateway-executed call, a
previous_response_idcontinuation must send that reasoning before the call and its output, as the live round does.#505 already fixed the order for tool search, web search and MCP. These tests fail on #505's parent (
248f382) and pass on main, so they guard that behaviour.Part of #482. file_search (#486, merged after #505) still reorders; I'll raise that on #482 separately, so it's not in this PR.
248f382(before #505)hosted_tool_search_projection_test::live_continuation_and_replay_show_the_model_the_same_search(now with reasoning before the search)web_search_tool_test::continuation_keeps_reasoning_before_the_web_search_it_led_to(new)mcp_tool_test::continuation_keeps_reasoning_before_the_mcp_call_it_led_to(new)The web search and MCP tests check the order of item types plus the reasoning item. Their continuations rebuild the call from the stored public item, so the call's id, arguments and output differ from the live round.
Test Plan
cargo test -p agentic-server-core --test hosted_tool_search_projection_test --test web_search_tool_test --test mcp_tool_teston main: 8, 36 and 18 passed.248f382: the three tests above fail at the continuation assertion (user, function_call, function_call_output, reasoning); all others pass.cargo fmt --all -- --check;cargo clippy -p agentic-server-core --all-targets -- -D warnings.AI assistance (Claude) was used to investigate and write these tests; I checked every line of code and ran them locally before opening this PR.