Skip to content

test: keep reasoning before gateway calls on previous_response_id continuations - #519

Merged
franciscojavierarceo merged 1 commit into
vllm-project:mainfrom
Than0316:test/482-continuation-order
Nov 13, 2026
Merged

franciscojavierarceo merged 1 commit into
vllm-project:mainfrom
Than0316:test/482-continuation-order

Conversation

@Than0316

Copy link
Copy Markdown
Contributor

Summary

Adds the regression tests #482 asked for: when the model reasons before a gateway-executed call, a previous_response_id continuation must send that reasoning before the call and its output, as the live round does.

#505 already fixed the order for tool search, web search and MCP. These tests fail on #505's parent (248f382) and pass on main, so they guard that behaviour.

Part of #482. file_search (#486, merged after #505) still reorders; I'll raise that on #482 separately, so it's not in this PR.

Gateway tool Test 248f382 (before #505) main
tool search hosted_tool_search_projection_test::live_continuation_and_replay_show_the_model_the_same_search (now with reasoning before the search) fails passes
web search web_search_tool_test::continuation_keeps_reasoning_before_the_web_search_it_led_to (new) fails passes
MCP mcp_tool_test::continuation_keeps_reasoning_before_the_mcp_call_it_led_to (new) fails passes

The web search and MCP tests check the order of item types plus the reasoning item. Their continuations rebuild the call from the stored public item, so the call's id, arguments and output differ from the live round.

Test Plan

  • cargo test -p agentic-server-core --test hosted_tool_search_projection_test --test web_search_tool_test --test mcp_tool_test on main: 8, 36 and 18 passed.
  • The same tests on 248f382: the three tests above fail at the continuation assertion (user, function_call, function_call_output, reasoning); all others pass.
  • cargo fmt --all -- --check; cargo clippy -p agentic-server-core --all-targets -- -D warnings.
  • Mock upstream and SQLite, as in the neighbouring tests; no live model.

AI assistance (Claude) was used to investigate and write these tests; I checked every line of code and ran them locally before opening this PR.

…tinuations

Signed-off-by: Than0316 <tassel0316@gmail.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block the test-only regression coverage for reasoning order in gateway-tool continuations.

No open issues 🎉

What changed in this PR

Adds regression coverage for #482, guarding the reasoning order preserved by #505 when previous_response_id continuations rebuild gateway-tool history. This PR changes tests only; file-search ordering remains out of scope.

Changes:

  • Adds web-search and MCP tests comparing live-round and continuation item order and reasoning content.
  • Extends hosted tool-search coverage to verify reasoning stays before the call across live execution, continuation, and store: false replay.
File Description
crates/​agentic-server-core/​tests/​web_search_tool_test.rs Adds reasoning-order coverage for web-search continuations.
crates/​agentic-server-core/​tests/​mcp_tool_test.rs Adds reasoning-order coverage for MCP continuations.
crates/​agentic-server-core/​tests/​hosted_tool_search_projection_test.rs Includes reasoning in live, continuation, and replay prefix comparisons.

Review effort: Balanced

@maralbahari maralbahari left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the tests!

file_search still reorders on main, as you noted on #482: record_gateway_results in executor/engine/agent_turn.rs stores its model-facing call and output in new_input_items, which persistence writes before the turn's outputs, so on a previous_response_id continuation they land ahead of the reasoning. Could you open a separate PR for that? It needs OpenAI and gateway cassette recordings of the file search continuation, so please take it only if you have the resources to record them; otherwise Francisco or I will pick it up.

PS: history no longer stores model-facing items. We keep only client-facing items, and the model-facing shapes are built at the one lowering point when the request is sent; model-facing shapes in stored history have been removed from main and are not encouraged. The file search path above is a leftover of that older storage, so the fix should store the client-facing file_search_call and lower it like the other gateway tools, not keep model-facing calls in history.

// live round's; the order must not.
let bodies = fixture.request_bodies().await;
assert_eq!(bodies.len(), 3);
let kinds = |body: &Value| -> Vec<Value> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the same kinds closure is written again in web_search_tool_test.rs. A small helper in tests/support that returns the input item types of a request body would let both tests share it.

@franciscojavierarceo
franciscojavierarceo merged commit 9ee42f1 into vllm-project:main Nov 13, 2026
1 check passed
@Than0316

Copy link
Copy Markdown
Contributor Author

@maralbahari thanks for the review and for explaining the storage direction. I can't record the gateway cassettes for the file_search continuation (I don't have a GPU to run a reasoning model), so I'll leave the file_search fix to you or Francisco. For reference, #542 was opened today for this case.

I'll open a small follow-up for your nit (a shared input_item_types helper in tests/support).

Is there anything else you'd like help with that can be tested against the mock upstream?

maralbahari pushed a commit that referenced this pull request Nov 16, 2026
…sts (#545)

## Summary

Follow-up to @maralbahari's nit on #519: the MCP and web search
continuation tests each defined the same closure to list the `type` of
every input item in a request body sent upstream. This moves it to
`tests/support` as `input_item_types` and uses it in both tests.

## Test Plan

- `cargo test -p agentic-server-core --test mcp_tool_test --test
web_search_tool_test --test hosted_tool_search_projection_test` on main:
18, 36 and 8 passed.
- `cargo fmt -- --check`; `cargo clippy --all-targets -- -D warnings`
(whole workspace, since `tests/support` is compiled into every core test
binary).
- `pre-commit run` on the changed files.

Signed-off-by: Than0316 <tassel0316@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants