Skip to content

fix: keep file_search reasoning before its call on continuations - #542

Closed
justrightnow wants to merge 1 commit into
vllm-project:mainfrom
justrightnow:fix/482-file-search-reasoning-order
Closed

justrightnow wants to merge 1 commit into
vllm-project:mainfrom
justrightnow:fix/482-file-search-reasoning-order

Conversation

@justrightnow

Copy link
Copy Markdown
Contributor

Summary

Continuations reorder a turn's reasoning around gateway-executed calls (#482). Hosted tool/web/MCP searches were fixed in #505 with regression coverage in #519. file_search — added later in #486 — still had the remaining gap, as flagged in the issue's follow-up.

When a gateway-run file search emits reasoning before the function call, the live second inference round already sees the model-facing items in the right order:

user, reasoning, function_call(file_search), function_call_output

But continuing with previous_response_id persisted that model-facing call/output pair at the front of the turn's history (new_input_items), ahead of the turn's public output (which holds the reasoning). The next turn therefore saw:

user, function_call, function_call_output, reasoning, message, user

This happens because the public file_search_call omits retrieved results by default (include_results defaults to false), and OutputItem::to_input_item returns None for it, so the model-facing pair (whose output carries retrieved_passages) had to be carried separately — it was just carried in the wrong place.

This change:

  • Adds RequestContext::file_search_history, recording each file search's model-facing call/output pair in public-output order.
  • Splices each pair at the public file_search_call item's position when assembling turn history (turn_items), instead of prepending it. Pairs are still consumed in order under in-turn compaction so multiple file searches stay aligned.
  • Leaves the in-turn (enriched_request.input) path and the public response shape untouched: the public response still exposes reasoning, file_search_call, message, with file_search_call.results omitted by default.

After the fix a continuation shows the model exactly the same prefix the live round did.

Test plan

Pure local mocks (mock /v1/responses + deterministic /v1/embeddings, in-memory shared-cache SQLite, temp file store) — no cassette and no OpenAI key required.

  • New regression test agentic-server-core/tests/file_search_history_order.rs::file_search_continuation_keeps_reasoning_before_the_call. It:
    • runs a [reasoning, file_search_call] -> message stored turn, then a previous_response_id turn;
    • asserts the live round-2 model input is user, reasoning, function_call(file_search), function_call_output with matching call_id;
    • asserts the continuation's prefix is byte-identical to that live prefix, followed by the assistant answer and the new user message;
    • asserts the public response keeps reasoning, file_search_call, message and that the public file_search_call omits results.
    • Before the fix it fails at "continuation shares the live round's prefix" (call/output precede reasoning); after, it passes.
  • cargo test -p agentic-server-core — all targets pass (including file_search_service, hosted_tool_search_*, web_search_tool_test, mcp_tool_test, stateful_conversation_integration, stateful_responses_integration, session_continuation_test, storage_*, compaction, and the replay/cassette suites).
  • cargo test -p agentic-server — passes, including file_search_http_test, conversations_test, responses_test, and compaction.
  • cargo clippy --all-targets -- -D warnings and cargo fmt -- --check — clean.
  • Environment-only failures unrelated to this diff on this machine: two messages_test "unreachable upstream" cases expect 502/connection-refused but the local transparent proxy (ALL_PROXY=socks5h://127.0.0.1:1024) turns the bound-but-unlistened loopback port into a 504 timeout; and one responses_websocket_test steering timing case passes 3/3 in isolation.

Thanks @Than0316 for the file-search follow-up and the failing-test analysis in #482, and to the #505/#519 authors whose fix this mirrors.

When a gateway-run file search produced reasoning before the function
call, continuing with previous_response_id persisted the model-facing
function call/output pair ahead of the turn's output, so the next turn
showed the model call, output, reasoning instead of reasoning, call,
output. Hosted tool/web/MCP searches were fixed for this in vllm-project#505 with
regression coverage in vllm-project#519; file search was the remaining gap tracked
in vllm-project#482 because its public file_search_call omits results by default.

Record each file search's model-facing call/output pair in public-output
order and splice it at the public file_search_call's position when
assembling turn history, so a continuation sees the same prefix the live
round did, without changing the public response shape.

Fixes vllm-project#482.

Signed-off-by: Jack Wei <weike@chinamobile.com>
@justrightnow

Copy link
Copy Markdown
Contributor Author

Verification (before vs after)

Local, deterministic mocks only — mock /v1/responses + /v1/embeddings, in-memory SQLite, temp file store; no OpenAI key/cassette.

Turn 1 upstream output: [reasoning, file_search_call("coral")] → terminal message; turn 2 continues with previous_response_id.

Round 2 model input — live (already correct, both before & after):

user, reasoning, function_call(file_search), function_call_output, message(assistant)

Continuation turn model input — before the fix:

user, function_call(file_search), function_call_output, reasoning, message, user

(call + output jumped ahead of the reasoning — #482)

Continuation turn model input — after the fix:

user, reasoning, function_call(file_search), function_call_output, message(assistant), user

The continuation prefix is now byte-identical to the live round (same function call_id, same generated output id, same retrieved passages), and the public response still returns reasoning, file_search_call, message with file_search_call.results omitted by default.

cargo test -p agentic-server-core --test file_search_history_order
# before: FAILED  ("continuation shares the live round's prefix")
# after:  ok. 1 passed
cargo test -p agentic-server-core      # all pass
cargo clippy --all-targets -- -D warnings && cargo fmt -- --check   # clean

@maralbahari

Copy link
Copy Markdown
Collaborator

@justrightnow Thanks however we cant accept this. #536 is already working on acceptable solution with proper cassettes recording for file history and continuation.

@justrightnow

Copy link
Copy Markdown
Contributor Author

OK

@justrightnow
justrightnow deleted the fix/482-file-search-reasoning-order branch November 16, 2026 08:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants