Skip to content

fix(messages): pass through upstream-run native server tools - #573

Open
justrightnow wants to merge 1 commit into
vllm-project:mainfrom
justrightnow:deliver/409-upstream-native-server-tools
Open

justrightnow wants to merge 1 commit into
vllm-project:mainfrom
justrightnow:deliver/409-upstream-native-server-tools

Conversation

@justrightnow

Copy link
Copy Markdown
Contributor

Summary

Fixes the upstream-owned half of #409 (server_tool_use blocks are missing / not replayed for native server tools on /v1/messages).

The gateway-executed half of that issue—returning server_tool_use and result blocks, sealing, usage.server_tool_use, and replaying the blocks as tool_use/tool_result—was already delivered by the merged #494. This change covers the complementary case: an upstream that runs native server tools itself (e.g. Anthropic's API) currently cannot, because the gateway always rewrites web_search_20250305 and runs it with its own provider, then hides the call from the client.

  • Add an UpstreamServerTools policy, configured with MESSAGES_UPSTREAM_SERVER_TOOLS or messages_gateway.upstream_server_tools (comma-separated native tool types, e.g. web_search_20250305). Default is empty, preserving current behavior.
  • Skip gateway tool registration and native-tool binding for upstream-owned tool types, so their declarations and blocks are forwarded unchanged through /v1/messages, streaming, and /v1/messages/count_tokens.
  • Restore the input field on streamed server_tool_use blocks so blocks replayed by clients stay valid for model-only upstreams.
  • Add HTTP-level tests for nonstreaming and streaming pass-through (asserting the gateway search provider is never called), mixed-owner behavior, and replay.

Test Plan

Verified in an isolated worktree based on the current origin/main (branch contains only this change):

  • cargo fmt -- --check — passed.
  • cargo clippy --workspace --all-targets -- -D warnings — passed, 0 warnings.
  • cargo check --workspace --all-targets — passed.
  • cargo test --workspace --no-fail-fast — 128 test binaries passed; focused suites: messages_server_tools lib 20 passed, dynamo_messages_test 3 passed, messages_stream_test 29 passed, server messages_server_tools_test 5 passed.
  • Two test targets reported failures that are pre-existing and environmental, reproduced on pristine origin/main:
    • messages_test::{messages_returns_anthropic_error_for_unreachable_upstream, messages_gateway_loop_returns_anthropic_error_for_unreachable_upstream}: gateway returns HTTP 504 where the test expects 502 on this machine.
    • agentic_process::tests::catalog_selection_retries_{a_warming_gateway, an_empty_catalog}: timing flake under heavy parallel load; both pass in isolation in this worktree and on pristine origin/main.

No cassettes or provider evidence were hand-authored; all test evidence uses the existing in-tree recording/recorder workflow.

Issue vllm-project#409 has two halves. The gateway-executed half (returning server_tool_use and result blocks, sealing, usage counts, and replaying them as tool_use/tool_result) was already delivered by PR vllm-project#494. This change completes the other half: an upstream that runs native server tools itself can be configured to own them, so declarations and blocks are forwarded unchanged instead of being rewritten and executed by the gateway.

- Add UpstreamServerTools policy (MESSAGES_UPSTREAM_SERVER_TOOLS / messages_gateway.upstream_server_tools), default empty for backward compatibility.
- Skip gateway registration and native-tool binding for upstream-owned tool types across /v1/messages, streaming, and count_tokens.
- Restore the input field on streamed server_tool_use blocks so replayed blocks stay valid.
- Cover nonstreaming and streaming pass-through at the HTTP level, asserting the gateway search provider is not called, plus mixed-owner and replay tests.

Signed-off-by: Jack Wei <weike@chinamobile.com>
@franciscojavierarceo

Copy link
Copy Markdown
Collaborator

this adds a new /v1/messages upstream-hosted tool wire path, including streamed server_tool_use blocks and mixed gateway rounds, but the coverage uses a scripted mock upstream. we should record and replay a real upstream case for the non-streaming and streaming pass-through, including the input_json_delta to input reconstruction and a mixed round, so we can verify the provider contract before merging.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants