Repository navigation
fix(messages): pass through upstream-run native server tools - #573
Open
justrightnow wants to merge 1 commit into
Open
justrightnow wants to merge 1 commit into
justrightnow wants to merge 1 commit into
Conversation
Issue vllm-project#409 has two halves. The gateway-executed half (returning server_tool_use and result blocks, sealing, usage counts, and replaying them as tool_use/tool_result) was already delivered by PR vllm-project#494. This change completes the other half: an upstream that runs native server tools itself can be configured to own them, so declarations and blocks are forwarded unchanged instead of being rewritten and executed by the gateway. - Add UpstreamServerTools policy (MESSAGES_UPSTREAM_SERVER_TOOLS / messages_gateway.upstream_server_tools), default empty for backward compatibility. - Skip gateway registration and native-tool binding for upstream-owned tool types across /v1/messages, streaming, and count_tokens. - Restore the input field on streamed server_tool_use blocks so replayed blocks stay valid. - Cover nonstreaming and streaming pass-through at the HTTP level, asserting the gateway search provider is not called, plus mixed-owner and replay tests. Signed-off-by: Jack Wei <weike@chinamobile.com>
justrightnow
requested review from
bbrowning,
franciscojavierarceo,
haoshan98,
jiahuei,
leseb,
maralbahari,
noobHappylife,
qandrew and
tjtanaa
as code owners
November 19, 2026 11:49
4 tasks
Collaborator
|
this adds a new |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the upstream-owned half of #409 (
server_tool_useblocks are missing / not replayed for native server tools on/v1/messages).The gateway-executed half of that issue—returning
server_tool_useand result blocks, sealing,usage.server_tool_use, and replaying the blocks astool_use/tool_result—was already delivered by the merged #494. This change covers the complementary case: an upstream that runs native server tools itself (e.g. Anthropic's API) currently cannot, because the gateway always rewritesweb_search_20250305and runs it with its own provider, then hides the call from the client.UpstreamServerToolspolicy, configured withMESSAGES_UPSTREAM_SERVER_TOOLSormessages_gateway.upstream_server_tools(comma-separated native tooltypes, e.g.web_search_20250305). Default is empty, preserving current behavior./v1/messages, streaming, and/v1/messages/count_tokens.inputfield on streamedserver_tool_useblocks so blocks replayed by clients stay valid for model-only upstreams.Test Plan
Verified in an isolated worktree based on the current
origin/main(branch contains only this change):cargo fmt -- --check— passed.cargo clippy --workspace --all-targets -- -D warnings— passed, 0 warnings.cargo check --workspace --all-targets— passed.cargo test --workspace --no-fail-fast— 128 test binaries passed; focused suites:messages_server_toolslib 20 passed,dynamo_messages_test3 passed,messages_stream_test29 passed, servermessages_server_tools_test5 passed.origin/main:messages_test::{messages_returns_anthropic_error_for_unreachable_upstream, messages_gateway_loop_returns_anthropic_error_for_unreachable_upstream}: gateway returns HTTP 504 where the test expects 502 on this machine.agentic_process::tests::catalog_selection_retries_{a_warming_gateway, an_empty_catalog}: timing flake under heavy parallel load; both pass in isolation in this worktree and on pristineorigin/main.No cassettes or provider evidence were hand-authored; all test evidence uses the existing in-tree recording/recorder workflow.