Skip to content

feat: resolve Responses file_id references from the Files API - #536

Merged
franciscojavierarceo merged 21 commits into
vllm-project:mainfrom
LeaderOnePro:feat/responses-file-id-hydration
Nov 18, 2026
Merged

franciscojavierarceo merged 21 commits into
vllm-project:mainfrom
LeaderOnePro:feat/responses-file-id-hydration

Conversation

@LeaderOnePro

@LeaderOnePro LeaderOnePro commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Responses clients can reference an uploaded file by file_id instead of sending inline content:
input_file and input_image message parts, plus structured function/custom tool-call outputs.
This PR resolves those references from the gateway Files API and inlines the content in the
request sent to the model, while stored history keeps the client's reference (#356).

Design, following RFC #500:

  • Request validation accepts file_id and rejects a part that also carries its inline field
    (file_data, filename, file_url, or image_url) with OpenAI's
    mutually_exclusive_parameters envelope.
  • executor::upstream_request is now async. Immediately before lowering, it calls
    ExecutionContext::file_handler().resolve on ctx.enriched_request.input and passes the
    resulting lookup to RequestPayload::to_upstream_request, so the lookup and the model-facing
    body are built from the same input and cannot get out of step. The lookup is used only by
    normalized_model_input and is never stored, checkpointed, or snapshotted.
  • execute.rs::prepare performs the same resolve and normalized_model_input pass before tool
    preparation, so a missing or invalid reference returns its 404/400/413 before any tool work or
    response event.
  • Model-facing lowering turns a text reference into an input_text part (File: <filename>
    prefix when a filename is known) and an image reference into an inline base64 image_url data
    URL. Each distinct file is read once per request even when referenced more than once. Stored
    Responses/conversation history, compaction windows, and multi-agent tree snapshots keep
    file_id, so each continuation resolves it again against the current file store.
  • Structured tool-output parts that carry file_id follow the same rules. Other tool-output
    parts keep their existing pass-through content contract and are not inspected.
  • Missing files return OpenAI's 404 invalid_request_error, code: null, param: null,
    Files [<ids joined ", ">] were not found. Wrong file types return OpenAI's 400 messages with
    param: input. Without a configured file store, the request fails with 400 rather than
    inventing content. Resolved content counts toward the existing
    max_request_body_size_bytes limit; exceeding it returns 413 body_too_large.
  • Split execution cannot reach the gateway file store, so in_process_feature() reports
    file_id references and agentic-llm-d rejects such requests instead of dropping content.
  • Files API: accept the OpenAI upload purposes fine-tune, assistants, batch, user_data,
    vision, and evals. A vision upload must use an image media type, and
    GET /v1/files/{file_id}/content serves vision uploads only; another purpose returns 400
    Not allowed to download files of purpose: <purpose>. The gateway rows reproduce the OpenAI
    reference rows for upload rules, download policy, unknown-ID envelopes, and the delete object.
  • RequestPayload::to_upstream_request now takes the resolved-file lookup (&ResolvedFiles);
    embedders calling it directly resolve references first or pass an empty map when the payload
    has none. RequestContext, rehydration, multi-agent contexts, and compaction need no change.
  • Docs, CHANGELOG.md, and the cassette recorder workflow are updated with the new behavior and
    the new Files API scenarios.

Behavior contract

Input Result
Message input_file file_id → stored UTF-8 text file Model-facing part becomes input_text, prefixed File: <filename>\n when a filename is known; stored item unchanged
Message input_image file_id → stored .jpeg/.jpg/.png/.gif/.webp file Model-facing part becomes an inline data:image/...;base64,... URL; detail and other fields preserved; stored item unchanged
Tool-output part with file_id Resolved like the corresponding message part
Tool-output part without file_id Passed through unchanged, not validated against message rules
File absent (one or many) 404 invalid_request_error, code: null, param: null, Files [<id1>, <id2>] were not found
input_file: file_id + file_data / filename / file_url 400 mutually_exclusive_parameters, param: input[i].content[p], Mutually exclusive parameters: '<param>'. Ensure you are only providing one of: 'file_id' or '<field>'.
input_image: file_id + image_url Same envelope, 'image_url'
Same conflict in a tool output Same envelope, param: input[i].output[p]
input_image file_id resolves to a non-image filename 400, code: null, param: input, Invalid input: Expected image type to be a supported format: .jpeg, .jpg, .png, .gif, .webp but got .<ext>.
input_file file_id resolves to a non-text filename 400, code: null, param: input, Invalid input: Expected context stuffing file type to be a supported format: <OpenAI list> but got .<ext>.
No gateway file store configured 400 invalid_request_error, message <location>: input_file with file_id requires the gateway file store; configure file storage or send the content inline
Resolved content exceeds max_request_body_size_bytes (default 10 MiB) 413 body_too_large, resolved file_id references exceed the <N>-byte request body limit
Split execution with a reference 400, file_id references is not supported for split execution

resolve collects references in request order with one shared collector, reads each distinct ID
once, and reports every missing ID in a single 404. prepare and upstream_request both resolve
enriched_request.input; the initial pass fails before tool preparation, and each upstream body
is built with a lookup of exactly the input it lowers.

Intentional limits and divergences

  • input_file accepts UTF-8 text only; file_url is never fetched. This is existing behavior
    and documented.
  • Image media type comes from the stored filename extension; the gateway does not sniff image
    bytes. FileObject exposes no client-visible content type, matching the OpenAI behavior.
  • Deleting a file makes later continuations fail with 404, including image continuations that
    OpenAI's recording answers with 200
    because OpenAI stored the turned-in image media. Under
    RFC [RFC]: one typed path for rehydrating and persisting history #500 the gateway stores only client-facing items and re-resolves each turn, so a deleted
    file is gone. This divergence is pinned by the image cassette and is intentional.
  • Resolved image size is the exact data-URL length; resolved text size is the exact
    JSON-escaped length of the text value, including the File: <filename> prefix. This counts
    the bytes the model-facing body actually carries rather than the raw stored bytes.
  • The no-store 400 is gateway-specific; OpenAI always has a file store, so no OpenAI envelope
    exists for it.
  • Purpose-filtered GET /v1/files?purpose= listing and Files list-parameter validation messages
    stay gateway-specific and are excluded from the row comparison; the gateway does not implement
    the purpose list filter.
  • Syntactically invalid file IDs are rejected by the Files API validator first; only valid but
    absent IDs produce Files [...] were not found.

Test Plan

cargo fmt -- --check                                        pass
cargo clippy --all-targets -- -D warnings                   exit 0
cargo test --workspace --no-fail-fast                       exit 0, 131 test targets, 3196 passed, 0 failures
uvx --python 3.12 pre-commit run --all-files                all hooks pass

The full suite was rerun after merging upstream main at da31b7c (typed MCP
require_approval policy).

End-to-end replay through the HTTP gateway
(crates/agentic-server/tests/file_id_reference_replay_test.rs, 5 tests):

  • text upload → Responses → continuation → delete → 404;
  • image upload → Responses → delete → 404 → re-upload → wrong-type 400;
  • Files API endpoint rows replay without inference;
  • the recorded gateway Files rows reproduce the OpenAI reference rows for all 7 scenario pairs
    (same lifecycle rows, statuses, upload rules, download policy, unknown-ID envelope, delete
    object);
  • Files API upload and lookup errors replay without inference, and reference conflicts keep the
    recorded status and part position.

The 7 gateway cassettes are paired with the OpenAI references in the parity test:

  • files-api-text-gateway-Qwen-Qwen2.5-3B-Instruct.yaml
  • files-api-errors-gateway-Qwen-Qwen2.5-3B-Instruct.yaml
  • files-api-endpoints-gateway-Qwen-Qwen2.5-3B-Instruct.yaml
  • files-api-input-gateway-Qwen-Qwen2.5-3B-Instruct.yaml
  • files-api-search-gateway-Qwen-Qwen2.5-3B-Instruct.yaml
  • files-api-compact-gateway-Qwen-Qwen2.5-3B-Instruct.yaml
  • files-api-image-gateway-Qwen-Qwen2.5-VL-3B-Instruct.yaml

Recorded through a real gateway + vLLM with
crates/agentic-server-core/tests/cassettes/record_text_file_input_cassettes.sh.

Unit and integration coverage also includes: dedupe and aggregated missing IDs, no store,
wrong image/text types, body limit, compaction treating a file_id image as meaningful context,
restored multi-agent histories plus new input, in_process_feature, and the existing
pass-through tool-output tests
(control_custom_tool_output_file_is_preserved,
control_structured_tool_output_file_is_preserved, all three tool-image continuation tests).

## Summary

Message `input_image` and `input_file` parts may reference an uploaded
file by `file_id`. Rehydration now reads the file from the gateway file
store and replaces the reference with a base64 data URL in the
model-facing request, so stateless backends receive the media inline.
The client's reference stays in `original_request` and canonical
history, so each continuation resolves it again.

- `rehydrate/files.rs` hydrates the enriched input after history
  assembly and before final validation. An `input_image` reference must
  resolve to an `image/*` file; an `input_file` reference is subject to
  the existing UTF-8 text media-type and filename rules.
- Early validation gains a pre-hydration mode that accepts a
  reference-only `input_file` while still rejecting every other
  unsupported form. Strict validation runs again after hydration.
- `FileSearchService::read_file` returns the stored bytes, filename, and
  content type that `FileObject` intentionally omits, and uploads now
  accept purpose `vision`.
- A missing store, deleted file, inline/reference conflict, non-image
  image reference, or invalid stored content type fails explicitly with
  the item and part position. Resolved data URLs are capped at 64 MiB
  per request (413 `body_too_large`).

Refs: vllm-project#356

## Test Plan

- Unit tests in `rehydrate/files.rs` cover inlining both part kinds with
  field preservation, inline conflicts, a non-image `input_image`, an
  invalid stored content type, missing files, a missing file store, and
  the per-request budget.
- `rehydrate/tests.rs` covers new input, a missing store, and a
  continuation that resolves a reference from stored history, asserting
  the canonical copy keeps `file_id` and the upstream request carries
  inline media.
- Integration tests cover `read_file` metadata and the `vision` purpose.
- Updated the two existing rejection tests for the new message.
- `cargo test --workspace --no-fail-fast`, `cargo clippy --workspace
  --all-targets -- -D warnings`, `cargo fmt --all -- --check`, and all
  pre-commit hooks pass.

Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
## Summary

- Reject `input_image.file_id` in strict validation when a request path does
  not rehydrate. Stored multi-agent trees and any other caller outside
  rehydration now fail explicitly instead of forwarding an unresolved
  reference to a stateless backend. The hydration path clears `file_id`
  before strict validation runs, so the accepted path is unchanged.
- Prefix user-facing file-store errors raised while reading a referenced file
  with the part position (`input[i].content[j]:`) while preserving the 404,
  400, or 409 status.
- Normalize the stored media type to lowercase and require a valid
  `type/subtype` token before building the data URL, so downstream data-URL
  parsing does not have to handle case or malformed prefixes.
- Document that compaction windows and multi-agent snapshots derive from the
  resolved input and can retain the data URL, and that inline `file_data`
  counts toward `max_request_body_size_bytes` while a `file_id` reference
  does not.

Refs: vllm-project#356

## Test Plan

- `image_references_pass_before_hydration_but_not_strict_validation` covers
  the new strict-validation rejection and its part position.
- `continuation_reports_a_deleted_file_reference` continues a stored response
  after deleting the referenced file and asserts a 404 with the part position.
- `normalizes_the_stored_media_type` uploads `IMAGE/PNG` and asserts the
  lowercase data URL; `rejects_an_invalid_stored_content_type` also covers a
  colon-bearing media type.
- The hydration test asserts `prompt_cache_breakpoint` survives on an
  `input_file` reference; the missing-file test asserts the position prefix and
  that the file ID is not echoed.
- `cargo test --workspace --no-fail-fast`, `cargo clippy --workspace
  --all-targets -- -D warnings`, `cargo fmt --all -- --check`, and all
  pre-commit hooks pass.

Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
Copilot AI balanced review requested due to automatic review settings November 13, 2026 11:06

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@maralbahari maralbahari left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking up #356. Keeping the client's file_id in history and resolving it per request is the right direction, but the resolution happens in the place RFC #500 (one typed path for rehydrating and persisting history, delivered in #501, #502, #503, #505 and #506) moved away from. History holds only what the client sent and received, and ResponsesInput::normalized_model_input is the one point that builds the model-facing form for each upstream request (ARCHITECTURE.md, "Output-to-input conversion for continuation rounds"). Inline file_data already works that way: it is lowered to input_text at that point while history keeps the original input_file.

Client-facing and model-facing shapes are different things. The client-facing shape is what the client sent and received: an input_file or input_image with its file_id, public tool items, compaction items. The model-facing shape is what one upstream needs for one request: here a base64 data URL in place of the file_id, elsewhere a gateway call lowered to a function call and its output, or a compaction item turned into a summary. It depends on the upstream and is rebuilt for every request. The gateway must always store the client history shape, never the internal model-facing one, in response and conversation history, compaction windows, session checkpoints and multi-agent tree snapshots alike.

We are gating changes to stay on that single path, and this PR adds what RFC #500 removed: a new rehydration step (hydrate_file_references), a second validation mode (validate_before_hydration with an allow_file_id flag), and model-facing data in persisted state. Because it rewrites enriched_request.input, compaction carries the resolved data URL in its window and multi-agent tree snapshots retain it, which the PR documents as intended and leaves to #534. We can't take a new rehydration, validation or persistence path. Please resolve file_id as part of the existing message-file lowering, where file_data already becomes input_text, so rehydration, validation and storage stay as they are and only ever see the client's file_id; then #534 is not needed.

Where the file access can live. The design can follow ConversationHandler and ResponseHandler in executor/modes/, which own their stores and keep the stored entries intact (ARCHITECTURE.md, "The hard boundary: no direct storage access"): a file handler there wraps the file service, and both the conversation and the response handler use it inside their existing rehydrate and persist. Persistence then keeps the client's file_id entry exactly as sent, and rehydration resolves it only for the model-facing request, without a new step in rehydrate.rs or a direct file_search call from the executor.

The behavior also needs OpenAI evidence rather than a chosen reading, for /v1/files itself and for the references to it. There are no OpenAI and gateway recordings of /v1/files yet, so please record both sides for: uploads with the content types and purposes this PR relies on (text, image/*, and ones it rejects), retrieving and deleting a file, and the errors for an invalid upload; then a Responses request that references an uploaded file through input_file and input_image, whether a previous_response_id continuation still works after the file is deleted, and the error status, code and message for a missing file or one with the wrong type. record_text_file_input_cassettes.sh (tests/cassettes/text_files/) already records inline input_file against both providers and is the natural place for these scenarios. Please match the gateway to the OpenAI recordings. Details inline.

Comment thread docs/guides/responses-compaction.md Resolved
Record the gateway behavior for /v1/files uploads, metadata and content retrieval, deletion, and Responses input_file and input_image file_id references with vLLM-hosted Qwen2.5 models. Extend the recorder with a files mode that captures multipart and binary bodies, and document the files scenarios.

Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
@LeaderOnePro

LeaderOnePro commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor Author

@maralbahari Thanks for the detailed review. The RFC #500 argument is clear, and I agree the resolution belongs at the model-facing lowering point rather than in a new rehydration step. Before I rework the branch, here are the gateway recordings and the two points I want to confirm.

Gateway recordings

I recorded the gateway side with vLLM 0.33.0 (Qwen/Qwen2.5-3B-Instruct for text and Qwen/Qwen2.5-VL-3B-Instruct for images) through the existing script:

TEXT_FILE_RECORD_SET=gateway TEXT_FILE_SCENARIOS="files-text files-image files-errors" \
MODEL=Qwen/Qwen2.5-3B-Instruct FILES_IMAGE_MODEL=Qwen/Qwen2.5-VL-3B-Instruct \
GATEWAY_URL=http://localhost:9000 \
bash crates/agentic-server-core/tests/cassettes/record_text_file_input_cassettes.sh

Three cassettes are committed in crates/agentic-server-core/tests/cassettes/text_files/responses/ (commit 9375f6e):

  • files-api-text-gateway-Qwen-Qwen2.5-3B-Instruct.yaml (11 turns): upload notes.txt (user_data), get metadata and content, reference it through input_file.file_id, continue through previous_response_id, delete it, retry the continuation, reference it again, upload it again, reference it through input_image (rejected), delete.
  • files-api-image-gateway-Qwen-Qwen2.5-VL-3B-Instruct.yaml (10 turns): the same lifecycle for input_image.file_id with the red-blue-64.png fixture uploaded as colors.png (vision), including the text-as-image rejection.
  • files-api-errors-gateway-Qwen-Qwen2.5-3B-Instruct.yaml (6 turns): invalid purpose, missing file, missing multipart body, and missing-file metadata/content/delete errors.

What the gateway does today:

  • A previous_response_id continuation after the file is deleted returns 404 {"error":{"code":"not_found","message":"input[0].content[1]: File not found","type":"not_found"}}.
  • A text file referenced as input_image returns 400 {"error":{"code":"invalid_request_error","message":"invalid request: input[0].content[1]: input_image.file_id must reference a file with an image media type","type":"invalid_request_error"}}.
  • Invalid uploads return 400 invalid_request_error; missing files return 404 not_found with File not found.

OpenAI side

My environment is temporarily inconvenient for recording the OpenAI side, so I could not record that reference set. Could you run the same recording on your side?

TEXT_FILE_RECORD_SET=openai TEXT_FILE_SCENARIOS="files-text files-image files-errors" \
OPENAI_MODEL=gpt-4o \
bash crates/agentic-server-core/tests/cassettes/record_text_file_input_cassettes.sh

I will match the gateway to those recordings, including whether the continuation after deletion still succeeds and the exact status, code, and message for missing files and wrong file types.

Before I rework the pipeline

  1. Synchronous lowering vs async file access. normalized_model_input and the public to_upstream_request are synchronous. A missing file also has to surface as 404 not_found (as recorded), while InputValidationError maps to 400. My plan is to read the referenced files in the file handler during the existing rehydration step, keep enriched_request.input and every stored history shape exactly as the client sent them, and pass a file_id -> bytes map into the synchronous lowering. Is that the split you have in mind, or would you rather make lowering async and read there?

  2. Input assembled outside the handlers. from_responses (stateless), cached session parents, and the multi-agent tree build the input in rehydrate.rs without calling ConversationHandler or ResponseHandler, and the current request's new items are merged there as well. A resolve inside a handler's rehydrate cannot see that final merged input. Would you rather route those paths through the handlers, or let the file handler resolve at that merge point, with the handler owned by ExecutionContext rather than a direct file_search call from the executor?

Next

I will remove hydrate_file_references, validate_before_hydration, the allow_file_id flag and validate_allow_file_id, and the 64 MiB budget; make file_id valid input in the single validate; resolve references in the message-file lowering; persist only client-facing items; and close #534 as unnecessary. #535 stays. I plan to count the resolved content's encoded length against max_request_body_size_bytes rather than a separate budget, which also means threading that limit into core.

Replay the recorded Files API and Responses file_id lifecycles through the HTTP gateway, remapping recorded file and response IDs and asserting the hydrated model input, stored client references, and error mapping.

Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
@maralbahari

Copy link
Copy Markdown
Collaborator

@maralbahari Thanks for the detailed review. The RFC #500 argument is clear, and I agree the resolution belongs at the model-facing lowering point rather than in a new rehydration step. Before I rework the branch, here are the gateway recordings and the two points I want to confirm.

Gateway recordings

I recorded the gateway side with vLLM 0.33.0 (Qwen/Qwen2.5-3B-Instruct for text and Qwen/Qwen2.5-VL-3B-Instruct for images) through the existing script:

TEXT_FILE_RECORD_SET=gateway TEXT_FILE_SCENARIOS="files-text files-image files-errors" \
MODEL=Qwen/Qwen2.5-3B-Instruct FILES_IMAGE_MODEL=Qwen/Qwen2.5-VL-3B-Instruct \
GATEWAY_URL=http://localhost:9000 \
bash crates/agentic-server-core/tests/cassettes/record_text_file_input_cassettes.sh

Three cassettes are committed in crates/agentic-server-core/tests/cassettes/text_files/responses/ (commit 9375f6e):

  • files-api-text-gateway-Qwen-Qwen2.5-3B-Instruct.yaml (11 turns): upload notes.txt (user_data), get metadata and content, reference it through input_file.file_id, continue through previous_response_id, delete it, retry the continuation, reference it again, upload it again, reference it through input_image (rejected), delete.
  • files-api-image-gateway-Qwen-Qwen2.5-VL-3B-Instruct.yaml (10 turns): the same lifecycle for input_image.file_id with the red-blue-64.png fixture uploaded as colors.png (vision), including the text-as-image rejection.
  • files-api-errors-gateway-Qwen-Qwen2.5-3B-Instruct.yaml (6 turns): invalid purpose, missing file, missing multipart body, and missing-file metadata/content/delete errors.

What the gateway does today:

  • A previous_response_id continuation after the file is deleted returns 404 {"error":{"code":"not_found","message":"input[0].content[1]: File not found","type":"not_found"}}.
  • A text file referenced as input_image returns 400 {"error":{"code":"invalid_request_error","message":"invalid request: input[0].content[1]: input_image.file_id must reference a file with an image media type","type":"invalid_request_error"}}.
  • Invalid uploads return 400 invalid_request_error; missing files return 404 not_found with File not found.

OpenAI side

My environment is temporarily inconvenient for recording the OpenAI side, so I could not record that reference set. Could you run the same recording on your side?

TEXT_FILE_RECORD_SET=openai TEXT_FILE_SCENARIOS="files-text files-image files-errors" \
OPENAI_MODEL=gpt-4o \
bash crates/agentic-server-core/tests/cassettes/record_text_file_input_cassettes.sh

I will match the gateway to those recordings, including whether the continuation after deletion still succeeds and the exact status, code, and message for missing files and wrong file types.

Before I rework the pipeline

  1. Synchronous lowering vs async file access. normalized_model_input and the public to_upstream_request are synchronous. A missing file also has to surface as 404 not_found (as recorded), while InputValidationError maps to 400. My plan is to read the referenced files in the file handler during the existing rehydration step, keep enriched_request.input and every stored history shape exactly as the client sent them, and pass a file_id -> bytes map into the synchronous lowering. Is that the split you have in mind, or would you rather make lowering async and read there?
  2. Input assembled outside the handlers. from_responses (stateless), cached session parents, and the multi-agent tree build the input in rehydrate.rs without calling ConversationHandler or ResponseHandler, and the current request's new items are merged there as well. A resolve inside a handler's rehydrate cannot see that final merged input. Would you rather route those paths through the handlers, or let the file handler resolve at that merge point, with the handler owned by ExecutionContext rather than a direct file_search call from the executor?

Next

I will remove hydrate_file_references, validate_before_hydration, the allow_file_id flag and validate_allow_file_id, and the 64 MiB budget; make file_id valid input in the single validate; resolve references in the message-file lowering; persist only client-facing items; and close #534 as unnecessary. #535 stays. I plan to count the resolved content's encoded length against max_request_body_size_bytes rather than a separate budget, which also means threading that limit into core.

Thanks for recording the gateway side; the way you recorded it is correct. I will record the OpenAI references and push them to this branch, and add a few more cases, such as continuing from a response that was deleted. Please implement the gateway behavior with the OpenAI recordings as the ground truth, including the status, code and message for a missing or deleted file and for a wrong file type; the gateway's current 404 not_found is not the reference.

  1. enriched_request.input keeps the role described in ARCHITECTURE.md ("Original and enriched requests"): the resolved history in the client's shape, with the file_id as sent. The model-facing form is produced only when the upstream request is built, through to_upstream_request and normalized_model_input, the same way file_data is lowered to input_text today. So the split you describe works: read the referenced files asynchronously and let the synchronous lowering use what was read. That lookup must stay out of everything stored or retained; stored items, the checkpoint, session parents and the multi-agent tree snapshot all keep the file_id.

  2. Those paths don't call a handler's rehydrate on purpose: their history is already in the request, the session or the tree snapshot, so there is nothing to load from storage, and the handlers are the boundary to the database. Please keep it that way rather than routing them through the handlers. Since every path, including each agent of a multi-agent run, ends in the same lowering, the file access only needs the final input: once rehydration has assembled it (where the final enriched_request.input.validate() runs), a file handler in executor/modes/, owned by ExecutionContext and wrapping the file service, can read the files it references, including the restored agents' histories. No direct file_search call from the executor.

The rest of the plan looks right. For the size check, reuse the existing max_request_body_size_bytes value, passed into core once at startup, rather than adding a new setting.

Side note: while implementing, please make sure you fully understand the existing functionality and reuse the code that is already there rather than adding new blocks. If the existing code lacks something, or isn't extendable or maintainable, a small refactor of it is better than introducing new wrappers or workarounds.

Signed-off-by: maral <maralbahari.98@gmail.com>
@maralbahari

Copy link
Copy Markdown
Collaborator

@LeaderOnePro
I have recorded the OpenAI side for you and pushed it to this branch. The recorder now has four more Files scenarios next to files-text, files-image and files-errors, and all seven OpenAI references (gpt-4o) are in crates/agentic-server-core/tests/cassettes/text_files/responses/files-api-*-openai-gpt-4o.yaml:

Scenario What it records
files-api every upload purpose with text, image, PDF, empty and binary content; listing with limit, after, purpose and invalid parameters; metadata and content retrieval; repeated deletion
files-input input_file by file_id (text, PDF, image), inline file_data, file_url, file_id combined with filename, file_data or image_url, input_image by file_id, a file-only message, bare input_file/input_image items, files in function_call_output, a missing file, store: false, streaming, then continuing after deleting the response, the file, or a conversation item's file
files-search vector stores, attachments with attributes, search, and the file_search tool, including missing and empty stores
files-compact /v1/responses/compact over a file by file_id, file_data and previous_response_id, continuing from each window, then again after the files are deleted

Please record the gateway side of the new scenarios the same way you recorded the first three, and match the gateway to these references:

TEXT_FILE_RECORD_SET=gateway TEXT_FILE_SCENARIOS="files-api files-input files-search files-compact" \
MODEL=Qwen/Qwen2.5-3B-Instruct GATEWAY_URL=http://localhost:9000 \
bash crates/agentic-server-core/tests/cassettes/record_text_file_input_cassettes.sh

What the OpenAI contract shows

Case OpenAI
A missing or deleted file, by file_id in a message, in function_call_output, on a previous_response_id continuation, in a conversation, or in a compacted window 404, type: invalid_request_error, code: null, param: null, message Files [<id>] were not found (the ids are listed together)
Continuing from a response whose input_image file was deleted 200; the image is answered from the stored history
file_id together with filename, file_data or image_url 400, code: mutually_exclusive_parameters, param: input[0].content[1], message Mutually exclusive parameters: 'input[0].content[1]'. Ensure you are only providing one of: 'file_id' or '<field>'.
A text or PDF file as input_image 400, code: null, param: input, message Invalid input: Expected image type to be a supported format: .jpeg, .jpg, .png, .gif, .webp but got .txt. (.pdf for a PDF)
An image as input_file 400, code: null, param: input, message Invalid input: Expected context stuffing file type to be a supported format: ... but got .png.
A message whose only content is an input_file accepted, with or without "type": "message"
input_file or input_image as a top-level input item 400, code: invalid_value
file_id inside function_call_output content resolved like a message file
store: false with a file_id works
Streaming with a missing file a JSON 404 before any stream starts
/v1/responses/compact over a file the window keeps the client's message with the same file_id (OpenAI adds "detail": "auto"), followed by the compaction item; continuing from it after the file is deleted returns the 404 above
GET /v1/files/{id}/content for user_data or assistants files 400, message Not allowed to download files of purpose: <purpose>; vision files download
/v1/files endpoints with an unknown id 404, param: id, message No such File object: <id>
Uploads vision accepts only images (400 Invalid file format text/plain. ...); an empty file returns 400 File is empty.; an unknown purpose returns 400 with param: purpose

So the reference stays in history and is resolved again each time the model needs it, including in compacted windows; nothing resolved is kept. The gateway's current 404 not_found / File not found and 400 invalid_request_error envelopes need to follow the rows above.

The design to follow

flowchart TD
    A["Request"] --> B["RequestPayload input.validate()<br/>file_id is valid input in the single validate<br/>mutually exclusive fields: 400 as recorded"]
    B --> C["rehydrate_with_continuation, branches unchanged<br/>conversation, previous response, session parent,<br/>stateless input, restored agent tree"]
    C --> D["enriched_request.input: client-facing history<br/>input_file and input_image keep file_id"]
    D --> E["exec_ctx.file_handler.resolve(final input and restored agent histories)<br/>one call at the end of rehydration<br/>missing ids: 404, wrong type: 400, as recorded<br/>resolved bytes count against max_request_body_size_bytes"]
    E --> F["RequestContext.files: per-turn lookup, never persisted or retained"]
    F --> G["upstream_request, for every round, agent and compaction summary<br/>to_upstream_request and normalized_model_input with the lookup"]
    G --> H["Lowering, on the existing file_data path<br/>input_file file_id to input_text 'File: name ...'<br/>input_image file_id to an image data URL<br/>the same for function_call_output content"]
    H --> I["Upstream model"]
    D --> J["Persistence unchanged: checkpoint, conversation items,<br/>compaction window and tree snapshot keep file_id"]
    K["executor/modes/files.rs: FileHandler<br/>wraps FileSearchService, owned by ExecutionContext"] -.-> E
Loading
  1. types/io/input/validation.rs and file_text.rs: a file_id part is valid input in the single validate, and the mutually exclusive rule is checked there as recorded. The InputFileContent to InputTextContent conversion takes the resolved bytes and filename for a file_id, through the same code that inline file_data uses.
  2. executor/modes/files.rs: a FileHandler next to ConversationHandler and ResponseHandler. Like they wrap their stores, it wraps the FileSearchService that ExecutionContext already holds; it only reads, so it has one resolve method and no persist. resolve collects the file_ids from message parts and function_call_output content, reads each through the existing get_file and file_content, and returns every missing id in one typed error carrying OpenAI's envelope. No change to the service itself is needed.
  3. executor/request.rs: ExecutionContext owns the FileHandler, and RequestContext carries the per-turn lookup. The agent and compaction contexts built in multi_agent/context.rs, multi_agent/rounds.rs and compaction.rs share it.
  4. rehydrate.rs: one resolve call after the branches have assembled the final input, where the final enriched_request.input.validate() runs. The branches stay as they are, and enriched_request.input is never rewritten.
  5. to_upstream_request and normalized_model_input take the lookup; their callers are upstream.rs, the preflight check in execute.rs, and the compaction summary request.
  6. Persistence, storage, checkpoints, session retention and the tree snapshot do not change.

With this in place, please remove hydrate_file_references, validate_before_hydration, the allow_file_id flag and validate_allow_file_id, the 64 MiB constant, and the documentation that says compacted windows or tree snapshots hold resolved data.

Resolve Responses input_file and input_image file_id references from the
Files API once per turn, after history assembly and before inference, and
use the lookup only for the model-facing projection. Stored history,
compaction windows, and multi-agent tree snapshots keep the client's
reference, following the single typed path from RFC vllm-project#500.

- Accept file_id references in messages and structured tool outputs and
  reject a part that also carries its inline field with OpenAI's
  mutually_exclusive_parameters envelope.
- Return OpenAI's missing-file and wrong-type envelopes, and count the
  resolved escaped size toward the request body limit.
- Accept the vision upload purpose and drop the unused FileContent API.
- Record the Files API and file_id reference cassettes and replay them
  through the HTTP gateway.

Refs vllm-project#356

Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
Bring in model capability validation (vllm-project#512), unknown conversation id
rejection (vllm-project#531), persisted-turn documentation (vllm-project#533), and the shared
input-item test helper (vllm-project#545).

Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>

@maralbahari maralbahari left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the rework. Validation, the FileHandler seam, lowering through the file_data path and keeping file_id in every stored shape all match what we agreed, and the old rehydration step and validation mode are gone.

Seeing the per-turn lookup implemented, I'd like to change one part of what I suggested. As a RequestContext field it is set to HashMap::default() at every literal (21 sites, plus 64 test calls passing an empty map), copied by hand into AgentContext, CompactionPlan, summarize_items and three sub-request contexts, and forced into the split context in agentic-llm-d. Because an empty map can also mean "not resolved", the lowering needs a new internal error class to catch a skipped stage. The file_id is already part of the checkpoint through the stored history rows, session history, compaction windows and tree snapshots, and the resolved bytes must stay out of all of them, so they should not be a context field either.

Please build the lookup where the upstream body is built instead: make upstream_request async, call exec.file_handler.resolve there on the input it lowers, and pass the result to to_upstream_request. Keep the early 404/400/413 by doing the same in the existing preflight in execute.rs::prepare. Then RequestContext, rehydration, the multi-agent contexts and compaction need no change, and referenced_items, UnresolvedFileId and is_internal can go.

Two more things before this can merge:

  • Gateway recordings for files-api, files-input, files-search and files-compact, with replay tests against the OpenAI references. The Files endpoint rows (download by purpose, unknown id, upload rules, the delete object) are not implemented yet.
  • Please keep the existing InvalidInput message, code and param unchanged except for the new mutually exclusive case; other input errors change without recordings now.

I'll decide separately on the deleted input_image continuation (OpenAI 200, gateway 404); keep the 404 and the note for now.

Move the per-turn Files API lookup out of `RequestContext` and rehydration.
`upstream_request` is now async and resolves the references in the input it
lowers, and `execute.rs::prepare` repeats that lookup so a missing file, a
wrong file type, or an oversized body still fails before tool preparation.

- Share one `file_references` collector between the resolver and split-
  execution detection, and remove `file_references.rs`, `ReferenceKind`,
  `is_internal`, and `UnresolvedFileId`.
- Check a resolved file's kind only where it is lowered, mapping the recorded
  wrong-type envelope through one `From<InputValidationError>`.
- Keep the file-kind rules in `file_text/kinds.rs` and keep one file-service
  handle on `ExecutionContext`; remove the test-only service setters.
- Carry `max_request_body_size` on the core `Config`, read by `from_config`.
- Implement the recorded Files API rows: upload purposes and vision media
  validation, download by purpose, unknown-id envelopes for lookup, content,
  delete, and list cursors, and the `object: "file"` delete row.
- Record the files-api, files-input, files-search, and files-compact gateway
  cassettes, re-record files-image with the vision model, and compare every
  gateway Files row with the OpenAI reference.

Refs vllm-project#356

Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>

@maralbahari maralbahari left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is much closer. The lookup is off RequestContext and built with the upstream body, rehydration is back to main, and the gateway recordings are in. What's left is mostly duplication, and removing it also makes conversions.rs and resolved_file.rs much smaller.

1. normalized_model_input can stay close to main.

Most of its new code is not the lookup itself:

  • File failures live in two error types, ResolvedFileError and InputFileError. Every call site merges them through a file_error(error, |reason| ...) callback, written out four times.
  • The conversions are free functions (resolved_file, file_error, lower_tool_output) next to the types that own them: InputFileContent, InputImageContent and ToolCallOutput.
  • A missing lookup entry hits an expect in a production path.

With one error type and the conversions on their owners, each branch is one call again:

InputContent::InputFile(file) => {
    let text = file.to_text(files).map_err(|reason| InputValidationError::unsupported_file(model.len(), part_index, reason))?;
    *part = InputContent::InputText(text);
}
InputContent::InputImage(image) => {
    image.detail.get_or_insert_with(|| "auto".to_owned());
    image.inline_file(files).map_err(|reason| InputValidationError::unsupported_file(model.len(), part_index, reason))?;
}
InputItem::FunctionCallOutput(mut output) => {
    output.output.inline_files(files, model.len())?;
    model.push(InputItem::FunctionCallOutput(output));
}
  • InputFileContent::to_text(&files) covers inline file_data and file_id in one place, replacing InputTextContent::try_from(&InputFileContent) for both.
  • InputImageContent::inline_file(&files) and ToolCallOutput::inline_files(&files, item_index) replace lower_tool_output.
  • A missing entry returns an error instead of panicking.
  • The two recorded wrong-kind messages become InputFileError variants, reusing the NotImage this PR adds but never constructs. unsupported_file gives them the recorded envelope (param: input, no position).

That removes ResolvedFileError, InputValidationError::resolved_file and resolved_file_message, the callback, the three free functions and the expect. resolved_file.rs then only needs the file's name and bytes, which FileObject and file_content already give.

2. New objects restate ones that already exist.

New in this PR Already exists Please
ResolvedFile { filename, bytes } FileObject (its filename) plus the bytes from file_content use them
ResolvedFileError::{NotImage, NotText, Content} InputFileError::NotText. This PR also adds InputFileError::NotImage, which nothing constructs one error: InputFileError
InputValidationError::resolved_file and resolved_file_message InputValidationError already reports InputFileError remove
data_url_len, text_json_len and the size loop in FileHandler::resolve the upstream body that upstream_request already serializes check the limit on that body
ReferenceRead { Missing, Failed } FileSearchError::NotFound versus the rest match FileSearchError
FilesApiError, FilesApiErrorEnvelope, files_api_error, file_not_found, file_row FileSearchError → ExecutorError → executor_error_response, and the OpenAI envelope in error_response.rs extend that path
the purpose check in the HTTP handler the same check and message in upload_file keep the service's
"NotFound or InvalidRequest → No such File object", four times in the HTTP handler the service's NotFound have the service return NotFound for an unusable ID
VISION_CONTENT_TYPES kinds::image_media_type, with the same five formats one list in kinds.rs

Two more points:

  • validate_tool_output validates tool-output file parts, but the OpenAI recordings have no such rejection. files-api-input t21 to t24 are success and a missing file only, and tool outputs are not validated on main. Please remove it.
  • FileHandler<'a> borrows ExecutionContext.file_search. Please make it a plain handler like ConversationHandler and ResponseHandler, owning the service through FileHandler::new(service) and replacing file_search, so there is still one handle. Until the follow-up below, the HTTP handler can reach the service through it.

Follow-up PR. The /v1/files endpoints still call FileSearchService straight from the HTTP handler. Conversations and responses go through their mode handler: the transport calls exec_ctx.conv_handler or exec_ctx.resp_handler, which calls the store. Once this PR is in, please open a follow-up that gives FileHandler the endpoint methods and has the HTTP handlers call them:

impl FileHandler {
    pub async fn upload(&self, ...) -> ExecutorResult<FileObject> { ... }
    pub async fn list(&self, params: &ListParams) -> ExecutorResult<ListResponse<FileObject>> { ... }
    pub async fn retrieve(&self, id: &str) -> ExecutorResult<FileObject> { ... }
    pub async fn content(&self, id: &str) -> ExecutorResult<Vec<u8>> { ... }
    pub async fn delete(&self, id: &str) -> ExecutorResult<DeleteObject> { ... }
}

Done when:

  • there is one wrong-kind error, on InputFileError;
  • normalized_model_input has one call per file branch, with no callback, free function or expect;
  • ResolvedFileError and the size estimators are gone;
  • there is one Files error envelope;
  • validation.rs does not validate tool outputs;
  • FileHandler owns the one file service.

@LeaderOnePro

Copy link
Copy Markdown
Contributor Author

@maralbahari Thanks — agreed on all of these. The remaining work is the deduplication pass you described; I'll keep the behavior pinned by the recordings and replay tests.

Planned changes:

  • one wrong-kind error on InputFileError; InputFileContent::to_text, InputImageContent::inline_file and
    ToolCallOutput::inline_files own the conversions, so normalized_model_input has one call per branch with no
    callback, free function or expect;
  • drop ResolvedFile, ResolvedFileError, data_url_len, text_json_len and the FileHandler::resolve size loop;
    check the limit on the serialized upstream body in upstream_request;
  • one Files error envelope through FileSearchError → ExecutorError → executor_error_response; keep the purpose
    check in upload_file and have the service return NotFound for an unusable ID;
  • remove tool-output validation from validation.rs;
  • make FileHandler own the one FileSearchService, built in ExecutionContext::from_config, and drop the lifetime
    and file_handler();
  • match FileSearchError directly instead of ReferenceRead, and keep one image-format list in kinds.rs.

I'll also remove the stray blank line in conversation.rs.

One extra: the CI Python client test still expects files.content to download an assistants upload. The recordings show OpenAI rejects that purpose, so I'll update the script to assert the recorded 400 and keep a vision download roundtrip.

I'll open the /v1/files handler follow-up after this PR merges.

Resolve every file_id reference through one placement path and report
Files API row failures with the recorded OpenAI envelope:

- Lower a resolved message part in place: InputImageContent::inline_file
  and InputFileContent::to_text own the image and text conversion, and
  ToolCallOutput::inline_files does the same for structured tool output.
  Drop ResolvedFile, ResolvedFileError, the size estimators and the
  resolve-time size loop; upstream_request checks the serialized body
  against max_request_body_size when it carries resolved content.
- FileHandler now owns the executor's one FileSearchService in
  ExecutionContext, so request execution and the /v1/files handlers use
  the same handle. prepare() builds the model-facing body before tool
  preparation, keeping early 404, 400 and 413 failures.
- FileSearchService reports any unusable row ID as FileNotFound and
  validates uploads itself through FileRowRejected; ExecutorError
  renders both with invalid_request_error, code null and param
  id/after/before. The vision format list comes from the shared
  IMAGE_MEDIA_TYPES constant.
- Message validation no longer inspects structured tool outputs; their
  file_id lowering reports through the output position.

Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
An assistants upload cannot be downloaded: OpenAI rejects GET
/v1/files/{id}/content for every purpose but vision, and the gateway
now reproduces that row. Assert the recorded 400 message and add a
vision PNG upload, download and delete roundtrip.

Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
Document that FileHandler owns the executor's one file service, that
the model-facing body is size-checked after serialization, and that
Files API rows use the recorded invalid_request_error envelope.

Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
A resolved file whose kind does not match a structured tool-output part
reported a positioned `invalid request` error, while the recorded row
(files-api-input t22) uses the same bare message, `param: input` and
`code: null` as a message part. Route it through the shared resolved-kind
envelope, as unsupported_file already does.

Also pin what the dedup pass left untested: the serialized-body 413 for
resolved content in upstream_request, the to_text / inline_file /
inline_files conversions (NotTextFile, NotImage, Unresolved, and
positioned content failures), the image media-type mapping behind
IMAGE_MEDIA_TYPES, and an HTTP guard that replays the t22 tool-output
turn.

FileHandler::resolve no longer matches FileSearchError::NotFound: the
service maps every unusable lookup to FileNotFound.

Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
ExecutionContext::with_file_search_service is gone; embedders wrap their
service with FileHandler::new and assign ExecutionContext.file_handler.
Update both file-search pages and the field comment.

Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>

@maralbahari maralbahari left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this round removes most of what we discussed. The wrong-kind error lives on InputFileError, the conversions are methods on the content types, the expect is gone, and conversions.rs is about 110 lines shorter.

Four new files remain that should not be needed. Their contents belong in existing code:

New file Where its contents go
types/io/input/resolved_file.rs (only the ResolvedFiles alias) content.rs, next to to_text and inline_file, which take it
types/io/input/tool_output.rs (ToolCallOutput::inline_files) content.rs, next to the two conversions it calls
types/file_search/file_row.rs (FILE_PURPOSES, FileRowParam) FILE_PURPOSES goes back in types/file_search.rs. In place of FileRowParam, the two Files variants carry the parameter as &'static str, as ExecutorError reports parameters elsewhere
types/file_search/error.rs (FileSearchError, moved, plus FileNotFound and FileRowRejected) FileSearchError, with the two Files variants, goes back in types/file_search.rs. That comes to 497 production lines, under the 500-line limit

A note on the next revision: several changes in this round satisfy the wording of a review comment rather than its intent. A method we named got its own module, "the parameter" became a new enum, and our description of the OpenAI recordings ("rows") became type names. Please read each comment for what it asks the code to become, and extend the existing types and modules above rather than adding new ones. If the next revision again adds new types, modules, wrappers or names that only answer a comment's wording, we can't move forward with this PR and will close it.

Move the remaining helpers back into the modules that own the types, so
there is no standalone module for one alias or one method:

- ResolvedFiles and ToolCallOutput::inline_files move into
  types/io/input/content.rs, next to to_text and inline_file.
- FILE_PURPOSES and FileSearchError move back into
  types/file_search.rs; the two Files variants carry their parameter as
  a &'static str instead of a dedicated enum, and FileRejected replaces
  FileRowRejected.
- ExecutorError::error_param reports those parameters directly.

No behavior changes; types/file_search.rs is 497 production lines,
within the 500-line limit.

Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
Signed-off-by: maral <maralbahari.98@gmail.com>

@maralbahari maralbahari left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving with one fix pushed directly (13d348b).

ctx.enriched_request.input.normalized_model_input(&[])?;
// Fail before any tool preparation or response event if the model can't be given this input:
// a missing referenced file returns its JSON 404 and a mismatched file type its 400 first.
let files = exec.file_handler.resolve(&ctx.enriched_request.input).await?;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed a mistake on this line and pushed the fix directly to be faster. It called upstream_request(exec, &ctx, false, None, None), which built a full upstream request with placeholder arguments (no stream, guidance or withheld tools) only to throw it away, and the first round then built it again. The last review didn't mean for that to happen. The fix puts back the previous file resolve and input check here, and the body-size check stays in upstream_request, on the request that is actually sent.

Signed-off-by: maral <maralbahari.98@gmail.com>

# Conflicts:
#	CHANGELOG.md
#	crates/agentic-server-core/src/executor/request.rs
#	crates/agentic-server-core/src/executor/upstream.rs
Signed-off-by: maral <maralbahari.98@gmail.com>

# Conflicts:
#	CHANGELOG.md
Signed-off-by: maral <maralbahari.98@gmail.com>
@LeaderOnePro

Copy link
Copy Markdown
Contributor Author

@maralbahari Thanks for the approval and for pushing the prepare fix directly. Resolving the files there and lowering with normalized_model_input is cleaner than building the whole upstream body just for the preflight. I also appreciate that you merged main and resolved the conflicts on the branch. Thanks as well for your patience through the revisions; the rule about extending the existing types and modules rather than adding new ones is clear.

@franciscojavierarceo
franciscojavierarceo merged commit 8c1cd74 into vllm-project:main Nov 18, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants