Repository navigation
feat: resolve Responses file_id references from the Files API - #536
franciscojavierarceo merged 21 commits into
Conversation
## Summary Message `input_image` and `input_file` parts may reference an uploaded file by `file_id`. Rehydration now reads the file from the gateway file store and replaces the reference with a base64 data URL in the model-facing request, so stateless backends receive the media inline. The client's reference stays in `original_request` and canonical history, so each continuation resolves it again. - `rehydrate/files.rs` hydrates the enriched input after history assembly and before final validation. An `input_image` reference must resolve to an `image/*` file; an `input_file` reference is subject to the existing UTF-8 text media-type and filename rules. - Early validation gains a pre-hydration mode that accepts a reference-only `input_file` while still rejecting every other unsupported form. Strict validation runs again after hydration. - `FileSearchService::read_file` returns the stored bytes, filename, and content type that `FileObject` intentionally omits, and uploads now accept purpose `vision`. - A missing store, deleted file, inline/reference conflict, non-image image reference, or invalid stored content type fails explicitly with the item and part position. Resolved data URLs are capped at 64 MiB per request (413 `body_too_large`). Refs: vllm-project#356 ## Test Plan - Unit tests in `rehydrate/files.rs` cover inlining both part kinds with field preservation, inline conflicts, a non-image `input_image`, an invalid stored content type, missing files, a missing file store, and the per-request budget. - `rehydrate/tests.rs` covers new input, a missing store, and a continuation that resolves a reference from stored history, asserting the canonical copy keeps `file_id` and the upstream request carries inline media. - Integration tests cover `read_file` metadata and the `vision` purpose. - Updated the two existing rejection tests for the new message. - `cargo test --workspace --no-fail-fast`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo fmt --all -- --check`, and all pre-commit hooks pass. Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
## Summary - Reject `input_image.file_id` in strict validation when a request path does not rehydrate. Stored multi-agent trees and any other caller outside rehydration now fail explicitly instead of forwarding an unresolved reference to a stateless backend. The hydration path clears `file_id` before strict validation runs, so the accepted path is unchanged. - Prefix user-facing file-store errors raised while reading a referenced file with the part position (`input[i].content[j]:`) while preserving the 404, 400, or 409 status. - Normalize the stored media type to lowercase and require a valid `type/subtype` token before building the data URL, so downstream data-URL parsing does not have to handle case or malformed prefixes. - Document that compaction windows and multi-agent snapshots derive from the resolved input and can retain the data URL, and that inline `file_data` counts toward `max_request_body_size_bytes` while a `file_id` reference does not. Refs: vllm-project#356 ## Test Plan - `image_references_pass_before_hydration_but_not_strict_validation` covers the new strict-validation rejection and its part position. - `continuation_reports_a_deleted_file_reference` continues a stored response after deleting the referenced file and asserts a 404 with the part position. - `normalizes_the_stored_media_type` uploads `IMAGE/PNG` and asserts the lowercase data URL; `rejects_an_invalid_stored_content_type` also covers a colon-bearing media type. - The hydration test asserts `prompt_cache_breakpoint` survives on an `input_file` reference; the missing-file test asserts the position prefix and that the file ID is not echoed. - `cargo test --workspace --no-fail-fast`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo fmt --all -- --check`, and all pre-commit hooks pass. Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
There was a problem hiding this comment.
Thanks for picking up #356. Keeping the client's file_id in history and resolving it per request is the right direction, but the resolution happens in the place RFC #500 (one typed path for rehydrating and persisting history, delivered in #501, #502, #503, #505 and #506) moved away from. History holds only what the client sent and received, and ResponsesInput::normalized_model_input is the one point that builds the model-facing form for each upstream request (ARCHITECTURE.md, "Output-to-input conversion for continuation rounds"). Inline file_data already works that way: it is lowered to input_text at that point while history keeps the original input_file.
Client-facing and model-facing shapes are different things. The client-facing shape is what the client sent and received: an input_file or input_image with its file_id, public tool items, compaction items. The model-facing shape is what one upstream needs for one request: here a base64 data URL in place of the file_id, elsewhere a gateway call lowered to a function call and its output, or a compaction item turned into a summary. It depends on the upstream and is rebuilt for every request. The gateway must always store the client history shape, never the internal model-facing one, in response and conversation history, compaction windows, session checkpoints and multi-agent tree snapshots alike.
We are gating changes to stay on that single path, and this PR adds what RFC #500 removed: a new rehydration step (hydrate_file_references), a second validation mode (validate_before_hydration with an allow_file_id flag), and model-facing data in persisted state. Because it rewrites enriched_request.input, compaction carries the resolved data URL in its window and multi-agent tree snapshots retain it, which the PR documents as intended and leaves to #534. We can't take a new rehydration, validation or persistence path. Please resolve file_id as part of the existing message-file lowering, where file_data already becomes input_text, so rehydration, validation and storage stay as they are and only ever see the client's file_id; then #534 is not needed.
Where the file access can live. The design can follow ConversationHandler and ResponseHandler in executor/modes/, which own their stores and keep the stored entries intact (ARCHITECTURE.md, "The hard boundary: no direct storage access"): a file handler there wraps the file service, and both the conversation and the response handler use it inside their existing rehydrate and persist. Persistence then keeps the client's file_id entry exactly as sent, and rehydration resolves it only for the model-facing request, without a new step in rehydrate.rs or a direct file_search call from the executor.
The behavior also needs OpenAI evidence rather than a chosen reading, for /v1/files itself and for the references to it. There are no OpenAI and gateway recordings of /v1/files yet, so please record both sides for: uploads with the content types and purposes this PR relies on (text, image/*, and ones it rejects), retrieving and deleting a file, and the errors for an invalid upload; then a Responses request that references an uploaded file through input_file and input_image, whether a previous_response_id continuation still works after the file is deleted, and the error status, code and message for a missing file or one with the wrong type. record_text_file_input_cassettes.sh (tests/cassettes/text_files/) already records inline input_file against both providers and is the natural place for these scenarios. Please match the gateway to the OpenAI recordings. Details inline.
Record the gateway behavior for /v1/files uploads, metadata and content retrieval, deletion, and Responses input_file and input_image file_id references with vLLM-hosted Qwen2.5 models. Extend the recorder with a files mode that captures multipart and binary bodies, and document the files scenarios. Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
|
@maralbahari Thanks for the detailed review. The RFC #500 argument is clear, and I agree the resolution belongs at the model-facing lowering point rather than in a new rehydration step. Before I rework the branch, here are the gateway recordings and the two points I want to confirm. Gateway recordings I recorded the gateway side with vLLM 0.33.0 ( Three cassettes are committed in
What the gateway does today:
OpenAI side My environment is temporarily inconvenient for recording the OpenAI side, so I could not record that reference set. Could you run the same recording on your side? I will match the gateway to those recordings, including whether the continuation after deletion still succeeds and the exact status, code, and message for missing files and wrong file types. Before I rework the pipeline
Next I will remove |
Replay the recorded Files API and Responses file_id lifecycles through the HTTP gateway, remapping recorded file and response IDs and asserting the hydrated model input, stored client references, and error mapping. Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
Thanks for recording the gateway side; the way you recorded it is correct. I will record the OpenAI references and push them to this branch, and add a few more cases, such as continuing from a response that was deleted. Please implement the gateway behavior with the OpenAI recordings as the ground truth, including the status, code and message for a missing or deleted file and for a wrong file type; the gateway's current
The rest of the plan looks right. For the size check, reuse the existing Side note: while implementing, please make sure you fully understand the existing functionality and reuse the code that is already there rather than adding new blocks. If the existing code lacks something, or isn't extendable or maintainable, a small refactor of it is better than introducing new wrappers or workarounds. |
Signed-off-by: maral <maralbahari.98@gmail.com>
|
@LeaderOnePro
Please record the gateway side of the new scenarios the same way you recorded the first three, and match the gateway to these references: What the OpenAI contract shows
So the reference stays in history and is resolved again each time the model needs it, including in compacted windows; nothing resolved is kept. The gateway's current The design to follow flowchart TD
A["Request"] --> B["RequestPayload input.validate()<br/>file_id is valid input in the single validate<br/>mutually exclusive fields: 400 as recorded"]
B --> C["rehydrate_with_continuation, branches unchanged<br/>conversation, previous response, session parent,<br/>stateless input, restored agent tree"]
C --> D["enriched_request.input: client-facing history<br/>input_file and input_image keep file_id"]
D --> E["exec_ctx.file_handler.resolve(final input and restored agent histories)<br/>one call at the end of rehydration<br/>missing ids: 404, wrong type: 400, as recorded<br/>resolved bytes count against max_request_body_size_bytes"]
E --> F["RequestContext.files: per-turn lookup, never persisted or retained"]
F --> G["upstream_request, for every round, agent and compaction summary<br/>to_upstream_request and normalized_model_input with the lookup"]
G --> H["Lowering, on the existing file_data path<br/>input_file file_id to input_text 'File: name ...'<br/>input_image file_id to an image data URL<br/>the same for function_call_output content"]
H --> I["Upstream model"]
D --> J["Persistence unchanged: checkpoint, conversation items,<br/>compaction window and tree snapshot keep file_id"]
K["executor/modes/files.rs: FileHandler<br/>wraps FileSearchService, owned by ExecutionContext"] -.-> E
With this in place, please remove |
Resolve Responses input_file and input_image file_id references from the Files API once per turn, after history assembly and before inference, and use the lookup only for the model-facing projection. Stored history, compaction windows, and multi-agent tree snapshots keep the client's reference, following the single typed path from RFC vllm-project#500. - Accept file_id references in messages and structured tool outputs and reject a part that also carries its inline field with OpenAI's mutually_exclusive_parameters envelope. - Return OpenAI's missing-file and wrong-type envelopes, and count the resolved escaped size toward the request body limit. - Accept the vision upload purpose and drop the unused FileContent API. - Record the Files API and file_id reference cassettes and replay them through the HTTP gateway. Refs vllm-project#356 Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
Bring in model capability validation (vllm-project#512), unknown conversation id rejection (vllm-project#531), persisted-turn documentation (vllm-project#533), and the shared input-item test helper (vllm-project#545). Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
maralbahari
left a comment
There was a problem hiding this comment.
Thanks for the rework. Validation, the FileHandler seam, lowering through the file_data path and keeping file_id in every stored shape all match what we agreed, and the old rehydration step and validation mode are gone.
Seeing the per-turn lookup implemented, I'd like to change one part of what I suggested. As a RequestContext field it is set to HashMap::default() at every literal (21 sites, plus 64 test calls passing an empty map), copied by hand into AgentContext, CompactionPlan, summarize_items and three sub-request contexts, and forced into the split context in agentic-llm-d. Because an empty map can also mean "not resolved", the lowering needs a new internal error class to catch a skipped stage. The file_id is already part of the checkpoint through the stored history rows, session history, compaction windows and tree snapshots, and the resolved bytes must stay out of all of them, so they should not be a context field either.
Please build the lookup where the upstream body is built instead: make upstream_request async, call exec.file_handler.resolve there on the input it lowers, and pass the result to to_upstream_request. Keep the early 404/400/413 by doing the same in the existing preflight in execute.rs::prepare. Then RequestContext, rehydration, the multi-agent contexts and compaction need no change, and referenced_items, UnresolvedFileId and is_internal can go.
Two more things before this can merge:
- Gateway recordings for
files-api,files-input,files-searchandfiles-compact, with replay tests against the OpenAI references. The Files endpoint rows (download by purpose, unknown id, upload rules, the delete object) are not implemented yet. - Please keep the existing
InvalidInputmessage, code and param unchanged except for the new mutually exclusive case; other input errors change without recordings now.
I'll decide separately on the deleted input_image continuation (OpenAI 200, gateway 404); keep the 404 and the note for now.
Move the per-turn Files API lookup out of `RequestContext` and rehydration. `upstream_request` is now async and resolves the references in the input it lowers, and `execute.rs::prepare` repeats that lookup so a missing file, a wrong file type, or an oversized body still fails before tool preparation. - Share one `file_references` collector between the resolver and split- execution detection, and remove `file_references.rs`, `ReferenceKind`, `is_internal`, and `UnresolvedFileId`. - Check a resolved file's kind only where it is lowered, mapping the recorded wrong-type envelope through one `From<InputValidationError>`. - Keep the file-kind rules in `file_text/kinds.rs` and keep one file-service handle on `ExecutionContext`; remove the test-only service setters. - Carry `max_request_body_size` on the core `Config`, read by `from_config`. - Implement the recorded Files API rows: upload purposes and vision media validation, download by purpose, unknown-id envelopes for lookup, content, delete, and list cursors, and the `object: "file"` delete row. - Record the files-api, files-input, files-search, and files-compact gateway cassettes, re-record files-image with the vision model, and compare every gateway Files row with the OpenAI reference. Refs vllm-project#356 Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
maralbahari
left a comment
There was a problem hiding this comment.
Thanks, this is much closer. The lookup is off RequestContext and built with the upstream body, rehydration is back to main, and the gateway recordings are in. What's left is mostly duplication, and removing it also makes conversions.rs and resolved_file.rs much smaller.
1. normalized_model_input can stay close to main.
Most of its new code is not the lookup itself:
- File failures live in two error types,
ResolvedFileErrorandInputFileError. Every call site merges them through afile_error(error, |reason| ...)callback, written out four times. - The conversions are free functions (
resolved_file,file_error,lower_tool_output) next to the types that own them:InputFileContent,InputImageContentandToolCallOutput. - A missing lookup entry hits an
expectin a production path.
With one error type and the conversions on their owners, each branch is one call again:
InputContent::InputFile(file) => {
let text = file.to_text(files).map_err(|reason| InputValidationError::unsupported_file(model.len(), part_index, reason))?;
*part = InputContent::InputText(text);
}
InputContent::InputImage(image) => {
image.detail.get_or_insert_with(|| "auto".to_owned());
image.inline_file(files).map_err(|reason| InputValidationError::unsupported_file(model.len(), part_index, reason))?;
}
InputItem::FunctionCallOutput(mut output) => {
output.output.inline_files(files, model.len())?;
model.push(InputItem::FunctionCallOutput(output));
}InputFileContent::to_text(&files)covers inlinefile_dataandfile_idin one place, replacingInputTextContent::try_from(&InputFileContent)for both.InputImageContent::inline_file(&files)andToolCallOutput::inline_files(&files, item_index)replacelower_tool_output.- A missing entry returns an error instead of panicking.
- The two recorded wrong-kind messages become
InputFileErrorvariants, reusing theNotImagethis PR adds but never constructs.unsupported_filegives them the recorded envelope (param: input, no position).
That removes ResolvedFileError, InputValidationError::resolved_file and resolved_file_message, the callback, the three free functions and the expect. resolved_file.rs then only needs the file's name and bytes, which FileObject and file_content already give.
2. New objects restate ones that already exist.
| New in this PR | Already exists | Please |
|---|---|---|
ResolvedFile { filename, bytes } |
FileObject (its filename) plus the bytes from file_content |
use them |
ResolvedFileError::{NotImage, NotText, Content} |
InputFileError::NotText. This PR also adds InputFileError::NotImage, which nothing constructs |
one error: InputFileError |
InputValidationError::resolved_file and resolved_file_message |
InputValidationError already reports InputFileError |
remove |
data_url_len, text_json_len and the size loop in FileHandler::resolve |
the upstream body that upstream_request already serializes |
check the limit on that body |
ReferenceRead { Missing, Failed } |
FileSearchError::NotFound versus the rest |
match FileSearchError |
FilesApiError, FilesApiErrorEnvelope, files_api_error, file_not_found, file_row |
FileSearchError → ExecutorError → executor_error_response, and the OpenAI envelope in error_response.rs |
extend that path |
| the purpose check in the HTTP handler | the same check and message in upload_file |
keep the service's |
"NotFound or InvalidRequest → No such File object", four times in the HTTP handler |
the service's NotFound |
have the service return NotFound for an unusable ID |
VISION_CONTENT_TYPES |
kinds::image_media_type, with the same five formats |
one list in kinds.rs |
Two more points:
validate_tool_outputvalidates tool-output file parts, but the OpenAI recordings have no such rejection.files-api-inputt21 to t24 are success and a missing file only, and tool outputs are not validated onmain. Please remove it.FileHandler<'a>borrowsExecutionContext.file_search. Please make it a plain handler likeConversationHandlerandResponseHandler, owning the service throughFileHandler::new(service)and replacingfile_search, so there is still one handle. Until the follow-up below, the HTTP handler can reach the service through it.
Follow-up PR. The /v1/files endpoints still call FileSearchService straight from the HTTP handler. Conversations and responses go through their mode handler: the transport calls exec_ctx.conv_handler or exec_ctx.resp_handler, which calls the store. Once this PR is in, please open a follow-up that gives FileHandler the endpoint methods and has the HTTP handlers call them:
impl FileHandler {
pub async fn upload(&self, ...) -> ExecutorResult<FileObject> { ... }
pub async fn list(&self, params: &ListParams) -> ExecutorResult<ListResponse<FileObject>> { ... }
pub async fn retrieve(&self, id: &str) -> ExecutorResult<FileObject> { ... }
pub async fn content(&self, id: &str) -> ExecutorResult<Vec<u8>> { ... }
pub async fn delete(&self, id: &str) -> ExecutorResult<DeleteObject> { ... }
}Done when:
- there is one wrong-kind error, on
InputFileError; normalized_model_inputhas one call per file branch, with no callback, free function orexpect;ResolvedFileErrorand the size estimators are gone;- there is one Files error envelope;
validation.rsdoes not validate tool outputs;FileHandlerowns the one file service.
|
@maralbahari Thanks — agreed on all of these. The remaining work is the deduplication pass you described; I'll keep the behavior pinned by the recordings and replay tests. Planned changes:
I'll also remove the stray blank line in One extra: the CI Python client test still expects I'll open the |
Resolve every file_id reference through one placement path and report Files API row failures with the recorded OpenAI envelope: - Lower a resolved message part in place: InputImageContent::inline_file and InputFileContent::to_text own the image and text conversion, and ToolCallOutput::inline_files does the same for structured tool output. Drop ResolvedFile, ResolvedFileError, the size estimators and the resolve-time size loop; upstream_request checks the serialized body against max_request_body_size when it carries resolved content. - FileHandler now owns the executor's one FileSearchService in ExecutionContext, so request execution and the /v1/files handlers use the same handle. prepare() builds the model-facing body before tool preparation, keeping early 404, 400 and 413 failures. - FileSearchService reports any unusable row ID as FileNotFound and validates uploads itself through FileRowRejected; ExecutorError renders both with invalid_request_error, code null and param id/after/before. The vision format list comes from the shared IMAGE_MEDIA_TYPES constant. - Message validation no longer inspects structured tool outputs; their file_id lowering reports through the output position. Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
An assistants upload cannot be downloaded: OpenAI rejects GET
/v1/files/{id}/content for every purpose but vision, and the gateway
now reproduces that row. Assert the recorded 400 message and add a
vision PNG upload, download and delete roundtrip.
Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
Document that FileHandler owns the executor's one file service, that the model-facing body is size-checked after serialization, and that Files API rows use the recorded invalid_request_error envelope. Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
A resolved file whose kind does not match a structured tool-output part reported a positioned `invalid request` error, while the recorded row (files-api-input t22) uses the same bare message, `param: input` and `code: null` as a message part. Route it through the shared resolved-kind envelope, as unsupported_file already does. Also pin what the dedup pass left untested: the serialized-body 413 for resolved content in upstream_request, the to_text / inline_file / inline_files conversions (NotTextFile, NotImage, Unresolved, and positioned content failures), the image media-type mapping behind IMAGE_MEDIA_TYPES, and an HTTP guard that replays the t22 tool-output turn. FileHandler::resolve no longer matches FileSearchError::NotFound: the service maps every unusable lookup to FileNotFound. Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
ExecutionContext::with_file_search_service is gone; embedders wrap their service with FileHandler::new and assign ExecutionContext.file_handler. Update both file-search pages and the field comment. Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
maralbahari
left a comment
There was a problem hiding this comment.
Thanks, this round removes most of what we discussed. The wrong-kind error lives on InputFileError, the conversions are methods on the content types, the expect is gone, and conversions.rs is about 110 lines shorter.
Four new files remain that should not be needed. Their contents belong in existing code:
| New file | Where its contents go |
|---|---|
types/io/input/resolved_file.rs (only the ResolvedFiles alias) |
content.rs, next to to_text and inline_file, which take it |
types/io/input/tool_output.rs (ToolCallOutput::inline_files) |
content.rs, next to the two conversions it calls |
types/file_search/file_row.rs (FILE_PURPOSES, FileRowParam) |
FILE_PURPOSES goes back in types/file_search.rs. In place of FileRowParam, the two Files variants carry the parameter as &'static str, as ExecutorError reports parameters elsewhere |
types/file_search/error.rs (FileSearchError, moved, plus FileNotFound and FileRowRejected) |
FileSearchError, with the two Files variants, goes back in types/file_search.rs. That comes to 497 production lines, under the 500-line limit |
A note on the next revision: several changes in this round satisfy the wording of a review comment rather than its intent. A method we named got its own module, "the parameter" became a new enum, and our description of the OpenAI recordings ("rows") became type names. Please read each comment for what it asks the code to become, and extend the existing types and modules above rather than adding new ones. If the next revision again adds new types, modules, wrappers or names that only answer a comment's wording, we can't move forward with this PR and will close it.
Move the remaining helpers back into the modules that own the types, so there is no standalone module for one alias or one method: - ResolvedFiles and ToolCallOutput::inline_files move into types/io/input/content.rs, next to to_text and inline_file. - FILE_PURPOSES and FileSearchError move back into types/file_search.rs; the two Files variants carry their parameter as a &'static str instead of a dedicated enum, and FileRejected replaces FileRowRejected. - ExecutorError::error_param reports those parameters directly. No behavior changes; types/file_search.rs is 497 production lines, within the 500-line limit. Signed-off-by: LeaderOnePro <LeaderOnePro@outlook.com>
Signed-off-by: maral <maralbahari.98@gmail.com>
maralbahari
left a comment
There was a problem hiding this comment.
Approving with one fix pushed directly (13d348b).
| ctx.enriched_request.input.normalized_model_input(&[])?; | ||
| // Fail before any tool preparation or response event if the model can't be given this input: | ||
| // a missing referenced file returns its JSON 404 and a mismatched file type its 400 first. | ||
| let files = exec.file_handler.resolve(&ctx.enriched_request.input).await?; |
There was a problem hiding this comment.
I noticed a mistake on this line and pushed the fix directly to be faster. It called upstream_request(exec, &ctx, false, None, None), which built a full upstream request with placeholder arguments (no stream, guidance or withheld tools) only to throw it away, and the first round then built it again. The last review didn't mean for that to happen. The fix puts back the previous file resolve and input check here, and the body-size check stays in upstream_request, on the request that is actually sent.
Signed-off-by: maral <maralbahari.98@gmail.com> # Conflicts: # CHANGELOG.md # crates/agentic-server-core/src/executor/request.rs # crates/agentic-server-core/src/executor/upstream.rs
Signed-off-by: maral <maralbahari.98@gmail.com> # Conflicts: # CHANGELOG.md
Signed-off-by: maral <maralbahari.98@gmail.com>
|
@maralbahari Thanks for the approval and for pushing the |
Summary
Responses clients can reference an uploaded file by
file_idinstead of sending inline content:input_fileandinput_imagemessage parts, plus structured function/custom tool-call outputs.This PR resolves those references from the gateway Files API and inlines the content in the
request sent to the model, while stored history keeps the client's reference (#356).
Design, following RFC #500:
file_idand rejects a part that also carries its inline field(
file_data,filename,file_url, orimage_url) with OpenAI'smutually_exclusive_parametersenvelope.executor::upstream_requestis now async. Immediately before lowering, it callsExecutionContext::file_handler().resolveonctx.enriched_request.inputand passes theresulting lookup to
RequestPayload::to_upstream_request, so the lookup and the model-facingbody are built from the same input and cannot get out of step. The lookup is used only by
normalized_model_inputand is never stored, checkpointed, or snapshotted.execute.rs::prepareperforms the same resolve andnormalized_model_inputpass before toolpreparation, so a missing or invalid reference returns its 404/400/413 before any tool work or
response event.
input_textpart (File: <filename>prefix when a filename is known) and an image reference into an inline base64
image_urldataURL. Each distinct file is read once per request even when referenced more than once. Stored
Responses/conversation history, compaction windows, and multi-agent tree snapshots keep
file_id, so each continuation resolves it again against the current file store.file_idfollow the same rules. Other tool-outputparts keep their existing pass-through content contract and are not inspected.
invalid_request_error,code: null,param: null,Files [<ids joined ", ">] were not found. Wrong file types return OpenAI's 400 messages withparam: input. Without a configured file store, the request fails with 400 rather thaninventing content. Resolved content counts toward the existing
max_request_body_size_byteslimit; exceeding it returns 413body_too_large.in_process_feature()reportsfile_id referencesandagentic-llm-drejects such requests instead of dropping content.fine-tune,assistants,batch,user_data,vision, andevals. Avisionupload must use an image media type, andGET /v1/files/{file_id}/contentservesvisionuploads only; another purpose returns 400Not allowed to download files of purpose: <purpose>. The gateway rows reproduce the OpenAIreference rows for upload rules, download policy, unknown-ID envelopes, and the delete object.
RequestPayload::to_upstream_requestnow takes the resolved-file lookup (&ResolvedFiles);embedders calling it directly resolve references first or pass an empty map when the payload
has none.
RequestContext, rehydration, multi-agent contexts, and compaction need no change.CHANGELOG.md, and the cassette recorder workflow are updated with the new behavior andthe new Files API scenarios.
Behavior contract
input_filefile_id→ stored UTF-8 text fileinput_text, prefixedFile: <filename>\nwhen a filename is known; stored item unchangedinput_imagefile_id→ stored.jpeg/.jpg/.png/.gif/.webpfiledata:image/...;base64,...URL;detailand other fields preserved; stored item unchangedfile_idfile_idinvalid_request_error,code: null,param: null,Files [<id1>, <id2>] were not foundinput_file:file_id+file_data/filename/file_urlmutually_exclusive_parameters,param: input[i].content[p],Mutually exclusive parameters: '<param>'. Ensure you are only providing one of: 'file_id' or '<field>'.input_image:file_id+image_url'image_url'param: input[i].output[p]input_imagefile_idresolves to a non-image filenamecode: null,param: input,Invalid input: Expected image type to be a supported format: .jpeg, .jpg, .png, .gif, .webp but got .<ext>.input_filefile_idresolves to a non-text filenamecode: null,param: input,Invalid input: Expected context stuffing file type to be a supported format: <OpenAI list> but got .<ext>.invalid_request_error, message<location>: input_file with file_id requires the gateway file store; configure file storage or send the content inlinemax_request_body_size_bytes(default 10 MiB)body_too_large,resolved file_id references exceed the <N>-byte request body limitfile_id references is not supported for split executionresolvecollects references in request order with one shared collector, reads each distinct IDonce, and reports every missing ID in a single 404.
prepareandupstream_requestboth resolveenriched_request.input; the initial pass fails before tool preparation, and each upstream bodyis built with a lookup of exactly the input it lowers.
Intentional limits and divergences
input_fileaccepts UTF-8 text only;file_urlis never fetched. This is existing behaviorand documented.
bytes.
FileObjectexposes no client-visible content type, matching the OpenAI behavior.OpenAI's recording answers with 200 because OpenAI stored the turned-in image media. Under
RFC [RFC]: one typed path for rehydrating and persisting history #500 the gateway stores only client-facing items and re-resolves each turn, so a deleted
file is gone. This divergence is pinned by the image cassette and is intentional.
JSON-escaped length of the
textvalue, including theFile: <filename>prefix. This countsthe bytes the model-facing body actually carries rather than the raw stored bytes.
exists for it.
GET /v1/files?purpose=listing and Files list-parameter validation messagesstay gateway-specific and are excluded from the row comparison; the gateway does not implement
the
purposelist filter.absent IDs produce
Files [...] were not found.Test Plan
The full suite was rerun after merging upstream
mainatda31b7c(typed MCPrequire_approvalpolicy).End-to-end replay through the HTTP gateway
(
crates/agentic-server/tests/file_id_reference_replay_test.rs, 5 tests):(same lifecycle rows, statuses, upload rules, download policy, unknown-ID envelope, delete
object);
recorded status and part position.
The 7 gateway cassettes are paired with the OpenAI references in the parity test:
files-api-text-gateway-Qwen-Qwen2.5-3B-Instruct.yamlfiles-api-errors-gateway-Qwen-Qwen2.5-3B-Instruct.yamlfiles-api-endpoints-gateway-Qwen-Qwen2.5-3B-Instruct.yamlfiles-api-input-gateway-Qwen-Qwen2.5-3B-Instruct.yamlfiles-api-search-gateway-Qwen-Qwen2.5-3B-Instruct.yamlfiles-api-compact-gateway-Qwen-Qwen2.5-3B-Instruct.yamlfiles-api-image-gateway-Qwen-Qwen2.5-VL-3B-Instruct.yamlRecorded through a real gateway + vLLM with
crates/agentic-server-core/tests/cassettes/record_text_file_input_cassettes.sh.Unit and integration coverage also includes: dedupe and aggregated missing IDs, no store,
wrong image/text types, body limit, compaction treating a
file_idimage as meaningful context,restored multi-agent histories plus new input,
in_process_feature, and the existingpass-through tool-output tests
(
control_custom_tool_output_file_is_preserved,control_structured_tool_output_file_is_preserved, all three tool-image continuation tests).