Skip to content

fix: refuse a conversation turn that resends a stored item before inference - #562

Open
Baiiiiiiiiii wants to merge 1 commit into
vllm-project:mainfrom
Baiiiiiiiiii:fix-conversation-resent-items
Open

Baiiiiiiiiii wants to merge 1 commit into
vllm-project:mainfrom
Baiiiiiiiiii:fix-conversation-resent-items

Conversation

@Baiiiiiiiiii

Copy link
Copy Markdown

Summary

A client that keeps its whole history and passes conversation resends items the conversation already holds, as in the function-calling guide's loop. The gateway ran inference, then failed with 500 failed to persist response (#561). Now from_conversation refuses such a turn before inference, with the duplicate-input error the gateway already returns for an id repeated in one request. ARCHITECTURE.md states the rule.

Changes by API

API What changed
Responses (/v1/responses) A conversation turn whose input repeats one of the conversation's items returns 400 invalid_request_error, param: "input", code: null, "Duplicate item found with id …", before inference. Before: 500 after inference; a stream sent the answer, then an error event. A store: false turn like this used to complete, with the item twice in the model's input; it now gets the same 400.

Test Plan

  • Evidence. Recorded in response-replayed-items-openai.yaml: t6 is OpenAI's 400 for an id repeated in one input, the error this PR returns; t3 is OpenAI's 200 for a stored item sent back without conversation, which this PR does not change. Documented: the conversation reference says the conversation's items "are prepended to input_items". Quoted, not recorded: openai-agents-python#1789 quotes the same 400 for this request in September 2025.

  • No new recording: I have no OpenAI API key. The recording this needs: extend the replayed-items scenario in record_cassette.py with a conversation turn that calls a function tool, then the resend (user message, the stored function_call, its output), once with store: true and once with store: false. Then run RESPONSES_RECORD_SET=all RESPONSES_SCENARIO=replayed-items bash crates/agentic-server-core/tests/cassettes/record_responses_api_cassettes.sh. I did not see OpenAI's reply for store: false. If the recording differs, I will match the gateway to it.

  • conversation_turn_resending_a_stored_item_is_refused_before_inference, 3 cases. Each resends the stored function call with its output. It checks the 400 body, zero inference requests and an unchanged conversation. Then it sends only the output and gets the answer.

    Case main this branch
    store: true 500 failed to persist response, after 1 inference request 400, no inference request
    store: true, stream: true 200, then the answer, then an error event, after 1 inference request 400, no inference request
    store: false 200, after 1 inference request 400, no inference request

    On main, the model's input held the resent item twice in all three cases.

  • conversation_turn_accepts_items_stored_in_another_conversation: the same items sent to a new conversation still complete.

  • cargo test --locked: 3172 passed, 0 failed. cargo clippy --all-targets -- -D warnings: clean. pre-commit run --all-files: passed.

…erence

Signed-off-by: Baiiiiiiiiii <2002paipai@gmail.com>
// The conversation's items are prepended to the input, so resending one of them repeats it in the input.
let new_ids: HashSet<&str> = ctx.new_input_items.iter().filter_map(InOutItem::id).collect();
if let Some(id) = items.iter().filter_map(InOutItem::id).find(|id| new_ids.contains(id)) {
return Err(ExecutorError::duplicate_input_item(id));

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the proper fix is to let exec_ctx.conv_handler.rehydrate to gate the duplication. similar to how exec_ctx.response_handler.rehydrate calls on resolve_input_items that way there is only one single iterator in the history of items. seems like this adds overhead to the rehydration path but if during rehydration we can identify duplicate and avoid it then we can avoid overhead


impl ExecutorError {
/// The error `OpenAI` returns for a model input that holds the item `id` twice.
pub(crate) fn duplicate_input_item(id: &str) -> Self {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there is already an error type specific for duplicate item or conflict error perhaps it can be reused?

@maralbahari

Copy link
Copy Markdown
Collaborator

I took a closer look and noticed a correct solution is to resolve the turn's input against the history it continues inside ResponseHandler::rehydrate, for every history source and only for stored turns, instead of adding a second check in from_conversation.

OpenAI accepts the same resend with store: false, and refuses it with previous_response_id and store: true as well, so the check has to be gated on store and has to cover previous_response_id too. This PR refuses the store: false turn that OpenAI accepts, and a previous_response_id turn that resends an item is still accepted and stored twice.

Suggested changes:

  1. ResponseHandler::rehydrate(ctx: &mut RequestContext, conversation: Option<ConversationSnapshot>) -> ExecutorResult<Option<ResponseCheckpoint>> is the one call in rehydrate_with_continuation. It takes the conversation snapshot, or the session's retained parent, or loads the previous_response_id chain, runs continue_from_compaction, puts the history in checkpoint.history, and then calls resolve_input_items:

    let snapshot = OptionFuture::from(conversation.then(|| exec_ctx.conv_handler.rehydrate(&ctx)))
        .await
        .transpose()?;
    match exec_ctx.resp_handler.rehydrate(&mut ctx, snapshot).await? {
        Some(checkpoint) if conversation => from_conversation(&mut ctx, checkpoint, exec_ctx)?,
        loaded if previous => from_response(&mut ctx, loaded, exec_ctx)?,
        _ => from_responses(&mut ctx)?,
    }
  2. resolve_input_items(items, history) keeps admit_all, the duplicate check within the input and the stored lookup. After the input loop, seen already holds the input's IDs, so the history is scanned once against it. An unstored turn passes no history.

    let repeated = items
        .iter()
        .filter_map(InOutItem::id)
        .find(|id| !seen.insert((*id).to_owned()))
        .or_else(|| history.filter_map(InputItem::id).find(|id| seen.contains(*id)));
  3. Reuse the duplicate error resolve_input_items already returns; it is exactly OpenAI's. ItemAlreadyInConversation and ItemConflict give a different code, param and message, so duplicate_input_item is not needed.

  4. from_conversation and from_response take the checkpoint and move checkpoint.history into the model input with mem::take; they no longer load or resolve anything. from_response hands a session turn to from_session_response, and commit() calls rehydrate(&mut ctx, None).

  5. Tests: a unit test in executor/rehydrate/tests.rs for conversation and previous_response_id with store: true (400, no inference) and store: false (accepted). The recordings extend the replayed-items scenario with these four cases for OpenAI and the gateway.

@franciscojavierarceo

Copy link
Copy Markdown
Collaborator

the new conversation duplicate check returns the same 400 for store: true and store: false, but the OpenAI recordings do not cover a turn that resends an item already stored in that conversation. the PR notes that the store: false response is still unknown, so we cannot confirm this wire behavior yet. we should record both forms against OpenAI and the gateway with the existing replayed-items scenario, then assert the status and error envelope before approving the new rejection.

@Baiiiiiiiiii

Copy link
Copy Markdown
Author

Thank you for the detailed design! I'll rework the PR as you suggested:

  • ResponseHandler::rehydrate loads the history the turn continues, whatever the source.
  • resolve_input_items checks the input against that history, only for stored turns.
  • It reuses the existing duplicate error.
  • Tests cover conversation and previous_response_id, each with store: true and store: false.

I'll also add the four cases to the replayed-items scenario and record them against OpenAI and the gateway. For the gateway I only have a T4 GPU, so I'd record with a small model such as Qwen/Qwen3-4B. Does that work?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants