Skip to content

feat: run background responses on the HTTP Responses API - #496

Open
Zheng-Lu wants to merge 6 commits into
vllm-project:mainfrom
Zheng-Lu:feat/357-background-responses
Open

Zheng-Lu wants to merge 6 commits into
vllm-project:mainfrom
Zheng-Lu:feat/357-background-responses

Conversation

@Zheng-Lu

@Zheng-Lu Zheng-Lu commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A client that sends background: true to POST /v1/responses gets 400 today, so long agentic turns must hold a
connection open for their whole run. This PR adds OpenAI's background mode: the gateway stores the response as
queued, returns it at once, and a supervised job runs it on the existing engine path. Clients poll
GET /v1/responses/{id} and stop a run with the new POST /v1/responses/{id}/cancel. Closes #357.

Since the first review: rebased onto the storage refactor (#501–#506); the request's input is now stored when the
response is queued, so input_items and previous_response_id work while it runs and after it is cancelled or fails
(haoshan98's review); and a new OpenAI recording set the cancel errors and when a conversation gains a background
turn's items.

Changes by API

API What changed
Responses (/v1/responses) background: true returns queued at once; GET reports queued / in_progress / completed / incomplete / failed / cancelled; new POST /v1/responses/{id}/cancel (idempotent; a finished or foreground response returns 400 with OpenAI's null code); input_items and continuation see the input from the moment it is queued. background with store: false, stream: true, or multi_agent, and GET with stream=true / starting_after, return 400 for now. Full capacity returns 503 background_capacity_exceeded; shutdown returns 503 server_shutting_down.
Conversations (/v1/conversations) As in OpenAI's recording, a background turn's input and output join the conversation together when it finishes; a cancelled or failed turn adds nothing.
Messages (/v1/messages) no wire change
Files and vector stores no wire change

Flow

flowchart LR
    C[Client] -->|"POST background: true"| A["admit: try_acquire (new)"]
    A -->|full| E["503"]
    A --> P["prepare (unchanged)"]
    P --> Q["persist::enqueue_background (new): row + input items + queued snapshot"]
    Q -->|"200 queued"| C
    Q --> S["Supervisor (new): run slot, claim, renew lease"]
    S --> R["run_blocking (unchanged engine)"]
    R --> F["persist: TurnOutput::Background (new): hold row, append output"]
    S -. "cancel / timeout / shutdown / lost lease" .-> X["abort + join, record status"]
Loading

The database row is authoritative. Every status change is one conditional UPDATE naming its writer: the owning
job (background_owner matches), a client cancel, or recovery (background_lease_expires_at < now). Terminal
statuses are never left, so two replicas on one database need no coordinator. Each job renews a 30 s lease every
10 s; a stopped instance's rows become failed at startup or when next read.

Concurrency follows AGENTS.md: a semaphore of max_concurrent + max_queued permits taken with try_acquire (no
buffering, no channels), a second semaphore for run slots, a TaskTracker with child CancellationTokens, and a
std::sync::Mutex registry never held across an await. Stopping a job aborts and joins its task, so the upstream
request and gateway tool calls are dropped before the outcome is written.

New modules

  • executor/background.rs (with jobs.rs, error.rs): admission, leases, cancellation, recovery, and shutdown;
    never inference. Sits next to executor/session.rs, the other owner of in-flight turns.
  • executor/engine/background.rs: the driver and supervisor; next to engine/execute.rs, which hands it the request.
  • executor/modes/response/background.rs: ResponseHandler's background operations, keeping the stores behind
    modes/.
  • storage/models/background.rs, storage/response/background.rs, storage/types/background.rs: the lifecycle SQL,
    the store methods, and BackgroundStatus / QueuedTurn, next to the response model, store, and types.
  • config/background.rs: the three limits, next to config/responses.rs.
Class diagram
classDiagram
    class BackgroundJobs {
        admit() Admission
        cancel(id) ResponsePayload
        stop_local(id)
        recover() usize
        begin_shutdown()
        shutdown()
    }
    class BackgroundJob {
        response_id
        token: CancellationToken
        run_slot()
        lease_deadline()
    }
    class BackgroundRun {
        owner
        created_at
        queued_items
    }
    class TurnOutput {
        <<enum>>
        Response
        Empty
        Compaction
        Background (new)
    }
    class QueuedTurn {
        response_id
        items
        snapshot
        metadata
        claim: Claim
    }
    ExecutionContext --> BackgroundJobs
    BackgroundJobs --> ResponseHandler
    BackgroundJobs ..> BackgroundJob : admits
    BackgroundJob ..> BackgroundRun : runs with
    BackgroundRun ..> TurnOutput : Background
    ResponseHandler --> ResponseStore
    ConversationHandler --> ConversationStore
    ResponseStore ..> QueuedTurn
    ConversationStore ..> QueuedTurn
Loading

Test Plan

  • Recordings: responses/response-background-openai.yaml and responses/response-background-gateway.yaml
    (25 turns each, recorded with the new background suite of record_responses_api_cassettes.sh; the gateway one
    against this branch's release build in front of OpenAI). They cover lifecycle, cancel (queued/running, twice,
    finished, foreground, unknown), input_items and continuation while running and after cancel, and a conversation
    before, during, and after a background turn. The two agree on every turn but the last: status codes, statuses, error
    type/code/param and message, and every item listing. responses_api_test::background replays both through
    the gateway with a mocked model, checking the gateway against OpenAI's turn at the same position. The last turn
    (store: false) is not replayed: OpenAI queues it, and the gateway still returns 400.
  • New tests (every await bounded by a timeout):
    • background_responses_test (12): lifecycle; store: false / stream: true / multi_agent → 400; cancel during
      inference and during a gateway tool call; idempotent cancel, finished, foreground, and unknown IDs;
      input_items and continuation while running and after cancel; delete while running; startup recovery; a
      conversation that gains a background turn's items only when it finishes, after a turn that finished first, and
      none from a cancelled one; MCP credentials never stored.
    • background_admission_test (5): admission ±1, cancel before start, max_concurrent=2 / max_queued=0, timeout,
      shutdown with running and queued jobs, a job whose lease expired cannot overwrite its row.
    • background_shutdown_test: SIGTERM against the real binary records the running job failed.
    • Core unit tests (5, 300 ms lease): renewals and a cancel from another instance; lease boundary ±1 through recovery
      and the conditional update; recovery past one full batch; queued input listed and output appended; a lost row
      stores none of its items.
    • Config: defaults and limits ±1 for all three knobs, file → environment precedence.
    • PostgreSQL (ignored, run by CI's postgres job): each writer's transitions, lease boundary, hold-and-finish, and an
      EXPLAIN showing recovery uses idx_responses_background_active.
  • Commands:
    • cargo fmt -- --check, cargo clippy --all-targets -- -D warnings: clean.
    • cargo test: 3,218 passed, 0 failed, 27 ignored.
    • PostgreSQL 17.11 (postgres:17-alpine, removed afterwards): background_responses_test 12 and
      background_admission_test 5 passed with AGENTIC_BACKGROUND_TEST_POSTGRES_URL; with TEST_POSTGRES_URL and
      --ignored, storage::schema::tests 4, postgres_storage_integration 6, and postgres_tenant_isolation_test 6
      passed.
    • Negative control on main (19b392c) with only the new test files: background_responses_test 0 of 12,
      background_shutdown_test 0 of 1, and background_admission_test does not compile.
    • Negative controls for the replay: the previous cancel errors fail it at t4, the input joining a conversation at
      queue time at t19, and items without IDs assigned at enqueue at t24 (the conversation's copy gets a new ID).
    • pre-commit run --all-files: every hook passed, including the production file-size check.

Known limitation: retrieval, cancel, and delete look a response up by ID with no principal check (#107); background
responses inherit that.

}

/// Reject continuing from a background response that has not finished.
pub(crate) fn ensure_continuable(id: &str, snapshot: &ResponsePayload) -> Result<(), BackgroundError> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cancelled or failed background responses can't be continued or listed

A background response stores none of its input until it completes. After a cancel or a failure, continuing from it with previous_response_id returns 400 and listing its input items returns 409; listing also fails while it is queued or running. OpenAI returns 200 with the original input in these cases. Suggested fix: store the input items when the response is queued.

@maralbahari

Copy link
Copy Markdown
Collaborator

@Zheng-Lu can we put this a bit on hold I have found several issues in DB schema that requires massive clean up in storage later I am working on the PRs mentioned in #500 we cant have more features in before fixing these.

@maralbahari

Copy link
Copy Markdown
Collaborator

@Zheng-Lu refactoring the storage layer has been done and merge. you can start maintaining this PR now for review.

@maralbahari
maralbahari marked this pull request as draft November 16, 2026 08:23
@maralbahari

Copy link
Copy Markdown
Collaborator

@Zheng-Lu I converted into drat once everything is upto date open for review again thanks

@Zheng-Lu
Zheng-Lu force-pushed the feat/357-background-responses branch from a0fea62 to bda4407 Compare November 19, 2026 00:58
background: true on POST /v1/responses stores the prepared turn as queued,
with the request's items as a finished turn stores them, and returns at once.
A supervised job runs it on the existing engine path and appends its output
when it finishes. POST /v1/responses/{response_id}/cancel cancels a queued or
running response. Admission, the run slots, and each job's run time are
bounded per instance; instances that share a database own their rows through
a renewable lease, and a stopped instance's rows become failed.

Closes vllm-project#357

Signed-off-by: Zheng Lu <Lz429671594@gmail.com>
Signed-off-by: Zheng Lu <Lz429671594@gmail.com>
Records a completed background response, one cancelled while it runs, the
same in a conversation, and store: false. Status polls go to the provider
directly, so the recorded turns are fixed.

Signed-off-by: Zheng Lu <Lz429671594@gmail.com>
…on timing

The OpenAI background recording shows two differences:

- Cancelling a synchronous or finished response returns 400 with a null
  code and param ("Cannot cancel a synchronous response." /
  "Cannot cancel a completed response.").
- A background turn's input does not join its conversation while it runs.
  The input and output join together when the turn finishes, and a
  cancelled turn leaves the conversation unchanged.

The queued input is now stored outside the conversation, under IDs assigned
at enqueue. Finishing adds that input and the output to the conversation and
makes the turn the latest one.

Signed-off-by: Zheng Lu <Lz429671594@gmail.com>
Adds response-background-openai.yaml, recorded against OpenAI with the
background suite of record_responses_api_cassettes.sh. The replay runs it
through the gateway with a mocked model. The mock answers each prompt with
its recorded completion and holds the long prompt until a cancel drops it.

The final store: false turn is not replayed: OpenAI queues it, and the
gateway still rejects it.

Signed-off-by: Zheng Lu <Lz429671594@gmail.com>
Record the background suite against the gateway in front of OpenAI and replay it, checking each
status code and error against OpenAI's turn at the same position. The two recordings agree on every
turn but the last, the store: false request that OpenAI queues and the gateway rejects.

Signed-off-by: Zheng Lu <Lz429671594@gmail.com>
@Zheng-Lu
Zheng-Lu force-pushed the feat/357-background-responses branch from bda4407 to 6101ee6 Compare November 19, 2026 01:22
@Zheng-Lu
Zheng-Lu marked this pull request as ready for review November 19, 2026 02:33

@maralbahari maralbahari left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Zheng-Lu we need openai cassettes recording to verify the gateway behavior for this feature. before reviewing PR

franciscojavierarceo pushed a commit that referenced this pull request Nov 19, 2026
…#570)

## Summary

A Responses request carrying vLLM's `previous_input_messages` extension
was forwarded upstream after rehydration had already built the item
history. On gpt-oss, vLLM rendered that history in front of the
rehydrated one: a `previous_response_id` turn with a contradictory
`previous_input_messages` went from 177 to 204 input tokens and still
returned 200. Sent directly, vLLM rejects that combination, but behind
the gateway it never sees `previous_response_id`.
`enable_response_messages: true` was also forwarded, and the requested
`input_messages`/`output_messages` never reached the client. Both are
now rejected at admission in `validate_modeled_extensions()`, next to
the `top_logprobs` check. Fixes #571; refs #314,
vllm-project/vllm#68575.

## Changes by API

| API | What changed |
| --- | --- |
| Responses (`/v1/responses`, HTTP JSON/SSE and WebSocket,
split-execution hydration) | Non-empty `previous_input_messages` and
`enable_response_messages: true` return `400 invalid_request_error`,
`code: "unsupported_parameter"`, with the member as `param`. Absent,
`null`, `[]` and `false` are still accepted. |

## Test Plan

- Recordings: none added. I cannot record the OpenAI reference (no
OpenAI credentials); the [create-response
reference](https://developers.openai.com/api/reference/resources/responses/methods/create)
defines neither member. The gateway behavior is admission-only (no
upstream request) and is covered the same way as `background` and
`prompt`.
- Extended the existing rejection tests with both members:
`test_unsupported_parameters_are_rejected_for_both_store_values` (HTTP
envelope, `store` × `stream`),
`websocket_rejects_unsupported_parameters_and_keeps_the_connection`,
`unsupported_parameters_test` (executor in both stream modes with no
upstream call, and `rehydrate_conversation`), and the `unsupported.rs`
unit tests (`param`/`code`; `null`/`[]`/`false` accepted).
- `cargo fmt --all -- --check` and `cargo clippy --workspace
--all-targets -- -D warnings`: pass.
- `cargo test --workspace --no-fail-fast`: 3222 passed, 4 failed, 25
ignored; `main` (`8c1cd74`) on the same runner: 3219 passed, 6 failed,
25 ignored. All failures on both are
`executor::inference::tests::error_bodies::streaming_error_timeout_*`
idle-timeout tests (the branch's 4 are a subset of main's 6); run alone,
they pass on both.
- `pre-commit run --all-files` and `mkdocs build`: pass.

This touches the same rejection-test arrays and docs section as #496;
I'll rebase whichever lands second.

AI assistance was used for the analysis and the test harness; I reviewed
the change and the results.

Signed-off-by: Dong Wang <dongw2019@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support asynchronous/background Responses

3 participants