Repository navigation
Conversation
| } | ||
|
|
||
| /// Reject continuing from a background response that has not finished. | ||
| pub(crate) fn ensure_continuable(id: &str, snapshot: &ResponsePayload) -> Result<(), BackgroundError> { |
There was a problem hiding this comment.
Cancelled or failed background responses can't be continued or listed
A background response stores none of its input until it completes. After a cancel or a failure, continuing from it with previous_response_id returns 400 and listing its input items returns 409; listing also fails while it is queued or running. OpenAI returns 200 with the original input in these cases. Suggested fix: store the input items when the response is queued.
|
@Zheng-Lu refactoring the storage layer has been done and merge. you can start maintaining this PR now for review. |
|
@Zheng-Lu I converted into drat once everything is upto date open for review again thanks |
a0fea62 to
bda4407
Compare
background: true on POST /v1/responses stores the prepared turn as queued,
with the request's items as a finished turn stores them, and returns at once.
A supervised job runs it on the existing engine path and appends its output
when it finishes. POST /v1/responses/{response_id}/cancel cancels a queued or
running response. Admission, the run slots, and each job's run time are
bounded per instance; instances that share a database own their rows through
a renewable lease, and a stopped instance's rows become failed.
Closes vllm-project#357
Signed-off-by: Zheng Lu <Lz429671594@gmail.com>
Signed-off-by: Zheng Lu <Lz429671594@gmail.com>
Records a completed background response, one cancelled while it runs, the same in a conversation, and store: false. Status polls go to the provider directly, so the recorded turns are fixed. Signed-off-by: Zheng Lu <Lz429671594@gmail.com>
…on timing
The OpenAI background recording shows two differences:
- Cancelling a synchronous or finished response returns 400 with a null
code and param ("Cannot cancel a synchronous response." /
"Cannot cancel a completed response.").
- A background turn's input does not join its conversation while it runs.
The input and output join together when the turn finishes, and a
cancelled turn leaves the conversation unchanged.
The queued input is now stored outside the conversation, under IDs assigned
at enqueue. Finishing adds that input and the output to the conversation and
makes the turn the latest one.
Signed-off-by: Zheng Lu <Lz429671594@gmail.com>
Adds response-background-openai.yaml, recorded against OpenAI with the background suite of record_responses_api_cassettes.sh. The replay runs it through the gateway with a mocked model. The mock answers each prompt with its recorded completion and holds the long prompt until a cancel drops it. The final store: false turn is not replayed: OpenAI queues it, and the gateway still rejects it. Signed-off-by: Zheng Lu <Lz429671594@gmail.com>
Record the background suite against the gateway in front of OpenAI and replay it, checking each status code and error against OpenAI's turn at the same position. The two recordings agree on every turn but the last, the store: false request that OpenAI queues and the gateway rejects. Signed-off-by: Zheng Lu <Lz429671594@gmail.com>
bda4407 to
6101ee6
Compare
maralbahari
left a comment
There was a problem hiding this comment.
@Zheng-Lu we need openai cassettes recording to verify the gateway behavior for this feature. before reviewing PR
…#570) ## Summary A Responses request carrying vLLM's `previous_input_messages` extension was forwarded upstream after rehydration had already built the item history. On gpt-oss, vLLM rendered that history in front of the rehydrated one: a `previous_response_id` turn with a contradictory `previous_input_messages` went from 177 to 204 input tokens and still returned 200. Sent directly, vLLM rejects that combination, but behind the gateway it never sees `previous_response_id`. `enable_response_messages: true` was also forwarded, and the requested `input_messages`/`output_messages` never reached the client. Both are now rejected at admission in `validate_modeled_extensions()`, next to the `top_logprobs` check. Fixes #571; refs #314, vllm-project/vllm#68575. ## Changes by API | API | What changed | | --- | --- | | Responses (`/v1/responses`, HTTP JSON/SSE and WebSocket, split-execution hydration) | Non-empty `previous_input_messages` and `enable_response_messages: true` return `400 invalid_request_error`, `code: "unsupported_parameter"`, with the member as `param`. Absent, `null`, `[]` and `false` are still accepted. | ## Test Plan - Recordings: none added. I cannot record the OpenAI reference (no OpenAI credentials); the [create-response reference](https://developers.openai.com/api/reference/resources/responses/methods/create) defines neither member. The gateway behavior is admission-only (no upstream request) and is covered the same way as `background` and `prompt`. - Extended the existing rejection tests with both members: `test_unsupported_parameters_are_rejected_for_both_store_values` (HTTP envelope, `store` × `stream`), `websocket_rejects_unsupported_parameters_and_keeps_the_connection`, `unsupported_parameters_test` (executor in both stream modes with no upstream call, and `rehydrate_conversation`), and the `unsupported.rs` unit tests (`param`/`code`; `null`/`[]`/`false` accepted). - `cargo fmt --all -- --check` and `cargo clippy --workspace --all-targets -- -D warnings`: pass. - `cargo test --workspace --no-fail-fast`: 3222 passed, 4 failed, 25 ignored; `main` (`8c1cd74`) on the same runner: 3219 passed, 6 failed, 25 ignored. All failures on both are `executor::inference::tests::error_bodies::streaming_error_timeout_*` idle-timeout tests (the branch's 4 are a subset of main's 6); run alone, they pass on both. - `pre-commit run --all-files` and `mkdocs build`: pass. This touches the same rejection-test arrays and docs section as #496; I'll rebase whichever lands second. AI assistance was used for the analysis and the test harness; I reviewed the change and the results. Signed-off-by: Dong Wang <dongw2019@gmail.com>
Summary
A client that sends
background: truetoPOST /v1/responsesgets400today, so long agentic turns must hold aconnection open for their whole run. This PR adds OpenAI's background mode: the gateway stores the response as
queued, returns it at once, and a supervised job runs it on the existing engine path. Clients pollGET /v1/responses/{id}and stop a run with the newPOST /v1/responses/{id}/cancel. Closes #357.Since the first review: rebased onto the storage refactor (#501–#506); the request's input is now stored when the
response is queued, so
input_itemsandprevious_response_idwork while it runs and after it is cancelled or fails(haoshan98's review); and a new OpenAI recording set the cancel errors and when a conversation gains a background
turn's items.
Changes by API
/v1/responses)background: truereturnsqueuedat once;GETreportsqueued/in_progress/completed/incomplete/failed/cancelled; newPOST /v1/responses/{id}/cancel(idempotent; a finished or foreground response returns400with OpenAI's nullcode);input_itemsand continuation see the input from the moment it is queued.backgroundwithstore: false,stream: true, ormulti_agent, andGETwithstream=true/starting_after, return400for now. Full capacity returns503 background_capacity_exceeded; shutdown returns503 server_shutting_down./v1/conversations)/v1/messages)Flow
flowchart LR C[Client] -->|"POST background: true"| A["admit: try_acquire (new)"] A -->|full| E["503"] A --> P["prepare (unchanged)"] P --> Q["persist::enqueue_background (new): row + input items + queued snapshot"] Q -->|"200 queued"| C Q --> S["Supervisor (new): run slot, claim, renew lease"] S --> R["run_blocking (unchanged engine)"] R --> F["persist: TurnOutput::Background (new): hold row, append output"] S -. "cancel / timeout / shutdown / lost lease" .-> X["abort + join, record status"]The database row is authoritative. Every status change is one conditional
UPDATEnaming its writer: the owningjob (
background_ownermatches), a client cancel, or recovery (background_lease_expires_at < now). Terminalstatuses are never left, so two replicas on one database need no coordinator. Each job renews a 30 s lease every
10 s; a stopped instance's rows become
failedat startup or when next read.Concurrency follows AGENTS.md: a semaphore of
max_concurrent + max_queuedpermits taken withtry_acquire(nobuffering, no channels), a second semaphore for run slots, a
TaskTrackerwith childCancellationTokens, and astd::sync::Mutexregistry never held across an await. Stopping a job aborts and joins its task, so the upstreamrequest and gateway tool calls are dropped before the outcome is written.
New modules
executor/background.rs(withjobs.rs,error.rs): admission, leases, cancellation, recovery, and shutdown;never inference. Sits next to
executor/session.rs, the other owner of in-flight turns.executor/engine/background.rs: the driver and supervisor; next toengine/execute.rs, which hands it the request.executor/modes/response/background.rs:ResponseHandler's background operations, keeping the stores behindmodes/.storage/models/background.rs,storage/response/background.rs,storage/types/background.rs: the lifecycle SQL,the store methods, and
BackgroundStatus/QueuedTurn, next to the response model, store, and types.config/background.rs: the three limits, next toconfig/responses.rs.Class diagram
classDiagram class BackgroundJobs { admit() Admission cancel(id) ResponsePayload stop_local(id) recover() usize begin_shutdown() shutdown() } class BackgroundJob { response_id token: CancellationToken run_slot() lease_deadline() } class BackgroundRun { owner created_at queued_items } class TurnOutput { <<enum>> Response Empty Compaction Background (new) } class QueuedTurn { response_id items snapshot metadata claim: Claim } ExecutionContext --> BackgroundJobs BackgroundJobs --> ResponseHandler BackgroundJobs ..> BackgroundJob : admits BackgroundJob ..> BackgroundRun : runs with BackgroundRun ..> TurnOutput : Background ResponseHandler --> ResponseStore ConversationHandler --> ConversationStore ResponseStore ..> QueuedTurn ConversationStore ..> QueuedTurnTest Plan
responses/response-background-openai.yamlandresponses/response-background-gateway.yaml(25 turns each, recorded with the new
backgroundsuite ofrecord_responses_api_cassettes.sh; the gateway oneagainst this branch's release build in front of OpenAI). They cover lifecycle, cancel (queued/running, twice,
finished, foreground, unknown),
input_itemsand continuation while running and after cancel, and a conversationbefore, during, and after a background turn. The two agree on every turn but the last: status codes, statuses, error
type/code/paramand message, and every item listing.responses_api_test::backgroundreplays both throughthe gateway with a mocked model, checking the gateway against OpenAI's turn at the same position. The last turn
(
store: false) is not replayed: OpenAI queues it, and the gateway still returns400.background_responses_test(12): lifecycle;store: false/stream: true/multi_agent→ 400; cancel duringinference and during a gateway tool call; idempotent cancel, finished, foreground, and unknown IDs;
input_itemsand continuation while running and after cancel; delete while running; startup recovery; aconversation that gains a background turn's items only when it finishes, after a turn that finished first, and
none from a cancelled one; MCP credentials never stored.
background_admission_test(5): admission ±1, cancel before start,max_concurrent=2/max_queued=0, timeout,shutdown with running and queued jobs, a job whose lease expired cannot overwrite its row.
background_shutdown_test: SIGTERM against the real binary records the running jobfailed.and the conditional update; recovery past one full batch; queued input listed and output appended; a lost row
stores none of its items.
EXPLAINshowing recovery usesidx_responses_background_active.cargo fmt -- --check,cargo clippy --all-targets -- -D warnings: clean.cargo test: 3,218 passed, 0 failed, 27 ignored.postgres:17-alpine, removed afterwards):background_responses_test12 andbackground_admission_test5 passed withAGENTIC_BACKGROUND_TEST_POSTGRES_URL; withTEST_POSTGRES_URLand--ignored,storage::schema::tests4,postgres_storage_integration6, andpostgres_tenant_isolation_test6passed.
main(19b392c) with only the new test files:background_responses_test0 of 12,background_shutdown_test0 of 1, andbackground_admission_testdoes not compile.queue time at t19, and items without IDs assigned at enqueue at t24 (the conversation's copy gets a new ID).
pre-commit run --all-files: every hook passed, including the production file-size check.Known limitation: retrieval, cancel, and delete look a response up by ID with no principal check (#107); background
responses inherit that.