The stateful, agentic API layer for vLLM, written in Rust 🦀
Run OpenAI-grade agentic workloads (Responses and Messages APIs, server-side tools, Codex, Claude Code) on your own GPUs.
vLLM gives you state-of-the-art inference throughput. But real agentic applications need more than raw tokens: they need conversation state, tool-call loops, and multi-turn orchestration. Today, all of that complexity lives in your client code.
Agentic API moves it server-side. It is a Rust-native gateway that sits in front of vLLM and owns the stateful agentic APIs, starting with an OpenAI-compatible Responses API. vLLM is one supported backend, not part of the Agentic API product name. Your application makes one API call and the server handles the rest: state hydration, tool execution, streaming, and continuation.
flowchart LR
C(["🧑💻 Client<br/>Codex · Claude Code · SDKs · curl"]) -->|"📮 <code>POST /v1/responses</code> · <code>/v1/messages</code><br/>🌐 HTTP 📡 SSE 🔌 WebSocket"| A
subgraph A ["⚡ Agentic API (Rust 🦀)"]
direction TB
S["🔄 State hydration<br/><code>previous_response_id</code>"]
T["🛠️ Server-side tools<br/>web search · web fetch · MCP · code interpreter"]
P["💾 Persistence<br/>SQLite · PostgreSQL"]
end
A -->|"🚀 <code>POST /v1/responses</code> · <code>/v1/messages</code><br/>⚙️ stateless 🤝 OpenAI/Anthropic-compatible"| V(["🚀 vLLM core<br/>inference engine"])
classDef client fill:#FFE8B3,stroke:#F59E0B,stroke-width:2px,color:#7C2D12
classDef inner fill:#E0E7FF,stroke:#6366F1,stroke-width:2px,color:#312E81
classDef engine fill:#DCFCE7,stroke:#22C55E,stroke-width:2px,color:#14532D
class C client
class S,T,P inner
class V engine
style A fill:#F5F3FF,stroke:#8B5CF6,stroke-width:2px,color:#5B21B6
linkStyle 0 stroke:#F59E0B,stroke-width:2px
linkStyle 1 stroke:#22C55E,stroke-width:2px
Tip
Point OpenAI Codex at Agentic API and drive it entirely with open models served by vLLM. No OpenAI account required.
- 🔄 Stateful conversations: the server manages history through
previous_response_idor the Conversations API, so clients stop replaying full transcripts. - 🛠️ Server-side tools: an explicit ownership model decides what runs where. Web search (You.com, Brave, Tavily, Serply, Keenable, or SearXNG), web fetch, MCP, file search, and an opt-in embedded code interpreter run on the gateway.
- 🤝 Multi-agent orchestration: one Responses request can spawn and coordinate subagents server-side (details).
- 📡 Every transport: non-streaming HTTP, SSE streaming, and WebSocket with
stream_idmultiplexing. - 🧰 Codex and Claude Code ready: Codex-shaped Responses traffic and Anthropic Messages traffic keep the tool declarations and item shapes each client depends on.
- 🗜️ Compaction: automatic and explicit context compaction through
/v1/responses/compact(guide). - 🔭 Observability: opt-in OpenTelemetry traces and metrics over OTLP (guide).
- ✅ Compatibility tested: validated against the Open Responses suite, with replay-cassette tests for OpenAI, vLLM, SGLang, and NVIDIA Dynamo traffic.
| Endpoint | Description |
|---|---|
POST /v1/responses · GET /v1/responses (WebSocket) |
Responses API with state, tools, and streaming |
GET · DELETE /v1/responses/{id} · GET /v1/responses/{id}/input_items |
Retrieve, delete, and list the input of a stored response |
POST /v1/responses/compact |
Compact direct input or a stored response chain |
/v1/conversations · /v1/conversations/{id}/items |
Conversations API |
POST /v1/messages · POST /v1/messages/count_tokens |
Anthropic Messages API with a server-side loop for gateway-owned tools |
/v1/files · /v1/vector_stores |
Files and vector stores for the file_search tool (guide) |
GET /v1/models · GET /v1/models/{model} |
Model listing from the upstream |
/v1/chat/completions · /v1/completions · /v1/embeddings · /v1/audio/* |
Forwarded to the upstream unchanged |
GET /health · GET /ready |
Liveness and readiness probes |
GET /swagger-ui · GET /openapi.json |
OpenAPI spec, served when ENABLE_OPENAPI_DOCS=true |
Planned: stateful Messages and the Interactions API. See the API reference for routing, authentication, retrieval, and WebSocket details.
1. Serve a model with vLLM. Any recipe from recipes.vllm.ai works:
vllm serve Qwen/Qwen3-30B-A3B-FP8 \
--tool-call-parser qwen3_coder --enable-auto-tool-choice \
--reasoning-parser qwen3 --port 5050NVIDIA Dynamo and SGLang upstreams work too.
2. Start Agentic API in front of it. Web search is optional; pick a provider and set its key:
AGENTIC_WEB_SEARCH_PROVIDER=brave BRAVE_API_KEY=<your-brave-api-key> \
cargo run -p agentic-server -- --llm-api-base http://127.0.0.1:5050Or use the published package instead of building (see Python installation).
3. Make a stateful call:
curl http://localhost:9000/v1/responses \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-30B-A3B-FP8",
"input": "What is new in vLLM this month?",
"tools": [{"type": "web_search"}]
}'Pass the returned id as previous_response_id to continue, and the server rehydrates the history for you.
The agentic CLI starts the gateway and launches a harness with an isolated configuration:
cargo build -p agentic-server --bins
./target/debug/agentic run codex --upstream http://127.0.0.1:5050
./target/debug/agentic run claude --upstream http://127.0.0.1:5050 --model Qwen/Qwen3-30B-A3B-FP8
./target/debug/agentic validate --upstream http://127.0.0.1:5050 --harness codex # preflight checks onlyThe model defaults to the first one the upstream lists. SQLite is the default store; pass --database-url postgresql://... for a shared session. See the quickstart guide and
harness CLI testing, including
Claude Code with a self-hosted model.
The agentic-api package on PyPI bundles the Rust gateway, the
agentic CLI, and a small Python launcher. Wheels cover Linux x86_64 (glibc 2.17+), macOS Intel, and macOS Apple
Silicon on Python 3.10+.
uvx --from agentic-api==0.10.0 agentic --version
uvx --from agentic-api==0.10.0 agentic serve --upstream http://existing-vllm:8000
python -m pip install agentic-api==0.10.0 # remote mode, no vLLM
python -m pip install "agentic-api[local]==0.10.0" # also installs the pinned vLLM runtimeWith uv:
uv pip install agentic-api==0.10.0
uv pip install "agentic-api[local]==0.10.0"
uvx --from agentic-api==0.10.0 agentic-api doctor
uvx --from agentic-api==0.10.0 agentic-api serve --vllm-base-url http://existing-vllm:8000See Python installation and workflows for managed vLLM, doctor output, and
known-good model profiles.
Agentic API speaks the Responses wire protocol Codex expects, including WebSockets. Add a provider to
~/.codex/config.toml:
[model_providers.agentic-api]
name = "OpenAI"
base_url = "http://localhost:9000/v1"
wire_api = "responses"
requires_openai_auth = false
supports_websockets = truecodex --disable image_generation -c model_provider=agentic-api -m Qwen/Qwen3-30B-A3B-FP8For the desktop app, see Codex Desktop with local models. With OIDC enabled, replace
requires_openai_auth = false with Codex's command-backed bearer authentication; see
GitHub authentication with Dex.
Agentic API serves the Anthropic Messages protocol at /v1/messages, so Claude Code (CLI or Agent SDK) runs against
open models:
export ANTHROPIC_BASE_URL="http://localhost:9000"
export ANTHROPIC_API_KEY="<your-key>"
export ANTHROPIC_MODEL="Qwen/Qwen3-30B-A3B-FP8" # match the served model
claude -p "summarize the files in this directory"Claude Code's own tools stay client-owned. Its native web search and web fetch tools run on the gateway; see
Gateway server tools on /v1/messages. With OIDC enabled, send the token in
ANTHROPIC_AUTH_TOKEN and leave ANTHROPIC_API_KEY unset (GitHub authentication with Dex).
Agentic API reads ~/.agentic-api/config.toml and generates it on first run. CLI arguments and environment variables
take precedence over the file, so secrets can stay in the environment:
llm_api_base = "http://127.0.0.1:5050"
[web_search]
provider = "brave"- Configuration: storage, request and rate limits, model capabilities, response limits, and MCP servers.
- Responses capability policies: per-model parameter limits, prompt-cache controls, and service tiers.
- Web search providers: provider settings and how each one maps the shared tool contract.
- Embedded code interpreter: the opt-in
embedded-code-interpreterbuild feature. - Deploying: containers, Kubernetes, and observability.
Every tool call has exactly one execution path, so nothing runs by accident:
| Ownership | Who executes it | Examples |
|---|---|---|
| Gateway-owned | Agentic API, which then continues the loop | Web search, web fetch, MCP, file search, code interpreter (opt-in) |
| Client-owned | Returned to the client | Shell tool, Codex editor tools, your functions |
| Provider-owned | Passed through to vLLM or an upstream provider | Provider-native tools |
Unknown or ambiguous tool shapes are never executed by default. They are preserved and returned.
Set multi_agent.enabled and the gateway runs the
Responses multi-agent loop itself: the root
agent spawns subagents, each with its own history and tools, and the response returns their attributed
multi_agent_call, multi_agent_call_output, and agent_message items.
curl http://localhost:9000/v1/responses \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-30B-A3B-FP8",
"store": true,
"multi_agent": {"enabled": true, "max_concurrent_subagents": 3},
"input": "Have one agent review correctness and another review test coverage, then summarize."
}'It works over HTTP, SSE, and WebSocket. Continue a stored agent tree with previous_response_id; with store: false,
send the complete item history on each request. See the API reference and
multi-agent execution for the details and limits.
crates/
├── agentic-server/ # Axum binary, transport handlers (HTTP/SSE/WS), configuration, `agentic` CLI
├── agentic-server-core/ # Protocol types, executor, tool framework, persistence
├── agentic-llm-d/ # Split-execution state backend for the llm-d coordinator
├── agentic-cli-docs/ # Build-time generator for the CLI reference docs
└── agentic-praxis/ # Praxis gateway integration
python/agentic_api/ # Python distribution and launcher for the packaged gateway
docs/ # MkDocs documentation, ADRs, and design notes
cargo build # build
cargo test # test
cargo clippy --all-targets -- -D warnings # lint
cargo fmt -- --check # format check
uv run mkdocs serve # docs (after: uv pip install -r docs/requirements.txt)Read ARCHITECTURE and CONTRIBUTING before opening a PR. Design decisions live in docs/adr/ and docs/design/.
- Responses API hydration: stateful continuation with
previous_response_id - Codex support: practical Codex sessions through the Responses API
- Server-side tool execution: explicit ownership, web search, web fetch, MCP, and code interpreter built in
- Messages API: Claude Code support with a server-side gateway tool loop
- Conversations API: OpenAI-compatible conversation and item management
- Multi-agent Responses: stateful and stateless execution over HTTP and WebSocket
- File search: Files and vector-store APIs with a gateway-executed
file_searchtool - Stateful Messages: Messages continuation on the same persistence and execution primitives
- Interactions API: durable, higher-level agentic workflows
- Production hardening: tenant-scoped state and cached-prefix continuation
Licensed under the Apache License 2.0.
⭐ If Agentic API saves you from writing one more client-side tool loop, star the repo! ⭐