Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions agents/hermes-agent.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,22 @@ Hermes keeps its configuration in `~/.hermes/`:

You can point Hermes at any [model from our catalog](https://deepinfra.com/models/text-generation) — for example `deepseek-ai/DeepSeek-V4-Flash` — either by editing `config.yaml` or interactively with `hermes model`. The agent's model calls are billed as regular inference usage against your account.

## Updating

You update Hermes yourself, whenever you want — DeepInfra doesn't update it for you. New versions are listed on the [Hermes releases page](https://github.com/NousResearch/hermes-agent/releases).

Click **Update** on the instance to get the exact commands, with copy buttons. Connect over SSH, then run:

```bash
git -C /opt/hermes/src remote set-branches origin main && hermes update
```

The `git` part makes the install track upstream's `main` branch, which is what `hermes update` pulls. The update takes a few minutes and keeps your settings and chat history. Since it brings in the latest upstream code, consider [creating a backup](/agents/introduction#backups) first so you can roll back.

<Note>
The version shown in the dashboard is the one DeepInfra installed, and it doesn't change when you update Hermes yourself. Run `hermes --version` on the instance to see what you're running.
</Note>

## FAQ

**Why is there no dashboard link for my instance?**
Expand Down
4 changes: 3 additions & 1 deletion agents/introduction.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ Stopping an instance shuts it down so you're not billed for compute while it's i
From the instance list or its details page you can:

- **Start / Stop** — pause compute billing without losing state.
- **Update** — when a newer framework version is available, upgrade in place. No restart needed; the agent may be unresponsive for a few minutes.
- **Update** — for OpenClaw, when a newer version is available, upgrade in place. No restart needed; the agent may be unresponsive for a few minutes. For Hermes-Agent, **Update** shows the commands to update it yourself over SSH, whenever you want (see [Updating Hermes-Agent](/agents/hermes-agent#updating)).
- **Backup** — trigger a snapshot on demand (see below).
- **Delete** — stop and permanently remove the instance. This cannot be undone.

Expand Down Expand Up @@ -132,6 +132,8 @@ curl -X POST https://api.deepinfra.com/v1/agents/{instance_id}/update \
-H "Authorization: Bearer $DEEPINFRA_API_KEY"
```

`/update` applies to OpenClaw. Hermes-Agent instances are [updated over SSH](/agents/hermes-agent#updating) instead, so for them `/update` returns `409` with the command to run.

Delete:

```bash
Expand Down
86 changes: 25 additions & 61 deletions integrations/crewai.mdx
Original file line number Diff line number Diff line change
@@ -1,86 +1,54 @@
---
title: CrewAI
description: Run CrewAI agents on DeepInfra models through the OpenAI-compatible API, embed knowledge with DeepInfra embedding models, and execute agent code in DeepInfra Sandboxes.
description: Run CrewAI agents on DeepInfra models through CrewAI's built-in DeepInfra provider, embed knowledge with DeepInfra embedding models, and execute agent code in DeepInfra Sandboxes.
icon: people-group
---

[CrewAI](https://docs.crewai.com/) is a Python framework for building crews of role-playing agents that collaborate on tasks. Its `LLM` class ships a native OpenAI client that accepts a custom base URL, so it talks to DeepInfra directly with no LiteLLM dependency. Tool calling, streaming, JSON-schema structured outputs and token usage reporting all work over this path, with any [chat model from our catalog](https://deepinfra.com/models/text-generation).
[CrewAI](https://docs.crewai.com/) is a Python framework for building crews of role-playing agents that collaborate on tasks. DeepInfra is one of its built-in providers: prefix a model id with `deepinfra/` and CrewAI sends the requests through its native OpenAI client, reading your key from `DEEPINFRA_API_KEY`, with no base URL to configure and no LiteLLM dependency. Tool calling, streaming, JSON-schema structured outputs and token usage reporting all work over this path, with any [chat model from our catalog](https://deepinfra.com/models/text-generation).

## Installation

```bash
pip install crewai
pip install "crewai>=1.15.27"
```

Get your API key from the [Dashboard](https://deepinfra.com/dash/api_keys) and export it as `DEEPINFRA_API_KEY`. It is the same key you use for inference, embeddings and Sandboxes.
The DeepInfra provider shipped in `crewai` 1.15.27 and needs no extras. Get your API key from the [Dashboard](https://deepinfra.com/dash/api_keys) and export it as `DEEPINFRA_API_KEY`. It is the same key you use for inference, embeddings and Sandboxes.

```bash
export DEEPINFRA_API_KEY="<your DeepInfra API key here>"
```

## Configuration

Create the LLM with `custom_openai=True` and DeepInfra's base URL. The model id is passed through unchanged, so use the exact `org/model` id from the catalog.
Pass the model id with the `deepinfra/` prefix. DeepInfra ids already have the form `org/model`, so a full reference has three segments; everything after the prefix reaches the API unchanged, so use the exact id from the catalog.

```python
import os

from crewai import LLM

llm = LLM(
model="deepseek-ai/DeepSeek-V4-Flash-0731",
custom_openai=True,
base_url="https://api.deepinfra.com/v1/openai",
api_key=os.environ["DEEPINFRA_API_KEY"],
temperature=0.2,
)
llm = LLM(model="deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731", temperature=0.2)
```

`custom_openai=True` selects CrewAI's native OpenAI client and skips the check against OpenAI's own model list, which is what lets a DeepInfra model id through. `base_url` is required when you set it. Pass the `llm` object to every `Agent` that should run on DeepInfra; different agents in one crew can use different models.
Pass the `llm` object to every `Agent` that should run on DeepInfra; different agents in one crew can use different models. Because the whole configuration fits in the model string, it also works wherever CrewAI takes a model name instead of an `LLM` object, including `agents.yaml`:

<Note>
Agents defined in `agents.yaml` take `llm` as a plain string, which cannot carry a base URL. In a `@CrewBase` class, build the `LLM` in a method decorated with `@llm` and reference the method name from YAML; CrewAI resolves the name to the object when it loads the agent.

```python
from crewai.project import CrewBase, agent, crew, llm, task


@CrewBase
class ResearchCrew:
@llm
def deepinfra_llm(self) -> LLM:
return LLM(
model="deepseek-ai/DeepSeek-V4-Flash-0731",
custom_openai=True,
base_url="https://api.deepinfra.com/v1/openai",
api_key=os.environ["DEEPINFRA_API_KEY"],
)
```
```yaml
researcher:
role: Researcher
goal: Find and summarise the facts the task asks for
backstory: A careful analyst who cites what the tools return.
llm: deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731
```

```yaml
researcher:
role: Researcher
llm: deepinfra_llm
```
</Note>
To route requests through a proxy or gateway, set `DEEPINFRA_BASE_URL`; passing `base_url` and `api_key` to `LLM` directly works as well.

## Build a crew

The example below gives an agent a tool and runs a single task. The agent calls the tool through OpenAI-style function calling, receives the result, and writes the answer.

```python
import os

from crewai import Agent, Crew, LLM, Task
from crewai.tools import tool

llm = LLM(
model="deepseek-ai/DeepSeek-V4-Flash-0731",
custom_openai=True,
base_url="https://api.deepinfra.com/v1/openai",
api_key=os.environ["DEEPINFRA_API_KEY"],
temperature=0.2,
)
llm = LLM(model="deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731", temperature=0.2)


@tool("Exchange rate")
Expand Down Expand Up @@ -128,12 +96,7 @@ import os
from crewai import Agent, Crew, LLM, Task
from crewai.knowledge.source.string_knowledge_source import StringKnowledgeSource

llm = LLM(
model="deepseek-ai/DeepSeek-V4-Flash-0731",
custom_openai=True,
base_url="https://api.deepinfra.com/v1/openai",
api_key=os.environ["DEEPINFRA_API_KEY"],
)
llm = LLM(model="deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731")

facts = StringKnowledgeSource(
content=(
Expand Down Expand Up @@ -216,11 +179,11 @@ class DeepInfraSandboxPython(BaseTool):
return f"exit={result.returncode}\nstdout:\n{result.stdout}\nstderr:\n{result.stderr}"
```

Add `DeepInfraSandboxPython()` to an agent's `tools` list like any other tool. A nano sandbox is running within about half a second of the create call, and the context manager terminates it when the call returns, so you pay only for the seconds the code runs. Files the agent needs to keep between calls belong in `/workspace`; to carry state across calls, create one `Sandbox` in the tool's constructor and reuse it instead of the context manager. Note the account limit of five active sandboxes when many agents run in parallel.
Add `DeepInfraSandboxPython()` to an agent's `tools` list like any other tool. A nano sandbox is running within about half a second of the create call, and the context manager terminates it when the call returns, so you pay only for the seconds the code runs. Files the agent needs to keep between calls belong in `/workspace`; to carry state across calls, create one `Sandbox` in the tool's constructor and reuse it instead of the context manager. Note the account limit of 25 active sandboxes when many agents run in parallel.

## Choosing a model

The agent loop depends on reliable tool calling. The models below are verified with CrewAI's function-calling path on DeepInfra; context lengths are from the [model catalog](https://api.deepinfra.com/v1/openai/models?filter=with_meta&sort_by=crewai), which is public and also carries current pricing.
The agent loop depends on reliable tool calling. The models below are verified with CrewAI's function-calling path on DeepInfra; prefix any of them with `deepinfra/` when you pass it to `LLM`. Context lengths are from the [model catalog](https://api.deepinfra.com/v1/openai/models?filter=with_meta&sort_by=crewai), which is public and also carries current pricing.

| Model | Context | Good for |
| --- | --- | --- |
Expand All @@ -232,7 +195,7 @@ The agent loop depends on reliable tool calling. The models below are verified w
| `google/gemma-4-31B-it-turbo` | 262K | Cheap, capable generalist |

<Note>
To compare other models, list the catalog with context windows and pricing via [`/v1/openai/models?filter=with_meta&sort_by=crewai`](https://api.deepinfra.com/v1/openai/models?filter=with_meta&sort_by=crewai). Each entry's `metadata` block has `context_length`, `max_tokens`, and per-million-token `pricing`; the `id` is what you pass as `model`.
To compare other models, list the catalog with context windows and pricing via [`/v1/openai/models?filter=with_meta&sort_by=crewai`](https://api.deepinfra.com/v1/openai/models?filter=with_meta&sort_by=crewai). Each entry's `metadata` block has `context_length`, `max_tokens`, and per-million-token `pricing`; the `id` is what you pass as `model`, after the `deepinfra/` prefix.

```bash
curl -s "https://api.deepinfra.com/v1/openai/models?filter=with_meta&sort_by=crewai" \
Expand All @@ -242,16 +205,17 @@ The agent loop depends on reliable tool calling. The models below are verified w

## Tips

- **Context window.** CrewAI looks model ids up in a table of OpenAI models to size its context management, and assumes 8,192 tokens for ids it does not know. With the default `respect_context_window=True`, an agent on a DeepInfra model starts summarising its conversation at roughly 7,000 tokens even when the model supports far more. Set `respect_context_window=False` on agents that need the full window; CrewAI then sends the full history and surfaces the provider error if a prompt ever exceeds the model's real limit.
- **The `deepinfra/` prefix.** `LLM(model="deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731")` routes through LiteLLM, which reads `DEEPINFRA_API_KEY` from the environment. It needs `pip install "crewai[litellm]"`; without LiteLLM installed, constructing the `LLM` raises an `ImportError`. The `custom_openai=True` form above uses CrewAI's native client and has no extra dependency.
- **Context window.** CrewAI looks model ids up in a table of OpenAI models to size its context management, and assumes 8,192 tokens for ids it does not know, DeepInfra models included. With the default `respect_context_window=True`, an agent on a DeepInfra model starts summarising its conversation at roughly 7,000 tokens even when the model supports far more. Set `respect_context_window=False` on agents that need the full window; CrewAI then sends the full history and surfaces the provider error if a prompt ever exceeds the model's real limit.
- **Model id format.** The part after `deepinfra/` must be `org/model`. A reference without the organisation, such as `deepinfra/DeepSeek-V4-Flash-0731`, raises a `ValueError` when the `LLM` is constructed rather than failing on the first request.
- **Older CrewAI versions.** Before `crewai` 1.15.27 the `deepinfra/` prefix routed through LiteLLM and needed `pip install "crewai[litellm]"`. If you cannot upgrade, create the LLM with `custom_openai=True`, `base_url="https://api.deepinfra.com/v1/openai"` and `api_key=os.environ["DEEPINFRA_API_KEY"]` instead; that form uses CrewAI's native OpenAI client and still works on current versions. On current versions, `is_litellm=True` keeps the LiteLLM path for those who need it.
- **Structured outputs.** Pass a Pydantic model as `response_format` on the `LLM` or as `output_pydantic` on a `Task`. CrewAI sends it as a strict JSON schema, which DeepInfra honours.
- **Streaming.** Set `stream=True` on the `LLM` to receive tokens as they are generated. Usage is still reported on the final chunk.

## Learn more

<CardGroup cols={2}>
<Card title="CrewAI LLM docs" icon="book" href="https://docs.crewai.com/en/concepts/llms">
Every `LLM` parameter, provider routing, and the YAML configuration format.
Every `LLM` parameter, the DeepInfra provider section, and the YAML configuration format.
</Card>
<Card title="Chat Completions" icon="comments" href="/chat/overview">
DeepInfra's OpenAI-compatible endpoint, including [Tool Calling](/chat/tool-calling) and [Structured Outputs](/chat/structured-outputs).
Expand All @@ -265,5 +229,5 @@ The agent loop depends on reliable tool calling. The models below are verified w
</CardGroup>

<Note>
Tested with `crewai` 1.15.21 and `deepinfra` 0.3.0.
Tested with `crewai` 1.15.27 and `deepinfra` 0.3.0.
</Note>
4 changes: 2 additions & 2 deletions sandboxes/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ Every sandbox boots as its own microVM — via Kata Containers running on QEMU/K

## Limits and quotas

- **Active sandboxes** — up to **5 non-stopped sandboxes per account** at a time. `creating`, `starting`, `running`, and `stopping` all count against this; `stopped` doesn't. Going over it returns a `429` (`RateLimitError`, aliased as `TooManySandboxesError` in the SDK) instead of a silent failure.
- **Active sandboxes** — up to **25 non-stopped sandboxes per account** at a time. `creating`, `starting`, `running`, and `stopping` all count against this; `stopped` doesn't. Going over it returns a `429` (`RateLimitError`, aliased as `TooManySandboxesError` in the SDK) instead of a silent failure.
- **Fleet capacity** — sandboxes run on a shared fleet, so a capacity crunch across all customers can occasionally return a `503` (`CapacityError`) on creation even when you're well under your own limit. Retry with backoff.
- **File transfers** — both `fs.write()` and `fs.read()` are capped at 100 MiB per call (see above).
- **Disk** — scales with plan (see the [catalog](#plans-and-pricing) for exact numbers); the container's own disk outside `/workspace` is only a couple GB regardless of plan.
Expand Down Expand Up @@ -128,7 +128,7 @@ Tags are set once at creation — there's no endpoint to update them afterward.
| Sandbox not found (or not yours) | `404` | `NotFoundError` |
| Wrong state for this operation | `409` | `ConflictError` |
| File transfer body over 100 MiB (either direction) | `413` | `ContentTooLargeError` |
| Over the 5-active-sandboxes cap | `429` | `RateLimitError` / `TooManySandboxesError` |
| Over the 25-active-sandboxes cap | `429` | `RateLimitError` / `TooManySandboxesError` |
| Shared fleet capacity exhausted, or creation temporarily disabled | `503` | `CapacityError` |
| Internal error | `5xx` | `InternalServerError` |

Expand Down