Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
2722f35
refactor(sandbox): unify declared compute suspension protocol
Sep 30, 2026
5c0b304
test(store): isolate synthetic suspension receipts from worker startup
Sep 30, 2026
4e0331a
test(store): use current protocol in worker recovery fixture
Sep 30, 2026
9d4f833
fix(runtime): validate allocation protocol and use shared idle clock
Sep 30, 2026
3dbcd0b
test(installer): isolate synthetic supplementary groups
Sep 30, 2026
ed9fb3b
test(store): isolate suspension execution configuration fixture
Sep 30, 2026
6656a47
feat(e2b): implement idle suspension through shared provider lifecycle
Sep 30, 2026
9ac1203
test(runtime): assert ingestion clock and refresh activity fence
Sep 30, 2026
2e7b36a
Merge branch 'codex/unified-suspension' into codex/e2b-unified-pause
Sep 30, 2026
fba894c
test(e2b): qualify suspension policy and isolate generation downgrade…
Sep 30, 2026
022875c
test(migrations): pin historical suspension policy in downgrade fixtures
Sep 30, 2026
3a50815
Merge origin/main and adapt suspension checks to execution writer
Sep 30, 2026
1c60b74
Merge origin/main and preserve suspension across module boundaries
Sep 30, 2026
c6239ac
Merge origin/main and move suspension receipts into deployment
Oct 1, 2026
173d120
test(modelconfiguration): use valid caller-owned observation fixture
Oct 1, 2026
11b24ff
Merge origin/main and preserve suspension documentation
Oct 1, 2026
03d36bc
Raise direct runtime capacity to 100 active sandboxes
Oct 3, 2026
bca4102
Configure direct sandbox capacity through Core settings
Oct 3, 2026
c93a19d
fix: suspend idle sandboxes before their first Turn
Oct 3, 2026
20a053c
test: align release checks with sandbox capacity and idle policy
Oct 3, 2026
14199e4
Merge main and preserve shared sandbox capacity settings
Oct 3, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions contracts/agents-api/node-generation-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,20 +44,21 @@ Each operation carries its own arguments and returns the following result on suc
| `command` | `RunCommand` | `command` | `command` |
| `observe` | `Observe` | `observation` | `sample` |
| `initial` | `Initial` | None | `compute` |
| `new_compute` | `NewCompute` | Positive compute `generation` and optional `snapshot` | `compute` |
| `new_compute` | `NewCompute` | Positive `generation` and optional `retained` | `compute` |
| `compute` | `GetCompute` | `compute` | `state` |
| `renew_compute` | `RenewCompute` | Exact current `compute` | `state` |
| `kill_compute` | `KillCompute` | `compute` | None |
| `resume_compute` | `ResumeCompute` | `compute` | `state` |
| `command_compute` | `RunCommandCompute` | `compute` and `command` | `command` |
| `suspend` | `Suspend` | `suspend` | `state` |
| `resume` | `Resume` | `resume` | `state` |
| `delete_snapshot` | `DeleteSnapshot` | `snapshot` | None |
| `delete_retained` | `DeleteRetained` | `retained` | None |

A request whose `connection_id`, `owner_epoch` or `sequence` does not match closes the connection. A malformed request gets an `invalid` response. A node without generation management accepts only its enrolled `deployment_generation`; a generation-managing node runs the request on that generation's provider and answers `unconfirmed` when it cannot. Core sends `create` and a `resume` that is not observe-only only to a generation that is ready on that node, and keeps at most 32 requests pending per connection.
A request whose `connection_id`, `owner_epoch` or `sequence` does not match closes the connection. A malformed request gets an `invalid` response. A node without generation management accepts only its enrolled `deployment_generation`; a generation-managing node runs the request on that generation's provider and answers `unconfirmed` when it cannot. Core sends `create` and a `resume` that is not reconciliation-only only to a generation that is ready on that node, and keeps at most 32 requests pending per connection.

The budget is relative: the node anchors `timeout_ms` to its own clock on receipt and consumes it while the request waits in its queue, so the hosts' clocks need not agree. Core still bounds its own wait. A full node queue closes the connection.

The `response` frame carries `id` and `connection_id`. A successful response carries the result named in the operation table, with no result field for `kill`, `kill_compute` or `delete_snapshot`. A failed response carries an `error_code`:
The `response` frame carries `id` and `connection_id`. A successful response carries the result named in the operation table, with no result field for `kill`, `kill_compute` or `delete_retained`. A failed response carries an `error_code`:

| `error_code` | Meaning |
| --- | --- |
Expand Down
11 changes: 6 additions & 5 deletions contracts/agents-api/zh/node-generation-protocol.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "沙箱节点协议"
source: contracts/agents-api/node-generation-protocol.md
source_hash: 1ee43dfcdd0eec0806ea3bc8a4c1227e10bd8ac5e69486505cb113a98e3f548a
source_hash: 187637e03b594296f75883bf67949c430f1b3978f7a059677979243d46177fca
---

沙箱节点在其主机上运行 Docker 或 microsandbox Provider,并通过一个 WebSocket 与 Core 相连。Core 通过该连接发送 Provider 操作;节点针对本地 Provider 执行这些操作,并报告就绪状态、主机测量值及其持有的部署代次。Core 始终是唯一的生命周期所有者:节点绝不重试变更操作或调度工作。帧和校验器位于 [`services/core/internal/sandbox/node`](https://github.com/MiniMax-AI/OpenAgentCore/tree/main/services/core/internal/sandbox/node)(`wire.go`、`generation_wire.go`);节点用于注册和读取配置的 HTTP 路由位于[机器连接 API](machine-api.md#node-routes)。
Expand Down Expand Up @@ -46,20 +46,21 @@ Core 发送包含以下内容的 `request` 帧:
| `command` | `RunCommand` | `command` | `command` |
| `observe` | `Observe` | `observation` | `sample` |
| `initial` | `Initial` | 无 | `compute` |
| `new_compute` | `NewCompute` | 大于零的计算 `generation` 和可选的 `snapshot` | `compute` |
| `new_compute` | `NewCompute` | 大于零的计算 `generation` 和可选的 `retained` | `compute` |
| `compute` | `GetCompute` | `compute` | `state` |
| `renew_compute` | `RenewCompute` | 精确的当前 `compute` | `state` |
| `kill_compute` | `KillCompute` | `compute` | 无 |
| `resume_compute` | `ResumeCompute` | `compute` | `state` |
| `command_compute` | `RunCommandCompute` | `compute` 和 `command` | `command` |
| `suspend` | `Suspend` | `suspend` | `state` |
| `resume` | `Resume` | `resume` | `state` |
| `delete_snapshot` | `DeleteSnapshot` | `snapshot` | 无 |
| `delete_retained` | `DeleteRetained` | `retained` | 无 |

只要 `connection_id`、`owner_epoch` 或 `sequence` 中任一值不匹配,请求就会关闭连接。格式错误的请求会得到 `invalid` 响应。未启用代次管理的节点仅接受其登记的 `deployment_generation`;支持代次管理的节点在对应代次的 Provider 上运行请求,无法运行时回复 `unconfirmed`。Core 仅向节点上已就绪的代次发送 `create` 和非 observe-only 的 `resume`,并且每条连接最多保留 32 个待处理请求。
只要 `connection_id`、`owner_epoch` 或 `sequence` 中任一值不匹配,请求就会关闭连接。格式错误的请求会得到 `invalid` 响应。未启用代次管理的节点仅接受其登记的 `deployment_generation`;支持代次管理的节点在对应代次的 Provider 上运行请求,无法运行时回复 `unconfirmed`。Core 仅向节点上已就绪的代次发送 `create` 和非 reconciliation-only 的 `resume`,并且每条连接最多保留 32 个待处理请求。

预算采用相对计时:节点收到请求时以自己的时钟为基准锚定 `timeout_ms`,并在请求排队等待期间持续消耗该预算,因此各主机的时钟无需保持一致。Core 仍会限制自身等待时长。节点队列已满时会关闭连接。

`response` 帧包含 `id` 和 `connection_id`。成功响应携带操作表中指定的结果;对于 `kill`、`kill_compute` 或 `delete_snapshot`,响应不含结果字段。失败响应携带一个 `error_code`:
`response` 帧包含 `id` 和 `connection_id`。成功响应携带操作表中指定的结果;对于 `kill`、`kill_compute` 或 `delete_retained`,响应不含结果字段。失败响应携带一个 `error_code`:

| `error_code` | 含义 |
| --- | --- |
Expand Down
2 changes: 2 additions & 0 deletions deploy/compose/compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,8 @@ services:
OAC_PROVIDER_STATE_ROOT: /state
OAC_NATIVE_INSTALLER_DIR: /opt/oac/native-installers
OAC_EXECUTION_CONCURRENCY: ${OAC_EXECUTION_CONCURRENCY:-}
OAC_SANDBOX_MAX_ACTIVE: ${OAC_SANDBOX_MAX_ACTIVE:-}
OAC_SANDBOX_MAX_RETAINED: ${OAC_SANDBOX_MAX_RETAINED:-}
OAC_DEFAULT_HARNESS: ${OAC_DEFAULT_HARNESS:-}
OAC_HARNESSES: ${OAC_HARNESSES:-}
OAC_WRITE_AUDIT_RETENTION: ${OAC_WRITE_AUDIT_RETENTION:-}
Expand Down
14 changes: 12 additions & 2 deletions deploy/compose/test_compose.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,17 @@ def rendered_compose(directory):

class ComposeTests(unittest.TestCase):
@classmethod
def render(cls, public_url=None):
def render(cls, public_url=None, settings=None):
env = dict(os.environ)
env.pop('OAC_PUBLIC_URL', None)
env.pop('OAC_HOST', None)
env.pop('OAC_WEB_PORT', None)
env.pop('OAC_SANDBOX_MAX_ACTIVE', None)
env.pop('OAC_SANDBOX_MAX_RETAINED', None)
for name in ('OAC_IMAGE_CORE', 'OAC_IMAGE_WEB', 'OAC_IMAGE_INGRESS'):
env.pop(name, None)
env['OAC_DATA_DIR'] = '/tmp/oac-compose-fixture'
env.update(settings or {})
if public_url is not None:
env['OAC_PUBLIC_URL'] = public_url
return json.loads(subprocess.check_output(
Expand Down Expand Up @@ -69,9 +72,16 @@ def test_compose_uses_private_services_and_ordered_initialization(self):
self.assertEqual(services['web']['healthcheck']['test'], ['CMD', '/usr/local/bin/oac-web', 'healthcheck'])
self.assertNotIn('python3', json.dumps(self.compose))
self.assertEqual(services['init']['environment']['OAC_REVISION'], 'd' * 40)
for name in ('OAC_EXECUTION_CONCURRENCY', 'OAC_DEFAULT_HARNESS', 'OAC_HARNESSES', 'OAC_WRITE_AUDIT_RETENTION', 'OAC_LOG_LEVEL'):
for name in ('OAC_SANDBOX_MAX_ACTIVE', 'OAC_SANDBOX_MAX_RETAINED', 'OAC_EXECUTION_CONCURRENCY', 'OAC_DEFAULT_HARNESS', 'OAC_HARNESSES', 'OAC_WRITE_AUDIT_RETENTION', 'OAC_LOG_LEVEL'):
self.assertEqual(services['core']['environment'][name], '', name)

def test_direct_sandbox_capacity_reaches_core(self):
settings = {'OAC_SANDBOX_MAX_ACTIVE': '7', 'OAC_SANDBOX_MAX_RETAINED': '31'}
services = self.render(settings=settings)['services']
for key, value in settings.items():
self.assertEqual(services['core']['environment'][key], value)
self.assertNotIn(key, services['web']['environment'])

def test_public_url_can_be_configured_after_initial_startup(self):
for value in (None, '', 'https://oac.example.test', 'http://localhost:9080'):
with self.subTest(public_url=value):
Expand Down
1 change: 1 addition & 0 deletions deploy/node/test_node_install.py
Original file line number Diff line number Diff line change
Expand Up @@ -645,6 +645,7 @@ def run_as(account, function, *arguments):
service_account=lambda: self.account),
mock.patch.object(installer.shutil, "which", side_effect=lambda tool: None if tool == "docker" and not self.docker_installed else "/usr/bin/" + tool),
mock.patch.object(installer.grp, "getgrnam", side_effect=lambda name: SimpleNamespace(gr_mem=["oac-node"] if self.joined else [])),
mock.patch.object(installer.os, "getgrouplist", side_effect=lambda name, gid: [gid]),
mock.patch.object(installer.grp, "getgrgid", side_effect=lambda gid: SimpleNamespace(gr_name=self.device_group))):
patch.start()
self.addCleanup(patch.stop)
Expand Down
6 changes: 6 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,8 @@ To change it, point the reverse proxy at the new address first, then edit `OAC_P
| `OAC_LOG_FORMAT` | `auto` | `auto`, `text` or `json` |
| `OAC_LOG_ADD_SOURCE` | unset | `1` adds source locations |
| `OAC_EXECUTION_CONCURRENCY` | `4` | Concurrent execution work, from 1 to 1024 |
| `OAC_SANDBOX_MAX_ACTIVE` | `100` | Active sandbox limit for direct Providers with suspension, from 1 to 100000. Independent of execution concurrency and node capacity |
| `OAC_SANDBOX_MAX_RETAINED` | `400` | Retained sandbox limit for direct Providers, from 1 to 100000, including active, suspended and unconfirmed cleanup. Must be at least the active limit |
| `OAC_DEFAULT_HARNESS` | `codex` | Harness used when a request does not name one |
| `OAC_HARNESSES` | Every registered Harness | Comma-separated Harnesses to enable besides the default one. Unknown names stop startup |
| `OAC_WRITE_AUDIT_RETENTION` | `2160h` | Minimum `1h` |
Expand All @@ -73,6 +75,10 @@ Runtime settings live in Core's database. Change them in Web; scripts use the sa

Which harnesses are enabled, and the default one, are process settings (`core.harnesses`, `core.default_harness`); System shows them read-only. The [Core administration API](../contracts/agents-api/admin-api.md) lists every Core API route, and the [deployment contract](../contracts/agents-api/sandbox-deployment.md) defines the sandbox fields, limits and change rules.

### Direct Provider capacity

Set `OAC_SANDBOX_MAX_ACTIVE` and `OAC_SANDBOX_MAX_RETAINED` in `.env`, then run `oac apply`. Core uses these limits for direct Providers with suspension enabled. Every unreleased allocation consumes retained capacity. Lowering a limit stops no existing sandbox; new allocations wait until usage falls below both limits. These settings are independent of `OAC_EXECUTION_CONCURRENCY` and enrolled node capacity.

### Node capacity

Core approves a node's capacity when you generate its Add node command: **Sandboxes at once** (`max_active`, default 2) and, for microsandbox only, **Retained sandboxes** (`max_retained`, default 8), with `max_retained >= max_active >= 1`. Docker never suspends sandboxes, so Web doesn't ask for it and Core keeps `max_retained` equal to `max_active`. Change them later with **Edit node**. Reservations and cleanup that is not confirmed count against capacity; lowering a limit stops no running sandbox. A node's own files can't change its capacity, size or Runtime.
Expand Down
4 changes: 4 additions & 0 deletions docs/runtime-bootstrap.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,10 @@ The Runtime validates the input and owns authentication and connection. A succes

Self-hosted executors and operator-provisioned devices get their daemon identity in other ways; the [machine connection API](../contracts/agents-api/machine-api.md#credentials) lists every credential source. All of them enter the same Runtime execution loop.

## Hosted suspension control

The private hosted park/wake control-file path is authored as `SuspendControlFile` in `internal/runtimebootstrap/bootstrap.go`. Core recovery and native Go adapters read that value; the E2B helper contract generator projects it into the template builder. Managed startup prepares its private directory and supplies `OAC_RUNTIME_DAEMON_SUSPEND_PID_FILE` to enable Runtime suspension. This is a packaged protocol setting. The shared Sandbox Provider registration owns idle and retention defaults; the adapter owns its native lease timeout.

## Verification

`go test ./internal/runtimebootstrap ./apps/daemon/internal/cli` covers the input contract, the exclusivity of credential sources and restart behavior. Provider tests verify delivery and file permissions without relying on the Runtime's private storage.
Loading
Loading