feat(delegate): drive the public @uipath/delegate-stdio host [PILOT-7854]

[Mihaiii]

Issue

PILOT-7854

Warning

Breaking changes. The auth and backend variable names stay as on main. What changes:

Before (main)	Now
npm install @uipath/delegate-sdk	npm install -g @uipath/delegate-stdio, version 1.203.0 or later (the first release with the auth init option and per-call usage frames)
DELEGATE_SDK_PATH → @uipath/delegate-sdk/dist/index.mjs	DELEGATE_STDIO_PATH → @uipath/delegate-stdio/dist/delegate_stdio.mjs; any other file is an AgentConfigError
DELEGATE_SDK_NODE_MODULES → root holding @uipath/delegate-sdk; home directory searched	Removed. A global install is found through the delegate-stdio shim on PATH; a local install still wins
agent.effort (Delegate-only field)	agent.sdk_options.effort, the key Claude Code already uses
Host env carried AUTH_TOKEN / TENANT_ID / ORG_ID / …	Removed from the host env; auth reaches the host as its auth init option on stdin

Unchanged: DELEGATE_ENV, DELEGATE_BACKEND_URL, and DELEGATE_AUTH_TOKEN / DELEGATE_TENANT_ID / DELEGATE_ORG_ID / DELEGATE_ORG_SLUG / DELEGATE_TENANT_SLUG, each with its bare spelling as a fallback.

Summary

• Replace the first-party agents/delegate/delegate_host.mjs with the @uipath/delegate-stdio host (^1.203.0); it installs @uipath/delegate-sdk and the interop runtime itself
• Port DelegateAgent to the host's protocol: event frames, per-call usage frames, terminal result / error, destroyed
• Send auth, the org/tenant slugs, the backend URL and the env slug as init options (auth, backendUrl, env); remove the host's own names (AUTH_TOKEN, TENANT_ID, ORG_ID, ORG_LOGICAL_NAME, TENANT_NAME, BACKEND_URL) and DELEGATE_AUTH_TOKEN from its env, so agent shells do not inherit the token and a stray BACKEND_URL cannot route the host
• Keep token usage and cost for the finished calls of a turn cut at max_turns or by an early stop
• Make crash retry independent of the host's stderr text (build 13599116 retried "terminated" on four tasks but not on one whose id contains "guardrail")
• Match HTTP status codes as whole words in the shared categorize_error: a port or GUID holding 401 (127.0.0.1:54013) no longer turns a transient crash into a non-retryable auth error, on any agent

Changes

Adapter — src/coder_eval/agents/delegate_agent.py

• _resolve_host_bundle replaces _resolve_sdk_entry: DELEGATE_STDIO_PATH, then the cwd, its ancestors and this agent's own agents/delegate/, then a global install found through the delegate-stdio shim on PATH
• _env (namespaced first, bare fallback) + _auth_option build the auth init option; _HOST_ENV_REMOVED lists what leaves the host env
• Read toolArgs / toolResult / toolStatus, Anthropic-convention usage, result.model; num_turns = len(result.turnUsages)
• isStepStart: false deltas extend one text block; enableSkills is sent explicitly (the host default is false); LoadSkill is recorded as the canonical Skill call
• Sum the per-call usage frames; the result's usage (the turn total) replaces the sum
• A tool result with no open call keeps its name and args; toolStatus: "interrupted" is an error
• Categorize WAF blocks (content filter, not retried), SSE connect timeouts (connection, retried) and session conflicts (a new conversation only while no turn of it has finished; after a finished turn, or with a pinned session_id, the retry keeps the conversation)
• Init errors: only auth / missing slugs / unknown env / a whole-word 401 or 403 are non-retryable AgentConfigError (with a hint that names coder_eval's variables); a mid-run respawn keeps that classification; other init errors and the 60 s init deadline are retryable
• No crash reason carries the stderr tail; it is logged at WARNING
• get_sdk_options() returns the init options with auth reduced to its field names and backendUrl to its host; a cancelled stdout drain no longer logs at ERROR; _force_kill_host clears the process handle
• Remove LLMGW_* from the host env when a token file is set
• A DELEGATE_BACKEND_URL that urlparse cannot parse records no host (None) instead of raising inside the orchestrator's _finalize_result, which lost task.json
• _build_init_options applies sdk_options first, so workingDirectory, enableComputerUse and model win even over a writer that skips validation (model_copy)

Errors — src/coder_eval/errors/categorization.py

• _mentions_status matches 401 / 402 / 429 / 502 / 503 / 504 as whole words; the phrase patterns are unchanged

Overrides — src/coder_eval/orchestration/overrides.py

• The sdk_options override guard accepts every registered agent kind whose config declares sdk_options (registry lookup), not only claude-code

Docs, CI, tests

• docs/agents/DELEGATE.md (global install, 1.203.0 floor, the env scrub described as defense in depth with its known gaps), docs/USER_GUIDE.md, docs/agents/HARNESS_PARITY.md, .env.example, .claude/notes/agents.md
• docs/TASK_DEFINITION_GUIDE.md, experiments/default.yaml, docs/REPORT_SCHEMA.md and the EvaluationResult.sdk_options description: the sdk_options keys and the recorded shape depend on the agent type
• pr-checks.yml delegate-live-tests: installs @uipath/delegate-stdio in agents/delegate/, sets the DELEGATE_* names from the existing DELEGATE_* secrets
• Golden-master scenarios for Delegate (tests/_fixtures/golden_streams/delegate_fixtures.py: a tool call, a max_turns cut, an error-frame crash, an orphaned tool) replace the _NO_GOLDEN_COVERAGE exemption. The frame builders and the fake host moved there from tests/test_delegate_agent.py; the 1.202.1 frames mirror a live transcript, and the 1.203.0 usage / result fields follow the host source of UiPath/Autopilot#6656. An autouse fixture clears the developer's own DELEGATE_* values
• The override-guard tests match the fixed part of the message, and a throwaway plugin kind proves the registry-based guard

Implementation Notes

• The host gives each auth field priority over its env var; the SDK reads none of them. It takes the credentials from the TokenAuthProvider the host builds, and an explicit backendUrl replaces the BACKEND_URL default its bundle reads at load. Host side: UiPath/Autopilot#6656. A refresh source (token file, LLMGW_*, saved login) still writes its token into the host's own process.env.AUTH_TOKEN.
• @uipath/delegate-stdio 1.203.0 is not published yet. It ships from UiPath/Autopilot#6656, which merges first. Until the release exists, npm install in agents/delegate/ cannot resolve ^1.203.0, so the delegate-live-tests job fails at install.
• The verbose host trace does not log the token. With DELEGATE_STDIO_VERBOSE=1, 1.203.0 logs each stdin command with auth reduced to its field names (redactInitAuth).
• Cut-turn usage relies on frame order: call N's usage frame precedes the tool results of call N. The max_turns boundary does not move.
• max_turns stays client-side: live, maxSteps: 2 ran 7 steps.

Testing

• test_delegate_agent*.py, test_delegate_agent_config.py, test_error_handling.py: 206 passed, 6 skipped. New tests: the auth option, namespaced-over-bare, removed host names, redacted sdk_options, the DELEGATE_STDIO_PATH file check, global-install resolution (POSIX symlink and Windows .cmd shims), the init-error hint, whole-word 401/403 (a port and a GUID stay retryable), the respawn classification, stderr-tail categorization, init retry, drain cancel, kill() handle
• Each review-fix test failed on the previous adapter code and passes now
• Golden master: uv run pytest tests/test_agent_golden_master.py -k delegate runs the 4 Delegate snapshots and their reconciliation invariant (8 passed); CI runs them in Quality Gate and Windows Smoke Test. New unit tests: test_reconciliation_invariant (four buckets summed across messages equal token_usage) and test_framework_owned_init_keys_win_over_sdk_options (fails on the old order)
• Review fixes, each with a test that fails on the old code: status-code rows in test_retry_logic_comprehensive.py and a should_retry check on every Delegate init-error case; a 409 after a finished turn and with a pinned session_id keeps the session; test_a_malformed_backend_url_records_no_host
• On eabfe326's code: full suite 6034 passed, coverage 90.68 %; pyright 0 errors; ruff and prose budget clean; custom lint 734 passed. Local Windows-only failures: 10 tests (symlink privilege, a node_modules in a parent of the temp dir) and CE033 test_drift_is_detected (an encoding issue in a test from feat(plugin): ship coder_eval as a Claude Code plugin + marketplace #82)
• Live against alpha, host built from UiPath/Autopilot#6656, DELEGATE_STDIO_VERBOSE=1: PONG, 12,250 tokens, $0.00095; the host log shows Using init-option / env var auth (not the saved login); the token is in neither the host's stderr nor sdk_options
• Full suite at 404b0994: 6013 passed, coverage 90.66 %; custom lint (735), ruff, prose budget and pyright (0 errors) clean. The 10 local failures are environment-only (Windows symlink privilege, a node_modules in a parent of the temp dir)

🤖 Generated with Claude Code

[github-actions]

Claude finished @Mihaiii's task in 1m 42s —— View job

---

PR Review in Progress

Review Checklist

• Read .github/code_review.md for review standards
• Read CLAUDE.md for project conventions
• Analyze the full diff (git diff origin/main...HEAD)
• Review each changed file with full context
• Check cross-file consistency
• Perform "what's missing" analysis
• Conduct design-level scrutiny
• Post comprehensive review feedback

Starting review now...

[Mihaiii]

[Axis 1] Pinned @uipath/delegate-stdio floor (^1.202.1) ignores the auth init option while env-token vars are removed, so the token auth path fails or silently uses the ambient login

This PR relies on the Autopilot related PR being merged first. I set ^1.203.0, even though that version is not yet released. "Delegate Live E2E (ROPC)" will fail for this PR because the package is not released (whereas in 1.202.1 it failed because of the blocker issue :))

[Axis 4] Bearer token travels in the stdin init frame. With DELEGATE_STDIO_VERBOSE=1 the host echoes it to stderr twice, and coder_eval logs that stderr unredacted to task.log and error_log_tail.

The Autopilot host already redacts it. delegate_stdio.ts:1581 logs redactInitAuth(cmd). No coder_eval-side redaction is needed.

[Axis 8] Removing DelegateAgentConfig.effort makes every Delegate task.json from v0.12.5 to v0.12.9 unloadable: --resume silently re-runs finished tasks, and run.json recovery drops the rows

This was not addressed. The project has no backward-compatibility burden, and addressing this would only make the code harder to read and more complicated. How often was the Delegate Agent from the coder_eval repo used before this change, such that we need to worry about backward compatibility?

[Mihaiii]

New run: https://dev.azure.com/uipath/coder_eval/_build/results?buildId=13640767&view=results

[uipreliga]

Review: coder_eval — pr:207 (25 files) axis:1,2,3,4,5,6,7,8 — #207 'feat(delegate): drive the public @uipath/delegate-stdio host [PILOT-7854]' by Mihaiii (head chore/move-delegate-sdk-2 → main)

Scope: pr:207 (25 files) axis:1,2,3,4,5,6,7,8 — #207 'feat(delegate): drive the public @uipath/delegate-stdio host [PILOT-7854]' by Mihaiii (head chore/move-delegate-sdk-2 → main) · branch chore/move-delegate-sdk-2 · 94e4a5a · 2026-10-03T15:35Z · workflow variant

Change class: complex — replaces the bespoke Node host with the public @uipath/delegate-stdio protocol driver: new subprocess/JSON-RPC control flow, crash-retry and error categorization, auth-as-init-option, env scrubbing, and a breaking env-var rename (DELEGATE_SDK_PATH → DELEGATE_STDIO_PATH, AUTH_TOKEN → DELEGATE_AUTH_TOKEN)

The Delegate move to the public delegate-stdio host is in good condition (9.4/10: strong types, careful credential redaction and thorough docs), but four defects can change a task's result or record: a substring retry classifier makes some transient init/send crashes non-retryable, a 409 session reset lets a dialog continue without its earlier context and still get a grade, a bad DELEGATE_BACKEND_URL loses task.json, and verbose host mode writes the bearer token to task.log; fix the first three before merge and the token leak soon after.

Summary

Axis	Score	🔴	🟠	🟡	🔵	Top Issue
1. Code Quality & Style	9.3 / 10	0	0	1	2	Branch complexity in DelegateAgent is still above the anchor, and _spawn_and_init is newly C-rated
2. Type Safety	9.9 / 10	0	0	0	1	DelegateAgentConfig.sdk_options replaces the typed effort field with an untyped one-key dict and a hand-written validator, unlike the sibling agents
3. Test Health	9.3 / 10	0	0	1	2	Delegate stays exempt from golden-master snapshots after the stdio protocol rewrite; the PR defers the snapshots and documents the deferral
4. Security	9.4 / 10	0	0	1	1	With DELEGATE_STDIO_VERBOSE=1 (a knob this PR's docs point to), the host echoes the raw init frame, including auth.accessToken, to stderr. coder_eval logs each stderr line in clear text to task.log and to the HTML report's log tail.
5. Architecture & Design	9.3 / 10	0	0	1	2	Delegate's get_sdk_options() init dict is rendered by Claude-shaped report code, which drops the Plugins row from Delegate reports
6. Error Handling & Resilience	8.9 / 10	0	0	2	1	The Delegate init-error classifier is a second substring classifier, and categorize_error still matches the raw '401' substring, so the whole-word 401/403 guard does not make port/GUID errors retryable
7. API Surface & Maintainability	9.7 / 10	0	0	0	3	The resolved delegate-stdio host is never checked against the documented 1.203.0 minimum version
8. Evaluation Harness Quality	9.2 / 10	0	0	1	3	Session-conflict recovery drops the conversation id with only a WARNING log, so later dialog turns continue without context and the run record has no reset marker

Overall Score: 9.4 / 10 · Weakest Axis: Error Handling & Resilience at 8.9 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 7 · 🔵 15 across 8 axes.

Blockers

None.

Non-blocking, but please consider before merge

1. [Axis 1] Branch complexity in DelegateAgent is still above the anchor, and _spawn_and_init is newly C-rated (src/coder_eval/agents/delegate_agent.py:732) — radon (routed) lists these functions, which are one theme: communicate L732 D(27), _handle_event L856 D(22), _spawn_and_init L528 C(13), _finalize_turn L1041 C(12). Compared with origin/main (radon on git show origin/main:), communicate is unchanged at 27 and _handle_event dropped from 24 to 22. _spawn_and_init is the one this PR made worse, because it adds inline env scrubbing (L530-540): env = dict(os.environ) ... if stripped := _strip_redundant_gateway_creds(env): ... for name in _HOST_ENV_REMOVED: env.pop(name, None). Move the env construction into a pure module-level _host_env(path_prepend, plugin_tools_dir) -> dict[str, str], which you can also unit-test without a fake process. In communicate, move the frame dispatch (L799-813, result/error/usage/event) into a helper, so the loop handles only deadline, stop and max_turns. This module is not a hot module, so the anchor is Medium, not High.
2. [Axis 3] Delegate stays exempt from golden-master snapshots after the stdio protocol rewrite; the PR defers the snapshots and documents the deferral (tests/test_agent_golden_master.py:240) — The PR replaces the whole stdio protocol: frame envelope, usage convention, per-call usage frames, turnUsages call count and tool-arg shapes. But it keeps AgentKind.DELEGATE: "no recorded delegate-stdio scenarios yet — tests/test_delegate_agent.py replays the frames" in _NO_GOLDEN_COVERAGE (line 240). The old reason ('field shapes UNVERIFIED') is gone, and .claude/notes/agents.md:882 now says "The frame shapes are now confirmed live, so record real scenarios to lift this". As a result, the persisted TurnRecord is rebuilt by EventCollector from Delegate events but is only checked field-by-field on a hand-picked subset (for example record.agent_output, record.commands[0].result_status, record.token_usage.uncached_input_tokens). A field that stops being mirrored drops silently. Also, no Delegate test checks the CLAUDE.md reconciliation invariant (Σ buckets across record.messages == token_usage). OpenCode and Pi each have test_reconciliation_invariant in their unit suites (tests/test_opencode_agent.py:138, tests/test_pi_agent.py:121), and the golden suite runs assert_reconciliation for every covered agent. I ran a probe: the invariant holds today for a result-only turn and for a usage-frame + result turn, so this is a coverage gap, not a live bug. Fix: add DELEGATE_SCENARIOS (happy path with a tool, max_turns cut with usage frames, crash on an error frame) to SCENARIOS_BY_AGENT, built from the existing _ev/_result/_usage/_tool_call builders, and remove the exemption. As a minimum, add a test_reconciliation_invariant to tests/test_delegate_agent.py that mirrors the OpenCode and Pi tests.
3. [Axis 4] With DELEGATE_STDIO_VERBOSE=1 (a knob this PR's docs point to), the host echoes the raw init frame, including auth.accessToken, to stderr. coder_eval logs each stderr line in clear text to task.log and to the HTML report's log tail. (src/coder_eval/agents/delegate_agent.py:1276) — Verified by running the host locally (@uipath/delegate-stdio 1.203.0-preview at /Users/religa/src/coder_eval/tmp/ds/inst2) with DELEGATE_STDIO_VERBOSE=1. An init frame that carried a sentinel token came back on stderr twice, in clear text: [delegate-stdio] [dispatch] Received line (321 chars): {"cmd":"init","options":{..."auth":{"accessToken":"SENTINELTOKEN_abcdef123456",...}}} and [handleInit] Called with options: {..."accessToken":"SENTINELTOKEN_..."...}. Without the variable, stderr held no token.

How the token reaches persisted output: _drain_stderr keeps every line (self._stderr_lines.append(text), line 1275) and logs it (logger.debug("delegate[stderr]: %s", text), line 1276). orchestrator.py:526 always opens task_log_handler(...) at its DEBUG default, and logging_config.py:282-292 attaches a DEBUG FileHandler plus the _LogTailBuffer that the HTML report embeds. _sanitise_log_text only strips ANSI and control bytes, so the bearer token lands in runs//.../task.log and in the report. On a crash, _log_stderr_tail (line 1132) also prints the same lines at WARNING. CI uploads runs/delegate-live/ as an artifact. CI does not set the variable today.

This is new in this PR. The deleted in-tree delegate_host.mjs never echoed init options. This PR moves the token in-band (options["auth"] = auth, line 602) to a third-party host that traces it, and docs/agents/DELEGATE.md:56 directs operators to DELEGATE_STDIO_VERBOSE=1 (trace every frame to stderr) with no warning.

Fix: in _drain_stderr, redact the known secret values (the accessToken from self._init_options["auth"]) before you append or log a line. Alternatively, mask any "accessToken":"..." JSON field with a regex. Add a warning at DELEGATE.md:56 that verbose mode writes the token to task.log. Add a test: a fake host echoes the init line on stderr, then assert that the token is not in caplog. CVSS:3.1/AV:L/AC:H/PR:L/UI:R/S:U/C:H/I:N/A:N
4. [Axis 5] Delegate's get_sdk_options() init dict is rendered by Claude-shaped report code, which drops the Plugins row from Delegate reports (src/coder_eval/agents/delegate_agent.py:695) — This PR adds def get_sdk_options(self) -> dict[str, Any] | None: (delegate_agent.py:695), which returns the host init dict. results.py:675-681 now describes the field as 'the shape depends on the agent', but nothing in the record says which shape it holds. The core consumers that read this field still assume the Claude Code shape. (1) orchestrator.py:1614 sdk_env = sdk_options.get("env") reads env as the ClaudeAgentOptions env dict that carries PATH. Delegate's dict uses the same key for a different value, the cloud slug string (delegate_agent.py:600 options["env"] = environment). Today only the isinstance(sdk_env, dict) guard keeps the two shapes apart, so Delegate's shellPathPrepend is never used to set the criteria PATH. (2) reports/markdown.py:57 resolve_agent_settings and reports/html.py:1025 prefer sdk_options over agent_config whenever it is present. A Delegate report therefore renders the Claude-specific rows from the init dict. It shows 'Permission Mode: N/A' and 'System Prompt Mode: unknown', and it loses the 'Plugins' row that the agent_config fallback used to show, because the init dict has bundledSkillsPath and no plugins key. This is the first non-Claude implementer of this channel, so the PR is what makes the shared contract ambiguous. Fix: do not parse an untyped per-agent dict in core. Move the PATH sync behind an Agent method (for example get_command_path()), and have the report renderers dispatch on agent_config.type, or persist a discriminator next to sdk_options. As a minimum, do not reuse the Claude key name env for a different meaning in the persisted dict.
5. [Axis 6] The Delegate init-error classifier is a second substring classifier, and categorize_error still matches the raw '401' substring, so the whole-word 401/403 guard does not make port/GUID errors retryable (src/coder_eval/agents/delegate_agent.py:579) — _is_config_init_error (line 132-136, _INIT_CONFIG_ERROR_STATUS = re.compile(r"\b40[13]\b")) only chooses between AgentConfigError and AgentCrashError. The retry decision comes later, in errors/categorization.py, and its substring patterns run before the AgentCrashError fallback (if any(pat in error_str for pat in ["authentication", "unauthorized", "invalid api key", "401"])). So line 579 raise AgentCrashError(f"Delegate SDK init failed: {message}") is still not retried for the cases this PR says it fixes. I ran categorize_error to check: AgentCrashError('Delegate SDK init failed: connect ECONNREFUSED 127.0.0.1:54013') returns AGENT_AUTH_ERROR, and should_retry(AGENT_AUTH_ERROR, 0) is False. The GUID case tenant c7a3f401-... also returns AGENT_AUTH_ERROR. The send path has the same leak: line 1172 reason = f"Delegate send failed: {message}" gives AGENT_AUTH_ERROR for the same port message. The SSE rewrite has it too: lines 401-402 defang only 'timeout', so a URL with port 4401 in the tail still returns AGENT_AUTH_ERROR. Other incidental substrings ('429', '402', 'invalid', 'credit') also turn a crash non-retryable. Two docs over-promise as a result: docs/agents/DELEGATE.md:138 ('Any other init error ... is retryable') and :173 ('Every other crash ends the turn as a retryable AgentCrashError'). The test at tests/test_delegate_agent.py:273 (("connect ECONNREFUSED 127.0.0.1:54013", AgentCrashError)) checks only the exception type. Unlike test_init_timeout_is_retryable at line 312, it does not call categorize_error. Fix options: defang or strip the status-like digit runs in the reason, as the SSE path does for 'timeout'; or move categorize_error's status codes to a whole-word regex. Add categorize_error(...) is ErrorCategory.AGENT_CRASH to the port and GUID test cases.
6. [Axis 6] A malformed DELEGATE_BACKEND_URL makes get_sdk_options() raise ValueError inside the unguarded _finalize_result, so task.json is lost and the original error is hidden (src/coder_eval/agents/delegate_agent.py:707) — Line 707 options["backendUrl"] = urlparse(options["backendUrl"]).hostname raises ValueError: Invalid IPv6 URL for a bracket typo such as DELEGATE_BACKEND_URL='https://[fd00::1:8080/api'. I reproduced this directly on PR head. _init_options is set at line 564 before the init handshake, so the bad value is stored even when init then fails. The orchestrator calls self.result.sdk_options = self.agent.get_sdk_options() (orchestrator.py:1061) inside _finalize_result, which runs in run()'s finally with no guard. The comment at orchestrator.py:1014 says an unguarded raise there 'would skip persistence and lose task.json'. Result: the init error is replaced by an unrelated ValueError, and task.json is never written. The rubric says an artifact the harness must always produce must degrade, and the diagnostic path must not hide the original error. Fix: wrap the hostname extraction, e.g. try: host = urlparse(url).hostname except ValueError: host = None. Also fix the same call in get_environment_info at line 723.
7. [Axis 8] Session-conflict recovery drops the conversation id with only a WARNING log, so later dialog turns continue without context and the run record has no reset marker (src/coder_eval/agents/delegate_agent.py:1171) — _crash_on_host_error handles a 409 'already being generated' by setting self._session_id = None (line 1171). Its only trace is a logger.warning("delegate: session %s is still generating a reply; the retry starts a new conversation", ...) (lines 1166-1169). The execute_with_retry attempt that follows sends "sessionId": None and opens a new backend conversation. On iteration 1 this is a clean retry. On iteration >= 2 (criteria-feedback loop or dialog/simulation mode) the agent loses every earlier turn's context, but the row is graded as one continuous trajectory. Before this PR, the same condition crashed again and ended as ERROR. Now it can finish as a normal PASS/FAIL row on a different trajectory. get_environment_info records only the final delegate_session_id (lines 726-727), so task.json cannot tell this row apart from a normal one, and cross-run comparison mixes the two populations. Fix: (a) record the reset on the result, e.g. environment_info["delegate_session_resets"] or the iteration where it happened, and/or (b) drop the session only when no earlier turn completed in this conversation (self._iteration == 1), and otherwise let the crash stay terminal so a context-less continuation is not scored. Add a test that runs a successful turn 1, then a 409 on turn 2, and asserts the recorded marker (or the terminal crash). Nightly: not reached. The coder-eval-uipath nightly runs the out-of-tree delegate-sdk kind (experiments/delegate.yaml type: delegate-sdk), not the in-tree delegate.

Nits

1. [Axis 1] get_environment_info re-reads the routing env vars that get_sdk_options now also persists (src/coder_eval/agents/delegate_agent.py:720) — The PR adds get_sdk_options() (L695-708). It persists the init dict, including backendUrl reduced to its host (options["backendUrl"] = urlparse(options["backendUrl"]).hostname, L707), env, and model. get_environment_info still reads os.environ.get("DELEGATE_BACKEND_URL") / os.environ.get("DELEGATE_ENV") again on its own (L720-725), which repeats _build_init_options L595-600. It also records delegate_model (L713). So task.json carries the same routing facts twice, from two reads of the environment at different times. The PR already removed delegate_effort from environment info for this reason. Do the same for delegate_backend_url_host / delegate_env / delegate_model, or derive them from self._init_options, so that only one place reads the env.
2. [Axis 1] The env-var names in _INIT_CONFIG_ERROR_HINT and _HOST_BUNDLE_REL_PATH are copies of constants defined next to them (src/coder_eval/agents/delegate_agent.py:139) — _INIT_CONFIG_ERROR_HINT (L139-144) types out "DELEGATE_AUTH_TOKEN / DELEGATE_TENANT_ID / DELEGATE_ORG_ID / DELEGATE_ORG_SLUG / DELEGATE_TENANT_SLUG" and "AUTH_TOKEN / TENANT_ID / ORG_ID / ORG_LOGICAL_NAME / TENANT_NAME". These are the names in _AUTH_OPTION_FIELDS (L194-200) and the first five entries of _HOST_ENV_REMOVED (L209-215). In the same way, _HOST_BUNDLE_REL_PATH = Path("node_modules") / "@uipath" / "delegate-stdio" / ... (L92) repeats _HOST_PACKAGE = "@uipath/delegate-stdio" (L89). Build these strings from the tuples instead, for example " / ".join(f"DELEGATE_{n}" for _, n in _AUTH_OPTION_FIELDS) and Path("node_modules", *_HOST_PACKAGE.split("/"), "dist", _HOST_BUNDLE_NAME). Then a new auth field cannot leave the hint wrong.
3. [Axis 2] DelegateAgentConfig.sdk_options replaces the typed effort field with an untyped one-key dict and a hand-written validator, unlike the sibling agents (src/coder_eval/models/agent_config.py:419) — sdk_options: dict[str, Any] = Field(default_factory=dict, ...) (line 419) and _validate_sdk_options_keys (lines 454-465: unknown = sorted(set(v) - _DELEGATE_SDK_OPTION_FIELDS) ... if "effort" in v and not isinstance(v["effort"], str)) re-implement in Python what Pydantic can enforce from a type. The runtime validation is complete, so there is no type hole. But the static type and the JSON schema now say Any, and the validator must be edited by hand for every new key. Use a TypedDict, like the LocalPluginConfig already in this file: class DelegateSdkOptions(TypedDict, total=False): __pydantic_config__ = ConfigDict(extra="forbid"); effort: str, and type the field sdk_options: DelegateSdkOptions. I checked this with pydantic: it accepts {'effort':'high'} and rejects {'effort':5}, {'x':1} and {'effort':None}. The value stays a plain dict at runtime, so options.update(...) and the == {"effort": "high"} tests do not change, and the hand-written validator can be deleted. Before you do this, confirm that config_merge.classify_annotation / _merge_value treat the TypedDict annotation as a dict. _merge_dict_by_model takes the strategy from matching[0], so the field is shared with ClaudeCodeAgentConfig.sdk_options.
4. [Axis 3] Several new Delegate branches have no test: bundle-name guard, 409 inverse guard, cancel/generic crash paths, frame-tolerance branches, spawn cwd, mid-run respawn error, and the sdk_options guard message (src/coder_eval/agents/delegate_agent.py:298) — if candidate.name == _HOST_BUNDLE_NAME and candidate.is_file(): is the only thing that stops _global_install_candidates()'s shim_path.resolve() (the delegate-stdio.cmd file on Windows) from being returned and run with node. I changed it to if candidate.is_file(): and all 8 TestResolveHostBundle tests still pass. The cause: test_global_install_found_beside_a_windows_cmd_shim (tests/test_delegate_agent.py:231-235) only points shutil.which at prefix / "delegate-stdio.cmd" and never creates that file, so is_file() already rejects it. Fix: create the .cmd shim file in that test (and in test_local_install_wins_over_global), so that the name guard is the check that is actually tested.
5. [Axis 3] The Delegate get_sdk_options() now puts a string under the env key, which the orchestrator PATH sync reads as Claude's env dict. No test covers this. (src/coder_eval/agents/delegate_agent.py:600) — options["env"] = environment (line 600, the DELEGATE_ENV slug such as 'alpha') is returned by the new non-None get_sdk_options(). That return now goes to Orchestrator._sync_sandbox_command_path_with_agent, which does sdk_env = sdk_options.get("env") (orchestrator.py:1614) and expects ClaudeAgentOptions' env dict. Today only the isinstance(sdk_env, dict) guard keeps this harmless. No test pins that a Delegate sdk_options dict with a string env leaves PATH sync a no-op and writes no command_base_path. Fix: add a small test that feeds a started DelegateAgent's get_sdk_options() (with DELEGATE_ENV set) through _sync_sandbox_command_path_with_agent and asserts that environment_info has no command_base_path, so that the key collision is pinned.
6. [Axis 4] The new credential-handling host dependency floats on a caret range, with no lockfile and no --ignore-scripts. CI installs it with the minimum-package-age gate skipped, in the job that then gives it a live token. (src/coder_eval/agents/delegate/package.json:7) — "@uipath/delegate-stdio": "^1.203.0" (package.json:7). The directory has no committed package-lock.json (git ls-files shows only .gitignore and package.json). In .github/workflows/pr-checks.yml, the install step runs npm install --safe-chain-skip-minimum-package-age. The next steps put DELEGATE_AUTH_TOKEN into the env of a process that runs this package, and that package now receives the bearer token in-band. The shipped dist/delegate_stdio.mjs is obfuscated (a _0x... string-table bundle), so you cannot audit what it does with the token.

The base had the same caret, no-lock shape for @uipath/delegate-sdk, so the extra risk is small. But this PR swaps in a different package and makes it the component that holds the credential.

Fix: pin an exact version ("1.203.0") and commit a package-lock.json. In CI, use npm ci --ignore-scripts, or keep the age gate on for packages other than the delegate-runtime-* platform binaries. CVSS:3.1/AV:N/AC:H/PR:H/UI:N/S:U/C:L/I:L/A:N
7. [Axis 5] Host-error recategorization changes the crash message text to steer the substring categorizer in errors/categorization.py (src/coder_eval/agents/delegate_agent.py:402) — _describe_host_error (lines 380-408), which this PR adds, sets the error category by editing the message text. For a WAF block it adds 'content filter' so the message matches categorization.py:104, and it keeps the word 'connection' so the message matches line 108. For an SSE connect timeout it removes the word 'timeout': defanged = re.sub("timeout", "time-out", original, flags=re.IGNORECASE) (line 402), so that categorization.py:98 if "timeout" in error_str: does not match. errors/agent.py states the convention: routing by isinstance is 'preferred over substring matching on a message, which can silently re-categorise a reworded RuntimeError'. The PR's own notes record a shipped defect of this same class: in build 13599116 the word 'guardrail' in a task path made a transient error non-retryable. The coupling to categorization.py exists only in the wording of the message. One test (test_known_host_errors_are_recategorized) pins it. Fix: raise typed exceptions instead, for example AgentCrashError subclasses or an explicit category=/retryable= attribute that _categorize_by_exception_type reads, so that a change to a pattern in categorization.py cannot silently change how Delegate failures are retried.
8. [Axis 5] UiPath-internal adapter logic is ported from the coder_eval_uipath plugin into the core wheel (src/coder_eval/agents/delegate_agent.py:223) — The PR moves the host to a public npm package, which is good. It also ports UiPath-specific handling into the importable core from the out-of-tree delegate-sdk adapter in coder_eval_uipath (.claude/notes/agents.md: 'ported from the out-of-tree delegate-sdk adapter'; HARNESS_PARITY.md: 'the built-in delegate replaced' it). Examples: _GATEWAY_S2S_ENV_VARS = ("LLMGW_CLIENT_ID", "LLMGW_CLIENT_SECRET", "LLMGW_URL") (line 223), which is the UiPath LLM Gateway S2S credential names, and _WAF_BLOCK_PAGE_MARKERS = ("continue with uipath platform", ...) (line 369), which is a fingerprint of UiPath's Cloudflare page. The agent registers unconditionally from core, and its [uipath] extra is empty (pyproject.toml comment). This goes against the agnostic-core / UiPath-opt-in direction for the OSS release. The impact today is small, because the names are public through the host package. Fix: before the public release, decide whether DelegateAgent should ship as a coder_eval.plugins entry point in a separate package (the SPI exists for this), or document it in CLAUDE.md as an intended built-in vendor agent.
9. [Axis 6] _drain_stdout re-raises a non-cancel failure, and _cancel_drain_tasks then re-raises that stored exception on every later respawn and on stop(), so the agent stays stuck and _mark_stopped() is skipped (src/coder_eval/agents/delegate_agent.py:1259) — _drain_stdout ends with except BaseException: logger.exception(...); await self._stdout_queue.put(None); raise (lines 1259-1262). The task therefore finishes holding its exception, for example a RecursionError from json.loads on a deeply nested line. _cancel_drain_tasks suppresses only cancellation: lines 681-682 are with contextlib.suppress(asyncio.CancelledError): await task. The await re-raises the stored exception, and lines 683-684 (self._stdout_task = None) never run. So the next communicate() respawn (line 747 -> line 542) raises the same stale exception before _begin_turn(), every time, until retries are used up. Also stop() -> _teardown_host (line 693) raises before self._mark_stopped() (line 612), and the stderr task is never cancelled. The comment at lines 676-677 says the drains 'already log and handle their own non-cancellation failures internally', which is not true for stdout. The trigger is rare. The PR changed this function (it added the CancelledError clause at 1257-1258), and 1259-1262 is a routed coverage gap. Fix: return instead of raise after putting the None sentinel in _drain_stdout, or use suppress(BaseException) / collect task.exception() in _cancel_drain_tasks so teardown always completes. Add a test that crashes the drain with a non-cancel exception and then calls stop() and respawns.
10. [Axis 7] The resolved delegate-stdio host is never checked against the documented 1.203.0 minimum version (src/coder_eval/agents/delegate_agent.py:298) — _resolve_host_bundle takes the first match: if candidate.name == _HOST_BUNDLE_NAME and candidate.is_file(): return candidate (line 298). This includes any global delegate-stdio on PATH, such as an older install left by the sibling coder_eval_uipath plugin. DELEGATE.md and package.json (^1.203.0) require 1.203.0 for the auth init option and the usage frame. The adapter removes AUTH_TOKEN from the host env and sends auth only as an init option, so an older host fails at init with an auth error. The _INIT_CONFIG_ERROR_HINT then points at env-var names, not at the version. Read version from the package.json beside the bundle (<bundle>/../../package.json). Raise AgentConfigError that names the found version and the floor, or log a warning if the file cannot be read.
11. [Axis 7] The host's own env names are removed from the host env without a log line, so a user who follows the package README is rerouted with no message (src/coder_eval/agents/delegate_agent.py:539) — for name in _HOST_ENV_REMOVED: env.pop(name, None) (lines 539-540) removes BACKEND_URL, ORG_LOGICAL_NAME and TENANT_NAME. coder_eval does not read these names as fallbacks. The PR calls the delegate-stdio README the wire-protocol SSOT, and that README documents these names. A user who sets BACKEND_URL=http://localhost:5002 the way the README says gets no backendUrl override: the host connects through DELEGATE_ENV or its own default, and nothing is logged. Log one WARNING for each removed name that was set and is not a credential (for example 'BACKEND_URL is ignored; set DELEGATE_BACKEND_URL'), as _strip_redundant_gateway_creds already logs its own removals.
12. [Axis 7] The user-facing env-var table and .env.example no longer name the install-override variable DELEGATE_STDIO_PATH (docs/USER_GUIDE.md:432) — On main the USER_GUIDE env table listed DELEGATE_SDK_NODE_MODULES / DELEGATE_SDK_PATH. The new row (line 432: "DELEGATE_ENV / DELEGATE_BACKEND_URL / DELEGATE_AUTH_TOKEN / ... | Delegate agent backend routing & auth") removes them and adds no row for the replacement DELEGATE_STDIO_PATH. .env.example line 93 also says only 'coder_eval finds the install itself.' A user whose old DELEGATE_SDK_PATH is now ignored without a message must find the rename in DELEGATE.md's 'CI only' note. Add DELEGATE_STDIO_PATH to the USER_GUIDE row and add a commented example to .env.example.
13. [Axis 8] Delegate init options let sdk_options overwrite framework-owned keys; only the model validator prevents it, and the downstream plugin skips that validator (src/coder_eval/agents/delegate_agent.py:588) — _build_init_options sets "workingDirectory": self.working_directory (583) and options["model"] = self.config.model (587), then runs options.update(self.config.sdk_options) (588). Any sdk_options key therefore overwrites workingDirectory, enableComputerUse or model. Today only DelegateAgentConfig._validate_sdk_options_keys (allowlist {"effort"}) stops this. Cross-repo interaction introduced here: DelegateAgentConfig now declares sdk_options, so coder_eval_uipath's _overrides_patch._kind_supports_sdk_options (true for any non-claude-code kind that declares the field) now also takes over in-tree delegate. Its _merge_sdk_options applies the merge with task.agent.model_copy(update={"sdk_options": merged}), which skips pydantic validation. With that plugin installed, --type delegate -D agent.sdk_options.model=X (or ...workingDirectory=/) passes unvalidated. The host then gets a model, or a cwd outside the sandbox, that differs from the recorded agent_config.model, and core lineage is not written. Fix: apply sdk_options first and set the framework-owned keys after it, or filter with _DELEGATE_SDK_OPTION_FIELDS at the point of use, so the agent does not depend on every writer validating. Separately, the downstream patch 1 is now redundant with this PR's registry-based guard (_kinds_accepting_sdk_options) and can be deleted when the pin moves past this release. Nightly: not affected (nightly uses delegate-sdk).
14. [Axis 8] REPORT_SCHEMA says sdk_options is null on non-Claude/non-Delegate agents, but the nightly's plugin agents persist non-null sdk_options (docs/REPORT_SCHEMA.md:145) — The new text says the shape is "the host's init options with credentials redacted on Delegate, null on the other agents". REPORT_SCHEMA.md is the cross-repo contract for task.json consumers (the coder-eval-uipath eval-runner and dashboard). The nightly's plugin agents delegate-sdk and studio-web override get_sdk_options() (coder_eval_uipath delegate_sdk_agent.py:1793 return self._init_options). Their rows carry non-null sdk_options with that plugin's own shape, and this repo does not control whether it is redacted. Reword to cover built-in agents only, e.g. "null on the other built-in agents; a plugin agent defines its own shape", so downstream consumers do not assume null (or redaction) for plugin kinds.
15. [Axis 8] HARNESS_PARITY calls delegate-sdk records historical and drops the open timing-bounds action, but the nightly still produces them (docs/agents/HARNESS_PARITY.md:522) — The PR removes the 'Known divergences' bullet for out-of-tree delegate-sdk, which included the open action "Mirror the Codex change in coder_eval_uipath (audit P3-1)". It adds a paragraph that puts delegate-sdk under "Older records ..." and calls it "the out-of-tree agent in coder_eval_uipath that the built-in delegate replaced" (lines 520-524), ending "No built-in agent writes this shape now" (531-532). But the coder-eval-uipath nightly still runs type: delegate-sdk (experiments/delegate.yaml, eval_runner skills/run.py), and that agent still sets no execution_started_at/execution_completed_at. Every new nightly Delegate row therefore still books its tool time as Unaccounted. Reword the paragraph to say current delegate-sdk records still have this shape, and keep the P3-1 action open, so people comparing timing buckets across agents do not take the nightly's Unaccounted inflation for historical data.

What's Missing

Nightly pipeline:

• 🟡 The PR does not say what happens to the nightly. Its feat(delegate)! commits bump the next release to 0.13.0 (semantic_release major_on_zero = false, main is 0.12.10). coder_eval_uipath pins coder-eval>=0.9.5,<0.13.0 (its pyproject.toml:11), so the nightly will not take this release, or any fix released after it, until someone moves the pin. Moving the pin turns on several cross-repo effects at once. (a) _overrides_patch._kind_supports_sdk_options starts to handle in-tree delegate sdk_options overrides without validation. (b) eval_runner/scripts/autopilot/ci/ado-run.sh:296 already exports DELEGATE_STDIO_PATH for delegate-sdk, and the in-tree delegate now reads the same name, with a 1.203.0 minimum version. (c) The REPORT_SCHEMA and HARNESS_PARITY wording about delegate-sdk changes. Add a 'Nightly impact' paragraph to the PR body that covers the version bump, the pin and these three effects. (trigger: pyproject.toml)

Parallel paths:

• 🔵 HARNESS_PARITY.md:523 says the built-in delegate replaced out-of-tree delegate-sdk, but the two config surfaces were not aligned. (1) Both agents read DELEGATE_STDIO_PATH, but only the plugin reads DELEGATE_STDIO_NODE_MODULES, so the built-in agent ignores it without a message. (2) The built-in resolver probes a global install on PATH. The plugin's .azure-pipelines/README.md:353 says its resolver never does. (3) delegate-sdk accepts unknown sdk_options keys and ignores them (DELEGATE_AGENT_GUIDE.md:151: 'one YAML can drive both a Claude and a delegate-sdk variant'). The built-in agent rejects every key except effort. I checked this: experiment defaults with sdk_options: {effort: high, max_thinking_tokens: 8000} make every delegate variant fail with a ValidationError. Add a migration note from delegate-sdk to delegate in DELEGATE.md, or make the two surfaces the same. (trigger: docs/agents/HARNESS_PARITY.md)
• 🔵 The contract change ('the shape depends on the agent') is in EvaluationResult.sdk_options (results.py:675) and REPORT_SCHEMA.md. Two places that define or read the field were not updated. Agent.get_sdk_options in src/coder_eval/agent.py:308 still says 'raw SDK options used for the last agent query'. resolve_agent_settings in reports/markdown.py:52 still says 'Prefers sdk_options (full SDK dump)'. Update both docstrings so they state the per-agent shape. (trigger: src/coder_eval/models/results.py) (restates: Axis 5: Delegate's get_sdk_options() init dict is rendered by Claude-shaped report code, which drops the Plugins row from Delegate reports)

Tests:

• 🔵 No config-merge test runs Delegate sdk_options through layers 1-4. test_merge_characterization.py has only a changed error-message match, and test_overrides_engine.py tests only layer 5 (-D). Two cases are not pinned: (a) effort: high in experiment defaults reaching a delegate variant, which is the 'works for both agents' claim at DELEGATE.md:174; (b) Claude-only keys in experiment defaults failing a delegate variant, which I reproduced. _merge_dict_by_model takes its strategy from matching[0], and this field is shared with ClaudeCodeAgentConfig. Add both cases to test_merge_characterization.py. (trigger: src/coder_eval/models/agent_config.py) (restates: Axis 2: DelegateAgentConfig.sdk_options replaces the typed effort field with an untyped one-key dict and a hand-written validator, unlike the sibling agents)

Downstream consumers:

• 🔵 Run-level reports pick the Agent Settings section from the first row that has data: reports/markdown.py:57 sdk_opts_list[0] and reports/html.py:1202 next(r for r in eval_results if r.sdk_options or ...). Before this PR, only Claude Code rows had sdk_options. Delegate rows now have it too. In a mixed Claude/Delegate A/B experiment, the report can show Delegate's init dict as the settings for the whole run, depending on row order. Neither reporter was changed, and no test covers a mixed-agent run. (trigger: src/coder_eval/agents/delegate_agent.py) (restates: Axis 5: Delegate's get_sdk_options() init dict is rendered by Claude-shaped report code, which drops the Plugins row from Delegate reports)
• 🔵 The text of the sdk_options guard error is now built from the sorted registry: 'only supported for agents.' coder_eval_uipath tests/test_overrides_patch.py:89 still asserts match="only supported for claude-code". That test passes only while claude-code sorts first among the kinds that accept sdk_options. A plugin kind whose name sorts before it (for example agent-x) would break the downstream test without warning. Tell the downstream repo about the new message, or keep a stable message prefix. (trigger: src/coder_eval/orchestration/overrides.py)

Display & mapping dicts:

• 🔵 The row table in collect_agent_settings_rows (reports/markdown.py:104-145, also used by html.py) has no rows for the init keys that Delegate now persists: env, backendUrl, projectId, enableComputerUse and enableSkills/bundledSkillsPath. So the routing facts that get_sdk_options() records do not appear in the report. Because the init dict has no allowed_tools key, the report also prints 'Allowed Tools: (all)', and the Plugins row is gone. Add rows for the Delegate keys, or use a separate per-agent row mapping. (trigger: src/coder_eval/agents/delegate_agent.py) (restates: Axis 5: Delegate's get_sdk_options() init dict is rendered by Claude-shaped report code, which drops the Plugins row from Delegate reports)

Harness & Lint Improvements

Static checks (lint / type):

• [ce-lint] CE069 (new BaseRule tests/lint/rules/ce069_categorizer_whole_word_tokens.py, wired in tests/lint/runner.py and added to the pyproject CE list): in src/coder_eval/errors/categorization.py, forbid a pat in error_str / any(pat in error_str for pat in [...]) membership test whose literal is all digits ("401", "429", "502", "503", "504") or is 3 characters or fewer ("oom", "git", "pip"). Such tokens must go through one shared whole-word helper (re.search(rf"\b{tok}\b", s)). The rule scans the list and tuple literals that feed in error_str. Prevents: [A6/A1 medium] 'Delegate SDK init failed: connect ECONNREFUSED 127.0.0.1:54013' and the GUID 'c7a3f401-...' categorize as AGENT_AUTH_ERROR (not retried) because categorization.py:81 matches the raw substring '401'. The same applies to the send path (delegate_agent.py:1172) and to a port 4401 in the SSE tail. It also catches the verifier's side note that 'boom' categorizes as OUT_OF_MEMORY through 'oom' (categorization.py:118). The second classifier _INIT_CONFIG_ERROR_STATUS = re.compile(r"\b40[13]\b") (delegate_agent.py:128) exists only because the shared categorizer was not whole-word.
• [ce-lint] Delete before guard: give AgentCrashError an explicit category: ErrorCategory | None (or retryable: bool) that _categorize_by_exception_type reads before _categorize_by_message. Then add CE070 (rule tests/lint/rules/ce070_no_message_steering_of_categorizer.py): under src/coder_eval/agents/, flag re.sub(...) / str.replace(...) whose pattern argument is a string literal that also appears in categorization.py's pattern lists (load that set from the module AST so the two cannot drift), and flag f-string/concatenated text in a raised AgentCrashError(...) that adds a categorizer token ('content filter', 'connection', 'timeout') as a literal. Known blind spot: tokens built at runtime. Prevents: [A5 low] _describe_host_error changes the retry category by editing message text: re.sub("timeout", "time-out", ...) at delegate_agent.py:402, and the added 'content filter' / 'connection' wording that categorization.py:104/108 then matches. The category of shipped defect is the one the notes record for build 13599116 ('guardrail' in a task path made a transient error non-retryable).
• [ce-lint] CE071 (rule tests/lint/rules/ce071_no_raw_stderr_logging.py): under src/coder_eval/agents/, a value read from a subprocess stderr stream (<x>.stderr.readline() / .read() / iteration, plus a name assigned from it and decoded) must not reach logger.<level>(...), list.append into a *_stderr* attribute, or an f-string in a raised exception unless it first passes through one shared redact_secrets(text, secrets) helper (new, in a shared module such as streaming/ or agents/_redact.py). AST taint is followed only inside one function body, and that is the documented blind spot. Prevents: [A4 medium] With DELEGATE_STDIO_VERBOSE=1 the host echoes the init frame with auth.accessToken. _drain_stderr keeps that line (delegate_agent.py:1275) and logs it at DEBUG (:1276) into task.log and the HTML log tail. _log_stderr_tail (:1132) repeats it at WARNING on a crash. Any future agent that drains a credential-holding child's stderr gets the same gate.
• [ce-lint] CE072 (rule tests/lint/rules/ce072_no_raw_urlparse_hostname.py): forbid urlparse(...).hostname / .port / .netloc attribute access anywhere in src/coder_eval outside one helper url_host(url: str | None) -> str | None (new, for example in path_utils.py or a small urls.py). The helper catches ValueError ('Invalid IPv6 URL') and returns None. Today there are 5 call sites: orchestrator.py:1761, codex_agent.py:1144 and :1499, delegate_agent.py:707 and :723. Prevents: [A6 medium] A malformed DELEGATE_BACKEND_URL ('https://[fd00::1:8080/api') makes get_sdk_options() raise ValueError at delegate_agent.py:707 inside the unguarded _finalize_result (orchestrator.py:1061). task.json is lost and the original init error is hidden. The rule also removes the same latent raise at delegate_agent.py:723 and the three Codex/orchestrator sites.
• [ce-lint] CE073 (rule tests/lint/rules/ce073_sdk_options_merged_first.py): in src/coder_eval/agents/, a <d>.update(<...>.sdk_options) (or {**..., **sdk_options}) must be the FIRST write to <d> in its function. Any subscript assignment or dict-literal key written to <d> before the update is a violation, because the user-authored passthrough can then overwrite a framework-owned key (workingDirectory, model, enableComputerUse). Prevents: [A8 low] _build_init_options sets workingDirectory (delegate_agent.py:583) and model (:587) and then runs options.update(self.config.sdk_options) (:588). Only the model validator stops an override, and coder_eval_uipath's _merge_sdk_options skips that validator through model_copy(update=...).
• [ce-lint] CE074 (rule tests/lint/rules/ce074_no_sdk_options_key_reads_in_core.py, reusing tests/lint/rules/_layers for the core/reports split as CE004/CE066 do): outside src/coder_eval/agents/, forbid .get("<literal>") / ["<literal>"] on a value that comes from get_sdk_options() or from a .sdk_options attribute. Core asks the agent through a typed Agent method (for example get_command_path() -> str | None, get_settings_rows()), so a per-agent dict with a different shape cannot be read as the Claude shape. Prevents: [A5 medium] orchestrator.py:1614 sdk_options.get("env") reads Delegate's DELEGATE_ENV slug string as ClaudeAgentOptions' env dict. Only isinstance(sdk_env, dict) saves it. reports/markdown.py:57 resolve_agent_settings and reports/html.py:1025 prefer the Delegate init dict and drop the Plugins row. [A3 low] This also removes the need for the separate pin test on the string env key.
• [ce-lint] Widen yaml_models_forbid_extras (or add CE075): in src/coder_eval/models/, a user-authored field annotated dict[str, Any] on a model that also has a field_validator doing an allowlist (set(v) - <frozenset>) or a per-key isinstance check must be a TypedDict with __pydantic_config__ = ConfigDict(extra="forbid") instead (the LocalPluginConfig pattern already in agent_config.py). An explicit EXEMPT entry with a reason covers true passthroughs such as ClaudeCodeAgentConfig.sdk_options. Pyright then types self.config.sdk_options["effort"] as str. Prevents: [A1/A2/A5/A7 low] DelegateAgentConfig.sdk_options: dict[str, Any] plus the hand-written _validate_sdk_options_keys (agent_config.py:419, 454-465), which must be edited by hand for every new key and has an Any static type and JSON schema. The 94e4a5a fix ('reject an sdk_options.effort that is not a string') is the hand-validator patch that the TypedDict makes unnecessary.
• [ruff] Turn on ruff C901 with [tool.ruff.lint.mccabe] max-complexity = 20 (below the current worst agent offender, per the cap-rule convention). Mark the existing offenders with a visible # noqa: C901 debt marker, the same way PLR0912/PLR0915 are already handled at max-branches=25 / max-statements=80. PLR0912 counts branches, not cyclomatic paths, so it does not fire on communicate today. Prevents: [A1 medium] DelegateAgent.communicate D(27) (delegate_agent.py:732) and _handle_event D(22) (:856) would carry noqa debt markers, and the next growth would fail make check. It does not catch the +3 regression of _spawn_and_init to C(13); the ratchet in the harness bucket covers that.
• [ci-gate] New whole-tree @pytest.mark.lint class in tests/test_custom_lint.py (CE076, npm supply-chain pin): every package.json under src/ must pin exact versions (no ^/~/*) and have a committed package-lock.json beside it. Every npm install / npm i step in .github/workflows/*.yml must be npm ci --ignore-scripts, and must not carry --safe-chain-skip-minimum-package-age unless an allowlist entry names the platform-binary package and gives a reason. Prevents: [A4 low] "@uipath/delegate-stdio": "^1.203.0" (src/coder_eval/agents/delegate/package.json:7) has no lockfile, and pr-checks.yml installs it with the age gate skipped, in the job that then passes it a live DELEGATE_AUTH_TOKEN.
• [ci-gate] Widen CE027 (tests/lint/doc_env_parity.py): add DELEGATE_ to FRAMEWORK_ENV_PREFIXES, and add the reverse direction. Every os.environ / os.getenv read of a framework-prefixed name in src/ must appear in the docs/USER_GUIDE.md environment table or in .env.example, unless an EXEMPT entry gives a reason. Also flag a name in a removal tuple (env.pop(name) over a module-level tuple) that a user-facing doc tells people to set. Prevents: [A7 low] DELEGATE_STDIO_PATH is consumed but missing from the USER_GUIDE.md:432 row and from .env.example:93. The renamed DELEGATE_SDK_PATH now fails without a message. The pop-tuple half also flags _HOST_ENV_REMOVED silently removing BACKEND_URL / ORG_LOGICAL_NAME / TENANT_NAME (delegate_agent.py:539-540), which the host README tells users to set.

Harness improvements (not statically reachable):

• Complexity ratchet in CI: a make complexity-ratchet target (run by make verify) that runs radon cc -j on the changed files at HEAD and at the merge-base, and fails when any function's cyclomatic score rises across a grade boundary (B->C, C->D) without a # noqa: C901-style debt note in the diff. Why not static: It compares two revisions. A single-tree AST rule sees only the absolute value, so it cannot tell that _spawn_and_init went from B(10) on origin/main to C(13) when C(13) is below any reasonable absolute cap. Prevents: [A1 medium] the +3 regression in DelegateAgent._spawn_and_init (delegate_agent.py:528) from the inline env scrub, and later small growth in communicate/_handle_event.
• Agent finalization fault-injection contract test, parametrized over every registered AgentKind (tests/test_agent_contract.py): start each agent with malformed routing env (bracket-typo URLs, empty strings, non-ASCII), drive Orchestrator._finalize_result, and assert that task.json is written and that the original error is the recorded one. Make _finalize_result guard each agent-provided getter (get_sdk_options, get_environment_info) so that one raising getter degrades to None and is logged. Why not static: It needs a live Orchestrator run and real exception flow through finally-blocks. CE072 removes the known urlparse shape, but any other raise in an agent getter only shows up at runtime. Prevents: [A6 medium] the malformed DELEGATE_BACKEND_URL ValueError at delegate_agent.py:707 that loses task.json via orchestrator.py:1061.
• Lift the Delegate golden-master exemption: add DELEGATE_SCENARIOS (happy path with a tool call, a max_turns cut with per-call usage frames, a crash on an error frame) to SCENARIOS_BY_AGENT in tests/test_agent_golden_master.py, built from the existing _ev/_result/_usage/_tool_call frame builders, and delete the _NO_GOLDEN_COVERAGE[AgentKind.DELEGATE] entry. Also give _NO_GOLDEN_COVERAGE entries a required until: marker (a tracking issue or a date) that a test checks, so a deferral cannot stay after its stated precondition ('frame shapes confirmed live') is met. Why not static: A golden snapshot diffs a full TurnRecord that EventCollector rebuilds from a replayed event stream. Field drift appears only when the stream is run. Prevents: [A3 medium->low] the stdio protocol rewrite is checked only field-by-field on a hand-picked subset. A field that stops being mirrored into task.json would drop without a failure.
• Report-rendering parity test over all agents: for each AgentKind with a fixture agent_config that has plugins, render the markdown and HTML agent-settings rows from a finished EvaluationResult (sdk_options included), and assert that the Plugins/Skills row and the Model row are present. Also check the PATH-sync result (command_base_path) against the agent's declared value. Why not static: The defect is the rendered output of a per-agent dict shape through shared renderers. CE074 forbids the key reads, but whether a row appears is a semantic property of the output. Prevents: [A5 medium] Delegate reports lose the Plugins row because reports/markdown.py:57 / reports/html.py:1025 prefer the init dict. [A3 low] It also pins that Delegate's string env key never writes command_base_path.
• Secret-leak canary in the Delegate (and every credential-taking agent) test suite: a fake host script that echoes its whole stdin init frame to stderr. Run a turn with a sentinel token and with DELEGATE_STDIO_VERBOSE=1, and assert that the sentinel is absent from caplog, from task.log, from the HTML log tail and from task.json. Run the same canary against the real host in the live CI job. Why not static: It needs the third-party host's runtime behavior (an obfuscated bundle whose trace output cannot be read statically) and the full logging pipeline. CE071 guards only the adapter side of the taint path. Prevents: [A4 medium] the accessToken echoed by DELEGATE_STDIO_VERBOSE=1 reaching task.log and the report through delegate_agent.py:1275-1276 and :1132.
• Mutation testing for agent adapters: a nightly (or make mutation-agents) mutmut/cosmic-ray run scoped to src/coder_eval/agents/delegate_agent.py, with a surviving-mutant budget. Then fix the test fixtures it exposes: create the .cmd shim file in test_global_install_found_beside_a_windows_cmd_shim and test_local_install_wins_over_global, so the _HOST_BUNDLE_NAME guard is what the test exercises. Why not static: Whether a guard is really tested depends on whether the test fails when the guard is removed. Only running mutated code shows that. Prevents: [A3 low] the bundle-name guard at delegate_agent.py:298, the 409 inverse guard, cancel/generic crash paths, the spawn cwd and the mid-run respawn error: branches whose removal keeps all tests green.
• Shared drain-task teardown helper for all subprocess agents (cancel_and_reap(tasks) that awaits each task, suppresses and logs any BaseException, and always clears the handles), plus a fault-injection test per agent: crash the stdout drain with a non-cancel exception (for example RecursionError from json.loads), then assert that stop() completes _mark_stopped() and that a respawn starts cleanly. Why not static: The failure needs a task that ends holding an exception and then a later await on it. That is runtime task state across calls, and an AST rule cannot see it reliably. Prevents: [A6 low] _drain_stdout re-raises (delegate_agent.py:1259-1262) and _cancel_drain_tasks suppresses only CancelledError (:681-682), so every later respawn and stop() raises the stale exception.
• Dialog-mode continuity check: in simulation mode, when an agent drops its backend conversation between turns, the agent must record a reset marker (for example environment_info['_session_resets'] with the iteration), and the scorer must flag or exclude rows whose trajectory is not continuous. Add a Delegate test: turn 1 succeeds, turn 2 gets a 409 'already being generated', and the test asserts the marker (or a terminal crash when iteration >= 2). Why not static: Context loss is a property of the backend conversation across turns. Detecting it needs the multi-turn run and the session id sequence. Prevents: [A8 medium] the 409 recovery at delegate_agent.py:1171 sets _session_id = None with only a WARNING. Later dialog turns continue without context, and the row is still graded as one continuous trajectory.
• Host version floor check at spawn: read version from the package.json beside the resolved delegate-stdio bundle and raise AgentConfigError (naming the version found and the 1.203.0 floor) when it is older. Also add an integration test that resolves a stale global install ahead of the floor. Why not static: The installed host version is a property of the machine, not of the source tree. Prevents: [A7 low] _resolve_host_bundle (delegate_agent.py:298) accepts any global delegate-stdio, for example an older one left by coder_eval_uipath. That host then fails init with an auth hint that points at env-var names and not at the version.
• Cross-repo contract job: a scheduled CI job that installs the pinned coder_eval_uipath plugin, runs one fixture task per plugin agent kind (delegate-sdk, studio-web), and checks the persisted task.json against REPORT_SCHEMA.md's per-field claims (sdk_options null or redacted, execution_started_at present) and against HARNESS_PARITY.md's list of agents that still write a given shape. Why not static: The records come from another repo's agents at runtime. This repo's docs can only be checked against what those agents actually write. Prevents: [A8 low] REPORT_SCHEMA.md:145 says sdk_options is null on non-Delegate agents, but the nightly's plugin agents persist non-null sdk_options. [A8 low] HARNESS_PARITY.md:520-532 calls delegate-sdk records historical and removes the open P3-1 timing action, but the nightly still writes them.

Top 5 Priority Actions

1. Make transient Delegate crashes retryable as the docs promise: at src/coder_eval/errors/categorization.py:81 the auth pattern matches the raw '401' substring, so 'Delegate SDK init failed: connect ECONNREFUSED 127.0.0.1:54013' (delegate_agent.py:579) and the send path (delegate_agent.py:1172) end as non-retryable AGENT_AUTH_ERROR; match status codes as whole words (or remove status-like digits from the reason), add categorize_error(...) is AGENT_CRASH assertions to the port and GUID cases at tests/test_delegate_agent.py:273, and correct docs/agents/DELEGATE.md:138 and :173.
2. Stop silent context loss after a 409: at src/coder_eval/agents/delegate_agent.py:1171 the agent sets self._session_id = None with only a WARNING, so in dialog/simulation mode turn 2 or later continues in a new backend conversation but still gets a grade as one trajectory; drop the session only when no earlier turn completed (iteration 1), let the crash stay terminal otherwise or record an explicit delegate_session_resets marker in environment_info, and add a test with a good turn 1 and a 409 on turn 2.
3. Guard the hostname extraction at src/coder_eval/agents/delegate_agent.py:707 (urlparse(...).hostname raises ValueError on a malformed IPv6 DELEGATE_BACKEND_URL): get_sdk_options() runs unguarded in Orchestrator._finalize_result (src/coder_eval/orchestrator.py:1061), so the error hides the original init failure and task.json is not written; catch ValueError and fall back to None (also harden line 723).
4. Keep the bearer token out of logs: with DELEGATE_STDIO_VERBOSE=1 the host echoes the init frame with auth.accessToken to stderr, and _drain_stderr (src/coder_eval/agents/delegate_agent.py:1275-1276, plus the WARNING tail at :1132) writes it to task.log and the HTML report; redact the known token or mask "accessToken":"..." before you append or log a line, add a caplog test, and add a warning at docs/agents/DELEGATE.md:56.
5. Make the framework-owned init keys win: at src/coder_eval/agents/delegate_agent.py:588 options.update(self.config.sdk_options) runs after workingDirectory/model are set, so an unvalidated writer (the downstream overrides patch uses model_copy) can send a different model or a cwd outside the sandbox than the one recorded in agent_config; apply sdk_options first or filter with _DELEGATE_SDK_OPTION_FIELDS at the point of use, then add DELEGATE_SCENARIOS and remove the exemption at tests/test_agent_golden_master.py:240 to snapshot the rewritten protocol.

---

Stats: 0 🔴 · 0 🟠 · 7 🟡 · 15 🔵 across 8 axes reviewed.

[Mihaiii]

@uipreliga Addressed the top 5 priorities, except priority number 4:

As mentioned in my previous comment, "The Autopilot host already redacts it. delegate_stdio.ts:1581 logs redactInitAuth(cmd). No coder_eval-side redaction is needed." - this happens in the Autopilot PR that goes hand in hand with this one.

I noticed @uipath/delegate-stdio 1.203.0-preview being mentioned and I think this is the source of the misunderstanding. Please use version 0.0.1-dev.37 from the internal feed to also capture the code from the equivalent branch. This is also the version used in the above mentioned pipeline run.