[austinintelligence]

What changed

Adds cargo xtask bench-model: runs a real agent CLI (command template with {prompt}, {mcp}, {model}) through 15 device goals in tools/bench/tasks.json, each with reset commands and verifiers that decide success from au reads of device state. Two au builds (--a, --b) are run interleaved, --runs per task (default 5). MCP calls are counted by a relay shim (cargo xtask bench-shim) the agent launches as its MCP server. The report gives, per build and per task, success rate, calls, input tokens (when the agent CLI reports them), and wall time as mean with 95% CI, plus A−B with a Welch interval, as JSON and a table. Spend guard: --max-runs, --max-cost. Documented in docs/contributing/testing.md, including how to take the 1.2.0 baseline. CHANGELOG line added.

Why

Every later 1.3.0 gate needs a measurement with a real model, not the scripted benchmark-live policy.

Safety and compatibility

• No device-specific serials, tokens, private paths, media, or signing material are included.
• No new way for an agent to run a shell, adb, or page JavaScript; command forms changed in computer/src/spec.rs only. (Contributor tooling only; the agent still gets the unchanged au serve --mcp.)
• Destructive/privacy-sensitive behavior has explicit confirmation coverage. (Tasks avoid account, camera, microphone, and location actions.)
• Temporary files, processes, emulators, and device state have cleanup coverage. (Agent is killed on timeout; shim exits with its client; files stay under target/bench-model/.)

Verification

• cargo fmt --all -- --check
• cargo clippy --workspace --all-targets --all-features -- -D warnings
• cargo test --workspace --all-targets
• cargo xtask verify — cargo xtask check, docs, and size pass (automation 2,128 of 2,350); verify stops here at npm run lint because oxlint is not installed in this container.
• npm test and npm run lint — not run (no oxlint).

New unit tests (tools/src/bench_model.rs), each failing before the module existed:
the_shipped_task_file_parses_with_a_verifier_per_task, a_template_fills_placeholders_and_keeps_the_prompt_one_argument, calls_are_counted_from_the_shim_log, usage_sums_input_token_fields_and_reads_cost, runs_interleave_builds_and_flip_order_each_round, confidence_intervals_use_the_t_distribution, the_report_compares_builds_per_task_and_overall.

Also smoke-tested without a device: the shim relays a real au serve --mcp session and logs both calls, and a stub agent run reports tokens and trips the cost guard.

Release impact

• No breaking change.
• Breaking change documented in the changelog and migration guidance.

Not done here

• The actual 1.2.0 baseline run (needs an emulator and agent API keys).
• Task verifier strings (for example Use Wi-Fi=off, Pair new device) are written from the documented screen format and have not been checked against a live emulator; expect to adjust some on the first real run.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YAbe34cvkmUtsvXKH65Cq1

---

Generated by Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 5d2ef323-a0e3-4918-b7a5-d001c6bc01af

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}