fix(sim): report failed, canceled and incomplete runs instead of a false pass

[scott-lowe-vapi]

Value

V.A.L.U.E. tier: project — PR 2 of 10 for inline simulation PR checks (TEST-141). Stacked on #57.

• Problem: npm run sim reported every run as passed. It read a results field the simulation-run API doesn't return and counted status === "pass" (items are passed/failed), so it always summarised 0/0 and exited 0, failing runs included.
• Who it affects: anyone gating on npm run sim, locally or in CI. The PR check (later in this stack) reuses this verdict, so it has to be right.
• What changes:
  • A strict verdict (src/sim-result.ts).
  • Item fetching that handles both response shapes and late results.
  • The run link printed, plus each failing judge with expected vs extracted values.
  • --timeout and Ctrl-C both cancel the run.
  • Exit codes: 0 passed, 1 failed, 2 usage, 3 incomplete.
  • A config-free client that never retries run creation on a 5xx, because the run may already be queued.

Evidence of value

Same stub API, one passed and one failed item:

	Result
main's sim.ts	{"pass":0,"fail":0} → exits 0 (false green)
This branch	failed — 1 of 1 simulations failed → exits 1

Live, against a test org (chat transport):

Suite	Exit	Output
Designed to fail (judge: "open 24 hours?")	1	✗ … open-24h (expected = true, got false) — run
Designed to pass (judge: "open 8–5 on Fridays?")	0	passed — 1 of 1 simulations passed — run

The temporary resources were deleted afterwards.

Testing plan

• tests/sim-result.test.ts is a verdict table covering:
  • the old false-green shape (no results);
  • 0 items, a short item list, and a count mismatch;
  • a failed item, with the failing judge listed;
  • canceled items;
  • all required evaluations skipped, and an optional skip alongside a scored required judge;
  • missing itemCounts, and a run that hasn't ended.
• tests/sim-run.test.ts runs runSimulation against a local HTTP stub:
  • pass, and fail using the bare-array item shape;
  • late item results;
  • timeout cancels the run, and an interrupt cancels it with the 400 "already ended" swallowed;
  • no retry of a 502 on create;
  • --no-watch;
  • pagination with overlapping pages deduped.
• npm run build and npm test pass (377 tests).
• Not tested: a live run that's still running when it's canceled (cancel was only exercised against the stub), and voice transport (the live runs used chat). Default behaviour change: unknown CLI arguments are now an error (exit 2) instead of being silently ignored.

Refs TEST-141

🤖 Generated with Claude Code

[scott-lowe-vapi]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• feat(promotion): gate promotion out of an org on a passing check #66
• feat(check): add the opt-in Vapi checks PR workflow and user docs #64
• feat(check): run checks live and report Vapi Evals statuses #63
• feat(check): fail closed on tools and servers in inline check payloads #62
• feat(check): build inline simulation-run payloads and add a dry-run CLI #61
• feat(check): parse vapi-checks.yml check configuration #60
• refactor: move resource parsing and org connection helpers into config-free modules #59
• fix(sim): report failed, canceled and incomplete runs instead of a false pass #58 👈 (View in Graphite)
• ci: type-check tests/ and send a User-Agent on sim API requests #57
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.