[abrichr]

What this adds

openadapt bench medplum --agent <scripted|claude|cmd:...|py:...> is a public benchmark that a vendor reruns with their own agent. It runs against a local Medplum (Apache-2.0, pinned by digest, loopback only, synthetic people).

• 37 cases (packages/flow/benchmark/fhir_medplum/public/cases.yaml), written for this benchmark, across Patient, AllergyIntolerance and Appointment writes:
  • 14 clean, including three without the element hint and three with another user's writes;
  • 23 faulted: wrong value, wrong patient, duplicate, lost reply, refused write, fake 201, collateral, late collateral, and pre-existing duplicates.
• A fault proxy sits between the agent and Medplum and applies each case's fault once.
• An end-state oracle reads the server before and after each case, with its own comparisons, not the verifier's.
• Four judges on the same run:
  • the agent's claim;
  • a read-back of the record the agent names;
  • OpenAdapt: the agent's claim, gated by the ensure_record verifier;
  • the verifier alone.
• The scorecard shows harms caught and missed, false stops, and person minutes from the fair test's costs_2.json, with a per-class breakdown.

Results (RESULTS.md)

Claude Sonnet 5.5 ($1.22, 20 min):

• OpenAdapt missed 1 of 9 harms; the agent's claim and the read-back missed 8. The one miss is A10, a record written after the check.
• OpenAdapt had 9 false stops. In each, Claude had corrected the proxy's changed value itself with a PUT, and the contract sends any change after the create to review.

Scripted agent: OpenAdapt missed 1 of 18 harms; the read-back missed 8.

Cold run of the README by a subagent with no context: 5 min 12 s from clone to scorecard, with the images already pulled.

Verifier changes (ensure_record, effects/fhir.py)

• An instant comparison.
• List filters with a dotted key and ^=, which read an Appointment's patient and practitioner participants apart.
• A filter on a missing key no longer matches the text None.
• An identity value with no search parameter is compared instead of searched. Before, it raised KeyError; the live run found this.

Review

An adversarial review of the fault set and the scorecard maths ran before the live runs. All eight of its should-fix items, and most of the minor ones, are addressed in bench(medplum): address the adversarial review. Its open-core and licensing checks found nothing; the one phone number it shared with a private set has been changed.

Tests

• packages/flow/tests/test_bench_medplum_public.py
• tests/test_bench_medplum_command.py
• new cases in test_ensure_record.py and test_fhir_extract_path_filter.py

[abrichr]

Adversarial review of the case set and the scorecard maths

An independent reviewer read the code, not this PR's conclusions, before any live run. It found no blocker. It raised 8 should-fix items, 3 of which tilted the comparison toward OpenAdapt, plus minor items. Dispositions are in commit bench(medplum): address the adversarial review.

#	Finding	Disposition
1	The OpenAdapt arm ignored the agent's claim; the read-back arm didn't	Fixed. OpenAdapt = agent's claim AND verifier. The verifier alone is a separate column.
2	Verifier read failures counted as stops	Fixed. They are left out as infra and listed.
3	A _search or $ POST could use up the fault	Fixed. Skipped; the proxy records whether each fault was delivered.
4	Undelivered faults were still scored	Fixed. Left out as not_run and listed.
5	+1 phone judged harmful by the oracle	Fixed in the oracle. The verifier stays strict; a false stop there is counted.
6	P10 untouched cost 420 s as a caught harm	Fixed. It is now a right stop, 120 s.
7	The element hint hides false stops	Three clean cases without the hint (P12, A11, AP12); README states the limit.
8	The case mix favours a whole-server diff; missing classes	Added a fake 201, a wrong patient on an appointment, and another user writing the same patient at write time. Per-class counts added. Update tasks not covered; README says so.
minor	Links bypass the proxy; read-back parses _history; oracle blind spot; unlocked log; early stop unreported; phone suffix shared with a private set	All fixed. The phone number was changed.
minor	A10 can't be passed; P11/P13 stop-vs-pass cost inversion; found-case benign re-save	Kept and documented in the README.

The verifier change (instant compare, prefix filter, leaf counting) was judged fail-closed. Open-core and licensing checks were clean.

The live run then found one more verifier bug (an identity value with no search parameter raised KeyError), fixed with a test before the recorded runs.