Skip to content

Run 3 levers: generate FHIR ensure-record contracts, and a one-tap stop card - #1653

Merged
abrichr merged 12 commits into
mainfrom
claude/contract-autogen-run3
Oct 29, 2026
Merged

abrichr merged 12 commits into
mainfrom
claude/contract-autogen-run3

Conversation

@abrichr

@abrichr abrichr commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

Fair test run 1 lost on person time: 30 minutes of contract setup and 420 s per caught harm. This PR goes after both costs and measures the result blind.

What it adds

  • openadapt_flow/generate/ensure_contract.py. Generates an openadapt.ensure-record/v1 contract from three demonstrations, the CapabilityStatement and the SearchParameter definitions. It regenerates all 16 hand-written Medplum Patient lines. It refuses when unsure, and a type with no reviewed identity template needs a person to confirm.
  • benchmark/stop_card/. A stop card that shows the exact difference and who else wrote. It offers Undo or Fix (each reads back) or Open in the app. Its cost model was pre-registered in PREREG_REVIEW.md before any timing.
  • benchmark/contract_autogen/. The blind protocol (BLIND.md), the recorded adversarial review (REVIEW.md), and the run.

Blind run 1 (Immunization, independent author, suite frozen by sha256)

  • Setup: 78 s of person time, 2 lines edited. The generator proposed identity [patient, vaccine_code]. That is wrong: the task's same-fact rule also needs the date. The person declined it and added the date. The generator did not admit the contract; it has no input for a different rule. Two deviations are recorded in BLIND.md.
  • Suite: 23 of 24 right, 0 silent passes, 1 false stop. The false stop comes from a look-alike that the contract can't exclude by status. That needs a verifier change; none is made here.
  • Card: 16 stops, 14 one-tap, 0 failed read-backs. Modelled 42.3 s per caught harm (×1) against V1's 420, with a pre-registered bar of 210.
  • Card defect: the false stop got Undo, which deleted a correct record. Per the pre-registration, the card isn't used in run 3 until this is fixed and re-measured.

Full numbers: packages/flow/benchmark/contract_autogen/results/RESULTS.md. Per-case results are private in openadapt-internal.

Not changed

The verifier (ensure_record.py), runtime/effects, identity and the compiler are untouched. No refusal is loosened. Run 1 and run 2 are unchanged.

Follow-ups

  • A status filter in ensure-record identity (excluding entered-in-error records). This is a verifier change and needs an adversarial review. It fixes both the false stop and the card defect.
  • A generator input for a person-supplied identity rule, so that an edit is admitted rather than hand-applied.
  • If-Match on card writes, to close the read-then-write race.
  • Casefold of non-ASCII in ensure_record (review finding 9).

@abrichr
abrichr marked this pull request as ready for review October 29, 2026 20:44
@abrichr
abrichr merged commit e5a412d into main Oct 29, 2026
45 checks passed
@abrichr
abrichr deleted the claude/contract-autogen-run3 branch October 29, 2026 21:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant