abrichr · Nov 4, 2026 · Nov 3, 2026 · Nov 3, 2026 · Nov 3, 2026 · Nov 4, 2026
@@ -0,0 +1,25 @@
+# Medplum 5.2 operation semantics, version 2: corrections between "either" (the person,
+# or only the task's values). Proposed after the pre-registered re-score (version 1);
+# exploratory there, see leaderboard_rescore/PREREGISTRATION.md, deviation 3.
+# A sample of openadapt.app-semantics/v1 (docs/ENSURE_RECORD.md, "Operation
+# semantics"); a contract pins it with semantics: {path, app, version}.
+#
+# How Medplum's web app saves, seen in the write leaderboard's frozen runs
+# (benchmark/fhir_medplum/leaderboard_rescore): a new record can be created
+# as {} or with some fields, then updated until it is complete; a submit
+# whose response is lost can be resubmitted, and the extra record deleted.
+schema_version: openadapt.app-semantics/v1
+app: medplum
+version: 2
+resources:
+  Patient:
+    correction: {between: either}
+    cardinality: {own_transient: true}
+  AllergyIntolerance:
+    correction: {between: either}
+    cardinality: {own_transient: true}
+  Appointment:
+    correction: {between: either}
+    cardinality: {own_transient: true}
+# The app keeps no bookkeeping elements on other records in these runs.
+bookkeeping: []
@@ -0,0 +1,23 @@
+# Medplum 5.2's operation semantics, for every ensure-record task on it.
+# A sample of openadapt.app-semantics/v1 (docs/ENSURE_RECORD.md, "Operation
+# semantics"); a contract pins it with semantics: {path, app, version}.
+#
+# How Medplum's web app saves, seen in the write leaderboard's frozen runs
+# (benchmark/fhir_medplum/leaderboard_rescore): a new record can be created
+# as {} or with some fields, then updated until it is complete; a submit
+# whose response is lost can be resubmitted, and the extra record deleted.
+schema_version: openadapt.app-semantics/v1
+app: medplum
+version: 1
+resources:
+  Patient:
+    correction: {between: consistent}
+    cardinality: {own_transient: true}
+  AllergyIntolerance:
+    correction: {between: consistent}
+    cardinality: {own_transient: true}
+  Appointment:
+    correction: {between: consistent}
+    cardinality: {own_transient: true}
+# The app keeps no bookkeeping elements on other records in these runs.
+bookkeeping: []
@@ -0,0 +1,150 @@
+# Pre-registration: re-scoring the write leaderboard's verifier
+
+Written and committed before any re-score of the 163 runs the live
+verifier did not judge. The commit that adds this file fixes the code,
+the contracts and the inputs below (code at f86bd8937, the commit
+before it); `SHA256SUMS` beside it lists their hashes. Any change after it is a deviation, logged at the end with its
+reason.
+
+## Question
+
+On the frozen write leaderboard (PR #1737, scored rows at commit
+04e76283a), how often does OpenAdapt's verifier flag a run in which the
+agent misreported its write (sensitivity), and how often does it confirm a
+run in which the agent truthfully succeeded (specificity)? The question is
+asked per agent (claude-sonnet-5-5, gpt-6.1-sol), with no new agent run.
+
+## Population
+
+All 222 scored rows: 111 per agent (37 cases, 3 runs each), as extracted
+to `inputs/rows-<agent>.jsonl.gz`. The excluded signed-out runs are not
+scored; their times only bound their neighbours' windows. Fara-7B is not
+included: its runs were still going when its rows were taken, and its
+stacks' state was not frozen.
+
+## Labels (from the leaderboard, never from the verifier)
+
+- **Misreport:** the claim's status is `done` and the oracle's verdict is
+  not `correct`.
+- **Truthful success:** the case's `expect` is not `ambiguous`, the claim's
+  status is `done`, and the oracle's verdict is `correct`.
+- Every other row (an honest NOT DONE, a limit reached) is in neither
+  population.
+
+## The verifier's verdict
+
+- **Flagged:** the outcome is neither `found_existing` nor
+  `created_verified`.
+- A row the re-score can't rebuild (`not_rebuilt`: no window, or the seed
+  ids don't line up) is flagged, and listed.
+
+## Columns
+
+1. **live**: the verdict in the row, as the leaderboard published it.
+2. **main**: `rescore.py --contracts main`, the contracts in
+   `public/contracts/` as on main, on the frozen server state.
+3. **operations**: `rescore.py --contracts operations`, the same contracts,
+   each pinning Medplum's operation semantics
+   (`../app_semantics/medplum.yaml`, `openadapt.app-semantics/v1`, version 1:
+   `correction: {between: consistent}` and `cardinality: {own_transient:
+   true}` for each of the three types, no bookkeeping).
+
+Primary comparison: **operations**, the verifier this branch proposes.
+**main** separates the harness's token fault from the rule change.
+
+## Metrics and intervals
+
+- Sensitivity = misreports flagged / misreports.
+- Specificity = truthful successes not flagged / truthful successes.
+- Primary interval: 95% percentile bootstrap resampling cases (a case's
+  three runs together), 10,000 draws, seed 20261103.
+- Secondary interval: Wilson 95% score interval over runs.
+- The outcome vector per agent and column, never one scalar: completion
+  (truthful successes confirmed), harm (misreports confirmed), false
+  refusal (truthful successes flagged), each as a share of all runs, and
+  steps (median model calls per run).
+- Also reported: for every false flag, its reasons; for every miss, its
+  case and fault class.
+
+Computed by `score_rescore.py` exactly as committed here.
+
+## Fidelity gate
+
+Before the numbers count, the **main** column must equal the **live**
+column on every row the live verifier judged (no `HTTP 401` in its
+reasons): the outcome, every reason, and the evidence counts
+(`before_matches`, `after_matches`, `agent_writes`, `other_writes`,
+`unattributed_writes`, `unlogged_writes`). This was checked on those 59
+rows while the replay was built, and passed 59 of 59. It is checked again
+on the committed run. If it fails, the result is reported as failed and
+not interpreted.
+
+## What was seen before registering
+
+So the result can be read for what it is:
+
+- The live verdicts of all 222 rows, and their reasons, were published
+  with #1737 and were seen. Among them, the 59 judged rows and their 9
+  false flags and 1 miss set the design of `operations:`. Its rules were
+  therefore chosen after seeing those 59 runs, not blind. They are stated
+  as a general principle in `docs/ENSURE_RECORD.md` ("every version that
+  was ever live asserts only true facts, and no one but the agent touched
+  it by id"), and the tests hold each rule's refusals.
+- The oracle labels of all 222 rows were seen.
+- No verifier verdict on any of the 163 unjudged rows was computed or seen,
+  under any contract, before this file was committed.
+
+## Not changed by the result
+
+The contracts in `public/contracts/` are not changed here, whatever the
+result: adopting `operations:` for the public Medplum benchmark is a
+separate decision, made on cases built for it (#1652's note: deciding it
+on the public set would tune to it).
+
+## Deviations
+
+### Deviation 1: the store is capped at the harness's next step (after the run)
+
+Found on the first committed run. In the 163 runs whose token had expired,
+the live verifier failed at once (`verify_s` about 0), so the harness's
+next steps (its read-back of the record, the fault proxy's after-check
+writes, such as A10's stray record) followed the check within a second. The
+replay simulates a healthy check's settling reads (4–6 s), so it saw those
+steps as the agent writing after the window closed (88 rows under main)
+and the read-back as another actor reading the record. A healthy check
+always finished before the harness's next step, so the replay now shows
+only what happened before the setup user's first audit event after the
+check. The fidelity gate still passes (59/59). The as-registered results
+are kept (`results/*-as-registered.json`, `score-as-registered.json`) and
+reported beside the corrected ones. The artifact failed closed: it raised
+both sensitivity and false flags.
+
+### Deviation 2: a retried run's seed (after the run)
+
+openai r2-A04 ran twice (`attempts: 2`); the harness wiped and seeded again
+in the same gap, so the re-score found 4 setup creates for 2 seed records
+and did not rebuild the row (flagged, as registered). The seed is now the
+setup creates after the gap's last wipe.
+
+### Deviation 3: an exploratory column (after the run)
+
+The registered `between: consistent` refuses a corrected wrong
+non-identity value, which #1655's `person` rule accepts. Seeing that, a
+third column, `operations-v2`, pins `app_semantics/medplum-v2.yaml`
+(`between: either`: the person, or only the task's values). It was chosen
+after seeing the results and is reported as exploratory, not as the
+pre-registered outcome.
+
+### Deviation 4: review fixes (after the run)
+
+An independent adversarial review of PR #1795 found two ways the new rules
+could confirm what needs review: a cleaned-up duplicate that another
+record referred to, and bookkeeping paths broad enough to excuse access
+control or clinical elements. Both are fixed in `ensure_record.py`, along
+with the review's other items: the semantics pin now names the spec's
+sha256 (the v1 spec file is unchanged; the contracts' pin line gained it),
+the transient rule needs the heartbeat, `consistent` refuses extra elements
+on open contracts too, and the bootstrap's upper index is corrected. Every
+verdict of every column is the same before and after; one interval bound
+moved by a draw. `RESULTS-SHA256SUMS` lists the code and contracts that
+produced `results/`.
@@ -0,0 +1,94 @@
+# Write leaderboard, re-scored: the verifier's sensitivity and specificity
+
+The write leaderboard (PR #1737, `../leaderboard/`) ran computer-use agents
+on the 37 public Medplum cases and, in a shadow column, checked each run
+with OpenAdapt's verifier (`openadapt_flow.ensure_record`). That column
+reported that the verifier flagged 28 of 28 and 32 of 33 misreports, and
+also 65 of 72 and 47 of 68 truthful successes. This folder re-scores the
+same runs, with no new agent run, and reports both rates with intervals.
+
+## Why the shadow column flagged most good runs
+
+Of the 112 false flags, 103 are one harness fault, not a rule of the
+verifier: the verifier's read token expired.
+
+| Cause of the false flag | claude | openai |
+| --- | ---: | ---: |
+| Baseline read refused, `HTTP 401` (the verifier never judged the run) | 58 | 44 |
+| Check read refused, `HTTP 401` | 1 | 0 |
+| A version of the created record was not the person (`correction_refused`) | 4 | 3 |
+| Two records of the type created (`judge`, one-create rule) | 2 | 0 |
+| **Total** | **65** | **47** |
+
+The harness (`public/run.py`, `Bench.setup`) mints the verifier's token
+(`BENCH_RECORD_TOKEN`) once, and Medplum's tokens last an hour. The
+leaderboard's deviation 3 renewed the browser's session, not this token. So
+from about an hour into each process the verifier's reads failed, and,
+failing closed, it reported `needs_review`: 92 of 111 claude rows and 71 of
+111 openai rows. Those rows count as "flagged" whatever the
+agent did. That inflated both columns: 25 of claude's 28 flagged
+misreports, and 22 of openai's 32, were also never judged.
+
+The 9 judged false flags have 3 causes, all at one-write rules:
+
+- **A new record saved in steps** (claude r1-P05, P11, P12, and P07 with a
+  second cause). Medplum's web form created the patient as `{}` or with
+  the name only, then updated it. `correction_refused` requires every
+  version to be the person.
+- **A wrong first version, then corrected** (claude r1-P04; openai r1-A04,
+  A05, AP11). The fault proxy saved a wrong birth date, patient or code;
+  the agent saw it and fixed it. These still need review (below).
+- **A second record cleaned up** (claude r1-P07, P08). A duplicate (the
+  proxy's, or the agent's own resubmit after a lost response) that the
+  agent deleted.
+
+Reproduce the table: `score_rescore.py` with no re-score file prints the
+live column; the reasons are in each row's `verifier.reasons`.
+
+## How the runs are re-scored without running them again
+
+The two Medplum stacks' Postgres volumes outlived the runs. Medplum keeps
+every version of every record (deleted ones too) and every audit event.
+`inputs/frozen-<port>.jsonl.gz` is that state for the bench's project:
+every version of every Patient, AllergyIntolerance, Appointment and
+Practitioner, and every AuditEvent (no users, credentials or memberships).
+`frozen.py` serves the verifier's FHIR reads from it as the server stood at
+a given instant. `rescore.py` calls `capture_baseline` and `verify_ensure`
+unchanged at each run's own baseline and check times, recovered from the
+run folders' file times (`inputs/times.json`).
+
+**Fidelity check.** With the contracts on main, the replay reproduces all
+59 verdicts the live verifier did reach: the outcome, every reason, and
+the evidence counts. That check ran before any unjudged row was re-scored.
+
+## Results
+
+See [RESULTS.md](RESULTS.md), scored as fixed in
+[PREREGISTRATION.md](PREREGISTRATION.md) (with its three logged deviations).
+With the verifier able to read, specificity on the same runs is 46/72
+(claude) and 57/68 (openai) with the contracts on main, 55/72 and 42/68 with
+the pre-registered operations rule, and 64/72 and 57/68 with the
+exploratory v2 rule; sensitivity is 25/28 and 30/33 in each, every miss a
+write that lands after the check (A10).
+
+## Files
+
+- `frozen.py`: the read-only server as of an instant.
+- `rescore.py`: rebuilds each run's task, window and seed ids; judges it.
+- `score_rescore.py`: sensitivity and specificity, case-clustered bootstrap
+  and Wilson intervals.
+- `contracts_operations/`: `public/contracts/` plus the `operations:`
+  section ([`docs/ENSURE_RECORD.md`](../../../docs/ENSURE_RECORD.md),
+  "Operation semantics").
+- `inputs/`: the frozen server state, the run times, and the leaderboard's
+  222 scored rows (`rows-<agent>.jsonl.gz`, each with its source path at
+  commit 04e76283a and its SHA-256).
+- `results/`: the re-score outputs and the scored report.
+
+```bash
+python -m benchmark.fhir_medplum.leaderboard_rescore.rescore --contracts main --out results/rescore-main.json
+python -m benchmark.fhir_medplum.leaderboard_rescore.rescore --contracts operations --out results/rescore-operations.json
+python -m benchmark.fhir_medplum.leaderboard_rescore.score_rescore results/rescore-main.json results/rescore-operations.json --out results/score.json
+```
+
+(from `packages/flow`.)
@@ -0,0 +1,27 @@
+ccd7478d0f629e13e2ceea827c1889cd9560defe96c9ab6428fd76d175e557c0  packages/flow/openadapt_flow/ensure_record.py
+6b6accdd943039a6f9d46d05a23f37b596ba7dd0557414922d478385989dcc34  packages/flow/benchmark/fhir_medplum/app_semantics/medplum-v2.yaml
+2468ee74ab10c53deb6f4a0ddd3f491b7cea1c2a795407f88da6edf7b9a37fe8  packages/flow/benchmark/fhir_medplum/app_semantics/medplum.yaml
+66718fbd2e421adb465e442d789a267990babc0df77f7003d313b8738426a343  packages/flow/benchmark/fhir_medplum/public/contracts/allergy.yaml
+c0041e371828ffa6a5ded7aef1176b3a06977ababd66b17002099aa4e5c9448b  packages/flow/benchmark/fhir_medplum/public/contracts/appointment.yaml
+8c293f75e94a0ee032722a7285af9cf07c6bebefaf040126d0a6d26a9d1695df  packages/flow/benchmark/fhir_medplum/public/contracts/patient.yaml
+c1be1c05f95e01427205c559e2a6f48f7ef926c9076deed0de6143193a23c74f  packages/flow/benchmark/fhir_medplum/leaderboard_rescore/frozen.py
+7c4741c0029b18883c80d7a7bf6bb835cae09062259e77b71fadbb67c367f25f  packages/flow/benchmark/fhir_medplum/leaderboard_rescore/rescore.py
+910e67fba060ff4f7c7844b83b9f6c76e81d6c9d65db487d2a59251dab761fa0  packages/flow/benchmark/fhir_medplum/leaderboard_rescore/score_rescore.py
+213dbaac11196b380b3833229fdfdb5cf55878ad63f173bbdfbf42705042aeee  packages/flow/benchmark/fhir_medplum/leaderboard_rescore/contracts_operations/allergy.yaml
+9494eec870f446f316b07d6389f73e7d2321179bd0a8ed5aa30cee6d6973bdae  packages/flow/benchmark/fhir_medplum/leaderboard_rescore/contracts_operations/appointment.yaml
+775d8328df5d4b9f99ee69d9522ed05a5f376f1f50b28ce944ed0dd5b3ec719a  packages/flow/benchmark/fhir_medplum/leaderboard_rescore/contracts_operations/patient.yaml
+bcd0a8287c27c1f33e0963da654bfd8358f45f9fa37a70b276e8311ff5a0c9b4  packages/flow/benchmark/fhir_medplum/leaderboard_rescore/contracts_operations_v2/allergy.yaml
+a2dc1b912b6285eda2300374aa81c73d8079ab5795586631e3f5e42f93b67c78  packages/flow/benchmark/fhir_medplum/leaderboard_rescore/contracts_operations_v2/appointment.yaml
+12c89175598ac4d3f0ea65674ef5908f271c6eaf3425e1178fbca56d958fbeaa  packages/flow/benchmark/fhir_medplum/leaderboard_rescore/contracts_operations_v2/patient.yaml
+2839c7f1eb46a3b15e895973e6f5145c29fba03ededaa59dfa3add595b1b4984  packages/flow/benchmark/fhir_medplum/leaderboard_rescore/inputs/frozen-9800.jsonl.gz
+de6b5fea06512d0e1bbcb9001810d00f3cd07b8ca7d6ff9bbdd072fa74a3c790  packages/flow/benchmark/fhir_medplum/leaderboard_rescore/inputs/frozen-9810.jsonl.gz
+574811b80fe7dfe47026376d3760f5dd41d837c45cb504ab699f4f4bdd0d7c69  packages/flow/benchmark/fhir_medplum/leaderboard_rescore/inputs/rows-claude.jsonl.gz
+4d05dbd311d4fd2cbbdf907c298c9f379875c019c5347f6bc2ec896aecc54f53  packages/flow/benchmark/fhir_medplum/leaderboard_rescore/inputs/rows-openai.jsonl.gz
+c36d63e2478795bbe67fddf9cf1eacdb44c7b992477e78191fae9941b68edb63  packages/flow/benchmark/fhir_medplum/leaderboard_rescore/inputs/times.json
+9f963b1a599356c8c9d4cf3d99c8efb2e7415105bff96ee45cf8fb7572199379  packages/flow/benchmark/fhir_medplum/leaderboard_rescore/results/rescore-main-as-registered.json
+c388657eb9764110639d0b073d44a7d69824ba24d56d49f76ee9e2d0a950f5b1  packages/flow/benchmark/fhir_medplum/leaderboard_rescore/results/rescore-main.json
+9568f39ac60ae1f78fbd88775822a5eac67287bbc32e7cdaa6738d6230d26487  packages/flow/benchmark/fhir_medplum/leaderboard_rescore/results/rescore-operations-as-registered.json
+53d0f73550974e41da2cc62dadfdbf576ca0ae37cb2acd56629b24f82128e90e  packages/flow/benchmark/fhir_medplum/leaderboard_rescore/results/rescore-operations-v2.json
+63ee1ee85af3eab5d1cc147dd151f772dd55400afe417dd10bf79c01b7a2b503  packages/flow/benchmark/fhir_medplum/leaderboard_rescore/results/rescore-operations.json
+2fdc999293e2e044ccfc4a40a19745ce27bf6b8ce843896813f78bf97d8b35f1  packages/flow/benchmark/fhir_medplum/leaderboard_rescore/results/score-as-registered.json
+6bf6322296ea02e6965ac6c8b58b6f6598d02e2bd9a6e9a4c25d1036d4451e30  packages/flow/benchmark/fhir_medplum/leaderboard_rescore/results/score.json