Skip to content
2 changes: 1 addition & 1 deletion CAPABILITIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ improvising or retrying when something is off.
| Judge an "ensure the record exists" write (found, created once with every value, or needs review) through a FHIR server's read path only: searches, AuditEvent and `_history`, never the database; the coverage claim names that read path. An agent's own correction of the record it created counts as one write when the audit trail and history show only that agent, no other reader, and the same record in every version | On main ([#1640](https://github.com/OpenAdaptAI/OpenAdapt/pull/1640), corrections [#1655](https://github.com/OpenAdaptAI/OpenAdapt/pull/1655)) | Real app (Medplum 5.2.8; three blind fault suites: 17/18, 17/20, 19/20; no fault passed in the last two. Corrections, on 15 blind cases and the 37 public ones: false stops 8 to 2 and 9 to 4 for Claude, no harm passed that the old rule caught) | `openadapt_flow/ensure_record.py`, [ENSURE_RECORD.md](packages/flow/docs/ENSURE_RECORD.md) |
| Generate an ensure-record contract from three demonstrations, the server's CapabilityStatement and its SearchParameter definitions: field paths and `[system=…]` filters, comparison kinds, value maps, identity, search parameters, the closed record and the change feed. It refuses when unsure, asks a person to confirm an identity for a type with no reviewed template, and self-checks every demonstration plus a perturbation per field | On main ([#1653](https://github.com/OpenAdaptAI/OpenAdapt/pull/1653)) | Real app (Medplum 5.2.8: all 16 hand-written Patient lines regenerated; one blind write type, Immunization: 78 s of person time, 2 lines edited after the person declined the proposed identity; blind suite 23/24, no fault passed) | `openadapt_flow/generate/ensure_contract.py`, [RESULTS.md](packages/flow/benchmark/contract_autogen/results/RESULTS.md) |
| A stop card for a stopped write: the exact record difference and who else wrote, with Undo or Fix that reads back, or Open in the app; cost modelled from published operators and timed machine steps, as pre-registered | On main ([#1653](https://github.com/OpenAdaptAI/OpenAdapt/pull/1653)) | Real app (Medplum 5.2.8, blind suite: 16 stops, 14 one-tap with no failed read-back; modelled 42 s per caught harm against 420; one wrong category on a false stop, gone in the re-measure with `void`: 24/24, 15 stops, no wrong category, no failed read-back) | `benchmark/stop_card/`, [PREREG_REVIEW.md](packages/flow/benchmark/stop_card/PREREG_REVIEW.md) |
| Generate `openadapt.verify/v1` contracts from a MySQL, MariaDB or PostgreSQL schema, the kinds of write with their parameter names, and one demonstration write per example: one contract per field set (an update gets one for each set of fields it may change), list parameters by length, a `found` twin for a record already in place, the whole-database frame on every dialect (it compares the contract's own tables too, so an excused column holds only on the rows the write declares, and the recheck sees a late change to a look-alike row), rows the app adds on its own declared by count, a recheck window the person confirms, and `frame.versions` (a lost update) from a version column or the app's audit trail of saves. Every undecided point is one numbered question; every contract is loaded and dry-read against the live store before it is written; a needed `TRIGGER` grant or view is applied with a flag or the kind is refused. Not yet: OpenAPI metadata | [#1672](https://github.com/OpenAdaptAI/OpenAdapt/pull/1672) | Real app (first study, instance copies; questions for a person: Dolibarr 4, Redmine 2, EspoCRM 3, against 11 hand decisions each; every in-scope planted fault stopped (22/22, 19/19, 21/21); correct writes confirmed 32/43, 32/38, 27/40; evidence in the private study repository) | `python -m openadapt_flow.generate.verify_contract`, [GENERATE_CONTRACTS.md](packages/flow/docs/GENERATE_CONTRACTS.md) |
| Generate `openadapt.verify/v1` contracts from a MySQL, MariaDB or PostgreSQL schema, the kinds of write with their parameter names, and one demonstration write per example: one contract per field set (an update gets one for each set of fields it may change), list parameters by length, a `found` twin for a record already in place, the whole-database frame on every dialect (it compares the contract's own tables too, so an excused column holds only on the rows the write declares, and the recheck sees a late change to a look-alike row), rows the app adds on its own declared by count, a recheck window the person confirms, and `frame.versions` (a lost update) from a version column or the app's audit trail of saves. Equal ids are never taken for a join (a row's own key, the column a path arrived by), a join only equal ids support is a question when a supported place also fits, and a pin needs two rows that agree. Every undecided point is one numbered question; every contract is loaded and dry-read against the live store before it is written; a needed `TRIGGER` grant or view is applied with a flag or the kind is refused. Not yet: OpenAPI metadata | [#1672](https://github.com/OpenAdaptAI/OpenAdapt/pull/1672) | Real app (first study, instance copies; questions for a person: Dolibarr 4, Redmine 2, EspoCRM 3, against 11 hand decisions each; every in-scope planted fault stopped (22/22, 19/19, 21/21); correct writes confirmed 32/43, 32/38, 27/40; evidence in the private study repository) | `python -m openadapt_flow.generate.verify_contract`, [GENERATE_CONTRACTS.md](packages/flow/docs/GENERATE_CONTRACTS.md) |
| `void` in an ensure-record contract: values that make a record assert nothing (`status="entered-in-error"`). A record holding one is never the person, is not a duplicate, and a created one needs review; it names values to leave out, never values to keep, and can't name an identity field | On main ([#1657](https://github.com/OpenAdaptAI/OpenAdapt/pull/1657)) | Real app (Medplum 5.2.8: 8 live cases written in advance, 8/8 with the line, 5/8 and one silent pass without it; frozen blind Immunization suite 24/24); adversarial review recorded | `openadapt_flow/ensure_record.py`, [RESULTS.md](packages/flow/benchmark/fhir_medplum/void/results/RESULTS.md), [REVIEW_VOID.md](packages/flow/benchmark/fhir_medplum/void/REVIEW_VOID.md) |
| Check a patient registration on a commercial EHR through its public FHIR sandbox: Epic (open.epic, R4), read through FHIR search and read with SMART backend auth; the frame is the search set, since the sandbox has no AuditEvent or `_history` | On main ([#1656](https://github.com/OpenAdaptAI/OpenAdapt/pull/1656)) | Real app (Epic public sandbox, August 2026 release; blind round 12/13 with none passed as verified, the miss a false refusal fixed in the contract; caught Epic dropping the phone on a 201 create) | `benchmark/commercial_ehr/`, [FINDINGS.md](packages/flow/benchmark/commercial_ehr/FINDINGS.md) |
| A public benchmark a vendor reruns with their own agent: `openadapt bench medplum --agent <scripted|claude|cmd:...|py:...>` runs 37 written-for-it cases (Patient, AllergyIntolerance, Appointment) against a local Medplum through a fault proxy, and scores the agent's claim, a read-back and OpenAdapt's verifier against an end-state oracle | On main ([#1652](https://github.com/OpenAdaptAI/OpenAdapt/pull/1652)) | Real app (Medplum 5.2.8; Claude Sonnet 5.5: OpenAdapt missed 1 of 9 harms with 4 false stops, all corrections that changed the record's identity, against 8 missed by the claim and the read-back; cold run from clone to scorecard 5 min) | [README](packages/flow/benchmark/fhir_medplum/public/README.md), [RESULTS.md](packages/flow/benchmark/fhir_medplum/public/RESULTS.md) |
Expand Down
48 changes: 46 additions & 2 deletions packages/flow/docs/GENERATE_CONTRACTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,28 @@ keeps, in this order:
the table defines first when they hold the same value in every row;
3. the place another kind's demonstrations used for the same table.

If places are still tied, that is a question. A value that is also in a row
If places are still tied, that is a question.

Equal ids are not a join. Demonstrations made on a fresh install often give
a new row the same id as an unrelated row, so:

- A row's own whole single-column key holding a generated number is never
a key to another table, and a new row's such key never links it to its
record, unless a declared foreign key says so. A member of a composite key
(an order line's `order_id` in `(order_id, line_no)`) and a copied token
(an extension table keyed by its parent's UUID) still link.
- A value is never placed in the column a path arrived by (a child row's
link column holds the record's key by construction).
- A list item's link is the column that holds the record's key in every
item row of every demonstration. When two always agree and no declared
foreign key picks one, the list is refused until a demonstration tells
them apart.
- When the best place goes through a key only equal values support (no
declared foreign key, no column named after its table) and a place whose
keys are all supported fits every demonstration too, that is a question
(`locations.<kind>`), never a silent pick.

A value that is also in a row
the write added beside the record (a call's lead is its parent and an
attendee row) is checked in both places.

Expand Down Expand Up @@ -193,7 +214,16 @@ too. A list parameter gets a contract per length, from 1 to 10 items.
user, to the value every demonstration held, or to a copy of a column of the
row a key points to (an invoice line's price from its product). A key, a
link to a row the write added, the year, month, week or day of a date the
task gives, and a generated token (a UUID) are recognized as such. Columns
task gives, and a generated token (a UUID) are recognized as such. A link
and a column that holds the row's own key need two demonstrations; the own
key must sit in the same text in every row (`INV-{key}`), never as a mere
substring. Such a column is listed in the question, never checked or
left open on its own. A date part must differ between the demonstrations: a constant
that happens to equal the month of every demonstration's date is a
constant. A copy is taken only through a declared key or a column named
after its table, and only when the source differs between demonstrations:
a column and its source that hold one value in every demonstration fit a
constant and a copy alike, so the question lists the column. Columns
that differ between demonstrations, free text every demonstration left
empty, and the write's time stamps are listed in the bookkeeping question,
and each is left unchecked only if the answer names it (`unchecked`). Free
Expand All @@ -205,6 +235,20 @@ columns that vary or are time stamps, so accepting it is explicit, and
leaves free text out: a person adds a free-text column to `unchecked` only
on purpose.

A pin needs two rows that agree. A value seen beside one write only (a field
set one demonstration alone gives, a row only one demonstration added) is
pinned only when every row of that table the kind's demonstrations wrote
holds it. Otherwise the bookkeeping question lists it as one the
demonstrations can't tell, it is left out of the proposed answer, and the
kind has no contract until a person names it under `unchecked` or records a
demonstration where it differs. A field set whose own demonstrations leave
such a column open, while the kind's full set pins it, gets no contract
(`rejected` in `report.json`). A column that holds the acting user's id in
every row is the actor, unless it is also a copy of a column of a row it
points to, or a link: then the demonstrations can't tell, and the question
lists it. Pins and widening count demonstrations, not rows. A row the app adds beside an update may hold the target's key: that is a
link to the record, never a pinned id.

**Rows the app adds on its own.** A row type the app added the same number of
times in every demonstration (an agenda event, a stream note, a price-history
row) is declared with that count and its constant columns. A table whose
Expand Down
Loading
Loading