Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
95 commits
Select commit Hold shift + click to select a range
f38f8c7
WIP: verify-contract generator from a SQL schema and demonstrations
abrichr Oct 30, 2026
4c5b7c5
WIP: generator trigger grant, demo and validation scripts
abrichr Oct 30, 2026
db75fd8
WIP: by-name update fields; keep answered questions counted
abrichr Oct 30, 2026
90af2ff
WIP: copied-column name preference
abrichr Oct 30, 2026
15e6c20
WIP: copied column prefers the table its key names
abrichr Oct 30, 2026
cc82064
WIP: compare number and date columns by value; docs
abrichr Oct 30, 2026
1173b66
WIP: free text unchecked on new records; co-owned child rows; demos f…
abrichr Oct 30, 2026
2865491
WIP: format; summarize script; connection hygiene in demos
abrichr Oct 30, 2026
1532b24
WIP: found flavor for updates already in place
abrichr Oct 30, 2026
da9d887
WIP: validation runs the declared recheck on late-write faults
abrichr Oct 30, 2026
a7696ab
WIP: side rows in the bookkeeping question; Dolibarr results
abrichr Oct 30, 2026
b059887
WIP: empty child rows, locations shared across kinds, exact-name upda…
abrichr Oct 30, 2026
1e86346
WIP: typed sample values for fields placed by name
abrichr Oct 30, 2026
bd284fd
WIP: nullable fields optional on create; date parts unchecked; Postgr…
abrichr Oct 30, 2026
af9a85c
WIP: found contracts check rows beside the record; unique pin columns
abrichr Oct 30, 2026
86c6488
WIP: tidy found handling
abrichr Oct 30, 2026
da944ea
WIP: copied columns need two rows
abrichr Oct 30, 2026
06ef14a
WIP: EspoCRM shapes: anchor scoring, tie-breaks, dispatch child rows,…
abrichr Oct 30, 2026
a50021b
WIP: derived-copy tie-break; logs per kind
abrichr Oct 30, 2026
bea29df
WIP: identical columns tie-break
abrichr Oct 30, 2026
3fdcb36
WIP: views in their own schema
abrichr Oct 30, 2026
3e9c5e1
WIP: new referenced rows per dispatch value
abrichr Oct 30, 2026
6c41eea
WIP: replaced linked rows declared with their count; demonstrated set…
abrichr Oct 30, 2026
43fbadc
WIP: empty pins only where the app left the column empty; generated t…
abrichr Oct 30, 2026
c68b85b
docs: generator decision rules
abrichr Oct 30, 2026
86d6f9d
WIP: created rows' tables append-only in the whole-database frame, so…
abrichr Oct 30, 2026
0a2abe0
WIP: email and phone values always compared in canonical form; docs link
abrichr Oct 30, 2026
34aba77
WIP: answered location questions still counted
abrichr Oct 30, 2026
ba048d0
WIP: Dolibarr results
abrichr Oct 30, 2026
cd1bac4
WIP: frame.versions from a version column or the app's audit trail
abrichr Oct 30, 2026
266b266
Generator results so far; CAPABILITIES; review brief
abrichr Oct 30, 2026
baf86e7
Versions from an audit trail: narrow only by the record's type
abrichr Oct 30, 2026
6ed3c6c
docs: lost-update versions in generated contracts
abrichr Oct 30, 2026
c51437b
Versions: a column named as a version; the author column when several…
abrichr Oct 30, 2026
ab43f2a
Lint, types, public artifact inventory, capability status
abrichr Oct 30, 2026
a1d9674
Final real-app results on the rebased branch
abrichr Oct 30, 2026
f48868d
Round 1 review: confirmed phone rule, row-scoped bookkeeping, found p…
abrichr Oct 30, 2026
14a6e40
Whole-database frame and recheck on PostgreSQL (#1673)
abrichr Oct 31, 2026
bd23973
WIP: verify-contract generator from a SQL schema and demonstrations
abrichr Oct 30, 2026
55adc3f
WIP: generator trigger grant, demo and validation scripts
abrichr Oct 30, 2026
07faa68
WIP: by-name update fields; keep answered questions counted
abrichr Oct 30, 2026
c149d37
WIP: copied-column name preference
abrichr Oct 30, 2026
a4a63fa
WIP: copied column prefers the table its key names
abrichr Oct 30, 2026
3ee84dd
WIP: compare number and date columns by value; docs
abrichr Oct 30, 2026
debc57a
WIP: free text unchecked on new records; co-owned child rows; demos f…
abrichr Oct 30, 2026
f3361fb
WIP: format; summarize script; connection hygiene in demos
abrichr Oct 30, 2026
ac6ad51
WIP: found flavor for updates already in place
abrichr Oct 30, 2026
03bfb9f
WIP: validation runs the declared recheck on late-write faults
abrichr Oct 30, 2026
1b8437c
WIP: side rows in the bookkeeping question; Dolibarr results
abrichr Oct 30, 2026
faf64ce
WIP: empty child rows, locations shared across kinds, exact-name upda…
abrichr Oct 30, 2026
7a38647
WIP: typed sample values for fields placed by name
abrichr Oct 30, 2026
0837f30
WIP: nullable fields optional on create; date parts unchecked; Postgr…
abrichr Oct 30, 2026
7033e39
WIP: found contracts check rows beside the record; unique pin columns
abrichr Oct 30, 2026
1098a2f
WIP: tidy found handling
abrichr Oct 30, 2026
7982898
WIP: copied columns need two rows
abrichr Oct 30, 2026
10559df
WIP: EspoCRM shapes: anchor scoring, tie-breaks, dispatch child rows,…
abrichr Oct 30, 2026
7ff9b39
WIP: derived-copy tie-break; logs per kind
abrichr Oct 30, 2026
a7bd394
WIP: identical columns tie-break
abrichr Oct 30, 2026
25c11b4
WIP: views in their own schema
abrichr Oct 30, 2026
a9a5c2a
WIP: new referenced rows per dispatch value
abrichr Oct 30, 2026
952491e
WIP: replaced linked rows declared with their count; demonstrated set…
abrichr Oct 30, 2026
0828bdf
WIP: empty pins only where the app left the column empty; generated t…
abrichr Oct 30, 2026
f2e9a8e
docs: generator decision rules
abrichr Oct 30, 2026
d39f04b
WIP: created rows' tables append-only in the whole-database frame, so…
abrichr Oct 30, 2026
1756cb5
WIP: email and phone values always compared in canonical form; docs link
abrichr Oct 30, 2026
fde99bf
WIP: answered location questions still counted
abrichr Oct 30, 2026
eb8fee6
WIP: Dolibarr results
abrichr Oct 30, 2026
c440987
WIP: frame.versions from a version column or the app's audit trail
abrichr Oct 30, 2026
bc399ca
Generator results so far; CAPABILITIES; review brief
abrichr Oct 30, 2026
ad5557f
Versions from an audit trail: narrow only by the record's type
abrichr Oct 30, 2026
759d060
docs: lost-update versions in generated contracts
abrichr Oct 30, 2026
cbe0e0c
Versions: a column named as a version; the author column when several…
abrichr Oct 30, 2026
0e01dcc
Lint, types, public artifact inventory, capability status
abrichr Oct 30, 2026
f3008fa
Final real-app results on the rebased branch
abrichr Oct 30, 2026
bd267c9
Round 1 review: confirmed phone rule, row-scoped bookkeeping, found p…
abrichr Oct 30, 2026
6f5b0f6
A row the app changes by key is excused for every kind, on that row only
abrichr Oct 30, 2026
7272965
found: a row the app changes by key is excused there too
abrichr Oct 30, 2026
d18b123
Uniqueness over every value the task reads through one key together
abrichr Oct 31, 2026
b11cfed
PostgreSQL: the whole-database frame and the recheck question, as on …
abrichr Oct 31, 2026
3332427
Keep the pre-rebase branch history (the rebase onto main is in the co…
abrichr Oct 31, 2026
64a59ee
Only a row named by a literal key is excused app-wide: a row the task…
abrichr Oct 31, 2026
2b6ab77
A changed row the task's values name applies only to field sets that …
abrichr Oct 31, 2026
0bd47c6
logs_add: any table the demonstrations added rows to, a log for every…
abrichr Oct 31, 2026
f4cb759
Versions read: filter on an event code no create used (Dolibarr's COM…
abrichr Oct 31, 2026
ea9b936
Versions filter: letters-only codes, never an id column
abrichr Oct 31, 2026
f743c61
Final real-app results on the round-1 branch: PostgreSQL frame and re…
abrichr Oct 31, 2026
d075d29
Move the generator's study evidence to the private study repository
abrichr Oct 31, 2026
cd4d481
CAPABILITIES, artifact inventory and review brief without the moved e…
abrichr Oct 31, 2026
99db906
Round 2 review: versions targets, per-surface fields, strict answers,…
abrichr Oct 31, 2026
cc32945
Bookkeeping proposal names only columns that vary or are time stamps;…
abrichr Oct 31, 2026
893a613
Free text of a row the app adds is pinned empty (NULL or ''), not to …
abrichr Oct 31, 2026
2bc534f
Bookkeeping proposal names every open column; rows the app adds keep …
abrichr Oct 31, 2026
abaa14c
Round 3 review: free text stays pinned, read-code deny-list, SET ROLE…
abrichr Oct 31, 2026
19d7413
Read codes: an audit row naming a record the demonstration added or c…
abrichr Oct 31, 2026
b1ce256
CAPABILITIES: counts after the round-3 fixes
abrichr Oct 31, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions CAPABILITIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,7 @@ improvising or retrying when something is off.
| Judge an "ensure the record exists" write (found, created once with every value, or needs review) through a FHIR server's read path only: searches, AuditEvent and `_history`, never the database; the coverage claim names that read path. An agent's own correction of the record it created counts as one write when the audit trail and history show only that agent, no other reader, and the same record in every version | On main ([#1640](https://github.com/OpenAdaptAI/OpenAdapt/pull/1640), corrections [#1655](https://github.com/OpenAdaptAI/OpenAdapt/pull/1655)) | Real app (Medplum 5.2.8; three blind fault suites: 17/18, 17/20, 19/20; no fault passed in the last two. Corrections, on 15 blind cases and the 37 public ones: false stops 8 to 2 and 9 to 4 for Claude, no harm passed that the old rule caught) | `openadapt_flow/ensure_record.py`, [ENSURE_RECORD.md](packages/flow/docs/ENSURE_RECORD.md) |
| Generate an ensure-record contract from three demonstrations, the server's CapabilityStatement and its SearchParameter definitions: field paths and `[system=…]` filters, comparison kinds, value maps, identity, search parameters, the closed record and the change feed. It refuses when unsure, asks a person to confirm an identity for a type with no reviewed template, and self-checks every demonstration plus a perturbation per field | On main ([#1653](https://github.com/OpenAdaptAI/OpenAdapt/pull/1653)) | Real app (Medplum 5.2.8: all 16 hand-written Patient lines regenerated; one blind write type, Immunization: 78 s of person time, 2 lines edited after the person declined the proposed identity; blind suite 23/24, no fault passed) | `openadapt_flow/generate/ensure_contract.py`, [RESULTS.md](packages/flow/benchmark/contract_autogen/results/RESULTS.md) |
| A stop card for a stopped write: the exact record difference and who else wrote, with Undo or Fix that reads back, or Open in the app; cost modelled from published operators and timed machine steps, as pre-registered | On main ([#1653](https://github.com/OpenAdaptAI/OpenAdapt/pull/1653)) | Real app (Medplum 5.2.8, blind suite: 16 stops, 14 one-tap with no failed read-back; modelled 42 s per caught harm against 420; one wrong category on a false stop, gone in the re-measure with `void`: 24/24, 15 stops, no wrong category, no failed read-back) | `benchmark/stop_card/`, [PREREG_REVIEW.md](packages/flow/benchmark/stop_card/PREREG_REVIEW.md) |
| Generate `openadapt.verify/v1` contracts from a MySQL, MariaDB or PostgreSQL schema, the kinds of write with their parameter names, and one demonstration write per example: one contract per field set (an update gets one for each set of fields it may change), list parameters by length, a `found` twin for a record already in place, the whole-database frame on every dialect, rows the app adds on its own declared by count, a recheck window the person confirms, and `frame.versions` (a lost update) from a version column or the app's audit trail of saves. Every undecided point is one numbered question; every contract is loaded and dry-read against the live store before it is written; a needed `TRIGGER` grant or view is applied with a flag or the kind is refused. Not yet: OpenAPI metadata | [#1672](https://github.com/OpenAdaptAI/OpenAdapt/pull/1672) | Real app (first study, instance copies; questions for a person: Dolibarr 4, Redmine 2, EspoCRM 3, against 11 hand decisions each; every in-scope planted fault stopped (22/22, 19/19, 21/21); correct writes confirmed 32/43, 32/38, 27/40; evidence in the private study repository) | `python -m openadapt_flow.generate.verify_contract`, [GENERATE_CONTRACTS.md](packages/flow/docs/GENERATE_CONTRACTS.md) |
| `void` in an ensure-record contract: values that make a record assert nothing (`status="entered-in-error"`). A record holding one is never the person, is not a duplicate, and a created one needs review; it names values to leave out, never values to keep, and can't name an identity field | On main ([#1657](https://github.com/OpenAdaptAI/OpenAdapt/pull/1657)) | Real app (Medplum 5.2.8: 8 live cases written in advance, 8/8 with the line, 5/8 and one silent pass without it; frozen blind Immunization suite 24/24); adversarial review recorded | `openadapt_flow/ensure_record.py`, [RESULTS.md](packages/flow/benchmark/fhir_medplum/void/results/RESULTS.md), [REVIEW_VOID.md](packages/flow/benchmark/fhir_medplum/void/REVIEW_VOID.md) |
| Check a patient registration on a commercial EHR through its public FHIR sandbox: Epic (open.epic, R4), read through FHIR search and read with SMART backend auth; the frame is the search set, since the sandbox has no AuditEvent or `_history` | On main ([#1656](https://github.com/OpenAdaptAI/OpenAdapt/pull/1656)) | Real app (Epic public sandbox, August 2026 release; blind round 12/13 with none passed as verified, the miss a false refusal fixed in the contract; caught Epic dropping the phone on a 201 create) | `benchmark/commercial_ehr/`, [FINDINGS.md](packages/flow/benchmark/commercial_ehr/FINDINGS.md) |
| A public benchmark a vendor reruns with their own agent: `openadapt bench medplum --agent <scripted|claude|cmd:...|py:...>` runs 37 written-for-it cases (Patient, AllergyIntolerance, Appointment) against a local Medplum through a fault proxy, and scores the agent's claim, a read-back and OpenAdapt's verifier against an end-state oracle | On main ([#1652](https://github.com/OpenAdaptAI/OpenAdapt/pull/1652)) | Real app (Medplum 5.2.8; Claude Sonnet 5.5: OpenAdapt missed 1 of 9 harms with 4 false stops, all corrections that changed the record's identity, against 8 missed by the claim and the read-back; cold run from clone to scorecard 5 min) | [README](packages/flow/benchmark/fhir_medplum/public/README.md), [RESULTS.md](packages/flow/benchmark/fhir_medplum/public/RESULTS.md) |
Expand All @@ -155,8 +156,8 @@ improvising or retrying when something is off.
| Check that the saved record matches its source document, not only the parameters: intake keeps each value's line, a second reading must agree before a guarded save, and a round trip compares the read-back with the document | On main | Synthetic (MockMed, text referral) | `openadapt_flow/provenance.py`, `intake`, `verify-run --provenance`, guarded save |
| Check one write from Python, without the CLI: one API (`openadapt_flow.verifier`: `begin_write`, `verify_write`; 2.4.1's `capture` form still works, deprecated) with one outcome (`CONFIRMED`, `REFUTED`, `UNCERTAIN`), the receipt, which parts were checked, and the result's scope (read path, stores read, time window, what it doesn't cover) | On main | Synthetic (SQLite; MockMed's fault server over real HTTP) | `openadapt_flow/verifier.py`, [VERIFIER_API.md](packages/flow/docs/VERIFIER_API.md) |
| Never send a write twice while its outcome is unknown: an append-only ledger on local disk (pluggable backend), keyed by the write's identity, that every adapter, the MCP tools and the CLI consult before sending; a read-back against the attempt's own baseline resolves it, and a refuted write waits for a person (`openadapt verify ledger list`, `reconcile`, `release`) | On main | Synthetic (MockMed's fault server: a worker killed between the send and the read-back, killed during a timed-out send, killed before the send, a late write, and two worker processes started together on one write) | `openadapt_flow/write_ledger.py`, [VERIFIER_API.md](packages/flow/docs/VERIFIER_API.md#the-uncertain-write-ledger) |
| Catch a write that lands after the verdict (a background job, a trigger, a delayed side record): a `CONFIRMED` with a recheck due is provisional (`provisional`, `recheck_due_at`); the recheck reads the whole database again and refutes the earlier result when a change since the post-save read has no explanation, and a recheck that doesn't confirm holds the write in the ledger. Run it from Python (`verifier.recheck`) or MCP (`verify_recheck`). A write after the recheck's read is not seen (`frame.database.recheck_after_s`, no default window) | On main ([#1639](https://github.com/OpenAdaptAI/OpenAdapt/pull/1639)); provisional result, MCP tool and ledger hold ([#1669](https://github.com/OpenAdaptAI/OpenAdapt/pull/1669)) | Real app (Dolibarr 21 and EspoCRM 10 on MariaDB 11.4, 180 s window: 4 of 4 planted late writes at 125 to 130 s refuted at the recheck, correct writes confirmed; EspoCRM's cron daemon needs its job tables declared); synthetic (SQLite); opt-in live OpenEMR test | `openadapt_flow/verifier.py`, `runtime/effects/late_write.py`, [VERIFY_RUN.md](packages/flow/docs/VERIFY_RUN.md#writes-that-land-later) |
| Diff the whole database for the frame (`frame.database`) on SQLite, MySQL and MariaDB. On MySQL and MariaDB every column type is read exactly (binary strings, `BIT`, geometries and vectors as hex, `FLOAT` widened to `DOUBLE`, `JSON` as stored text, decimals at full precision), so a row the save adds, changes or removes outside the declared records, such as an extra planned call or a stream note, is refuted unless a declared effect, a side record, an attributed concurrent write or an allow-listed bookkeeping table explains it | On main ([#1661](https://github.com/OpenAdaptAI/OpenAdapt/pull/1661)) | Real app (Dolibarr 21 on MariaDB 11.4: all 387 tables read where the frame raised `TypeError`; EspoCRM on MariaDB 11.4: the first-contact study's extra call and stream note refuted, correct opportunities verified); live MariaDB type tests | [VERIFY_RUN.md](packages/flow/docs/VERIFY_RUN.md#the-whole-database), `runtime/effects/db_snapshot.py` |
| Catch a write that lands after the verdict (a background job, a trigger, a delayed side record): a `CONFIRMED` with a recheck due is provisional (`provisional`, `recheck_due_at`); the recheck reads the whole database again and refutes the earlier result when a change since the post-save read has no explanation, and a recheck that doesn't confirm holds the write in the ledger. Run it from Python (`verifier.recheck`) or MCP (`verify_recheck`). A write after the recheck's read is not seen (`frame.database.recheck_after_s`, no default window) | On main ([#1639](https://github.com/OpenAdaptAI/OpenAdapt/pull/1639)); provisional result, MCP tool and ledger hold ([#1669](https://github.com/OpenAdaptAI/OpenAdapt/pull/1669)) | Real app (Dolibarr 21 and EspoCRM 10 on MariaDB 11.4, 180 s window: 4 of 4 planted late writes at 125 to 130 s refuted at the recheck, correct writes confirmed; EspoCRM's cron daemon needs its job tables declared; Redmine 6.1 on PostgreSQL 17, 180 s window: the first-contact study's late writes F19 and F20 refuted at the recheck, correct writes confirmed); synthetic (SQLite); opt-in live OpenEMR test | `openadapt_flow/verifier.py`, `runtime/effects/late_write.py`, [VERIFY_RUN.md](packages/flow/docs/VERIFY_RUN.md#writes-that-land-later) |
| Diff the whole database for the frame (`frame.database`) on SQLite, MySQL, MariaDB and PostgreSQL. On MySQL and MariaDB every column type is read exactly (binary strings, `BIT`, geometries and vectors as hex, `FLOAT` widened to `DOUBLE`, `JSON` as stored text, decimals at full precision). On PostgreSQL every column type is read as text by a declared rule in a session pinned to UTC and exact float output, and a type without a rule, a table the account can't read in full, or an unreadable sequence is refused. So a row the save adds, changes or removes outside the declared records, such as an extra planned call or a stream note, is refuted unless a declared effect, a side record, an attributed concurrent write or an allow-listed bookkeeping table explains it | On main ([#1661](https://github.com/OpenAdaptAI/OpenAdapt/pull/1661)); PostgreSQL ([#1673](https://github.com/OpenAdaptAI/OpenAdapt/pull/1673)) | Real app (Dolibarr 21 on MariaDB 11.4: all 387 tables read where the frame raised `TypeError`; EspoCRM on MariaDB 11.4: the first-contact study's extra call and stream note refuted, correct opportunities verified; Redmine 6.1 on PostgreSQL 17: 58 tables read, a stray upload in a table the contract doesn't read refuted, correct writes verified); live MariaDB and PostgreSQL type tests | [VERIFY_RUN.md](packages/flow/docs/VERIFY_RUN.md#the-whole-database), `runtime/effects/db_snapshot.py` |
| Catch a lost update: another user saves a change to the record while the agent works, and the agent saves its stale copy over it. Before and after look like a correct update, so `frame.versions` reads the target's versions (FHIR or SQL version history, a version column, or the app's audit or action-history table) before and after the write. A version by another author, or more than one version step when no author is read, refutes the write; a version that can't be read or attributed stops it | On main ([#1662](https://github.com/OpenAdaptAI/OpenAdapt/pull/1662)) | Real app (HAPI FHIR and EspoCRM, first-contact study fault scripts: both lost updates refuted, correct updates verified, every other case's verdict unchanged) | [VERIFY_RUN.md](packages/flow/docs/VERIFY_RUN.md#lost-updates), `runtime/effects/versions.py` |
| Guarded save: a safety layer any agent wraps around one write (check the record, hold for approval, read back) | On main ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)) | Synthetic | `GUARDED_SAVE.md` |
| A person's approval in Desktop is signed with a key kept behind Touch ID on a Mac: one-prompt setup in the app, a prompt for every answer, and Flow never sees the key | On main ([#1500](https://github.com/OpenAdaptAI/OpenAdapt/pull/1500)) | Unit tests; opt-in test on a real Secure Enclave | `packages/desktop/openadapt_desktop/approver_keychain.py`, [APPROVAL_MODES.md](packages/flow/docs/APPROVAL_MODES.md#the-approver-key) |
Expand Down
Loading
Loading