Skip to content

feat(flow): one verifier API and a durable uncertain-write ledger (standalone part of #1628) #1642

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
abrichr wants to merge 1 commit into from
Closed

feat(flow): one verifier API and a durable uncertain-write ledger (standalone part of #1628) #1642

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CAPABILITIES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,8 @@ improvising or retrying when something is off.
| Prove that nothing outside the declared change moved (frame conditions), and attribute concurrent edits by other users through the app's audit log or a modified-by field; the receipt signs which parts were checked | On main | Synthetic (MockMed, SQLite audit log) | `verify-run`, [VERIFICATION_COVERAGE.md](packages/flow/docs/VERIFICATION_COVERAGE.md) |
| Admit a record check only after a fault suite written to a store copy shows it catches every fault class it claims, including writes outside its read (another table, a side row for the saved record), and sign each class's result into the receipt | On main | Synthetic (in-memory REST store, SQLite copy); OpenEMR stand-in built from the pinned fixture's dump | `openadapt_flow/fault_gate.py`, `scaffold-verifier --fault-gate`, `verify-run` `admission:` |
| Check that the saved record matches its source document, not only the parameters: intake keeps each value's line, a second reading must agree before a guarded save, and a round trip compares the read-back with the document | On main | Synthetic (MockMed, text referral) | `openadapt_flow/provenance.py`, `intake`, `verify-run --provenance`, guarded save |
| Check one write from Python, without the CLI: read the store before the write, then judge it against the same contract (`capture`, `verify_write`), with the receipt and which parts were checked | On main | Synthetic (SQLite) | `openadapt_flow/verifier.py` |
| Check one write from Python, without the CLI: one API (`openadapt_flow.verifier`: `begin_write`, `verify_write`; 2.4.1's `capture` form still works, deprecated) with one outcome (`CONFIRMED`, `REFUTED`, `UNCERTAIN`), the receipt, which parts were checked, and the result's scope (read path, stores read, time window, what it doesn't cover) | On main | Synthetic (SQLite; MockMed's fault server over real HTTP) | `openadapt_flow/verifier.py`, [VERIFIER_API.md](packages/flow/docs/VERIFIER_API.md) |
| Never send a write twice while its outcome is unknown: an append-only ledger on local disk (pluggable backend), keyed by the write's identity, that `begin_write` consults before anything is sent; a read-back against the attempt's own baseline resolves it, and a refuted write waits for a person (`reconcile`, `release` in the Python API) | On main | Synthetic (MockMed's fault server: a worker killed between the send and the read-back, killed during a timed-out send, killed before the send, a late write, and two worker processes started together on one write) | `openadapt_flow/write_ledger.py`, [VERIFIER_API.md](packages/flow/docs/VERIFIER_API.md#the-uncertain-write-ledger) |
| Guarded save: a safety layer any agent wraps around one write (check the record, hold for approval, read back) | On main ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)) | Synthetic | `GUARDED_SAVE.md` |
| A person's approval in Desktop is signed with a key kept behind Touch ID on a Mac: one-prompt setup in the app, a prompt for every answer, and Flow never sees the key | On main ([#1500](https://github.com/OpenAdaptAI/OpenAdapt/pull/1500)) | Unit tests; opt-in test on a real Secure Enclave | `packages/desktop/openadapt_desktop/approver_keychain.py`, [APPROVAL_MODES.md](packages/flow/docs/APPROVAL_MODES.md#the-approver-key) |
| The approver root Desktop trusts can be pinned by an administrator, in a file this user can't write (macOS, Linux, Windows); Desktop checks the owner, mode or ACL and refuses links. A Mac without one uses the root it set up behind Touch ID, read with Keychain prompts off; the two disagreeing is refused | On main | Unit tests per attack on real files; Windows ACL rules unit-tested, real-ACL test on Windows CI; opt-in test against the real login Keychain | `openadapt-desktop approver-pin`, `packages/desktop/openadapt_desktop/approver_anchor.py`, [APPROVAL_MODES.md](packages/flow/docs/APPROVAL_MODES.md#the-approver-key) |
Expand Down
96 changes: 95 additions & 1 deletion llms-full.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -544,7 +544,8 @@ improvising or retrying when something is off.
| Prove that nothing outside the declared change moved (frame conditions), and attribute concurrent edits by other users through the app's audit log or a modified-by field; the receipt signs which parts were checked | On main | Synthetic (MockMed, SQLite audit log) | `verify-run`, [VERIFICATION_COVERAGE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFICATION_COVERAGE.md) |
| Admit a record check only after a fault suite written to a store copy shows it catches every fault class it claims, including writes outside its read (another table, a side row for the saved record), and sign each class's result into the receipt | On main | Synthetic (in-memory REST store, SQLite copy); OpenEMR stand-in built from the pinned fixture's dump | `openadapt_flow/fault_gate.py`, `scaffold-verifier --fault-gate`, `verify-run` `admission:` |
| Check that the saved record matches its source document, not only the parameters: intake keeps each value's line, a second reading must agree before a guarded save, and a round trip compares the read-back with the document | On main | Synthetic (MockMed, text referral) | `openadapt_flow/provenance.py`, `intake`, `verify-run --provenance`, guarded save |
| Check one write from Python, without the CLI: read the store before the write, then judge it against the same contract (`capture`, `verify_write`), with the receipt and which parts were checked | On main | Synthetic (SQLite) | `openadapt_flow/verifier.py` |
| Check one write from Python, without the CLI: one API (`openadapt_flow.verifier`: `begin_write`, `verify_write`; 2.4.1's `capture` form still works, deprecated) with one outcome (`CONFIRMED`, `REFUTED`, `UNCERTAIN`), the receipt, which parts were checked, and the result's scope (read path, stores read, time window, what it doesn't cover) | On main | Synthetic (SQLite; MockMed's fault server over real HTTP) | `openadapt_flow/verifier.py`, [VERIFIER_API.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFIER_API.md) |
| Never send a write twice while its outcome is unknown: an append-only ledger on local disk (pluggable backend), keyed by the write's identity, that `begin_write` consults before anything is sent; a read-back against the attempt's own baseline resolves it, and a refuted write waits for a person (`reconcile`, `release` in the Python API) | On main | Synthetic (MockMed's fault server: a worker killed between the send and the read-back, killed during a timed-out send, killed before the send, a late write, and two worker processes started together on one write) | `openadapt_flow/write_ledger.py`, [VERIFIER_API.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFIER_API.md#the-uncertain-write-ledger) |
| Guarded save: a safety layer any agent wraps around one write (check the record, hold for approval, read back) | On main ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)) | Synthetic | `GUARDED_SAVE.md` |
| A person's approval in Desktop is signed with a key kept behind Touch ID on a Mac: one-prompt setup in the app, a prompt for every answer, and Flow never sees the key | On main ([#1500](https://github.com/OpenAdaptAI/OpenAdapt/pull/1500)) | Unit tests; opt-in test on a real Secure Enclave | `packages/desktop/openadapt_desktop/approver_keychain.py`, [APPROVAL_MODES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/APPROVAL_MODES.md#the-approver-key) |
| The approver root Desktop trusts can be pinned by an administrator, in a file this user can't write (macOS, Linux, Windows); Desktop checks the owner, mode or ACL and refuses links. A Mac without one uses the root it set up behind Touch ID, read with Keychain prompts off; the two disagreeing is refused | On main | Unit tests per attack on real files; Windows ACL rules unit-tested, real-ACL test on Windows CI; opt-in test against the real login Keychain | `openadapt-desktop approver-pin`, `packages/desktop/openadapt_desktop/approver_anchor.py`, [APPROVAL_MODES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/APPROVAL_MODES.md#the-approver-key) |
Expand Down Expand Up @@ -1556,6 +1557,99 @@ The binding ties the click to the entry the run made. It doesn't check that
the entry is right: the values typed come from the run's parameters, and only
a record check can confirm what the application saved.

A labelled save on a create-new form (Trac's **Create ticket**, NetBox's
**Create**) has the same problem: no existing record is on screen next to it,
so it compiled with no identity check, and a policy that refuses unarmed saves
refused it. Its record is the one the run is typing into the form. When the
recording shows the save is the submit button of the one form that owns every
field typed since the last save, the compiler gives it a **form identity**
(`Anchor.form_identity`). Before the click, a run requires (identity mode
`form`):

- the button is still that form's submit button, and the form owns exactly the
fields this run typed into, each holding exactly what the run typed;
- the form submits to the same origin, path and query parameter names with the
same method, and would send the same set of field names and types, hidden
and file ones included, so an edit form that looks like the create form, or
an address, method or record id a script changed or added, halts;
- every field the run did not type into (a hidden parent or record id, a
select left on its default, a checked box, a query parameter's value) holds
its demonstrated value, stored as a salted digest. Only a per-page token is
not compared: a hidden field or query value named like one (`csrf`, `token`,
`nonce` and the like) whose value is shaped like one (16 characters or more,
not all digits) and is not shaped like a record id (hex groups joined by `-`
or `_`, as a UUID is) unless it changed while the demonstration was filled
in. A short, numeric or record-shaped value under such a name, and a value a
script set while the form was filled in, are compared. A program that
should create under a different parent on each run therefore halts until
that parent is checked another way.

The checks above read the page before the click; they are pre-checks. The
safety layer is the wire check (`openadapt_flow/wire_check.py`, shared with
taught browser writes): from the check until the next step, a Playwright route
on the browser context holds every write request from any of its pages (a
popup or new tab the click opens included), and a GET navigation to the
form's address, at the network, where page scripts can't reach it. A request
leaves only when its method, address, query and fields equal what the checked
button would send (form fields or a flat JSON object, every value except the
exempt tokens), and only once; anything else is aborted before it is sent. So
a page listener that runs after the check (a submit, capture or `formdata`
listener, a click handler that fills in or enables a field, a timer), a save
sent by script (`form.submit()`, `fetch`), a write from a popup, a submit from
another form and a second submit can't send anything the check didn't see. A
save sent in another shape (nested JSON, JSON with a repeated key, a custom
encoding) can't be compared, so it is held too. The run watches the wire check
for two seconds after the click, then again at the start of the next step and
at the end of the run, and halts on any held request.

What the wire check doesn't see:

- **Service workers.** Their requests skip routes. The runner's browsers block
service workers; on a page one controls anyway, the save is refused before
the click, and one that starts while the save is held halts the run.
- **WebSockets.** One opened while the save is held is never connected to the
server. A message sent on one opened earlier can't be stopped: the run
halts on it, after it was sent. A socket that a save opens just as the page
navigates away closes with the page; it isn't connected, and the run may
not report it.
- **Redirects.** A 307 or 308 answer to the checked save makes the browser send
the same request again to the address the server names, possibly another
site. The server that accepted the save chooses that.
- **GET requests.** Reads pass. An app whose GET addresses change records
(some do, OpenEMR among them) is outside the wire check; its record check
still applies.
- **Late requests.** A submission a script delays past the end of the run is
still held until the page closes, but the run, already over, doesn't
report it.

A page whose own listener changes a field as the form is sent changes what is
sent after the check, so its save halts even when the demonstration did the
same: a false halt, in the safe direction. The compiler treats a click on the
submit button of the form the demonstration typed into as a save, whatever
the screen did afterwards, so a slow recording still arms the check. When the
check can't be armed on such a save, the step names why, and the save is
refused.

The digests of untouched values are salted; with the per-bundle salt stored in
the bundle (the default, unless `OPENADAPT_FLOW_IDENTITY_SALT` is set), a
low-entropy value such as a numeric id or a select option can be recovered by
trying candidates. The recording itself holds those values in plain text, as
it holds the typed ones.

A backend that can't read the page leaves the identity unreadable, which halts
too. Each field must hold exactly what the run typed, so a controlled input
that reformats the value halts: one that reformats on every keystroke stops at
the typed-value read-back, and one that reformats when it loses focus (a
phone-number mask, for example) stops at the form check. A field that a React
keyed remount replaces with a new element, even with the same value, also
halts: the new element isn't one the run typed into. A submit button outside
the `<form>` element counts when its `form=` attribute names that form. One
with no form owner at all (a React portal that submits from script) gets no
form identity, so its save stays unarmed.

The form identity ties the click to the entry the run made in the demonstrated
form. It doesn't prove the app saved it correctly: only a record check does.

For consequential or entity-sensitive workflows, require a policy that refuses
unarmed actions, review the inferred risk and tags, and test identity against
real near-match and same-name records. Pure-pixel substrates should expect more
Expand Down
Loading
Loading