diff --git a/CAPABILITIES.md b/CAPABILITIES.md index e3e7e830d..f27cfb77e 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -147,7 +147,8 @@ improvising or retrying when something is off. | Prove that nothing outside the declared change moved (frame conditions), and attribute concurrent edits by other users through the app's audit log or a modified-by field; the receipt signs which parts were checked | On main | Synthetic (MockMed, SQLite audit log) | `verify-run`, [VERIFICATION_COVERAGE.md](packages/flow/docs/VERIFICATION_COVERAGE.md) | | Admit a record check only after a fault suite written to a store copy shows it catches every fault class it claims, including writes outside its read (another table, a side row for the saved record), and sign each class's result into the receipt | On main | Synthetic (in-memory REST store, SQLite copy); OpenEMR stand-in built from the pinned fixture's dump | `openadapt_flow/fault_gate.py`, `scaffold-verifier --fault-gate`, `verify-run` `admission:` | | Check that the saved record matches its source document, not only the parameters: intake keeps each value's line, a second reading must agree before a guarded save, and a round trip compares the read-back with the document | On main | Synthetic (MockMed, text referral) | `openadapt_flow/provenance.py`, `intake`, `verify-run --provenance`, guarded save | -| Check one write from Python, without the CLI: read the store before the write, then judge it against the same contract (`capture`, `verify_write`), with the receipt and which parts were checked | On main | Synthetic (SQLite) | `openadapt_flow/verifier.py` | +| Check one write from Python, without the CLI: one API (`openadapt_flow.verifier`: `begin_write`, `verify_write`; 2.4.1's `capture` form still works, deprecated) with one outcome (`CONFIRMED`, `REFUTED`, `UNCERTAIN`), the receipt, which parts were checked, and the result's scope (read path, stores read, time window, what it doesn't cover) | On main | Synthetic (SQLite; MockMed's fault server over real HTTP) | `openadapt_flow/verifier.py`, [VERIFIER_API.md](packages/flow/docs/VERIFIER_API.md) | +| Never send a write twice while its outcome is unknown: an append-only ledger on local disk (pluggable backend), keyed by the write's identity, that `begin_write` consults before anything is sent; a read-back against the attempt's own baseline resolves it, and a refuted write waits for a person (`reconcile`, `release` in the Python API) | On main | Synthetic (MockMed's fault server: a worker killed between the send and the read-back, killed during a timed-out send, killed before the send, a late write, and two worker processes started together on one write) | `openadapt_flow/write_ledger.py`, [VERIFIER_API.md](packages/flow/docs/VERIFIER_API.md#the-uncertain-write-ledger) | | Guarded save: a safety layer any agent wraps around one write (check the record, hold for approval, read back) | On main ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)) | Synthetic | `GUARDED_SAVE.md` | | A person's approval in Desktop is signed with a key kept behind Touch ID on a Mac: one-prompt setup in the app, a prompt for every answer, and Flow never sees the key | On main ([#1500](https://github.com/OpenAdaptAI/OpenAdapt/pull/1500)) | Unit tests; opt-in test on a real Secure Enclave | `packages/desktop/openadapt_desktop/approver_keychain.py`, [APPROVAL_MODES.md](packages/flow/docs/APPROVAL_MODES.md#the-approver-key) | | The approver root Desktop trusts can be pinned by an administrator, in a file this user can't write (macOS, Linux, Windows); Desktop checks the owner, mode or ACL and refuses links. A Mac without one uses the root it set up behind Touch ID, read with Keychain prompts off; the two disagreeing is refused | On main | Unit tests per attack on real files; Windows ACL rules unit-tested, real-ACL test on Windows CI; opt-in test against the real login Keychain | `openadapt-desktop approver-pin`, `packages/desktop/openadapt_desktop/approver_anchor.py`, [APPROVAL_MODES.md](packages/flow/docs/APPROVAL_MODES.md#the-approver-key) | diff --git a/llms-full.txt b/llms-full.txt index 8ee360be8..7934daeb4 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -544,7 +544,8 @@ improvising or retrying when something is off. | Prove that nothing outside the declared change moved (frame conditions), and attribute concurrent edits by other users through the app's audit log or a modified-by field; the receipt signs which parts were checked | On main | Synthetic (MockMed, SQLite audit log) | `verify-run`, [VERIFICATION_COVERAGE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFICATION_COVERAGE.md) | | Admit a record check only after a fault suite written to a store copy shows it catches every fault class it claims, including writes outside its read (another table, a side row for the saved record), and sign each class's result into the receipt | On main | Synthetic (in-memory REST store, SQLite copy); OpenEMR stand-in built from the pinned fixture's dump | `openadapt_flow/fault_gate.py`, `scaffold-verifier --fault-gate`, `verify-run` `admission:` | | Check that the saved record matches its source document, not only the parameters: intake keeps each value's line, a second reading must agree before a guarded save, and a round trip compares the read-back with the document | On main | Synthetic (MockMed, text referral) | `openadapt_flow/provenance.py`, `intake`, `verify-run --provenance`, guarded save | -| Check one write from Python, without the CLI: read the store before the write, then judge it against the same contract (`capture`, `verify_write`), with the receipt and which parts were checked | On main | Synthetic (SQLite) | `openadapt_flow/verifier.py` | +| Check one write from Python, without the CLI: one API (`openadapt_flow.verifier`: `begin_write`, `verify_write`; 2.4.1's `capture` form still works, deprecated) with one outcome (`CONFIRMED`, `REFUTED`, `UNCERTAIN`), the receipt, which parts were checked, and the result's scope (read path, stores read, time window, what it doesn't cover) | On main | Synthetic (SQLite; MockMed's fault server over real HTTP) | `openadapt_flow/verifier.py`, [VERIFIER_API.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFIER_API.md) | +| Never send a write twice while its outcome is unknown: an append-only ledger on local disk (pluggable backend), keyed by the write's identity, that `begin_write` consults before anything is sent; a read-back against the attempt's own baseline resolves it, and a refuted write waits for a person (`reconcile`, `release` in the Python API) | On main | Synthetic (MockMed's fault server: a worker killed between the send and the read-back, killed during a timed-out send, killed before the send, a late write, and two worker processes started together on one write) | `openadapt_flow/write_ledger.py`, [VERIFIER_API.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFIER_API.md#the-uncertain-write-ledger) | | Guarded save: a safety layer any agent wraps around one write (check the record, hold for approval, read back) | On main ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)) | Synthetic | `GUARDED_SAVE.md` | | A person's approval in Desktop is signed with a key kept behind Touch ID on a Mac: one-prompt setup in the app, a prompt for every answer, and Flow never sees the key | On main ([#1500](https://github.com/OpenAdaptAI/OpenAdapt/pull/1500)) | Unit tests; opt-in test on a real Secure Enclave | `packages/desktop/openadapt_desktop/approver_keychain.py`, [APPROVAL_MODES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/APPROVAL_MODES.md#the-approver-key) | | The approver root Desktop trusts can be pinned by an administrator, in a file this user can't write (macOS, Linux, Windows); Desktop checks the owner, mode or ACL and refuses links. A Mac without one uses the root it set up behind Touch ID, read with Keychain prompts off; the two disagreeing is refused | On main | Unit tests per attack on real files; Windows ACL rules unit-tested, real-ACL test on Windows CI; opt-in test against the real login Keychain | `openadapt-desktop approver-pin`, `packages/desktop/openadapt_desktop/approver_anchor.py`, [APPROVAL_MODES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/APPROVAL_MODES.md#the-approver-key) | @@ -1556,6 +1557,99 @@ The binding ties the click to the entry the run made. It doesn't check that the entry is right: the values typed come from the run's parameters, and only a record check can confirm what the application saved. +A labelled save on a create-new form (Trac's **Create ticket**, NetBox's +**Create**) has the same problem: no existing record is on screen next to it, +so it compiled with no identity check, and a policy that refuses unarmed saves +refused it. Its record is the one the run is typing into the form. When the +recording shows the save is the submit button of the one form that owns every +field typed since the last save, the compiler gives it a **form identity** +(`Anchor.form_identity`). Before the click, a run requires (identity mode +`form`): + +- the button is still that form's submit button, and the form owns exactly the + fields this run typed into, each holding exactly what the run typed; +- the form submits to the same origin, path and query parameter names with the + same method, and would send the same set of field names and types, hidden + and file ones included, so an edit form that looks like the create form, or + an address, method or record id a script changed or added, halts; +- every field the run did not type into (a hidden parent or record id, a + select left on its default, a checked box, a query parameter's value) holds + its demonstrated value, stored as a salted digest. Only a per-page token is + not compared: a hidden field or query value named like one (`csrf`, `token`, + `nonce` and the like) whose value is shaped like one (16 characters or more, + not all digits) and is not shaped like a record id (hex groups joined by `-` + or `_`, as a UUID is) unless it changed while the demonstration was filled + in. A short, numeric or record-shaped value under such a name, and a value a + script set while the form was filled in, are compared. A program that + should create under a different parent on each run therefore halts until + that parent is checked another way. + +The checks above read the page before the click; they are pre-checks. The +safety layer is the wire check (`openadapt_flow/wire_check.py`, shared with +taught browser writes): from the check until the next step, a Playwright route +on the browser context holds every write request from any of its pages (a +popup or new tab the click opens included), and a GET navigation to the +form's address, at the network, where page scripts can't reach it. A request +leaves only when its method, address, query and fields equal what the checked +button would send (form fields or a flat JSON object, every value except the +exempt tokens), and only once; anything else is aborted before it is sent. So +a page listener that runs after the check (a submit, capture or `formdata` +listener, a click handler that fills in or enables a field, a timer), a save +sent by script (`form.submit()`, `fetch`), a write from a popup, a submit from +another form and a second submit can't send anything the check didn't see. A +save sent in another shape (nested JSON, JSON with a repeated key, a custom +encoding) can't be compared, so it is held too. The run watches the wire check +for two seconds after the click, then again at the start of the next step and +at the end of the run, and halts on any held request. + +What the wire check doesn't see: + +- **Service workers.** Their requests skip routes. The runner's browsers block + service workers; on a page one controls anyway, the save is refused before + the click, and one that starts while the save is held halts the run. +- **WebSockets.** One opened while the save is held is never connected to the + server. A message sent on one opened earlier can't be stopped: the run + halts on it, after it was sent. A socket that a save opens just as the page + navigates away closes with the page; it isn't connected, and the run may + not report it. +- **Redirects.** A 307 or 308 answer to the checked save makes the browser send + the same request again to the address the server names, possibly another + site. The server that accepted the save chooses that. +- **GET requests.** Reads pass. An app whose GET addresses change records + (some do, OpenEMR among them) is outside the wire check; its record check + still applies. +- **Late requests.** A submission a script delays past the end of the run is + still held until the page closes, but the run, already over, doesn't + report it. + +A page whose own listener changes a field as the form is sent changes what is +sent after the check, so its save halts even when the demonstration did the +same: a false halt, in the safe direction. The compiler treats a click on the +submit button of the form the demonstration typed into as a save, whatever +the screen did afterwards, so a slow recording still arms the check. When the +check can't be armed on such a save, the step names why, and the save is +refused. + +The digests of untouched values are salted; with the per-bundle salt stored in +the bundle (the default, unless `OPENADAPT_FLOW_IDENTITY_SALT` is set), a +low-entropy value such as a numeric id or a select option can be recovered by +trying candidates. The recording itself holds those values in plain text, as +it holds the typed ones. + +A backend that can't read the page leaves the identity unreadable, which halts +too. Each field must hold exactly what the run typed, so a controlled input +that reformats the value halts: one that reformats on every keystroke stops at +the typed-value read-back, and one that reformats when it loses focus (a +phone-number mask, for example) stops at the form check. A field that a React +keyed remount replaces with a new element, even with the same value, also +halts: the new element isn't one the run typed into. A submit button outside +the `
` element counts when its `form=` attribute names that form. One +with no form owner at all (a React portal that submits from script) gets no +form identity, so its save stays unarmed. + +The form identity ties the click to the entry the run made in the demonstrated +form. It doesn't prove the app saved it correctly: only a record check does. + For consequential or entity-sensitive workflows, require a policy that refuses unarmed actions, review the inferred risk and tags, and test identity against real near-match and same-name records. Pure-pixel substrates should expect more diff --git a/packages/flow/docs/VERIFIER_API.md b/packages/flow/docs/VERIFIER_API.md new file mode 100644 index 000000000..467f86fee --- /dev/null +++ b/packages/flow/docs/VERIFIER_API.md @@ -0,0 +1,316 @@ +# Verifier API + +One public module checks one write against the system of record: +`openadapt_flow.verifier`. It replaces the two overlapping APIs proposed in +#1588 (`openadapt_flow.verifier`) and #1589 (`openadapt_flow.verify`). It +wraps `verify_run` and checks exactly what verify-run checks. The module and +the ledger (`openadapt_flow/write_ledger.py`) stand on their own. The +`openadapt_flow.verify` wrapper, the MCP verify tools and the `openadapt +verify ledger` commands described below belong to #1589's surface and move +onto this API with it. + +## The call + +```python +from openadapt_flow.verifier import Actor, Contract, begin_write, verify_write + +contract = Contract.load("verify.yaml") # openadapt.verify/v1 +pending = begin_write(contract, params, actor=Actor("agent", "intake-bot")) +... # the caller sends the write +result = verify_write(pending) +if result.outcome is Outcome.CONFIRMED: + print(result.receipt_sha256) +else: + print(result.outcome, result.reasons) # a person checks; don't resend +``` + +`begin_write` does three things, in this order: + +1. Consults the uncertain-write ledger for this write's identity. If an + earlier attempt is still open or unresolved, it raises (see + [Before the write](#before-the-write)) and nothing is sent. +2. Reads the system of record (the baseline). If the store can't be read, it + raises `BaselineUnreadable` and nothing is sent. +3. Records the attempt in the ledger, then returns a `PendingWrite`. + +`verify_write` reads the store again, judges the write against the baseline, +and records the outcome in the ledger. It never retries or resends the write. + +`verified_write(...)` is the same pair as a context manager. + +## One outcome vocabulary + +`result.outcome` is one of three values. They answer one question: does the +system of record show the declared write, and only it? + +| Outcome | Meaning | Resend? | +|---|---|---| +| `CONFIRMED` | Every declared effect was read back from the system of record at the contract's minimum strength. Nothing else in the read set moved, except what the frame excuses. The source check held or wasn't asked for. | Not needed | +| `REFUTED` | The store shows something other than the declared write: a wrong value, another record, a duplicate, or a change outside the declared write. | Never. A person checks. | +| `UNCERTAIN` | The store doesn't show the write, couldn't be read, or was read too weakly to certify it. The write may still land. | Not until reconciliation resolves it | + +`result.verified` is `outcome is Outcome.CONFIRMED`. `result.retry_safe` is +true in one case only: the caller said the write was never sent and the store +shows nothing landed (verify-run's `halted_before_effect`). + +A caller that can't tell whether the write was sent, such as a tool that +raised before or while acting, leaves `write_attempted=True`. A store that +shows nothing is then `UNCERTAIN`, never `retry_safe`. + +**Delivery is the caller's call.** `verify_write(pending, delivered=False)` +records the outcome but marks it undelivered, for example when a tool call +was cancelled or timed out after the read-back. The next identical call +raises `AlreadyWritten` with that result instead of sending again. + +**Who declared each value.** `begin_write(..., declared_by=...)` takes one +name (`"agent"`) or a map from value name to who set it (`{"clinic": +"operator", "note": "agent"}`). `to_dict()` returns it as +`values_declared_by`. To bind it into the receipt as well, put it in +`Actor.build`, which verify-run digests. + +### Mapping from the old names + +| Source | Old name | New outcome | +|---|---|---| +| verify-run receipt `outcome` | `verified` | `CONFIRMED` | +| | `halted_before_effect` | `REFUTED`, `retry_safe=True` | +| | `rejected_policy` (self-approved) | `REFUTED` | +| | `reconciliation_required` | `REFUTED` or `UNCERTAIN` (rule below) | +| | `failed_platform` | `UNCERTAIN` | +| #1588 `WriteResult.status` | `verified` / `refuted` / `unconfirmed` | `CONFIRMED` / `REFUTED` / `UNCERTAIN` | +| #1589 `verdict` | `VERIFIED` / `REFUTED` / `RECONCILIATION_REQUIRED` | `CONFIRMED` / `REFUTED` / `UNCERTAIN` | +| #1609 adapter reply | `NOT_WRITTEN` | No outcome: the write was blocked before it was sent (`written: false`) | + +`reconciliation_required` splits with #1588's rule. It's `REFUTED` when the +frame or the source check is refuted, or when a failed effect was observed +`present` or `conflicting`. It's `UNCERTAIN` when every failed effect was +observed `absent`, or a part wasn't checked. One behavior change from #1589: +a write that lands on another patient now comes back `REFUTED` when the frame +sees the other record move. #1589 called it `RECONCILIATION_REQUIRED`. Both +outcomes block a resend. + +The receipt keeps verify-run's `outcome` unchanged. `result.receipt_outcome` +carries it, and the per-part `checked` map (`delta`, `frame`, `source`: each +`held`, `refuted` or `not_checked`) stays as it is. + +## What a result establishes + +Every result carries `result.scope`, so a reader can tell what was and wasn't +checked: + +```python +@dataclass(frozen=True) +class Scope: + read_paths: tuple[str, ...] # "database", "api", "fhir", "audit_log", "screen", "file" + stores: tuple[str, ...] # each store read, e.g. "sql", "rest", "frame.database", "audit" + window: Window # baseline_read_at, read_at, max_execution_window_s + strength: str # "independent_system_of_record" | "independent_session" | "unchecked" + not_covered: tuple[str, ...] # what this result says nothing about +``` + +The read path comes from each verifier's substrate: `sql` is `database`; +`rest` and `graphql` are `api`; `fhir` is `fhir`; the frame's audit read is +`audit_log`; `onscreen`, `ui_readback` and `fresh_session` are `screen`. + +`not_covered` always lists: + +- writes after `read_at` (delayed or queued writes); +- stores the contract doesn't read, by name when known: email, other + services, files; +- for a contract with no `frame.database`, tables outside the read set. + +`result.to_dict()` is the JSON reply for agents and the CLI. It holds closed +fields only: no record values. + +## Before the write + +`begin_write` raises a subclass of `WriteBlocked` when the write must not be +sent. Adapters turn it into a `written: false` reply. + +| Exception | When | +|---|---| +| `BaselineUnreadable` | The store can't be read, so nothing could verify the write afterwards. | +| `WriteHeld` | An earlier attempt with the same identity is in flight in another worker, `UNCERTAIN`, or `REFUTED` and not yet released by a person. | +| `AlreadyWritten` | An earlier attempt was `CONFIRMED` but its reply never reached the caller (a cancellation, a timeout, a crash). It carries the earlier result, and the write isn't sent again. | + +## The uncertain-write ledger + +The ledger stops a restart or a second worker from resending a write whose +outcome isn't known. + +**Identity.** A write's key is an HMAC-SHA256 over three things: + +- the contract's identity (`Contract.identity`): each effect's kind, target + and values, unbound, and the store (`Contract.store`: the verifier's kind + and location and `environment_id`, without credentials; a loopback host + is one host); +- the bound values, normalized so that `7` and `"7"` are the same write; +- an optional caller `write_key`. + +Timeouts, the window and the frame are left out of the identity, so editing +them between a crash and a restart can't make the same write look new. An +address is spelled one way: + +- scheme and host lower case; +- a loopback host as `loopback`; +- no default port; +- no trailing slash. + +Environment variable names (`*_env`, for example the token that picks a +tenant) stay in the identity; secret values don't. The same write to another +store is its own write. But the same effects and values might also be +recorded under another address for the same store: the same verifier kind, +`environment_id` and host, with the address split or spelled differently. +If that other attempt isn't free (it's in flight, unresolved, or already +made), `begin_write` holds the write: a hold beats a resend. Give each +store its own `environment_id`, so two stores on one host never hold each +other. + +If the store's address is edited between a crash and the restart, the old +baseline was read by a different verifier and can't be judged. The attempt +stays held until a person checks the record. + +The HMAC key is a secret in the ledger folder (mode 0600), so the ledger +never holds values or plain digests of them. A landed write without a +`write_key` is handed back to an identical call for `repeat_s` (default 10 +minutes); after that, the same values are a new write. Inside that window, +two intended writes with the same values need distinct `write_key`s. With a +`write_key`, a landed write is never made again under it. The MCP tool +`verify_begin` takes an optional `write_key`. + +**Storage.** `FileLedger` is the default, in `~/.openadapt/ledger/` +(`OPENADAPT_LEDGER_DIR` overrides it). It keeps one append-only JSON Lines +file per key. Events are `begun`, `resolved`, `delivered` and `released`. +Nothing is rewritten or deleted. A `LedgerBackend` protocol (`events`, +`append`, `lease`) lets a deployment keep it elsewhere, such as a shared +database for workers on several hosts. `MemoryLedger` is for tests. +`ledger=None` turns the ledger off; use it only for a sandbox that resets +between runs, such as RL episodes. + +**Leases.** `begin_write` takes an exclusive lease on the key, without +waiting (`flock` with `LOCK_NB` on the lease's own open file), and holds it +until `verify_write` returns. Whichever thread runs `verify_write` releases +it, and a second `begin_write` in the same process is refused, never +blocked. The operating system +releases a lease when its process dies. So a second worker that can't get the +lease knows the first is alive and refuses (`WriteHeld`). A worker that gets +the lease while an attempt is still open knows its owner died between the +send and the read-back, and reconciles it before anything else. + +**State of a key.** This is decided from its latest attempt: + +| Latest attempt | `begin_write` | +|---|---| +| none, `retry_safe`, or landed and delivered more than `repeat_s` ago (no `write_key`) | new attempt | +| open, lease held elsewhere | `WriteHeld` (in flight) | +| open, lease free (owner died) | reconcile, then decide again | +| landed (`CONFIRMED`, or `landed`): undelivered, inside `repeat_s`, or under a `write_key` | `AlreadyWritten` with the stored result | +| `UNCERTAIN` | reconcile, then decide again | +| `REFUTED` | `WriteHeld` until `release` | + +**Reconciliation.** `reconcile(contract, key)` resolves an attempt by reading +back. Callers include `begin_write`, `openadapt verify ledger reconcile`, and +an operator's job. + +- It judges the attempt against the attempt's own baseline, so a record that + was there before is never taken for this write. If the process died between + the send and the read-back, that baseline is unused. If an `UNCERTAIN` + check already used it, reconciliation reads a private copy. That's still + the same write, and the lease stops any other attempt at it from being + sent in the meantime. A copy carries `remediates` with the earlier + receipt's digest. +- `CONFIRMED` and `REFUTED` resolve the attempt. A `CONFIRMED` reconciliation + is handed back once (`AlreadyWritten`) instead of a resend. +- `UNCERTAIN` keeps the attempt held. Past the contract's + `max_execution_window_s`, the baseline proves nothing, so the attempt stays + held. A held attempt is released by a person. An attempt the store still + shows `absent` is also released once the ledger's `settle_s` has passed. + `settle_s` defaults to off, because a queued write can land late. +- An attempt whose process died before it sent anything looks the same as + one that died just after: it stays held until a person, or `settle_s`, + releases it. + +**Other writes in the window.** `verify_write` and reconciliation both check +the result against attempts of other keys that were open during its window, +from just before its baseline read to the end of its read-back. + +- **Same values.** One record can't prove two writes. If another attempt + with the same values was open during the window, a `CONFIRMED` result + becomes `UNCERTAIN` and is held for a person. That other attempt may have + written the record, or this attempt's send may have stored nothing. +- **Added records from other writes.** A correct write can see records that + other workers added in the same window. The excuse applies only when all + of these hold: + - its declared records landed exactly (`delta` held); + - the only frame breach is added records; + - there are no more of them than other attempts on the same store wrote + in the window, counting attempts read back as landed or still holding + a live lease. + + Then the result is `UNCERTAIN` with `landed: true` and `review: true`. It + isn't confirmed, and it's never sent again. It stays listed in `openadapt + verify ledger list` until a person releases it, since a count can't name + the records. While it's under review, an identical call is handed back + even after `repeat_s`, so no later attempt hides it. An abandoned attempt, + or one on another store, never excuses anything, and a breach nothing + explains stays `REFUTED`. +- **Which open attempts count.** An attempt still open counts while its + owner holds the lease, or for the contract's window after it began, since + a write it sent before its owner died may still land. Once a person + releases an attempt, it's closed. Two concurrent tasks with the same + values and different write keys both end `UNCERTAIN` and wait for a + person. That's the conservative call. +- **Lease checks.** Checking whether another key's owner is alive takes + that key's lease for an instant. A `begin_write` for that key at the same + moment is refused as in flight. The refusal is safe, but the caller sees + it. + +**What's kept on disk.** While an attempt may still need a read-back, its +baseline and values stay in its private folder (mode 0700, files 0600) next +to the ledger, as verify-run keeps baselines today. Once it resolves for +good (landed, refuted, retry-safe or released), they're removed. Receipts +stay; they hold no record values. + +**People.** `openadapt verify ledger list` shows open and held keys (no +values). `release KEY --by NAME --reason TEXT --found absent|landed` records what a person found after they checked the record. `absent` frees the key, so the write may be sent again. `landed` resolves the attempt as landed, so it's handed back and never sent again. `reconcile [KEY]` reads back. + +## Modules + +| Module | After this change | +|---|---| +| `openadapt_flow.verifier` | The one public API: `Contract`, `Actor`, `Source`, `begin_write`, `verify_write`, `verified_write`, `reconcile`, `WriteResult`, `Outcome`, `Scope`, `PendingWrite`, the `WriteBlocked` family, `VerifierError`, the ledger classes, #1588's `capture_database`/`diff_database`, and its deprecated `capture`/`Snapshot`. Tests pin the signatures. | +| `openadapt_flow.verify` (#1589) | A thin, deprecated wrapper with the same signatures. It warns on import. Results are one `WriteResult`, but `isinstance(r, VerifiedWrite)` still means `CONFIRMED`. `to_dict()` keeps `verdict`, `seal`, `seal_path`, `values_declared_by` and `authorization`. Its `BaselineUnreadable` is `WriteBlocked`, so existing `except BaselineUnreadable` blocks also refuse held writes. The message says which refusal it was; an `AlreadyWritten` carries the earlier `.result`. | +| `openadapt.verify` (launcher) | Re-exports `openadapt_flow.verifier`. | +| `agent/verify_tools.py` (MCP) | Calls `begin_write`/`verify_write`; the ledger applies through them. | +| #1609 `WriteGuard` | Its in-memory `_held` and `_lost_replies` become ledger reads. The owner of #1609 drops them and catches `AlreadyWritten`. | + +#1588's `capture(contract, params)` and `verify_write(contract, before, +params=..., actor=...)` shipped in 2.4.1, so they keep working, deprecated: +`capture` warns, and a snapshot still verifies once, only under the state +directory it was captured in (`OPENADAPT_VERIFIER_STATE_DIR`). The result is +the one `WriteResult`: `status` gives 2.4.1's `verified`, `refuted` or +`unconfirmed`, and 2.4.1's `outcome` string (verify-run's outcome) is now +`receipt_outcome`, since `outcome` is `CONFIRMED`, `REFUTED` or `UNCERTAIN`. +A snapshot doesn't go through the ledger, so move to +`begin_write(contract, params, actor=...)` and `verify_write(pending)`. + +## What this doesn't do + +- It doesn't make a write idempotent in the target system. It stops this + process, a restart, or another worker that shares the ledger from resending + while the outcome is unknown. A writer that bypasses the verifier isn't + covered. +- A store address respelled by hand between a write and its retry (a host + alias such as `127.1`, a hostname for an IP, a DNS alias) isn't + recognized as the same store, and the write could be sent again. Keep + store addresses stable while writes are open, and give each store its + own `environment_id`. Tenants told apart only by URL path also need + distinct `environment_id`s. Without them, one tenant's unresolved write + holds the same write for the other. +- Workers on several hosts need a shared backend. `FileLedger` locks work on + one machine's local disk, not on network file systems. +- If a process crashes after `verify_write` records a delivered outcome but + before the reply reaches the agent, the reply is lost, and an identical + call without a `write_key` counts as a new write. A cancellation the + caller sees isn't affected (`delivered=False`). `write_key` closes the + remaining gap for callers that have a stable task id. diff --git a/packages/flow/openadapt_flow/verifier.py b/packages/flow/openadapt_flow/verifier.py index 0416a33ab..652cda825 100644 --- a/packages/flow/openadapt_flow/verifier.py +++ b/packages/flow/openadapt_flow/verifier.py @@ -1,16 +1,16 @@ -"""Check one write against its contract: the stable API. +"""Check one write against the system of record: the one public verifier API. - from openadapt_flow.verifier import Contract, capture, verify_write + from openadapt_flow.verifier import Actor, Contract, begin_write, verify_write contract = Contract.load("verify.yaml") - before = capture(contract, params) # read the system of record first + pending = begin_write(contract, params, actor=Actor("agent", "intake-bot")) ... # the agent, skill or program writes - result = verify_write(contract, before, params=params, actor="openadapt_actor") - if not result.verified: - print(result.checked, result.reasons) + result = verify_write(pending) + result.outcome # CONFIRMED, REFUTED or UNCERTAIN + result.scope # what the result establishes -MCP skills, the RL reward and the registry all call this module; it wraps -``verify_run`` and checks exactly what verify-run checks: +Every adapter, the MCP verify tools and ``openadapt verify`` call this +module. It wraps ``verify_run`` and checks exactly what verify-run checks: - **delta:** the declared records were written with the declared values; - **frame:** nothing else changed, except what the audit trail attributes to @@ -18,38 +18,31 @@ - **source** (when a :class:`Source` is given): the saved values match the document they came from. -Fail-closed: an unreadable read or a missing or stale baseline gives -``verified=False`` with that part ``not_checked``. A failed check never -raises. :class:`VerifierError` is raised for a contract that can't be loaded -or bound (an unbound parameter), a snapshot taken for another contract, and a -snapshot that was already used. Any call uses up its snapshot: a refused one, -and one whose write is ``unconfirmed`` (reconcile, then capture again; don't -re-verify). Whoever can write a snapshot controls the verdict (its records, -and its salt, which is what single use is keyed on), so the writer being -judged must never have access to it, nor to the state directory where -captures are recorded (``OPENADAPT_VERIFIER_STATE_DIR``, default -``~/.cache/openadapt/verifier``): a snapshot verifies only under the state -directory it was captured under, so not across machines. Capture once per -write: two snapshots taken before one write each verify it, and a capture -never verified leaves its record behind. No model calls. - -The public names are ``Contract``, ``Snapshot``, ``Source``, ``capture``, -``verify_write``, ``WriteResult`` and ``VerifierError`` -(``tests/test_verifier_api.py`` pins their signatures). +:func:`begin_write` consults the uncertain-write ledger +(:mod:`openadapt_flow.write_ledger`) before anything is read or sent, so a +restart or a second worker can't resend a write whose outcome is unknown; it +raises a :class:`WriteBlocked` when the write must not be sent. +:func:`verify_write` records each outcome there. A failed check never raises. +No model calls. See ``docs/VERIFIER_API.md``. """ from __future__ import annotations +import enum import hashlib import json import os +import re import secrets import tempfile import threading +import time +import warnings from collections.abc import Callable, Mapping, Sequence +from contextlib import contextmanager from dataclasses import dataclass, field from pathlib import Path -from typing import Any, Optional +from typing import Any, Iterator, Optional, Union from openadapt_flow.runtime.effects.db_snapshot import Snapshot as DatabaseSnapshot from openadapt_flow.runtime.effects.db_snapshot import TableDiff, diff, take_snapshot @@ -59,28 +52,56 @@ VerifyRunError, _write_private, capture_baseline, - load_verify_config, parse_verify_config, receipt_sha256, verify_run, + write_result, +) +from openadapt_flow.write_ledger import ( + Attempt, + FileLedger, + Ledger, + LedgerBackend, + LedgerError, + MemoryLedger, + default_ledger, + prune, ) __all__ = [ "Actor", + "AlreadyWritten", "BaselineUnreadable", "Contract", "DatabaseSnapshot", + "FileLedger", + "Ledger", + "LedgerBackend", + "LedgerError", + "MemoryLedger", + "Outcome", + "PendingWrite", + "Scope", "Snapshot", "Source", "TableDiff", "VerifierError", + "Window", + "WriteBlocked", + "WriteHeld", "WriteResult", + "begin_write", "capture", "capture_database", "diff_database", + "reconcile", + "verified_write", "verify_write", ] +DEFAULT_WORKDIR = "~/.openadapt/verify" +_OPAQUE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{7,127}$") + #: The receipt's executor kinds, with the names callers also use. _KINDS = { "program": "flow_replay", @@ -91,14 +112,74 @@ "human": "human", } +#: Each verifier substrate's read path. +READ_PATHS = { + "sql": "database", + "database": "database", + "rest": "api", + "graphql": "api", + "fhir": "fhir", + "onscreen": "screen", + "ui_readback": "screen", + "fresh_session": "screen", + "desktop_readback": "screen", + "file": "file", + "fs": "file", + "document": "file", + "email": "email", +} + + +#: Who declared the values: one name for all (``"agent"``), or each value's +#: name mapped to who set it (``{"clinic": "operator", "note": "agent"}``). +#: Names only, never values. +DeclaredBy = Union[str, Mapping[str, str]] + + +class Outcome(str, enum.Enum): + """Does the system of record show the declared write, and only it?""" + + #: Every declared effect was read back from the system of record, and + #: nothing else in the read set moved that the frame doesn't excuse. + CONFIRMED = "CONFIRMED" + #: The store shows something other than the declared write: a wrong + #: value, another record, a duplicate, a change outside it. Never resend. + REFUTED = "REFUTED" + #: The store doesn't show the write, couldn't be read, or was read too + #: weakly to certify it. It may still land: reconcile before any resend. + UNCERTAIN = "UNCERTAIN" + class VerifierError(ValueError): """The contract can't be loaded or bound, or a call is misused.""" -class BaselineUnreadable(VerifierError): - """The system of record can't be read before the write. Don't write: - nothing could verify it afterwards.""" +class WriteBlocked(VerifierError): + """Don't send the write. ``reason`` says why, in a plain sentence.""" + + def __init__(self, reason: str, *, key: str = "") -> None: + super().__init__(reason) + self.reason = reason + self.key = key + + +class BaselineUnreadable(WriteBlocked): + """The system of record can't be read before the write, so nothing + could verify it afterwards.""" + + +class WriteHeld(WriteBlocked): + """An earlier attempt of this write is in flight in another worker, + ``UNCERTAIN``, or ``REFUTED`` and not yet released by a person.""" + + +class AlreadyWritten(WriteBlocked): + """This write was already made and checked; its reply never reached the + caller. ``result`` is that earlier reply. Nothing is sent again.""" + + def __init__(self, reason: str, *, key: str, result: Mapping[str, Any]) -> None: + super().__init__(reason, key=key) + self.result = dict(result) @dataclass(frozen=True) @@ -123,25 +204,1228 @@ def run_record(self) -> dict[str, Any]: return record -@dataclass(frozen=True) class Contract: - """A parsed ``verify.yaml`` (``openadapt.verify/v1``).""" + """A parsed ``verify.yaml`` (``openadapt.verify/v1``). - config: VerifyConfig + ``writes`` (optional, ``"METHOD /path"``) are the requests a taught + program under this contract may send; any other write is stopped before + it is sent. + """ + + def __init__( + self, config: VerifyConfig, *, name: str = "", writes: Sequence[str] = () + ) -> None: + self.config = config + self.name = name + self.writes = tuple(_parse_write(w) for w in writes) @classmethod - def load(cls, path: str | Path) -> "Contract": + def load(cls, path: Union[str, Path]) -> "Contract": + import yaml + + path = Path(path) try: - return cls(load_verify_config(path)) - except VerifyRunError as exc: - raise VerifierError(str(exc)) from exc + data = yaml.safe_load(path.read_text(encoding="utf-8")) or {} + except (OSError, yaml.YAMLError) as exc: + raise VerifierError(f"{path.name}: {exc}") from exc + if not isinstance(data, Mapping): + raise VerifierError(f"{path.name}: a contract is a YAML mapping") + contract = cls.from_dict(data, name=path.stem) + contract.config._base_dir = path.resolve().parent + return contract @classmethod - def from_dict(cls, doc: Mapping[str, Any]) -> "Contract": + def from_dict(cls, data: Mapping[str, Any], *, name: str = "") -> "Contract": + data = dict(data) + writes = data.pop("writes", None) or () + if isinstance(writes, str) or not isinstance(writes, (list, tuple)): + raise VerifierError('writes is a list of "METHOD /path" strings') try: - return cls(parse_verify_config(dict(doc))) + config = parse_verify_config(data, source=name or "contract") except VerifyRunError as exc: raise VerifierError(str(exc)) from exc + return cls(config, name=name, writes=[str(w) for w in writes]) + + @property + def digest(self) -> str: + """SHA-256 of the contract as parsed: part of a write's identity.""" + + body = json.dumps( + { + "config": self.config.model_dump(mode="json"), + "writes": list(self.writes), + }, + sort_keys=True, + separators=(",", ":"), + default=str, + ) + return hashlib.sha256(body.encode("utf-8")).hexdigest() + + @property + def store(self) -> str: + """SHA-256 of which system of record the contract writes to: the + verifier's kind and location and the ``environment_id``. Timings and + credentials are left out, and a loopback host is one host.""" + + return _digest( + { + "environment": self.config.environment_id, + "verifier": _strip( + self.config.verifier.model_dump(mode="json"), drop_secrets=True + ), + } + ) + + @property + def shape(self) -> str: + """SHA-256 of the effects, the verifier kind, the ``environment_id`` + and the hosts (or database files) the verifier reads: the same write + to the same named environment and host, however the address is + split or spelled. Used only to hold, never to hand back.""" + + hosts: set[str] = set() + + def walk(value: Any, name: str = "") -> None: + if isinstance(value, Mapping): + for k, v in value.items(): + walk(v, str(k)) + elif isinstance(value, (list, tuple)): + for v in value: + walk(v, name) + elif isinstance(value, str) and _URL.match(value): + hosts.add(_location(value).split("://", 1)[1].split("/", 1)[0]) + elif name.endswith(("database", "_env")) or name in _HOST_KEYS: + # A database file or server, or the credential that picks + # a tenant: each tells one store from another. + hosts.add(f"{name}={_location(str(value)).lower()}") + + verifier = self.config.verifier.model_dump(mode="json") + walk(verifier) + return _digest( + { + "effects": [ + _strip(e.model_dump(mode="json")) for e in self.config.effects + ], + "kind": verifier.get("kind"), + "environment": self.config.environment_id, + "hosts": sorted(hosts), + } + ) + + @property + def identity(self) -> str: + """SHA-256 of what the write is and where: each effect's kind, target + and values, unbound, and :attr:`store`. Timings, the window and the + frame are left out, so editing them between a crash and a restart + can't make the same write look new.""" + + return _digest( + { + "effects": [ + _strip(e.model_dump(mode="json")) for e in self.config.effects + ], + "store": self.store, + } + ) + + @property + def params(self) -> list[str]: + """Every ``{"param": name}`` the contract's reads and effects bind.""" + + found: set[str] = set() + + def walk(value: Any) -> None: + if isinstance(value, Mapping): + name = value.get("param") + if isinstance(name, str) and name: + found.add(name) + for item in value.values(): + walk(item) + elif isinstance(value, (list, tuple)): + for item in value: + walk(item) + + walk(self.config.model_dump(mode="json")) + return sorted(found) + + +#: Settings that change how long a read waits, never what is written. +_TIMINGS = frozenset( + {"timeout_s", "poll_interval_s", "approval_wait_s", "max_execution_window_s"} +) +#: Keys whose values are secrets themselves. ``*_env`` keys name an +#: environment variable, which tells tenants apart and holds no secret. +_SECRET = re.compile(r"password|secret|token|cookie|header|credential|value$", re.I) +_HOST_KEYS = frozenset({"host", "hostname", "port", "dbname", "db", "unix_socket"}) +_URL = re.compile(r"^[A-Za-z][A-Za-z0-9+.-]*://") +_LOOPBACK = {"localhost", "127.0.0.1", "::1"} +_DEFAULT_PORTS = {"http": 80, "https": 443, "mysql": 3306, "postgresql": 5432} + + +def _location(value: str) -> str: + """One spelling per address: scheme and host lower case, a loopback host + as ``loopback``, no default port, no trailing slash.""" + + if not _URL.match(value): + return value.rstrip("/") if value.endswith("/") and len(value) > 1 else value + from urllib.parse import urlsplit, urlunsplit + + try: + parts = urlsplit(value) + port = parts.port + except ValueError: + return value + scheme = parts.scheme.lower() + host = (parts.hostname or "").lower() + if host in _LOOPBACK: + host = "loopback" + if port is not None and port != _DEFAULT_PORTS.get(scheme): + host = f"{host}:{port}" + return urlunsplit((scheme, host, parts.path.rstrip("/"), parts.query, "")) + + +def _strip(value: Any, *, drop_secrets: bool = False) -> Any: + if isinstance(value, Mapping): + out: dict[str, Any] = {} + for k, v in value.items(): + name = str(k) + if name in _TIMINGS: + continue + secret = ( + drop_secrets + and not name.endswith("_env") + and not isinstance(v, (Mapping, list, tuple)) + and _SECRET.search(name) + ) + if not secret: + out[name] = _strip(v, drop_secrets=drop_secrets) + return out + if isinstance(value, (list, tuple)): + return [_strip(v, drop_secrets=drop_secrets) for v in value] + if isinstance(value, str): + return _location(value) + return value + + +def _digest(value: Any) -> str: + body = json.dumps(value, sort_keys=True, separators=(",", ":"), default=str) + return hashlib.sha256(body.encode("utf-8")).hexdigest() + + +def _parse_write(text: str) -> str: + method, _, path = text.strip().partition(" ") + method = method.upper() + path = path.strip() + if method not in ("POST", "PUT", "PATCH", "DELETE") or not path.startswith("/"): + raise VerifierError( + f'writes entry {text!r}: use "METHOD /path" (POST, PUT, PATCH or DELETE)' + ) + return f"{method} {path}" + + +@dataclass(frozen=True) +class Source: + """Where the written values came from: provenance and the documents.""" + + provenance: Any + documents: Mapping[str, Any] + + +@dataclass(frozen=True) +class Window: + """When the store was read: before the write, and after it.""" + + baseline_read_at: Optional[str] + read_at: Optional[str] + max_execution_window_s: Optional[int] + + +@dataclass(frozen=True) +class Scope: + """What a result establishes, and what it says nothing about.""" + + #: ``database``, ``api``, ``fhir``, ``audit_log``, ``screen``, ``file``. + read_paths: tuple[str, ...] + #: Each store that was read: the verifier's substrates, ``frame.database`` + #: and ``audit``. + stores: tuple[str, ...] + #: Stores the contract names that couldn't be read this time. + unread: tuple[str, ...] + window: Window + #: ``independent_system_of_record``, ``independent_session`` or + #: ``unchecked``. + strength: str + #: Plain sentences: what the result does not cover. + not_covered: tuple[str, ...] + + def to_dict(self) -> dict[str, Any]: + return { + "read_paths": list(self.read_paths), + "stores": list(self.stores), + "unread": list(self.unread), + "window": { + "baseline_read_at": self.window.baseline_read_at, + "read_at": self.window.read_at, + "max_execution_window_s": self.window.max_execution_window_s, + }, + "strength": self.strength, + "not_covered": list(self.not_covered), + } + + +#: The old names, for ``to_dict()``'s ``verdict`` and ``openadapt_flow.verify``. +_VERDICTS = { + Outcome.CONFIRMED: "VERIFIED", + Outcome.REFUTED: "REFUTED", + Outcome.UNCERTAIN: "RECONCILIATION_REQUIRED", +} +_STATUSES = { + Outcome.CONFIRMED: "verified", + Outcome.REFUTED: "refuted", + Outcome.UNCERTAIN: "unconfirmed", +} + + +@dataclass(frozen=True) +class WriteResult: + #: ``CONFIRMED``, ``REFUTED`` or ``UNCERTAIN``. + outcome: Outcome + #: What was read, when, and what this result doesn't cover. + scope: Scope + #: ``{"delta", "frame", "source"}``: ``held``, ``refuted`` or + #: ``not_checked`` (``source`` is ``instruction`` when none was given). + checked: dict[str, Any] = field(default_factory=dict) + #: Plain sentences, without record values. + reasons: list[str] = field(default_factory=list) + #: True only when the caller said the write was never sent and the store + #: shows nothing landed. + retry_safe: bool = False + #: The verify-run receipt's ``outcome`` (``verified``, + #: ``reconciliation_required``...), or "" when no receipt was issued. + receipt_outcome: str = "" + #: ``present``, ``absent``, ``conflicting`` or ``unknown``. + observed: str = "unknown" + receipt: dict[str, Any] = field(default_factory=dict, repr=False) + receipt_sha256: str = "" + receipt_path: Optional[str] = None + evidence: dict[str, Any] = field(default_factory=dict, repr=False) + write_id: str = "" + #: The write's ledger key (an HMAC, no values), or "" without a ledger. + write_key: str = "" + declared_by: DeclaredBy = "agent" + authorization: str = "unbound" + seal: str = "off" + seal_path: Optional[str] = None + #: Not confirmed, but the store holds the declared records: the write is + #: never sent again, and it isn't held for a person. + landed: bool = False + #: A person should look at the store: shown by ``openadapt verify ledger + #: list`` until released, though it doesn't stop a retry being handed back. + review: bool = False + + @property + def verified(self) -> bool: + return self.outcome is Outcome.CONFIRMED + + @property + def verdict(self) -> str: + """The outcome under its old name (``VERIFIED``, ``REFUTED``, + ``RECONCILIATION_REQUIRED``).""" + + return _VERDICTS[self.outcome] + + @property + def status(self) -> str: + """The outcome under #1588's names: ``verified``, ``refuted`` or + ``unconfirmed``.""" + + return _STATUSES[self.outcome] + + @property + def reason(self) -> str: + return "; ".join(self.reasons) + + def to_dict(self) -> dict[str, Any]: + """The reply an agent or the CLI reads: closed fields only.""" + + return { + "verified": self.verified, + "outcome": self.outcome.value, + "verdict": self.verdict, + "retry_safe": self.retry_safe, + "reason": self.reason, + "checked": {k: v for k, v in self.checked.items() if k != "coverage"}, + "observed": self.observed, + "scope": self.scope.to_dict(), + "write_id": self.write_id, + "write_key": self.write_key, + "receipt_outcome": self.receipt_outcome, + "receipt_sha256": self.receipt_sha256 or None, + "receipt_path": self.receipt_path, + "values_declared_by": self.declared_by + if isinstance(self.declared_by, str) + else dict(self.declared_by), + "authorization": self.authorization, + "seal": self.seal, + "seal_path": self.seal_path, + "landed": self.landed, + "review": self.review, + } + + +_NO_LEDGER: Any = object() # sentinel: the caller passed no ``ledger`` + + +@dataclass +class PendingWrite: + """A write that may now be sent: the store was read before it, and the + ledger holds its lease. Single use: :func:`verify_write` consumes it.""" + + contract: Contract = field(repr=False) + params: Mapping[str, Any] = field(repr=False) + actor: Actor + write_id: str + workdir: Path + authorization_ref: str = field(repr=False) + #: An operator's authorization reference was given (not defaulted). + authorized: bool = False + declared_by: DeclaredBy = "agent" + write_key: str = "" + values_key: str = field(default="", repr=False) + #: When the baseline read began (epoch seconds): the window's start. + started: float = field(default=0.0, repr=False) + ledger: Optional[Ledger] = field(default=None, repr=False) + write_key_bound: bool = False + used: bool = False + _lease: Any = field(default=None, repr=False) + + @property + def baseline_path(self) -> Path: + return self.workdir / "baseline.json" + + def release(self) -> None: + """Give up the ledger lease without verifying (the process keeps + running but abandons the write). The attempt stays open, so the next + caller reconciles it before any resend.""" + + lease, self._lease = self._lease, None + if lease is not None: + lease.release() + + +def _resolve_ledger(ledger: Any) -> Optional[Ledger]: + if ledger is _NO_LEDGER: + return default_ledger() + if ledger is None or isinstance(ledger, Ledger): + return ledger + return Ledger(ledger) # a LedgerBackend + + +def begin_write( + contract: Contract, + params: Mapping[str, Any], + *, + actor: Actor, + ledger: Any = _NO_LEDGER, + write_key: Optional[str] = None, + authorization_ref: str = "", + workdir: Union[str, Path, None] = None, + declared_by: DeclaredBy = "agent", +) -> PendingWrite: + """Consult the ledger, then read the system of record before the write. + + Args: + contract: What the write must do (and, with a frame, must not do). + params: The values the write will save, and any that name the record. + actor: Who writes. Its own changes are never excused as another + user's. + ledger: The uncertain-write ledger: a :class:`Ledger`, a + :class:`LedgerBackend`, or None to turn it off (only for a + sandbox that resets between runs). Default: :class:`FileLedger` + in ``OPENADAPT_LEDGER_DIR`` or ``~/.openadapt/ledger``. + write_key: The caller's id for this one write (a task id). With it, + a confirmed write is never made again under the same key; + without it, two writes with the same values are the same write + only while the first one's outcome is open. + authorization_ref: An opaque reference to what authorized the write; + only its salted digest enters the receipt. + workdir: Where this write's private folder goes (default: the + ledger's, or ``~/.openadapt/verify`` without one). + declared_by: Who declared the values, for the reply: a name, or each + value's name mapped to who set it (operator-pinned or agent). + To bind it into the receipt too, put it in ``actor.build``. + + Raises :class:`WriteBlocked` (:class:`BaselineUnreadable`, + :class:`WriteHeld` or :class:`AlreadyWritten`) when the write must not + be sent, and :class:`VerifierError` for a value the contract needs that + ``params`` lacks. + """ + + actor.run_record() # an unknown kind fails before anything is read + if authorization_ref and not _OPAQUE.match(authorization_ref): + raise VerifierError( + "authorization_ref must be an opaque id (8-128 of A-Za-z0-9._:-)" + ) + names = contract.params + missing = [n for n in names if n not in params] + if missing: + raise VerifierError(f"values missing for: {', '.join(missing)}") + values = {n: params[n] for n in names} + try: + led = _resolve_ledger(ledger) + except LedgerError as exc: + raise WriteHeld(f"the write ledger can't be used ({exc})") from exc + write_id = f"write-{secrets.token_hex(12)}" + if led is None: + folder = _new_folder(Path(workdir or DEFAULT_WORKDIR), write_id) + return _capture( + contract, params, actor, write_id, folder, authorization_ref, declared_by + ) + try: + key = led.key_for(contract.identity, values, write_key) + values_key = led.key_for(contract.identity, values, None) + shape_key = led.key_for(contract.shape, values, write_key) + lease = led.backend.lease(key) + except LedgerError as exc: + raise WriteHeld(f"the write ledger can't be used ({exc})") from exc + if lease is None: + raise WriteHeld( + "this write is in flight in another worker; wait for its result", + key=key, + ) + try: + _settle(led, key, contract, params, actor) + _hold_elsewhere(led, key, shape_key) + folder = ( + _new_folder(Path(workdir), write_id) + if workdir is not None + else led.backend.attempt_dir(write_id) + ) + started = time.time() # before the baseline: this attempt's window opens + pending = _capture( + contract, params, actor, write_id, folder, authorization_ref, declared_by + ) + pending.started = started + _write_private( + folder / "attempt.json", + {"values": values, "authorization_ref": pending.authorization_ref}, + ) + led.begun( + key, + write_id, + folder=str(folder), + contract=contract.identity, + values_key=values_key, + store=contract.store, + shape_key=shape_key, + t0=started, + records=sum(int(e.expected_count or 1) for e in contract.config.effects), + actor_kind=actor.kind, + actor_id=actor.id, + write_key_bound=write_key is not None, + authorized=pending.authorized, + declared_by=declared_by + if isinstance(declared_by, str) + else {str(k): str(v) for k, v in declared_by.items()}, + ) + except LedgerError as exc: + lease.release() + raise WriteHeld(f"the write ledger can't be used ({exc})", key=key) from exc + except BaseException: + lease.release() + raise + pending.ledger = led + pending.write_key = key + pending.values_key = values_key + pending.write_key_bound = write_key is not None + pending._lease = lease + return pending + + +def _new_folder(root: Path, write_id: str) -> Path: + folder = root.expanduser() / write_id + folder.mkdir(parents=True, exist_ok=False) + folder.chmod(0o700) + return folder + + +def _capture( + contract: Contract, + params: Mapping[str, Any], + actor: Actor, + write_id: str, + folder: Path, + authorization_ref: str, + declared_by: DeclaredBy, +) -> PendingWrite: + pending = PendingWrite( + contract=contract, + params=dict(params), + actor=actor, + write_id=write_id, + workdir=folder, + authorization_ref=authorization_ref or write_id, + authorized=bool(authorization_ref), + declared_by=declared_by, + ) + try: + payload = capture_baseline( + contract.config, pending.params, pending.baseline_path + ) + except Exception as exc: # a store that can't be reached, a bad binding + raise BaselineUnreadable( + "the system of record could not be read before the write " + f"({type(exc).__name__})" + ) from None + state = payload.get("state") or {} + if not state.get("reachable"): + raise BaselineUnreadable( + "the system of record could not be read before the write" + ) + database = contract.config.frame.database if contract.config.frame else None + if database is not None and FRAME_DB_DETAIL not in (state.get("detail") or {}): + raise BaselineUnreadable("the database could not be read for the frame") + return pending + + +def _settle( + led: Ledger, + key: str, + contract: Contract, + params: Mapping[str, Any], + actor: Actor, +) -> None: + """With the lease held: resolve the key's earlier attempt, or raise.""" + + attempt = led.latest(key) + if attempt is not None and attempt.state in ("open", "uncertain"): + # Open with a free lease: its owner died between the send and the + # read-back. Read back before anything is sent. + _reconcile(led, key, attempt, contract, params) + attempt = led.latest(key) + state = led.state(attempt) + if state == "free": + return + if state == "undelivered": + assert attempt is not None and attempt.resolved is not None + result = dict(attempt.resolved.get("result") or {}) + result["not_sent_again"] = True + if not attempt.delivered: + led.delivered(key, attempt.attempt_id) + raise AlreadyWritten( + ( + "this write was already made: the store holds it, though it " + "wasn't confirmed; its result is attached, and it was not sent " + "again" + if result.get("landed") + else "this write was already made and checked; its result is " + "attached, and it was not sent again" + ), + key=key, + result=result, + ) + if state == "held": + raise WriteHeld( + "an earlier attempt of this write was refuted; a person must check " + "the record and release it (openadapt verify ledger release)", + key=key, + ) + raise WriteHeld( + "an earlier attempt of this write may still land; the store doesn't " + "show it yet, so it is held until a read-back finds it or a person " + "releases it", + key=key, + ) + + +def _hold_elsewhere(led: Ledger, key: str, shape_key: str) -> None: + """Hold when the same write, under another store address (an edited + contract, an alias), is open or unresolved: a hold beats a resend.""" + + for other, begun, _resolved in led.attempts(): + if other == key or begun.get("shape_key") != shape_key: + continue + attempt = led.latest(other) + if attempt is None or attempt.attempt_id != begun.get("attempt"): + continue + if led.state(attempt) != "free": + raise WriteHeld( + "the same write, under another address for the store, is in " + "flight, unresolved or already made; a person checks which " + "store it went to (openadapt verify ledger list)", + key=key, + ) + + +def _reconcile( + led: Ledger, + key: str, + attempt: Attempt, + contract: Contract, + params: Optional[Mapping[str, Any]] = None, +) -> None: + """Resolve ``attempt`` by reading back. The caller holds the lease. + + The attempt is judged against its own baseline, so a record that was + there before it is never taken for its write. A baseline a check already + used is copied for this read: it is the same write, and the lease keeps + any other attempt of it from being sent meanwhile. Past the contract's + window the baseline proves nothing, and the attempt stays uncertain. + """ + + folder = attempt.folder + baseline = folder / "baseline.json" + try: + saved = json.loads((folder / "attempt.json").read_text(encoding="utf-8")) + payload = json.loads(baseline.read_text(encoding="utf-8")) + except (OSError, ValueError): + return # nothing to judge with: stays as it is, for a person + workdir = folder + if baseline.with_name(baseline.name + ".used").exists(): + workdir = folder / f"reconcile-{secrets.token_hex(6)}" + workdir.mkdir(mode=0o700) + _write_private(workdir / "baseline.json", payload) + previous = (attempt.resolved or {}).get("receipt_sha256") + pending = PendingWrite( + contract=contract, + params={**(saved.get("values") or {}), **dict(params or {})}, + actor=Actor( + str(attempt.begun.get("actor_kind") or "agent"), + str(attempt.begun.get("actor_id") or "unidentified"), + ), + write_id=attempt.attempt_id, + workdir=workdir, + authorization_ref=str(saved.get("authorization_ref") or attempt.attempt_id), + authorized=bool(attempt.begun.get("authorized")), + declared_by=attempt.begun.get("declared_by") or "agent", + write_key=key, + ) + result = _judge( + pending, + write_attempted=True, + source=None, + seal=True, + remediates=f"sha256:{previous}" if previous else None, + ) + result = _check_overlap( + led, + key, + attempt.attempt_id, + str(attempt.begun.get("values_key") or ""), + str(attempt.begun.get("store") or ""), + float(attempt.begun.get("t0") or 0.0), + result, + ) + if ( + result.outcome is not Outcome.UNCERTAIN + or attempt.resolved is None + or _landed(result) + ): + _record( + led, + key, + attempt.attempt_id, + result, + delivered=False, + via="reconcile", + write_key_bound=bool(attempt.begun.get("write_key_bound")), + folder=folder, + ) + if ( + result.outcome is Outcome.UNCERTAIN + and result.observed == "absent" + and led.settle_s is not None + and _age_s(attempt) >= led.settle_s + ): + led.resolved( + key, + attempt.attempt_id, + outcome=Outcome.UNCERTAIN.value, + retry_safe=True, + delivered=True, + via="settled", + reason=f"the store showed nothing {int(led.settle_s)}s after the attempt", + ) + + +def _age_s(attempt: Attempt) -> float: + import calendar + + try: + began = calendar.timegm( + time.strptime(str(attempt.begun.get("at")), "%Y-%m-%dT%H:%M:%SZ") + ) + except ValueError: + return 0.0 + return time.time() - began + + +def verify_write( + pending: Union[PendingWrite, Contract], + before: Optional["Snapshot"] = None, + *, + write_attempted: bool = True, + source: Optional[Source] = None, + seal: bool = True, + delivered: bool = True, + params: Optional[Mapping[str, object]] = None, + actor: Optional[Actor] = None, + authorization_ref: str = "verify_write", +) -> WriteResult: + """Read the system of record now and judge the write against the + baseline :func:`begin_write` took. Never resends the write. + + Args: + pending: From :func:`begin_write`. Single use. + write_attempted: False only when the caller knows the write was + never sent (a stop before the save); then a store that shows + nothing makes the result ``retry_safe``. When it can't know (a + tool raised before or while acting), leave it True: a store + that shows nothing is then ``UNCERTAIN``, never retry_safe. + source: The written values' provenance and documents, to compare the + saved record with its source. + seal: Sign a confirmed receipt with this computer's Seal key. + delivered: False when the reply won't reach the caller (a + cancelled tool call, a timeout); the next identical call then + raises :class:`AlreadyWritten` with this result instead of a + resend. The caller decides this, not the verifier. + + Deprecated form (#1588): ``verify_write(contract, before, params=..., + actor=...)`` with a :func:`capture` snapshot judges the write the same + way, without the ledger; ``authorization_ref`` belongs to it only. + """ + + if isinstance(pending, Contract): + if before is None or params is None or actor is None: + raise VerifierError( + "verify_write(contract, before, params=..., actor=...) needs all three" + ) + return _verify_snapshot( + pending, + before, + params=params, + actor=actor, + source=source, + authorization_ref=authorization_ref, + write_attempted=write_attempted, + ) + if before is not None or params is not None or actor is not None: + raise VerifierError( + "verify_write(pending) takes no snapshot, params or actor: " + "begin_write bound them" + ) + if pending.used: + raise VerifierError("this write was already checked; begin a new one") + pending.used = True + # Without the lease (released), another worker may be reconciling this + # attempt: judge, but leave the ledger to it. + holds_lease = pending.ledger is not None and pending._lease is not None + try: + result = _judge( + pending, write_attempted=write_attempted, source=source, seal=seal + ) + if holds_lease and pending.ledger is not None: + result = _check_overlap( + pending.ledger, + pending.write_key, + pending.write_id, + pending.values_key, + pending.contract.store, + pending.started, + result, + ) + _record( + pending.ledger, + pending.write_key, + pending.write_id, + result, + delivered=delivered, + via="verify_write", + write_key_bound=pending.write_key_bound, + folder=pending.workdir, + ) + finally: + pending.release() + return result + + +def _record( + led: Ledger, + key: str, + attempt_id: str, + result: WriteResult, + *, + delivered: bool, + via: str, + write_key_bound: bool, + folder: Optional[Path] = None, +) -> None: + led.resolved( + key, + attempt_id, + outcome=result.outcome.value, + retry_safe=result.retry_safe, + landed=result.landed, + review=result.review, + delivered=delivered, + via=via, + write_key_bound=write_key_bound, + receipt_sha256=result.receipt_sha256 or None, + result=result.to_dict(), + ) + if folder is not None and _landed(result): + prune(folder) + + +def _landed(result: WriteResult) -> bool: + """Resolved for good: nothing will read this attempt back again.""" + + return result.outcome is not Outcome.UNCERTAIN or result.landed or result.retry_safe + + +def _downgrade( + result: WriteResult, reason: str, *, landed: bool = False, review: bool = False +) -> WriteResult: + import dataclasses + + return dataclasses.replace( + result, + outcome=Outcome.UNCERTAIN, + reasons=[*result.reasons, reason], + retry_safe=False, + landed=landed, + review=review, + ) + + +def _overlapping( + led: Ledger, own_key: str, since: float, until: float, window_s: float +) -> list[tuple[str, dict[str, Any], Optional[dict[str, Any]]]]: + """Attempts of other keys whose write could have landed between + ``since`` (this attempt's baseline read began) and ``until`` (its + read-back ended). The same key's earlier attempts resolved before this + one began: its lease orders them. An attempt still open counts while its + owner is alive, or for ``window_s`` after it began (its owner died, and + a write it sent may still land); a released one is closed.""" + + out: list[tuple[str, dict[str, Any], Optional[dict[str, Any]]]] = [] + for key, begun, resolved in led.attempts(): + began = float(begun.get("t0") or begun.get("t") or 0.0) + if key == own_key or began > until: + continue + if resolved is not None and resolved.get("event") == "released": + continue # released before any outcome: a person found nothing sent + if resolved is not None: + if float(resolved.get("t") or until) >= since: + out.append((key, begun, resolved)) + elif began >= since - window_s or _live(led, key): + out.append((key, begun, resolved)) + return out + + +def _live(led: Ledger, key: str) -> bool: + """Another caller holds ``key``'s lease right now.""" + + try: + lease = led.backend.lease(key) + except LedgerError: + return True + if lease is None: + return True + lease.release() + return False + + +def _check_overlap( + led: Ledger, + key: str, + attempt_id: str, + values_key: str, + store: str, + since: float, + result: WriteResult, +) -> WriteResult: + """Judge a result against the other attempts open during its window. + + - Same values: one record can't prove two writes, so a ``CONFIRMED`` is + ``UNCERTAIN`` for a person (it may be the other attempt's record, or + this attempt's send may have stored nothing). + - A frame breach of only added records, no more than other attempts on + the same store wrote in the window (read back as landed, or live + now): the declared write landed and the extras are most likely + theirs. ``UNCERTAIN`` with ``landed`` (never resent, not held) and + ``review`` (a person looks, since a count can't name the records). + """ + + window_s = float(result.scope.window.max_execution_window_s or 3600) + others = _overlapping(led, key, since, time.time(), window_s) + if result.outcome is Outcome.CONFIRMED: + if values_key and any( + b.get("values_key") == values_key for _k, b, _r in others + ): + return _downgrade( + result, + "another attempt with the same values was open during this " + "write, so the record may be that write; a person checks " + "which one landed", + ) + return result + if result.outcome is not Outcome.REFUTED or result.checked.get("delta") != "held": + return result + if result.checked.get("source") == "refuted": + return result + frame = result.evidence.get("frame") or {} + kinds = set(frame.get("collateral_kinds") or []) + if frame.get("status") != "refuted" or not kinds or kinds - {"added"}: + return result + written = 0 + for key, begun, resolved in others: + if not store or begun.get("store") != store: + continue # another system of record can't explain this one's rows + if resolved is not None: + if resolved.get("outcome") == "CONFIRMED" or resolved.get("landed"): + written += int(begun.get("records") or 1) + elif _live(led, key): + written += int(begun.get("records") or 1) + if not written or int(frame.get("collateral_count") or 0) > written: + return result + return _downgrade( + result, + "the declared records landed; the records added beside them could be " + "other writes made on the same store in the same window, so this " + "write isn't confirmed or sent again, and a person checks the extras", + landed=True, + review=True, + ) + + +def _judge( + pending: PendingWrite, + *, + write_attempted: bool, + source: Optional[Source], + seal: bool, + remediates: Optional[str] = None, +) -> WriteResult: + run_record = pending.actor.run_record() + base: dict[str, Any] = { + "write_id": pending.write_id, + "write_key": pending.write_key, + "declared_by": pending.declared_by, + "authorization": "operator" if pending.authorized else "unbound", + } + try: + doc = verify_run( + pending.contract.config, + params=pending.params, + run_record=run_record, + authorization_ref=pending.authorization_ref, + baseline_path=pending.baseline_path, + private_dir=pending.workdir, + correlation_id=pending.write_id, + remediates=remediates, + write_attempted=write_attempted, + provenance=None if source is None else source.provenance, + documents=None if source is None else source.documents, + ) + path: Optional[str] = str(write_result(doc, pending.workdir)) + except Exception as exc: # noqa: BLE001 - unreadable store, timeout, used baseline + return WriteResult( + outcome=Outcome.UNCERTAIN, + scope=_scope(pending.contract, {}), + reasons=[ + "the system of record could not be read back " + f"({type(exc).__name__}); the write may still land" + ], + **base, + ) + receipt = dict(doc["receipt"]) + evidence = dict(doc["evidence"]) + receipt_outcome = str(receipt.get("outcome") or "") + checked = dict(receipt.get("checked") or {}) + outcome, retry_safe = classify(receipt_outcome, checked, evidence) + seal_path, seal_state = None, "off" + if outcome is Outcome.CONFIRMED and seal and path: + seal_path, seal_state = _seal(path, pending.write_id) + return WriteResult( + outcome=outcome, + scope=_scope(pending.contract, evidence, receipt), + checked=checked, + reasons=_reasons(evidence, receipt_outcome), + retry_safe=retry_safe, + receipt_outcome=receipt_outcome, + observed=_observed(evidence), + receipt=receipt, + receipt_sha256=receipt_sha256(receipt), + receipt_path=path, + evidence=evidence, + seal=seal_state, + seal_path=seal_path, + **base, + ) + + +def classify( + receipt_outcome: str, checked: Mapping[str, Any], evidence: Mapping[str, Any] +) -> tuple[Outcome, bool]: + """A verify-run receipt's outcome as (:class:`Outcome`, retry_safe).""" + + if receipt_outcome == "verified": + return Outcome.CONFIRMED, False + if receipt_outcome == "halted_before_effect": + # The caller never sent it, and the store shows nothing landed. + return Outcome.REFUTED, True + if receipt_outcome == "rejected_policy": + return Outcome.REFUTED, False # landed, but approved by its own executor + if receipt_outcome != "reconciliation_required": + return Outcome.UNCERTAIN, False + if checked.get("frame") == "refuted" or checked.get("source") == "refuted": + return Outcome.REFUTED, False + if checked.get("delta") != "refuted": + return Outcome.UNCERTAIN, False + rows = (evidence.get("readback") or {}).get("effects") or [] + failed = [ + r for r in rows if isinstance(r, Mapping) and r.get("verdict") != "CONFIRMED" + ] + if failed and all(r.get("observed_effect") == "absent" for r in failed): + return Outcome.UNCERTAIN, False + return Outcome.REFUTED, False + + +def _observed(evidence: Mapping[str, Any]) -> str: + rows = (evidence.get("readback") or {}).get("effects") or [] + seen = {str(r.get("observed_effect")) for r in rows if isinstance(r, Mapping)} + if (evidence.get("frame") or {}).get("status") == "refuted": + seen.add("conflicting") + if not seen or "unknown" in seen or "None" in seen: + return "unknown" + if "conflicting" in seen or seen == {"absent", "present"}: + return "conflicting" + return "absent" if "absent" in seen else "present" + + +def _scope( + contract: Contract, + evidence: Mapping[str, Any], + receipt: Optional[Mapping[str, Any]] = None, +) -> Scope: + cfg = contract.config + readback = evidence.get("readback") or {} + rows = [r for r in readback.get("effects") or [] if isinstance(r, Mapping)] + frame = evidence.get("frame") or {} + frame_read = bool(frame) and frame.get("status") != "not_checked" + stores: list[str] = [] + unread: list[str] = [] + for row in rows: + name = str(row.get("substrate") or cfg.verifier.kind) + target = unread if row.get("unavailable") or row.get("stale") else stores + if name not in target: + target.append(name) + if not rows: + unread.append(cfg.verifier.kind) + spec = cfg.frame + if spec is not None and getattr(spec, "database", None) is not None: + (stores if frame_read and "database" in frame else unread).append( + "frame.database" + ) + attribution = getattr(spec, "attribution", None) if spec is not None else None + if attribution is not None and attribution.kind == "audit_read": + (stores if frame_read else unread).append("audit") + paths: list[str] = [] + for name in stores: + path = ( + "database" + if name == "frame.database" + else "audit_log" + if name == "audit" + else READ_PATHS.get(name, name) + ) + if path not in paths: + paths.append(path) + contracts = (receipt or {}).get("contracts") or {} + strength = str(contracts.get("observed_effect_strength") or "unchecked") + not_covered = [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + ] + if spec is None or getattr(spec, "database", None) is None: + not_covered.append("records outside the read set (no frame.database)") + return Scope( + read_paths=tuple(paths), + stores=tuple(stores), + unread=tuple(unread), + window=Window( + baseline_read_at=readback.get("baseline_read_at"), + read_at=readback.get("read_at"), + max_execution_window_s=readback.get("max_execution_window_s"), + ), + strength=strength, + not_covered=tuple(not_covered), + ) + + +def _reasons(evidence: Mapping[str, Any], receipt_outcome: str) -> list[str]: + out: list[str] = [] + for row in (evidence.get("readback") or {}).get("effects") or []: + if not isinstance(row, Mapping) or str(row.get("verdict")) == "CONFIRMED": + continue + out.append( + f"{row.get('kind')} was {str(row.get('verdict')).lower()}: observed " + f"{row.get('observed_effect')}, {row.get('observed_count')} of " + f"{row.get('expected_count')} expected record(s)" + ) + frame = evidence.get("frame") or {} + if frame.get("status") == "refuted": + kinds = ", ".join(frame.get("collateral_kinds") or []) + out.append( + f"{frame.get('collateral_count', 0)} record(s) outside the declared " + f"write moved ({kinds}); " + f"{frame.get('unattributed_count', 0)} of them no actor explains" + ) + elif frame.get("status") == "not_checked": + out.append("what else changed could not be read, so the frame is not checked") + source = evidence.get("source") or {} + if isinstance(source, Mapping) and source.get("status") == "refuted": + fields = ", ".join(source.get("mismatched_fields") or []) + out.append(f"the saved record differs from its source in {fields}") + if receipt_outcome == "rejected_policy": + out.append("the write's approver is its own executor") + if not out and receipt_outcome not in ("verified", ""): + out.append( + "the read was too weak to certify the write " + f"({(evidence.get('readback') or {}).get('verification_tier')})" + ) + return out + + +def _seal(receipt_path: str, write_id: str) -> tuple[Optional[str], str]: + """Sign the receipt file with this computer's Seal key. Returns the Seal's + path and ``sealed``; or no path and ``off`` (Seals turned off) or + ``unsealed: ``. The outcome stands either way: the Seal attests who + issued the receipt, not the write.""" + + from openadapt_flow import offline_seal + + if not offline_seal.mode_enabled(): + return None, "off" + try: + sealed = offline_seal.sign_file( + Path(receipt_path), subject_kind="record", run_id=write_id + ) + out = Path(receipt_path).with_name("receipt.seal.json") + _write_private(out, sealed.model_dump(mode="json")) + except Exception as exc: # noqa: BLE001 - say why, never fail the write + return ( + None, + f"unsealed: this computer's Seal key couldn't be used ({type(exc).__name__})", + ) + return str(out), "sealed" + + +# The deprecated #1588 form: capture() and verify_write(contract, before, ...). +# Kept as #1588 shipped it, so a snapshot verifies once and only under the +# state directory it was captured in; results are the one WriteResult. @dataclass @@ -255,41 +1539,18 @@ def _consume(payload: Any) -> None: raise VerifierError(f"can't record the snapshot's use: {exc}") from exc -@dataclass(frozen=True) -class Source: - """Where the written values came from: provenance and the documents.""" - - provenance: Any - documents: Mapping[str, Any] - - -@dataclass(frozen=True) -class WriteResult: - #: True only when every part checked held. - verified: bool - #: ``verified``; ``refuted`` (the store shows a different, extra or - #: duplicate write: don't retry blindly); or ``unconfirmed`` (the store - #: doesn't show the write, or couldn't be read: it may still land, so - #: reconcile before retrying). - status: str - #: ``verified``, ``reconciliation_required``, ``halted_before_effect``, - #: or another verify-run outcome: never ``verified`` on a failed check. - outcome: str - #: ``{"delta": ..., "frame": ..., "source": ...}``: ``held``, - #: ``refuted`` or ``not_checked`` (``source`` is ``instruction`` when no - #: source was given). - checked: dict[str, str] - #: Plain sentences, without record values. - reasons: list[str] = field(default_factory=list) - #: The effect receipt, its SHA-256, and its evidence, as verify-run - #: issues them. - receipt: dict[str, Any] = field(default_factory=dict) - receipt_sha256: str = "" - evidence: dict[str, Any] = field(default_factory=dict) +def capture(contract: Contract, params: Mapping[str, object]) -> Snapshot: + """Deprecated (#1588): read the system of record before the write. + Use :func:`begin_write`, which also holds the write in the uncertain-write + ledger; a snapshot doesn't, so a restart can't tell it was sent. + """ -def capture(contract: Contract, params: Mapping[str, object]) -> Snapshot: - """Read the system of record before the write.""" + warnings.warn( + "verifier.capture is deprecated; use begin_write", + DeprecationWarning, + stacklevel=2, + ) with tempfile.TemporaryDirectory(prefix="openadapt-verify-") as tmp: try: @@ -309,119 +1570,121 @@ def capture(contract: Contract, params: Mapping[str, object]) -> Snapshot: return snapshot -def _status(checked: Mapping[str, str], evidence: Mapping[str, Any]) -> str: - """``refuted`` when the store shows something other than the declared - write; ``unconfirmed`` when it doesn't show the write (or couldn't be - read), so the write may still land.""" - - if checked.get("frame") == "refuted" or checked.get("source") == "refuted": - return "refuted" - if checked.get("delta") != "refuted": - return "unconfirmed" - rows = (evidence.get("readback") or {}).get("effects") or [] - failed = [ - r for r in rows if isinstance(r, Mapping) and r.get("verdict") != "CONFIRMED" - ] - if failed and all(r.get("observed_effect") == "absent" for r in failed): - return "unconfirmed" - return "refuted" - - -def _reasons(evidence: Mapping[str, Any]) -> list[str]: - out: list[str] = [] - for row in (evidence.get("readback") or {}).get("effects") or []: - if not isinstance(row, Mapping) or str(row.get("verdict")) == "CONFIRMED": - continue - out.append( - f"{row.get('kind')} was {str(row.get('verdict')).lower()}: observed " - f"{row.get('observed_effect')}, {row.get('observed_count')} of " - f"{row.get('expected_count')} expected record(s)" - ) - frame = evidence.get("frame") or {} - if frame.get("status") == "refuted": - kinds = ", ".join(frame.get("collateral_kinds") or []) - out.append( - f"{frame.get('collateral_count', 0)} record(s) outside the declared " - f"write moved ({kinds}); " - f"{frame.get('unattributed_count', 0)} of them no actor explains" - ) - elif frame.get("status") == "not_checked": - out.append("what else changed could not be read, so the frame is not checked") - source = evidence.get("source") or {} - if isinstance(source, Mapping) and source.get("status") == "refuted": - fields = ", ".join(source.get("mismatched_fields") or []) - out.append(f"the saved record differs from its source in {fields}") - return out - - -def verify_write( +def _verify_snapshot( contract: Contract, before: Snapshot, *, params: Mapping[str, object], actor: Actor, - source: Optional[Source] = None, - authorization_ref: str = "verify_write", - write_attempted: bool = True, + source: Optional[Source], + authorization_ref: str, + write_attempted: bool, ) -> WriteResult: - """Read the system of record now and judge the write against ``before``. - - Args: - contract: What the write must do (and, with a frame, must not do). - before: :func:`capture` taken before the write. Single use. - params: The run's inputs, which the contract's values bind. - actor: Who wrote. Its own changes are never excused as another - user's. - source: The written values' provenance and documents, to compare the - saved record with its source. - authorization_ref: An opaque reference to what authorized the write; - only its salted digest enters the receipt. - write_attempted: False when the write was never tried (a stop before - the save), so the receipt says no effect was attempted. - """ - run_record = actor.run_record() if before.used: raise VerifierError("this snapshot was already used; capture again") before.used = True _consume(before.payload) - - def run(path: Path) -> dict[str, Any]: + # Judge the payload whose salt was just used, never the file as it is now. + with tempfile.TemporaryDirectory(prefix="openadapt-verify-") as tmp: try: - return verify_run( + doc = verify_run( contract.config, params=params, run_record=run_record, write_attempted=write_attempted, authorization_ref=authorization_ref, - baseline_path=path, + baseline_path=before.save(Path(tmp) / "baseline.json"), provenance=None if source is None else source.provenance, documents=None if source is None else source.documents, ) except (VerifyRunError, OSError) as exc: raise VerifierError(str(exc)) from exc - - # Judge the payload whose salt was just used, never the file as it is now. - with tempfile.TemporaryDirectory(prefix="openadapt-verify-") as tmp: - result = run(before.save(Path(tmp) / "baseline.json")) - receipt = dict(result["receipt"]) - evidence = dict(result["evidence"]) - outcome = str(receipt.get("terminal_outcome") or receipt.get("outcome") or "") + receipt = dict(doc["receipt"]) + evidence = dict(doc["evidence"]) + receipt_outcome = str(receipt.get("outcome") or "") checked = dict(receipt.get("checked") or {}) - verified = outcome == "verified" - status = "verified" if verified else _status(checked, evidence) + outcome, retry_safe = classify(receipt_outcome, checked, evidence) return WriteResult( - verified=verified, - status=status, outcome=outcome, + scope=_scope(contract, evidence, receipt), checked=checked, - reasons=_reasons(evidence), + reasons=_reasons(evidence, receipt_outcome), + retry_safe=retry_safe, + receipt_outcome=receipt_outcome, + observed=_observed(evidence), receipt=receipt, receipt_sha256=receipt_sha256(receipt), evidence=evidence, ) +@dataclass +class WriteBlock: + """What :func:`verified_write` yields; ``result`` is set after the block.""" + + pending: PendingWrite + write_attempted: bool = True + result: Optional[WriteResult] = None + + +@contextmanager +def verified_write( + contract: Contract, + params: Mapping[str, Any], + *, + actor: Actor, + **options: Any, +) -> Iterator[WriteBlock]: + """``with verified_write(...) as w: send()`` then read ``w.result``. + + ``options`` go to :func:`begin_write`. If the block raises before the + write is sent, set ``w.write_attempted = False`` first; the read-back + runs either way, because a half-sent write may have landed. + """ + + block = WriteBlock(begin_write(contract, params, actor=actor, **options)) + try: + yield block + finally: + block.result = verify_write( + block.pending, write_attempted=block.write_attempted + ) + + +def reconcile( + contract: Contract, + key: str, + *, + ledger: Any = _NO_LEDGER, +) -> Optional[str]: + """Resolve the key's open or uncertain attempt by reading back. + + Returns the attempt's state afterwards (``free``, ``undelivered``, + ``uncertain``, ``held``, ``open``), or None when the key has no + attempt. Raises :class:`WriteHeld` when another worker holds the key. + """ + + led = _resolve_ledger(ledger) + if led is None: + raise VerifierError("reconcile needs a ledger") + lease = led.backend.lease(key) + if lease is None: + raise WriteHeld("this write is in flight in another worker", key=key) + try: + attempt = led.latest(key) + if attempt is None: + return None + if attempt.state in ("open", "uncertain"): + _reconcile(led, key, attempt, contract) + attempt = led.latest(key) + return attempt.state if attempt is not None else None + finally: + lease.release() + + +# -- the database frame, without a contract (from #1588) ---------------------- + #: The key the snapshots of this process share, when the caller gives none: #: two captures taken in one process compare. Saved snapshots need an #: explicit ``key``. diff --git a/packages/flow/openadapt_flow/write_ledger.py b/packages/flow/openadapt_flow/write_ledger.py new file mode 100644 index 000000000..bc11713eb --- /dev/null +++ b/packages/flow/openadapt_flow/write_ledger.py @@ -0,0 +1,597 @@ +"""The uncertain-write ledger: no write is sent twice while its outcome is unknown. + +:func:`openadapt_flow.verifier.begin_write` consults it before every write +and :func:`~openadapt_flow.verifier.verify_write` records each outcome, so a +restart, a retry or a second worker that shares the ledger can't resend a +write that may already have landed. See ``docs/VERIFIER_API.md``. + +Each write identity (a key, see :meth:`Ledger.key_for`) has an append-only +log of events: + +``begun`` + An attempt: its baseline was read and the write may now be sent. +``resolved`` + The attempt's outcome (``CONFIRMED``, ``REFUTED`` or ``UNCERTAIN``), + from a read-back. ``delivered`` says the caller received it. +``delivered`` + A stored outcome was handed to a later caller instead of a resend. +``released`` + A person checked the record and released a held attempt. + +Nothing is rewritten or deleted. Events hold closed fields only: no record +values, and keys are HMACs under a secret kept with the ledger. + +A lease (:meth:`LedgerBackend.lease`) gives one caller a key from +``begin_write`` until ``verify_write`` returns. :class:`FileLedger` leases +are ``flock`` locks, which the operating system drops when the process dies: +a free lease on an open attempt means its owner died between the send and +the read-back. +""" + +from __future__ import annotations + +import fcntl +import hashlib +import hmac +import json +import os +import secrets +import threading +import time +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Mapping, Optional, Protocol, Sequence + +__all__ = [ + "Attempt", + "FileLedger", + "Lease", + "Ledger", + "LedgerBackend", + "LedgerError", + "MemoryLedger", + "default_ledger", +] + +#: ``OPENADAPT_LEDGER_DIR`` overrides it. +DEFAULT_DIR = "~/.openadapt/ledger" +_KEY = 64 # hex chars of an HMAC-SHA256 + + +class LedgerError(RuntimeError): + """The ledger can't be read or written. Nothing may be sent.""" + + +class Lease(Protocol): + def release(self) -> None: ... + + +class LedgerBackend(Protocol): + """Where the ledger lives. Implement it for a shared store (a database) + when workers run on several hosts; :class:`FileLedger` is one machine.""" + + def secret(self) -> bytes: + """The HMAC key for write identities, the same for every worker.""" + + def events(self, key: str) -> list[dict[str, Any]]: + """Every event appended for ``key``, oldest first.""" + + def append(self, key: str, event: Mapping[str, Any]) -> None: + """Append one event durably before returning.""" + + def lease(self, key: str) -> Optional[Lease]: + """An exclusive lease on ``key`` without waiting, or None when + another live caller holds it. It must end when its holder dies.""" + + def keys(self) -> list[str]: + """Every key with at least one event.""" + + def attempt_dir(self, attempt_id: str) -> Path: + """A new private folder for one attempt's baseline and values.""" + + +def _private_dir(path: Path) -> Path: + try: + path.mkdir(mode=0o700, parents=True, exist_ok=True) + path.chmod(0o700) + except OSError as exc: + raise LedgerError(f"can't keep the write ledger: {exc}") from exc + info = os.lstat(path) + if ( + not os.path.isdir(path) + or os.path.islink(path) + or info.st_uid != os.geteuid() + or info.st_mode & 0o077 + ): + # Another user could remove entries (a resend) or plant them. + raise LedgerError(f"{path} must be a private directory owned by this user") + return path + + +def _check_key(key: str) -> str: + if len(key) != _KEY or any(c not in "0123456789abcdef" for c in key): + raise LedgerError("a ledger key is 64 lowercase hex characters") + return key + + +class _FileLease: + def __init__(self, handle: Any) -> None: + self._handle = handle + + def release(self) -> None: + handle, self._handle = self._handle, None + if handle is not None: + fcntl.flock(handle.fileno(), fcntl.LOCK_UN) + handle.close() + + +class FileLedger: + """The ledger on this machine's local disk (the default). + + ``root``/``entries/.jsonl`` holds each key's events, one JSON object + per line, appended and fsynced. ``locks/.lock`` carries the lease, + ``attempts//`` each attempt's baseline and values, and + ``ledger.key`` the HMAC key (all private to this user). ``flock`` locks + don't hold across hosts or on network file systems. + """ + + def __init__(self, root: str | Path) -> None: + self.root = _private_dir(Path(root).expanduser()) + self._entries = _private_dir(self.root / "entries") + self._locks = _private_dir(self.root / "locks") + self._attempts = _private_dir(self.root / "attempts") + + def secret(self) -> bytes: + path = self.root / "ledger.key" + try: + fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + except FileExistsError: + pass + else: + with os.fdopen(fd, "w") as fh: + fh.write(secrets.token_hex(32)) + fh.flush() + os.fsync(fh.fileno()) + # Another worker may be writing it: wait for the whole key. + for _ in range(50): + try: + text = path.read_text().strip() + except OSError as exc: + raise LedgerError(f"can't read the ledger key: {exc}") from exc + if len(text) == 64: + try: + return bytes.fromhex(text) + except ValueError: + break + time.sleep(0.01) + raise LedgerError( + f"{path} is not a ledger key (a crash while it was written?). " + "A person checks that no write is open, then removes it" + ) + + def events(self, key: str) -> list[dict[str, Any]]: + path = self._entries / f"{_check_key(key)}.jsonl" + try: + lines = path.read_text(encoding="utf-8").splitlines() + except FileNotFoundError: + return [] + except OSError as exc: + raise LedgerError(f"can't read the write ledger: {exc}") from exc + out: list[dict[str, Any]] = [] + for line in lines: + try: + event = json.loads(line) + except ValueError: + # A line torn by a crash mid-append: its call never returned, + # so nothing it would have allowed happened. + continue + if isinstance(event, dict): + out.append(event) + return out + + def append(self, key: str, event: Mapping[str, Any]) -> None: + path = self._entries / f"{_check_key(key)}.jsonl" + line = json.dumps(dict(event), sort_keys=True, separators=(",", ":")) + try: + fd = os.open(path, os.O_RDWR | os.O_APPEND | os.O_CREAT, 0o600) + try: + size = os.fstat(fd).st_size + torn = size > 0 and os.pread(fd, 1, size - 1) != b"\n" + os.write(fd, (("\n" if torn else "") + line + "\n").encode("utf-8")) + os.fsync(fd) + finally: + os.close(fd) + except OSError as exc: + raise LedgerError(f"can't write the write ledger: {exc}") from exc + + def lease(self, key: str) -> Optional[Lease]: + path = self._locks / f"{_check_key(key)}.lock" + try: + handle = open(path, "a") # noqa: SIM115 - the lease owns it + except OSError as exc: + raise LedgerError(f"can't lease a write ledger key: {exc}") from exc + try: + os.chmod(path, 0o600) + fcntl.flock(handle.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB) + except BlockingIOError: + handle.close() + return None + except OSError as exc: + handle.close() + raise LedgerError(f"can't lease a write ledger key: {exc}") from exc + return _FileLease(handle) + + def keys(self) -> list[str]: + return sorted(p.stem for p in self._entries.glob("*.jsonl")) + + def attempt_dir(self, attempt_id: str) -> Path: + path = self._attempts / attempt_id + try: + path.mkdir(mode=0o700) + except OSError as exc: + raise LedgerError(f"can't keep the attempt: {exc}") from exc + return path + + +class _MemoryLease: + def __init__(self, lock: threading.Lock) -> None: + self._lock: Optional[threading.Lock] = lock + + def release(self) -> None: + lock, self._lock = self._lock, None + if lock is not None: + lock.release() + + +class MemoryLedger: + """The ledger in this process only: for tests. It survives no restart.""" + + def __init__(self, root: Optional[str | Path] = None) -> None: + import tempfile + + self._root = Path(root or tempfile.mkdtemp(prefix="openadapt-ledger-")) + self._secret = secrets.token_bytes(32) + self._events: dict[str, list[dict[str, Any]]] = {} + self._locks: dict[str, threading.Lock] = {} + self._guard = threading.Lock() + + def secret(self) -> bytes: + return self._secret + + def events(self, key: str) -> list[dict[str, Any]]: + with self._guard: + return [dict(e) for e in self._events.get(key, [])] + + def append(self, key: str, event: Mapping[str, Any]) -> None: + with self._guard: + self._events.setdefault(_check_key(key), []).append(dict(event)) + + def lease(self, key: str) -> Optional[Lease]: + with self._guard: + lock = self._locks.setdefault(key, threading.Lock()) + return _MemoryLease(lock) if lock.acquire(blocking=False) else None + + def keys(self) -> list[str]: + with self._guard: + return sorted(self._events) + + def attempt_dir(self, attempt_id: str) -> Path: + path = self._root / attempt_id + path.mkdir(mode=0o700, parents=True) + return path + + +@dataclass +class Attempt: + """The latest attempt of a key, folded from its events.""" + + attempt_id: str + begun: dict[str, Any] + resolved: Optional[dict[str, Any]] = None + delivered: bool = False + released: bool = False + events: list[dict[str, Any]] = field(default_factory=list) + + @property + def state(self) -> str: + """The state with no repeat window (see :meth:`state_with`).""" + + return self.state_with(0.0) + + def state_with(self, repeat_s: float) -> str: + """``open`` (no outcome yet), ``free`` (a new attempt may start), + ``undelivered`` (an outcome to hand back instead of a resend), + ``uncertain`` (reconcile first) or ``held`` (a person checks). + + A write that landed (``CONFIRMED``, or found in the store on a + reconcile) is handed back instead of sent again while its reply may + not have reached the caller, and for ``repeat_s`` after it resolved: + a retry inside that window gets the result. A caller-given write + key names one write, made once: it is handed back for good. + """ + + if self.released: + return "free" # a person checked the record + r = self.resolved + if r is None: + return "open" + if r.get("retry_safe"): + return "free" + if r.get("outcome") == "CONFIRMED" or r.get("landed"): + if r.get("write_key_bound") or r.get("review") or not self.delivered: + # Under review: handed back until a person releases it, so + # no later attempt hides it from the list. + return "undelivered" + return "free" if _age(r.get("at")) >= repeat_s else "undelivered" + if r.get("outcome") == "REFUTED": + return "held" + return "uncertain" + + @property + def review(self) -> bool: + """Landed, but a person should look at the store; until released.""" + + return bool(self.resolved and self.resolved.get("review")) and not self.released + + @property + def folder(self) -> Path: + return Path(str(self.begun.get("folder", ""))) + + +def _age(at: Any) -> float: + import calendar + + try: + then = calendar.timegm(time.strptime(str(at), "%Y-%m-%dT%H:%M:%SZ")) + except ValueError: + return 0.0 + return time.time() - then + + +def prune(folder: Path) -> None: + """Remove an attempt's raw values and baseline once nothing needs them + (it resolved for good). Receipts stay; they hold no record values.""" + + if not str(folder) or not folder.is_absolute(): + return # an attempt with no folder of its own + for name in ("attempt.json", "baseline.json"): + (folder / name).unlink(missing_ok=True) + for copy in folder.glob("reconcile-*/baseline.json"): + copy.unlink(missing_ok=True) + + +def fold(events: Sequence[Mapping[str, Any]]) -> Optional[Attempt]: + """The latest attempt in ``events``, or None when there is none.""" + + latest: Optional[Attempt] = None + for event in events: + kind = event.get("event") + if kind == "begun": + latest = Attempt(str(event.get("attempt")), dict(event)) + continue + if latest is None or event.get("attempt") != latest.attempt_id: + continue + if kind == "resolved": + latest.resolved = dict(event) + latest.delivered = bool(event.get("delivered")) + elif kind == "delivered": + latest.delivered = True + elif kind == "released": + latest.released = True + if latest is not None: + latest.events = [ + dict(e) for e in events if e.get("attempt") == latest.attempt_id + ] + return latest + + +def _scalar(value: Any) -> Any: + """One form per stored value: ``7`` and ``"7"`` are the same write.""" + + if isinstance(value, bool): + return "true" if value else "false" + if value is None: + return "" + if isinstance(value, (int, float, str)): + return str(value) + if isinstance(value, Mapping): + return {str(k): _scalar(v) for k, v in value.items()} + if isinstance(value, (list, tuple)): + return [_scalar(v) for v in value] + return str(value) + + +def _canonical(value: Any) -> bytes: + return json.dumps(value, sort_keys=True, separators=(",", ":"), default=str).encode( + "utf-8" + ) + + +class Ledger: + """The ledger's logic over a :class:`LedgerBackend`. + + ``settle_s``: how long after an attempt began a read-back that shows + nothing proves the write never landed. Default None: such an attempt + stays held until a person releases it, because a queued write can land + late. + + ``repeat_s``: how long a landed write without a write key is handed + back to an identical call instead of being sent again (an agent's retry + after a lost reply). Default 10 minutes. Two intended writes with the + same values inside it need distinct write keys. + """ + + def __init__( + self, + backend: LedgerBackend, + *, + settle_s: Optional[float] = None, + repeat_s: float = 600.0, + ): + self.backend = backend + self.settle_s = settle_s + self.repeat_s = repeat_s + + def key_for( + self, + contract_digest: str, + values: Mapping[str, Any], + write_key: Optional[str] = None, + ) -> str: + """A write's identity: its contract, its bound values, and the + caller's ``write_key`` (a task id) when given.""" + + body = _canonical( + { + "contract": contract_digest, + "values": _scalar(dict(values)), + "key": write_key, + } + ) + return hmac.new(self.backend.secret(), body, hashlib.sha256).hexdigest() + + def latest(self, key: str) -> Optional[Attempt]: + return fold(self.backend.events(key)) + + def state(self, attempt: Optional[Attempt]) -> str: + return "free" if attempt is None else attempt.state_with(self.repeat_s) + + def attempts(self) -> list[tuple[str, dict[str, Any], Optional[dict[str, Any]]]]: + """Every attempt in the ledger: (key, begun event, resolved event). + + A released attempt counts as resolved when a person released it. + """ + + out = [] + for key in self.backend.keys(): + begun: dict[str, dict[str, Any]] = {} + resolved: dict[str, dict[str, Any]] = {} + for event in self.backend.events(key): + attempt = str(event.get("attempt")) + if event.get("event") == "begun": + begun[attempt] = dict(event) + elif event.get("event") in ("resolved", "released"): + resolved.setdefault(attempt, dict(event)) + out += [(key, b, resolved.get(a)) for a, b in begun.items()] + return out + + def begun(self, key: str, attempt_id: str, **fields: Any) -> None: + self.backend.append( + key, + { + "event": "begun", + "attempt": attempt_id, + "at": _now(), + "t": time.time(), + **fields, + }, + ) + + def resolved(self, key: str, attempt_id: str, **fields: Any) -> None: + self.backend.append( + key, + { + "event": "resolved", + "attempt": attempt_id, + "at": _now(), + "t": time.time(), + **fields, + }, + ) + + def delivered(self, key: str, attempt_id: str) -> None: + self.backend.append( + key, {"event": "delivered", "attempt": attempt_id, "at": _now()} + ) + + def release(self, key: str, *, by: str, reason: str, found: str) -> Attempt: + """A person checked the record: release the key's held attempt. + + ``found`` is what they saw in the store. ``absent``: the write isn't + there, so the key is free and it may be sent again. ``landed``: it + is there, so the attempt resolves as landed and an identical call is + handed back, never resent. + + Raises :class:`LedgerError` when another caller holds the key, or + its latest attempt isn't held, uncertain, open or under review. + """ + + if not by.strip() or not reason.strip(): + raise LedgerError("a release names who checked the record, and why") + if found not in ("absent", "landed"): + raise LedgerError("found is absent or landed") + lease = self.backend.lease(_check_key(key)) + if lease is None: + raise LedgerError("the write is in flight in another worker") + try: + attempt = self.latest(key) + if attempt is None or ( + attempt.state not in ("held", "uncertain", "open") + and not attempt.review + ): + raise LedgerError("nothing to release for that key") + if found == "landed": + self.backend.append( + key, + { + "event": "resolved", + "attempt": attempt.attempt_id, + "at": _now(), + "t": time.time(), + "outcome": "UNCERTAIN", + "landed": True, + "delivered": False, + "via": "person", + "by": by.strip()[:128], + "reason": reason.strip()[:500], + "write_key_bound": bool(attempt.begun.get("write_key_bound")), + "result": { + "verified": False, + "outcome": "UNCERTAIN", + "landed": True, + "reason": "a person found the write in the store", + }, + }, + ) + prune(attempt.folder) + return attempt + self.backend.append( + key, + { + "event": "released", + "attempt": attempt.attempt_id, + "at": _now(), + "t": time.time(), + "by": by.strip()[:128], + "reason": reason.strip()[:500], + }, + ) + attempt.released = True + prune(attempt.folder) + return attempt + finally: + lease.release() + + def unresolved(self) -> list[tuple[str, Attempt]]: + """Every key whose latest attempt is open, uncertain or held, or + landed with something a person should look at (``review``).""" + + out = [] + for key in self.backend.keys(): + attempt = self.latest(key) + if attempt is None: + continue + if attempt.state in ("open", "uncertain", "held") or attempt.review: + out.append((key, attempt)) + return out + + +def _now() -> str: + return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) + + +def default_ledger() -> Ledger: + """The :class:`FileLedger` in ``OPENADAPT_LEDGER_DIR`` or + ``~/.openadapt/ledger``.""" + + return Ledger(FileLedger(os.environ.get("OPENADAPT_LEDGER_DIR") or DEFAULT_DIR)) diff --git a/packages/flow/tests/conftest.py b/packages/flow/tests/conftest.py index e90d98135..145f115e4 100644 --- a/packages/flow/tests/conftest.py +++ b/packages/flow/tests/conftest.py @@ -57,4 +57,6 @@ def _isolate_durable_authority(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) # The frame's host secret: a per-test value, so a test never creates or # reads the real ~/.config/openadapt/frame.key. monkeypatch.setenv("OPENADAPT_FRAME_SECRET", "11" * 32) + # The uncertain-write ledger: per test, never ~/.openadapt/ledger. + monkeypatch.setenv("OPENADAPT_LEDGER_DIR", str(tmp_path / ".write-ledger")) monkeypatch.delenv("OPENADAPT_FRAME_SECRET_FILE", raising=False) diff --git a/packages/flow/tests/test_verifier_api.py b/packages/flow/tests/test_verifier_api.py index 439874cf1..3afd1891f 100644 --- a/packages/flow/tests/test_verifier_api.py +++ b/packages/flow/tests/test_verifier_api.py @@ -1,14 +1,12 @@ -"""The stable verifier API: capture, write, verify_write. +"""The one verifier API: begin_write, the write, verify_write. -MCP skills, the RL reward and the registry call ``openadapt_flow.verifier``. -These tests pin its public signatures and check it end to end on a SQLite -store. Records are made up. +These tests pin its public signatures and outcome vocabulary, and check it +end to end on a SQLite store. Records are made up. """ from __future__ import annotations import inspect -import json import sqlite3 from pathlib import Path from typing import Any @@ -20,11 +18,13 @@ Actor, BaselineUnreadable, Contract, - Snapshot, + MemoryLedger, + Outcome, VerifierError, WriteResult, - capture, + begin_write, capture_database, + classify, diff_database, verify_write, ) @@ -33,12 +33,9 @@ @pytest.fixture(autouse=True) -def _used_dir(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: - """Each test records used snapshots in its own directory.""" - - used = tmp_path / "used" - monkeypatch.setattr(verifier, "USED_DIR", used) - return used +def _ledger_dir(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("OPENADAPT_LEDGER_DIR", str(tmp_path / "ledger")) + monkeypatch.setenv("OPENADAPT_SEAL_HOME", str(tmp_path / "seal")) PARAMS = {"fname": "Ada", "lname": "Quill", "email": "ada@example.test"} @@ -94,56 +91,105 @@ def _write(db: Path, *statements: str) -> None: def _check(tmp_path: Path, *statements: str) -> WriteResult: db = _store(tmp_path) - contract = _contract(db) - before = capture(contract, PARAMS) + pending = begin_write(_contract(db), PARAMS, actor=BOT, ledger=MemoryLedger()) _write(db, *statements) - return verify_write(contract, before, params=PARAMS, actor=BOT) + return verify_write(pending) -def test_a_correct_write_is_verified(tmp_path: Path) -> None: +def test_a_correct_write_is_confirmed(tmp_path: Path) -> None: result = _check(tmp_path, SAVE) - assert result.verified and result.outcome == "verified" - assert result.status == "verified" + assert result.outcome is Outcome.CONFIRMED and result.verified + assert result.receipt_outcome == "verified" and result.verdict == "VERIFIED" assert len(result.receipt_sha256) == 64 assert result.checked["delta"] == "held" and result.checked["frame"] == "held" - assert result.reasons == [] - assert result.receipt["outcome"] == "verified" + assert result.reasons == [] and result.observed == "present" + + +def test_a_result_says_what_it_establishes(tmp_path: Path) -> None: + scope = _check(tmp_path, SAVE).scope + assert scope.read_paths == ("database",) and scope.stores == ("sql",) + assert scope.unread == () + assert scope.strength == "independent_system_of_record" + assert scope.window.baseline_read_at and scope.window.read_at + assert scope.window.max_execution_window_s == 3600 + joined = " ".join(scope.not_covered) + for gap in ("after read_at", "email", "other services", "authorized"): + assert gap in joined + # No frame.database: what's outside the read set isn't covered either. + assert "outside the read set" in joined -def test_a_wrong_value_is_not_verified(tmp_path: Path) -> None: +def test_a_wrong_value_is_not_confirmed(tmp_path: Path) -> None: wrong = SAVE.replace("'Ada', 'Quill'", "'Eda', 'Quill'") result = _check(tmp_path, wrong) - assert not result.verified - assert result.checked["delta"] != "held" - assert result.reasons + assert result.outcome is not Outcome.CONFIRMED + assert result.checked["delta"] != "held" and result.reasons -def test_a_stray_write_is_not_verified(tmp_path: Path) -> None: +def test_a_stray_write_is_refuted(tmp_path: Path) -> None: result = _check(tmp_path, SAVE, "UPDATE patient SET phone = '2' WHERE id = 1") - assert not result.verified and result.status == "refuted" + assert result.outcome is Outcome.REFUTED and not result.retry_safe assert result.checked["frame"] == "refuted" assert any("outside the declared write" in r for r in result.reasons) -def test_no_write_is_not_verified(tmp_path: Path) -> None: +def test_a_duplicate_write_is_refuted(tmp_path: Path) -> None: + assert _check(tmp_path, SAVE, SAVE).outcome is Outcome.REFUTED + + +def test_no_write_is_uncertain(tmp_path: Path) -> None: result = _check(tmp_path) - assert not result.verified # The store doesn't show the write: it may still land, so reconcile. - assert result.status == "unconfirmed" + assert result.outcome is Outcome.UNCERTAIN and not result.retry_safe + assert result.verdict == "RECONCILIATION_REQUIRED" + + +def test_a_write_never_sent_and_absent_is_retry_safe(tmp_path: Path) -> None: + db = _store(tmp_path) + pending = begin_write(_contract(db), PARAMS, actor=BOT, ledger=MemoryLedger()) + result = verify_write(pending, write_attempted=False) + assert result.receipt_outcome == "halted_before_effect" + assert result.outcome is Outcome.REFUTED and result.retry_safe def test_an_unreadable_store_stops_before_the_write(tmp_path: Path) -> None: contract = _contract(tmp_path / "missing" / "app.sqlite") with pytest.raises(BaselineUnreadable): - capture(contract, PARAMS) + begin_write(contract, PARAMS, actor=BOT, ledger=MemoryLedger()) + + +def test_an_unknown_actor_kind_is_refused_before_reading(tmp_path: Path) -> None: + with pytest.raises(VerifierError): + begin_write( + _contract(_store(tmp_path)), PARAMS, actor=Actor("robot", "x"), ledger=None + ) + + +def test_a_missing_value_is_refused(tmp_path: Path) -> None: + with pytest.raises(VerifierError, match="values missing"): + begin_write(_contract(_store(tmp_path)), {"fname": "Ada"}, actor=BOT) -def test_an_unknown_actor_kind_is_refused(tmp_path: Path) -> None: +def test_a_pending_write_is_checked_once(tmp_path: Path) -> None: db = _store(tmp_path) - contract = _contract(db) - before = capture(contract, PARAMS) + pending = begin_write(_contract(db), PARAMS, actor=BOT, ledger=MemoryLedger()) + _write(db, SAVE) + assert verify_write(pending).verified with pytest.raises(VerifierError): - verify_write(contract, before, params=PARAMS, actor=Actor("robot", "x")) + verify_write(pending) + + +def test_without_a_ledger_the_folder_is_the_workdir(tmp_path: Path) -> None: + db = _store(tmp_path) + pending = begin_write( + _contract(db), PARAMS, actor=BOT, ledger=None, workdir=tmp_path / "w" + ) + assert pending.workdir.parent == tmp_path / "w" + assert pending.workdir.stat().st_mode & 0o777 == 0o700 + _write(db, SAVE) + result = verify_write(pending) + assert result.verified and result.write_key == "" + assert Path(str(result.receipt_path)).parent == pending.workdir def test_the_database_diff_names_what_moved_without_values(tmp_path: Path) -> None: @@ -155,70 +201,120 @@ def test_the_database_diff_names_what_moved_without_values(tmp_path: Path) -> No assert set(moved) == {"patient"} assert (moved["patient"].added, moved["patient"].changed) == (1, 1) assert moved["patient"].changed_columns == ["phone"] - # Undone: back where it was. _write(db, "DELETE FROM patient WHERE id = 2", "UPDATE patient SET phone = '1'") assert diff_database(before, capture_database(read, "sqlite")) == {} -def test_a_snapshot_verifies_one_write(tmp_path: Path) -> None: - db = _store(tmp_path) - contract = _contract(db) - before = capture(contract, PARAMS) - _write(db, SAVE) - verify_write(contract, before, params=PARAMS, actor=BOT) - with pytest.raises(VerifierError): - verify_write(contract, before, params=PARAMS, actor=BOT) - - -def test_a_snapshot_crosses_a_process_boundary(tmp_path: Path) -> None: - db = _store(tmp_path) - contract = _contract(db) - path = capture(contract, PARAMS).save(tmp_path / "before.json") - assert path.stat().st_mode & 0o777 == 0o600 - _write(db, SAVE) - result = verify_write(contract, Snapshot.load(path), params=PARAMS, actor=BOT) - assert result.verified - - def test_a_bad_contract_raises(tmp_path: Path) -> None: with pytest.raises(VerifierError): Contract.from_dict({"schema_version": "openadapt.verify/v1"}) +def test_the_contract_names_the_values_it_binds(tmp_path: Path) -> None: + assert _contract(_store(tmp_path)).params == ["email", "fname", "lname"] + + +@pytest.mark.parametrize( + "receipt_outcome, checked, rows, expected", + [ + ("verified", {}, [], (Outcome.CONFIRMED, False)), + ("halted_before_effect", {}, [], (Outcome.REFUTED, True)), + ("rejected_policy", {}, [], (Outcome.REFUTED, False)), + ("failed_platform", {}, [], (Outcome.UNCERTAIN, False)), + ("reconciliation_required", {"frame": "refuted"}, [], (Outcome.REFUTED, False)), + ( + "reconciliation_required", + {"source": "refuted"}, + [], + (Outcome.REFUTED, False), + ), + ( + "reconciliation_required", + {"delta": "not_checked"}, + [], + (Outcome.UNCERTAIN, False), + ), + ( + "reconciliation_required", + {"delta": "refuted"}, + [{"verdict": "REFUTED", "observed_effect": "absent"}], + (Outcome.UNCERTAIN, False), + ), + ( + "reconciliation_required", + {"delta": "refuted"}, + [{"verdict": "REFUTED", "observed_effect": "present"}], + (Outcome.REFUTED, False), + ), + ], +) +def test_every_receipt_outcome_maps_to_one_of_three( + receipt_outcome: str, checked: dict, rows: list, expected: tuple +) -> None: + evidence = {"readback": {"effects": rows}} + assert classify(receipt_outcome, checked, evidence) == expected + + def test_the_public_api_is_stable() -> None: assert sorted(verifier.__all__) == [ "Actor", + "AlreadyWritten", "BaselineUnreadable", "Contract", "DatabaseSnapshot", + "FileLedger", + "Ledger", + "LedgerBackend", + "LedgerError", + "MemoryLedger", + "Outcome", + "PendingWrite", + "Scope", "Snapshot", "Source", "TableDiff", "VerifierError", + "Window", + "WriteBlocked", + "WriteHeld", "WriteResult", + "begin_write", "capture", "capture_database", "diff_database", + "reconcile", + "verified_write", "verify_write", ] + assert [o.value for o in Outcome] == ["CONFIRMED", "REFUTED", "UNCERTAIN"] def params(fn: Any) -> list[tuple[str, str]]: return [ (name, p.kind.name) for name, p in inspect.signature(fn).parameters.items() ] - assert params(capture) == [ + assert params(begin_write) == [ ("contract", "POSITIONAL_OR_KEYWORD"), ("params", "POSITIONAL_OR_KEYWORD"), + ("actor", "KEYWORD_ONLY"), + ("ledger", "KEYWORD_ONLY"), + ("write_key", "KEYWORD_ONLY"), + ("authorization_ref", "KEYWORD_ONLY"), + ("workdir", "KEYWORD_ONLY"), + ("declared_by", "KEYWORD_ONLY"), ] assert params(verify_write) == [ - ("contract", "POSITIONAL_OR_KEYWORD"), + ("pending", "POSITIONAL_OR_KEYWORD"), + # before, params, actor and authorization_ref: the deprecated + # (contract, before) form; test_verifier_snapshot_api.py pins it. ("before", "POSITIONAL_OR_KEYWORD"), + ("write_attempted", "KEYWORD_ONLY"), + ("source", "KEYWORD_ONLY"), + ("seal", "KEYWORD_ONLY"), + ("delivered", "KEYWORD_ONLY"), ("params", "KEYWORD_ONLY"), ("actor", "KEYWORD_ONLY"), - ("source", "KEYWORD_ONLY"), ("authorization_ref", "KEYWORD_ONLY"), - ("write_attempted", "KEYWORD_ONLY"), ] assert [n for n in inspect.signature(capture_database).parameters] == [ "read", @@ -230,15 +326,12 @@ def params(fn: Any) -> list[tuple[str, str]]: "before", "actor_names", ] - assert [f for f in WriteResult.__dataclass_fields__] == [ - "verified", - "status", + assert list(WriteResult.__dataclass_fields__)[:5] == [ "outcome", + "scope", "checked", "reasons", - "receipt", - "receipt_sha256", - "evidence", + "retry_safe", ] @@ -263,118 +356,16 @@ def test_diff_database_applies_the_profile_and_before(tmp_path: Path) -> None: "UPDATE users_secure SET login_fail_counter = 1, last_update = 'b' WHERE id = 2", ) after = capture_database(read, "sqlite", before=before, **args) - # The acting user's own sign-in bookkeeping: back where it was. assert diff_database(before, after, **args) == {} _write(db, "UPDATE users_secure SET password = 'x' WHERE id = 1") after = capture_database(read, "sqlite", before=before, **args) assert set(diff_database(before, after, **args)) == {"users_secure"} -def test_a_reloaded_or_copied_snapshot_is_still_single_use(tmp_path: Path) -> None: - """Adversarial review of #1588: one write must not earn two rewards.""" - - import copy - - db = _store(tmp_path) - contract = _contract(db) - first = capture(contract, PARAMS) - path = first.save(tmp_path / "before.json") - _write(db, SAVE) - assert verify_write( - contract, Snapshot.load(path), params=PARAMS, actor=BOT - ).verified - for reuse in ( - Snapshot.load(path), - first, - Snapshot(copy.deepcopy(first.payload)), - ): - with pytest.raises(VerifierError): - verify_write(contract, reuse, params=PARAMS, actor=BOT) - - -def test_a_snapshot_from_another_contract_or_unbound_params_raise_verifier_error( - tmp_path: Path, -) -> None: - db = _store(tmp_path) - contract = _contract(db) - (tmp_path / "other").mkdir() - other_db = _store(tmp_path / "other") - with pytest.raises(VerifierError): - verify_write( - _contract(other_db), capture(contract, PARAMS), params=PARAMS, actor=BOT - ) - with pytest.raises(VerifierError): - verify_write( - contract, capture(contract, PARAMS), params={"fname": "Ada"}, actor=BOT - ) - - -def test_a_snapshot_repr_shows_no_values_or_salt(tmp_path: Path) -> None: - snapshot = capture(_contract(_store(tmp_path)), PARAMS) - assert str(snapshot.payload["salt"]) not in repr(snapshot) - assert "Wendell" not in repr(snapshot) - - -def test_a_used_snapshot_is_refused_in_another_process( - tmp_path: Path, _used_dir: Path -) -> None: - """Re-review of #1588: a copied file, or a snapshot pickled to a worker, - must not verify a second write in a new process.""" - - import shutil - import subprocess - import sys - - db = _store(tmp_path) - contract = _contract(db) - first = capture(contract, PARAMS) - path = first.save(tmp_path / "before.json") - _write(db, SAVE) - assert verify_write( - contract, Snapshot.load(path), params=PARAMS, actor=BOT - ).verified - copy = tmp_path / "copy.json" - shutil.copy(path, copy) - script = ( - "import sys; from pathlib import Path\n" - "from openadapt_flow import verifier as v\n" - "v.USED_DIR = Path(sys.argv[3])\n" - "c = v.Contract.from_dict(__import__('json').loads(sys.argv[2]))\n" - "try:\n" - " v.verify_write(c, v.Snapshot.load(sys.argv[1])," - " params={'fname': 'Ada', 'lname': 'Quill', 'email': 'ada@example.test'}," - " actor=v.Actor(kind='agent', id='bot'))\n" - " print('verified')\n" - "except v.VerifierError:\n" - " print('refused')\n" +def test_a_pending_write_repr_shows_no_values(tmp_path: Path) -> None: + pending = begin_write( + _contract(_store(tmp_path)), PARAMS, actor=BOT, ledger=MemoryLedger() ) - out = subprocess.run( - [sys.executable, "-c", script, str(copy), json.dumps(_doc(db)), str(_used_dir)], - capture_output=True, - text=True, - check=True, - ) - assert out.stdout.strip() == "refused" - - -def test_the_snapshot_fields_are_stable() -> None: - from dataclasses import fields - - assert [f.name for f in fields(Snapshot)] == ["payload", "used", "path"] - assert list(inspect.signature(Snapshot.load).parameters) == ["path"] - assert list(inspect.signature(Snapshot.save).parameters) == ["self", "path"] - - -def test_a_snapshot_verifies_only_where_it_was_captured( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch -) -> None: - """Re-review of #1588: a worker with another HOME (another state - directory) must refuse, not replay.""" - - db = _store(tmp_path) - contract = _contract(db) - before = capture(contract, PARAMS) - _write(db, SAVE) - monkeypatch.setattr(verifier, "USED_DIR", tmp_path / "elsewhere") - with pytest.raises(VerifierError, match="captured under this state"): - verify_write(contract, before, params=PARAMS, actor=BOT) + text = repr(pending) + assert "Ada" not in text and "example.test" not in text + pending.release() diff --git a/packages/flow/tests/test_verifier_snapshot_api.py b/packages/flow/tests/test_verifier_snapshot_api.py new file mode 100644 index 000000000..34596abf6 --- /dev/null +++ b/packages/flow/tests/test_verifier_snapshot_api.py @@ -0,0 +1,389 @@ +"""The deprecated #1588 form of the verifier API: capture, write, +verify_write(contract, before, ...). + +It shipped in 2.4.1, so it keeps working over the one API: these are #1588's +tests, with results in the one outcome vocabulary. ``test_verifier_api.py`` +covers begin_write. Records are made up. +""" + +from __future__ import annotations + +import inspect +import json +import sqlite3 +from pathlib import Path +from typing import Any + +import pytest + +from openadapt_flow import verifier +from openadapt_flow.verifier import ( + Actor, + BaselineUnreadable, + Contract, + Outcome, + Snapshot, + VerifierError, + WriteResult, + capture, + capture_database, + diff_database, + verify_write, +) + +BOT = Actor(kind="agent", id="bot") + + +pytestmark = pytest.mark.filterwarnings("ignore::DeprecationWarning") + + +@pytest.fixture(autouse=True) +def _used_dir(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: + """Each test records used snapshots in its own directory.""" + + used = tmp_path / "used" + monkeypatch.setattr(verifier, "USED_DIR", used) + return used + + +PARAMS = {"fname": "Ada", "lname": "Quill", "email": "ada@example.test"} + + +def _store(tmp_path: Path) -> Path: + path = tmp_path / "app.sqlite" + con = sqlite3.connect(path) + con.executescript( + "CREATE TABLE patient (id INTEGER PRIMARY KEY, fname TEXT, lname TEXT, " + "email TEXT, phone TEXT);" + "INSERT INTO patient VALUES (1, 'Wendell', 'Prior', 'w@example.test', '1');" + ) + con.commit() + con.close() + return path + + +def _doc(db: Path) -> dict[str, Any]: + return { + "schema_version": "openadapt.verify/v1", + "verifier": { + "kind": "sql", + "sqlite_database": str(db), + "sql_query": "SELECT * FROM patient", + }, + "record": { + "name": "patient", + "match": ["email", "lname"], + "fields": {"fname": "fname", "lname": "lname", "email": "email"}, + }, + "frame": {"identity_field": "id"}, + } + + +def _contract(db: Path) -> Contract: + return Contract.from_dict(_doc(db)) + + +def _write(db: Path, *statements: str) -> None: + con = sqlite3.connect(db) + for statement in statements: + con.execute(statement) + con.commit() + con.close() + + +SAVE = ( + "INSERT INTO patient (fname, lname, email) " + "VALUES ('Ada', 'Quill', 'ada@example.test')" +) + + +def _check(tmp_path: Path, *statements: str) -> WriteResult: + db = _store(tmp_path) + contract = _contract(db) + before = capture(contract, PARAMS) + _write(db, *statements) + return verify_write(contract, before, params=PARAMS, actor=BOT) + + +def test_a_correct_write_is_verified(tmp_path: Path) -> None: + result = _check(tmp_path, SAVE) + assert result.verified and result.outcome is Outcome.CONFIRMED + assert result.receipt_outcome == "verified" + assert result.status == "verified" + assert len(result.receipt_sha256) == 64 + assert result.checked["delta"] == "held" and result.checked["frame"] == "held" + assert result.reasons == [] + assert result.receipt["outcome"] == "verified" + + +def test_a_wrong_value_is_not_verified(tmp_path: Path) -> None: + wrong = SAVE.replace("'Ada', 'Quill'", "'Eda', 'Quill'") + result = _check(tmp_path, wrong) + assert not result.verified + assert result.checked["delta"] != "held" + assert result.reasons + + +def test_a_stray_write_is_not_verified(tmp_path: Path) -> None: + result = _check(tmp_path, SAVE, "UPDATE patient SET phone = '2' WHERE id = 1") + assert not result.verified and result.status == "refuted" + assert result.checked["frame"] == "refuted" + assert any("outside the declared write" in r for r in result.reasons) + + +def test_no_write_is_not_verified(tmp_path: Path) -> None: + result = _check(tmp_path) + assert not result.verified + # The store doesn't show the write: it may still land, so reconcile. + assert result.status == "unconfirmed" + + +def test_an_unreadable_store_stops_before_the_write(tmp_path: Path) -> None: + contract = _contract(tmp_path / "missing" / "app.sqlite") + with pytest.raises(BaselineUnreadable): + capture(contract, PARAMS) + + +def test_an_unknown_actor_kind_is_refused(tmp_path: Path) -> None: + db = _store(tmp_path) + contract = _contract(db) + before = capture(contract, PARAMS) + with pytest.raises(VerifierError): + verify_write(contract, before, params=PARAMS, actor=Actor("robot", "x")) + + +def test_the_database_diff_names_what_moved_without_values(tmp_path: Path) -> None: + db = _store(tmp_path) + read = {"sqlite_database": str(db)} + before = capture_database(read, "sqlite") + _write(db, SAVE, "UPDATE patient SET phone = '2' WHERE id = 1") + moved = diff_database(before, capture_database(read, "sqlite")) + assert set(moved) == {"patient"} + assert (moved["patient"].added, moved["patient"].changed) == (1, 1) + assert moved["patient"].changed_columns == ["phone"] + # Undone: back where it was. + _write(db, "DELETE FROM patient WHERE id = 2", "UPDATE patient SET phone = '1'") + assert diff_database(before, capture_database(read, "sqlite")) == {} + + +def test_a_snapshot_verifies_one_write(tmp_path: Path) -> None: + db = _store(tmp_path) + contract = _contract(db) + before = capture(contract, PARAMS) + _write(db, SAVE) + verify_write(contract, before, params=PARAMS, actor=BOT) + with pytest.raises(VerifierError): + verify_write(contract, before, params=PARAMS, actor=BOT) + + +def test_a_snapshot_crosses_a_process_boundary(tmp_path: Path) -> None: + db = _store(tmp_path) + contract = _contract(db) + path = capture(contract, PARAMS).save(tmp_path / "before.json") + assert path.stat().st_mode & 0o777 == 0o600 + _write(db, SAVE) + result = verify_write(contract, Snapshot.load(path), params=PARAMS, actor=BOT) + assert result.verified + + +def test_a_bad_contract_raises(tmp_path: Path) -> None: + with pytest.raises(VerifierError): + Contract.from_dict({"schema_version": "openadapt.verify/v1"}) + + +def test_the_snapshot_api_stays_public() -> None: + assert { + "Actor", + "BaselineUnreadable", + "Contract", + "DatabaseSnapshot", + "Snapshot", + "Source", + "TableDiff", + "VerifierError", + "WriteResult", + "capture", + "capture_database", + "diff_database", + "verify_write", + } <= set(verifier.__all__) + + def params(fn: Any) -> list[tuple[str, str]]: + return [ + (name, p.kind.name) for name, p in inspect.signature(fn).parameters.items() + ] + + assert params(capture) == [ + ("contract", "POSITIONAL_OR_KEYWORD"), + ("params", "POSITIONAL_OR_KEYWORD"), + ] + legacy = dict(params(verify_write)) + assert list(legacy)[:2] == ["pending", "before"] # (contract, before) + for name in ("params", "actor", "source", "authorization_ref", "write_attempted"): + assert legacy[name] == "KEYWORD_ONLY" + assert [n for n in inspect.signature(capture_database).parameters] == [ + "read", + "dialect", + "allow_tables", + "max_rows", + "key", + "profile", + "before", + "actor_names", + ] + for name in ( + "verified", + "status", + "outcome", + "checked", + "reasons", + "receipt", + "receipt_sha256", + "evidence", + ): + assert hasattr(WriteResult, name) or name in WriteResult.__dataclass_fields__ + + +def test_the_two_forms_dont_mix(tmp_path: Path) -> None: + db = _store(tmp_path) + contract = _contract(db) + with pytest.raises(VerifierError): + verify_write(contract, capture(contract, PARAMS), params=PARAMS) + + +def test_diff_database_applies_the_profile_and_before(tmp_path: Path) -> None: + db = tmp_path / "app.sqlite" + con = sqlite3.connect(db) + con.executescript( + "CREATE TABLE users (id INTEGER PRIMARY KEY, username TEXT, uuid TEXT," + " last_updated TEXT);" + "CREATE TABLE users_secure (id INTEGER PRIMARY KEY, username TEXT," + " password TEXT, login_fail_counter INTEGER, last_update TEXT);" + "INSERT INTO users VALUES (1, 'clinic2', 'u1', 'a'), (2, 'bot', 'u2', 'a');" + "INSERT INTO users_secure VALUES (1, 'clinic2', 'h1', 0, 'a'), (2, 'bot', 'h2', 0, 'a');" + ) + con.commit() + con.close() + read = {"sqlite_database": str(db)} + args = {"profile": "openemr", "actor_names": ["bot"]} + before = capture_database(read, "sqlite", **args) + _write( + db, + "UPDATE users_secure SET login_fail_counter = 1, last_update = 'b' WHERE id = 2", + ) + after = capture_database(read, "sqlite", before=before, **args) + # The acting user's own sign-in bookkeeping: back where it was. + assert diff_database(before, after, **args) == {} + _write(db, "UPDATE users_secure SET password = 'x' WHERE id = 1") + after = capture_database(read, "sqlite", before=before, **args) + assert set(diff_database(before, after, **args)) == {"users_secure"} + + +def test_a_reloaded_or_copied_snapshot_is_still_single_use(tmp_path: Path) -> None: + """Adversarial review of #1588: one write must not earn two rewards.""" + + import copy + + db = _store(tmp_path) + contract = _contract(db) + first = capture(contract, PARAMS) + path = first.save(tmp_path / "before.json") + _write(db, SAVE) + assert verify_write( + contract, Snapshot.load(path), params=PARAMS, actor=BOT + ).verified + for reuse in ( + Snapshot.load(path), + first, + Snapshot(copy.deepcopy(first.payload)), + ): + with pytest.raises(VerifierError): + verify_write(contract, reuse, params=PARAMS, actor=BOT) + + +def test_a_snapshot_from_another_contract_or_unbound_params_raise_verifier_error( + tmp_path: Path, +) -> None: + db = _store(tmp_path) + contract = _contract(db) + (tmp_path / "other").mkdir() + other_db = _store(tmp_path / "other") + with pytest.raises(VerifierError): + verify_write( + _contract(other_db), capture(contract, PARAMS), params=PARAMS, actor=BOT + ) + with pytest.raises(VerifierError): + verify_write( + contract, capture(contract, PARAMS), params={"fname": "Ada"}, actor=BOT + ) + + +def test_a_snapshot_repr_shows_no_values_or_salt(tmp_path: Path) -> None: + snapshot = capture(_contract(_store(tmp_path)), PARAMS) + assert str(snapshot.payload["salt"]) not in repr(snapshot) + assert "Wendell" not in repr(snapshot) + + +def test_a_used_snapshot_is_refused_in_another_process( + tmp_path: Path, _used_dir: Path +) -> None: + """Re-review of #1588: a copied file, or a snapshot pickled to a worker, + must not verify a second write in a new process.""" + + import shutil + import subprocess + import sys + + db = _store(tmp_path) + contract = _contract(db) + first = capture(contract, PARAMS) + path = first.save(tmp_path / "before.json") + _write(db, SAVE) + assert verify_write( + contract, Snapshot.load(path), params=PARAMS, actor=BOT + ).verified + copy = tmp_path / "copy.json" + shutil.copy(path, copy) + script = ( + "import sys; from pathlib import Path\n" + "from openadapt_flow import verifier as v\n" + "v.USED_DIR = Path(sys.argv[3])\n" + "c = v.Contract.from_dict(__import__('json').loads(sys.argv[2]))\n" + "try:\n" + " v.verify_write(c, v.Snapshot.load(sys.argv[1])," + " params={'fname': 'Ada', 'lname': 'Quill', 'email': 'ada@example.test'}," + " actor=v.Actor(kind='agent', id='bot'))\n" + " print('verified')\n" + "except v.VerifierError:\n" + " print('refused')\n" + ) + out = subprocess.run( + [sys.executable, "-c", script, str(copy), json.dumps(_doc(db)), str(_used_dir)], + capture_output=True, + text=True, + check=True, + ) + assert out.stdout.strip() == "refused" + + +def test_the_snapshot_fields_are_stable() -> None: + from dataclasses import fields + + assert [f.name for f in fields(Snapshot)] == ["payload", "used", "path"] + assert list(inspect.signature(Snapshot.load).parameters) == ["path"] + assert list(inspect.signature(Snapshot.save).parameters) == ["self", "path"] + + +def test_a_snapshot_verifies_only_where_it_was_captured( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """Re-review of #1588: a worker with another HOME (another state + directory) must refuse, not replay.""" + + db = _store(tmp_path) + contract = _contract(db) + before = capture(contract, PARAMS) + _write(db, SAVE) + monkeypatch.setattr(verifier, "USED_DIR", tmp_path / "elsewhere") + with pytest.raises(VerifierError, match="captured under this state"): + verify_write(contract, before, params=PARAMS, actor=BOT) diff --git a/packages/flow/tests/test_write_ledger.py b/packages/flow/tests/test_write_ledger.py new file mode 100644 index 000000000..b2cd21b90 --- /dev/null +++ b/packages/flow/tests/test_write_ledger.py @@ -0,0 +1,849 @@ +"""The uncertain-write ledger: no write is sent twice while its outcome is unknown. + +Each case runs against MockMed's fault server, a real HTTP app with its own +record store. Workers are separate processes that share one ledger folder, +as a restarted worker or a second worker on the same machine would. The +store is the ground truth: every case counts the records that landed. +""" + +from __future__ import annotations + +import json +import os +import subprocess +import sys +import textwrap +import time +from pathlib import Path + +import pytest + +from openadapt_flow.guard_demo import FakeAgent, verify_yaml +from openadapt_flow.mockmed.fault_server import serve as fault_serve +from openadapt_flow.verifier import ( + Actor, + AlreadyWritten, + Contract, + FileLedger, + Ledger, + MemoryLedger, + Outcome, + WriteHeld, + begin_write, + reconcile, + verify_write, +) +from openadapt_flow.write_ledger import LedgerError, fold + +NOTE = "Follow-up in two weeks" +VALUES = {"patient_id": "p1", "note": NOTE} +BOT = Actor("agent", "intake-bot") + + +@pytest.fixture +def sor(): + url, db, stop = fault_serve() + try: + yield url.rstrip("/"), db + finally: + stop() + + +@pytest.fixture +def ledger_dir(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: + path = tmp_path / "ledger" + monkeypatch.setenv("OPENADAPT_LEDGER_DIR", str(path)) + return path + + +def _contract(base: str) -> Contract: + return Contract.from_dict(verify_yaml(base), name="referral") + + +def _landed(db) -> int: + return sum( + 1 + for r in db.snapshot()["records"] + if r.get("patient_id") == "p1" and r.get("note") == NOTE + ) + + +#: A worker in its own process. argv: base URL, what to do, write_key or "". +WORKER = textwrap.dedent( + """ + import json, os, signal, sys, time + from pathlib import Path + from openadapt_flow.guard_demo import FakeAgent, verify_yaml + from openadapt_flow.verifier import ( + Actor, AlreadyWritten, Contract, WriteHeld, begin_write, verify_write, + ) + + base, mode, key = sys.argv[1], sys.argv[2], sys.argv[3] or None + go = os.environ.get("WORKER_GO") + if go: # start together with the other worker + while not Path(go).exists(): + time.sleep(0.005) + contract = Contract.from_dict(verify_yaml(base), name="referral") + values = {"patient_id": "p1", "note": "Follow-up in two weeks"} + try: + pending = begin_write( + contract, values, actor=Actor("agent", "worker"), write_key=key + ) + except AlreadyWritten as exc: + print(json.dumps({"worker": "already", "result": exc.result})); sys.exit(0) + except WriteHeld as exc: + print(json.dumps({"worker": "held", "reason": str(exc)})); sys.exit(0) + if mode == "die_before_send": + os.kill(os.getpid(), signal.SIGKILL) + agent = FakeAgent(base) + if mode == "die_after_timeout": + try: + agent.click_save("p1", values["note"], fault="timeout") + except Exception: + pass + os.kill(os.getpid(), signal.SIGKILL) + agent.click_save("p1", values["note"]) + if mode == "die_after_send": + os.kill(os.getpid(), signal.SIGKILL) + hold = os.environ.get("WORKER_HOLD") + if hold: # keep the lease while the other worker tries + Path(hold + ".sent").write_text("") + while not Path(hold).exists(): + time.sleep(0.005) + result = verify_write(pending) + print(json.dumps({"worker": "sent", "outcome": result.outcome.value})) + """ +) + + +def _worker( + base: str, mode: str, ledger_dir: Path, key: str = "", **env: str +) -> subprocess.Popen: + return subprocess.Popen( + [sys.executable, "-c", WORKER, base, mode, key], + env={ + **os.environ, + "OPENADAPT_LEDGER_DIR": str(ledger_dir), + **env, + }, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + ) + + +def _reply(proc: subprocess.Popen) -> dict: + out, err = proc.communicate(timeout=120) + assert proc.returncode == 0, err + return json.loads(out.strip().splitlines()[-1]) + + +def _wait_for(path: Path, proc: subprocess.Popen) -> None: + deadline = time.monotonic() + 60 + while not path.exists(): + assert proc.poll() is None, proc.communicate()[1] + assert time.monotonic() < deadline, "the worker never sent its write" + time.sleep(0.01) + + +# -- kill and restart ---------------------------------------------------------- + + +def test_a_worker_killed_after_the_send_is_not_resent_on_restart( + sor, ledger_dir: Path +) -> None: + base, db = sor + dead = _worker(base, "die_after_send", ledger_dir) + dead.communicate(timeout=120) + assert dead.returncode == -9 and _landed(db) == 1 + + # The restarted worker makes the same call: the ledger reads the write back + # with the dead worker's own baseline, and hands back that result. + with pytest.raises(AlreadyWritten) as caught: + begin_write(_contract(base), VALUES, actor=BOT) + assert caught.value.result["outcome"] == "CONFIRMED" + assert caught.value.result["not_sent_again"] is True + assert len(caught.value.result["receipt_sha256"]) == 64 + assert _landed(db) == 1 + + # An identical call inside the repeat window is a retry: handed back too. + with pytest.raises(AlreadyWritten): + begin_write(_contract(base), VALUES, actor=BOT) + # After it (repeat_s=0 here), the same values are a new write. + no_repeat = Ledger(FileLedger(ledger_dir), repeat_s=0) + pending = begin_write(_contract(base), VALUES, actor=BOT, ledger=no_repeat) + FakeAgent(base).click_save("p1", NOTE) + # Its own baseline holds the first note, so the second is its new record. + assert verify_write(pending).verified + assert _landed(db) == 2 + + +def test_a_worker_killed_while_its_write_timed_out_is_not_resent( + sor, ledger_dir: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + from openadapt_flow.mockmed import fault_server + + monkeypatch.setattr(fault_server, "TIMEOUT_HANG_S", 0.5) + base, db = sor + dead = _worker(base, "die_after_timeout", ledger_dir) + dead.communicate(timeout=120) + assert dead.returncode == -9 and _landed(db) == 1 + with pytest.raises(AlreadyWritten): + begin_write(_contract(base), VALUES, actor=BOT) + assert _landed(db) == 1 + + +def test_a_worker_killed_before_the_send_is_held_until_released( + sor, ledger_dir: Path +) -> None: + base, db = sor + dead = _worker(base, "die_before_send", ledger_dir) + dead.communicate(timeout=120) + assert dead.returncode == -9 and _landed(db) == 0 + + # Nothing shows, but a send can't be ruled out: held, never resent. + with pytest.raises(WriteHeld, match="may still land") as caught: + begin_write(_contract(base), VALUES, actor=BOT) + key = caught.value.key + with pytest.raises(WriteHeld): + begin_write(_contract(base), VALUES, actor=BOT) + assert _landed(db) == 0 + + # A person checks the record and releases it; then the write may go. + ledger = Ledger(FileLedger(ledger_dir)) + assert [k for k, _ in ledger.unresolved()] == [key] + ledger.release( + key, by="nurse-lead", reason="checked the chart: no referral", found="absent" + ) + pending = begin_write(_contract(base), VALUES, actor=BOT) + FakeAgent(base).click_save("p1", NOTE) + assert verify_write(pending).verified and _landed(db) == 1 + + +def test_a_settle_time_releases_an_attempt_the_store_never_showed( + sor, ledger_dir: Path +) -> None: + base, db = sor + dead = _worker(base, "die_before_send", ledger_dir) + dead.communicate(timeout=120) + ledger = Ledger(FileLedger(ledger_dir), settle_s=0) + pending = begin_write(_contract(base), VALUES, actor=BOT, ledger=ledger) + FakeAgent(base).click_save("p1", NOTE) + assert verify_write(pending).verified and _landed(db) == 1 + + +def test_a_late_write_found_on_reconcile_is_never_resent(sor, ledger_dir: Path) -> None: + base, db = sor + pending = begin_write(_contract(base), VALUES, actor=BOT) + # The read-back runs before the write lands (a queued write). + assert verify_write(pending).outcome is Outcome.UNCERTAIN + with pytest.raises(WriteHeld): + begin_write(_contract(base), VALUES, actor=BOT) + FakeAgent(base).click_save("p1", NOTE) # it lands late + + # Reconciliation judges it against the attempt's own baseline: confirmed, + # handed back, and never sent again. + with pytest.raises(AlreadyWritten) as caught: + begin_write(_contract(base), VALUES, actor=BOT) + assert caught.value.result["outcome"] == "CONFIRMED" + assert caught.value.result["receipt_outcome"] == "verified" + assert _landed(db) == 1 + + +def test_a_refuted_write_is_held_across_restarts(sor, ledger_dir: Path) -> None: + base, db = sor + pending = begin_write(_contract(base), VALUES, actor=BOT) + FakeAgent(base).click_save("p1", NOTE) + FakeAgent(base).click_save("p1", NOTE) # saved twice + assert verify_write(pending).outcome is Outcome.REFUTED + fresh = Ledger(FileLedger(ledger_dir)) # a new process, the same disk + with pytest.raises(WriteHeld, match="refuted"): + begin_write(_contract(base), VALUES, actor=BOT, ledger=fresh) + assert _landed(db) == 2 + + +def test_reconcile_resolves_an_attempt_by_reading_back(sor, ledger_dir: Path) -> None: + base, db = sor + dead = _worker(base, "die_after_send", ledger_dir) + dead.communicate(timeout=120) + ledger = Ledger(FileLedger(ledger_dir)) + [(key, attempt)] = ledger.unresolved() + assert attempt.state == "open" + assert reconcile(_contract(base), key) == "undelivered" + resolved = ledger.latest(key) + assert resolved is not None and resolved.resolved is not None + assert resolved.resolved["outcome"] == "CONFIRMED" + assert resolved.resolved["via"] == "reconcile" + + +# -- two workers ----------------------------------------------------------------- + + +def test_a_second_worker_waits_while_the_first_has_the_write( + sor, ledger_dir: Path, tmp_path: Path +) -> None: + base, db = sor + hold = tmp_path / "hold" + first = _worker(base, "send", ledger_dir, key="task-0001", WORKER_HOLD=str(hold)) + _wait_for(Path(str(hold) + ".sent"), first) + # The first worker sent the write and hasn't read it back yet. + second = _reply(_worker(base, "send", ledger_dir, key="task-0001")) + assert second["worker"] == "held" and "in flight" in second["reason"] + hold.write_text("") + assert _reply(first) == {"worker": "sent", "outcome": "CONFIRMED"} + # A write key names one write: the retry gets the result, every time. + for _ in range(2): + again = _reply(_worker(base, "send", ledger_dir, key="task-0001")) + assert again["worker"] == "already" + assert again["result"]["outcome"] == "CONFIRMED" + assert _landed(db) == 1 + + +def test_two_workers_started_together_send_one_write( + sor, ledger_dir: Path, tmp_path: Path +) -> None: + base, db = sor + for round_ in range(3): + go = tmp_path / f"go-{round_}" + key = f"task-race-{round_}" + workers = [ + _worker(base, "send", ledger_dir, key=key, WORKER_GO=str(go)) + for _ in range(2) + ] + time.sleep(0.5) + go.write_text("") + replies = sorted(_reply(w)["worker"] for w in workers) + assert replies.count("sent") == 1, replies + assert set(replies) <= {"sent", "held", "already"} + assert _landed(db) == round_ + 1 + + +# -- the ledger itself ----------------------------------------------------------- + + +def test_a_cancelled_reply_is_handed_back_instead_of_a_resend( + sor, ledger_dir: Path +) -> None: + base, db = sor + pending = begin_write(_contract(base), VALUES, actor=BOT) + FakeAgent(base).click_save("p1", NOTE) + verify_write(pending, delivered=False) # the tool call was cancelled + with pytest.raises(AlreadyWritten) as caught: + begin_write(_contract(base), VALUES, actor=BOT) + assert caught.value.result["outcome"] == "CONFIRMED" + assert _landed(db) == 1 + + +def test_the_ledger_holds_no_values_and_is_private(sor, ledger_dir: Path) -> None: + base, _db = sor + pending = begin_write(_contract(base), VALUES, actor=BOT) + FakeAgent(base).click_save("p1", NOTE) + verify_write(pending) + entries = list((ledger_dir / "entries").glob("*.jsonl")) + assert len(entries) == 1 + text = entries[0].read_text() + assert NOTE not in text and '"p1"' not in text + for path in [ledger_dir, *ledger_dir.rglob("*")]: + mode = path.stat().st_mode & 0o777 + assert mode in (0o700, 0o600), (path, oct(mode)) + + +def test_a_torn_last_line_is_ignored_and_appends_stay_whole( + tmp_path: Path, +) -> None: + backend = FileLedger(tmp_path / "ledger") + key = "a" * 64 + backend.append(key, {"event": "begun", "attempt": "write-1"}) + path = tmp_path / "ledger" / "entries" / f"{key}.jsonl" + with path.open("a") as fh: + fh.write('{"event": "resol') # a crash mid-append + assert [e["event"] for e in backend.events(key)] == ["begun"] + backend.append( + key, {"event": "resolved", "attempt": "write-1", "outcome": "REFUTED"} + ) + attempt = fold(backend.events(key)) + assert attempt is not None and attempt.state == "held" + + +def test_a_ledger_folder_others_can_write_is_refused(tmp_path: Path) -> None: + root = tmp_path / "ledger" + FileLedger(root) + # A folder this user owns is reset to 0700; a symlink can't be trusted. + (root / "entries").rmdir() + (root / "entries").symlink_to(tmp_path) + with pytest.raises(LedgerError): + FileLedger(root) + + +def test_a_release_names_who_and_why(sor, ledger_dir: Path) -> None: + base, _db = sor + pending = begin_write(_contract(base), VALUES, actor=BOT) + FakeAgent(base).click_save("p1", NOTE) + FakeAgent(base).click_save("p1", NOTE) + verify_write(pending) + ledger = Ledger(FileLedger(ledger_dir)) + [(key, _)] = ledger.unresolved() + with pytest.raises(LedgerError): + ledger.release(key, by="", reason="", found="absent") + ledger.release(key, by="nurse-lead", reason="removed the duplicate", found="landed") + assert ledger.unresolved() == [] + + +def test_threads_in_one_process_share_the_lease_rules(sor, ledger_dir: Path) -> None: + """Adapters begin and verify on pool threads: a second begin_write in the + same process is refused at once, and whichever thread verifies releases.""" + + from concurrent.futures import ThreadPoolExecutor + + base, db = sor + pool = ThreadPoolExecutor(max_workers=2) + pending = pool.submit( + begin_write, _contract(base), VALUES, actor=BOT, write_key="task-t" + ).result() + started = time.monotonic() + with pytest.raises(WriteHeld, match="in flight"): + pool.submit( + begin_write, _contract(base), VALUES, actor=BOT, write_key="task-t" + ).result(timeout=5) + assert time.monotonic() - started < 5 # refused, never waited + FakeAgent(base).click_save("p1", NOTE) + assert pool.submit(verify_write, pending).result().verified # another thread + with pytest.raises(AlreadyWritten): + pool.submit( + begin_write, _contract(base), VALUES, actor=BOT, write_key="task-t" + ).result() + pool.shutdown() + assert _landed(db) == 1 + + +def test_who_declared_each_value_reaches_the_reply(sor, ledger_dir: Path) -> None: + base, _db = sor + declared = {"patient_id": "operator", "note": "agent"} + pending = begin_write(_contract(base), VALUES, actor=BOT, declared_by=declared) + FakeAgent(base).click_save("p1", NOTE) + assert verify_write(pending).to_dict()["values_declared_by"] == declared + + +# -- review findings (adversarial review of #1628) ------------------------------- + + +def test_editing_read_settings_after_a_crash_keeps_the_hold( + sor, ledger_dir: Path +) -> None: + base, db = sor + dead = _worker(base, "die_after_send", ledger_dir) + dead.communicate(timeout=120) + moved = Contract.from_dict( + verify_yaml(base.replace("127.0.0.1", "localhost")), name="referral" + ) + assert moved.identity == _contract(base).identity + # Another address for the store: the old baseline was read by another + # verifier, so nothing can judge the attempt. Held, never resent. + with pytest.raises(WriteHeld): + begin_write(moved, VALUES, actor=BOT) + doc = verify_yaml(base) + doc["effects"][0]["timeout_s"] = 2.0 + doc["max_execution_window_s"] = 1800 + edited = Contract.from_dict(doc, name="referral") + assert edited.digest != _contract(base).digest + assert edited.identity == _contract(base).identity + # Only timings changed: the attempt is judged and handed back. + with pytest.raises(AlreadyWritten): + begin_write(edited, VALUES, actor=BOT) + assert _landed(db) == 1 + + +def test_a_value_typed_differently_is_the_same_write(sor, ledger_dir: Path) -> None: + base, _db = sor + pending = begin_write(_contract(base), VALUES, actor=BOT) + with pytest.raises(WriteHeld, match="in flight"): + begin_write(_contract(base), {"patient_id": "p1", "note": NOTE}, actor=BOT) + ledger = Ledger(MemoryLedger()) + assert ledger.key_for("c", {"n": 7}) == ledger.key_for("c", {"n": "7"}) + pending.release() + + +def test_another_tasks_identical_write_is_never_taken_for_this_one( + sor, ledger_dir: Path +) -> None: + base, db = sor + # task-1 reads its baseline and dies before sending. + dead = _worker(base, "die_before_send", ledger_dir, key="task-1") + dead.communicate(timeout=120) + # task-2 makes the same write and it lands. task-1 may have sent too, so + # one record can't confirm task-2 either: both wait for a person. + pending = begin_write(_contract(base), VALUES, actor=BOT, write_key="task-2") + FakeAgent(base).click_save("p1", NOTE) + assert verify_write(pending).outcome is Outcome.UNCERTAIN + # task-1 restarts: the one record may be task-2's, so it isn't confirmed. + with pytest.raises(WriteHeld, match="may still land"): + begin_write(_contract(base), VALUES, actor=BOT, write_key="task-1") + assert _landed(db) == 1 + + +def test_other_workers_records_dont_hold_a_correct_write(sor, ledger_dir: Path) -> None: + base, db = sor + contract = _contract(base) + a = begin_write(contract, VALUES, actor=BOT) + other = {"patient_id": "p2", "note": NOTE} + b = begin_write(contract, other, actor=BOT) + FakeAgent(base).click_save("p1", NOTE) + FakeAgent(base).click_save("p2", NOTE) + for pending in (a, b): + result = verify_write(pending) + # Each sees the other's record beside its own: landed, not confirmed, + # never held and never resent. + assert result.outcome is Outcome.UNCERTAIN and result.landed + assert result.review # a person looks at the extra record + with pytest.raises(AlreadyWritten): + begin_write(contract, VALUES, actor=BOT) + listed = Ledger(FileLedger(ledger_dir)).unresolved() + assert len(listed) == 2 and all(a.review for _k, a in listed) + + +def test_a_stray_record_no_attempt_explains_stays_refuted( + sor, ledger_dir: Path +) -> None: + base, _db = sor + pending = begin_write(_contract(base), VALUES, actor=BOT) + FakeAgent(base).click_save("p1", NOTE) + FakeAgent(base).click_save("p3", "a stray write") # no attempt in the ledger + result = verify_write(pending) + assert result.outcome is Outcome.REFUTED and not result.landed + + +def test_a_write_reconciled_after_others_landed_is_not_held( + sor, ledger_dir: Path +) -> None: + base, db = sor + dead = _worker(base, "die_after_send", ledger_dir) + dead.communicate(timeout=120) + # Another write lands before the restart. + other = {"patient_id": "p2", "note": NOTE} + pending = begin_write(_contract(base), other, actor=BOT) + FakeAgent(base).click_save("p2", NOTE) + verify_write(pending) + with pytest.raises(AlreadyWritten) as caught: + begin_write(_contract(base), VALUES, actor=BOT) + assert caught.value.result["landed"] is True + assert _landed(db) == 1 + + +def test_an_unusable_ledger_blocks_the_write(sor, tmp_path: Path, monkeypatch) -> None: + base, _db = sor + bad = tmp_path / "not-private" + bad.mkdir() + (bad / "entries").symlink_to(tmp_path) + monkeypatch.setenv("OPENADAPT_LEDGER_DIR", str(bad)) + with pytest.raises(WriteHeld, match="ledger can't be used"): + begin_write(_contract(base), VALUES, actor=BOT) + + +def test_a_released_pending_write_leaves_the_ledger_alone( + sor, ledger_dir: Path +) -> None: + base, _db = sor + pending = begin_write(_contract(base), VALUES, actor=BOT) + FakeAgent(base).click_save("p1", NOTE) + pending.release() + verify_write(pending) # judged, but not recorded without the lease + [(key, attempt)] = Ledger(FileLedger(ledger_dir)).unresolved() + assert attempt.state == "open" + + +def test_a_resolved_attempt_keeps_no_values(sor, ledger_dir: Path) -> None: + base, _db = sor + pending = begin_write(_contract(base), VALUES, actor=BOT) + FakeAgent(base).click_save("p1", NOTE) + verify_write(pending) + assert not (pending.workdir / "attempt.json").exists() + assert not (pending.workdir / "baseline.json").exists() + assert (pending.workdir / "effect-receipt.json").exists() + + +# -- second review round --------------------------------------------------------- + + +def test_the_same_write_to_another_store_is_its_own_write(ledger_dir: Path) -> None: + url_a, db_a, stop_a = fault_serve() + url_b, db_b, stop_b = fault_serve() + try: + a, b = _contract(url_a.rstrip("/")), _contract(url_b.rstrip("/")) + assert a.identity != b.identity + for key in ("ticket-42", None): + pending = begin_write(a, VALUES, actor=BOT, write_key=key) + FakeAgent(url_a.rstrip("/")).click_save("p1", NOTE) + assert verify_write(pending).verified + pending = begin_write(b, VALUES, actor=BOT, write_key=key) + FakeAgent(url_b.rstrip("/")).click_save("p1", NOTE) + assert verify_write(pending).verified + assert _landed(db_a) == 2 and _landed(db_b) == 2 + finally: + stop_a() + stop_b() + + +def test_a_loopback_alias_is_the_same_store(sor, ledger_dir: Path) -> None: + base, _db = sor + alias = Contract.from_dict( + verify_yaml(base.replace("127.0.0.1", "localhost")), name="referral" + ) + assert alias.identity == _contract(base).identity + + +def test_an_earlier_tasks_record_never_confirms_a_later_unsent_one( + sor, ledger_dir: Path +) -> None: + base, db = sor + first = begin_write(_contract(base), VALUES, actor=BOT, write_key="task-1") + time.sleep(1.1) # the second task begins a second later + dead = _worker(base, "die_before_send", ledger_dir, key="task-2") + dead.communicate(timeout=120) + FakeAgent(base).click_save("p1", NOTE) + assert verify_write(first).outcome is Outcome.UNCERTAIN # task-2 was open + with pytest.raises(WriteHeld): + begin_write(_contract(base), VALUES, actor=BOT, write_key="task-2") + assert _landed(db) == 1 + + +def test_a_phantom_save_beside_an_identical_write_is_not_confirmed( + sor, ledger_dir: Path +) -> None: + base, db = sor + one = begin_write(_contract(base), VALUES, actor=BOT, write_key="task-1") + two = begin_write(_contract(base), VALUES, actor=BOT, write_key="task-2") + FakeAgent(base).click_save("p1", NOTE) + FakeAgent(base).click_save("p1", NOTE, fault="no_persist") + results = [verify_write(one), verify_write(two)] + assert not any(r.verified for r in results) + assert _landed(db) == 1 + + +def test_a_stale_attempt_never_excuses_a_stray_write(sor, ledger_dir: Path) -> None: + base, _db = sor + # An attempt for another patient, begun and abandoned (never sent). + dead = _worker(base, "die_before_send", ledger_dir) + dead.communicate(timeout=120) + other = {"patient_id": "p9", "note": NOTE} + pending = begin_write(_contract(base), other, actor=BOT) + FakeAgent(base).click_save("p9", NOTE) + FakeAgent(base).click_save("p3", "a stray note") + result = verify_write(pending) + assert result.outcome is Outcome.REFUTED and not result.landed + + +# -- third review round ---------------------------------------------------------- + + +def test_releasing_an_open_attempt_lifts_the_hold(sor, ledger_dir: Path) -> None: + base, db = sor + dead = _worker(base, "die_before_send", ledger_dir) + dead.communicate(timeout=120) + ledger = Ledger(FileLedger(ledger_dir)) + [(key, attempt)] = ledger.unresolved() + assert attempt.state == "open" + ledger.release( + key, by="nurse-lead", reason="checked the chart: nothing saved", found="absent" + ) + pending = begin_write(_contract(base), VALUES, actor=BOT) + FakeAgent(base).click_save("p1", NOTE) + assert verify_write(pending).verified and _landed(db) == 1 + + +def test_a_released_stale_attempt_never_holds_later_writes( + sor, ledger_dir: Path +) -> None: + base, db = sor + dead = _worker(base, "die_before_send", ledger_dir, key="task-old") + dead.communicate(timeout=120) + ledger = Ledger(FileLedger(ledger_dir)) + [(key, _)] = ledger.unresolved() + ledger.release(key, by="nurse-lead", reason="nothing saved", found="absent") + pending = begin_write(_contract(base), VALUES, actor=BOT, write_key="task-new") + FakeAgent(base).click_save("p1", NOTE) + assert verify_write(pending).verified + + +def test_a_write_under_review_stays_listed_past_the_repeat_window( + sor, ledger_dir: Path +) -> None: + base, _db = sor + ledger = Ledger(FileLedger(ledger_dir), repeat_s=0) + contract = _contract(base) + a = begin_write(contract, VALUES, actor=BOT, ledger=ledger) + b = begin_write( + contract, {"patient_id": "p2", "note": NOTE}, actor=BOT, ledger=ledger + ) + FakeAgent(base).click_save("p1", NOTE) + FakeAgent(base).click_save("p2", NOTE) + assert verify_write(a).review + verify_write(b) + for _ in range(2): # past repeat_s: still handed back, still listed + with pytest.raises(AlreadyWritten): + begin_write(contract, VALUES, actor=BOT, ledger=ledger) + assert any(a.review for _k, a in ledger.unresolved()) + + +@pytest.mark.parametrize( + "edit", + [ + lambda u: u + "/", + lambda u: u.replace("http://", "HTTP://"), + lambda u: u.replace("127.0.0.1", "LOCALHOST"), + ], +) +def test_a_respelled_store_address_is_the_same_store(sor, edit) -> None: + base, _db = sor + assert Contract.from_dict(verify_yaml(edit(base))).store == _contract(base).store + + +def test_default_ports_and_slashes_spell_one_address() -> None: + from openadapt_flow.verifier import _location + + assert _location("https://Api.Example.test:443/fhir/") == _location( + "https://api.example.test/fhir" + ) + assert _location("https://api.example.test:8443/") != _location( + "https://api.example.test/" + ) + + +def test_the_same_write_under_another_address_is_held_while_open( + sor, ledger_dir: Path +) -> None: + base, db = sor + dead = _worker(base, "die_after_send", ledger_dir, key="ticket-9") + dead.communicate(timeout=120) + doc = verify_yaml(base) + doc["verifier"]["base_url"] = base + "/../" # another spelling, same server + moved = Contract.from_dict(doc, name="referral") + assert moved.store != _contract(base).store + with pytest.raises(WriteHeld, match="another address"): + begin_write(moved, VALUES, actor=BOT, write_key="ticket-9") + assert _landed(db) == 1 + + +def test_tenants_chosen_by_credential_are_different_stores(sor) -> None: + base, _db = sor + docs = [] + for env in ("TENANT_A_TOKEN", "TENANT_B_TOKEN"): + doc = verify_yaml(base) + doc["verifier"]["auth"] = {"kind": "bearer", "bearer_env": env} + docs.append(doc) + try: + a, b = (Contract.from_dict(d) for d in docs) + except Exception as exc: # noqa: BLE001 - auth needs the env set + pytest.skip(f"auth shape not accepted here: {exc}") + assert a.store != b.store + + +# -- fourth review round --------------------------------------------------------- + + +def _split(base: str) -> Contract: + """The same store, its address split differently.""" + + doc = verify_yaml(base) + doc["verifier"]["base_url"] = base + "/api" + doc["verifier"]["records_path"] = "/db" + return Contract.from_dict(doc, name="referral") + + +@pytest.mark.parametrize("key", ["ticket-7", None]) +def test_a_made_write_is_never_resent_under_another_spelling( + sor, ledger_dir: Path, key +) -> None: + base, db = sor + pending = begin_write(_contract(base), VALUES, actor=BOT, write_key=key) + FakeAgent(base).click_save("p1", NOTE) + assert verify_write(pending).verified + assert _split(base).store != _contract(base).store + with pytest.raises(WriteHeld, match="another address"): + begin_write(_split(base), VALUES, actor=BOT, write_key=key) + assert _landed(db) == 1 + + +def test_a_hold_on_one_environment_never_holds_another(ledger_dir: Path) -> None: + url, db, stop = fault_serve() + try: + base = url.rstrip("/") + docs = [] + for env in ("clinic-north-prod", "clinic-south-prod"): + doc = verify_yaml(base) + doc["environment_id"] = env + docs.append(Contract.from_dict(doc, name="referral")) + north, south = docs + held = begin_write(north, VALUES, actor=BOT) # in flight on north + pending = begin_write(south, VALUES, actor=BOT) + pending.release() + held.release() + finally: + stop() + + +def test_releasing_a_stale_attempt_unblocks_the_write_it_held( + sor, ledger_dir: Path +) -> None: + base, db = sor + dead = _worker(base, "die_before_send", ledger_dir, key="task-old") + dead.communicate(timeout=120) + pending = begin_write(_contract(base), VALUES, actor=BOT, write_key="task-new") + FakeAgent(base).click_save("p1", NOTE) + assert verify_write(pending).outcome is Outcome.UNCERTAIN # task-old was open + ledger = Ledger(FileLedger(ledger_dir)) + old = next(k for k, a in ledger.unresolved() if a.state == "open") + ledger.release(old, by="nurse-lead", reason="task-old never sent", found="absent") + # task-new's retry reconciles: confirmed now, handed back, never resent. + with pytest.raises(AlreadyWritten) as caught: + begin_write(_contract(base), VALUES, actor=BOT, write_key="task-new") + assert caught.value.result["outcome"] == "CONFIRMED" + assert _landed(db) == 1 + + +def test_a_release_that_found_the_write_never_frees_it(sor, ledger_dir: Path) -> None: + base, db = sor + pending = begin_write(_contract(base), VALUES, actor=BOT, write_key="task-9") + assert verify_write(pending).outcome is Outcome.UNCERTAIN # read too early + FakeAgent(base).click_save("p1", NOTE) + ledger = Ledger(FileLedger(ledger_dir), settle_s=None) + [(key, _)] = ledger.unresolved() + ledger.release(key, by="nurse-lead", reason="the note is there", found="landed") + for _ in range(2): + with pytest.raises(AlreadyWritten): + begin_write(_contract(base), VALUES, actor=BOT, write_key="task-9") + assert _landed(db) == 1 + + +# -- fifth review round ---------------------------------------------------------- + + +def test_two_sql_servers_with_one_database_name_are_different_shapes() -> None: + def contract(host: str) -> Contract: + return Contract.from_dict( + { + "schema_version": "openadapt.verify/v1", + "verifier": { + "kind": "sql", + "sql_driver": "pymysql", + "sql_connect_args": { + "host": host, + "port": 3306, + "database": "openemr", + }, + "sql_query": "SELECT * FROM patient_data", + }, + "effects": [ + { + "kind": "record_written", + "match": {"fname": {"param": "fname"}}, + "expected_count": 1, + } + ], + } + ) + + try: + a, b = contract("emr-a.internal"), contract("emr-b.internal") + except Exception as exc: # noqa: BLE001 - driver options vary by install + pytest.skip(f"sql contract shape not accepted here: {exc}") + assert a.shape != b.shape and a.store != b.store