diff --git a/CAPABILITIES.md b/CAPABILITIES.md index e3d68257cd..7bf1771e1e 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -155,7 +155,7 @@ improvising or retrying when something is off. | Check that the saved record matches its source document, not only the parameters: intake keeps each value's line, a second reading must agree before a guarded save, and a round trip compares the read-back with the document | On main | Synthetic (MockMed, text referral) | `openadapt_flow/provenance.py`, `intake`, `verify-run --provenance`, guarded save | | Check one write from Python, without the CLI: one API (`openadapt_flow.verifier`: `begin_write`, `verify_write`; 2.4.1's `capture` form still works, deprecated) with one outcome (`CONFIRMED`, `REFUTED`, `UNCERTAIN`), the receipt, which parts were checked, and the result's scope (read path, stores read, time window, what it doesn't cover) | On main | Synthetic (SQLite; MockMed's fault server over real HTTP) | `openadapt_flow/verifier.py`, [VERIFIER_API.md](packages/flow/docs/VERIFIER_API.md) | | Never send a write twice while its outcome is unknown: an append-only ledger on local disk (pluggable backend), keyed by the write's identity, that every adapter, the MCP tools and the CLI consult before sending; a read-back against the attempt's own baseline resolves it, and a refuted write waits for a person (`openadapt verify ledger list`, `reconcile`, `release`) | On main | Synthetic (MockMed's fault server: a worker killed between the send and the read-back, killed during a timed-out send, killed before the send, a late write, and two worker processes started together on one write) | `openadapt_flow/write_ledger.py`, [VERIFIER_API.md](packages/flow/docs/VERIFIER_API.md#the-uncertain-write-ledger) | -| Catch a write that lands after the verdict (a background job, a trigger, a delayed side record): the receipt says when a recheck is due, and the recheck reads the whole database again and refutes the earlier result when a change since the post-save read has no explanation (`frame.database.recheck_after_s`, `verifier.recheck`) | On main ([#1639](https://github.com/OpenAdaptAI/OpenAdapt/pull/1639)) | Synthetic (SQLite); opt-in live OpenEMR test | `openadapt_flow/verifier.py`, `runtime/effects/late_write.py`, [VERIFY_RUN.md](packages/flow/docs/VERIFY_RUN.md#writes-that-land-later) | +| Catch a write that lands after the verdict (a background job, a trigger, a delayed side record): a `CONFIRMED` with a recheck due is provisional (`provisional`, `recheck_due_at`); the recheck reads the whole database again and refutes the earlier result when a change since the post-save read has no explanation, and a recheck that doesn't confirm holds the write in the ledger. Run it from Python (`verifier.recheck`) or MCP (`verify_recheck`). A write after the recheck's read is not seen (`frame.database.recheck_after_s`, no default window) | On main ([#1639](https://github.com/OpenAdaptAI/OpenAdapt/pull/1639)); provisional result, MCP tool and ledger hold ([#1669](https://github.com/OpenAdaptAI/OpenAdapt/pull/1669)) | Real app (Dolibarr 21 and EspoCRM 10 on MariaDB 11.4, 180 s window: 4 of 4 planted late writes at 125 to 130 s refuted at the recheck, correct writes confirmed; EspoCRM's cron daemon needs its job tables declared); synthetic (SQLite); opt-in live OpenEMR test | `openadapt_flow/verifier.py`, `runtime/effects/late_write.py`, [VERIFY_RUN.md](packages/flow/docs/VERIFY_RUN.md#writes-that-land-later) | | Diff the whole database for the frame (`frame.database`) on SQLite, MySQL and MariaDB. On MySQL and MariaDB every column type is read exactly (binary strings, `BIT`, geometries and vectors as hex, `FLOAT` widened to `DOUBLE`, `JSON` as stored text, decimals at full precision), so a row the save adds, changes or removes outside the declared records, such as an extra planned call or a stream note, is refuted unless a declared effect, a side record, an attributed concurrent write or an allow-listed bookkeeping table explains it | On main ([#1661](https://github.com/OpenAdaptAI/OpenAdapt/pull/1661)) | Real app (Dolibarr 21 on MariaDB 11.4: all 387 tables read where the frame raised `TypeError`; EspoCRM on MariaDB 11.4: the first-contact study's extra call and stream note refuted, correct opportunities verified); live MariaDB type tests | [VERIFY_RUN.md](packages/flow/docs/VERIFY_RUN.md#the-whole-database), `runtime/effects/db_snapshot.py` | | Catch a lost update: another user saves a change to the record while the agent works, and the agent saves its stale copy over it. Before and after look like a correct update, so `frame.versions` reads the target's versions (FHIR or SQL version history, a version column, or the app's audit or action-history table) before and after the write. A version by another author, or more than one version step when no author is read, refutes the write; a version that can't be read or attributed stops it | On main ([#1662](https://github.com/OpenAdaptAI/OpenAdapt/pull/1662)) | Real app (HAPI FHIR and EspoCRM, first-contact study fault scripts: both lost updates refuted, correct updates verified, every other case's verdict unchanged) | [VERIFY_RUN.md](packages/flow/docs/VERIFY_RUN.md#lost-updates), `runtime/effects/versions.py` | | Guarded save: a safety layer any agent wraps around one write (check the record, hold for approval, read back) | On main ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)) | Synthetic | `GUARDED_SAVE.md` | @@ -273,7 +273,7 @@ improvising or retrying when something is off. | MCP server: your agent decides what to enter, OpenAdapt makes the entry and checks it | Released | Synthetic sandbox | `openadapt-agent`, [packages/agent](packages/agent/README.md) | | Teach over MCP: an agent does a new task once through OpenAdapt; the run is read back, compiled and admitted (a person confirms the record check; the agent admits only under a person-signed policy), and becomes a tool it calls with no model calls. Web, and Windows, macOS, Linux, RDP and Citrix with a person approving each save | On main ([#1335](https://github.com/OpenAdaptAI/OpenAdapt/pull/1335)) | Synthetic (MockMed over MCP, and through the real RDP and Citrix backends) | `openadapt-agent serve --teach`, [packages/agent](packages/agent/README.md#teach-a-new-workflow-from-your-agent) | | Turn a read-only browser task into a tool for the coding agent you already use: `openadapt setup` registers the MCP server and a skill in Claude Code, Cursor, Codex and Gemini CLI (idempotent, `--dry-run`, `--undo`); the agent does the task once through `teach_start`/`teach_act`/`teach_finish`, the task must end in a download, the file is checked (columns, a column between two inputs or equal to one, fewest rows), the compiled steps are replayed once with no model, and the agent gets a `taught_` tool that downloads a checked file with new inputs. The browser sends no POST, PUT, PATCH or DELETE and stays on one site; a compiled run stops when a control is missing, matches more than one element, or is relabelled | On main ([#1502](https://github.com/OpenAdaptAI/OpenAdapt/pull/1502)) | Real headless Chromium on the local example portal (scripted MCP client, in CI); a real Claude Code agent taught the export from the guide's prompt and called the tool with new dates (2026-10-24, one run) | `openadapt setup`, `openadapt-agent serve --teach-browser`, [FIRST_TOOL.md](packages/flow/docs/FIRST_TOOL.md) | -| Verify any agent's own writes: `openadapt verify` reads the system of record before and after a write, against a contract the operator owns (REST, FHIR, SQL, GraphQL), and returns a verified receipt (`openadapt.effect-receipt/v1`) or a refusal (`REFUTED`, `UNCERTAIN`), never an unverified success. Python SDK (`openadapt.verify`, which is `openadapt_flow.verifier`: `begin_write`, `verify_write`, `verified_write`) wraps a write in any agent framework; the MCP server (`openadapt verify serve`) offers `verify_begin` and `verify_check` to any MCP client | On main ([#1589](https://github.com/OpenAdaptAI/OpenAdapt/pull/1589)) | Synthetic (MockMed's fault server over real HTTP: correct write, wrong record, not saved, saved twice, unreadable store) | `openadapt verify`, [VERIFY_WRITES.md](packages/flow/docs/VERIFY_WRITES.md) | +| Verify any agent's own writes: `openadapt verify` reads the system of record before and after a write, against a contract the operator owns (REST, FHIR, SQL, GraphQL), and returns a verified receipt (`openadapt.effect-receipt/v1`) or a refusal (`REFUTED`, `UNCERTAIN`), never an unverified success. Python SDK (`openadapt.verify`, which is `openadapt_flow.verifier`: `begin_write`, `verify_write`, `verified_write`) wraps a write in any agent framework; the MCP server (`openadapt verify serve`) offers `verify_begin`, `verify_check` and `verify_recheck` to any MCP client | On main ([#1589](https://github.com/OpenAdaptAI/OpenAdapt/pull/1589)) | Synthetic (MockMed's fault server over real HTTP: correct write, wrong record, not saved, saved twice, unreadable store) | `openadapt verify`, [VERIFY_WRITES.md](packages/flow/docs/VERIFY_WRITES.md) | | Skill cache for framework agents: an agent in any framework does a browser task through compile-agent's harness as plain tools (`BrowserTools`); when the system of record shows exactly the task's record and the operator confirms the checks, the run is canaried, admitted and emitted as an MCP server. A repeat of the same task (instruction, record, app, contract, environment) calls that tool instead of the agent: no model calls, the program's checks and this task's record check on every run, eviction after a failed run | On main | Synthetic (MockMed, LangGraph on `claude-sonnet-5-5`, 3 trials, repeats on the same record: run 1 6 model calls and $0.022; runs 2 to 10 27/27 VERIFIED, 0 model calls, 7.8 s median); Trac 1.6: run 1 verified, not admitted | `openadapt_flow.integrations.agents.skills`, [benchmark/skill_cache](packages/flow/benchmark/skill_cache/README.md) | | Verified write tools for agent frameworks: `verified(tool, contract=...)` wraps a Claude Agent SDK, OpenAI Agents SDK, LangGraph or CrewAI write tool in one line. The contract and any values the operator pins come from the operator, never the agent's arguments; the store is read before and after; the agent gets a receipt or a refusal, and a refused write is held instead of retried | On main | Synthetic (each framework's own agent loop with a scripted model, writing through the example app's order form over real HTTP: verified save, a save the app drops, a save it keeps twice); one real-model run per framework, opt-in | `openadapt_flow.integrations.agents`, extras `claude-agent-sdk`, `openai-agents`, `langgraph` (CrewAI: install `crewai` beside it); [AGENT_FRAMEWORKS.md](packages/flow/docs/AGENT_FRAMEWORKS.md) | | Teach a browser write as a verified tool: with write contracts (`serve --teach-browser --contracts DIR`), the agent teaches a save once; run 1 is verified in the system of record before the `taught_` tool exists, and every call makes the write and returns a verified receipt or a refusal. Sign-in values come from the server's environment and never reach the agent or the program. Controls in named iframes are found through their frame | On main ([#1589](https://github.com/OpenAdaptAI/OpenAdapt/pull/1589)) | Synthetic (the example app's sign-in and order form in headless Chromium over MCP: verified save, a save the app drops, a save it keeps twice) | `openadapt-agent serve --teach-browser --contracts`, [VERIFY_WRITES.md](packages/flow/docs/VERIFY_WRITES.md#teach-a-write-as-a-tool) | diff --git a/llms-full.txt b/llms-full.txt index d11a142d40..e060cc35ed 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -552,7 +552,7 @@ improvising or retrying when something is off. | Check that the saved record matches its source document, not only the parameters: intake keeps each value's line, a second reading must agree before a guarded save, and a round trip compares the read-back with the document | On main | Synthetic (MockMed, text referral) | `openadapt_flow/provenance.py`, `intake`, `verify-run --provenance`, guarded save | | Check one write from Python, without the CLI: one API (`openadapt_flow.verifier`: `begin_write`, `verify_write`; 2.4.1's `capture` form still works, deprecated) with one outcome (`CONFIRMED`, `REFUTED`, `UNCERTAIN`), the receipt, which parts were checked, and the result's scope (read path, stores read, time window, what it doesn't cover) | On main | Synthetic (SQLite; MockMed's fault server over real HTTP) | `openadapt_flow/verifier.py`, [VERIFIER_API.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFIER_API.md) | | Never send a write twice while its outcome is unknown: an append-only ledger on local disk (pluggable backend), keyed by the write's identity, that every adapter, the MCP tools and the CLI consult before sending; a read-back against the attempt's own baseline resolves it, and a refuted write waits for a person (`openadapt verify ledger list`, `reconcile`, `release`) | On main | Synthetic (MockMed's fault server: a worker killed between the send and the read-back, killed during a timed-out send, killed before the send, a late write, and two worker processes started together on one write) | `openadapt_flow/write_ledger.py`, [VERIFIER_API.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFIER_API.md#the-uncertain-write-ledger) | -| Catch a write that lands after the verdict (a background job, a trigger, a delayed side record): the receipt says when a recheck is due, and the recheck reads the whole database again and refutes the earlier result when a change since the post-save read has no explanation (`frame.database.recheck_after_s`, `verifier.recheck`) | On main ([#1639](https://github.com/OpenAdaptAI/OpenAdapt/pull/1639)) | Synthetic (SQLite); opt-in live OpenEMR test | `openadapt_flow/verifier.py`, `runtime/effects/late_write.py`, [VERIFY_RUN.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFY_RUN.md#writes-that-land-later) | +| Catch a write that lands after the verdict (a background job, a trigger, a delayed side record): a `CONFIRMED` with a recheck due is provisional (`provisional`, `recheck_due_at`); the recheck reads the whole database again and refutes the earlier result when a change since the post-save read has no explanation, and a recheck that doesn't confirm holds the write in the ledger. Run it from Python (`verifier.recheck`) or MCP (`verify_recheck`). A write after the recheck's read is not seen (`frame.database.recheck_after_s`, no default window) | On main ([#1639](https://github.com/OpenAdaptAI/OpenAdapt/pull/1639)); provisional result, MCP tool and ledger hold ([#1669](https://github.com/OpenAdaptAI/OpenAdapt/pull/1669)) | Real app (Dolibarr 21 and EspoCRM 10 on MariaDB 11.4, 180 s window: 4 of 4 planted late writes at 125 to 130 s refuted at the recheck, correct writes confirmed; EspoCRM's cron daemon needs its job tables declared); synthetic (SQLite); opt-in live OpenEMR test | `openadapt_flow/verifier.py`, `runtime/effects/late_write.py`, [VERIFY_RUN.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFY_RUN.md#writes-that-land-later) | | Diff the whole database for the frame (`frame.database`) on SQLite, MySQL and MariaDB. On MySQL and MariaDB every column type is read exactly (binary strings, `BIT`, geometries and vectors as hex, `FLOAT` widened to `DOUBLE`, `JSON` as stored text, decimals at full precision), so a row the save adds, changes or removes outside the declared records, such as an extra planned call or a stream note, is refuted unless a declared effect, a side record, an attributed concurrent write or an allow-listed bookkeeping table explains it | On main ([#1661](https://github.com/OpenAdaptAI/OpenAdapt/pull/1661)) | Real app (Dolibarr 21 on MariaDB 11.4: all 387 tables read where the frame raised `TypeError`; EspoCRM on MariaDB 11.4: the first-contact study's extra call and stream note refuted, correct opportunities verified); live MariaDB type tests | [VERIFY_RUN.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFY_RUN.md#the-whole-database), `runtime/effects/db_snapshot.py` | | Catch a lost update: another user saves a change to the record while the agent works, and the agent saves its stale copy over it. Before and after look like a correct update, so `frame.versions` reads the target's versions (FHIR or SQL version history, a version column, or the app's audit or action-history table) before and after the write. A version by another author, or more than one version step when no author is read, refutes the write; a version that can't be read or attributed stops it | On main ([#1662](https://github.com/OpenAdaptAI/OpenAdapt/pull/1662)) | Real app (HAPI FHIR and EspoCRM, first-contact study fault scripts: both lost updates refuted, correct updates verified, every other case's verdict unchanged) | [VERIFY_RUN.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFY_RUN.md#lost-updates), `runtime/effects/versions.py` | | Guarded save: a safety layer any agent wraps around one write (check the record, hold for approval, read back) | On main ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)) | Synthetic | `GUARDED_SAVE.md` | @@ -670,7 +670,7 @@ improvising or retrying when something is off. | MCP server: your agent decides what to enter, OpenAdapt makes the entry and checks it | Released | Synthetic sandbox | `openadapt-agent`, [packages/agent](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md) | | Teach over MCP: an agent does a new task once through OpenAdapt; the run is read back, compiled and admitted (a person confirms the record check; the agent admits only under a person-signed policy), and becomes a tool it calls with no model calls. Web, and Windows, macOS, Linux, RDP and Citrix with a person approving each save | On main ([#1335](https://github.com/OpenAdaptAI/OpenAdapt/pull/1335)) | Synthetic (MockMed over MCP, and through the real RDP and Citrix backends) | `openadapt-agent serve --teach`, [packages/agent](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md#teach-a-new-workflow-from-your-agent) | | Turn a read-only browser task into a tool for the coding agent you already use: `openadapt setup` registers the MCP server and a skill in Claude Code, Cursor, Codex and Gemini CLI (idempotent, `--dry-run`, `--undo`); the agent does the task once through `teach_start`/`teach_act`/`teach_finish`, the task must end in a download, the file is checked (columns, a column between two inputs or equal to one, fewest rows), the compiled steps are replayed once with no model, and the agent gets a `taught_` tool that downloads a checked file with new inputs. The browser sends no POST, PUT, PATCH or DELETE and stays on one site; a compiled run stops when a control is missing, matches more than one element, or is relabelled | On main ([#1502](https://github.com/OpenAdaptAI/OpenAdapt/pull/1502)) | Real headless Chromium on the local example portal (scripted MCP client, in CI); a real Claude Code agent taught the export from the guide's prompt and called the tool with new dates (2026-10-24, one run) | `openadapt setup`, `openadapt-agent serve --teach-browser`, [FIRST_TOOL.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/FIRST_TOOL.md) | -| Verify any agent's own writes: `openadapt verify` reads the system of record before and after a write, against a contract the operator owns (REST, FHIR, SQL, GraphQL), and returns a verified receipt (`openadapt.effect-receipt/v1`) or a refusal (`REFUTED`, `UNCERTAIN`), never an unverified success. Python SDK (`openadapt.verify`, which is `openadapt_flow.verifier`: `begin_write`, `verify_write`, `verified_write`) wraps a write in any agent framework; the MCP server (`openadapt verify serve`) offers `verify_begin` and `verify_check` to any MCP client | On main ([#1589](https://github.com/OpenAdaptAI/OpenAdapt/pull/1589)) | Synthetic (MockMed's fault server over real HTTP: correct write, wrong record, not saved, saved twice, unreadable store) | `openadapt verify`, [VERIFY_WRITES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFY_WRITES.md) | +| Verify any agent's own writes: `openadapt verify` reads the system of record before and after a write, against a contract the operator owns (REST, FHIR, SQL, GraphQL), and returns a verified receipt (`openadapt.effect-receipt/v1`) or a refusal (`REFUTED`, `UNCERTAIN`), never an unverified success. Python SDK (`openadapt.verify`, which is `openadapt_flow.verifier`: `begin_write`, `verify_write`, `verified_write`) wraps a write in any agent framework; the MCP server (`openadapt verify serve`) offers `verify_begin`, `verify_check` and `verify_recheck` to any MCP client | On main ([#1589](https://github.com/OpenAdaptAI/OpenAdapt/pull/1589)) | Synthetic (MockMed's fault server over real HTTP: correct write, wrong record, not saved, saved twice, unreadable store) | `openadapt verify`, [VERIFY_WRITES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFY_WRITES.md) | | Skill cache for framework agents: an agent in any framework does a browser task through compile-agent's harness as plain tools (`BrowserTools`); when the system of record shows exactly the task's record and the operator confirms the checks, the run is canaried, admitted and emitted as an MCP server. A repeat of the same task (instruction, record, app, contract, environment) calls that tool instead of the agent: no model calls, the program's checks and this task's record check on every run, eviction after a failed run | On main | Synthetic (MockMed, LangGraph on `claude-sonnet-5-5`, 3 trials, repeats on the same record: run 1 6 model calls and $0.022; runs 2 to 10 27/27 VERIFIED, 0 model calls, 7.8 s median); Trac 1.6: run 1 verified, not admitted | `openadapt_flow.integrations.agents.skills`, [benchmark/skill_cache](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/skill_cache/README.md) | | Verified write tools for agent frameworks: `verified(tool, contract=...)` wraps a Claude Agent SDK, OpenAI Agents SDK, LangGraph or CrewAI write tool in one line. The contract and any values the operator pins come from the operator, never the agent's arguments; the store is read before and after; the agent gets a receipt or a refusal, and a refused write is held instead of retried | On main | Synthetic (each framework's own agent loop with a scripted model, writing through the example app's order form over real HTTP: verified save, a save the app drops, a save it keeps twice); one real-model run per framework, opt-in | `openadapt_flow.integrations.agents`, extras `claude-agent-sdk`, `openai-agents`, `langgraph` (CrewAI: install `crewai` beside it); [AGENT_FRAMEWORKS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/AGENT_FRAMEWORKS.md) | | Teach a browser write as a verified tool: with write contracts (`serve --teach-browser --contracts DIR`), the agent teaches a save once; run 1 is verified in the system of record before the `taught_` tool exists, and every call makes the write and returns a verified receipt or a refusal. Sign-in values come from the server's environment and never reach the agent or the program. Controls in named iframes are found through their frame | On main ([#1589](https://github.com/OpenAdaptAI/OpenAdapt/pull/1589)) | Synthetic (the example app's sign-in and order form in headless Chromium over MCP: verified save, a save the app drops, a save it keeps twice) | `openadapt-agent serve --teach-browser --contracts`, [VERIFY_WRITES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFY_WRITES.md#teach-a-write-as-a-tool) | @@ -1755,6 +1755,16 @@ same-surface observations. A visible value does not prove it was durably stored. Masked input can prove that the field changed, not what secret value it now contains. +A write check sees the store as it was when it read. A write that lands +later, such as a delayed retry, a queued job or a trigger, is not in its +result. With `frame.database.recheck_after_s`, a `CONFIRMED` write check is +provisional (`provisional: true`, `recheck_due_at`) until a recheck reads the +whole database again, and the recheck refutes it when something moved that +nothing explains. A write after the recheck's read is still not seen, and +the window has no default: the contract's author picks one longer than the +slowest write that can follow a save +([`VERIFY_RUN.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFY_RUN.md#writes-that-land-later)). + `lint` reports vacuous postconditions and other coverage gaps. `certify` enforces the selected policy. Neither command understands the application's business semantics, and the permissive `replay` command does not require diff --git a/openadapt/cli.py b/openadapt/cli.py index 5e1fb17c92..dccb57cbef 100644 --- a/openadapt/cli.py +++ b/openadapt/cli.py @@ -1300,7 +1300,8 @@ def verify() -> None: help="Where baselines and receipts go (default ~/.openadapt/verify).", ) def verify_serve(contracts: str, state: Optional[str]) -> None: - """Serve verify_contracts, verify_begin and verify_check over MCP (stdio).""" + """Serve verify_contracts, verify_begin, verify_check and verify_recheck + over MCP (stdio).""" _require_supported_python() try: from openadapt_flow.agent.cli import main as agent_main diff --git a/packages/flow/CHANGELOG.md b/packages/flow/CHANGELOG.md index 43e8f68bb8..d328b95ea4 100644 --- a/packages/flow/CHANGELOG.md +++ b/packages/flow/CHANGELOG.md @@ -23,6 +23,8 @@ - A write that lands after the verdict can now be caught. With `frame.database.recheck_after_s`, a verified write's receipt says when a recheck is due (`recheck_due_at`), and `openadapt_flow.verifier.recheck` reads the whole database again at or after that time. It judges every change since the post-save read with the frame's rules: the app profile's bookkeeping, other users' writes the audit log attributes, and changes a later write by the same actor under the same contract saw in its own window (`excused_by`, any outcome). Anything else turns the earlier result into `REFUTED` (late). A change that may be what refuted a later write, a database that can't be read, or a saved state that wasn't issued here or doesn't match its receipt gives `UNCERTAIN`. The saved state holds keyed hashes, not values. `frame.database.settle_attempts` lets the reads after a save, and at a recheck, retry within a stated budget until two in a row match; rows added to append-only tables between them don't count as movement. Without these settings, nothing changes ([#1639](https://github.com/OpenAdaptAI/OpenAdapt/pull/1639)). +- A `CONFIRMED` write with a recheck due now says it is provisional (`WriteResult.provisional`, `provisional` in `to_dict()`). MCP clients of `openadapt verify serve` can run the recheck with the new `verify_recheck` tool. A recheck that gives `REFUTED` or `UNCERTAIN` now resolves the write's ledger attempt again for review, so a retry under the same key is held for a person instead of being handed the earlier `CONFIRMED`. A recheck from a saved `recheck.json` finds the attempt by the receipt's `correlation_id`; the result carries its ledger (`WriteResult.ledger`); when no attempt is re-resolved, the recheck's reasons say so. Over MCP, `verify_recheck` excuses the agent's own later writes under the same contract. + ### Fixes - The read-only SQL guard now reads the query's tokens instead of matching substrings. `REPLACE(phone, '-', '')`, a `;` inside a string literal, and a table quoted as `` `call` `` no longer refuse the query. Every write and multi-statement form it refused before is still refused. It now also refuses a backslash, a dollar-quoted string, an unterminated literal, and a call to `query_to_xml` or `eval`, which run a query passed as text. diff --git a/packages/flow/docs/LIMITS.md b/packages/flow/docs/LIMITS.md index 3006d5e85b..8f6f3ec839 100644 --- a/packages/flow/docs/LIMITS.md +++ b/packages/flow/docs/LIMITS.md @@ -574,6 +574,16 @@ same-surface observations. A visible value does not prove it was durably stored. Masked input can prove that the field changed, not what secret value it now contains. +A write check sees the store as it was when it read. A write that lands +later, such as a delayed retry, a queued job or a trigger, is not in its +result. With `frame.database.recheck_after_s`, a `CONFIRMED` write check is +provisional (`provisional: true`, `recheck_due_at`) until a recheck reads the +whole database again, and the recheck refutes it when something moved that +nothing explains. A write after the recheck's read is still not seen, and +the window has no default: the contract's author picks one longer than the +slowest write that can follow a save +([`VERIFY_RUN.md`](VERIFY_RUN.md#writes-that-land-later)). + `lint` reports vacuous postconditions and other coverage gaps. `certify` enforces the selected policy. Neither command understands the application's business semantics, and the permissive `replay` command does not require diff --git a/packages/flow/docs/VERIFIER_API.md b/packages/flow/docs/VERIFIER_API.md index d3278e89be..58818dd4d5 100644 --- a/packages/flow/docs/VERIFIER_API.md +++ b/packages/flow/docs/VERIFIER_API.md @@ -137,12 +137,25 @@ late = recheck(contract, result, params=params, actor=actor, late.outcome # CONFIRMED, REFUTED (late) or UNCERTAIN ``` -The caller can schedule it from `recheck_due_at` and keep working: before -then, `recheck` raises `RecheckNotDue` (`wait=True` sleeps instead). It reads -the whole database again and judges every change since the post-save read. -A later write passed in `excused_by` accounts only for what changed in its -own window. A change that may be what refuted that later write gives -`UNCERTAIN`. See [VERIFY_RUN.md](VERIFY_RUN.md#writes-that-land-later). +Until the recheck runs, the result is provisional: `result.provisional` is +true (`provisional` in `to_dict()`), and a write that lands before the due +time can still refute it. The caller can schedule the recheck from +`recheck_due_at` and keep working: before then, `recheck` raises +`RecheckNotDue` (`wait=True` sleeps instead). It reads the whole database +again and judges every change since the post-save read. A later write passed +in `excused_by` accounts only for what changed in its own window. A change +that may be what refuted that later write gives `UNCERTAIN`. + +A recheck that gives `REFUTED` or `UNCERTAIN` resolves the write's ledger +attempt again, for review: a retry under the same key is held for a person, +never handed the earlier `CONFIRMED`. The ledger is the one the result +carries from `begin_write` (`result.ledger`), else the default; `ledger=` +overrides it (`None`: no ledger). A saved `RecheckState` finds its attempt by +the receipt's `correlation_id`. When no attempt is re-resolved, the recheck's +`reasons` say so. Over MCP, the agent's later writes under the same contract +excuse their own rows; writes under other contracts can't. A write that +lands after the recheck's read is not seen. MCP clients run the recheck with +`verify_recheck`. See [VERIFY_RUN.md](VERIFY_RUN.md#writes-that-land-later). ## Before the write diff --git a/packages/flow/docs/VERIFY_RUN.md b/packages/flow/docs/VERIFY_RUN.md index 9b501300f8..11da921326 100644 --- a/packages/flow/docs/VERIFY_RUN.md +++ b/packages/flow/docs/VERIFY_RUN.md @@ -528,6 +528,9 @@ When it is set and the run is verified: the recheck without waiting on it. - A verified receipt that carries `recheck_due_at` is provisional until the recheck runs: a write that lands in the window can still refute it. + `WriteResult.provisional` is true, and so is `provisional` in + `to_dict()` and in the MCP `verify_check` reply. Don't treat the write as + final until the recheck confirms it. - `evidence.frame.recheck` says what the recheck compares (`database_since_post_save`), when the post-save read was taken, and a salted digest of the state it compares with. @@ -586,6 +589,87 @@ until then instead. Limits: - A later write whose database never settled after its save has no state, so it can't be passed as `excused_by`. Reconcile it first. +### Running the recheck + +In Python, call it at or after `recheck_due_at`, or let it sleep until then: + +```python +from openadapt_flow.verifier import recheck + +late = recheck(contract, result, params=values, actor=actor, wait=True) +late.outcome # CONFIRMED, REFUTED or UNCERTAIN +late.read_at # the recheck's own read; nothing after it is seen +``` + +A process that restarts in the window can load the saved state with +`RecheckState.load(path, params=values)` and pass it as `write`. + +Over MCP (`openadapt verify serve`), call `verify_recheck` with the +`write_id` of a `verify_check` reply that says `provisional: true`. Before the +due time it returns `due: false`, `recheck_due_at` and `wait_s`. The server +passes the agent's own later writes as `excused_by`: those under the same +contract, by the same agent, begun after this write's post-save read and +already checked with `verify_check`. While such a write is begun but not yet +checked, `verify_recheck` refuses and asks for that check first. Writes under +different contracts can't excuse each other: an agent that creates a contact +and then an opportunity inside the window gets the first write `REFUTED`. +The server holds the state in memory, so after a restart run the recheck +from Python. There is no CLI command for it. + +A recheck that gives `REFUTED` or `UNCERTAIN` also resolves the write's +ledger attempt again, for review. `REFUTED` holds it, and `UNCERTAIN` keeps it +uncertain, so a retry under the same key is never handed the earlier +`CONFIRMED`; `openadapt verify ledger list` shows it until a person releases +it. The ledger is the one the `WriteResult` carries from `begin_write`, or +the default ledger; `ledger=` overrides it. A `RecheckState` loaded from disk +finds its attempt by the receipt's `correlation_id` (the write id). When no +attempt is re-resolved (no ledger, a ledger without that attempt, or a newer +attempt of the same write), the recheck's `reasons` say so, and the write +must be held by hand. + +### Choosing the window + +`recheck_after_s` has no default. Make it longer than the slowest write +that can follow a save in your system: a client's delayed retry, a queued +job, a scheduled task, an integration that writes back. A write that lands +after the recheck's read is not seen by the recheck or by anything else +here; to cover a longer delay, make the window longer. The longer the +window, the more of the app's own background activity it also sees, and each +such change refutes unless the contract explains it (`allow_tables`, the app +profile, `ignore_columns`, `side_records`) or the audit log attributes it to +another user. + +Measured on two real apps, Dolibarr 21 and EspoCRM 10 on MariaDB 11.4, with +`recheck_after_s: 180`: + +- Four planted late writes (a changed phone, a second product, a delayed + duplicate opportunity, a look-alike lead's title) landed 125 to 130 s + after the save. Each check said `CONFIRMED`, and each recheck said + `REFUTED`. Correct writes of the same tasks stayed `CONFIRMED`. +- With the apps' schedulers off, `settle_s: 1` and `settle_attempts: 1` were + enough. +- With EspoCRM's cron daemon on, its job queue moved during every window. + Correct writes were refuted at the recheck (about 140 rows each), and a + recheck read while a job ran was `UNCERTAIN`. Declaring the scheduler's + bookkeeping fixed both: `scheduled_job_log_record` append-only, `job` + skipped with a reason (the queue adds and then updates its rows), + `scheduled_job.last_run` in `ignore_columns`, and `settle_attempts: 10`. + Then the correct writes were `CONFIRMED` again. An app with a scheduler + needs this kind of review before its rechecks mean anything. + +Each of those declarations stops checking something, at the check and at +the recheck: + +- A skipped table (`job`) is never read. Every write to it is unseen, + including a job the agent's own action queues (a mass email, a workflow, + a webhook) or one it adds by mistake. +- An append-only table (`scheduled_job_log_record`) lets any added row + through, whoever added it. Only an edited or removed row counts. +- An ignored column (`scheduled_job.last_run`) is not compared on any row. + +Prefer append-only, or `ignore_columns` on the columns that move, over +skipping a table when the table allows it. + ## Verdicts | What the store shows | `effect_verdict` | `outcome` | diff --git a/packages/flow/docs/VERIFY_WRITES.md b/packages/flow/docs/VERIFY_WRITES.md index 5d7e4b6bc3..5bcc461af6 100644 --- a/packages/flow/docs/VERIFY_WRITES.md +++ b/packages/flow/docs/VERIFY_WRITES.md @@ -64,13 +64,31 @@ The agent calls `verify_begin` before the write, makes the write its own way, th | Result | What it means | What to do | | --- | --- | --- | -| `outcome: CONFIRMED` (`verified: true`) | The system of record shows exactly the declared write. The result carries a receipt (`receipt_sha256`). | Nothing more. | +| `outcome: CONFIRMED` (`verified: true`) | The system of record shows exactly the declared write. The result carries a receipt (`receipt_sha256`). | Nothing more, unless it also says `provisional: true`. | +| `outcome: CONFIRMED` with `provisional: true` and `recheck_due_at` | The store showed exactly the declared write when it was read. The contract asks for a recheck, because a write can still land later and undo that. | Run the recheck at or after `recheck_due_at` (see below). Until then, treat the write as not yet final. | | `outcome: REFUTED` | The store shows a different write: a wrong value, another record, or a duplicate. | Don't retry. A person fixes the record. | | `outcome: UNCERTAIN` (`verdict: RECONCILIATION_REQUIRED`) | The store doesn't show the write, or couldn't be read. It may still land. | Don't retry, or it could save twice. The ledger holds it until a read-back finds it or a person checks. | | `ok_to_write: false` (from `verify_begin`), or `WriteBlocked` | The store couldn't be read before the write, or the ledger holds an earlier attempt of the same write. | Don't make the write. | Every result also says what it establishes (`scope`): the read path (database, API, FHIR, audit log or screen), the stores read, the time window, and what it doesn't cover, such as email, other services, and writes that land after the read. The ledger keeps a write that may have landed from being sent again, also after a restart or by another worker (`openadapt verify ledger list`). See [VERIFIER_API.md](VERIFIER_API.md). +### Writes that land later + +A check sees what is in the store when it reads. A write that lands after that read is not part of the result: a retry that fires two minutes later, a queued job, a trigger. To catch those, the contract can ask for a recheck. Set `frame.database.recheck_after_s` (a contract that reads the whole database), and every `CONFIRMED` result says `provisional: true` and when the recheck is due (`recheck_due_at`). The recheck reads the whole database again and judges every change since the first check: + +- In Python: `recheck(contract, result, params=values, actor=AGENT)` at or after `recheck_due_at`, or with `wait=True` to sleep until then. +- Over MCP: call `verify_recheck` with the `write_id`. Before the due time it returns `due: false` and how long to wait. Your agent's later writes under the same contract, once checked, account for their own rows. A write under another contract can't: an agent that creates a contact and then an opportunity inside the window gets the first write `REFUTED`. + +| Recheck outcome | What it means | What to do | +| --- | --- | --- | +| `CONFIRMED` | Nothing moved since the first check that the contract doesn't explain. | The write is final as of the recheck. | +| `REFUTED` | Something changed after the first check that nothing explains: the earlier `CONFIRMED` no longer stands. | Don't retry. A person checks the record. | +| `UNCERTAIN` | The database couldn't be read again, or was still changing. | Don't retry. A person checks the record. | + +A recheck that doesn't confirm also marks the write in the ledger, so a retry of it is held for a person (`openadapt verify ledger list`) instead of being handed the earlier `CONFIRMED`. That holds for a recheck run from a saved `recheck.json` too. If the recheck can't find the write in the ledger, its reasons say so: hold the write by hand. + +There is no default window: pick one longer than the slowest write that can follow a save in your system. A write after the recheck's read is not seen by anything. See [VERIFY_RUN.md](VERIFY_RUN.md#writes-that-land-later) for how the recheck judges changes and what was measured on real apps. + A verified result proves the system of record holds what was declared. It doesn't prove the write was authorized: `values_declared_by` says who declared the values (the agent), and `authorization` is `unbound` unless you pass an operator's authorization reference to `begin_write`. The receipt is sealed with this computer's Seal key (`seal_path`; manage the key with `openadapt-flow local-seal key`, or turn Seals off with `OPENADAPT_LOCAL_SEAL=off`), so someone else can check it was issued here and not written by the agent. `seal` says `sealed`, `off`, or `unsealed:` with why the key couldn't be used. ## Teach a write as a tool diff --git a/packages/flow/openadapt_flow/agent/cli.py b/packages/flow/openadapt_flow/agent/cli.py index 8761aa01c9..2714fd7861 100644 --- a/packages/flow/openadapt_flow/agent/cli.py +++ b/packages/flow/openadapt_flow/agent/cli.py @@ -226,7 +226,7 @@ def build_parser() -> argparse.ArgumentParser: metavar="CONTRACTS_DIR", help=( "Serve only the verify tools (verify_contracts, verify_begin, " - "verify_check): any agent calls them around its own writes, and gets " + "verify_check, verify_recheck): any agent calls them around its own writes, and gets " "a verified receipt or a refusal. CONTRACTS_DIR holds the write " "contracts (.yaml, openadapt.verify/v1)." ), diff --git a/packages/flow/openadapt_flow/agent/verify_tools.py b/packages/flow/openadapt_flow/agent/verify_tools.py index dfba24a97b..3849d8f244 100644 --- a/packages/flow/openadapt_flow/agent/verify_tools.py +++ b/packages/flow/openadapt_flow/agent/verify_tools.py @@ -18,7 +18,15 @@ shows a different write) or ``RECONCILIATION_REQUIRED`` (it doesn't show the write, which may still land); ``outcome`` says the same as ``CONFIRMED``, ``REFUTED`` or ``UNCERTAIN``. Never retry a refused write: - a person checks the record. + a person checks the record. When the contract sets + ``frame.database.recheck_after_s``, a ``CONFIRMED`` result also says + ``provisional: true`` and ``recheck_due_at``: a write that lands before + then can still refute it. +``verify_recheck`` + At or after ``recheck_due_at``. The server reads the whole database + again and returns ``CONFIRMED`` (the earlier verdict stands), ``REFUTED`` + (something moved since that nothing explains: the earlier ``CONFIRMED`` + no longer stands) or ``UNCERTAIN``. A write after this read is not seen. ``verify_begin`` consults the uncertain-write ledger, so a write whose outcome is unknown is refused, also after a restart or in another server. @@ -33,19 +41,24 @@ from typing import Any, Mapping, Optional from openadapt_flow.agent.bridge import BridgeError, ToolSpec +from openadapt_flow.runtime.effects.late_write import parse_time from openadapt_flow.verifier import ( Actor, AlreadyWritten, Contract, PendingWrite, + RecheckNotDue, + VerifierError, WriteBlocked, + WriteResult, begin_write, + recheck, verify_write, ) __all__ = ["VERIFY_TOOLS", "VerifyBridge", "contract_params"] -VERIFY_TOOLS = ("verify_contracts", "verify_begin", "verify_check") +VERIFY_TOOLS = ("verify_contracts", "verify_begin", "verify_check", "verify_recheck") _NAME = re.compile(r"^[a-z][a-z0-9_-]{0,63}$") _WRITE_KEY = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:@/-]{0,127}$") _AGENT_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:@-]{0,127}$") @@ -56,8 +69,11 @@ "the contract and the values you will save; if it refuses, don't write. " "Make the write your own way. Then call verify_check with the write_id. " "Only verified: true means the write is in the system of record as " - "declared. A refusal means a person must check the record: never retry " - "that write, and tell the person the verdict and reason." + "declared. If the result also says provisional: true, call " + "verify_recheck with the write_id at or after recheck_due_at: a write that " + "lands before then can still refute it. A refusal means a person must " + "check the record: never retry that write, and tell the person the " + "verdict and reason." ) @@ -96,6 +112,12 @@ def __init__(self, contracts_dir: Path, state_dir: Path) -> None: if not self.contracts: raise ValueError(f"no write contracts (*.yaml) in {self.contracts_dir}") self._pending: dict[str, PendingWrite] = {} + #: Provisional results whose recheck is still to run, by write_id. + #: Held in this server's memory only: after a restart, recheck from + #: Python with the saved ``recheck.json`` (``RecheckState.load``). + self._rechecks: dict[str, tuple[PendingWrite, WriteResult]] = {} + #: Every checked write with a recheck state, of any outcome. + self._checked: list[tuple[PendingWrite, WriteResult]] = [] self._lock = threading.Lock() # -- the MCP layer's interface ------------------------------------------ @@ -171,6 +193,23 @@ def list_tool_specs(self) -> list[ToolSpec]: ), annotations={"readOnlyHint": True, "openWorldHint": True}, ), + ToolSpec( + name="verify_recheck", + description=( + "Call at or after a provisional result's recheck_due_at. " + "Reads the whole database again: CONFIRMED (the verdict " + "stands), REFUTED (something moved after it that nothing " + "explains: the write is no longer verified) or UNCERTAIN. " + "Your later writes under the same contract, checked with " + "verify_check, account for their own rows; a write under " + "another contract does not, and refutes it. Before the due " + "time it returns due: false and the time to call again." + ), + input_schema=_obj( + {"write_id": {"type": "string", "maxLength": 64}}, ["write_id"] + ), + annotations={"readOnlyHint": True, "openWorldHint": True}, + ), ] def dispatch( @@ -191,6 +230,8 @@ def dispatch( return self._begin(args) if name == "verify_check": return self._check(args) + if name == "verify_recheck": + return self._recheck(args) raise BridgeError("unknown tool name") # -- tools ------------------------------------------------------------------ @@ -242,4 +283,121 @@ def _check(self, args: Mapping[str, Any]) -> dict[str, Any]: result = verify_write( entry, write_attempted=bool(args.get("write_attempted", True)) ) - return result.to_dict() + out = result.to_dict() + if result.recheck is not None: + # Any outcome: a later write's own window excuses its rows in an + # earlier write's recheck (``excused_by``). + with self._lock: + self._checked.append((entry, result)) + if result.recheck is not None and out.get("provisional"): + with self._lock: + self._rechecks[write_id] = (entry, result) + out["next"] = ( + "Provisional: call verify_recheck with this write_id at or after " + "recheck_due_at. A write that lands before then can still refute " + "it; a write after the recheck is not seen." + ) + return out + + def _later_writes( + self, entry: PendingWrite, result: WriteResult + ) -> list[WriteResult]: + """This session's checked writes under the same contract by the same + actor, begun after ``result``'s post-save read: each excuses only + what changed in its own window. A write under another contract, or + one begun before that read, excuses nothing.""" + + state = result.recheck + if state is None: + return [] + post = state.payload.get("post_state") or {} + try: + since = parse_time(str(post.get("read_at"))) + except ValueError: + return [] + out: list[WriteResult] = [] + for other_entry, other in self._checked: + if other is result or other_entry.contract is not entry.contract: + continue + if other_entry.actor != entry.actor or other.recheck is None: + continue + began = (other.recheck.payload.get("baseline") or {}).get( + "baseline_read_at" + ) + try: + if parse_time(str(began)) < since: + continue + except ValueError: + continue + out.append(other) + return out + + def _recheck(self, args: Mapping[str, Any]) -> dict[str, Any]: + write_id = str(args.get("write_id")) + with self._lock: + held = self._rechecks.get(write_id) + if held is None: + raise BridgeError( + "no provisional result for that write_id: only a CONFIRMED write " + "whose contract sets frame.database.recheck_after_s has a " + "recheck, each is rechecked once, and this server holds them in " + "memory only" + ) + entry, result = held + with self._lock: + unchecked = [ + p + for p in self._pending.values() + if p.contract is entry.contract and p.actor == entry.actor + ] + later = self._later_writes(entry, result) + if unchecked: + # Its rows would look unexplained: no verdict until it is checked. + raise BridgeError( + "a later write under this contract is not checked yet; call " + "verify_check for it, then verify_recheck again" + ) + try: + late = recheck( + entry.contract, + result, + params=entry.params, + actor=entry.actor, + excused_by=later, + ) + except RecheckNotDue as exc: + return { + "due": False, + "write_id": write_id, + "recheck_due_at": exc.due_at, + "wait_s": max(1, round(exc.wait_s)), + "next": "Not due yet: call verify_recheck again at or after " + "recheck_due_at.", + } + except VerifierError as exc: + with self._lock: + self._rechecks.pop(write_id, None) + raise BridgeError(f"the write can't be rechecked: {exc}") from exc + with self._lock: + self._rechecks.pop(write_id, None) + if late.verified: + nxt = "The verdict stands. A write after read_at is not seen." + elif late.late: + nxt = ( + "The earlier CONFIRMED no longer stands. Don't retry: a person " + "checks the record." + ) + else: + nxt = "Not a pass. Don't retry: a person checks the record." + return { + "due": True, + "write_id": write_id, + "verified": late.verified, + "outcome": late.outcome, + "late": late.late, + "reasons": list(late.reasons), + "recheck_due_at": late.due_at, + "read_at": late.read_at, + "receipt_sha256": late.receipt_sha256, + "next": nxt, + } diff --git a/packages/flow/openadapt_flow/verifier.py b/packages/flow/openadapt_flow/verifier.py index 6d447e82aa..2b2b41698f 100644 --- a/packages/flow/openadapt_flow/verifier.py +++ b/packages/flow/openadapt_flow/verifier.py @@ -560,6 +560,9 @@ class WriteResult: #: A person should look at the store: shown by ``openadapt verify ledger #: list`` until released, though it doesn't stop a retry being handed back. review: bool = False + #: The ledger the write was made under (``begin_write``'s), so + #: :func:`recheck` re-resolves the attempt there by default. + ledger: Optional[Ledger] = field(default=None, repr=False, compare=False) #: When the contract sets ``frame.database.recheck_after_s`` and the #: database settled after the save: the private state :func:`recheck` #: needs (saved as ``recheck.json`` in the write's folder). A confirmed @@ -575,6 +578,14 @@ class WriteResult: def verified(self) -> bool: return self.outcome is Outcome.CONFIRMED + @property + def provisional(self) -> bool: + """CONFIRMED, but a recheck is due (``recheck_due_at``): a write that + lands before it can still refute this result, and only the recheck + says the verdict stands. A write after the recheck is not seen.""" + + return self.verified and bool(self.recheck_due_at) + @property def verdict(self) -> str: """The outcome under its old name (``VERIFIED``, ``REFUTED``, @@ -619,6 +630,7 @@ def to_dict(self) -> dict[str, Any]: "landed": self.landed, "review": self.review, "recheck_due_at": self.recheck_due_at, + "provisional": self.provisional, } @@ -1199,6 +1211,10 @@ def verify_write( ) finally: pending.release() + if pending.ledger is not None: + import dataclasses + + result = dataclasses.replace(result, ledger=pending.ledger) return result @@ -1980,6 +1996,7 @@ def recheck( actor: Actor, excused_by: Sequence[WriteResult | RecheckState] = (), wait: bool = False, + ledger: Any = _NO_LEDGER, ) -> RecheckResult: """Read the whole database again and judge every change since a verified write's post-save read. @@ -2003,6 +2020,15 @@ def recheck( a loaded state needs its params (:meth:`RecheckState.load`). wait: Before the due time, sleep until it instead of raising :class:`RecheckNotDue`. + ledger: The ledger the write was made under (default: the one a + :class:`WriteResult` carries from :func:`begin_write`, else the + default ledger; ``None``: none). When the recheck doesn't + confirm the write, its attempt is resolved again there + (``REFUTED`` or ``UNCERTAIN``, for review), so a retry is held + for a person instead of being handed the earlier ``CONFIRMED``. + A saved :class:`RecheckState` finds its attempt by the receipt's + ``correlation_id``. When no attempt is re-resolved, ``reasons`` + says so. Changes the frame excuses after a save are excused here too: the app profile's bookkeeping, and another actor's write the audit log @@ -2012,8 +2038,117 @@ def recheck( ``UNCERTAIN``. Raises :class:`VerifierError` when ``write`` scheduled no recheck or an ``excused_by`` entry doesn't qualify, and :class:`RecheckNotDue` before the due time. + + A write that lands after the recheck's read (``read_at``) is not seen. + """ + + late = _judge_recheck( + contract, write, params=params, actor=actor, excused_by=excused_by, wait=wait + ) + if late.outcome != "CONFIRMED": + late = _hold_after_recheck(ledger, write, late) + return late + + +def _hold_after_recheck( + ledger: Any, write: WriteResult | RecheckState, late: RecheckResult +) -> RecheckResult: + """Resolve the write's ledger attempt again with the recheck's outcome, + for review: ``REFUTED`` holds it, ``UNCERTAIN`` keeps it uncertain. + Either way a retry is not handed the earlier ``CONFIRMED``. + + The attempt is the write's own: its ledger key and write id from a + :class:`WriteResult`, or, for a saved :class:`RecheckState`, the + receipt's ``correlation_id`` (the write id) looked up in the ledger. + The ledger is the one given, else the one the write was made under. + When no attempt is re-resolved (no ledger, no such attempt, or a newer + attempt of the key), the result says so: the write must be held by hand. """ + if isinstance(write, WriteResult): + write_id, key = write.write_id, write.write_key + digest = write.receipt_sha256 or None + if ledger is _NO_LEDGER and write.ledger is not None: + ledger = write.ledger + else: + receipt = write.payload.get("receipt") or {} + write_id = str(receipt.get("correlation_id") or "") + key = "" + digest = receipt_sha256(receipt) if receipt else None + + def unheld(why: str) -> RecheckResult: + return _replace( + late, + reasons=[ + *late.reasons, + f"no ledger attempt was re-resolved ({why}); hold this write by " + "hand so a retry isn't handed the earlier CONFIRMED", + ], + ) + + try: + led = _resolve_ledger(ledger) + if led is None: + return unheld("no ledger was given") + if not write_id: + return unheld("the write has no write id") + if not key: + found = [ + k + for k, begun, _ in led.attempts() + if str(begun.get("attempt")) == write_id + ] + if len(found) != 1: + return unheld("this ledger has no attempt with the write's id") + key = found[0] + attempt = led.latest(key) + if attempt is None: + return unheld("this ledger has no attempt with the write's id") + if attempt.attempt_id != write_id: + return unheld("a newer attempt of this write is in the ledger") + prior = attempt.resolved or {} + reason = "; ".join(late.reasons) or "the recheck did not confirm the write" + led.resolved( + key, + write_id, + outcome=late.outcome, + retry_safe=False, + landed=False, + review=True, + delivered=True, + via="recheck", + late=late.late, + write_key_bound=bool(prior.get("write_key_bound")), + receipt_sha256=digest, + result={ + "verified": False, + "outcome": late.outcome, + "late": late.late, + "reason": reason, + "receipt_sha256": digest, + }, + ) + except Exception as exc: # noqa: BLE001 - the verdict stands; say so + return _replace( + late, + reasons=[ + *late.reasons, + f"the ledger could not record it ({type(exc).__name__}); hold " + "this write by hand", + ], + ) + return late + + +def _judge_recheck( + contract: Contract, + write: WriteResult | RecheckState, + *, + params: Mapping[str, object], + actor: Actor, + excused_by: Sequence[WriteResult | RecheckState], + wait: bool, +) -> RecheckResult: state = write.recheck if isinstance(write, WriteResult) else write receipt = state.payload.get("receipt") if state is not None else None if ( diff --git a/packages/flow/tests/test_verifier_api.py b/packages/flow/tests/test_verifier_api.py index 2bdc4f6e5b..b00efe0683 100644 --- a/packages/flow/tests/test_verifier_api.py +++ b/packages/flow/tests/test_verifier_api.py @@ -351,6 +351,7 @@ def params(fn: Any) -> list[tuple[str, str]]: ("actor", "KEYWORD_ONLY"), ("excused_by", "KEYWORD_ONLY"), ("wait", "KEYWORD_ONLY"), + ("ledger", "KEYWORD_ONLY"), ] assert [f for f in RecheckResult.__dataclass_fields__] == [ "outcome", diff --git a/packages/flow/tests/test_verifier_recheck.py b/packages/flow/tests/test_verifier_recheck.py index ab82b6e188..2849e7e631 100644 --- a/packages/flow/tests/test_verifier_recheck.py +++ b/packages/flow/tests/test_verifier_recheck.py @@ -803,3 +803,138 @@ def test_settle_attempts_at_its_default_stays_out_of_the_digest() -> None: ) assert plain.digest() == explicit.digest() assert "settle_attempts" not in json.dumps(plain.model_dump()) + + +# -- a provisional result, and the ledger after the recheck ---------------------- + + +def test_a_confirmed_result_with_a_recheck_due_is_provisional(tmp_path: Path) -> None: + _, _, result = _first(tmp_path) + assert result.verified and result.provisional + assert result.to_dict()["provisional"] is True + (tmp_path / "plain").mkdir() + db = _db(tmp_path / "plain") + plain = _save(Contract.from_dict(_doc(db, recheck_after_s=None)), db, 2) + assert plain.verified and not plain.provisional + assert plain.to_dict()["provisional"] is False + + +def _keyed(tmp_path: Path, key: str) -> tuple[Contract, Path, WriteResult]: + db = _db(tmp_path) + contract = Contract.from_dict(_doc(db)) + before = begin_write(contract, _params(2), actor=BOT, write_key=key) + _sql(db, *_save_sql(2)) + result = verify_write(before) + assert result.verified and result.provisional, result.reasons + return contract, db, result + + +def test_a_late_refutation_holds_the_write_in_the_ledger( + tmp_path: Path, due_now: None +) -> None: + contract, db, result = _keyed(tmp_path, "order-77") + _sql(db, "INSERT INTO calendar (pid, title) VALUES (2, 'New Patient')") + late = _recheck(contract, result) + assert late.outcome == "REFUTED" + # A retry under the same key is held for a person, never handed the + # earlier CONFIRMED. + with pytest.raises(verifier.WriteHeld): + begin_write(contract, _params(2), actor=BOT, write_key="order-77") + from openadapt_flow.write_ledger import default_ledger + + rows = dict(default_ledger().unresolved()) + assert rows[result.write_key].resolved["via"] == "recheck" + assert rows[result.write_key].resolved["late"] is True + + +def test_an_uncertain_recheck_is_not_handed_back_as_confirmed( + tmp_path: Path, due_now: None +) -> None: + contract, db, result = _keyed(tmp_path, "order-78") + db.unlink() # the database can't be read again + late = _recheck(contract, result) + assert late.outcome == "UNCERTAIN" + with pytest.raises(verifier.WriteBlocked) as raised: + begin_write(contract, _params(2), actor=BOT, write_key="order-78") + assert not isinstance(raised.value, verifier.AlreadyWritten) + + +def test_a_confirming_recheck_leaves_the_ledger_as_it_was( + tmp_path: Path, due_now: None +) -> None: + contract, _, result = _keyed(tmp_path, "order-79") + assert _recheck(contract, result).outcome == "CONFIRMED" + with pytest.raises(verifier.AlreadyWritten) as raised: + begin_write(contract, _params(2), actor=BOT, write_key="order-79") + assert raised.value.result["outcome"] == "CONFIRMED" + + +def test_a_recheck_without_a_ledger_changes_no_ledger( + tmp_path: Path, due_now: None +) -> None: + contract, db, result = _keyed(tmp_path, "order-80") + _sql(db, "INSERT INTO calendar (pid, title) VALUES (2, 'New Patient')") + late = _recheck(contract, result, ledger=None) + assert late.outcome == "REFUTED" + # Nothing was held, and the result says so. + assert "no ledger attempt was re-resolved (no ledger was given)" in late.reasons[-1] + with pytest.raises(verifier.AlreadyWritten): + begin_write(contract, _params(2), actor=BOT, write_key="order-80") + + +def test_a_recheck_from_a_saved_state_holds_the_write( + tmp_path: Path, due_now: None +) -> None: + # The reviewer's probe: after a restart, the recheck runs from the saved + # recheck.json, which carries no ledger key. + contract, db, result = _keyed(tmp_path, "k-a1") + _sql(db, "INSERT INTO calendar (pid, title) VALUES (2, 'New Patient')") + assert result.recheck is not None and result.recheck.path is not None + saved = RecheckState.load(result.recheck.path, params=_params(2)) + late = _recheck(contract, saved) + assert late.outcome == "REFUTED" + assert not any("re-resolved" in r for r in late.reasons) + with pytest.raises(verifier.WriteHeld): + begin_write(contract, _params(2), actor=BOT, write_key="k-a1") + + +def test_a_saved_state_whose_attempt_is_not_in_the_ledger_says_so( + tmp_path: Path, due_now: None +) -> None: + contract, db, result = _keyed(tmp_path, "k-a2") + _sql(db, "INSERT INTO calendar (pid, title) VALUES (2, 'New Patient')") + saved = RecheckState.load(result.recheck.path, params=_params(2)) + other = verifier.Ledger(verifier.MemoryLedger()) + late = _recheck(contract, saved, ledger=other) + assert late.outcome == "REFUTED" + assert "this ledger has no attempt with the write's id" in late.reasons[-1] + + +def test_the_write_carries_its_own_ledger(tmp_path: Path, due_now: None) -> None: + # The reviewer's probe A2: begun under a custom ledger, rechecked without + # naming it. The recheck re-resolves the attempt there. + db = _db(tmp_path) + contract = Contract.from_dict(_doc(db)) + custom = verifier.Ledger(verifier.MemoryLedger()) + before = begin_write( + contract, _params(2), actor=BOT, ledger=custom, write_key="k-c" + ) + _sql(db, *_save_sql(2)) + result = verify_write(before) + assert result.provisional and result.ledger is custom + _sql(db, "INSERT INTO calendar (pid, title) VALUES (2, 'New Patient')") + assert _recheck(contract, result).outcome == "REFUTED" + with pytest.raises(verifier.WriteHeld): + begin_write(contract, _params(2), actor=BOT, ledger=custom, write_key="k-c") + + +def test_a_wrong_ledger_says_nothing_was_held(tmp_path: Path, due_now: None) -> None: + contract, db, result = _keyed(tmp_path, "k-w") + _sql(db, "INSERT INTO calendar (pid, title) VALUES (2, 'New Patient')") + wrong = verifier.Ledger(verifier.MemoryLedger()) + late = _recheck(contract, result, ledger=wrong) + assert late.outcome == "REFUTED" + assert ( + "re-resolved" in late.reasons[-1] + and "hold this write by hand" in late.reasons[-1] + ) diff --git a/packages/flow/tests/test_verify_tools.py b/packages/flow/tests/test_verify_tools.py index 6e5aef7167..a681cb68b7 100644 --- a/packages/flow/tests/test_verify_tools.py +++ b/packages/flow/tests/test_verify_tools.py @@ -8,6 +8,7 @@ from __future__ import annotations import json +import sqlite3 from pathlib import Path from typing import Any @@ -20,6 +21,8 @@ from mcp import ClientSession # noqa: E402 from mcp.shared.memory import create_client_server_memory_streams # noqa: E402 +from openadapt_flow import verifier # noqa: E402 +from openadapt_flow.agent.bridge import BridgeError # noqa: E402 from openadapt_flow.agent.cli import main as agent_main # noqa: E402 from openadapt_flow.agent.mcp import build_server # noqa: E402 from openadapt_flow.agent.verify_tools import VerifyBridge # noqa: E402 @@ -144,3 +147,194 @@ def test_serve_verify_refuses_other_modes(tmp_path: Path, capsys) -> None: assert "--verify serves only the verify tools" in capsys.readouterr().err assert agent_main(["serve", "--verify", str(tmp_path)]) == 2 assert "no write contracts" in capsys.readouterr().err + + +# -- a provisional result and verify_recheck (SQLite, made-up records) ---------- + +RECHECK_SCHEMA = """ +CREATE TABLE patient_data (id INTEGER PRIMARY KEY, fname TEXT, lname TEXT, + email TEXT); +CREATE TABLE calendar (id INTEGER PRIMARY KEY AUTOINCREMENT, pid INTEGER, title TEXT); +INSERT INTO patient_data VALUES (1, 'Wendell', 'Prior', 'w@example.test'); +""" + +VALUES = {"fname": "Ada", "lname": "Quill", "email": "ada@example.test"} +SAVE = ( + "INSERT INTO patient_data (id, fname, lname, email) " + "VALUES (2, 'Ada', 'Quill', 'ada@example.test')" +) + + +def _sql(db: Path, *statements: str) -> None: + con = sqlite3.connect(db) + for statement in statements: + con.execute(statement) + con.commit() + con.close() + + +@pytest.fixture +def sqlite_bridge( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> tuple[VerifyBridge, Path]: + monkeypatch.setenv("OPENADAPT_LEDGER_DIR", str(tmp_path / "ledger")) + monkeypatch.setattr(verifier, "USED_DIR", tmp_path / "used") + db = tmp_path / "app.sqlite" + con = sqlite3.connect(db) + con.executescript(RECHECK_SCHEMA) + con.commit() + con.close() + contracts = tmp_path / "contracts" + contracts.mkdir() + doc = { + "schema_version": "openadapt.verify/v1", + "verifier": { + "kind": "sql", + "sqlite_database": str(db), + "sql_query": "SELECT * FROM patient_data", + }, + "record": { + "name": "patient", + "match": ["email", "lname"], + "fields": {"fname": "fname", "lname": "lname", "email": "email"}, + }, + "frame": { + "identity_field": "id", + "database": { + "dialect": "sqlite", + "read": {"sqlite_database": str(db)}, + "views": {"patient_data": ""}, + "settle_s": 0, + "recheck_after_s": 60, + }, + }, + } + (contracts / "patient.yaml").write_text(yaml.safe_dump(doc)) + return VerifyBridge(contracts, tmp_path / "state"), db + + +def _provisional(bridge: VerifyBridge, db: Path) -> dict[str, Any]: + begun = bridge.dispatch("verify_begin", {"contract": "patient", "values": VALUES}) + _sql(db, SAVE) + checked = bridge.dispatch("verify_check", {"write_id": begun["write_id"]}) + assert checked["verified"] is True and checked["provisional"] is True + assert checked["recheck_due_at"] and "verify_recheck" in checked["next"] + return checked + + +def test_verify_recheck_is_listed_and_waits_for_its_due_time(sqlite_bridge) -> None: + bridge, db = sqlite_bridge + assert "verify_recheck" in [t.name for t in bridge.list_tool_specs()] + assert "verify_recheck" in bridge.instructions + checked = _provisional(bridge, db) + early = bridge.dispatch("verify_recheck", {"write_id": checked["write_id"]}) + assert early["due"] is False + assert early["recheck_due_at"] == checked["recheck_due_at"] + assert early["wait_s"] > 30 + + +def test_verify_recheck_confirms_then_is_spent(sqlite_bridge, monkeypatch) -> None: + bridge, db = sqlite_bridge + checked = _provisional(bridge, db) + monkeypatch.setattr(verifier, "seconds_until", lambda due: 0.0) + late = bridge.dispatch("verify_recheck", {"write_id": checked["write_id"]}) + assert late["due"] is True and late["outcome"] == "CONFIRMED" + assert late["verified"] is True and late["late"] is False + assert late["receipt_sha256"] == checked["receipt_sha256"] + with pytest.raises(BridgeError, match="no provisional result"): + bridge.dispatch("verify_recheck", {"write_id": checked["write_id"]}) + + +def test_verify_recheck_refutes_a_write_that_landed_late( + sqlite_bridge, monkeypatch +) -> None: + bridge, db = sqlite_bridge + checked = _provisional(bridge, db) + # After the verdict, before the recheck: a background job adds a row. + _sql(db, "INSERT INTO calendar (pid, title) VALUES (2, 'New Patient')") + monkeypatch.setattr(verifier, "seconds_until", lambda due: 0.0) + late = bridge.dispatch("verify_recheck", {"write_id": checked["write_id"]}) + assert late["outcome"] == "REFUTED" and late["late"] is True + assert late["verified"] is False and "no longer stands" in late["next"] + # The same write again is held for a person, not handed the CONFIRMED. + again = bridge.dispatch("verify_begin", {"contract": "patient", "values": VALUES}) + assert again["ok_to_write"] is False and "earlier_result" not in again + + +VALUES_3 = {"fname": "Bo", "lname": "Quill", "email": "bo@example.test"} +SAVE_3 = ( + "INSERT INTO patient_data (id, fname, lname, email) " + "VALUES (3, 'Bo', 'Quill', 'bo@example.test')" +) + + +def _second(bridge: VerifyBridge, db: Path, contract: str = "patient") -> dict: + begun = bridge.dispatch("verify_begin", {"contract": contract, "values": VALUES_3}) + _sql(db, SAVE_3) + return bridge.dispatch("verify_check", {"write_id": begun["write_id"]}) + + +def test_the_agents_next_write_in_the_window_does_not_refute_the_first( + sqlite_bridge, monkeypatch +) -> None: + # The reviewer's probe: two correct writes by one agent under one + # contract inside the window. Each recheck excuses the other's rows. + bridge, db = sqlite_bridge + first = _provisional(bridge, db) + second = _second(bridge, db) + assert second["provisional"] is True + monkeypatch.setattr(verifier, "seconds_until", lambda due: 0.0) + one = bridge.dispatch("verify_recheck", {"write_id": first["write_id"]}) + two = bridge.dispatch("verify_recheck", {"write_id": second["write_id"]}) + assert one["outcome"] == "CONFIRMED", one["reasons"] + assert two["outcome"] == "CONFIRMED", two["reasons"] + retry = bridge.dispatch("verify_begin", {"contract": "patient", "values": VALUES}) + assert retry["ok_to_write"] is False + assert retry["earlier_result"]["outcome"] == "CONFIRMED" + + +def test_a_late_row_after_the_next_write_still_refutes_the_first( + sqlite_bridge, monkeypatch +) -> None: + bridge, db = sqlite_bridge + first = _provisional(bridge, db) + _second(bridge, db) + _sql(db, "INSERT INTO calendar (pid, title) VALUES (2, 'New Patient')") + monkeypatch.setattr(verifier, "seconds_until", lambda due: 0.0) + one = bridge.dispatch("verify_recheck", {"write_id": first["write_id"]}) + assert one["outcome"] == "REFUTED" + + +def test_a_write_under_another_contract_is_not_excused( + sqlite_bridge, monkeypatch, tmp_path +) -> None: + bridge, db = sqlite_bridge + # The same read under another name: another contract excuses nothing. + doc = yaml.safe_load((tmp_path / "contracts" / "patient.yaml").read_text()) + bridge.contracts["intake"] = verifier.Contract.from_dict(doc, name="intake") + first = _provisional(bridge, db) + _second(bridge, db, contract="intake") + monkeypatch.setattr(verifier, "seconds_until", lambda due: 0.0) + one = bridge.dispatch("verify_recheck", {"write_id": first["write_id"]}) + assert one["outcome"] == "REFUTED" + + +def test_a_recheck_waits_for_a_later_write_still_unchecked( + sqlite_bridge, monkeypatch +) -> None: + bridge, db = sqlite_bridge + first = _provisional(bridge, db) + begun = bridge.dispatch("verify_begin", {"contract": "patient", "values": VALUES_3}) + _sql(db, SAVE_3) + monkeypatch.setattr(verifier, "seconds_until", lambda due: 0.0) + with pytest.raises(BridgeError, match="not checked yet"): + bridge.dispatch("verify_recheck", {"write_id": first["write_id"]}) + bridge.dispatch("verify_check", {"write_id": begun["write_id"]}) + one = bridge.dispatch("verify_recheck", {"write_id": first["write_id"]}) + assert one["outcome"] == "CONFIRMED", one["reasons"] + + +def test_a_write_without_a_recheck_has_none_to_run(sqlite_bridge) -> None: + bridge, _ = sqlite_bridge + with pytest.raises(BridgeError, match="no provisional result"): + bridge.dispatch("verify_recheck", {"write_id": "nope"})