diff --git a/CAPABILITIES.md b/CAPABILITIES.md index 7bf1771e1e..f23ae086bf 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -147,6 +147,7 @@ improvising or retrying when something is off. | Judge an "ensure the record exists" write (found, created once with every value, or needs review) through a FHIR server's read path only: searches, AuditEvent and `_history`, never the database; the coverage claim names that read path. An agent's own correction of the record it created counts as one write when the audit trail and history show only that agent, no other reader, and the same record in every version | On main ([#1640](https://github.com/OpenAdaptAI/OpenAdapt/pull/1640), corrections [#1655](https://github.com/OpenAdaptAI/OpenAdapt/pull/1655)) | Real app (Medplum 5.2.8; three blind fault suites: 17/18, 17/20, 19/20; no fault passed in the last two. Corrections, on 15 blind cases and the 37 public ones: false stops 8 to 2 and 9 to 4 for Claude, no harm passed that the old rule caught) | `openadapt_flow/ensure_record.py`, [ENSURE_RECORD.md](packages/flow/docs/ENSURE_RECORD.md) | | Generate an ensure-record contract from three demonstrations, the server's CapabilityStatement and its SearchParameter definitions: field paths and `[system=…]` filters, comparison kinds, value maps, identity, search parameters, the closed record and the change feed. It refuses when unsure, asks a person to confirm an identity for a type with no reviewed template, and self-checks every demonstration plus a perturbation per field | On main ([#1653](https://github.com/OpenAdaptAI/OpenAdapt/pull/1653)) | Real app (Medplum 5.2.8: all 16 hand-written Patient lines regenerated; one blind write type, Immunization: 78 s of person time, 2 lines edited after the person declined the proposed identity; blind suite 23/24, no fault passed) | `openadapt_flow/generate/ensure_contract.py`, [RESULTS.md](packages/flow/benchmark/contract_autogen/results/RESULTS.md) | | A stop card for a stopped write: the exact record difference and who else wrote, with Undo or Fix that reads back, or Open in the app; cost modelled from published operators and timed machine steps, as pre-registered | On main ([#1653](https://github.com/OpenAdaptAI/OpenAdapt/pull/1653)) | Real app (Medplum 5.2.8, blind suite: 16 stops, 14 one-tap with no failed read-back; modelled 42 s per caught harm against 420; one wrong category on a false stop, gone in the re-measure with `void`: 24/24, 15 stops, no wrong category, no failed read-back) | `benchmark/stop_card/`, [PREREG_REVIEW.md](packages/flow/benchmark/stop_card/PREREG_REVIEW.md) | +| Generate `openadapt.verify/v1` contracts from a MySQL, MariaDB or PostgreSQL schema, the kinds of write with their parameter names, and one demonstration write per example: one contract per field set (an update gets one for each set of fields it may change), list parameters by length, a `found` twin for a record already in place, the whole-database frame on every dialect, rows the app adds on its own declared by count, a recheck window the person confirms, and `frame.versions` (a lost update) from a version column or the app's audit trail of saves. Every undecided point is one numbered question; every contract is loaded and dry-read against the live store before it is written; a needed `TRIGGER` grant or view is applied with a flag or the kind is refused. Not yet: OpenAPI metadata | [#1672](https://github.com/OpenAdaptAI/OpenAdapt/pull/1672) | Real app (first study, instance copies; questions for a person: Dolibarr 4, Redmine 2, EspoCRM 3, against 11 hand decisions each; every in-scope planted fault stopped (22/22, 19/19, 21/21); correct writes confirmed 32/43, 32/38, 27/40; evidence in the private study repository) | `python -m openadapt_flow.generate.verify_contract`, [GENERATE_CONTRACTS.md](packages/flow/docs/GENERATE_CONTRACTS.md) | | `void` in an ensure-record contract: values that make a record assert nothing (`status="entered-in-error"`). A record holding one is never the person, is not a duplicate, and a created one needs review; it names values to leave out, never values to keep, and can't name an identity field | On main ([#1657](https://github.com/OpenAdaptAI/OpenAdapt/pull/1657)) | Real app (Medplum 5.2.8: 8 live cases written in advance, 8/8 with the line, 5/8 and one silent pass without it; frozen blind Immunization suite 24/24); adversarial review recorded | `openadapt_flow/ensure_record.py`, [RESULTS.md](packages/flow/benchmark/fhir_medplum/void/results/RESULTS.md), [REVIEW_VOID.md](packages/flow/benchmark/fhir_medplum/void/REVIEW_VOID.md) | | Check a patient registration on a commercial EHR through its public FHIR sandbox: Epic (open.epic, R4), read through FHIR search and read with SMART backend auth; the frame is the search set, since the sandbox has no AuditEvent or `_history` | On main ([#1656](https://github.com/OpenAdaptAI/OpenAdapt/pull/1656)) | Real app (Epic public sandbox, August 2026 release; blind round 12/13 with none passed as verified, the miss a false refusal fixed in the contract; caught Epic dropping the phone on a 201 create) | `benchmark/commercial_ehr/`, [FINDINGS.md](packages/flow/benchmark/commercial_ehr/FINDINGS.md) | | A public benchmark a vendor reruns with their own agent: `openadapt bench medplum --agent ` runs 37 written-for-it cases (Patient, AllergyIntolerance, Appointment) against a local Medplum through a fault proxy, and scores the agent's claim, a read-back and OpenAdapt's verifier against an end-state oracle | On main ([#1652](https://github.com/OpenAdaptAI/OpenAdapt/pull/1652)) | Real app (Medplum 5.2.8; Claude Sonnet 5.5: OpenAdapt missed 1 of 9 harms with 4 false stops, all corrections that changed the record's identity, against 8 missed by the claim and the read-back; cold run from clone to scorecard 5 min) | [README](packages/flow/benchmark/fhir_medplum/public/README.md), [RESULTS.md](packages/flow/benchmark/fhir_medplum/public/RESULTS.md) | @@ -155,8 +156,8 @@ improvising or retrying when something is off. | Check that the saved record matches its source document, not only the parameters: intake keeps each value's line, a second reading must agree before a guarded save, and a round trip compares the read-back with the document | On main | Synthetic (MockMed, text referral) | `openadapt_flow/provenance.py`, `intake`, `verify-run --provenance`, guarded save | | Check one write from Python, without the CLI: one API (`openadapt_flow.verifier`: `begin_write`, `verify_write`; 2.4.1's `capture` form still works, deprecated) with one outcome (`CONFIRMED`, `REFUTED`, `UNCERTAIN`), the receipt, which parts were checked, and the result's scope (read path, stores read, time window, what it doesn't cover) | On main | Synthetic (SQLite; MockMed's fault server over real HTTP) | `openadapt_flow/verifier.py`, [VERIFIER_API.md](packages/flow/docs/VERIFIER_API.md) | | Never send a write twice while its outcome is unknown: an append-only ledger on local disk (pluggable backend), keyed by the write's identity, that every adapter, the MCP tools and the CLI consult before sending; a read-back against the attempt's own baseline resolves it, and a refuted write waits for a person (`openadapt verify ledger list`, `reconcile`, `release`) | On main | Synthetic (MockMed's fault server: a worker killed between the send and the read-back, killed during a timed-out send, killed before the send, a late write, and two worker processes started together on one write) | `openadapt_flow/write_ledger.py`, [VERIFIER_API.md](packages/flow/docs/VERIFIER_API.md#the-uncertain-write-ledger) | -| Catch a write that lands after the verdict (a background job, a trigger, a delayed side record): a `CONFIRMED` with a recheck due is provisional (`provisional`, `recheck_due_at`); the recheck reads the whole database again and refutes the earlier result when a change since the post-save read has no explanation, and a recheck that doesn't confirm holds the write in the ledger. Run it from Python (`verifier.recheck`) or MCP (`verify_recheck`). A write after the recheck's read is not seen (`frame.database.recheck_after_s`, no default window) | On main ([#1639](https://github.com/OpenAdaptAI/OpenAdapt/pull/1639)); provisional result, MCP tool and ledger hold ([#1669](https://github.com/OpenAdaptAI/OpenAdapt/pull/1669)) | Real app (Dolibarr 21 and EspoCRM 10 on MariaDB 11.4, 180 s window: 4 of 4 planted late writes at 125 to 130 s refuted at the recheck, correct writes confirmed; EspoCRM's cron daemon needs its job tables declared); synthetic (SQLite); opt-in live OpenEMR test | `openadapt_flow/verifier.py`, `runtime/effects/late_write.py`, [VERIFY_RUN.md](packages/flow/docs/VERIFY_RUN.md#writes-that-land-later) | -| Diff the whole database for the frame (`frame.database`) on SQLite, MySQL and MariaDB. On MySQL and MariaDB every column type is read exactly (binary strings, `BIT`, geometries and vectors as hex, `FLOAT` widened to `DOUBLE`, `JSON` as stored text, decimals at full precision), so a row the save adds, changes or removes outside the declared records, such as an extra planned call or a stream note, is refuted unless a declared effect, a side record, an attributed concurrent write or an allow-listed bookkeeping table explains it | On main ([#1661](https://github.com/OpenAdaptAI/OpenAdapt/pull/1661)) | Real app (Dolibarr 21 on MariaDB 11.4: all 387 tables read where the frame raised `TypeError`; EspoCRM on MariaDB 11.4: the first-contact study's extra call and stream note refuted, correct opportunities verified); live MariaDB type tests | [VERIFY_RUN.md](packages/flow/docs/VERIFY_RUN.md#the-whole-database), `runtime/effects/db_snapshot.py` | +| Catch a write that lands after the verdict (a background job, a trigger, a delayed side record): a `CONFIRMED` with a recheck due is provisional (`provisional`, `recheck_due_at`); the recheck reads the whole database again and refutes the earlier result when a change since the post-save read has no explanation, and a recheck that doesn't confirm holds the write in the ledger. Run it from Python (`verifier.recheck`) or MCP (`verify_recheck`). A write after the recheck's read is not seen (`frame.database.recheck_after_s`, no default window) | On main ([#1639](https://github.com/OpenAdaptAI/OpenAdapt/pull/1639)); provisional result, MCP tool and ledger hold ([#1669](https://github.com/OpenAdaptAI/OpenAdapt/pull/1669)) | Real app (Dolibarr 21 and EspoCRM 10 on MariaDB 11.4, 180 s window: 4 of 4 planted late writes at 125 to 130 s refuted at the recheck, correct writes confirmed; EspoCRM's cron daemon needs its job tables declared; Redmine 6.1 on PostgreSQL 17, 180 s window: the first-contact study's late writes F19 and F20 refuted at the recheck, correct writes confirmed); synthetic (SQLite); opt-in live OpenEMR test | `openadapt_flow/verifier.py`, `runtime/effects/late_write.py`, [VERIFY_RUN.md](packages/flow/docs/VERIFY_RUN.md#writes-that-land-later) | +| Diff the whole database for the frame (`frame.database`) on SQLite, MySQL, MariaDB and PostgreSQL. On MySQL and MariaDB every column type is read exactly (binary strings, `BIT`, geometries and vectors as hex, `FLOAT` widened to `DOUBLE`, `JSON` as stored text, decimals at full precision). On PostgreSQL every column type is read as text by a declared rule in a session pinned to UTC and exact float output, and a type without a rule, a table the account can't read in full, or an unreadable sequence is refused. So a row the save adds, changes or removes outside the declared records, such as an extra planned call or a stream note, is refuted unless a declared effect, a side record, an attributed concurrent write or an allow-listed bookkeeping table explains it | On main ([#1661](https://github.com/OpenAdaptAI/OpenAdapt/pull/1661)); PostgreSQL ([#1673](https://github.com/OpenAdaptAI/OpenAdapt/pull/1673)) | Real app (Dolibarr 21 on MariaDB 11.4: all 387 tables read where the frame raised `TypeError`; EspoCRM on MariaDB 11.4: the first-contact study's extra call and stream note refuted, correct opportunities verified; Redmine 6.1 on PostgreSQL 17: 58 tables read, a stray upload in a table the contract doesn't read refuted, correct writes verified); live MariaDB and PostgreSQL type tests | [VERIFY_RUN.md](packages/flow/docs/VERIFY_RUN.md#the-whole-database), `runtime/effects/db_snapshot.py` | | Catch a lost update: another user saves a change to the record while the agent works, and the agent saves its stale copy over it. Before and after look like a correct update, so `frame.versions` reads the target's versions (FHIR or SQL version history, a version column, or the app's audit or action-history table) before and after the write. A version by another author, or more than one version step when no author is read, refutes the write; a version that can't be read or attributed stops it | On main ([#1662](https://github.com/OpenAdaptAI/OpenAdapt/pull/1662)) | Real app (HAPI FHIR and EspoCRM, first-contact study fault scripts: both lost updates refuted, correct updates verified, every other case's verdict unchanged) | [VERIFY_RUN.md](packages/flow/docs/VERIFY_RUN.md#lost-updates), `runtime/effects/versions.py` | | Guarded save: a safety layer any agent wraps around one write (check the record, hold for approval, read back) | On main ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)) | Synthetic | `GUARDED_SAVE.md` | | A person's approval in Desktop is signed with a key kept behind Touch ID on a Mac: one-prompt setup in the app, a prompt for every answer, and Flow never sees the key | On main ([#1500](https://github.com/OpenAdaptAI/OpenAdapt/pull/1500)) | Unit tests; opt-in test on a real Secure Enclave | `packages/desktop/openadapt_desktop/approver_keychain.py`, [APPROVAL_MODES.md](packages/flow/docs/APPROVAL_MODES.md#the-approver-key) | diff --git a/llms-full.txt b/llms-full.txt index e060cc35ed..64d75e9fde 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -544,6 +544,7 @@ improvising or retrying when something is off. | Judge an "ensure the record exists" write (found, created once with every value, or needs review) through a FHIR server's read path only: searches, AuditEvent and `_history`, never the database; the coverage claim names that read path. An agent's own correction of the record it created counts as one write when the audit trail and history show only that agent, no other reader, and the same record in every version | On main ([#1640](https://github.com/OpenAdaptAI/OpenAdapt/pull/1640), corrections [#1655](https://github.com/OpenAdaptAI/OpenAdapt/pull/1655)) | Real app (Medplum 5.2.8; three blind fault suites: 17/18, 17/20, 19/20; no fault passed in the last two. Corrections, on 15 blind cases and the 37 public ones: false stops 8 to 2 and 9 to 4 for Claude, no harm passed that the old rule caught) | `openadapt_flow/ensure_record.py`, [ENSURE_RECORD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/ENSURE_RECORD.md) | | Generate an ensure-record contract from three demonstrations, the server's CapabilityStatement and its SearchParameter definitions: field paths and `[system=…]` filters, comparison kinds, value maps, identity, search parameters, the closed record and the change feed. It refuses when unsure, asks a person to confirm an identity for a type with no reviewed template, and self-checks every demonstration plus a perturbation per field | On main ([#1653](https://github.com/OpenAdaptAI/OpenAdapt/pull/1653)) | Real app (Medplum 5.2.8: all 16 hand-written Patient lines regenerated; one blind write type, Immunization: 78 s of person time, 2 lines edited after the person declined the proposed identity; blind suite 23/24, no fault passed) | `openadapt_flow/generate/ensure_contract.py`, [RESULTS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/contract_autogen/results/RESULTS.md) | | A stop card for a stopped write: the exact record difference and who else wrote, with Undo or Fix that reads back, or Open in the app; cost modelled from published operators and timed machine steps, as pre-registered | On main ([#1653](https://github.com/OpenAdaptAI/OpenAdapt/pull/1653)) | Real app (Medplum 5.2.8, blind suite: 16 stops, 14 one-tap with no failed read-back; modelled 42 s per caught harm against 420; one wrong category on a false stop, gone in the re-measure with `void`: 24/24, 15 stops, no wrong category, no failed read-back) | `benchmark/stop_card/`, [PREREG_REVIEW.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/stop_card/PREREG_REVIEW.md) | +| Generate `openadapt.verify/v1` contracts from a MySQL, MariaDB or PostgreSQL schema, the kinds of write with their parameter names, and one demonstration write per example: one contract per field set (an update gets one for each set of fields it may change), list parameters by length, a `found` twin for a record already in place, the whole-database frame on every dialect, rows the app adds on its own declared by count, a recheck window the person confirms, and `frame.versions` (a lost update) from a version column or the app's audit trail of saves. Every undecided point is one numbered question; every contract is loaded and dry-read against the live store before it is written; a needed `TRIGGER` grant or view is applied with a flag or the kind is refused. Not yet: OpenAPI metadata | [#1672](https://github.com/OpenAdaptAI/OpenAdapt/pull/1672) | Real app (first study, instance copies; questions for a person: Dolibarr 4, Redmine 2, EspoCRM 3, against 11 hand decisions each; every in-scope planted fault stopped (22/22, 19/19, 21/21); correct writes confirmed 32/43, 32/38, 27/40; evidence in the private study repository) | `python -m openadapt_flow.generate.verify_contract`, [GENERATE_CONTRACTS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/GENERATE_CONTRACTS.md) | | `void` in an ensure-record contract: values that make a record assert nothing (`status="entered-in-error"`). A record holding one is never the person, is not a duplicate, and a created one needs review; it names values to leave out, never values to keep, and can't name an identity field | On main ([#1657](https://github.com/OpenAdaptAI/OpenAdapt/pull/1657)) | Real app (Medplum 5.2.8: 8 live cases written in advance, 8/8 with the line, 5/8 and one silent pass without it; frozen blind Immunization suite 24/24); adversarial review recorded | `openadapt_flow/ensure_record.py`, [RESULTS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/void/results/RESULTS.md), [REVIEW_VOID.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/void/REVIEW_VOID.md) | | Check a patient registration on a commercial EHR through its public FHIR sandbox: Epic (open.epic, R4), read through FHIR search and read with SMART backend auth; the frame is the search set, since the sandbox has no AuditEvent or `_history` | On main ([#1656](https://github.com/OpenAdaptAI/OpenAdapt/pull/1656)) | Real app (Epic public sandbox, August 2026 release; blind round 12/13 with none passed as verified, the miss a false refusal fixed in the contract; caught Epic dropping the phone on a 201 create) | `benchmark/commercial_ehr/`, [FINDINGS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/commercial_ehr/FINDINGS.md) | | A public benchmark a vendor reruns with their own agent: `openadapt bench medplum --agent ` runs 37 written-for-it cases (Patient, AllergyIntolerance, Appointment) against a local Medplum through a fault proxy, and scores the agent's claim, a read-back and OpenAdapt's verifier against an end-state oracle | On main ([#1652](https://github.com/OpenAdaptAI/OpenAdapt/pull/1652)) | Real app (Medplum 5.2.8; Claude Sonnet 5.5: OpenAdapt missed 1 of 9 harms with 4 false stops, all corrections that changed the record's identity, against 8 missed by the claim and the read-back; cold run from clone to scorecard 5 min) | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/public/README.md), [RESULTS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/public/RESULTS.md) | @@ -552,8 +553,8 @@ improvising or retrying when something is off. | Check that the saved record matches its source document, not only the parameters: intake keeps each value's line, a second reading must agree before a guarded save, and a round trip compares the read-back with the document | On main | Synthetic (MockMed, text referral) | `openadapt_flow/provenance.py`, `intake`, `verify-run --provenance`, guarded save | | Check one write from Python, without the CLI: one API (`openadapt_flow.verifier`: `begin_write`, `verify_write`; 2.4.1's `capture` form still works, deprecated) with one outcome (`CONFIRMED`, `REFUTED`, `UNCERTAIN`), the receipt, which parts were checked, and the result's scope (read path, stores read, time window, what it doesn't cover) | On main | Synthetic (SQLite; MockMed's fault server over real HTTP) | `openadapt_flow/verifier.py`, [VERIFIER_API.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFIER_API.md) | | Never send a write twice while its outcome is unknown: an append-only ledger on local disk (pluggable backend), keyed by the write's identity, that every adapter, the MCP tools and the CLI consult before sending; a read-back against the attempt's own baseline resolves it, and a refuted write waits for a person (`openadapt verify ledger list`, `reconcile`, `release`) | On main | Synthetic (MockMed's fault server: a worker killed between the send and the read-back, killed during a timed-out send, killed before the send, a late write, and two worker processes started together on one write) | `openadapt_flow/write_ledger.py`, [VERIFIER_API.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFIER_API.md#the-uncertain-write-ledger) | -| Catch a write that lands after the verdict (a background job, a trigger, a delayed side record): a `CONFIRMED` with a recheck due is provisional (`provisional`, `recheck_due_at`); the recheck reads the whole database again and refutes the earlier result when a change since the post-save read has no explanation, and a recheck that doesn't confirm holds the write in the ledger. Run it from Python (`verifier.recheck`) or MCP (`verify_recheck`). A write after the recheck's read is not seen (`frame.database.recheck_after_s`, no default window) | On main ([#1639](https://github.com/OpenAdaptAI/OpenAdapt/pull/1639)); provisional result, MCP tool and ledger hold ([#1669](https://github.com/OpenAdaptAI/OpenAdapt/pull/1669)) | Real app (Dolibarr 21 and EspoCRM 10 on MariaDB 11.4, 180 s window: 4 of 4 planted late writes at 125 to 130 s refuted at the recheck, correct writes confirmed; EspoCRM's cron daemon needs its job tables declared); synthetic (SQLite); opt-in live OpenEMR test | `openadapt_flow/verifier.py`, `runtime/effects/late_write.py`, [VERIFY_RUN.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFY_RUN.md#writes-that-land-later) | -| Diff the whole database for the frame (`frame.database`) on SQLite, MySQL and MariaDB. On MySQL and MariaDB every column type is read exactly (binary strings, `BIT`, geometries and vectors as hex, `FLOAT` widened to `DOUBLE`, `JSON` as stored text, decimals at full precision), so a row the save adds, changes or removes outside the declared records, such as an extra planned call or a stream note, is refuted unless a declared effect, a side record, an attributed concurrent write or an allow-listed bookkeeping table explains it | On main ([#1661](https://github.com/OpenAdaptAI/OpenAdapt/pull/1661)) | Real app (Dolibarr 21 on MariaDB 11.4: all 387 tables read where the frame raised `TypeError`; EspoCRM on MariaDB 11.4: the first-contact study's extra call and stream note refuted, correct opportunities verified); live MariaDB type tests | [VERIFY_RUN.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFY_RUN.md#the-whole-database), `runtime/effects/db_snapshot.py` | +| Catch a write that lands after the verdict (a background job, a trigger, a delayed side record): a `CONFIRMED` with a recheck due is provisional (`provisional`, `recheck_due_at`); the recheck reads the whole database again and refutes the earlier result when a change since the post-save read has no explanation, and a recheck that doesn't confirm holds the write in the ledger. Run it from Python (`verifier.recheck`) or MCP (`verify_recheck`). A write after the recheck's read is not seen (`frame.database.recheck_after_s`, no default window) | On main ([#1639](https://github.com/OpenAdaptAI/OpenAdapt/pull/1639)); provisional result, MCP tool and ledger hold ([#1669](https://github.com/OpenAdaptAI/OpenAdapt/pull/1669)) | Real app (Dolibarr 21 and EspoCRM 10 on MariaDB 11.4, 180 s window: 4 of 4 planted late writes at 125 to 130 s refuted at the recheck, correct writes confirmed; EspoCRM's cron daemon needs its job tables declared; Redmine 6.1 on PostgreSQL 17, 180 s window: the first-contact study's late writes F19 and F20 refuted at the recheck, correct writes confirmed); synthetic (SQLite); opt-in live OpenEMR test | `openadapt_flow/verifier.py`, `runtime/effects/late_write.py`, [VERIFY_RUN.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFY_RUN.md#writes-that-land-later) | +| Diff the whole database for the frame (`frame.database`) on SQLite, MySQL, MariaDB and PostgreSQL. On MySQL and MariaDB every column type is read exactly (binary strings, `BIT`, geometries and vectors as hex, `FLOAT` widened to `DOUBLE`, `JSON` as stored text, decimals at full precision). On PostgreSQL every column type is read as text by a declared rule in a session pinned to UTC and exact float output, and a type without a rule, a table the account can't read in full, or an unreadable sequence is refused. So a row the save adds, changes or removes outside the declared records, such as an extra planned call or a stream note, is refuted unless a declared effect, a side record, an attributed concurrent write or an allow-listed bookkeeping table explains it | On main ([#1661](https://github.com/OpenAdaptAI/OpenAdapt/pull/1661)); PostgreSQL ([#1673](https://github.com/OpenAdaptAI/OpenAdapt/pull/1673)) | Real app (Dolibarr 21 on MariaDB 11.4: all 387 tables read where the frame raised `TypeError`; EspoCRM on MariaDB 11.4: the first-contact study's extra call and stream note refuted, correct opportunities verified; Redmine 6.1 on PostgreSQL 17: 58 tables read, a stray upload in a table the contract doesn't read refuted, correct writes verified); live MariaDB and PostgreSQL type tests | [VERIFY_RUN.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFY_RUN.md#the-whole-database), `runtime/effects/db_snapshot.py` | | Catch a lost update: another user saves a change to the record while the agent works, and the agent saves its stale copy over it. Before and after look like a correct update, so `frame.versions` reads the target's versions (FHIR or SQL version history, a version column, or the app's audit or action-history table) before and after the write. A version by another author, or more than one version step when no author is read, refutes the write; a version that can't be read or attributed stops it | On main ([#1662](https://github.com/OpenAdaptAI/OpenAdapt/pull/1662)) | Real app (HAPI FHIR and EspoCRM, first-contact study fault scripts: both lost updates refuted, correct updates verified, every other case's verdict unchanged) | [VERIFY_RUN.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFY_RUN.md#lost-updates), `runtime/effects/versions.py` | | Guarded save: a safety layer any agent wraps around one write (check the record, hold for approval, read back) | On main ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)) | Synthetic | `GUARDED_SAVE.md` | | A person's approval in Desktop is signed with a key kept behind Touch ID on a Mac: one-prompt setup in the app, a prompt for every answer, and Flow never sees the key | On main ([#1500](https://github.com/OpenAdaptAI/OpenAdapt/pull/1500)) | Unit tests; opt-in test on a real Secure Enclave | `packages/desktop/openadapt_desktop/approver_keychain.py`, [APPROVAL_MODES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/APPROVAL_MODES.md#the-approver-key) | diff --git a/packages/flow/CHANGELOG.md b/packages/flow/CHANGELOG.md index c5f1334dc3..879ed736a2 100644 --- a/packages/flow/CHANGELOG.md +++ b/packages/flow/CHANGELOG.md @@ -19,6 +19,8 @@ ### Features +- The whole-database frame (`frame.database`) and its recheck now work on PostgreSQL (`dialect: postgresql`). The snapshot is one read-only repeatable-read transaction over every schema but the system ones. Every column type is read as text by a declared rule, in a session pinned to UTC and exact float output: `numeric` at full precision (with its stored scale when it has no declared one), floats exactly, `bytea` as hex, `json` as stored and `jsonb` canonical, arrays, ranges, enums, domains, `uuid`, network, time, `interval`, `money`, bit, `xml`, text-search and geometric types. A table without a primary key is keyed by the whole row, with duplicates counted. A column type without a rule, a table the account can't read in full (no `SELECT`, row-level security), a foreign table and an unreadable sequence are refused unless the allow-list skips them with a reason. A sequence a column owns is that table's counter, and a counter set back counts as a change; any other sequence is compared as a one-row table. Large objects are read by the SHA-256 of their content. `frame.database.schemas` means the same as on MySQL (each must be visible, and every visible schema is read); a schema is left out only by an `allow_tables` entry `.*` with `mode: skip` and a reason. + - A lost update now stops the run. `frame.versions` names a read of the target's versions (HAPI's `hfj_res_ver`, a version column, or an app's action-history table) and, where the app records one, the author of each version. The verifier reads them before and after the write; a version of a pre-existing target by anyone other than the agent, or more than one version step when no author is read, refutes the frame. A versions read that fails, a target with no version, or a version with no author leaves the frame not checked. The receipt holds counts and a problem code (`evidence.frame.versions`). Contracts without `frame.versions` keep their digests. - A write that lands after the verdict can now be caught. With `frame.database.recheck_after_s`, a verified write's receipt says when a recheck is due (`recheck_due_at`), and `openadapt_flow.verifier.recheck` reads the whole database again at or after that time. It judges every change since the post-save read with the frame's rules: the app profile's bookkeeping, other users' writes the audit log attributes, and changes a later write by the same actor under the same contract saw in its own window (`excused_by`, any outcome). Anything else turns the earlier result into `REFUTED` (late). A change that may be what refuted a later write, a database that can't be read, or a saved state that wasn't issued here or doesn't match its receipt gives `UNCERTAIN`. The saved state holds keyed hashes, not values. `frame.database.settle_attempts` lets the reads after a save, and at a recheck, retry within a stated budget until two in a row match; rows added to append-only tables between them don't count as movement. Without these settings, nothing changes ([#1639](https://github.com/OpenAdaptAI/OpenAdapt/pull/1639)). @@ -27,6 +29,8 @@ ### Fixes +- A composite primary key in the whole-database frame is now kept as a JSON array of its values. Before, the values were joined with `|`, so the keys (`x|y`, `z`) and (`x`, `y|z`) were the same, one row hid the other, and deleting it went unseen. This applies to every dialect. A snapshot or recheck state saved by an earlier version keys rows differently, so recheck a write with the version that checked it. + - The read-only SQL guard now reads the query's tokens instead of matching substrings. `REPLACE(phone, '-', '')`, a `;` inside a string literal, and a table quoted as `` `call` `` no longer refuse the query. Every write and multi-statement form it refused before is still refused. It now also refuses a backslash, a dollar-quoted string, an unterminated literal, and a call to `query_to_xml` or `eval`, which run a query passed as text. When the query has parameters, it also refuses a `%` placeholder inside a literal or quoted name (the driver splices the value in as SQL; `%%` is fine). It refuses a dollar quote right after a number, row locks (`FOR SHARE`, `FOR KEY SHARE`, `LOCK IN SHARE MODE`), and `get_lock`, `pg_advisory_*`, `pg_sleep_for`, `pg_sleep_until`, `master_pos_wait`, `source_pos_wait`, `pg_terminate_backend`, `pg_cancel_backend`, `load_file`, `pg_read_file`, `pg_ls_dir` and `pg_notify`. - The whole-database snapshot reads a SQLite `REAL` with 17 digits, so 0.3 and 0.1+0.2 no longer read the same, and the text `"\x00null"` no longer reads as NULL. A number's plain value, which selectors compare with, is the shortest text that reads back as the stored value (`1.1` for a MySQL `FLOAT`, `1.5` for a `DECIMAL(24,8)`), so a correct write is no longer stopped because a selector stopped matching. diff --git a/packages/flow/docs/GENERATE_CONTRACTS.md b/packages/flow/docs/GENERATE_CONTRACTS.md new file mode 100644 index 0000000000..b4d12cc9a5 --- /dev/null +++ b/packages/flow/docs/GENERATE_CONTRACTS.md @@ -0,0 +1,329 @@ +# Generate verify contracts from a SQL schema + +`python -m openadapt_flow.generate.verify_contract` writes the +`openadapt.verify/v1` contracts for an app's kinds of write. You give it a +read-only database account, a short description of each kind of write, and +one demonstration write per example. It asks a person only what it can't +decide alone, in a numbered list. The goal is three questions or fewer per +app. + +Use it for any app whose data is in MySQL, MariaDB or PostgreSQL. For a FHIR +server, use `openadapt_flow.generate.ensure_contract` (an +`openadapt.ensure-record/v1` contract). This guide needs no code reading. + +## What you need + +- **A read-only database account.** The contracts read with it, and so does + the generator. Put its connection in `read.yaml`, with the password in an + environment variable: + + ```yaml + sql_driver: pymysql # or psycopg2 for PostgreSQL + sql_connect_args: {host: 127.0.0.1, port: 3306, user: oa_reader, database: app} + sql_password_env: APP_READER_PASSWORD + ``` + + The generator refuses an account that can insert, update, delete or change + the schema. On MySQL and MariaDB the account also needs the `TRIGGER` + privilege, because the whole-database check must see triggers. The + generator can grant it (see [Setup steps it applies](#setup-steps-it-applies)). +- **`kinds.yaml`**: each kind of write, its parameter names, and one or two + example tasks. A null value means the task leaves that field out. A list + value (invoice lines) is a list of mappings. + + ```yaml + kinds: + - kind: update_customer + parameters: [customer, town, new_phone, new_email, new_address, new_zip] + examples: + - values: {customer: Transports Petit, town: Rouen, new_phone: 02 35 15 60 60} + - values: {customer: Pharmacie Moreau, town: Lille, new_phone: 03 20 06 07 08, + new_email: officine@pharmacie-moreau-lille.fr} + ``` + +- **The acting user's id**, as the caller passes it to the verifier, for + example `--actor-param bot_user_id=5`. A column that holds it (an author or + creator column) is checked against it. + +## Steps + +1. Check the account and read the schema: + + ```bash + python -m openadapt_flow.generate.verify_contract schema --read read.yaml + ``` + +2. For each example, make the write once, correctly, the way the agent will + (through the app's API or screens), between a snapshot and a record: + + ```bash + python -m openadapt_flow.generate.verify_contract snapshot --read read.yaml --out work/before.json + # ... make the write ... + python -m openadapt_flow.generate.verify_contract record --read read.yaml \ + --before work/before.json --kind update_customer \ + --values '{"customer": "Transports Petit", "town": "Rouen", "new_phone": "02 35 15 60 60"}' \ + --demos work/demos + ``` + + Skip an example whose record already exists ("make sure X exists"): the + right outcome there is no write. Snapshots hold row values. Keep the work + folder private, like a database export. + +3. Propose: + + ```bash + python -m openadapt_flow.generate.verify_contract propose --read read.yaml \ + --kinds kinds.yaml --demos work/demos --out work/out --actor-param bot_user_id=5 + ``` + +4. Read `work/out/QUESTIONS.md`. Write the answers in `answers.yaml` and run + `propose` again with `--answers answers.yaml`. Until every question is + answered, the manifest says so and `choose` refuses to pick a contract. + +## What it writes + +- `contracts//..yaml`: one contract per field set. The + flavor is `create` or `update`, and each has a `found` twin for a task + whose record already holds its values. +- `manifest.yaml`: which contract fits which task values. +- `QUESTIONS.md`: the numbered questions, the proposed answers, and per kind + how many contracts were written and why any were not. +- `report.json`: where each parameter was found, the rows the app adds on its + own, the tables treated as logs, and every contract that was not written, + with the reason. + +At run time, pick the contract for a task with `choose`, then call the +verifier as usual: + +```python +from openadapt_flow.generate.verify_contract import choose +from openadapt_flow.verifier import Actor, begin_write, verify_write + +contract, params = choose("work/out/manifest.yaml", "update_customer", task_values, + {"bot_user_id": "5"}) +pending = begin_write(contract, params, actor=Actor("agent", "bot")) +... # the agent writes +result = verify_write(pending) +``` + +`choose` raises `GenerateError` when no contract fits the task's values. Treat +that as a stop: the write must not be sent unchecked. + +## How it decides + +**The main record.** In each demonstration, the main record is the added row +(a create) or changed row (an update) that holds the most task values. + +**Where a value is.** Each value is looked up by exact rules only: + +| Rule | Example | +| --- | --- | +| The stored text equals the value | `subject` | +| Equal after case folding, for a value shaped like an email address | `Ana@Example.org` | +| Equal digits, only for a parameter a person confirmed as a phone number (the `phones` question), when the app stored other text with the same digits | `02 38 53 21 09` stored as `0238532109` | +| Equal as a number, a date or a date-time, compared in SQL | `9.20` stored as `9.2`; `2026-10-27 14:30` stored as `2026-10-27 14:30:00` | +| Minutes between two date-time columns of the row | a call's duration | +| A display column of the row a key column points to | `project_id` -> `projects.name` | +| Two text columns joined by one space | `firstname` and `lastname` | + +A display value behind a key must name exactly one row of its table that +isn't soft-deleted (a `deleted` or `is_deleted` flag, or a `deleted_at` +time): a second "Jon Park", or "Mary Ann Lee" split as `Mary Ann|Lee` and +`Mary|Ann Lee`, makes the value empty, so the write stops. At run time a +confirmed phone value that isn't written like a phone number (fewer than 7 +digits, or other characters) is compared exactly. +| A row the write added that holds the main record's key | a journal note, a custom field value | +| A row it added that links the record to the row holding the value | an email address, a call's attendee | + +A key column can be a declared foreign key, or a column whose value is +another table's primary key. When several places fit in every demonstration, +the generator keeps the shortest path, then a declared key, then a column +named after its table, then the most exact rule. Among places still tied, it +keeps, in this order: + +1. the column named like the parameter (`stage`, not `last_stage`); +2. the source column when another is its lower-cased or digits-only copy in + every row (`email_address.name`, not `email_address.lower`), or the column + the table defines first when they hold the same value in every row; +3. the place another kind's demonstrations used for the same table. + +If places are still tied, that is a question. A value that is also in a row +the write added beside the record (a call's lead is its parent and an +attendee row) is checked in both places. + +A parameter that no example gives is placed where another kind's +demonstrations found the parameter of the same name on the same main table +(`placed_from_another_kind`), else at the main table's column of the same +name after a leading `new_` or `set_` (`placed_by_name`). A parameter whose +value names the field to change (`field: status`, `value: In Process`) gets a +contract per field: the demonstrated ones, and every other text column of the +main table. + +**Field sets.** A task's field set is its non-null values. For an update, the +generator writes a contract for every non-empty set of the fields to change +that it found, up to 64. For a create, it writes one per demonstrated set, +then the sets with optional fields left out, up to 64 in all. A field is +optional when some example leaves it null or its place can be empty (a +nullable column, a row beside the record). A field left out of a create must +be empty in the new record, unless the demonstrations that left it out +filled that column anyway (a call's parent, which is its account when there +is no lead). A row the app adds with an empty value when a field is left out +(Redmine's custom field values, a journal with no note) is expected then +too. A list parameter gets a contract per length, from 1 to 10 items. + +**Create, found, update.** + +- `create`: exactly one new main record holds the task's values, and + everything else in the new row is what the demonstrations showed, or is + bookkeeping. +- `found`: the record already holds the task's values and, for a create, + the same constants the demonstrations' new rows held (a soft-delete flag, a + status, a type), and nothing in the database changed. `choose` reads the + store first and picks `found` when such a record is already there: a "make + sure it exists" task whose record exists, or an update that is already in + place. A trashed record with the task's values is not found. +- `update`: exactly one record matches the task's identity values and holds + the new values. Only the fields the task changes may change, plus the + target's own bookkeeping columns a person confirmed. When the + demonstrations replaced the record's rows beside it (an old email address + link removed), the earlier rows may go, and afterwards the record has as + many such rows as the demonstrations left. + +**Columns of a new row no value explains.** Each is pinned to the acting +user, to the value every demonstration held, or to a copy of a column of the +row a key points to (an invoice line's price from its product). A key, a +link to a row the write added, the year, month, week or day of a date the +task gives, and a generated token (a UUID) are recognized as such. Columns +that differ between demonstrations, free text every demonstration left +empty, and the write's time stamps are listed in the bookkeeping question, +and each is left unchecked only if the answer names it (`unchecked`). Free +text the answer doesn't name must stay empty: on the main record and its +child rows NULL or `''`, and on a row the app adds beside the write the +empty value every demonstration stored there. A kind with any other column +the answer doesn't name gets no contract. The proposed answer names the +columns that vary or are time stamps, so accepting it is explicit, and +leaves free text out: a person adds a free-text column to `unchecked` only +on purpose. + +**Rows the app adds on its own.** A row type the app added the same number of +times in every demonstration (an agenda event, a stream note, a price-history +row) is declared with that count and its constant columns. A table whose +added rows differ in number between demonstrations is listed in the +bookkeeping question and is a log only if the person names it (`logs_add`); +otherwise every added row stops the write. The person may also name a +counted table (an access log the app writes once per request: the agent may +make more requests than the demonstration did). A named table is a log only +for the kinds whose demonstrations added rows there: rows may be added to +it, and existing rows must not change. The generator never decides a table +is a log from its name. + +**Changes the app makes to existing rows.** A column of an existing row that +the demonstrations changed with no task value is excused only on that row: +the target itself, a row with the same key in every demonstration (a counter +row), or the row the task's values name through a key of the main record +(the account a contact belongs to). It is never excused by name on other +rows or tables. Time stamps set during the write, counters that went up and +columns set to the acting user are proposed; anything else is listed and +excused only if the person adds it (`columns_add`). A changed row the +generator can't identify is listed too, and a write that changes it stops. +Each declared row may change only its own table's excused columns +(`frame.target_fields_by_surface`): a parent row the app touches can't take +the target's changed fields, and the target can't take the parent's. + +**Answers.** `bookkeeping: accept` takes the proposal. A mapping may have +`accept: true` (take the proposed columns), `logs_add`, `columns_add`, +`columns_remove` and `unchecked`, and each can name only what the question +listed. Any other key, or a table or column the question didn't list, is +refused. A mapping without `accept: true` takes only what it names. + +**The frame.** Every contract uses the whole-database frame +(`frame.database`, with the store's dialect): a row added, removed or changed +in any table that the contract doesn't declare stops the write, except rows +added to a table the person named as a log. The main record's table and the +tables of rows that hold values are read by the contract's own reads, with +every column. On PostgreSQL the read account also needs `SELECT` on the +sequences (see [VERIFY_RUN.md](VERIFY_RUN.md)); without it the dry read +fails and no contract is written. + +**Lost updates.** An update's contracts set `frame.versions`, so another +user's save to the target inside the window stops the write. The generator +first takes a whole-number column named as a version that went up by one in +every demonstration (Redmine's `lock_version`): every existing record has a +version there. Such a column is trusted to step from the stored value on +every save. An app that writes back the version its form carried, without +checking it, makes another user's save and the agent's stale one a single +step, which this check can't see. An app with an audit trail of saves can +read that trail as well (a hand-written `frame.versions`), since authors +catch what a step count can't. Otherwise it takes the app's own audit trail +when every update demonstration added a row there that names the record and +the acting user (EspoCRM's `action_history_record`, Dolibarr's agenda +events); the audit table's own key never counts as the record's. The read +is narrowed to the record's type, and every entry counts as a save except +an event code shown to be a read: one a demonstration logged on a record it +didn't add or change, and no demonstration logged on one it did. A code no +demonstration showed (a bulk edit, an import) and a NULL code count as +saves. Only the main record is a target (`target_surface`). A target +the read before the write doesn't show as new is treated as existing, and +with no version before the write it stops as `missing` (a record saved +before the trail began, or outside the app). With neither, the update has no +version check, and `report.json` shows `versions: null`. If the app stops +writing the trail (an agenda setting turned off), the agent's own save has +no new version, and the verifier stops the write rather than pass it. + +**The recheck window.** One question asks how long the +app can still write on its own after a verified write. The answer is +`frame.database.recheck_after_s`; the proposal is 300 seconds. + +## Setup steps it applies + +The generator never leaves a step for a person to apply later. A contract +that needs something the store doesn't have is either set up, with a flag +and an admin connection you give, or not written, with the reason. + +The read account must be read-only. On PostgreSQL the check uses the +privileges of every role it can act as (a role it inherits, or +one it can `SET ROLE` to), and column grants count too: `INSERT`, +`UPDATE`, `DELETE`, `TRUNCATE`, `TRIGGER` or `REFERENCES` on any table, +`USAGE` or `UPDATE` on a sequence, or `CREATE` on a schema or the database +stops the generator. + +- `--grant-trigger --admin-read admin.yaml --reader-account "'oa_reader'@'%'"`: + grants `TRIGGER` on the database to the read account (MySQL, MariaDB). The + account name is split into user and host and escaped by the driver. + Without it, the generator stops and prints the `GRANT` statement for the + DBA. `TRIGGER` also lets the account create and drop triggers; the verifier + needs it only to see them, and `report.json` says so under `read_account`. + Keep that account's password away from the agent. +- `--create-views --admin-read admin.yaml --reader-account ...`: the + verifier's SQL guard refuses some table names in a read (EspoCRM's `call`). + This creates a view `oa_read_` in a schema of its own, + `_oa_read` (so the app's own restores and upgrades don't drop it), + and grants the read account `SELECT` on that schema. Without the flag, the + kind is refused with the reason. `report.json` lists what was applied under + `applied`. + +Every contract is loaded and read once against the live store before it is +written out. A contract whose read fails is not written; `report.json` says +why. + +## Limits + +- A parameter that no example gives is placed only by the name rule above. + Otherwise a task that gives it has no contract and is stopped. Add an + example and a demonstration. +- A value the demonstrations hold in no row, or in a place none of the rules + describe, is a question. +- A list item that appears twice in one task (the same product and quantity + twice) is stopped: each item must match one new row. +- Columns of a new record that differ between demonstrations and aren't + task values (an invoice's computed totals, its due date), free text every + demonstration left empty, and time stamps are not checked. The bookkeeping + question lists every one of them. +- With one demonstration of a create, every other column must hold the value + it held then, including one computed from the task's values (a price with + tax). `QUESTIONS.md` says so; record a second create with other values. +- REST metadata (OpenAPI) is not read yet. Apps without database access need + hand-written contracts. +- The frame doesn't attribute changes to other users: another user's edit to + an unrelated record during the window stops the write. Only the target's + own versions are read from the audit trail (lost updates). diff --git a/packages/flow/docs/VERIFY_RUN.md b/packages/flow/docs/VERIFY_RUN.md index 11da921326..85d5225c53 100644 --- a/packages/flow/docs/VERIFY_RUN.md +++ b/packages/flow/docs/VERIFY_RUN.md @@ -99,6 +99,9 @@ frame: identity_field: pid # default: id, else the whole record ignore_fields: [date] # fields that change on their own target_fields: [phone_home] # default: the fields the effects name + # target_fields_by_surface: # optional: per surface, replaces target_fields + # record: [phone_home] # a declared row of `record` may change these + # raw_account: [updated_at] # a table counts as the surface `views` maps it to attribution: # optional kind: audit_read # or record_field, with server_set: [...] actor_field: user @@ -260,7 +263,7 @@ This is the frame of an EspoCRM "create opportunity" contract on MariaDB: frame: identity_field: id database: - dialect: mysql # or sqlite + dialect: mysql # or postgresql, sqlite read: # read-only: SELECT, TRIGGER and SHOW VIEW sql_driver: pymysql sql_connect_args: {host: 127.0.0.1, port: 3306, user: frame_ro, database: espocrm} @@ -307,6 +310,84 @@ that still can't be read exactly makes the database unreadable, and the run stops; the error names the table. When the database can't be read at the baseline, `begin_write` says why. +### PostgreSQL + +With `dialect: postgresql` (driver `psycopg2` or `psycopg`), the snapshot is +one `REPEATABLE READ, READ ONLY` transaction. The read refuses to go on if the +server reports any other isolation level. + +- **Schemas.** The read covers every schema except `pg_catalog`, + `information_schema` and PostgreSQL's internal ones. A schema the read + account can't see is refused, not left out. `frame.database.schemas` + means what it means on MySQL: each named schema must be visible, and every + other visible schema is still read. Only an `allow_tables` entry + `.*` with `mode: skip` and a reason leaves a schema out; the + receipt lists it under `evidence.frame.database.schemas_skipped`. Leaving + a schema out also leaves out its code, such as a trigger function there + that runs on a table the frame reads. A table in the default schema + (`public`) is named without it (`issues`); a table in another schema is + named with it (`billing.invoice`). +- **The session.** The read pins its session so that each stored value has + one text: `TimeZone` UTC, ISO dates, ISO 8601 intervals, + `extra_float_digits = 3`, bytes as hex, and `lc_monetary` C. A reader + whose own settings differ reads the same thing. +- **Types.** Each column is read as text by a declared rule for its type: + - integers as integers; + - `numeric` at full precision (`NaN` and infinities too). A `numeric` + declared without a scale keeps the scale it was stored with: `1.5` + changed to `1.50` shows, but both still equal the number 1.5; + - `real` and `double precision` exactly, with the shortest text that + reads back as the same float as the plain value; + - `money` as a number; + - `bytea` as the hex of its bytes; + - `json` as its stored text, and `jsonb` as its canonical text; + - as their own text: strings, `uuid`, network addresses, dates, times, + timestamps (in UTC), `interval` (months, days and seconds kept apart, so + `1 day` to `24 hours` shows), bit strings, `xml`, `tsvector`, `tsquery` + and the geometric types; + - arrays, ranges, multiranges, enums and domains by their element, + subtype or base type's rule. + + A composite type, an extension's type, a `reg*` type, and an array of + `money` have no rule. A table with such a column is refused, and the error + names the column. +- **Refusals.** The read also refuses a table the account has no `SELECT` + on, a table whose row-level security hides rows from it, a foreign table, + and a sequence it can't read. A refused table can only be left out by an + `allow_tables` entry with `mode: skip` and a reason. +- **Tables without a primary key.** Their rows are keyed by the whole row, + and identical rows are counted, so a second copy of a row shows as added. +- **Partitions.** A partitioned table is read through its partitions, and + only once. +- **Sequences.** A sequence that a column owns (`serial`, `identity`) is + that table's counter, read just before the snapshot. A row written and + removed in the window moves it, which shows. So does an insert that + failed, because it draws a value too, and a counter set back with + `setval`, after which inserts would reuse numbers. Any other sequence is read as + a one-row table of its own. A value drawn from it is a change. Only an + `allow_tables` entry with `mode: skip` and a reason, which documents it as + the app's bookkeeping, leaves it out. +- **Large objects.** They live in `pg_catalog`, and a column that holds + an object's oid keeps it when the content changes. The read keeps each + object's oid, owner, privileges and the SHA-256 of its content as rows of + a table named `pg_largeobject`, so a changed, added or removed object + shows. An object the account can't read is refused unless `pg_largeobject` + is skipped with a reason. +- **Database code.** The read digests triggers (and whether each is + enabled), views, materialized views, functions, rules, row-level security + policies, event triggers and installed extensions. A change to any of them + in the window shows as `(database code)`. + +The read account needs `USAGE` on each schema and `SELECT` on its tables and +sequences, for example: + +```sql +GRANT USAGE ON SCHEMA public TO frame_ro; +GRANT SELECT ON ALL TABLES IN SCHEMA public TO frame_ro; +GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO frame_ro; +ALTER ROLE frame_ro SET default_transaction_read_only = on; +``` + ## Lost updates Two snapshots can't see a lost update. The agent reads a record, another @@ -333,6 +414,7 @@ frame: FROM hfj_res_ver v JOIN hfj_resource r ON r.res_id = v.res_id record_field: id # the version row's record target_field: id # the target's field in the read set (default: identity_field) + # target_surface: record # only records matched on this surface are targets version_field: ver # author_field: user_id # when the app records who saved each version # self: [{param: bot_user_id}] @@ -402,7 +484,10 @@ Anything else stops the run, never a pass: A record created in the window has no earlier version to overwrite and isn't judged. Whether a target existed is told by `frame.identity_field` (default `id`), never by `target_field`'s value after the write: a target that existed -with no version under its current `target_field` value is `missing`. Only targets are judged: another user's save to an unrelated record +with no version under its current `target_field` value is `missing`. A target +is new only when the read before the write shows it wasn't there: the +effect's own surface read for a `surfaces` verifier, else the verifier's +records. With no such read, or no identity, it is treated as existing. Only targets are judged: another user's save to an unrelated record leaves this check as it was (the frame and its attribution judge that record, as before). The versions are read before the records at the baseline, so a save between the two reads shows in the record baseline. The @@ -639,13 +724,19 @@ such change refutes unless the contract explains it (`allow_tables`, the app profile, `ignore_columns`, `side_records`) or the audit log attributes it to another user. -Measured on two real apps, Dolibarr 21 and EspoCRM 10 on MariaDB 11.4, with -`recheck_after_s: 180`: +Measured on three real apps, Dolibarr 21 and EspoCRM 10 on MariaDB 11.4 and +Redmine 6.1 on PostgreSQL 17, with `recheck_after_s: 180`: - Four planted late writes (a changed phone, a second product, a delayed duplicate opportunity, a look-alike lead's title) landed 125 to 130 s after the save. Each check said `CONFIRMED`, and each recheck said `REFUTED`. Correct writes of the same tasks stayed `CONFIRMED`. +- On Redmine, a time entry's hours changed 150 s after the save, and a + delayed retry created a duplicate issue 140 s after it. Each check said + `CONFIRMED`, and each recheck said `REFUTED`. Correct writes stayed + `CONFIRMED` at the check and the recheck. Redmine's own bookkeeping + needed one declaration: `user_preferences.others` in `ignore_columns` + (every save records the user's last project there). - With the apps' schedulers off, `settle_s: 1` and `settle_attempts: 1` were enough. - With EspoCRM's cron daemon on, its job queue moved during every window. diff --git a/packages/flow/docs/VERIFY_WRITES.md b/packages/flow/docs/VERIFY_WRITES.md index 5bcc461af6..f71443376a 100644 --- a/packages/flow/docs/VERIFY_WRITES.md +++ b/packages/flow/docs/VERIFY_WRITES.md @@ -26,6 +26,11 @@ effects: The contract belongs to whoever runs the agent. The agent can't change what it is checked against. +When the app keeps its data in MySQL, MariaDB or PostgreSQL, the generator +writes these contracts from the schema and one demonstration write per +example, and asks a person only what it can't decide: +[GENERATE_CONTRACTS.md](GENERATE_CONTRACTS.md). + ## 2. Wrap the write In Python, wrap your agent's own write function. The same code works as a tool in the Claude Agent SDK, OpenAI Agents, LangGraph, or a plain script: diff --git a/packages/flow/openadapt_flow/generate/demo_analysis.py b/packages/flow/openadapt_flow/generate/demo_analysis.py new file mode 100644 index 0000000000..3f54acd89f --- /dev/null +++ b/packages/flow/openadapt_flow/generate/demo_analysis.py @@ -0,0 +1,751 @@ +"""Find where each parameter of a kind of write lands, from demonstration diffs. + +A demonstration is one correct write of a kind, made by the person or the +setup agent, with the store snapshotted before and after +(:mod:`openadapt_flow.generate.sql_store`). This module reads the diffs and +says, for each parameter, which table and column holds its value, and by +which exact rule: + +- ``exact``: the stored text equals the value's text; +- ``email``: equal after case folding (``ValueExpr`` kind ``email``); +- ``phone``: equal digits (``ValueExpr`` kind ``phone``); +- ``number``, ``date``, ``datetime``: equal by value, compared in SQL after a + cast of the parameter to the column's type; +- ``minutes``: the number of minutes between two date-time columns of the + row; +- a display column of a row another column points to (a declared foreign + key, or a column whose value is that row's primary key), or two text + columns joined by one space (a person's first and last name); +- a row the write added beside the main record, linked to it by a column + that holds the main record's key. + +Every rule is exact: no fuzzy matching, no model. A parameter found in more +than one place in every demonstration, or in none, is reported, and the +generator turns it into a question for a person. +""" + +from __future__ import annotations + +import datetime as _dt +import decimal +import re +from collections.abc import Mapping, Sequence +from dataclasses import dataclass, field +from typing import Any, Optional + +from openadapt_flow.generate.sql_store import ( + Diff, + RowChange, + Schema, + Table, + TableRows, +) +from openadapt_flow.runtime.effects.effect import kind_form + +#: Rules a value can be found by, most exact first. +RULES = ("exact", "email", "phone", "number", "date", "datetime", "minutes") + +_IDLIKE = re.compile(r"(^id$|_id$|^id_|^fk_|_fk$|^parent$|^parent_id$)", re.I) +_IDENT = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$") +_EMAIL = re.compile(r"^[^@\s]+@[^@\s]+\.[^@\s]+$") +_PHONE = re.compile(r"^\+?[0-9][0-9 ().\-/]*$") +#: Fewest digits a value compared as a phone number may have. +PHONE_MIN_DIGITS = 7 + + +def phone_shaped(value: Any) -> bool: + """Is ``value`` written like a phone number (digits and ``+ ( ) - . /`` + and spaces, with at least :data:`PHONE_MIN_DIGITS` digits)?""" + + text = value_text(value) if value is not None else "" + return ( + bool(_PHONE.match(text)) + and sum(ch.isdigit() for ch in text) >= PHONE_MIN_DIGITS + ) + + +def value_text(value: Any) -> str: + """A parameter value as text, the way the verifier renders it.""" + + from openadapt_flow.runtime.authorization import runtime_param_text + + if isinstance(value, str): + return value + return runtime_param_text(value) + + +def _decimal(text: Any) -> Optional[decimal.Decimal]: + if text is None or isinstance(text, bool): + return None + try: + d = decimal.Decimal(str(text).strip()) + except (decimal.InvalidOperation, ValueError): + return None + return d if d.is_finite() else None + + +def _datetime(text: str) -> Optional[_dt.datetime]: + for fmt in ( + "%Y-%m-%d %H:%M:%S", + "%Y-%m-%d %H:%M", + "%Y-%m-%dT%H:%M:%S", + "%Y-%m-%dT%H:%M", + "%Y-%m-%dT%H:%M:%SZ", + "%Y-%m-%dT%H:%MZ", + ): + try: + return _dt.datetime.strptime(text.strip(), fmt) + except ValueError: + continue + return None + + +def rule_for(stored: Optional[str], column_kind: str, value: Any) -> Optional[str]: + """The most exact rule by which ``stored`` holds ``value``, or None.""" + + if stored is None or value is None or isinstance(value, (list, dict)): + return None + text = value_text(value) + if text == "": + return None + if column_kind == "text": + # An email address is compared case-folded even when this + # demonstration's text matches exactly (another write may store + # ``Ana@Example.org``). A phone number only when the app stored other + # text with the same digits, and only as a proposal: a person + # confirms it (the ``phones`` question), never the value's shape. + if _EMAIL.match(text) and stored.casefold() == text.casefold(): + return "email" + if ( + stored != text + and _PHONE.match(text) + and sum(ch.isdigit() for ch in text) >= PHONE_MIN_DIGITS + ): + a, b = kind_form("phone", stored), kind_form("phone", text) + if a is not None and a == b: + return "phone" + if stored == text and column_kind not in ("number", "bool", "date", "datetime"): + # A number or date column is compared by value even when this + # demonstration's text happens to match: another row of the same + # column may print as ``5.5000``. + return "exact" + if column_kind == "text": + return None + if column_kind in ("number", "bool"): + da, db = _decimal(stored), _decimal(text) + if da is not None and db is not None and da == db: + return "number" + return None + if column_kind == "date": + return "date" if stored[:10] == text and len(text) == 10 else None + if column_kind == "datetime": + ta, tb = _datetime(stored.split(".")[0]), _datetime(text) + if ta is not None and tb is not None and ta == tb: + return "datetime" + return None + return None + + +@dataclass(frozen=True) +class Hop: + """One step from a row to another row. + + ``fk``: this row's ``column`` holds the key ``ref_column`` of a row of + ``table``. ``child``: a row of ``table`` whose ``column`` holds this row's + primary key ``ref_column``. + """ + + kind: str # fk | child + table: str + column: str + ref_column: str + declared: bool = False + + def to_json(self) -> dict[str, Any]: + return { + "kind": self.kind, + "table": self.table, + "column": self.column, + "ref_column": self.ref_column, + "declared": self.declared, + } + + @classmethod + def from_json(cls, d: Mapping[str, Any]) -> "Hop": + return cls( + str(d["kind"]), + str(d["table"]), + str(d["column"]), + str(d["ref_column"]), + bool(d.get("declared", False)), + ) + + +@dataclass(frozen=True) +class Location: + """Where a value is: a path of hops from the main row, then the column + (or two columns joined by a space) that holds it, and the rule.""" + + hops: tuple[Hop, ...] + columns: tuple[str, ...] + rule: str + + @property + def table_hint(self) -> str: + return self.hops[-1].table if self.hops else "" + + def signature(self) -> str: + path = "/".join( + f"{h.kind}:{h.table}.{h.column}={h.ref_column}" for h in self.hops + ) + return f"{path}#{'+'.join(self.columns)}@{self.rule}" + + def describe(self, anchor: str) -> str: + where = anchor + for h in self.hops: + if h.kind == "fk": + where = f"{where}.{h.column} -> {h.table}" + else: + where = f"{h.table} (whose {h.column} is the {where} key)" + cols = " || ' ' || ".join(self.columns) + return f"{where}.{cols} ({self.rule})" + + def to_json(self) -> dict[str, Any]: + return { + "hops": [h.to_json() for h in self.hops], + "columns": list(self.columns), + "rule": self.rule, + } + + @classmethod + def from_json(cls, d: Mapping[str, Any]) -> "Location": + return cls( + tuple(Hop.from_json(h) for h in d.get("hops", [])), + tuple(d["columns"]), + str(d["rule"]), + ) + + +def _hop_cost(h: Hop) -> int: + """0 for a declared foreign key, 1 for a column named after the table it + points to (``status_id`` -> ``issue_statuses``), 2 for a value match only.""" + + if h.declared: + return 0 + stem = re.sub(r"(^fk_|_id$|_fk$|^id_)", "", h.column.lower()).rstrip("s") + table = h.table.lower() + return 1 if stem and len(stem) >= 3 and stem in table else 2 + + +def _rank(loc: Location) -> tuple[int, int, int, int]: + return ( + len(loc.hops), + sum(_hop_cost(h) for h in loc.hops), + RULES.index(loc.rule) if loc.rule in RULES else 99, + len(loc.columns), + ) + + +@dataclass +class Demo: + """One demonstration: the kind, the values given, and the diff.""" + + kind: str + values: dict[str, Any] + diff: Diff + after: Mapping[str, TableRows] + started_at: str = "" + finished_at: str = "" + name: str = "" + + +class _Index: + """Rows by table and single-column key, from one snapshot.""" + + def __init__(self, schema: Schema, rows: Mapping[str, TableRows]) -> None: + self.schema = schema + self.rows = rows + self._by_key: dict[str, dict[str, dict[str, Optional[str]]]] = {} + for name, table in schema.tables.items(): + t = rows.get(name) + if t is None or t.count_only is not None or len(table.primary_key) != 1: + continue + pk = table.primary_key[0] + self._by_key[name] = { + str(r.get(pk)): r for r in t.rows.values() if r.get(pk) is not None + } + + def row(self, table: str, key: str) -> Optional[dict[str, Optional[str]]]: + return self._by_key.get(table, {}).get(key) + + def keyed_tables(self) -> list[str]: + return list(self._by_key) + + +def _fk_targets( + schema: Schema, idx: _Index, table: Table, column: str, value: str +) -> list[tuple[str, str, bool]]: + """Rows ``table.column = value`` may point to: (table, key column, + declared).""" + + out = [] + for fk in table.foreign_keys: + if fk.column == column and idx.row(fk.table, value) is not None: + target = schema.tables.get(fk.table) + if target is not None and target.primary_key == [fk.ref_column]: + out.append((fk.table, fk.ref_column, True)) + if out: + return out + if not _IDLIKE.search(column) or value in ("", "0"): + return [] + for name in idx.keyed_tables(): + if idx.row(name, value) is not None: + out.append((name, schema.tables[name].primary_key[0], False)) + return out + + +def _display_matches( + table: Table, row: Mapping[str, Optional[str]], value: Any +) -> list[tuple[tuple[str, ...], str]]: + """Columns of ``row`` that hold ``value``, with the rule.""" + + out: list[tuple[tuple[str, ...], str]] = [] + for col in table.columns: + rule = rule_for(row.get(col.name), col.kind, value) + if rule is not None: + out.append(((col.name,), rule)) + if isinstance(value, str) and " " in value.strip(): + texts = [c for c in table.columns if c.kind == "text"] + for i, a in enumerate(texts): + for b in texts[i + 1 : i + 3]: + va, vb = row.get(a.name), row.get(b.name) + if va and vb and f"{va} {vb}" == value: + out.append(((a.name, b.name), "exact")) + if isinstance(value, (int, float)) and not isinstance(value, bool): + stamps = [c for c in table.columns if c.kind == "datetime"] + for a in stamps: + for b in stamps: + if a is b: + continue + ta, tb = row.get(a.name), row.get(b.name) + da = _datetime((ta or "").split(".")[0]) if ta else None + db = _datetime((tb or "").split(".")[0]) if tb else None + if da and db and db > da: + minutes = decimal.Decimal((db - da).total_seconds()) / 60 + if minutes == _decimal(value_text(value)): + out.append(((a.name, b.name), "minutes")) + return out + + +def locate( + schema: Schema, + demo: Demo, + anchor: RowChange, + value: Any, + *, + before: bool = False, + changed_only: bool = False, + max_hops: int = 2, + links: Optional[Mapping[str, set[str]]] = None, +) -> list[Location]: + """Every location of ``value`` from the main row ``anchor``. + + ``before``: look in the row as it was before the write (an update's + identity). ``changed_only``: at depth 0, only columns the write changed + (an update's new values). + """ + + idx = _Index(schema, demo.after) + table = schema.tables[anchor.table] + row = anchor.before if before else anchor.after + if row is None: + return [] + found: list[Location] = [] + changed = set(anchor.changed_columns) + added = set() + for ch in demo.diff.changes: + t2 = schema.tables.get(ch.table) + if ( + ch.kind == "added" + and ch.after + and t2 is not None + and len(t2.primary_key) == 1 + ): + added.add((ch.table, ch.after.get(t2.primary_key[0]))) + + def walk( + t: Table, r: Mapping[str, Optional[str]], hops: tuple[Hop, ...], depth: int + ) -> None: + for cols, rule in _display_matches(t, r, value): + if depth == 0 and changed_only and not (set(cols) & changed): + continue + found.append(Location(hops, cols, rule)) + if depth >= max_hops: + return + for col in t.columns: + v = r.get(col.name) + if v is None or (col.name in t.primary_key and hops): + continue + if depth == 0 and changed_only and col.name not in changed: + continue + for target, ref, declared in _fk_targets(schema, idx, t, col.name, v): + target_row = idx.row(target, v) + if target_row is None or (target == anchor.table and hops): + continue # back to the main record: not a location + if not declared and not hops and anchor.key and (target, v) in added: + # The main row's key columns point at rows that were + # there before; a row the write added is reached from a + # row that names the main record instead. + continue + walk( + schema.tables[target], + target_row, + hops + (Hop("fk", target, col.name, ref, declared),), + depth + 1, + ) + if depth == 0 and len(t.primary_key) == 1 and not before: + key = r.get(t.primary_key[0]) + if key is None: + return + for ch in demo.diff.changes: + if ch.kind != "added" or ch.table == t.name or ch.after is None: + continue + child = schema.tables.get(ch.table) + if child is None: + continue + for col in child.columns: + if ch.after.get(col.name) != key: + continue + if links is not None and col.name not in links.get( + ch.table, {col.name} + ): + continue + declared = any( + fk.column == col.name and fk.table == t.name + for fk in child.foreign_keys + ) + if not declared and not _IDLIKE.search(col.name): + continue + walk( + child, + ch.after, + hops + + ( + Hop( + "child", ch.table, col.name, t.primary_key[0], declared + ), + ), + depth + 1, + ) + + walk(table, row, (), 0) + unique: dict[str, Location] = {} + for loc in found: + unique.setdefault(loc.signature(), loc) + return sorted(unique.values(), key=_rank) + + +def link_columns( + schema: Schema, demos: Sequence[tuple["Demo", RowChange]] +) -> dict[str, set[str]]: + """Per table, the columns that hold the main record's key in every row + that table gained, across all demonstrations (a journal's + ``journalized_id``, not its ``user_id`` that happens to be equal once).""" + + out: dict[str, set[str]] = {} + for demo, anchor in demos: + t = schema.tables.get(anchor.table) + row = anchor.after or {} + if t is None or len(t.primary_key) != 1 or row.get(t.primary_key[0]) is None: + continue + key = row[t.primary_key[0]] + for ch in demo.diff.changes: + if ch.kind != "added" or ch.table == anchor.table or ch.after is None: + continue + cols = {c for c, v in ch.after.items() if v == key} + out[ch.table] = out[ch.table] & cols if ch.table in out else cols + return out + + +def best(locations: Sequence[Location]) -> tuple[Optional[Location], list[Location]]: + """The single best location, or None and the tied ones.""" + + if not locations: + return None, [] + ranked = sorted(locations, key=_rank) + top = [loc for loc in ranked if _rank(loc) == _rank(ranked[0])] + if len(top) == 1: + return top[0], [] + return None, top + + +def pick_anchor( + schema: Schema, demo: Demo, params: Sequence[str] +) -> tuple[Optional[RowChange], str, int]: + """The main row of a demonstration: the added row (a create) or changed + row (an update) that holds the most parameter values. ``mode`` is + ``create`` or ``update``; ``score`` how many values it holds.""" + + best_row: Optional[RowChange] = None + best_mode, best_score, best_direct = "", 0, 0 + for ch in demo.diff.changes: + if ch.kind == "removed" or ch.table not in schema.tables: + continue + mode = "create" if ch.kind == "added" else "update" + score = direct = 0 + for p in params: + v = demo.values.get(p) + if v is None or isinstance(v, (list, dict)): + continue + if mode == "update": + hit = locate(schema, demo, ch, v, before=True, max_hops=1) or locate( + schema, demo, ch, v, changed_only=True, max_hops=1 + ) + else: + hit = locate(schema, demo, ch, v, max_hops=1) + # Only the row's own columns, and keys named after the table they + # point to: a row the write added beside the main record (an + # issue's journal) reaches the record's values only through a + # link back, and a key equal to some row's id by chance proves + # nothing. + hit = [ + loc + for loc in hit + if all(h.kind == "fk" and _hop_cost(h) <= 1 for h in loc.hops) + ] + score += 1 if hit else 0 + direct += 1 if any(not loc.hops for loc in hit) else 0 + if (score, direct) > (best_score, best_direct): + best_row, best_mode, best_score, best_direct = ch, mode, score, direct + return best_row, best_mode, best_score + + +def is_identifier(value: Any) -> bool: + return isinstance(value, str) and bool(_IDENT.match(value)) + + +def snake(name: str) -> str: + return re.sub(r"(? bool: + if not stamp or not start or not end: + return False + when = _datetime(stamp.split(".")[0].split("+")[0]) + if when is None: + try: + when = _dt.datetime.strptime(stamp[:10], "%Y-%m-%d") + except ValueError: + return False + lo = _dt.datetime.fromisoformat(start.replace("Z", "")) - _dt.timedelta(days=1) + hi = _dt.datetime.fromisoformat(end.replace("Z", "")) + _dt.timedelta(days=1) + return lo <= when <= hi + + +def column_roles( + schema: Schema, + table: str, + rows: Sequence[tuple[Demo, Mapping[str, Optional[str]]]], + explained: set[str], + actor_values: Mapping[str, str], + link_values: Sequence[set[str]] = (), +) -> dict[str, ColumnRole]: + """Roles of the columns of new rows of ``table`` that no parameter + explains, across every demonstration row given (one row per demo or + several). ``link_values``: per row, keys of rows the same write added + (a column holding one is a link).""" + + t = schema.tables[table] + out: dict[str, ColumnRole] = {} + for col in t.columns: + name = col.name + if name in explained: + continue + values = [r.get(name) for _, r in rows] + if col.auto or (name in t.primary_key and len(t.primary_key) == 1): + out[name] = ColumnRole("key") + continue + if ( + col.kind in ("date", "datetime") + and all( + v is None or _within(v, d.started_at, d.finished_at) + for (d, _), v in zip(rows, values) + ) + and any(v is not None for v in values) + ): + out[name] = ColumnRole("timestamp") + continue + actor = [ + p + for p, av in actor_values.items() + if values and all(v is not None and v == av for v in values) + ] + if actor: + out[name] = ColumnRole("actor", detail={"param": actor[0]}) + continue + links = link_values or [set() for _ in rows] + if values and all( + v is not None + and (v in lv or (len(t.primary_key) == 1 and v == r.get(t.primary_key[0]))) + for v, lv, (_, r) in zip(values, links, rows) + ): + out[name] = ColumnRole("link") + continue + if ( + len(t.primary_key) == 1 + and values + and all( + v is not None + and r.get(t.primary_key[0]) is not None + and str(r.get(t.primary_key[0])) in v + and v != r.get(t.primary_key[0]) + for v, (_, r) in zip(values, rows) + ) + ): + out[name] = ColumnRole("varies", detail={"why": "holds the row's own key"}) + continue + part = _date_part(values, [d for d, _ in rows]) + if part is not None: + out[name] = ColumnRole( + "varies", detail={"why": f"the {part} of a date the task gives"} + ) + continue + copied = _copied(schema, t, name, rows) + if copied is not None: + out[name] = ColumnRole("copied", detail=copied) + continue + distinct = {v for v in values} + if col.kind == "text" and distinct <= {None, ""}: + # Free text the task doesn't give and the demonstrations left + # empty (a line's description, a note): an agent may fill it + # from the record it copies, so it isn't checked. + # The empty value the demonstrations stored: NULL, '' or both. + empty = ( + "null" + if distinct == {None} + else "text" + if distinct == {""} + else "mixed" + ) + out[name] = ColumnRole("free", detail={"empty": empty}) + continue + if len(distinct) == 1: + v = values[0] + default = col.default + if v is None: + out[name] = ColumnRole("constant", None) + else: + out[name] = ColumnRole( + "constant", + v, + detail={ + "is_default": default is not None and default.strip("'") == v + }, + ) + continue + out[name] = ColumnRole("varies") + return out + + +def _date_part( + values: Sequence[Optional[str]], demos: Sequence["Demo"] +) -> Optional[str]: + """Is the column the year, month, ISO week or day of one date value the + task gives, in every row? (Redmine stores a time entry's ``tyear``, + ``tmonth`` and ``tweek``.) The app computes it, so it isn't pinned.""" + + parts = { + "year": lambda d: d.year, + "month": lambda d: d.month, + "week": lambda d: d.isocalendar()[1], + "day": lambda d: d.day, + } + for label, fn in parts.items(): + ok = bool(values) + for v, demo in zip(values, demos): + dates = [] + for x in demo.values.values(): + if isinstance(x, str) and re.match(r"^\d{4}-\d{2}-\d{2}", x): + try: + dates.append(_dt.date.fromisoformat(x[:10])) + except ValueError: + pass + if v is None or not any(str(fn(dd)) == v for dd in dates): + ok = False + break + if ok: + return label + return None + + +def _copied( + schema: Schema, + t: Table, + name: str, + rows: Sequence[tuple[Demo, Mapping[str, Optional[str]]]], +) -> Optional[dict[str, Any]]: + """Is the column a copy of one column of the row an FK points to, in + every row? (An invoice line's unit price is its product's price.)""" + + col = t.column(name) + if col is None or col.kind not in ("number", "text") or len(rows) < 2: + # One row can't tell a copy from a coincidence (a user id equal to a + # project id). + return None + candidates: Optional[set[tuple[str, str, str, str]]] = None + for demo, r in rows: + v = r.get(name) + if v is None: + return None + idx = _Index(schema, demo.after) + here: set[tuple[str, str, str, str]] = set() + for other in t.columns: + if ( + other.name == name + or other.name in t.primary_key + or r.get(other.name) is None + ): + continue + for target, ref, _declared in _fk_targets( + schema, idx, t, other.name, str(r.get(other.name)) + ): + trow = idx.row(target, str(r.get(other.name))) + if trow is None: + continue + for tc in schema.tables[target].columns: + if tc.name == ref: + continue + if rule_for(trow.get(tc.name), tc.kind, v) in ("exact", "number"): + here.add((other.name, target, ref, tc.name)) + candidates = here if candidates is None else candidates & here + if not candidates: + return None + if candidates and len(candidates) > 1: + # Several columns of the row it points to hold the same value: keep + # the one named like this column (``tva_tx`` from ``tva_tx``, then + # ``subprice`` from ``price``), if exactly one is. + def stem(col: str) -> str: + return re.sub(r"(^fk_|_id$|_fk$|^id_)", "", col.lower()) + + for test in ( + lambda c: c[3] == name, + lambda c: c[3] in name, + lambda c: c[3] in name and c[1].lower().endswith(stem(c[0])), + ): + named = {c for c in candidates if test(c)} + if len(named) == 1: + candidates = named + break + if candidates and len(candidates) == 1: + via, target, ref, source = next(iter(candidates)) + return {"via": via, "table": target, "ref_column": ref, "column": source} + return None diff --git a/packages/flow/openadapt_flow/generate/ensure_contract.py b/packages/flow/openadapt_flow/generate/ensure_contract.py index 4660fd1455..a34a3747bb 100644 --- a/packages/flow/openadapt_flow/generate/ensure_contract.py +++ b/packages/flow/openadapt_flow/generate/ensure_contract.py @@ -824,7 +824,12 @@ def render_yaml(gen: Generated, *, generator: str = "") -> str: def main(argv: Optional[Sequence[str]] = None) -> int: - ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap = argparse.ArgumentParser( + description=__doc__.splitlines()[0] + + " For an app whose data is in MySQL, MariaDB or PostgreSQL, use " + "`python -m openadapt_flow.generate.verify_contract` instead " + "(docs/GENERATE_CONTRACTS.md)." + ) ap.add_argument("teach", help="teach directory") ap.add_argument("--out", help="write the contract YAML here when admitted") ap.add_argument("--report", help="write the JSON report here") diff --git a/packages/flow/openadapt_flow/generate/sql_store.py b/packages/flow/openadapt_flow/generate/sql_store.py new file mode 100644 index 0000000000..db55cab8b8 --- /dev/null +++ b/packages/flow/openadapt_flow/generate/sql_store.py @@ -0,0 +1,530 @@ +"""Read a SQL database's schema and rows for the contract generator. + +The generator (:mod:`openadapt_flow.generate.verify_contract`) needs two +things from the store a write lands in: + +- the **schema**, from ``information_schema`` (MySQL, MariaDB, PostgreSQL): + every base table, its columns with their types, defaults and nullability, + the primary key, declared foreign keys, and the columns the database fills + on its own (auto-increment keys, ``ON UPDATE CURRENT_TIMESTAMP``); +- **row snapshots** before and after a demonstration write, so a diff shows + which tables, rows and columns one kind of write touches. + +Everything is read through the read-only connection the contract will use +(``sql_driver``, ``sql_connect_args``, ``sql_password_env``), with the +driver's read-only session switched on. Nothing here writes, and nothing +calls a model. + +A snapshot holds the rows' values. It is written only into the generator's +own working directory, which the person keeps private, like any database +export. +""" + +from __future__ import annotations + +import datetime as _dt +import decimal +import importlib +import json +import os +from collections.abc import Callable, Mapping, Sequence +from dataclasses import asdict, dataclass, field +from pathlib import Path +from typing import Any, Optional + +#: Dialects this module reads. +DIALECTS = ("mysql", "postgresql") + +#: Above this many rows a table is snapshotted by count and largest key only; +#: a demonstration diff can't say which of its rows changed. +DEFAULT_MAX_ROWS = 20000 + + +class StoreError(RuntimeError): + """The store can't be read as the generator needs.""" + + +@dataclass +class Column: + name: str + data_type: str + nullable: bool + default: Optional[str] + #: The database fills it on insert (auto-increment, a sequence, identity). + auto: bool = False + #: The database rewrites it on every update (``ON UPDATE CURRENT_TIMESTAMP``). + on_update: bool = False + + @property + def kind(self) -> str: + """``text``, ``number``, ``date``, ``datetime``, ``time``, ``bool``, + ``binary`` or ``other``.""" + + t = self.data_type.lower() + if t in ("tinyint(1)", "boolean", "bool"): + return "bool" + if any( + t.startswith(p) + for p in ( + "int", + "tinyint", + "smallint", + "mediumint", + "bigint", + "integer", + "decimal", + "numeric", + "float", + "double", + "real", + ) + ): + return "number" + if t == "date": + return "date" + if t.startswith(("datetime", "timestamp")): + return "datetime" + if t.startswith("time"): + return "time" + if any( + t.startswith(p) + for p in ("char", "varchar", "text", "tinytext", "mediumtext", "longtext") + ) or t in ("character varying", "character", "citext", "enum", "set"): + return "text" + if t.startswith(("character varying", "character")): + return "text" + if any(t.startswith(p) for p in ("blob", "binary", "varbinary", "bytea")): + return "binary" + return "other" + + +@dataclass +class ForeignKey: + column: str + table: str + ref_column: str + + +@dataclass +class Table: + name: str + columns: list[Column] + primary_key: list[str] + foreign_keys: list[ForeignKey] = field(default_factory=list) + + def column(self, name: str) -> Optional[Column]: + for c in self.columns: + if c.name == name: + return c + return None + + @property + def column_names(self) -> list[str]: + return [c.name for c in self.columns] + + +@dataclass +class Schema: + dialect: str + database: str + tables: dict[str, Table] + #: Views the read account sees, by name (they are read, never written). + views: list[str] = field(default_factory=list) + + def to_json(self) -> dict[str, Any]: + return { + "dialect": self.dialect, + "database": self.database, + "views": list(self.views), + "tables": {n: asdict(t) for n, t in self.tables.items()}, + } + + @classmethod + def from_json(cls, data: Mapping[str, Any]) -> "Schema": + tables = {} + for name, t in dict(data["tables"]).items(): + tables[name] = Table( + name=t["name"], + columns=[Column(**c) for c in t["columns"]], + primary_key=list(t["primary_key"]), + foreign_keys=[ForeignKey(**f) for f in t.get("foreign_keys", [])], + ) + return cls( + str(data["dialect"]), + str(data["database"]), + tables, + list(data.get("views", [])), + ) + + +def dialect_of(read: Mapping[str, Any]) -> str: + """``mysql`` or ``postgresql``, from the read's driver.""" + + driver = str(read.get("sql_driver") or "") + if driver in ("pymysql", "MySQLdb", "mysql.connector"): + return "mysql" + if driver in ("psycopg2", "psycopg"): + return "postgresql" + raise StoreError( + f"sql_driver {driver!r}: the generator reads MySQL/MariaDB (pymysql) " + "or PostgreSQL (psycopg2)" + ) + + +def connector(read: Mapping[str, Any]) -> Callable[[], Any]: + """A function that opens one read-only connection, as the contract will.""" + + driver = str(read.get("sql_driver") or "") + dialect_of(read) + module = importlib.import_module(driver) + args = dict(read.get("sql_connect_args") or {}) + env = read.get("sql_password_env") + if env: + if env not in os.environ: + raise StoreError( + f"environment variable {env} (the read password) is not set" + ) + args["password"] = os.environ[str(env)] + + def connect() -> Any: + from openadapt_flow.runtime.effects.sql import read_only_connection + + conn = module.connect(**args) + return read_only_connection(conn, driver) + + return connect + + +def _rows(cur: Any, sql: str, params: Sequence[Any] = ()) -> list[tuple[Any, ...]]: + cur.execute(sql, tuple(params)) if params else cur.execute(sql) + return [tuple(r) for r in cur.fetchall()] + + +def read_schema(connect: Callable[[], Any], dialect: str) -> Schema: + """Every base table the read account sees, with columns, keys and FKs.""" + + conn = connect() + try: + cur = conn.cursor() + if dialect == "mysql": + return _mysql_schema(cur) + if dialect == "postgresql": + return _pg_schema(cur) + raise StoreError(f"dialect {dialect!r} is not one of {DIALECTS}") + finally: + try: + conn.rollback() + finally: + conn.close() + + +def _mysql_schema(cur: Any) -> Schema: + (db,) = _rows(cur, "SELECT DATABASE()")[0] + if not db: + raise StoreError("the connection names no database (sql_connect_args.database)") + kinds = dict( + _rows( + cur, + "SELECT TABLE_NAME, TABLE_TYPE FROM information_schema.TABLES " + "WHERE TABLE_SCHEMA = %s", + (db,), + ) + ) + cols = _rows( + cur, + "SELECT TABLE_NAME, COLUMN_NAME, COLUMN_TYPE, DATA_TYPE, IS_NULLABLE, " + "COLUMN_DEFAULT, EXTRA, COLUMN_KEY FROM information_schema.COLUMNS " + "WHERE TABLE_SCHEMA = %s ORDER BY TABLE_NAME, ORDINAL_POSITION", + (db,), + ) + keys = _rows( + cur, + "SELECT TABLE_NAME, COLUMN_NAME, CONSTRAINT_NAME, REFERENCED_TABLE_NAME, " + "REFERENCED_COLUMN_NAME FROM information_schema.KEY_COLUMN_USAGE " + "WHERE TABLE_SCHEMA = %s ORDER BY TABLE_NAME, CONSTRAINT_NAME, " + "ORDINAL_POSITION", + (db,), + ) + tables: dict[str, Table] = {} + for tname, cname, ctype, dtype, nullable, default, extra, _key in cols: + if kinds.get(tname) != "BASE TABLE": + continue + extra = str(extra or "").lower() + ctype = str(ctype).lower() + t = tables.setdefault(tname, Table(tname, [], [])) + t.columns.append( + Column( + name=str(cname), + data_type="tinyint(1)" + if ctype.startswith("tinyint(1)") + else str(dtype).lower(), + nullable=str(nullable).upper() == "YES", + default=None if default is None else str(default), + auto="auto_increment" in extra, + on_update="on update" in extra, + ) + ) + for tname, cname, cons, rtable, rcol in keys: + kt = tables.get(tname) + if kt is None: + continue + t = kt + if cons == "PRIMARY": + t.primary_key.append(str(cname)) + elif rtable: + t.foreign_keys.append(ForeignKey(str(cname), str(rtable), str(rcol))) + views = sorted(n for n, k in kinds.items() if k == "VIEW") + return Schema("mysql", str(db), tables, views) + + +def _pg_schema(cur: Any) -> Schema: + (schema,) = _rows(cur, "SELECT current_schema()")[0] + (db,) = _rows(cur, "SELECT current_database()")[0] + kinds = dict( + _rows( + cur, + "SELECT table_name, table_type FROM information_schema.tables " + "WHERE table_schema = %s", + (schema,), + ) + ) + cols = _rows( + cur, + "SELECT table_name, column_name, data_type, is_nullable, column_default, " + "is_identity FROM information_schema.columns WHERE table_schema = %s " + "ORDER BY table_name, ordinal_position", + (schema,), + ) + tables: dict[str, Table] = {} + for tname, cname, dtype, nullable, default, identity in cols: + if kinds.get(tname) != "BASE TABLE": + continue + t = tables.setdefault(tname, Table(tname, [], [])) + default_s = None if default is None else str(default) + t.columns.append( + Column( + name=str(cname), + data_type=str(dtype).lower(), + nullable=str(nullable).upper() == "YES", + default=default_s, + auto=(default_s or "").startswith("nextval(") + or str(identity).upper() == "YES", + ) + ) + # pg_catalog: information_schema hides constraints of tables the account + # doesn't own, and the read account owns none. + cons = _rows( + cur, + "SELECT c.conrelid::regclass::text, c.contype, a.attname, " + "c.confrelid::regclass::text, fa.attname " + "FROM pg_constraint c " + "JOIN pg_namespace n ON n.oid = c.connamespace " + "CROSS JOIN LATERAL unnest(c.conkey) WITH ORDINALITY AS k(attnum, ord) " + "JOIN pg_attribute a ON a.attrelid = c.conrelid AND a.attnum = k.attnum " + "LEFT JOIN pg_attribute fa ON fa.attrelid = c.confrelid " + "AND fa.attnum = c.confkey[k.ord] " + "WHERE n.nspname = %s AND c.contype IN ('p', 'f') " + "ORDER BY 1, c.conname, k.ord", + (schema,), + ) + for tname, ctype, cname, rtable, rcol in cons: + ct = tables.get(str(tname).split(".")[-1].strip('"')) + if ct is None: + continue + t = ct + if ctype == "p": + t.primary_key.append(str(cname)) + elif rtable and rcol: + t.foreign_keys.append( + ForeignKey(str(cname), str(rtable).split(".")[-1].strip('"'), str(rcol)) + ) + views = sorted(n for n, k in kinds.items() if k == "VIEW") + return Schema("postgresql", str(db), tables, views) + + +def quote(dialect: str, name: str) -> str: + """A quoted identifier. Names with a quote character are refused.""" + + if "`" in name or '"' in name or "\x00" in name: + raise StoreError(f"identifier {name!r} holds a quote character") + return f"`{name}`" if dialect == "mysql" else f'"{name}"' + + +def cell_text(value: Any) -> Optional[str]: + """One cell as text, the way the generator compares values: ``None`` + stays ``None``; bytes are ``0x`` + hex; dates are ISO; a Decimal keeps its + stored digits.""" + + if value is None: + return None + if isinstance(value, (bytes, bytearray, memoryview)): + return "0x" + bytes(value).hex() + if isinstance(value, _dt.datetime): + return value.isoformat(sep=" ") + if isinstance(value, (_dt.date, _dt.time)): + return value.isoformat() + if isinstance(value, decimal.Decimal): + return format(value, "f") + if isinstance(value, bool): + return "true" if value else "false" + if isinstance(value, float) and value.is_integer(): + return str(int(value)) + if isinstance(value, (dict, list)): + return json.dumps(value, sort_keys=True) + return str(value) + + +@dataclass +class TableRows: + #: Row key (the primary key's values joined by ``|``, or the whole row + #: for a table without one) -> column -> text. + rows: dict[str, dict[str, Optional[str]]] + #: Set when the table had more rows than the snapshot keeps. + count_only: Optional[int] = None + + +def row_key(table: Table, row: Mapping[str, Optional[str]]) -> str: + cols = table.primary_key or table.column_names + return "|".join("\x00" if row.get(c) is None else str(row.get(c)) for c in cols) + + +def snapshot( + connect: Callable[[], Any], + schema: Schema, + *, + max_rows: int = DEFAULT_MAX_ROWS, +) -> dict[str, TableRows]: + """Every table's rows as text, in one read-only transaction.""" + + conn = connect() + out: dict[str, TableRows] = {} + try: + cur = conn.cursor() + if schema.dialect == "mysql": + cur.execute("START TRANSACTION WITH CONSISTENT SNAPSHOT, READ ONLY") + else: + cur.execute("SET TRANSACTION ISOLATION LEVEL REPEATABLE READ") + for name in sorted(schema.tables): + table = schema.tables[name] + q = quote(schema.dialect, name) + (count,) = _rows(cur, f"SELECT COUNT(*) FROM {q}")[0] + if int(count) > max_rows: + out[name] = TableRows({}, count_only=int(count)) + continue + cols = table.column_names + cur.execute( + "SELECT " + + ", ".join(quote(schema.dialect, c) for c in cols) + + f" FROM {q}" + ) + rows: dict[str, dict[str, Optional[str]]] = {} + for raw in cur.fetchall(): + row = {c: cell_text(v) for c, v in zip(cols, raw)} + key = row_key(table, row) + while key in rows: # identical rows in a table without a key + key += "+" + rows[key] = row + out[name] = TableRows(rows) + finally: + try: + conn.rollback() + finally: + conn.close() + return out + + +def save_snapshot(path: Path, rows: Mapping[str, TableRows]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + data = {n: {"rows": t.rows, "count_only": t.count_only} for n, t in rows.items()} + tmp = path.with_suffix(".tmp") + tmp.write_text(json.dumps(data, sort_keys=True), encoding="utf-8") + os.chmod(tmp, 0o600) + tmp.replace(path) + + +def load_snapshot(path: Path) -> dict[str, TableRows]: + data = json.loads(path.read_text(encoding="utf-8")) + return {n: TableRows(dict(t["rows"]), t.get("count_only")) for n, t in data.items()} + + +@dataclass +class RowChange: + table: str + key: str + kind: str # added | removed | changed + before: Optional[dict[str, Optional[str]]] + after: Optional[dict[str, Optional[str]]] + + @property + def changed_columns(self) -> list[str]: + if self.before is None or self.after is None: + return [] + return sorted( + c + for c in set(self.before) | set(self.after) + if self.before.get(c) != self.after.get(c) + ) + + def to_json(self) -> dict[str, Any]: + return asdict(self) + + +@dataclass +class Diff: + changes: list[RowChange] + #: Tables too large to diff row by row whose count changed. + unread: dict[str, tuple[int, int]] = field(default_factory=dict) + + def tables(self) -> list[str]: + return sorted({c.table for c in self.changes} | set(self.unread)) + + def of(self, table: str, kind: Optional[str] = None) -> list[RowChange]: + return [ + c + for c in self.changes + if c.table == table and (kind is None or c.kind == kind) + ] + + def to_json(self) -> dict[str, Any]: + return { + "changes": [c.to_json() for c in self.changes], + "unread": {k: list(v) for k, v in self.unread.items()}, + } + + @classmethod + def from_json(cls, data: Mapping[str, Any]) -> "Diff": + return cls( + [RowChange(**c) for c in data.get("changes", [])], + { + k: (int(v[0]), int(v[1])) + for k, v in dict(data.get("unread", {})).items() + }, + ) + + +def diff(before: Mapping[str, TableRows], after: Mapping[str, TableRows]) -> Diff: + """Rows added, removed or changed, table by table.""" + + changes: list[RowChange] = [] + unread: dict[str, tuple[int, int]] = {} + for name in sorted(set(before) | set(after)): + a, b = before.get(name), after.get(name) + if a is None or b is None: + unread[name] = ( + 0 if a is None else len(a.rows), + 0 if b is None else len(b.rows), + ) + continue + if a.count_only is not None or b.count_only is not None: + ca = a.count_only if a.count_only is not None else len(a.rows) + cb = b.count_only if b.count_only is not None else len(b.rows) + if ca != cb: + unread[name] = (ca, cb) + continue + for key in sorted(set(a.rows) | set(b.rows)): + ra, rb = a.rows.get(key), b.rows.get(key) + if ra == rb: + continue + kind = "added" if ra is None else "removed" if rb is None else "changed" + changes.append(RowChange(name, key, kind, ra, rb)) + return Diff(changes, unread) diff --git a/packages/flow/openadapt_flow/generate/verify_contract.py b/packages/flow/openadapt_flow/generate/verify_contract.py new file mode 100644 index 0000000000..2b472d69da --- /dev/null +++ b/packages/flow/openadapt_flow/generate/verify_contract.py @@ -0,0 +1,3690 @@ +"""Propose ``openadapt.verify/v1`` contracts from a SQL schema and demonstrations. + + python -m openadapt_flow.generate.verify_contract --help + +A person setting up the verifier for an app used to write every contract by +hand, with 10 or 11 decisions per app. This generator writes them from: + +- the store's schema (``information_schema`` on MySQL, MariaDB or + PostgreSQL), read through the read-only account the contracts will use; +- a short description of each kind of write: its parameter names and one or + two example tasks (the ``kinds.yaml`` a setup agent is given); +- one demonstration write per example, which it snapshots before and after + to see which tables and columns the kind touches. + +For each kind it proposes one contract per field set the kind's tasks can +use (an update that may change the phone, the email or both gets a contract +for each), reading the whole database so an extra record anywhere is seen, +and a manifest that picks the contract for a task's values +(:func:`choose`). Whatever it can't decide alone becomes one numbered +question in ``QUESTIONS.md``; the goal is at most three per app. Answers go +in ``answers.yaml`` and the generator is run again. + +It never emits a contract that reads something that doesn't exist: every +contract is loaded and read once against the live store (a dry read) before +it is written out, and a contract that needs a database view gets it created +(``--create-views``, through an admin connection) or is refused with the +reason. See ``docs/GENERATE_CONTRACTS.md``. Nothing here calls a model. +""" + +from __future__ import annotations + +import argparse +import datetime as _dt +import hashlib +import itertools +import json +import os +import re +import sys +import tempfile +from collections.abc import Mapping, Sequence +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Optional + +import yaml + +from openadapt_flow.generate import sql_store as store +from openadapt_flow.generate.demo_analysis import ( + ColumnRole, + Demo, + Hop, + Location, + _hop_cost, + _rank, + best, + column_roles, + is_identifier, + link_columns, + locate, + pick_anchor, +) +from openadapt_flow.generate.sql_store import RowChange, Schema + +MANIFEST_SCHEMA = "openadapt.contract-manifest/v1" +#: Most contracts one kind gets (all subsets of up to six optional fields). +MAX_VARIANTS = 64 +#: Longest list a list parameter gets a contract for. +MAX_LIST = 10 +#: The recheck window proposed when the person hasn't answered. +DEFAULT_RECHECK_S = 300 + + +class GenerateError(RuntimeError): + """The generator can't go on; the message says why.""" + + +# -- inputs ------------------------------------------------------------------- + + +def load_yaml(path: Path) -> Any: + try: + return yaml.safe_load(Path(path).read_text(encoding="utf-8")) + except (OSError, yaml.YAMLError) as exc: + raise GenerateError(f"cannot read {path}: {exc}") from exc + + +def load_read(path: Path) -> dict[str, Any]: + """``read.yaml``: the read-only connection, as in a ``kind: sql`` + verifier (``sql_driver``, ``sql_connect_args``, ``sql_password_env``).""" + + data = load_yaml(path) + if not isinstance(data, dict) or not data.get("sql_driver"): + raise GenerateError(f"{path}: needs sql_driver and sql_connect_args") + if "password" in dict(data.get("sql_connect_args") or {}): + raise GenerateError( + f"{path}: put the password in an environment variable named by " + "sql_password_env, not in the file" + ) + return { + k: data[k] + for k in ("sql_driver", "sql_connect_args", "sql_password_env") + if k in data + } + + +@dataclass +class KindSpec: + kind: str + parameters: list[str] + description: str = "" + examples: list[dict[str, Any]] = field(default_factory=list) + + +def load_kinds(path: Path) -> list[KindSpec]: + data = load_yaml(path) or {} + kinds = data.get("kinds") if isinstance(data, dict) else None + if not isinstance(kinds, list) or not kinds: + raise GenerateError(f"{path}: needs a list under kinds:") + out = [] + for k in kinds: + if not isinstance(k, dict) or not k.get("kind") or not k.get("parameters"): + raise GenerateError(f"{path}: each kind needs kind: and parameters:") + out.append( + KindSpec( + str(k["kind"]), + [str(p) for p in k["parameters"]], + str(k.get("description") or ""), + [ + { + "instruction": e.get("instruction", ""), + "values": dict(e.get("values") or {}), + } + for e in k.get("examples") or [] + ], + ) + ) + return out + + +def load_demos(demo_dir: Path) -> list[Demo]: + demos = [] + for f in sorted(Path(demo_dir).glob("*.demo.json")): + d = json.loads(f.read_text(encoding="utf-8")) + after = store.load_snapshot(Path(demo_dir) / d["after_snapshot"]) + demos.append( + Demo( + kind=d["kind"], + values=dict(d["values"]), + diff=store.Diff.from_json(d["diff"]), + after=after, + started_at=d.get("started_at", ""), + finished_at=d.get("finished_at", ""), + name=f.name, + ) + ) + return demos + + +# -- questions ------------------------------------------------------------------ + + +@dataclass +class Question: + key: str + text: str + proposal: Any + options: list[str] = field(default_factory=list) + + +def _answer(answers: Mapping[str, Any], key: str) -> Any: + return (answers or {}).get(key) + + +# -- the plan for one kind -------------------------------------------------------- + + +@dataclass +class ChildGroup: + """Rows a write adds beside the main row that hold a parameter's value.""" + + table: str + link: Hop # child hop from the anchor + params: dict[str, Location] # param -> location relative to the child row + roles: dict[str, ColumnRole] + count: int = 1 + #: List parameter whose items these rows are, if any. + list_param: Optional[str] = None + changed_columns: list[str] = field(default_factory=list) + #: Other parameters this row also points to (a contact's account link + #: row also holds the title): with one of them given and this one not, + #: the row is still expected, with this value empty. + co_owners: list[str] = field(default_factory=list) + #: The app adds this row with an empty value when the task leaves the + #: parameter out (Redmine's custom field values): expected then too. + empty_when_null: bool = False + #: The parameter whose value the rows hold (``children`` may hold a + #: second group for it: a call's lead is its parent and an attendee). + param: str = "" + #: An update replaced earlier rows here (removed or changed rows linked + #: to the record), leaving ``after_count`` of them. + replaced: bool = False + after_count: int = 1 + + +@dataclass +class SideSpec: + table: str + count: int + roles: dict[str, ColumnRole] + #: Used only when this parameter is not in the field set (the table holds + #: that parameter's row otherwise). + unless: Optional[str] = None + #: Added rows of other tables these rows hold the key of (for ownership). + owner: Optional[str] = None + + +@dataclass +class KindPlan: + kind: str + mode: str = "" # create | update + anchor: str = "" + params: list[str] = field(default_factory=list) + optional: list[str] = field(default_factory=list) + identity: list[str] = field(default_factory=list) + changes: list[str] = field(default_factory=list) + locations: dict[str, Location] = field(default_factory=dict) + #: dispatch value -> (value param -> location) + dispatch: Optional[str] = None + dispatch_locations: dict[str, dict[str, Location]] = field(default_factory=dict) + children: dict[str, ChildGroup] = field(default_factory=dict) # by param + lists: dict[str, ChildGroup] = field(default_factory=dict) + added_refs: dict[str, dict[str, Any]] = field(default_factory=dict) + sides: list[SideSpec] = field(default_factory=list) + logs: set[str] = field(default_factory=set) + bookkeeping: dict[str, set[str]] = field(default_factory=dict) + anchor_changed: set[str] = field(default_factory=set) + demos: list[Demo] = field(default_factory=list) + anchors: list[RowChange] = field(default_factory=list) + unresolved: dict[str, list[Location]] = field(default_factory=dict) + refused: list[str] = field(default_factory=list) + removed: dict[str, int] = field(default_factory=dict) + #: dispatch value -> group name -> rows that hold the value param. + dispatch_children: dict[str, dict[str, "ChildGroup"]] = field(default_factory=dict) + #: unresolved key -> (parameter, dispatch value or None). + unresolved_meta: dict[str, tuple[str, Optional[str]]] = field(default_factory=dict) + #: Parameters placed by name, not by a demonstration. + by_name: list[str] = field(default_factory=list) + #: Parameters placed where another kind's demonstrations found them. + shared: list[str] = field(default_factory=list) + #: Existing rows (not the target) the demonstrations changed with no + #: task value: each the same row every time (by key) or the row a + #: located value names (an account behind the contact's key), with the + #: columns that moved and what kind of change each was. + touched: list[dict[str, Any]] = field(default_factory=list) + #: The target's own columns that moved with no task value, by kind. + anchor_extra: dict[str, str] = field(default_factory=dict) + #: Changed rows nothing identifies: a write that changes them stops. + unexcused: dict[str, set[str]] = field(default_factory=dict) + #: Tables whose added rows differ between demonstrations: logs only if a + #: person says so. + log_candidates: set[str] = field(default_factory=set) + #: Where the target's versions are read (``frame.versions``): a version + #: column that steps by one per save, or an audit table naming the + #: record and the user who saved it. None when the demonstrations show + #: neither. + versions: Optional[dict[str, Any]] = None + #: More places a value is in: rows the write added beside the record + #: (a call's lead is its parent and also an attendee row). + extra: dict[str, list[Location]] = field(default_factory=dict) + + +def _scalars(values: Mapping[str, Any]) -> dict[str, Any]: + return { + k: v + for k, v in values.items() + if v is not None and not isinstance(v, (list, dict)) + } + + +def _intersect(per_demo: list[list[Location]]) -> list[Location]: + if not per_demo: + return [] + common = {loc.signature(): loc for loc in per_demo[0]} + for locs in per_demo[1:]: + sigs = {loc.signature() for loc in locs} + common = {s: loc for s, loc in common.items() if s in sigs} + return list(common.values()) + + +def _as_change(table: str, row: Mapping[str, Optional[str]]) -> RowChange: + return RowChange(table, "", "added", None, dict(row)) + + +def analyze( + schema: Schema, + spec: KindSpec, + demos: Sequence[Demo], + actor_values: Mapping[str, str], +) -> KindPlan: + plan = KindPlan(spec.kind, params=list(spec.parameters)) + mine = [d for d in demos if d.kind == spec.kind and d.diff.changes] + if not mine: + plan.refused.append( + "no demonstration write of this kind (record one with " + "`verify_contract record`)" + ) + return plan + examples = [e["values"] for e in spec.examples] + [d.values for d in mine] + plan.optional = sorted( + p for p in spec.parameters if any(v.get(p) is None for v in examples) + ) + for d in mine: + anchor, mode, score = pick_anchor(schema, d, spec.parameters) + if anchor is None or score == 0: + plan.refused.append( + f"{d.name}: no row the write touched holds any of the task's values" + ) + return plan + if plan.anchor and (anchor.table != plan.anchor or mode != plan.mode): + plan.refused.append( + f"{d.name}: the main record is {anchor.table} ({mode}), but other " + f"demonstrations write {plan.anchor} ({plan.mode})" + ) + return plan + plan.anchor, plan.mode = anchor.table, mode + plan.demos.append(d) + plan.anchors.append(anchor) + lists = [ + p + for p in spec.parameters + if any(isinstance(d.values.get(p), list) for d in plan.demos) + ] + links = link_columns(schema, list(zip(plan.demos, plan.anchors))) + scalars = [p for p in spec.parameters if p not in lists] + + # Where each value is, in every demonstration that gives it. + found: dict[str, list[list[Location]]] = {p: [] for p in scalars} + roles_of: dict[str, set[str]] = {p: set() for p in scalars} + for d, a in zip(plan.demos, plan.anchors): + for p, v in _scalars(d.values).items(): + if p not in found: + continue + if plan.mode == "update": + change = locate(schema, d, a, v, changed_only=True, links=links) + ident = [ + loc + for loc in locate(schema, d, a, v, before=True) + if not loc.hops or loc.hops[0].kind == "fk" + ] + if ident and (not change or _rank(ident[0]) < _rank(change[0])): + found[p].append(ident) + roles_of[p].add("identity") + else: + found[p].append(change) + roles_of[p].add("change" if change else "missing") + else: + found[p].append(locate(schema, d, a, v, links=links)) + # A parameter whose value names a field (``field: status``). + for p in scalars: + values = [d.values.get(p) for d in plan.demos if d.values.get(p) is not None] + if ( + len(values) >= 2 + and all(is_identifier(v) for v in values) + and len(set(values)) == len(values) + and all(not locs for locs in found[p]) + and plan.dispatch is None + ): + plan.dispatch = p + for p in scalars: + if p == plan.dispatch: + continue + if plan.dispatch is not None and "identity" not in roles_of[p]: + by_value: dict[str, list[list[Location]]] = {} + for d, locs in zip( + [d for d in plan.demos if d.values.get(p) is not None], found[p] + ): + by_value.setdefault(str(d.values.get(plan.dispatch)), []).append(locs) + if len(by_value) > 1: + ok = True + for dv, per in by_value.items(): + loc, tied = _pick(p, _intersect(per), plan.demos, schema) + if loc is None: + key = f"{p} (when {plan.dispatch} is {dv})" + plan.unresolved[key] = tied + plan.unresolved_meta[key] = (p, dv) + ok = False + else: + plan.dispatch_locations.setdefault(dv, {})[p] = loc + if ok: + plan.changes.append(p) + continue + all_found = _intersect(found[p]) + loc, tied = _pick(p, all_found, plan.demos, schema) + if loc is not None: + plan.extra[p] = [ + x + for x in all_found + if x.hops + and x.hops[0].kind == "child" + and x != loc + # in another table than the main place, through keys named + # after their tables (a call's lead attendee row) + and (not loc.hops or x.hops[0].table != loc.hops[0].table) + and all(_hop_cost(h) <= 1 for h in x.hops[1:]) + ] + if loc is None: + if not found[p]: + named = _by_name(schema, plan, p) + if named is not None: + plan.locations[p] = named + plan.by_name.append(p) + if plan.mode == "update": + plan.changes.append(p) + continue # never given: no contract uses it + plan.unresolved[p] = tied + plan.unresolved_meta[p] = (p, None) + continue + plan.locations[p] = loc + if plan.mode == "update": + (plan.identity if "identity" in roles_of[p] else plan.changes).append(p) + # A column one value is already in is not another value's place. + progress = True + while progress and plan.unresolved: + progress = False + claimed = { + (loc.hops[0].column if loc.hops else c) + for loc in plan.locations.values() + for c in loc.columns + } + for name, tied in list(plan.unresolved.items()): + left = [ + loc + for loc in tied + if (loc.hops[0].column if loc.hops else loc.columns[0]) not in claimed + ] + if len(left) == 1 and name in plan.params: + plan.locations[name] = left[0] + plan.unresolved.pop(name) + if plan.mode == "update" and name not in plan.identity: + plan.changes.append(name) + progress = True + if plan.mode == "create": + # A create task may leave out any value whose place can be empty: a + # nullable column, a nullable key column, or a row beside the record. + anchor_t = schema.tables[plan.anchor] + for p, loc in plan.locations.items(): + col = None + if not loc.hops and len(loc.columns) == 1: + col = anchor_t.column(loc.columns[0]) + elif loc.hops and loc.hops[0].kind == "fk": + col = anchor_t.column(loc.hops[0].column) + if (col is not None and col.nullable) or ( + loc.hops and loc.hops[0].kind == "child" + ): + if p not in plan.optional: + plan.optional.append(p) + plan.optional.sort() + if plan.mode == "update" and not plan.identity: + plan.refused.append( + "no parameter names the record before the write, so the update's " + "target can't be told apart from other records" + ) + _children(schema, plan, lists) + _sides(schema, plan, actor_values, demos) + return plan + + +def _share_locations( + schema: Schema, plans: Sequence[KindPlan], kinds: Sequence[KindSpec] +) -> None: + """An update parameter no example of its kind gives (a priority), placed + where another kind's demonstrations found the parameter of the same name + on the same main table (``issues.priority_id -> enumerations.name``). + Placed this way, it replaces a placement by name.""" + + for plan in plans: + if plan.mode != "update" or not plan.anchor: + continue + for name in plan.params: + if name in plan.identity or name == plan.dispatch: + continue + if name in plan.locations and name not in plan.by_name: + continue + for other in plans: + if other is plan or other.anchor != plan.anchor: + continue + loc = other.locations.get(name) + if loc is None or name in other.by_name: + continue + if loc.hops and loc.hops[0].kind != "fk": + continue + plan.locations[name] = loc + if name in plan.by_name: + plan.by_name.remove(name) + plan.shared.append(name) + if name not in plan.changes: + plan.changes.append(name) + break + + +def _pick( + param: str, + found: Sequence[Location], + demos: Sequence[Demo] = (), + schema: Optional[Schema] = None, +) -> tuple[Optional[Location], list[Location]]: + """The best location. Among tied ones: the one whose column is named + like the parameter (``stage``, not ``last_stage``); else drop a column + that is only a copy of another tied one, lower-cased or digits only, in + every row of its table (EspoCRM's ``email_address.lower``, + ``phone_number.numeric``).""" + + loc, tied = best(found) + if loc is not None or not tied: + return loc, tied + name = re.sub(r"^(new|set)_", "", param) + named = [t for t in tied if t.columns and t.columns[-1] == name] + if len(named) == 1: + return named[0], [] + keep = [t for t in tied if not _derived_copy(t, tied, demos)] + if len(keep) == 1: + return keep[0], [] + same = _identical_columns(keep, demos, schema) + if same is not None: + return same, [] + return None, tied + + +def _identical_columns( + tied: Sequence[Location], demos: Sequence[Demo], schema: Optional[Schema] = None +) -> Optional[Location]: + """Tied columns of one row that hold the same value in every row of + their table can't be told apart by any data: keep the one the table + defines first (the source; a copy is added after it). Either compares + exactly, so a wrong pick stops a write, never passes one.""" + + if not demos or len(tied) < 2: + return None + first = tied[0] + if any( + t.hops != first.hops or len(t.columns) != 1 or t.rule != first.rule + for t in tied + ): + return None + table = first.hops[-1].table if first.hops else None + rows = demos[0].after.get(table) if table else None + if rows is None or rows.count_only is not None or not rows.rows: + return None + cols = [t.columns[0] for t in tied] + if not all(len({r.get(c) for c in cols}) == 1 for r in rows.rows.values()): + return None + if schema is None or table not in schema.tables: + return None + order = schema.tables[table].column_names + return min( + tied, key=lambda t: order.index(t.columns[0]) if t.columns[0] in order else 999 + ) + + +def _derived_copy( + loc: Location, tied: Sequence[Location], demos: Sequence[Demo] +) -> bool: + if len(loc.columns) != 1 or not loc.hops or not demos: + return False + table = loc.hops[-1].table + rows = demos[0].after.get(table) + if rows is None or rows.count_only is not None or not rows.rows: + return False + + def digits(v: str) -> str: + return "".join(ch for ch in v if ch.isdigit()) + + for other in tied: + if other is loc or other.hops != loc.hops or len(other.columns) != 1: + continue + a, b = other.columns[0], loc.columns[0] + for fn in (str.lower, digits): + if all( + r.get(b) == (fn(str(r[a])) if r.get(a) is not None else None) + for r in rows.rows.values() + ) and any(r.get(a) != r.get(b) for r in rows.rows.values()): + return True + return False + + +def _all_locations(plan: KindPlan) -> list[tuple[str, Location]]: + out = list(plan.locations.items()) + for locs in plan.dispatch_locations.values(): + out.extend(locs.items()) + for g in [*plan.children.values(), *plan.lists.values()]: + out.extend(g.params.items()) + return out + + +def _confirm_phones(plan: KindPlan, confirmed: set[str]) -> None: + """A phone rule a person didn't confirm compares the text exactly.""" + + def fix(name: str, loc: Location) -> Location: + if loc.rule == "phone" and name not in confirmed: + return Location(loc.hops, loc.columns, "exact") + return loc + + plan.locations = {k: fix(k, v) for k, v in plan.locations.items()} + plan.dispatch_locations = { + dv: {k: fix(k, v) for k, v in locs.items()} + for dv, locs in plan.dispatch_locations.items() + } + for g in [*plan.children.values(), *plan.lists.values()]: + g.params = {k: fix(k, v) for k, v in g.params.items()} + + +def _resolve(plan: KindPlan, key: str, loc: Location) -> None: + """Place an unresolved value (answered, or settled by a tie-break).""" + + param, dv = plan.unresolved_meta.get(key, (key, None)) + plan.unresolved.pop(key, None) + if dv is not None: + plan.dispatch_locations.setdefault(dv, {})[param] = loc + if param not in plan.changes: + plan.changes.append(param) + return + plan.locations[param] = loc + if plan.mode == "update" and param not in plan.identity + plan.changes: + plan.changes.append(param) + + +def _tail(plan: KindPlan, loc: Location) -> tuple[str, tuple[str, ...]]: + return (loc.hops[-1].table if loc.hops else plan.anchor, loc.columns) + + +def _break_ties_across(plans: Sequence[KindPlan]) -> None: + """A value tied between columns of one table (an email's ``name`` and + ``lower``) goes where another kind's demonstrations put the same kind + of value, if exactly one tied place is one of those.""" + + for plan in plans: + for key, tied in list(plan.unresolved.items()): + if not tied: + continue + seen = { + _tail(o, loc) + for o in plans + if o is not plan + for loc in [ + *o.locations.values(), + *[x for d in o.dispatch_locations.values() for x in d.values()], + ] + } + hits = [t for t in tied if _tail(plan, t) in seen] + if len(hits) == 1: + _resolve(plan, key, hits[0]) + + +def _by_name(schema: Schema, plan: KindPlan, param: str) -> Optional[Location]: + """An update parameter no demonstration gives (``new_zip``), placed at + the main table's column of the same name after a leading ``new_`` or + ``set_`` (``zip``), when one exists. Compared exactly by the column's + type. Reported as placed by name; a demonstration that gives it replaces + this.""" + + if not plan.anchor: + return None + name = re.sub(r"^(new|set)_", "", param) + col = schema.tables[plan.anchor].column(name) + if col is None or name in schema.tables[plan.anchor].primary_key: + return None + rule = { + "number": "number", + "bool": "number", + "date": "date", + "datetime": "datetime", + }.get(col.kind, "exact") + if col.kind not in ("text", "number", "bool", "date", "datetime"): + return None + return Location((), (name,), rule) + + +def _children( + schema: Schema, plan: KindPlan, lists: Sequence[str], only_missing: bool = False +) -> None: + """Group child-row locations, and find each list parameter's rows.""" + + todo: list[tuple[str, str, Location, Optional[str]]] = [] + for p, loc in plan.locations.items(): + todo.append((p, p, loc, None)) + for i, x in enumerate(plan.extra.get(p, [])): + todo.append((f"{p}#{i + 2}", p, x, None)) + for value, locs in plan.dispatch_locations.items(): + for p, loc in locs.items(): + todo.append((f"{p}@{value}", p, loc, value)) + for gname, p, loc, dv in todo: + if not loc.hops or loc.hops[0].kind != "child": + continue + if only_missing and ( + gname in plan.children or gname in plan.dispatch_children.get(dv or "", {}) + ): + continue + rel = Location(loc.hops[1:], loc.columns, loc.rule) + rows = [] + for d, a in zip(plan.demos, plan.anchors): + if d.values.get(p) is None: + continue + if dv is not None and str(d.values.get(plan.dispatch or "")) != dv: + continue + for ch in d.diff.of(loc.hops[0].table, "added"): + if ch.after is None: + continue + key = (a.after or {}).get(loc.hops[0].ref_column) + if ch.after.get(loc.hops[0].column) != key: + continue + if locate( + schema, + d, + _as_change(ch.table, ch.after), + d.values[p], + max_hops=len(rel.hops), + ): + rows.append((d, ch.after)) + break + explained = {loc.hops[0].column, *rel.columns} + explained |= {h.column for h in rel.hops[:1]} + co: list[str] = [] + for q in plan.params: + if q == p or not rows: + continue + hits = [ + locate( + schema, d, _as_change(loc.hops[0].table, r), d.values[q], max_hops=1 + ) + for d, r in rows + if d.values.get(q) is not None + and not isinstance(d.values.get(q), (list, dict)) + ] + if hits and all( + h + and h[0].hops + and _hop_cost(h[0].hops[0]) <= 1 + and h[0].hops[0].table != plan.anchor + for h in hits + ): + co.append(q) + explained |= {h[0].hops[0].column for h in hits} + group = ChildGroup( + loc.hops[0].table, + loc.hops[0], + {p: rel}, + column_roles(schema, loc.hops[0].table, rows, explained, {}), + 1, + co_owners=co, + param=p, + ) + if dv is None: + plan.children[gname] = group + else: + plan.dispatch_children.setdefault(dv, {})[gname] = group + for p in lists: + per_key: dict[str, list[list[Location]]] = {} + table = link = None + rows = [] + for d, a in zip(plan.demos, plan.anchors): + items = d.values.get(p) + if not isinstance(items, list): + continue + key = ( + (a.after or {}).get(schema.tables[plan.anchor].primary_key[0]) + if schema.tables[plan.anchor].primary_key + else None + ) + children = [ + ch + for ch in d.diff.changes + if ch.kind == "added" + and ch.after is not None + and ch.table != plan.anchor + and key is not None + and key in ch.after.values() + ] + used: set[int] = set() + for item in items: + if not isinstance(item, dict): + plan.refused.append(f"{p}: list items must be mappings") + return + for i, ch in enumerate(children): + if i in used or ch.after is None: + continue + after = ch.after + found_at = { + k: locate(schema, d, _as_change(ch.table, after), v, max_hops=1) + for k, v in item.items() + } + if all(found_at.values()): + used.add(i) + link_col = next(c for c, v in after.items() if v == key) + table = ch.table + link = Hop( + "child", + ch.table, + link_col, + schema.tables[plan.anchor].primary_key[0], + ) + for k, ls in found_at.items(): + per_key.setdefault(k, []).append(ls) + rows.append((d, after)) + break + else: + plan.unresolved[f"{p} item {item}"] = [] + if table is None or link is None: + plan.unresolved.setdefault(p, []) + continue + item_locs: dict[str, Location] = {} + for k, per in per_key.items(): + item_loc, tied = best(_intersect(per)) + if item_loc is None: + plan.unresolved[f"{p}.{k}"] = tied + return + item_locs[k] = item_loc + explained = {link.column} | {c for x in item_locs.values() for c in x.columns} + explained |= {x.hops[0].column for x in item_locs.values() if x.hops} + plan.lists[p] = ChildGroup( + table, + link, + item_locs, + column_roles(schema, table, rows, explained, {}), + list_param=p, + ) + + +def _all_groups(plan: KindPlan, dv: Optional[str] = None) -> list[ChildGroup]: + """Every child group; with ``dv``, the plain ones and that dispatch + value's own.""" + + out = list(plan.children.values()) + for value, groups in plan.dispatch_children.items(): + if dv is None or value == dv: + out.extend(groups.values()) + return out + + +def _demo_groups(plan: KindPlan, d: Demo) -> list[ChildGroup]: + dv = str(d.values.get(plan.dispatch)) if plan.dispatch else None + return _all_groups(plan, dv) if dv is not None else list(plan.children.values()) + + +def _sides( + schema: Schema, + plan: KindPlan, + actor_values: Mapping[str, str], + all_demos: Sequence[Demo] = (), +) -> None: + """Classify every other row the demonstrations touched.""" + + child_tables = {g.table: g.param for g in _all_groups(plan)} + list_tables = {g.table for g in plan.lists.values()} + pure: dict[str, list[list[tuple[Demo, dict[str, Optional[str]]]]]] = {} + counts: dict[str, list[int]] = {} + touched: list[tuple[Demo, RowChange, RowChange]] = [] + for d, a in zip(plan.demos, plan.anchors): + added_keys: set[str] = set() + for ch in d.diff.changes: + if ch.kind == "added" and ch.after is not None: + at = schema.tables.get(ch.table) + if at is not None and len(at.primary_key) == 1: + v = ch.after.get(at.primary_key[0]) + if v is not None: + added_keys.add(v) + per_table: dict[str, list[dict[str, Optional[str]]]] = {} + for ch in d.diff.changes: + if ch is a or ch.table == plan.anchor and ch.key == a.key: + continue + if ch.kind == "removed": + if ch.table in child_tables or ch.table in list_tables: + _mark_replaced(schema, plan, d, a, ch) + continue + plan.removed[ch.table] = plan.removed.get(ch.table, 0) + 1 + continue + if ch.kind == "changed": + linked = ( + ch.table in child_tables + and a.after is not None + and any( + v == (a.after or {}).get(plan_pk(schema, plan.anchor)) + for v in (ch.after or {}).values() + ) + ) + if linked: + _mark_replaced(schema, plan, d, a, ch) + g = next(x for x in _all_groups(plan) if x.table == ch.table) + g.changed_columns = sorted( + set(g.changed_columns) | set(ch.changed_columns) + ) + elif ch.table != plan.anchor or ch.key != a.key: + touched.append((d, a, ch)) + continue + # added + if ch.table in list_tables: + continue + if ch.table in child_tables and _is_param_row(schema, plan, d, ch): + per_table.setdefault(ch.table, []) + counts.setdefault(ch.table, [0] * len(plan.demos)) + continue + if ch.table in child_tables and _is_empty_param_row(schema, plan, d, a, ch): + continue + if _is_ref_row(schema, plan, d, ch): + continue + per_table.setdefault(ch.table, []).append(ch.after or {}) + i = plan.demos.index(d) + for t, added in per_table.items(): + counts.setdefault(t, [0] * len(plan.demos)) + counts[t][i] = max(len(added), 1 if t in child_tables else 0) + pure.setdefault(t, []).append([(d, r) for r in added]) + d.__dict__["_added_keys"] = added_keys + for t, cs in sorted(counts.items()): + rows = [r for group in pure.get(t, []) for r in group] + if len(set(cs)) == 1 and cs[0] > 0 and rows: + links: list[set[str]] = [getattr(d, "_added_keys", set()) for d, _ in rows] + roles = column_roles(schema, t, rows, set(), actor_values, links) + plan.sides.append(SideSpec(t, cs[0], roles, unless=child_tables.get(t))) + elif rows: + # The number of added rows differs between demonstrations: a log + # only when a person says so (bookkeeping question). Until then + # every added row stops the write. + plan.log_candidates.add(t) + _touched_rows(schema, plan, touched, actor_values) + if plan.mode == "update": + for a in plan.anchors: + plan.anchor_changed |= set(a.changed_columns) + explained = { + c for loc in plan.locations.values() if not loc.hops for c in loc.columns + } + explained |= { + loc.hops[0].column + for loc in plan.locations.values() + if loc.hops and loc.hops[0].kind == "fk" + } + for locs in plan.dispatch_locations.values(): + for loc in locs.values(): + explained |= set(loc.columns) if not loc.hops else {loc.hops[0].column} + extra = plan.anchor_changed - explained + for col in sorted(extra): + plan.anchor_extra[col] = _change_kind( + schema, plan.anchor, col, plan.demos, plan.anchors, actor_values + ) + + +_WHOLE_NUMBER = re.compile(r"^-?[0-9]+$") + +#: Kinds of change a person is offered as the app's own bookkeeping by +#: default. Anything else is listed but not excused unless they add it. +BOOKKEEPING_KINDS = ("timestamp", "counter", "actor") + + +def _change_kind( + schema: Schema, + table: str, + col: str, + demos: Sequence[Demo], + changes: Sequence[RowChange], + actor_values: Mapping[str, str], +) -> str: + """``timestamp`` (set to a time inside the demonstration), ``counter`` + (a whole number that went up), ``actor`` (set to the acting user) or + ``other``, the same in every demonstration that changed it.""" + + from openadapt_flow.generate.demo_analysis import _within + + column = schema.tables[table].column(col) + kinds = set() + for d, ch in zip(demos, changes): + before = (ch.before or {}).get(col) + after = (ch.after or {}).get(col) + if before == after: + continue + if ( + column is not None + and column.kind in ("date", "datetime") + and _within(after, d.started_at, d.finished_at) + ): + kinds.add("timestamp") + elif ( + column is not None + and column.kind in ("number", "text") + and before is not None + and after is not None + and _WHOLE_NUMBER.match(before) + and _WHOLE_NUMBER.match(after) + and int(after) > int(before) + ): + kinds.add("counter") + elif after is not None and after in set(actor_values.values()): + kinds.add("actor") + else: + kinds.add("other") + return kinds.pop() if len(kinds) == 1 else "other" + + +def _touched_rows( + schema: Schema, + plan: KindPlan, + touched: Sequence[tuple[Demo, RowChange, RowChange]], + actor_values: Mapping[str, str], +) -> None: + """Identify each existing row a demonstration changed with no task + value, so an excuse applies to that row only: the same row (by key) in + every demonstration (a counter row), or the row a located value names + through a key of the main record (the contact's account).""" + + by_table: dict[str, list[tuple[Demo, RowChange, RowChange]]] = {} + for item in touched: + by_table.setdefault(item[2].table, []).append(item) + for table, items in sorted(by_table.items()): + t = schema.tables.get(table) + cols = sorted({c for _, _, ch in items for c in ch.changed_columns}) + if t is None or len(t.primary_key) != 1: + plan.unexcused.setdefault(table, set()).update(cols) + continue + pk = t.primary_key[0] + keys = {ch.key for _, _, ch in items} + demos_with = {id(d) for d, _, _ in items} + selector: Optional[dict[str, Any]] = None + how = "" + if ( + len(keys) == 1 + and len(demos_with) == len(plan.demos) + and len(items) == len(plan.demos) + ): + key = (items[0][2].after or {}).get(pk) + if key is not None: + selector = {pk: {"literal": key}} + how = f"{table} row {pk}={key}" + if selector is None: + # Reached from the main record by a key column whose target row + # holds located values in single columns. + for p, loc in plan.locations.items(): + if ( + not loc.hops + or loc.hops[0].kind != "fk" + or loc.hops[0].table != table + ): + continue + hop = loc.hops[0] + if all( + (a.after or {}).get(hop.column) == (ch.after or {}).get(pk) + for _, a, ch in items + ): + sel = { + x.columns[0]: {"param": q} + for q, x in plan.locations.items() + if len(x.hops) == 1 + and x.hops[0] == hop + and len(x.columns) == 1 + and x.rule == "exact" + } + if sel: + selector = sel + how = f"the {table} row the task's {', '.join(sorted(v['param'] for v in sel.values()))} name" + break + if selector is None or len(items) != len({id(d) for d, _, _ in items}): + plan.unexcused.setdefault(table, set()).update(cols) + continue + kinds = { + c: _change_kind( + schema, + table, + c, + [d for d, _, _ in items], + [ch for _, _, ch in items], + actor_values, + ) + for c in cols + } + plan.touched.append( + { + "table": table, + "selector": selector, + "columns": kinds, + "row": how, + # A literal key, not one the task's values name. + "by_key": all("literal" in v for v in selector.values()), + } + ) + + +def _version_column(schema: Schema, plan: KindPlan) -> Optional[dict[str, Any]]: + """A whole-number column of the main record that went up by exactly one + in every update demonstration (Redmine's ``lock_version``).""" + + t = schema.tables[plan.anchor] + if len(t.primary_key) != 1: + return None + for col in t.columns: + # Named as a version (``lock_version``, ``version_number``): a user + # id that happens to go from 1 to 2 is not one. + if ( + col.kind != "number" + or col.name in t.primary_key + or "version" not in col.name.lower() + ): + continue + ok = bool(plan.anchors) + for a in plan.anchors: + b0 = (a.before or {}).get(col.name) + b1 = (a.after or {}).get(col.name) + if not ( + b0 is not None + and b1 is not None + and _WHOLE_NUMBER.match(b0) + and _WHOLE_NUMBER.match(b1) + and int(b1) == int(b0) + 1 + ): + ok = False + break + if ok: + return {"kind": "step", "table": plan.anchor, "column": col.name} + return None + + +def _audit_source( + schema: Schema, + plan: KindPlan, + actor_values: Mapping[str, str], + create_demos: Sequence[Demo] = (), + update_demos: Sequence[Demo] = (), +) -> Optional[dict[str, Any]]: + """A table that gained, in every update demonstration, a row naming the + main record (a column holding its key) and the acting user (a column + holding the actor's id): the app's own audit trail of saves (EspoCRM's + ``action_history_record``, Dolibarr's agenda events).""" + + t = schema.tables[plan.anchor] + if len(t.primary_key) != 1 or not actor_values: + return None + actors = set(actor_values.values()) + found: Optional[dict[str, Any]] = None + for name in sorted(schema.tables): + table = schema.tables[name] + if name == plan.anchor or len(table.primary_key) != 1: + continue + pairs: Optional[set[tuple[str, str, str]]] = None + rows: list[dict[str, Optional[str]]] = [] + for d, a in zip(plan.demos, plan.anchors): + key = (a.after or {}).get(t.primary_key[0]) + here: set[tuple[str, str, str]] = set() + for ch in d.diff.of(name, "added"): + r = ch.after or {} + for lc, lv in r.items(): + # The audit row's own id never names the record, even + # when it happens to equal the record's id. + if lv != key or lc in table.primary_key: + continue + for ac, av in r.items(): + if ac != lc and av in actors: + p = next(k for k, v in actor_values.items() if v == av) + here.add((lc, ac, p)) + rows.append(r) + pairs = here if pairs is None else pairs & here + if not pairs or len({p[0] for p in pairs}) != 1: + continue + # Several columns hold the acting user (author and assignee): the + # one named as the author, else as a user, else the first by name. + lc, ac, param = sorted( + pairs, + key=lambda p: ( + "author" not in p[1].lower(), + "user" not in p[1].lower(), + p[1], + ), + )[0] + # Only a column naming the record's type (``target_type: Lead``) + # narrows the read. Anything else the demonstrations had in common + # (the action, the client address) could hide another user's save, + # so it isn't used: an entry by anyone else stops the write. + where: dict[str, Any] = {} + names = {plan.anchor.lower(), plan.anchor.lower().split("_")[-1]} + for col in table.columns: + if col.name in (lc, ac) or col.kind != "text": + continue + values = {r.get(col.name) for r in rows} + if len(values) == 1: + v = next(iter(values)) + if v is not None and v.lower() in names and v.isidentifier(): + where[col.name] = v + # Every entry counts as a save except a code shown to be a read: a + # demonstration logged it on a record of this table it didn't add or + # change, and no demonstration logged it on one it did. An unseen + # code (a bulk edit, an import) or a NULL one counts as a save. + reads: dict[str, set[str]] = {} + writes: dict[str, set[str]] = {} + for d in [*create_demos, *update_demos]: + # Every record the demonstration added or changed, in any table: + # an audit row naming one of them is a save (a create logged on a + # contact, not a read of a lead). + moved = { + str((ch.after or {}).get(k)) + for ch in d.diff.changes + if ch.kind in ("added", "changed") and ch.table in schema.tables + for k in schema.tables[ch.table].primary_key + } + for ch in d.diff.of(name, "added"): + r = ch.after or {} + rid = r.get(lc) + if rid is None: + continue + side = writes if str(rid) in moved else reads + for col in table.columns: + v = r.get(col.name) + if v is not None: + side.setdefault(col.name, set()).add(v) + for col in table.columns: + if ( + col.name in (lc, ac) + or col.name in table.primary_key + or col.kind != "text" + or col.name.lower().endswith("id") + or col.name in where + ): + continue + deny = reads.get(col.name, set()) - writes.get(col.name, set()) + if deny and all(re.fullmatch(r"[A-Za-z_]{1,64}", x) for x in deny): + where[col.name] = {"not_in": sorted(deny)} + found = { + "kind": "audit", + "table": name, + "record_column": lc, + "author_column": ac, + "actor_param": param, + "where": where, + } + break + return found + + +def _versions_frame( + b: "SqlBuilder", plan: KindPlan, settings: "Settings" +) -> Optional[dict[str, Any]]: + v = plan.versions + if v is None: + return None + pk = b.schema.tables[plan.anchor].primary_key[0] + if v["kind"] == "step": + sql = ( + f"SELECT {b.text('t', plan.anchor, pk)} AS {b.q('id')}, " + f"{b.text('t', plan.anchor, v['column'])} AS {b.q('ver')} " + f"FROM {b.tref(plan.anchor)} t" + ) + return { + "read": {"kind": "sql", **settings.read, "sql_query": sql}, + "record_field": "id", + "target_field": "id", + "target_surface": "record", + "version_field": "ver", + } + table = v["table"] + vpk = b.schema.tables[table].primary_key[0] + + def cond(c: str, val: Any) -> str: + if isinstance(val, str): + return f"{b.text('t', table, c)} = {b.literal(val)}" + # A deny-list of read codes: a NULL code still counts as a save. + ref = b.text("t", table, c) + listed = ", ".join(b.literal(x) for x in val["not_in"]) + return f"({ref} IS NULL OR {ref} NOT IN ({listed}))" + + conds = " AND ".join(cond(c, val) for c, val in sorted(v["where"].items())) + sql = ( + f"SELECT {b.text('t', table, v['record_column'])} AS {b.q('id')}, " + f"{b.text('t', table, vpk)} AS {b.q('ver')}, " + f"{b.text('t', table, v['author_column'])} AS {b.q('author')} " + f"FROM {b.tref(table)} t" + (f" WHERE {conds}" if conds else "") + ) + return { + "read": {"kind": "sql", **settings.read, "sql_query": sql}, + "record_field": "id", + "target_field": "id", + "target_surface": "record", + "version_field": "ver", + "author_field": "author", + "self": [{"param": v["actor_param"]}], + } + + +def _mark_replaced( + schema: Schema, plan: KindPlan, d: Demo, a: RowChange, ch: RowChange +) -> None: + """An update removed or changed a row linked to the record in a table + that holds a value: count the record's rows there after the write.""" + + if plan.mode != "update" or a.after is None: + return + for g in _demo_groups(plan, d): + if g.table != ch.table: + continue + key = a.after.get(g.link.ref_column) + rows = d.after.get(g.table) + if rows is None or rows.count_only is not None: + continue + consts = [(c, r.value) for c, r in g.roles.items() if r.role == "constant"] + n = sum( + 1 + for r in rows.rows.values() + if r.get(g.link.column) == key and all(r.get(c) == v for c, v in consts) + ) + g.replaced, g.after_count = True, n + + +def plan_pk(schema: Schema, table: str) -> str: + pk = schema.tables[table].primary_key + return pk[0] if len(pk) == 1 else "" + + +def _is_param_row(schema: Schema, plan: KindPlan, d: Demo, ch: RowChange) -> bool: + for g in _demo_groups(plan, d): + p = g.param + if g.table != ch.table or d.values.get(p) is None or ch.after is None: + continue + rel = g.params[p] + if locate( + schema, + d, + _as_change(ch.table, ch.after), + d.values[p], + max_hops=len(rel.hops), + ): + return True + return False + + +def _is_empty_param_row( + schema: Schema, plan: KindPlan, d: Demo, a: RowChange, ch: RowChange +) -> bool: + """A row the app added for a parameter the task left out: linked to the + main record, holding every constant of that parameter's rows, and an + empty value where the parameter's value goes.""" + + if ch.after is None or a.after is None: + return False + for g in _demo_groups(plan, d): + p = g.param + if g.table != ch.table or d.values.get(p) is not None: + continue + if ch.after.get(g.link.column) != a.after.get(g.link.ref_column): + continue + rel = g.params[p] + cols = rel.columns if not rel.hops else (rel.hops[0].column,) + if any(ch.after.get(c) not in (None, "") for c in cols): + continue + consts = [(c, r.value) for c, r in g.roles.items() if r.role == "constant"] + if consts and all(ch.after.get(c) == v for c, v in consts): + g.empty_when_null = True + return True + return False + + +def _is_ref_row(schema: Schema, plan: KindPlan, d: Demo, ch: RowChange) -> bool: + """A row the write added that a parameter's location points to (a new + email address row a link row names): declared by the value it holds.""" + + if ch.after is None: + return False + for g in _demo_groups(plan, d): + p = g.param + rel = g.params[p] + if not rel.hops or rel.hops[0].table != ch.table or d.values.get(p) is None: + continue + col = rel.columns[0] if len(rel.columns) == 1 else None + if ( + col + and ch.after.get(col) is not None + and _rule_ok(ch.after.get(col), d.values[p], rel.rule) + ): + dv = str(d.values.get(plan.dispatch)) if plan.dispatch else None + plan.added_refs.setdefault( + f"{p}@{dv}" if dv else p, + { + "table": ch.table, + "column": col, + "rule": rel.rule, + "param": p, + "dv": dv, + }, + ) + return True + return False + + +def _rule_ok(stored: Any, value: Any, rule: str) -> bool: + from openadapt_flow.generate.demo_analysis import rule_for + + return rule_for(stored, "text", value) == rule + + +# -- SQL -------------------------------------------------------------------------- + + +class SqlBuilder: + """SQL text for one dialect. Every value from a task is bound as a + parameter; table and column names come from the schema and are quoted.""" + + def __init__(self, schema: Schema) -> None: + self.schema = schema + self.d = schema.dialect + self.params: dict[str, dict[str, str]] = {} + #: Tables read through a view (the SQL guard refuses their name). + self.views: dict[str, str] = {} + #: (table, key column) -> the displays the task's values are read + #: through that key with: checked together for uniqueness. + self.companions: dict[tuple[str, str], list[tuple[tuple[str, ...], str]]] = {} + #: The main record's table: a key back to it names the record by its + #: own key, not by a display value, so no uniqueness check applies. + self.anchor = "" + self._n = itertools.count() + + def q(self, name: str) -> str: + return store.quote(self.d, name) + + def tref(self, table: str) -> str: + """A table as a read names it: its view when it has one.""" + + return self.qt(self.views.get(table, table)) + + def qt(self, name: str) -> str: + """A table name, ``schema.table`` quoted part by part.""" + + return ".".join(self.q(part) for part in name.split(".")) + + def ph(self, param: str) -> str: + for k, v in self.params.items(): + if v == {"param": param}: + return f"%({k})s" + name = f"v{next(self._n)}" + self.params[name] = {"param": param} + return f"%({name})s" + + def text(self, alias: str, table: str, column: str) -> str: + col = self.schema.tables[table].column(column) + ref = f"{alias}.{self.q(column)}" + kind = col.kind if col else "other" + if self.d == "mysql": + if kind in ("binary", "other"): + return f"HEX({ref})" + return f"CAST({ref} AS CHAR)" + if kind == "binary": + return f"encode({ref}, 'hex')" + return f"CAST({ref} AS TEXT)" + + def display(self, alias: str, table: str, columns: Sequence[str], rule: str) -> str: + if rule == "minutes": + a, b = (f"{alias}.{self.q(c)}" for c in columns) + if self.d == "mysql": + return f"(TIMESTAMPDIFF(SECOND, {a}, {b}) / 60)" + return f"(EXTRACT(EPOCH FROM ({b} - {a})) / 60)" + if rule in ("number", "date", "datetime"): + return f"{alias}.{self.q(columns[0])}" + if len(columns) == 1: + return self.text(alias, table, columns[0]) + parts = [self.text(alias, table, c) for c in columns] + if self.d == "mysql": + return "CONCAT(" + ", ' ', ".join(parts) + ")" + return " || ' ' || ".join(parts) + + def path(self, alias: str, table: str, loc: Location, depth: int = 0) -> str: + if not loc.hops: + return self.display(alias, table, loc.columns, loc.rule) + hop, rest = loc.hops[0], Location(loc.hops[1:], loc.columns, loc.rule) + if hop.kind != "fk": + raise GenerateError("a child hop can't be read as one value") + sub = f"a{depth + 1}" + inner = self.path(sub, hop.table, rest, depth + 1) + live = self.live(sub, hop.table) + # A child row's link to its own record names the target by identity + # and needs no check; any other key into the main table (a parent + # issue named by its subject) does. + child_link = hop.table == self.anchor and table != self.anchor + if not rest.hops and not child_link: + # The value must name exactly one live row of its table: a + # second "Jon Park", or "Mary Ann|Lee" beside "Mary|Ann Lee", + # makes it NULL, so nothing matches and the write stops. + u = f"u{depth + 1}" + # Every value the task gives through this key together (a + # customer's name and town) must name one row: a name shared by + # two towns is fine when the town tells them apart. + displays = self.companions.get((hop.table, hop.column)) or [ + (rest.columns, rest.rule) + ] + conds = [ + self.same( + self.display(u, hop.table, cols, rule), + self.display(sub, hop.table, cols, rule), + False, + ) + for cols, rule in displays + ] + unique = ( + f"(SELECT COUNT(*) FROM {self.tref(hop.table)} {u} WHERE " + f"{' AND '.join(conds)}{self.live_and(u, hop.table)}) = 1" + ) + inner = f"CASE WHEN {unique} THEN {inner} END" + return ( + f"(SELECT {inner} FROM {self.tref(hop.table)} {sub} WHERE " + f"{self.same(f'{sub}.{self.q(hop.ref_column)}', f'{alias}.{self.q(hop.column)}', hop.declared)}" + f"{' AND ' + live if live else ''})" + ) + + def live(self, alias: str, table: str) -> str: + """The condition that a row isn't soft-deleted (a ``deleted`` or + ``is_deleted`` flag, or a ``deleted_at`` time), or "" when the table + has none: a trashed look-alike never resolves a key.""" + + t = self.schema.tables.get(table) + if t is None: + return "" + for name in ("deleted", "is_deleted"): + col = t.column(name) + if col is not None and col.kind in ("number", "bool"): + ref = f"{alias}.{self.q(name)}" + if self.d == "mysql" or col.kind == "number": + return f"COALESCE({ref}, 0) = 0" + return f"NOT COALESCE({ref}, false)" + if t.column("deleted_at") is not None: + return f"{alias}.{self.q('deleted_at')} IS NULL" + return "" + + def live_and(self, alias: str, table: str) -> str: + cond = self.live(alias, table) + return f" AND {cond}" if cond else "" + + def same(self, a: str, b: str, declared: bool = True) -> str: + if declared: + return f"{a} = {b}" + if self.d == "mysql": + return f"CAST({a} AS BINARY) = CAST({b} AS BINARY)" + return f"CAST({a} AS TEXT) = CAST({b} AS TEXT)" + + def flag(self, expr: str, rule: str, ph: str) -> str: + if self.d == "mysql": + if rule in ("number", "minutes"): + cond = ( + f"CAST({ph} AS CHAR) REGEXP '^-?[0-9]+([.][0-9]+)?$' AND " + f"{expr} = CAST({ph} AS DECIMAL(65,30))" + ) + elif rule == "date": + cond = f"{expr} = CAST({ph} AS DATE)" + else: + cond = f"{expr} = CAST({ph} AS DATETIME)" + else: + if rule in ("number", "minutes"): + cond = f"CAST({expr} AS numeric) = CAST({ph} AS numeric)" + elif rule == "date": + cond = f"{expr} = CAST({ph} AS date)" + else: + cond = f"{expr} = CAST({ph} AS timestamp)" + return f"CASE WHEN {cond} THEN '1' ELSE '0' END" + + def literal(self, text: str) -> str: + if "\\" in text or "\x00" in text: + raise GenerateError("a constant holds a backslash or NUL; not pinned") + return "'" + text.replace("'", "''") + "'" + + def pin(self, alias: str, table: str, column: str, value: Optional[str]) -> str: + """A flag: this column holds the demonstrated constant (or no value).""" + + col = self.schema.tables[table].column(column) + ref = f"{alias}.{self.q(column)}" + if value is None: + return f"CASE WHEN {ref} IS NULL THEN '1' ELSE '0' END" + kind = col.kind if col else "text" + if kind in ("number", "bool"): + cast = "DECIMAL(65,30)" if self.d == "mysql" else "numeric" + lhs = ref if self.d == "mysql" else f"CAST({ref} AS numeric)" + if kind == "bool" and self.d != "mysql": + lhs = f"CAST(CAST({ref} AS integer) AS numeric)" + value = "1" if value in ("1", "true", "True") else "0" + return f"CASE WHEN {lhs} = CAST({self.literal(value)} AS {cast}) THEN '1' ELSE '0' END" + if kind == "date": + return f"CASE WHEN {ref} = CAST({self.literal(value)} AS DATE) THEN '1' ELSE '0' END" + return f"CASE WHEN {self.text(alias, table, column)} = {self.literal(value)} THEN '1' ELSE '0' END" + + def empty(self, alias: str, table: str, column: str) -> str: + col = self.schema.tables[table].column(column) + ref = f"{alias}.{self.q(column)}" + if col is not None and col.kind == "text": + return f"CASE WHEN {ref} IS NULL OR {ref} = '' THEN '1' ELSE '0' END" + return f"CASE WHEN {ref} IS NULL THEN '1' ELSE '0' END" + + def copied( + self, alias: str, detail: Mapping[str, Any], table: str, column: str + ) -> str: + src = ( + f"(SELECT s.{self.q(detail['column'])} FROM {self.tref(detail['table'])} s " + f"WHERE s.{self.q(detail['ref_column'])} = {alias}.{self.q(detail['via'])})" + ) + ref = f"{alias}.{self.q(column)}" + return f"CASE WHEN {ref} = {src} THEN '1' ELSE '0' END" + + def key(self, alias: str, table: str) -> Optional[str]: + pk = self.schema.tables[table].primary_key + if not pk: + return None + if len(pk) == 1: + return self.text(alias, table, pk[0]) + parts = [self.text(alias, table, c) for c in pk] + if self.d == "mysql": + return "CONCAT_WS('|', " + ", ".join(parts) + ")" + return " || '|' || ".join(parts) + + +def _alias(name: str) -> str: + out = re.sub(r"[^A-Za-z0-9_]", "_", name) + return out[:60] + + +@dataclass +class Surface: + name: str + table: str + columns: list[tuple[str, str]] # (alias, expr) + alias: str = "t" + + def sql(self, b: SqlBuilder) -> str: + cols = ", ".join(f"{expr} AS {b.q(a)}" for a, expr in self.columns) + return f"SELECT {cols} FROM {b.qt(self.table)} {self.alias}" + + +def _base_columns(b: SqlBuilder, table: str, alias: str = "t") -> list[tuple[str, str]]: + t = b.schema.tables[table] + cols: list[tuple[str, str]] = [] + key = b.key(alias, table) + if key is not None: + cols.append(("id", key)) + for c in t.columns: + if c.name == "id" and key is not None: + continue + cols.append((c.name, b.text(alias, table, c.name))) + return cols + + +# -- one contract ------------------------------------------------------------------ + + +@dataclass +class Variant: + kind: str + fields: list[str] + lists: dict[str, int] = field(default_factory=dict) + dispatch_value: Optional[str] = None + files: dict[str, str] = field( + default_factory=dict + ) # create | found | update -> path + sample: dict[str, Any] = field(default_factory=dict) + + @property + def slug(self) -> str: + parts = sorted(self.fields) + [ + f"{k}x{n}" for k, n in sorted(self.lists.items()) + ] + if self.dispatch_value is not None: + parts.insert(0, f"{self.dispatch_value}") + text = "+".join(parts) + digest = hashlib.sha256(text.encode()).hexdigest()[:8] + short = re.sub(r"[^A-Za-z0-9_+]", "_", text)[:80] + return f"{short}.{digest}" + + +@dataclass +class Settings: + read: dict[str, Any] + actor_params: dict[str, str] + recheck_after_s: Optional[float] + environment_id: str + max_window_s: int = 900 + #: Tables a view replaces in surface reads (the SQL guard refuses the + #: table's name): table -> view name. + views: dict[str, str] = field(default_factory=dict) + #: Table -> columns of a new row the person named as not checked (they + #: vary, are free text left empty, or are the write's time stamp). + unchecked: dict[str, set[str]] = field(default_factory=dict) + + +#: Roles of a new row's column that no rule checks: named by the person +#: (``bookkeeping.unchecked``), else pinned empty (free) or the kind refused. +OPEN_ROLES = ("varies", "free", "timestamp") + + +def _value_expr(param: str, rule: str) -> dict[str, Any]: + out: dict[str, Any] = {"param": param} + if rule in ("email", "phone"): + out["kind"] = rule + return out + + +def build_contract( + schema: Schema, + plan: KindPlan, + variant: Variant, + flavor: str, + settings: Settings, + excused: Mapping[str, set[str]], + logs: set[str], +) -> dict[str, Any]: + """One ``openadapt.verify/v1`` document for one field set. ``excused``: + table -> columns a person confirmed as the app's own bookkeeping; each + is excused only on the rows it was seen changing (the target, or a row + identified by key or by the task's values), never by name elsewhere.""" + + b = SqlBuilder(schema) + b.views = dict(settings.views) + b.anchor = plan.anchor + wanted = set(variant.fields) | set(variant.lists) + list_keys = {k for g in plan.lists.values() for k in g.params} + for c_name, c_loc in _all_locations(plan): + if c_name not in wanted and c_name not in list_keys: + continue + if not c_loc.hops or c_loc.hops[-1].kind != "fk": + continue + c_key = (c_loc.hops[-1].table, c_loc.hops[-1].column) + c_entry = (tuple(c_loc.columns), c_loc.rule) + if c_entry not in b.companions.setdefault(c_key, []): + b.companions[c_key].append(c_entry) + fields = set(variant.fields) + anchor = plan.anchor + surfaces: dict[str, Surface] = {} + effects: list[dict[str, Any]] = [] + target_fields: set[str] = set() + #: Surface -> the fields its declared rows may change. + by_surface: dict[str, set[str]] = {} + named_tables: set[str] = {anchor} + + # The main record, with every column and the values its parameters reach. + main = Surface( + "record", settings.views.get(anchor, anchor), _base_columns(b, anchor) + ) + match: dict[str, Any] = {} + locations = dict(plan.locations) + if variant.dispatch_value is not None: + locations.update(plan.dispatch_locations.get(variant.dispatch_value, {})) + identity_params = [] + for p in sorted(fields): + loc = locations.get(p) + if loc is None or (loc.hops and loc.hops[0].kind == "child"): + continue + identity_params.append(p) + key = _place(b, main, anchor, p, loc) + if loc.rule in ("number", "date", "datetime", "minutes"): + match[key] = {"literal": "1"} + else: + match[key] = _value_expr(p, loc.rule) + if plan.mode == "update" and p in plan.changes: + target_fields.add(key) + target_fields.update(loc.columns if not loc.hops else [loc.hops[0].column]) + if plan.mode == "update" and flavor == "update": + # The target's own bookkeeping, excused on the target row only. + target_fields.update( + c for c in plan.anchor_extra if c in excused.get(anchor, set()) + ) + if plan.mode == "create": + # ``found`` pins the same constants as ``create`` (a soft-delete + # flag, a status, a type): a trashed record with the task's values + # is not "already there". Who created it may differ. + roles = _anchor_roles(schema, plan, variant, settings.actor_params) + for col, role in sorted(roles.items()): + if flavor == "found" and role.role in ("actor", "copied"): + continue + _pin( + b, + main, + anchor, + col, + role, + match, + settings.actor_params, + settings.unchecked, + ) + if plan.mode == "create" and flavor == "create": + support = _support(plan, variant) + for p in plan.optional: + if p in fields or p in variant.lists: + continue + loc = plan.locations.get(p) + if loc is None: + continue + cols = ( + list(loc.columns) + if not loc.hops + else ([loc.hops[0].column] if loc.hops[0].kind == "fk" else []) + ) + for c in cols: + # A column another value of this field set fills (a call's + # parent is its account, or its lead) is not pinned empty. + if any( + (a.after or {}).get(c) not in (None, "") + for d, a in support + if d.values.get(p) is None + ): + continue + alias = _add_column( + main, f"empty__{_alias(c)}", b.empty("t", anchor, c) + ) + match[alias] = {"literal": "1"} + effect: dict[str, Any] = { + "kind": "record_written", + "probe": "surface=record|the main record holds the task's values", + "match": match, + "expected_count": 1, + } + if plan.mode == "create" and flavor == "create": + effect["count_new_only"] = True + effects.append(effect) + surfaces["record"] = main + + # Rows beside the main record that hold a value (a note, a custom field, + # an email address link), and list items (invoice lines). + found = flavor == "found" + groups = [ + (p, g) + for g in _all_groups(plan, variant.dispatch_value) + for p in [g.param] + if p in fields + or (not found and g.empty_when_null) + or (not found and plan.mode == "create" and set(g.co_owners) & fields) + ] + groups += [(p, g) for p, g in plan.lists.items() if p in variant.lists] + # A child row names its record by the record's identity: for an update + # only the values that pick the target, never the ones it changes (the + # target's older rows would otherwise move with the change). + main_params = ( + [ip for ip in identity_params if ip in plan.identity] + if plan.mode == "update" + else identity_params + ) + for p, g in groups: + name = f"rows:{g.table}" + sname = _alias(name) + if sname not in surfaces: + scols = _base_columns(b, g.table) + link = g.link + # The main record's values, seen from the child row. + for ip in main_params: + loc = locations[ip] + up = Location( + (Hop("fk", anchor, link.column, link.ref_column, True),) + loc.hops, + loc.columns, + loc.rule, + ) + expr = b.path("t", g.table, up) + if loc.rule in ("number", "date", "datetime", "minutes"): + expr = b.flag(expr, loc.rule, b.ph(ip)) + scols.append((f"main__{_alias(ip)}", expr)) + surfaces[sname] = Surface( + sname, settings.views.get(g.table, g.table), scols + ) + named_tables.add(g.table) + surf = surfaces[sname] + items = range(variant.lists[p]) if g.list_param else [None] + for i in items: + m: dict[str, Any] = {} + for ip in main_params: + loc = locations[ip] + if loc.rule in ("number", "date", "datetime", "minutes"): + m[f"main__{_alias(ip)}"] = {"literal": "1"} + else: + m[f"main__{_alias(ip)}"] = _value_expr(ip, loc.rule) + for k, loc in g.params.items(): + pname = k if g.list_param is None else f"{p}.{i}.{k}" + if g.list_param is None and pname not in fields: + # Given another owner but not this value: empty. + for c in loc.columns if not loc.hops else [loc.hops[0].column]: + alias = f"empty__{_alias(c)}" + if alias not in {a for a, _ in surf.columns}: + surf.columns.append((alias, b.empty("t", g.table, c))) + m[alias] = {"literal": "1"} + continue + key = _place(b, surf, g.table, pname, loc) + if loc.rule in ("number", "date", "datetime", "minutes"): + m[key] = {"literal": "1"} + else: + m[key] = _value_expr(pname, loc.rule) + for col, role in sorted(g.roles.items()): + if role.role == "link" or (found and role.role in ("actor", "copied")): + continue + _pin( + b, + surf, + g.table, + col, + role, + m, + settings.actor_params, + settings.unchecked, + ) + effect = { + "kind": "record_written", + "probe": f"surface={sname}|a row beside the main record holds {p}", + "match": m, + "expected_count": 1, + } + if not found: + # Found: the row is already there, and nothing changes. + effect["count_new_only"] = True + effects.append(effect) + if plan.mode == "update" and g.replaced and not found: + # The demonstrations replaced the record's earlier rows here (an + # old email address link removed): afterwards the record has + # exactly as many such rows as they left, and its earlier ones + # may go. + m = { + k: v + for k, v in effects[-1]["match"].items() + if k.startswith(("main__", "pin__")) or k in g.roles + } + effects.append( + { + "kind": "record_written", + "probe": f"surface={sname}|the record's rows here after the write", + "match": m, + "expected_count": g.after_count, + } + ) + if plan.mode == "update" and not found: + by_surface.setdefault(sname, set()).update(g.changed_columns) + # Existing rows the app changes beside the write (a counter row, the + # account a contact belongs to): declared by key or by the task's values, + # with only the confirmed columns allowed to move, on those rows only. + for spec in plan.touched: + # A row changed by key (a user's preferences, which even a read may + # touch) is excused in ``found`` too; a row named by the task's + # values only when something is written. + if found and not spec.get("by_key"): + continue + # A row the task's values name applies only to a field set that + # gives them (a call logged on a contact names no lead). + if any( + "param" in v and v["param"] not in fields for v in spec["selector"].values() + ): + continue + cols = [c for c in spec["columns"] if c in excused.get(spec["table"], set())] + if not cols: + continue + name = f"raw_{_alias(spec['table'])}" + if name not in surfaces: + surfaces[name] = Surface( + name, + settings.views.get(spec["table"], spec["table"]), + _base_columns(b, spec["table"]), + ) + effects.append( + { + "kind": "record_written", + "probe": f"surface={name}|{spec['row']}, changed by the app", + "match": dict(spec["selector"]), + "expected_count": 1, + } + ) + # Only on this table's rows: a parent row the app touches can't take + # the target's fields, nor the target a parent's. + by_surface.setdefault(name, set()).update(cols) + if spec["table"] == anchor: + by_surface.setdefault("record", set()).update(cols) + # A new row a link row names (a new email address). + for ref in plan.added_refs.values(): + p = ref["param"] + if p not in fields or found or ref["dv"] not in (None, variant.dispatch_value): + continue + effects.append( + _raw_effect( + b, + schema, + surfaces, + settings, + ref["table"], + {ref["column"]: _value_expr(p, ref["rule"])}, + 1, + {}, + f"the new {ref['table']} row holds {p}", + ) + ) + for side in plan.sides: + if found or (side.unless is not None and side.unless in fields): + continue + m = {} + for col, role in sorted(side.roles.items()): + if role.role == "constant" and role.value is not None: + m[col] = {"literal": role.value} + elif role.role == "actor": + m[col] = {"param": role.detail["param"]} + effects.append( + _raw_effect( + b, + schema, + surfaces, + settings, + side.table, + m, + side.count, + side.roles, + f"{side.count} row(s) the app adds to {side.table}", + ) + ) + + verifier: dict[str, Any] = {"kind": "surfaces", "surfaces": {}} + for s in surfaces.values(): + sub: dict[str, Any] = {"kind": "sql", **settings.read, "sql_query": s.sql(b)} + used = {k: v for k, v in b.params.items() if f"%({k})s" in sub["sql_query"]} + if used: + sub["sql_query_params"] = used + verifier["surfaces"][s.name] = sub + # Main and child effects read named surfaces: no other read is judged + # against them, and the whole-database frame covers every other table. + frame: dict[str, Any] = {} + if not found: + by_surface.setdefault("record", set()).update(target_fields) + # Per surface: each declared row may change only its own table's fields. + frame["target_fields"] = sorted(set().union(*by_surface.values())) + frame["target_fields_by_surface"] = { + k: sorted(v) for k, v in sorted(by_surface.items()) if v + } + if plan.mode == "update": + # Another user's save to the target inside the window (a lost + # update) shows only in the target's versions. + versions = _versions_frame(b, plan, settings) + if versions is not None: + frame["versions"] = versions + # MySQL, MariaDB and PostgreSQL: the whole-database frame + allow: list[dict[str, Any]] = [] + for t in sorted(logs): + if t in named_tables: + continue + allow.append({"table": t}) + for t in sorted(named_tables): + if plan.mode == "create" or flavor == "found": + # The contract's own read judges the rows the write adds; the + # whole-database frame still sees any existing row change, + # now and at the recheck (a late edit to a look-alike). + allow.append({"table": t}) + continue + # An update changes a row here, which an append-only table + # can't allow: its own read judges this table, and the recheck + # (database only) doesn't see it. + allow.append( + { + "table": t, + "mode": "skip", + "reason": "read in full, every column, by this contract's own surface", + } + ) + views = {s.table: s.name for s in surfaces.values() if s.name.startswith("raw_")} + db: dict[str, Any] = { + "dialect": schema.dialect, + "read": dict(settings.read), + "allow_tables": allow, + "views": views, + "settle_s": 1, + "settle_attempts": 5, + } + nrf = { + s.table: sorted(getattr(s, "_new_row_fields", [])) + for s in surfaces.values() + if s.name.startswith("raw_") and getattr(s, "_new_row_fields", None) + } + if nrf: + db["new_row_fields"] = nrf + if settings.recheck_after_s: + db["recheck_after_s"] = settings.recheck_after_s + frame["database"] = db + doc: dict[str, Any] = { + "schema_version": "openadapt.verify/v1", + "environment_id": settings.environment_id, + "max_execution_window_s": settings.max_window_s, + "verifier": verifier, + "effects": effects, + "frame": frame, + } + return doc + + +def _all_columns(schema: Schema, tables: set[str]) -> set[str]: + return { + c.name for t in tables if t in schema.tables for c in schema.tables[t].columns + } + + +def _place(b: SqlBuilder, surf: Surface, table: str, param: str, loc: Location) -> str: + """Add the column that holds ``param``'s value to ``surf``; its name.""" + + if ( + not loc.hops + and len(loc.columns) == 1 + and loc.rule in ("exact", "email", "phone") + ): + return loc.columns[0] if loc.columns[0] != "id" else "id" + expr = b.path("t", table, loc) + alias = f"p__{_alias(param)}" + if loc.rule in ("number", "date", "datetime", "minutes"): + expr = b.flag(expr, loc.rule, b.ph(param)) + if alias not in {a for a, _ in surf.columns}: + surf.columns.append((alias, expr)) + return alias + + +def _pin( + b: SqlBuilder, + surf: Surface, + table: str, + col: str, + role: ColumnRole, + match: dict[str, Any], + actor_params: Mapping[str, str], + unchecked: Mapping[str, set[str]] = {}, +) -> None: + if role.role == "free" and col not in unchecked.get(table, set()): + # Empty in every demonstration and not named as unchecked: it stays + # empty. + alias = _add_column(surf, f"empty__{_alias(col)}", b.empty("t", table, col)) + match[alias] = {"literal": "1"} + return + if role.role == "constant": + column = b.schema.tables[table].column(col) + if role.value is not None and column is not None and column.kind == "text": + match.setdefault(col, {"literal": role.value}) + return + alias = _add_column( + surf, f"pin__{_alias(col)}", b.pin("t", table, col, role.value) + ) + match[alias] = {"literal": "1"} + elif role.role == "actor": + match.setdefault(col, {"param": role.detail["param"]}) + elif role.role == "copied": + alias = _add_column( + surf, f"pin__{_alias(col)}", b.copied("t", role.detail, table, col) + ) + match[alias] = {"literal": "1"} + + +def _add_column(surf: Surface, alias: str, expr: str) -> str: + """Add a computed column; the same name with another expression (two + rows of one table pinned to different constants) gets its own name.""" + + for a, e in surf.columns: + if a == alias and e == expr: + return alias + if any(a == alias for a, _ in surf.columns): + alias = f"{alias}__{hashlib.sha256(expr.encode()).hexdigest()[:6]}" + if any(a == alias for a, _ in surf.columns): + return alias + surf.columns.append((alias, expr)) + return alias + + +def _raw_effect( + b: SqlBuilder, + schema: Schema, + surfaces: dict[str, Surface], + settings: Settings, + table: str, + match: dict[str, Any], + count: int, + roles: Mapping[str, ColumnRole], + note: str, +) -> dict[str, Any]: + """An effect on a table's own read: rows the app adds beside the write. + + The table is judged by the whole-database frame, which also checks that + every column these rows set beyond ``match`` is declared in + ``new_row_fields`` or holds its default.""" + + name = f"raw_{_alias(table)}" + surf = surfaces.get(name) + if surf is None: + surf = Surface(name, settings.views.get(table, table), _base_columns(b, table)) + surfaces[name] = surf + t = schema.tables[table] + m = dict(match) + # Text and whole-number constants only: the snapshot's JSON form of + # a decimal or a date isn't the generator's text form. + for col, value in list(m.items()): + c = t.column(col) + lit = value.get("literal") if isinstance(value, dict) else None + if ( + c is not None + and lit is not None + and not ( + c.kind == "text" + or ( + c.kind in ("number", "bool") + and c.data_type.startswith( + ( + "int", + "tinyint", + "smallint", + "mediumint", + "bigint", + "integer", + ) + ) + and re.fullmatch(r"-?\d+", lit) + ) + ) + ): + m.pop(col) + nrf = set(getattr(surf, "_new_row_fields", set())) + for c in t.columns: + if c.name in m: + continue + role = roles.get(c.name) + if role is not None and role.role == "constant" and role.value is None: + continue + if ( + role is None + or role.role != "constant" + or (role.detail or {}).get("is_default") + ): + if role is not None and role.role in OPEN_ROLES: + # Set freely only when the person named it unchecked. Free + # text not named holds the empty value every demonstration + # stored (NULL or ''), matched on the row itself; anything + # else must hold its default. + if c.name in settings.unchecked.get(table, set()): + nrf.add(c.name) + elif role.role == "free": + empty = (role.detail or {}).get("empty") + if empty == "null": + m[c.name] = {"null": True} + nrf.add(c.name) + elif empty == "text": + m[c.name] = {"literal": ""} + nrf.add(c.name) + continue + if role is None or role.role in ("key", "link", "copied", "actor"): + nrf.add(c.name) + continue + nrf.add(c.name) + surf.__dict__["_new_row_fields"] = nrf + return { + "kind": "record_written", + "probe": f"surface={name}|{note}", + "match": m, + "expected_count": count, + "count_new_only": True, + } + + +def _anchor_roles( + schema: Schema, plan: KindPlan, variant: Variant, actor_params: Mapping[str, str] +) -> dict[str, ColumnRole]: + """The main row's columns no parameter explains, from the demonstrations + that give this field set (all of them when none gives exactly it).""" + + support = _support(plan, variant) + explained: set[str] = set() + for loc in plan.locations.values(): + if not loc.hops: + explained |= set(loc.columns) + elif loc.hops[0].kind == "fk": + explained.add(loc.hops[0].column) + rows = [(d, a.after or {}) for d, a in support] + roles = column_roles(schema, plan.anchor, rows, explained, actor_params) + if len(support) < len(plan.demos): + # A column that differs between all demonstrations and holds a + # generated token (a UUID) is the app's own, whichever ones support + # this field set. + everyone = [(d, a.after or {}) for d, a in zip(plan.demos, plan.anchors)] + for col, role in column_roles( + schema, plan.anchor, everyone, explained, actor_params + ).items(): + values = [r.get(col) for _, r in everyone] + if role.role == "varies" and all( + v is not None and _TOKEN.match(v) for v in values + ): + roles[col] = role + return roles + + +_TOKEN = re.compile( + r"^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}" + r"|[0-9a-fA-F]{16,})$" +) + + +def _support(plan: KindPlan, variant: Variant) -> list[tuple[Demo, RowChange]]: + """The demonstrations that give every field of this set (all of them + when none does).""" + + fields = set(variant.fields) | set(variant.lists) + if variant.dispatch_value is not None: + fields.add(plan.dispatch or "") + return [ + (d, a) + for d, a in zip(plan.demos, plan.anchors) + if fields <= {k for k, v in d.values.items() if v is not None} + ] or list(zip(plan.demos, plan.anchors)) + + +# -- variants ----------------------------------------------------------------------- + + +def variants( + schema: Schema, plan: KindPlan, spec: KindSpec, answers: Mapping[str, Any] +) -> list[Variant]: + out: list[Variant] = [] + lists = list(plan.lists) + if plan.mode == "update": + if plan.dispatch is not None: + values = dict(plan.dispatch_locations) + # By analogy: another column of the main table, written the way + # a demonstrated plain column is. + plain = [ + loc + for locs in values.values() + for loc in locs.values() + if not loc.hops and loc.rule == "exact" + ] + if plain: + vp = next(iter(next(iter(values.values())))) + for c in schema.tables[plan.anchor].columns: + for name in {c.name, _camel(c.name)}: + if ( + name in values + or c.name in schema.tables[plan.anchor].primary_key + ): + continue + if c.kind == "text": + values[name] = {vp: Location((), (c.name,), "exact")} + plan.dispatch_locations = values + for dv in sorted(values): + fields = sorted(set(plan.identity) | {plan.dispatch, *values[dv]}) + out.append(Variant(plan.kind, fields, dispatch_value=dv)) + return out + changes = [p for p in plan.changes if p in plan.locations] + for n in range(1, len(changes) + 1): + for combo in itertools.combinations(changes, n): + out.append(Variant(plan.kind, sorted(set(plan.identity) | set(combo)))) + return out[:MAX_VARIANTS] + located = set(plan.locations) + base_sets = [] + for d in plan.demos: + given = { + k for k, v in d.values.items() if v is not None and not isinstance(v, list) + } & (located | {g.param for g in plan.children.values()}) + base_sets.append(frozenset(given)) + optional = [p for p in plan.optional if p in located and p not in lists] + seen: set[frozenset[str]] = set(base_sets) + # Every demonstrated set first, then sets with fewer optional fields, + # one field left out at a time across demonstrations, so the cap keeps + # the sets closest to what was shown. + sets: list[frozenset[str]] = list(dict.fromkeys(base_sets)) + drops = {base: [p for p in optional if p in base] for base in base_sets} + for n in range(1, max((len(v) for v in drops.values()), default=0) + 1): + for base in base_sets: + for combo in itertools.combinations(drops[base], n): + s = frozenset(base - set(combo)) + if s not in seen: + seen.add(s) + sets.append(s) + # A field the examples leave out sometimes, and that one demonstration + # gives, may be added to a set from another demonstration. + for s in list(sets): + for p in optional: + if p not in s: + t = frozenset(s | {p}) + if t not in seen: + seen.add(t) + sets.append(t) + sets = sets[:MAX_VARIANTS] + for s in sets: + if lists: + for n in range(1, MAX_LIST + 1): + out.append(Variant(plan.kind, sorted(s), lists={lists[0]: n})) + else: + out.append(Variant(plan.kind, sorted(s))) + return out + + +def _camel(name: str) -> str: + head, *rest = name.split("_") + return head + "".join(w.title() for w in rest) + + +# -- validation ------------------------------------------------------------------------ + + +def flatten(values: Mapping[str, Any]) -> dict[str, Any]: + """Task values as the contracts bind them: ``lines: [{qty: 4}]`` is + ``lines.0.qty: 4``; nulls are left out.""" + + out: dict[str, Any] = {} + for k, v in values.items(): + if v is None: + continue + if isinstance(v, list): + for i, item in enumerate(v): + if isinstance(item, dict): + for ik, iv in item.items(): + if iv is not None: + out[f"{k}.{i}.{ik}"] = iv + else: + out[f"{k}.{i}"] = item + else: + out[k] = v + return out + + +def sample_values( + plan: KindPlan, variant: Variant, others: Sequence[Demo] = () +) -> dict[str, Any]: + """Values to dry-read a contract with: the demonstrations' own (another + kind's, for a parameter placed from that kind).""" + + out: dict[str, Any] = {} + for d in [*plan.demos, *others]: + for k, v in d.values.items(): + if v is not None and k not in out: + out[k] = v + if variant.dispatch_value is not None and plan.dispatch: + out[plan.dispatch] = variant.dispatch_value + for p, n in variant.lists.items(): + items = out.get(p) if isinstance(out.get(p), list) else [] + if items: + out[p] = [items[i % len(items)] for i in range(n)] + for k in variant.fields: + loc = plan.locations.get(k) + if k not in out and loc is not None: + # Placed by name, so no demonstration gave a value: any value of + # the right type reads the same columns. + out[k] = {"number": 1, "minutes": 1, "date": "2026-01-01", + "datetime": "2026-01-01 00:00"}.get(loc.rule, "x") # fmt: skip + keep = set(variant.fields) | set(variant.lists) + return {k: v for k, v in out.items() if k in keep} + + +def dry_read(path: Path, params: Mapping[str, Any], mysql: bool) -> Optional[str]: + """Load the contract and read the live store once, as ``begin_write`` + would. None when it reads; else why not.""" + + from openadapt_flow.verifier import Contract, VerifierError + from openadapt_flow.verify_run import ( + FRAME_DB_DETAIL, + VerifyRunError, + capture_baseline, + ) + + try: + contract = Contract.load(path) + except VerifierError as exc: + return f"does not load: {exc}" + with tempfile.TemporaryDirectory() as tmp: + try: + payload = capture_baseline(contract.config, params, Path(tmp) / "b.json") + except (VerifyRunError, ValueError, KeyError) as exc: + return f"dry read failed: {exc}" + state = payload.get("state", {}) + if not state.get("reachable"): + return "dry read: the store could not be read" + detail = state.get("detail") or {} + for name, sub in detail.items(): + if isinstance(sub, dict) and "reachable" in sub and not sub.get("reachable"): + if name == "openadapt:frame_audit": + continue + return f"dry read: surface {name!r} could not be read" + if mysql: + snap = detail.get(FRAME_DB_DETAIL) + if not isinstance(snap, dict): + from openadapt_flow.runtime.effects.db_snapshot import ( + mac_key, + take_snapshot, + ) + from openadapt_flow.verify_run import _audit_sql, _db_connect, _ref_columns + + frame = contract.config.frame + database: Any = frame.database if frame is not None else None + try: + take_snapshot( + _db_connect(database.read), + database.dialect, + key=mac_key(b"0" * 32), + modes=database.modes(), + max_rows=database.max_rows, + extra_queries=_audit_sql(contract.config), + ignore=database.all_ignore_columns(), + ref_columns=_ref_columns(contract.config), + schemas=database.schemas, + links=database.recheck_after_s is not None, + ) + why = "no error on a second read" + except Exception as exc: # noqa: BLE001 - reported + why = f"{type(exc).__name__}: {exc}" + return f"dry read: the whole-database frame could not be read ({why})" + if snap.get("code_visible") is False: + return ( + "dry read: the read account can't see triggers (MySQL needs the " + "TRIGGER privilege on the database), so the frame can't be checked" + ) + return None + + +def check_read_only(connect: Any, dialect: str) -> list[str]: + """Why the read account is not read-only (empty when it is).""" + + problems: list[str] = [] + conn = connect() + try: + cur = conn.cursor() + if dialect == "mysql": + cur.execute("SHOW GRANTS") + for (grant,) in cur.fetchall(): + g = str(grant).upper() + privs = g.split(" ON ")[0].replace("GRANT ", "") + for word in ( + "ALL PRIVILEGES", + "INSERT", + "UPDATE", + "DELETE", + "CREATE", + "DROP", + "ALTER", + "INDEX", + "EXECUTE", + "FILE", + "SUPER", + "EVENT", + ): + if re.search(rf"\b{word}\b", privs) and not privs.startswith( + "USAGE" + ): + problems.append(f"the read account has {word}: {grant}") + else: + # Every role the account is, inherits, or can SET ROLE to (a + # NOINHERIT membership): what any of them can do, the account can. + roles = ( + "(SELECT oid, rolname, rolsuper, rolcreaterole, rolcreatedb " + "FROM pg_roles WHERE pg_has_role(current_user, oid, 'MEMBER')) r" + ) + user_schemas = ( + "n.nspname NOT IN ('pg_catalog', 'information_schema') " + "AND n.nspname NOT LIKE 'pg\\_toast%' " + "AND n.nspname NOT LIKE 'pg\\_temp%'" + ) + + def who(role: str) -> str: + return "the read account" + ("" if role == user else f" (as {role})") + + cur.execute("SELECT current_user") + user = str(cur.fetchone()[0]) + cur.execute( + "SELECT r.rolname FROM " + roles + " WHERE r.rolsuper OR " + "r.rolcreaterole OR r.rolcreatedb ORDER BY 1" + ) + for (role,) in cur.fetchall(): + problems.append( + f"{who(role)} is a superuser or can create roles or databases" + ) + cur.execute( + "SELECT r.rolname, n.nspname, c.relname, p.priv FROM " + roles + " " + "CROSS JOIN pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace " + "CROSS JOIN unnest(ARRAY['INSERT', 'UPDATE', 'DELETE', 'TRUNCATE', " + "'TRIGGER', 'REFERENCES']) AS p(priv) " + "WHERE c.relkind IN ('r', 'p', 'v', 'm', 'f') AND " + + user_schemas + + " AND (has_table_privilege(r.oid, c.oid, p.priv) OR (p.priv IN " + "('INSERT', 'UPDATE', 'REFERENCES') AND " + "has_any_column_privilege(r.oid, c.oid, p.priv))) " + "ORDER BY 1, 2, 3, 4 LIMIT 20" + ) + for role, schema_name, t, priv in cur.fetchall(): + problems.append( + f"{who(role)} has {priv} on table {schema_name}.{t} " + "(or on one of its columns)" + ) + cur.execute( + "SELECT r.rolname, n.nspname, c.relname, p.priv FROM " + roles + " " + "CROSS JOIN pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace " + "CROSS JOIN unnest(ARRAY['USAGE', 'UPDATE']) AS p(priv) " + "WHERE c.relkind = 'S' AND " + + user_schemas + + " AND has_sequence_privilege(r.oid, c.oid, p.priv) " + "ORDER BY 1, 2, 3, 4 LIMIT 20" + ) + for role, schema_name, t, priv in cur.fetchall(): + problems.append( + f"{who(role)} has {priv} on sequence {schema_name}.{t} " + "(it can move the counter)" + ) + cur.execute( + "SELECT r.rolname, n.nspname FROM " + roles + " " + "CROSS JOIN pg_namespace n WHERE " + + user_schemas + + " AND has_schema_privilege(r.oid, n.oid, 'CREATE') ORDER BY 1, 2" + ) + for role, schema_name in cur.fetchall(): + problems.append(f"{who(role)} has CREATE on schema {schema_name}") + cur.execute( + "SELECT r.rolname FROM " + roles + " WHERE " + "has_database_privilege(r.oid, current_database(), 'CREATE') ORDER BY 1" + ) + for (role,) in cur.fetchall(): + problems.append(f"{who(role)} has CREATE on the database") + finally: + conn.rollback() + conn.close() + return problems + + +def read_account_notes(connect: Any, dialect: str) -> list[str]: + """What the read account can do beyond reading, that the contracts + still need (MySQL: TRIGGER, so the whole-database frame sees triggers; + it also lets the account create them).""" + + if dialect != "mysql": + return [] + return ( + [ + "the read account holds TRIGGER (needed to see triggers; it can also " + "create and drop triggers, so keep its password from the agent)" + ] + if _sees_triggers(connect) + else [] + ) + + +def guard_problem(sql: str) -> Optional[str]: + from openadapt_flow.runtime.effects.sql import assert_read_only_sql + + try: + assert_read_only_sql(sql) + except ValueError as exc: + return str(exc) + return None + + +def _admin_connect(admin_read: Mapping[str, Any]) -> Any: + import importlib + + module = importlib.import_module(str(admin_read["sql_driver"])) + args = dict(admin_read.get("sql_connect_args") or {}) + env = admin_read.get("sql_password_env") + if env: + if env not in os.environ: + raise GenerateError( + f"environment variable {env} (the admin password) is not set" + ) + args["password"] = os.environ[str(env)] + return module.connect(**args) + + +def _account(conn: Any, reader: str) -> str: + """``'user'@'host'`` from ``user@host`` or ``'user'@'host'``, each part + escaped by the driver: the account name never reaches SQL as written.""" + + m = re.fullmatch(r"'?([^'@]+)'?(?:@'?([^']*)'?)?", reader.strip()) + if m is None: + raise GenerateError(f"--reader-account {reader!r} is not user@host") + user, host = m.group(1), m.group(2) if m.group(2) is not None else "%" + return f"{conn.escape(user)}@{conn.escape(host)}" + + +def grant_trigger(admin_read: Mapping[str, Any], schema: Schema, reader: str) -> str: + """MySQL shows a table's triggers only to an account with the TRIGGER + privilege, and the whole-database frame must see them (a trigger planted + during a write could otherwise pass). Run only with ``--grant-trigger``.""" + + conn = _admin_connect(admin_read) + try: + sql = ( + f"GRANT TRIGGER ON {store.quote('mysql', schema.database)}.* TO " + f"{_account(conn, reader)}" + ) + conn.cursor().execute(sql) + conn.commit() + finally: + conn.close() + return sql + + +def _sees_triggers(connect: Any) -> bool: + """Does the account hold TRIGGER (or ALL) on the database? A database + with no triggers shows none either way, so the grant is what's checked.""" + + conn = connect() + try: + cur = conn.cursor() + cur.execute("SHOW GRANTS") + grants = [str(g[0]).upper() for g in cur.fetchall()] + finally: + conn.rollback() + conn.close() + for g in grants: + privs = g.split(" ON ")[0] + if "TRIGGER" in privs or "ALL PRIVILEGES" in privs: + return True + return False + + +def view_schema(schema: Schema) -> str: + """Where the generator's views live: a schema of their own, beside the + app's, so the app's own upgrades and restores never drop them and the + whole-database frame doesn't read them as app tables.""" + + return f"{schema.database}_oa_read" + + +def existing_views( + connect: Any, schema: Schema, tables: Sequence[str] +) -> dict[str, str]: + """Table -> ``schema.view`` for the views the read account can see.""" + + conn = connect() + try: + cur = conn.cursor() + cur.execute( + "SELECT table_name FROM information_schema.views WHERE table_schema = %s", + (view_schema(schema),), + ) + have = {str(r[0]) for r in cur.fetchall()} + finally: + conn.rollback() + conn.close() + return { + t: f"{view_schema(schema)}.oa_read_{t}" + for t in tables + if f"oa_read_{t}" in have + } + + +def create_views( + admin_read: Mapping[str, Any], schema: Schema, tables: Sequence[str], reader: str +) -> dict[str, str]: + """Create one read-only view per table the SQL guard refuses by name, in + the schema ``_oa_read``, and grant the read account SELECT on + that schema. Only run with ``--create-views``; the connection is the + admin one given.""" + + if schema.dialect != "mysql": + raise GenerateError("--create-views supports MySQL and MariaDB") + conn = _admin_connect(admin_read) + made = {} + vs = store.quote("mysql", view_schema(schema)) + try: + cur = conn.cursor() + cur.execute(f"CREATE DATABASE IF NOT EXISTS {vs}") + for t in tables: + q = store.quote("mysql", schema.database) + "." + store.quote("mysql", t) + v = vs + "." + store.quote("mysql", f"oa_read_{t}") + cur.execute( + f"CREATE OR REPLACE SQL SECURITY DEFINER VIEW {v} AS SELECT * FROM {q}" + ) + made[t] = f"{view_schema(schema)}.oa_read_{t}" + cur.execute(f"GRANT SELECT ON {vs}.* TO {_account(conn, reader)}") + conn.commit() + finally: + conn.close() + return made + + +# -- the whole run -------------------------------------------------------------------- + + +def propose( + *, + read: dict[str, Any], + kinds: list[KindSpec], + demo_dir: Path, + out: Path, + actor_params: Mapping[str, str], + answers: Mapping[str, Any], + environment_id: str = "", + admin_read: Optional[Mapping[str, Any]] = None, + create_views_flag: bool = False, + reader_account: str = "", + validate: bool = True, + grant_trigger_flag: bool = False, +) -> dict[str, Any]: + """Analyze, ask, emit and validate. Returns the report (also written).""" + + connect = store.connector(read) + dialect = store.dialect_of(read) + schema = store.read_schema(connect, dialect) + problems = check_read_only(connect, dialect) + if problems: + raise GenerateError( + "the read account is not read-only, so no contract is proposed: " + + "; ".join(problems) + ) + applied: list[str] = [] + if dialect == "mysql" and not _sees_triggers(connect): + if grant_trigger_flag: + if not admin_read or not reader_account: + raise GenerateError( + "--grant-trigger needs --admin-read and --reader-account" + ) + applied.append(grant_trigger(admin_read, schema, reader_account)) + if not _sees_triggers(connect): + raise GenerateError( + "the read account can't see triggers, so the whole-database frame " + "would never be checked (every write would be UNCERTAIN). MySQL shows " + "triggers only to an account with the TRIGGER privilege. Rerun with " + "--grant-trigger --admin-read --reader-account , " + "or have the DBA run: GRANT TRIGGER ON .* TO " + ) + demos = load_demos(demo_dir) + env = environment_id or f"generated-{schema.database}" + env = re.sub(r"[^A-Za-z0-9._:-]", "-", env) + if len(env) < 8: + env = (env + "-environment")[:64] + plans = [analyze(schema, k, demos, actor_params) for k in kinds] + # Versions of an update's target, chosen once every kind's mode is + # known: the creates' and the updates' audit rows tell write events apart. + create_kinds = {p.kind for p in plans if p.mode == "create"} + update_kinds = {p.kind for p in plans if p.mode == "update"} + for p in plans: + if p.mode == "update" and not p.refused: + # A version column first: every existing row has a version + # there. An audit trail misses records saved before it, or + # outside the app, and such a target stops as `missing`. + p.versions = _version_column(schema, p) or _audit_source( + schema, + p, + actor_params, + [d for d in demos if d.kind in create_kinds], + [d for d in demos if d.kind in update_kinds], + ) + questions: list[Question] = [] + + _share_locations(schema, plans, kinds) + _break_ties_across(plans) + location_questions: list[Question] = [] + # Q: per kind, a value the generator couldn't place. + for p in plans: + if not p.unresolved: + continue + chosen = dict(_answer(answers, f"locations.{p.kind}") or {}) + asked = dict(p.unresolved) + open_ = {} + for name, tied in list(p.unresolved.items()): + pick = chosen.get(name) + if isinstance(pick, int) and 0 <= pick < len(tied): + _resolve(p, name, tied[pick]) + else: + open_[name] = tied + p.unresolved = open_ + if asked: + parts = [] + for name, tied in asked.items(): + if tied: + opts = "; ".join( + f"[{i}] {loc.describe(p.anchor)}" for i, loc in enumerate(tied) + ) + parts.append(f"`{name}` is in several places: {opts}") + else: + parts.append( + f"`{name}` was not found in any row the demonstrations wrote; " + "tasks that give it are stopped" + ) + location_questions.append( + Question( + f"locations.{p.kind}", + f"Kind `{p.kind}`: " + + " ".join(parts) + + f" Answer `locations.{p.kind}: {{: }}` to pick one.", + None, + ) + ) + if not open_: + answers = {**answers, f"locations.{p.kind}": chosen} + # Q: phone numbers. Comparing by digits is a normalization a person + # confirms per parameter; it is never inferred from a value's shape. + phones = sorted( + { + f"{p.kind}.{name}" + for p in plans + for name, loc in _all_locations(p) + if loc.rule == "phone" + } + ) + if phones: + confirmed = _answer(answers, "phones") + confirmed = set(confirmed) if isinstance(confirmed, list) else set() + questions.append( + Question( + "phones", + "These values were stored as other text with the same digits (a " + "phone number written with spaces stored without): " + + ", ".join(phones) + + ". Compare them as phone numbers (the same digits, at least 7, " + "and a task value that isn't shaped like a phone number is " + "compared exactly)? Answer `phones: [, ...]` with " + "the ones that are phone numbers; any other is compared exactly.", + phones, + ) + ) + for p in plans: + _confirm_phones( + p, {x.split(".", 1)[1] for x in confirmed if x.startswith(p.kind + ".")} + ) + + # A row the app changes by key (a user's preferences, a counter) is the + # app's, whichever kind of write a demonstration happened to make when + # it changed: the same excuse, on that row only, applies to every kind. + keyed: dict[tuple[str, str], dict[str, Any]] = {} + for p in plans: + for row_spec in p.touched: + if row_spec.get("by_key"): + k = ( + row_spec["table"], + json.dumps(row_spec["selector"], sort_keys=True), + ) + merged = keyed.setdefault(k, {**row_spec, "columns": {}}) + merged["columns"].update(row_spec["columns"]) + for p in plans: + p.touched = [x for x in p.touched if not x.get("by_key")] + [ + {**x, "columns": dict(x["columns"])} for x in keyed.values() + ] + + # Q: bookkeeping, for the whole app at once. + ans = _answer(answers, "bookkeeping") + if ans is not None and ans != "accept" and not isinstance(ans, dict): + raise GenerateError( + "bookkeeping: answer `accept` or a mapping with accept, logs_add, " + "columns_add, columns_remove and unchecked" + ) + if isinstance(ans, dict): + unknown = set(ans) - { + "accept", + "logs_add", + "columns_add", + "columns_remove", + "unchecked", + } + if unknown: + raise GenerateError(f"bookkeeping: unknown answer keys {sorted(unknown)}") + if "accept" in ans and not isinstance(ans["accept"], bool): + raise GenerateError("bookkeeping.accept must be true or false") + # Only `accept` (or `accept: true`) takes the proposed columns; a mapping + # without it takes only what it names. + accepted = ans == "accept" or (isinstance(ans, dict) and ans.get("accept") is True) + ans = ans if isinstance(ans, dict) else {} + + def table_cols(key: str) -> dict[str, set[str]]: + raw = ans.get(key) or {} + if not isinstance(raw, dict) or not all( + isinstance(c, list) and all(isinstance(x, str) for x in c) + for c in raw.values() + ): + raise GenerateError(f"bookkeeping.{key}: a mapping of table -> [columns]") + return {str(t): set(c) for t, c in raw.items()} + + raw_logs = ans.get("logs_add") or [] + if not isinstance(raw_logs, list) or not all(isinstance(x, str) for x in raw_logs): + raise GenerateError("bookkeeping.logs_add: a list of tables") + log_add = set(raw_logs) + columns_add = table_cols("columns_add") + columns_remove = table_cols("columns_remove") + unchecked_named = table_cols("unchecked") + proposed: dict[str, set[str]] = {} + offered: dict[str, set[str]] = {} + where_seen: dict[str, set[str]] = {} + for p in plans: + for col, kind in p.anchor_extra.items(): + (proposed if kind in BOOKKEEPING_KINDS else offered).setdefault( + p.anchor, set() + ).add(col) + where_seen.setdefault(f"{p.anchor}", set()).add("the target row") + for row_spec in p.touched: + for col, kind in row_spec["columns"].items(): + (proposed if kind in BOOKKEEPING_KINDS else offered).setdefault( + row_spec["table"], set() + ).add(col) + where_seen.setdefault(row_spec["table"], set()).add(row_spec["row"]) + for t, cols in columns_add.items(): + extra = cols - proposed.get(t, set()) - offered.get(t, set()) + if extra: + raise GenerateError( + f"bookkeeping.columns_add names {t}: {', '.join(sorted(extra))}, " + "which the question didn't list" + ) + excused: dict[str, set[str]] = {} + for t in set(proposed) | set(columns_add): + took = proposed.get(t, set()) if accepted else set() + excused[t] = (took | columns_add.get(t, set())) - columns_remove.get(t, set()) + # Columns of a new row no rule checks, per kind and table. + open_by_kind: dict[str, dict[str, dict[str, str]]] = {} + for p in plans: + own: dict[str, dict[str, str]] = {} + if p.mode == "create" and p.demos: + for v in [Variant(p.kind, sorted(p.locations))]: + for col, role in _anchor_roles(schema, p, v, actor_params).items(): + if role.role in OPEN_ROLES: + own.setdefault(p.anchor, {})[col] = role.role + for g in [*p.children.values(), *p.lists.values()]: + for col, role in g.roles.items(): + if role.role in OPEN_ROLES: + own.setdefault(g.table, {})[col] = role.role + for sd in p.sides: + for col, role in sd.roles.items(): + if role.role in OPEN_ROLES: + own.setdefault(sd.table, {})[col] = role.role + open_by_kind[p.kind] = own + unchecked: dict[str, set[str]] = {} + for own_cols in open_by_kind.values(): + for t, roles_of in own_cols.items(): + unchecked.setdefault(t, set()).update(roles_of) + proposed_open: dict[str, list[str]] = {} + for own_cols in open_by_kind.values(): + for t, roles_of in own_cols.items(): + for c, role_name in roles_of.items(): + if role_name != "free": + proposed_open.setdefault(t, []) + if c not in proposed_open[t]: + proposed_open[t].append(c) + proposed_open = {t: sorted(c) for t, c in sorted(proposed_open.items())} + for t, cols in unchecked_named.items(): + extra = cols - unchecked.get(t, set()) + if extra: + raise GenerateError( + f"bookkeeping.unchecked names {t}: {', '.join(sorted(extra))}, " + "which the question didn't list" + ) + for p in plans: + # A column that varies or is the write's time stamp can't be pinned: + # unnamed, the kind has no contract. Free text left empty is pinned + # empty instead. + missing = sorted( + f"{t}.{c}" + for t, roles_of in open_by_kind[p.kind].items() + for c, role in roles_of.items() + if role != "free" and c not in unchecked_named.get(t, set()) + ) + if missing and not p.refused: + p.refused.append( + "new-row columns no rule checks and the answer doesn't name under " + "bookkeeping.unchecked: " + ", ".join(missing) + ) + # Tables a person may name as logs, per kind: those the kind's + # demonstrations added rows to beside the write. + log_offer = { + p.kind: set(p.log_candidates) | {s.table for s in p.sides} for p in plans + } + candidates = set().union(*log_offer.values()) if plans else set() + extra_logs = log_add - candidates + if extra_logs: + raise GenerateError( + "bookkeeping.logs_add names tables the question didn't list: " + + ", ".join(sorted(extra_logs)) + ) + sides = sorted({f"{s.table} ({s.count} per write)" for p in plans for s in p.sides}) + for p in plans: # a table the person calls a log is no longer counted + p.logs = log_add & log_offer[p.kind] + p.sides = [x for x in p.sides if x.table not in p.logs] + unexcused = {t: c for p in plans for t, c in p.unexcused.items()} + if proposed or offered or unchecked or candidates or sides or unexcused: + lines = [] + if proposed: + lines.append( + "Proposed as the app's own bookkeeping, excused only on the rows " + "the demonstrations changed (" + + "; ".join( + f"{t}: {', '.join(sorted(where_seen.get(t, set())))}" + for t in sorted(proposed) + ) + + "): " + + "; ".join( + f"{t}: {', '.join(sorted(c))}" for t, c in sorted(proposed.items()) + ) + + "." + ) + if offered: + lines.append( + "Also changed with no task value, and not excused unless you add " + "them under columns_add (a write that changes them stops): " + + "; ".join( + f"{t}: {', '.join(sorted(c))}" for t, c in sorted(offered.items()) + ) + + "." + ) + if unexcused: + lines.append( + "Changed on rows the generator can't identify, so a write that " + "changes them stops: " + + "; ".join( + f"{t}: {', '.join(sorted(c))}" for t, c in sorted(unexcused.items()) + ) + + "." + ) + if sides: + lines.append( + "Rows the app adds beside each write, expected exactly that many " + "times: " + ", ".join(sides) + "." + ) + if candidates: + lines.append( + "Tables the app added rows to beside the write. None is a log " + "unless you list it under logs_add (for the kinds whose " + "demonstrations added rows there, rows may then be added any " + "number of times, existing rows must not change): " + + ", ".join(sorted(candidates)) + + "." + ) + if unchecked: + lines.append( + "Columns of a new row no rule checks (they differ between " + "demonstrations, are free text every demonstration left empty, or " + "are the write's time stamp). Each is checked only as you name it " + "under unchecked: free text you don't name must stay empty, and a " + "kind with another column you don't name has no contract: " + + "; ".join( + f"{t}: {', '.join(sorted(c))}" for t, c in sorted(unchecked.items()) + ) + + "." + ) + questions.append( + Question( + "bookkeeping", + "Is this the app's own bookkeeping? " + + " ".join(lines) + + " Answer `bookkeeping: accept`, or a mapping with any of " + "accept (true takes the proposed columns), logs_add, columns_add, " + "columns_remove and unchecked. An answer can name only what this " + "question lists. Unanswered, nothing is excused, no table is a " + "log and no column is left unchecked.", + # The proposal names the columns that vary or are time stamps, + # so accepting it is explicit. Free text every demonstration + # left empty stays pinned empty unless the person adds it. + {"accept": True, "unchecked": proposed_open} + if proposed_open + else "accept", + ) + ) + # Q: the recheck window (frame.database.recheck_after_s). + recheck: Optional[float] = None + a = _answer(answers, "recheck_after_s") + recheck = float(a) if isinstance(a, (int, float)) and a > 0 else None + questions.append( + Question( + "recheck_after_s", + "How long after a verified write can the app still write on its own " + "(a background job, a delayed retry)? The database is read again that " + f"many seconds later and a late change refutes the write. Proposed: " + f"{DEFAULT_RECHECK_S}. Answer `recheck_after_s: `.", + DEFAULT_RECHECK_S, + ) + ) + for p in plans: # child groups of values placed after analysis + _children(schema, p, [], only_missing=True) + questions.extend(location_questions) + questions_open = [q for q in questions if _answer(answers, q.key) is None] + settings = Settings( + read, + dict(actor_params), + recheck or DEFAULT_RECHECK_S, + env, + unchecked=unchecked_named, + ) + + # Tables whose name the SQL guard refuses: a view, or a refusal. + blocked = sorted( + t + for t in schema.tables + if guard_problem(f"SELECT * FROM {store.quote(dialect, t)} t") is not None + ) + needed = sorted( + {p.anchor for p in plans if p.anchor in blocked} + | { + g.table + for p in plans + for g in list(p.children.values()) + list(p.lists.values()) + if g.table in blocked + } + ) + if needed: + existing = existing_views(connect, schema, needed) + missing = [t for t in needed if t not in existing] + if missing and create_views_flag: + if not admin_read or not reader_account: + raise GenerateError( + "--create-views needs --admin-read and --reader-account" + ) + made = create_views(admin_read, schema, missing, reader_account) + applied.extend(f"view {v} over {t}" for t, v in made.items()) + existing = existing_views(connect, schema, needed) + settings.views = existing + for p in plans: + tables = {p.anchor} | { + g.table for g in list(p.children.values()) + list(p.lists.values()) + } + for t in sorted(tables & set(blocked)): + if t not in existing: + p.refused.append( + f"the SQL guard refuses the table name {t!r} in a read; rerun " + "with --create-views --admin-read --reader-account " + " to read it through a view, or wait for the guard fix" + ) + + out.mkdir(parents=True, exist_ok=True) + manifest: dict[str, Any] = { + "schema_version": MANIFEST_SCHEMA, + "dialect": dialect, + "database": schema.database, + "actor_params": sorted(actor_params), + "questions_open": [q.key for q in questions_open], + "kinds": {}, + } + report: dict[str, Any] = { + "applied": applied, + "read_account": read_account_notes(connect, dialect), + "dialect": dialect, + "tables": len(schema.tables), + "questions": [ + { + "number": i + 1, + "key": q.key, + "text": q.text, + "proposal": q.proposal, + "answered": _answer(answers, q.key) is not None, + } + for i, q in enumerate(questions) + ], + "kinds": {}, + } + for p, spec in zip(plans, kinds): + entry: dict[str, Any] = { + "mode": p.mode, + "main_table": p.anchor, + "parameters": p.params, + "identity": sorted(p.identity), + "lists": sorted(p.lists), + "dispatch": p.dispatch, + "variants": [], + } + rep: dict[str, Any] = { + "mode": p.mode, + "main_table": p.anchor, + "locations": { + k: v.describe(p.anchor) for k, v in sorted(p.locations.items()) + }, + "placed_by_name": sorted(p.by_name), + "versions": p.versions, + "placed_from_another_kind": sorted(p.shared), + "demonstrations": len(p.demos), + "children": {k: g.table for k, g in p.children.items()}, + "dispatch_children": { + dv: {k: g.table for k, g in gs.items()} + for dv, gs in p.dispatch_children.items() + }, + "lists": {k: g.table for k, g in p.lists.items()}, + "side_rows": [ + {"table": s.table, "count": s.count, "unless": s.unless} + for s in p.sides + ], + "logs": sorted(p.logs), + "log_candidates": sorted(p.log_candidates), + "touched_rows": p.touched, + "unexcused": {t: sorted(c) for t, c in p.unexcused.items()}, + "refused": list(p.refused), + "unresolved": { + k: [loc.describe(p.anchor) for loc in v] + for k, v in sorted(p.unresolved.items()) + }, + "emitted": 0, + "rejected": [], + } + if p.removed: + rep["refused"].append( + "the demonstrations removed rows the generator can't explain: " + + ", ".join(f"{t} ({n})" for t, n in sorted(p.removed.items())) + ) + if p.refused or not p.mode: + manifest["kinds"][p.kind] = entry + report["kinds"][p.kind] = rep + continue + flavors = ["create", "found"] if p.mode == "create" else ["update", "found"] + for v in variants(schema, p, spec, answers): + if any( + f not in p.locations + and f != p.dispatch + and f not in p.children + and f not in (p.dispatch_locations.get(v.dispatch_value or "", {})) + for f in v.fields + ): + continue + ok = True + for flavor in flavors: + doc = build_contract( + schema, p, v, flavor, settings, excused, set(p.logs) + ) + rel = Path("contracts") / p.kind / f"{v.slug}.{flavor}.yaml" + path = out / rel + path.parent.mkdir(parents=True, exist_ok=True) + header = ( + f"# Generated by openadapt_flow.generate.verify_contract: kind {p.kind}, " + f"{flavor}, fields {', '.join(v.fields + [f'{k}x{n}' for k, n in v.lists.items()])}.\n" + "# Proposed, not admitted: a person answers QUESTIONS.md first.\n" + ) + path.write_text( + header + + yaml.safe_dump( + doc, sort_keys=False, allow_unicode=True, width=100 + ), + encoding="utf-8", + ) + why = None + for s in doc["verifier"]["surfaces"].values(): + why = why or guard_problem(s["sql_query"]) + if why is None and validate: + why = dry_read(path, flatten(sample_values(p, v, demos)), True) + if why is not None: + path.unlink() + rep["rejected"].append( + {"variant": v.slug, "flavor": flavor, "why": why} + ) + ok = False + break + v.files[flavor] = str(rel) + if ok: + rep["emitted"] += 1 + entry["variants"].append( + { + "fields": v.fields, + "lists": v.lists, + "dispatch_value": v.dispatch_value, + **{k: f for k, f in v.files.items()}, + } + ) + else: + for f in v.files.values(): + (out / f).unlink(missing_ok=True) + manifest["kinds"][p.kind] = entry + report["kinds"][p.kind] = rep + (out / "manifest.yaml").write_text( + yaml.safe_dump(manifest, sort_keys=False, allow_unicode=True), encoding="utf-8" + ) + (out / "report.json").write_text( + json.dumps(report, indent=2, ensure_ascii=False), encoding="utf-8" + ) + (out / "QUESTIONS.md").write_text( + _questions_md(questions, answers, report), encoding="utf-8" + ) + report["questions_open"] = [q.key for q in questions_open] + return report + + +def _questions_md( + questions: Sequence[Question], answers: Mapping[str, Any], report: Mapping[str, Any] +) -> str: + lines = ["# Questions for a person", ""] + if not questions: + lines.append( + "None: every decision followed from the schema and the demonstrations." + ) + for i, q in enumerate(questions, 1): + done = _answer(answers, q.key) + lines.append(f"{i}. {q.text}") + lines.append( + f" - Answer key: `{q.key}`. Proposed: `{json.dumps(q.proposal, ensure_ascii=False)}`." + ) + lines.append( + f" - {'Answered: `' + json.dumps(done, ensure_ascii=False) + '`' if done is not None else 'Not answered yet.'}" + ) + lines.append("") + lines.append("## Kinds") + lines.append("") + for kind, rep in report["kinds"].items(): + lines.append( + f"- `{kind}` ({rep['mode'] or 'no contract'}, main table `{rep['main_table']}`): {rep['emitted']} contract(s)." + ) + if rep.get("demonstrations") == 1 and rep["mode"] == "create": + lines.append( + " - One demonstration only: every column of a new record that the task " + "doesn't give must hold the value it held then. Record a second create " + "with other values, so a column computed from the values (a price with " + "tax) is told apart from a constant." + ) + for r in rep["refused"]: + lines.append(f" - Refused: {r}") + for r in rep["rejected"][:5]: + lines.append(f" - Not emitted ({r['variant']}, {r['flavor']}): {r['why']}") + return "\n".join(lines) + "\n" + + +# -- at run time --------------------------------------------------------------------------- + + +def choose( + manifest_path: Path, + kind: str, + values: Mapping[str, Any], + extra_params: Optional[Mapping[str, Any]] = None, + *, + require_answers: bool = True, +) -> tuple[Any, dict[str, Any]]: + """The contract for one task, and the params to call ``begin_write`` with. + + The field set is the task's non-null values (a list counts by length). + For a create kind, the store is read first: when a record with the + task's values already exists the ``found`` contract applies (nothing may + change), else the ``create`` one. Raises :class:`GenerateError` when no + contract fits: the write is then stopped, never sent unchecked. + """ + + manifest_path = Path(manifest_path) + manifest = load_yaml(manifest_path) or {} + if manifest.get("schema_version") != MANIFEST_SCHEMA: + raise GenerateError(f"{manifest_path} is not a contract manifest") + if require_answers and manifest.get("questions_open"): + raise GenerateError( + "the manifest still has open questions: " + + ", ".join(manifest["questions_open"]) + + " (answer them in answers.yaml and run propose again)" + ) + entry = (manifest.get("kinds") or {}).get(kind) + if not entry or not entry.get("variants"): + raise GenerateError(f"no contract for kind {kind!r}") + given = {k for k, v in values.items() if v is not None and not isinstance(v, list)} + lists = {k: len(v) for k, v in values.items() if isinstance(v, list)} + dispatch = entry.get("dispatch") + for var in entry["variants"]: + if set(var["fields"]) != given or dict(var.get("lists") or {}) != lists: + continue + if dispatch and str(values.get(dispatch)) != str(var.get("dispatch_value")): + continue + params = {**flatten(values), **dict(extra_params or {})} + base = manifest_path.parent + # The record already holds the task's values (found, or an update + # already in place): nothing may change. Else the create or update. + found = _load_for(base / var["found"], params) + if _exists(found, params): + return found, params + return _load_for(base / var[entry["mode"]], params), params + raise GenerateError( + f"no {kind} contract for the fields {sorted(given)}" + + (f" and lists {lists}" if lists else "") + + ": record a demonstration that gives them, or the task is stopped" + ) + + +def _load_for(path: Path, params: Mapping[str, Any]) -> Any: + """The contract at ``path`` for these values. A value the contract + compares as a phone number that isn't written like one (fewer than 7 + digits, or other characters) is compared exactly instead.""" + + from openadapt_flow.generate.demo_analysis import phone_shaped + from openadapt_flow.verifier import Contract + + doc = load_yaml(path) or {} + changed = False + for effect in doc.get("effects") or []: + for value in (effect.get("match") or {}).values(): + if ( + isinstance(value, dict) + and value.get("kind") == "phone" + and not phone_shaped(params.get(str(value.get("param")))) + ): + value.pop("kind") + changed = True + if not changed: + return Contract.load(path) + contract = Contract.from_dict(doc, name=Path(path).stem) + contract.config._base_dir = Path(path).resolve().parent + return contract + + +def _exists(contract: Any, params: Mapping[str, Any]) -> bool: + """Does every effect of the found contract hold now (the record, and each + row beside it that holds a value)?""" + + from openadapt_flow.runtime.effects.effect import record_matches + from openadapt_flow.runtime.effects.surfaces import surface_of + from openadapt_flow.verify_run import _build_verifier + + verifier = _build_verifier(contract.config, params) + state = verifier.capture_pre_state() + # Every effect of the found contract (the main record, and each row + # beside it that holds a value) must already hold. + for effect in contract.config.effects: + bound = effect.resolve(params) + sub = (state.detail or {}).get(surface_of(effect) or "record", {}) + if not sub.get("reachable"): + raise GenerateError("the store could not be read to choose the contract") + selector = dict(bound.match) + hits = sum( + 1 for r in sub.get("records", []) if record_matches(dict(r), selector) + ) + if hits != bound.expected_count: + return False + return True + + +# -- CLI ------------------------------------------------------------------------------------ + + +def _now() -> str: + return ( + _dt.datetime.now(_dt.timezone.utc) + .replace(tzinfo=None) + .isoformat(timespec="seconds") + ) + + +def _parse_actor(items: Sequence[str]) -> dict[str, str]: + out = {} + for item in items or []: + if "=" not in item: + raise GenerateError(f"--actor-param {item!r}: use NAME=VALUE") + k, v = item.split("=", 1) + out[k.strip()] = v.strip() + return out + + +def main(argv: Optional[Sequence[str]] = None) -> int: + ap = argparse.ArgumentParser( + prog="python -m openadapt_flow.generate.verify_contract", + description=( + "Propose openadapt.verify/v1 contracts for an app's kinds of write, from " + "its SQL schema (MySQL, MariaDB or PostgreSQL) and one demonstration " + "write per example. Steps: `schema` (check the read account), then for " + "each example `snapshot` before the write and `record` after it, then " + "`propose`. Answer QUESTIONS.md in answers.yaml and run `propose` again. " + "Guide: packages/flow/docs/GENERATE_CONTRACTS.md. For a FHIR ensure-record " + "contract use openadapt_flow.generate.ensure_contract instead." + ), + ) + sub = ap.add_subparsers(dest="command", required=True) + s = sub.add_parser( + "schema", help="read the schema and check the account is read-only" + ) + s.add_argument( + "--read", + required=True, + type=Path, + help="read.yaml: sql_driver, sql_connect_args, sql_password_env", + ) + s.add_argument("--out", type=Path, help="write schema.json here") + s = sub.add_parser( + "snapshot", help="snapshot every table before a demonstration write" + ) + s.add_argument("--read", required=True, type=Path) + s.add_argument( + "--out", + required=True, + type=Path, + help="e.g. work/before.json (holds row values: keep it private)", + ) + s = sub.add_parser( + "record", help="snapshot again after the write and save the demonstration" + ) + s.add_argument("--read", required=True, type=Path) + s.add_argument("--before", required=True, type=Path) + s.add_argument("--kind", required=True) + s.add_argument( + "--values", required=True, help="the task's values as JSON, or @file.json" + ) + s.add_argument( + "--demos", required=True, type=Path, help="directory of demonstrations" + ) + s.add_argument( + "--name", default="", help="file name stem (default: kind and a number)" + ) + s = sub.add_parser("propose", help="analyze the demonstrations and write contracts") + s.add_argument("--read", required=True, type=Path) + s.add_argument( + "--kinds", + required=True, + type=Path, + help="kinds.yaml: kinds, parameters, examples", + ) + s.add_argument("--demos", required=True, type=Path) + s.add_argument("--out", required=True, type=Path) + s.add_argument( + "--answers", type=Path, help="answers.yaml with the person's answers" + ) + s.add_argument( + "--actor-param", + action="append", + default=[], + help="NAME=VALUE the caller passes for the acting user (e.g. bot_user_id=5); a column holding it is the actor", + ) + s.add_argument("--environment-id", default="") + s.add_argument( + "--create-views", + action="store_true", + help="create a view for a table the SQL guard refuses by name (needs --admin-read)", + ) + s.add_argument( + "--admin-read", + type=Path, + help="admin.yaml: a connection allowed to CREATE VIEW and GRANT", + ) + s.add_argument( + "--reader-account", + default="", + help="the read account to GRANT SELECT on created views (MySQL: 'user'@'%%')", + ) + s.add_argument( + "--grant-trigger", + action="store_true", + help="MySQL: GRANT TRIGGER to the read account through --admin-read, so the frame can see triggers", + ) + s.add_argument( + "--no-validate", action="store_true", help="skip the dry read (tests only)" + ) + args = ap.parse_args(argv) + try: + read = load_read(args.read) + connect = store.connector(read) + dialect = store.dialect_of(read) + if args.command == "schema": + schema = store.read_schema(connect, dialect) + problems = check_read_only(connect, dialect) + if args.out: + args.out.write_text( + json.dumps(schema.to_json(), indent=1), encoding="utf-8" + ) + print( + f"{dialect} {schema.database}: {len(schema.tables)} tables, {len(schema.views)} views" + ) + for p in problems: + print(f"NOT READ-ONLY: {p}") + return 1 if problems else 0 + if args.command == "snapshot": + schema = store.read_schema(connect, dialect) + rows = store.snapshot(connect, schema) + store.save_snapshot(args.out, rows) + (args.out.with_suffix(".meta.json")).write_text( + json.dumps({"taken_at": _now()}), encoding="utf-8" + ) + print(f"snapshot of {len(rows)} tables -> {args.out}") + return 0 + if args.command == "record": + schema = store.read_schema(connect, dialect) + values = args.values + if values.startswith("@"): + values = Path(values[1:]).read_text(encoding="utf-8") + values = json.loads(values) + before = store.load_snapshot(args.before) + meta = args.before.with_suffix(".meta.json") + started = json.loads(meta.read_text())["taken_at"] if meta.exists() else "" + after = store.snapshot(connect, schema) + d = store.diff(before, after) + args.demos.mkdir(parents=True, exist_ok=True) + n = len(list(args.demos.glob(f"{args.kind}-*.demo.json"))) + 1 + stem = args.name or f"{args.kind}-{n}" + snap = f"{stem}.after.json" + store.save_snapshot(args.demos / snap, after) + doc = { + "kind": args.kind, + "values": values, + "started_at": started, + "finished_at": _now(), + "after_snapshot": snap, + "diff": d.to_json(), + } + path = args.demos / f"{stem}.demo.json" + path.write_text( + json.dumps(doc, indent=1, ensure_ascii=False), encoding="utf-8" + ) + os.chmod(path, 0o600) + print( + f"{path}: {len(d.changes)} row change(s) in {', '.join(d.tables()) or 'no table'}" + ) + return 0 + answers = load_yaml(args.answers) if args.answers else {} + admin = load_read(args.admin_read) if args.admin_read else None + report = propose( + read=read, + kinds=load_kinds(args.kinds), + demo_dir=args.demos, + out=args.out, + actor_params=_parse_actor(args.actor_param), + answers=answers or {}, + environment_id=args.environment_id, + admin_read=admin, + create_views_flag=args.create_views, + reader_account=args.reader_account, + validate=not args.no_validate, + grant_trigger_flag=args.grant_trigger, + ) + print( + f"questions: {len(report['questions'])} ({len(report['questions_open'])} open) -> {args.out / 'QUESTIONS.md'}" + ) + for kind, rep in report["kinds"].items(): + status = ( + "; ".join(rep["refused"]) + if rep["refused"] + else f"{rep['emitted']} contract(s)" + ) + print(f" {kind}: {status}") + return 0 + except (GenerateError, store.StoreError) as exc: + print(f"error: {exc}", file=sys.stderr) + return 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/packages/flow/openadapt_flow/runtime/effects/db_snapshot.py b/packages/flow/openadapt_flow/runtime/effects/db_snapshot.py index 93029d104a..3fc813e06d 100644 --- a/packages/flow/openadapt_flow/runtime/effects/db_snapshot.py +++ b/packages/flow/openadapt_flow/runtime/effects/db_snapshot.py @@ -20,16 +20,21 @@ primary key. - **One consistent read.** Everything is read on one connection inside one read-only transaction (``START TRANSACTION WITH CONSISTENT SNAPSHOT, READ - ONLY`` on MySQL and MariaDB; ``BEGIN`` on SQLite). The audit log is read in - the same transaction (``extra_queries``). MySQL's ``AUTO_INCREMENT`` isn't - part of a snapshot, so it is read just before the snapshot starts: a row - committed in between is in the snapshot, never missing from it. + ONLY`` on MySQL and MariaDB; ``BEGIN ISOLATION LEVEL REPEATABLE READ, READ + ONLY`` on PostgreSQL; ``BEGIN`` on SQLite). The audit log is read in the + same transaction (``extra_queries``). MySQL's ``AUTO_INCREMENT`` and + PostgreSQL's sequences aren't part of a snapshot, so they are read just + before the snapshot starts: a row committed in between is in the snapshot, + never missing from it. - **Schema and code.** Each table's definition and the database's triggers, views, routines and events are digested. On MySQL a trigger is visible only to an account with ``TRIGGER``; without it the snapshot says so (``code_visible``), and the frame is not checked. -- **Counters.** ``sqlite_sequence`` and ``AUTO_INCREMENT`` (on the column the - database numbers, key or not) show a row written and deleted in the window. +- **Counters.** ``sqlite_sequence``, ``AUTO_INCREMENT`` and a PostgreSQL + sequence a column owns (on the column the database numbers, key or not) + show a row written and deleted in the window. Any other PostgreSQL sequence + is read as a one-row table of its own, so a value drawn from it is a + change unless the allow-list skips it with a reason. - **The clock.** ``CURRENT_TIMESTAMP`` at the snapshot, so a column the database fills with the time can be held to the window. - **Allow-listed tables** are append-only by default: their existing rows @@ -38,12 +43,16 @@ and MariaDB a row is read as ``JSON_OBJECT``; a type whose value it can't carry as exact text is read another way (:func:`_mysql_encoding`: bytes, bit strings and geometries as hex, ``FLOAT`` widened to ``DOUBLE``, JSON as - its stored text), and numbers parse as ``Decimal``. :func:`canonical` gives - each value one exact text form. + its stored text), and numbers parse as ``Decimal``. On PostgreSQL each + column is read as text by a rule for its type (:func:`_pg_rule`), in a + session pinned to UTC and to exact float output; a type without a rule + makes the table unreadable. :func:`canonical` gives each value one exact + text form. Read-only: SQLite files open with ``mode=ro``; a driver connection should use -an account with SELECT (and, on MySQL, TRIGGER and SHOW VIEW) on every schema -the app can write. The baseline must still be protected like the records: +an account with SELECT (and, on MySQL, TRIGGER and SHOW VIEW; on PostgreSQL, +USAGE on each schema and SELECT on its sequences) on every schema the app can +write. The baseline must still be protected like the records: whoever holds both it and the host secret can test guesses against the MACs. """ @@ -53,6 +62,7 @@ import hashlib import json import os +import re import secrets import struct from collections.abc import Callable, Mapping, Sequence @@ -61,7 +71,7 @@ from pathlib import Path from typing import Any, Literal, Optional -Dialect = Literal["sqlite", "mysql"] +Dialect = Literal["sqlite", "mysql", "postgresql"] TableMode = Literal["compare", "append_only", "skip"] #: Columns per JSON object call: SQLite caps a function at 127 arguments. @@ -157,6 +167,8 @@ def canonical(value: Any) -> str: return "\x00null" if isinstance(value, bool): return "true" if value else "false" + if isinstance(value, StoredNumber): + return value.stored if isinstance(value, (int, float, Decimal)): number = Decimal(str(value)) if not number.is_finite(): @@ -181,6 +193,24 @@ def canonical(value: Any) -> str: return str(value) +class StoredNumber(Decimal): + """A PostgreSQL ``numeric`` declared without a scale, which keeps the + scale it was stored with: ``1.5`` and ``1.50`` are different stored + values. It equals, and prints as, its plain value (``1.5``), so a + selector or parameter compares by value; :func:`canonical` gives the + stored text, so a change of scale alone still shows.""" + + stored: str + + +def _stored_number(text: str) -> StoredNumber: + number = Decimal(text) + exact = Context(prec=max(len(number.as_tuple().digits), 1)) + value = StoredNumber(format(number.normalize(exact), "f")) + value.stored = text + return value + + def _interval_text(value: datetime.timedelta) -> str: micro = (value.days * 86400 + value.seconds) * 1_000_000 + value.microseconds sign, micro = ("-", -micro) if micro < 0 else ("", micro) @@ -307,6 +337,8 @@ class Snapshot: objects: str = "" code_visible: bool = True schemas: list[str] = field(default_factory=list) + #: Schemas the allow-list leaves out (``.*``, skip, with a reason). + schemas_skipped: list[str] = field(default_factory=list) tables: dict[str, Table] = field(default_factory=dict) #: In memory only: rows of ``extra_queries``, read in the same transaction. extra: dict[str, list[dict[str, Any]]] = field(default_factory=dict, repr=False) @@ -320,6 +352,11 @@ def to_json(self) -> dict[str, Any]: "objects": self.objects, "code_visible": self.code_visible, "schemas": self.schemas, + **( + {"schemas_skipped": self.schemas_skipped} + if self.schemas_skipped + else {} + ), "tables": { name: { "mode": t.mode, @@ -349,6 +386,7 @@ def from_json(cls, data: Mapping[str, Any]) -> "Snapshot": objects=str(data.get("objects", "")), code_visible=bool(data.get("code_visible", True)), schemas=list(data.get("schemas", [])), + schemas_skipped=list(data.get("schemas_skipped", [])), tables={ name: Table( mode=t["mode"], @@ -422,6 +460,8 @@ def _default_text(raw: Any, dialect: Dialect) -> Optional[str]: if raw is None: return None + if dialect == "postgresql": + return _pg_default_text(str(raw)) text = str(raw).strip() lowered = text.lower().strip("()") if lowered.startswith( @@ -435,6 +475,40 @@ def _default_text(raw: Any, dialect: Dialect) -> Optional[str]: return text +#: PostgreSQL defaults the database fills with the time (``pg_get_expr``). +_PG_CLOCK = re.compile( + r"(now\(\)|current_timestamp|current_date|localtimestamp|" + r"statement_timestamp\(\)|transaction_timestamp\(\)|clock_timestamp\(\)|" + r"\('now'::text\)::[a-z ]+)(::[a-z ()0-9]+)?", + re.IGNORECASE, +) + + +def _pg_default_text(expr: str) -> Optional[str]: + """A PostgreSQL default (``pg_get_expr``) as the value it stores. + + Only literals are read: ``NULL``, a quoted string with a cast + (``'open'::character varying``), a number (``0``, ``(-1)``) and + ``true``/``false``. A time function is ``CLOCK_DEFAULT``. Any other + expression (``nextval(...)``, ``gen_random_uuid()``) is kept as its + text, which no stored value equals, so a new row that relies on it must + list the column in ``new_row_fields``. + """ + + text = expr.strip() + if _PG_CLOCK.fullmatch(text): + return CLOCK_DEFAULT + if re.fullmatch(r"NULL(::[^']+)?", text, re.IGNORECASE): + return None + m = re.fullmatch(r"'((?:[^']|'')*)'::[^']+", text, re.S) + if m: + return m.group(1).replace("''", "'") + m = re.fullmatch(r"\(?(-?\d+(?:\.\d+)?(?:[eE][+-]?\d+)?)\)?(::[^']+)?", text) + if m: + return m.group(1) + return text + + # -- reading ------------------------------------------------------------------ @@ -454,10 +528,19 @@ class _Session: def __init__(self, connect: Callable[[], Any], dialect: Dialect) -> None: self.dialect = dialect self.con = connect() + if dialect == "postgresql": + # psycopg opens a transaction on the first statement unless told + # not to; the snapshot's own BEGIN must be the one that counts. + self.con.autocommit = True self.cur = self.con.cursor() + if dialect == "postgresql": + for sql in _PG_SESSION: + self.cur.execute(sql) def begin(self) -> None: - if self.dialect == "mysql": + if self.dialect == "postgresql": + self.cur.execute("BEGIN ISOLATION LEVEL REPEATABLE READ, READ ONLY") + elif self.dialect == "mysql": # Only statements that can't fail: a driver may end the session # on any error. MySQL 8 caches AUTO_INCREMENT unless told not to; # MariaDB has no such variable. @@ -514,6 +597,16 @@ class _Def: #: The column the database numbers (AUTO_INCREMENT, or SQLite's rowid #: alias), whether or not it is the primary key. auto_column: Optional[str] = None + #: The quoted relation to read (PostgreSQL: ``"schema"."table"``). + relation: str = "" + #: Why the table can't be read exactly (PostgreSQL); it is refused + #: unless the allow-list skips it. + unreadable: str = "" + #: Rows read already (a PostgreSQL sequence's one row), not from a table. + fixed: Optional[list[dict[str, Any]]] = None + #: The SELECT that reads the rows, when it isn't the table's own + #: (PostgreSQL large objects), one text column per ``columns``. + select: str = "" def _digest(parts: Sequence[Any]) -> str: @@ -784,11 +877,568 @@ def _row_part(table: str, chunk: Any) -> dict[str, Any]: return dict(parsed) +# -- PostgreSQL ----------------------------------------------------------------- + +#: The session a PostgreSQL snapshot reads in. Each setting fixes how a type +#: prints, so one stored value has one text: timestamps in UTC (no repeated +#: hour), ISO dates and intervals, floats with every digit they need +#: (shortest exact on PostgreSQL 12 and later, 17 digits before), bytes as +#: hex, ``money`` in the C locale (two decimals, read as ``numeric``). +_PG_SESSION = ( + "SET TimeZone = 'UTC'", + "SET DateStyle = 'ISO, YMD'", + "SET IntervalStyle = 'iso_8601'", + "SET extra_float_digits = 3", + "SET bytea_output = 'hex'", + "SET lc_monetary = 'C'", + "SET client_encoding = 'UTF8'", +) + +#: Schemas a PostgreSQL snapshot never reads: the catalogs, TOAST storage +#: and other sessions' temporary tables. +_PG_SYSTEM_SCHEMAS = ("pg_catalog", "information_schema") + + +def _pg_system(schema: str) -> bool: + return schema in _PG_SYSTEM_SCHEMAS or schema.startswith(("pg_toast", "pg_temp_")) + + +#: Built-in PostgreSQL types (in ``pg_catalog``) and the rule each is read +#: by. Every rule reads the column as text in the pinned session +#: (:data:`_PG_SESSION`), whose output is exact for these types: +#: +#: - ``int``: integers and ``oid``, parsed as ``int``. +#: - ``numeric``: parsed as ``Decimal``, every digit kept (``NaN`` and the +#: infinities too). Declared without a scale, as a :class:`StoredNumber`, +#: which keeps the scale it was stored with. +#: - ``float4``/``float8``: parsed as ``Decimal``; the plain value is the +#: shortest text that reads back as the same stored float. +#: - ``money``: read as ``numeric`` (in the C locale, hundredths). +#: - ``bool``: ``true``/``false`` (its text cast) as ``bool``. +#: - ``bytea``: the hex of the stored bytes, as ``bytes``. +#: - ``text``: the type's own text. Strings, ``uuid``, network addresses, +#: dates and times (UTC), ``interval`` (ISO 8601, months, days and +#: microseconds kept apart), bit strings, ``xml``, ``json`` (its stored +#: text), ``jsonb`` (its canonical text: keys sorted, duplicates gone), +#: ``tsvector`` and ``tsquery`` (sorted lexemes), and the geometric types +#: (their floats printed exactly). Arrays, ranges, multiranges, enums and +#: domains are read by their element, subtype or base type's rule. +_PG_RULES: dict[str, str] = { + **dict.fromkeys(("int2", "int4", "int8", "oid"), "int"), + "numeric": "numeric", + "float4": "float4", + "float8": "float8", + "money": "money", + "bool": "bool", + "bytea": "bytea", + **dict.fromkeys( + ( + "text", + "varchar", + "bpchar", + "name", + "char", + "uuid", + "inet", + "cidr", + "macaddr", + "macaddr8", + "date", + "time", + "timetz", + "timestamp", + "timestamptz", + "interval", + "bit", + "varbit", + "xml", + "json", + "jsonb", + "jsonpath", + "tsvector", + "tsquery", + "point", + "line", + "lseg", + "box", + "path", + "polygon", + "circle", + "pg_lsn", + ), + "text", + ), +} + + +@dataclass +class _PgType: + name: str + schema: str + kind: str # pg_type.typtype: b(ase), d(omain), e(num), r(ange), m(ultirange), c(omposite), p(seudo) + category: str + element: int + base: int + typmod: int = -1 # a domain's own modifier (a numeric's scale) + subtype: int = 0 # a range's subtype, or a multirange's range + + +def _pg_rule( + types: Mapping[int, _PgType], oid: int, typmod: int = -1, depth: int = 0 +) -> str: + """The rule a PostgreSQL column type is read by (:data:`_PG_RULES`). + + Raises ``ValueError`` naming the type when there is none: an extension's + type, a composite, a ``reg*`` alias (it prints by ``search_path``), and + anything else whose text this module hasn't been shown to be exact. + """ + + t = types.get(oid) + if t is None or depth > 8: + raise ValueError(f"type oid {oid} is not in the catalog") + label = f"{t.schema}.{t.name}" + if t.kind == "d": + return _pg_rule(types, t.base, t.typmod, depth + 1) + if t.kind == "e": + return "text" + if t.kind in ("r", "m"): + inner = _pg_rule(types, t.subtype, -1, depth + 1) + if inner == "money": + raise ValueError(f"{label} holds money, which prints by locale") + return "text" + if t.kind == "b" and t.category == "A" and t.element: + inner = _pg_rule(types, t.element, -1, depth + 1) + if inner == "money": + raise ValueError(f"{label} holds money, which prints by locale") + return "text" + if t.kind == "b" and t.schema == "pg_catalog" and t.name in _PG_RULES: + if t.name == "numeric" and typmod == -1: + return "numeric_free" + return _PG_RULES[t.name] + raise ValueError(f"no exact reading rule for type {label}") + + +def _pg_cell(column: str, rule: str) -> str: + q = _quote("postgresql", column) + if rule == "bytea": + return f"encode({q}, 'hex')" + if rule == "money": + return f"({q})::numeric::text" + if rule == "numeric_free": + return f"({q})::text" + return f"({q})::text" + + +def _pg_value(rule: str, text: Optional[str]) -> Any: + """A column's text (read by :func:`_pg_cell`) as its plain value.""" + + if text is None: + return None + if rule == "int": + return int(text) + if rule == "bool": + if text not in ("true", "false"): + raise SnapshotError(f"a boolean read as {text!r}") + return text == "true" + if rule == "bytea": + return bytes.fromhex(text) + if rule == "numeric_free": + return _stored_number(text) + if rule in ("numeric", "money", "float4", "float8"): + number = Decimal(text) + if not number.is_finite(): + return number + if rule == "float4": + return _float32_text(number) + if rule == "float8": + return Decimal(repr(float(number))) + exact = Context(prec=max(len(number.as_tuple().digits), 1)) + return Decimal(format(number.normalize(exact), "f")) + return str(text) + + +def _pg_in(names: Sequence[str]) -> str: + return "(" + ", ".join(_literal(n) for n in names) + ")" + + +@dataclass +class _PgSequence: + name: str + relation: str + increment: int + readable: bool + definition: list[Any] + #: The table and column that own it (``serial``, identity), if any. + table: Optional[str] = None + column: Optional[str] = None + #: The last value drawn (the start minus one step when none was). + last: Optional[int] = None + #: ``last_value`` and ``is_called`` as stored. + raw: Optional[int] = None + is_called: Optional[bool] = None + + +def _pg_scope( + s: _Session, schemas: Sequence[str], skipped: Sequence[str] = () +) -> tuple[list[str], str]: + """The schemas a PostgreSQL snapshot reads, and the default one (whose + tables are named without it). + + Every schema but the system ones is read, as on MySQL. ``schemas`` must + each be visible. A schema is left out only when the allow-list skips it + (``skipped``: ``.*`` with mode ``skip`` and a reason); any other + schema the account can't see is refused rather than left out.""" + + every = { + str(n): bool(u) + for n, u in s.rows( + "SELECT nspname, has_schema_privilege(oid, 'USAGE') FROM pg_namespace" + ) + if not _pg_system(str(n)) + } + unknown = sorted(set(skipped) - set(every)) + if unknown: + raise SnapshotError( + "allow-listed schemas not in the database: " + ", ".join(unknown) + ) + missing = sorted(x for x in schemas if not every.get(x) or x in skipped) + if missing: + raise SnapshotError( + "the read account can't see schema(s): " + ", ".join(missing) + ) + hidden = sorted(n for n, u in every.items() if not u and n not in skipped) + if hidden: + raise SnapshotError( + "the read account can't see schema(s) " + + ", ".join(hidden) + + ": grant it USAGE and SELECT there, or allow-list each as " + "'.*' with mode skip and a reason" + ) + scope = sorted(n for n in every if n not in skipped) + default = str(s.rows("SELECT current_schema()")[0][0] or "") + return scope, default + + +def _pg_name(schema: str, table: str, default: str) -> str: + return table if schema == default else f"{schema}.{table}" + + +def _pg_relation(schema: str, table: str) -> str: + return f"{_quote('postgresql', schema)}.{_quote('postgresql', table)}" + + +def _pg_sequences( + s: _Session, scope: Sequence[str], default: str +) -> dict[str, _PgSequence]: + """Every sequence in scope, with its value. Read before the snapshot + starts (a sequence isn't transactional): a row committed in between is + in the snapshot, so the counter never runs ahead of what it shows.""" + + out: dict[str, _PgSequence] = {} + for row in s.rows( + "SELECT n.nspname, c.relname, s.seqincrement, s.seqstart, s.seqmin, " + "s.seqmax, s.seqcycle, format_type(s.seqtypid, NULL), " + "has_sequence_privilege(c.oid, 'SELECT'), tn.nspname, t.relname, a.attname " + "FROM pg_sequence s JOIN pg_class c ON c.oid = s.seqrelid " + "JOIN pg_namespace n ON n.oid = c.relnamespace " + "LEFT JOIN pg_depend d ON d.classid = 'pg_class'::regclass " + "AND d.objid = c.oid AND d.refclassid = 'pg_class'::regclass " + "AND d.deptype IN ('a', 'i') AND d.refobjsubid > 0 " + "LEFT JOIN pg_class t ON t.oid = d.refobjid " + "LEFT JOIN pg_namespace tn ON tn.oid = t.relnamespace " + "LEFT JOIN pg_attribute a ON a.attrelid = d.refobjid " + "AND a.attnum = d.refobjsubid " + f"WHERE n.nspname IN {_pg_in(scope)} ORDER BY 1, 2" + ): + schema, name, inc, start, low, high, cycle, typ, ok, tschema, table, col = row + out[_pg_name(str(schema), str(name), default)] = _PgSequence( + _pg_name(str(schema), str(name), default), + _pg_relation(str(schema), str(name)), + int(inc), + bool(ok), + [int(inc), int(start), int(low), int(high), bool(cycle), str(typ)], + None if table is None else _pg_name(str(tschema), str(table), default), + None if col is None else str(col), + ) + readable = [q for q in out.values() if q.readable] + if readable: + for name, last, called in s.rows( + " UNION ALL ".join( + f"SELECT {_literal(q.name)}, last_value, is_called FROM {q.relation}" + for q in readable + ) + ): + q = out[str(name)] + q.raw, q.is_called = int(last), bool(called) + q.last = q.raw if called else q.raw - q.increment + return out + + +def _pg_definitions( + s: _Session, scope: Sequence[str], default: str +) -> tuple[dict[str, _Def], str]: + """Every PostgreSQL table's definition (partitions, not their parents), + and the digest of the database's code.""" + + within = _pg_in(scope) + types: dict[int, _PgType] = {} + for oid, name, schema, kind, cat, elem, base, mod in s.rows( + "SELECT t.oid, t.typname, n.nspname, t.typtype, t.typcategory, t.typelem, " + "t.typbasetype, t.typtypmod FROM pg_type t " + "JOIN pg_namespace n ON n.oid = t.typnamespace" + ): + types[int(oid)] = _PgType( + str(name), str(schema), str(kind), str(cat), int(elem), int(base), int(mod) + ) + # pg_range.rngmultitypid exists from PostgreSQL 14; read through JSON so + # an older server gives NULL instead of an error. + for rng, sub, multi in s.rows( + "SELECT r.rngtypid, r.rngsubtype, (to_jsonb(r) ->> 'rngmultitypid')::oid " + "FROM pg_range r" + ): + types[int(rng)].subtype = int(sub) + if multi is not None and int(multi) in types: + types[int(multi)].subtype = int(rng) + tables: dict[int, tuple[str, str, list[Any]]] = {} + out: dict[str, _Def] = {} + parents: list[Any] = [] + for row in s.rows( + "SELECT c.oid, n.nspname, c.relname, c.relkind, c.relpersistence, " + "c.relrowsecurity, c.relforcerowsecurity, has_table_privilege(c.oid, 'SELECT'), " + "CASE WHEN c.relrowsecurity THEN row_security_active(c.oid) ELSE false END " + "FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace " + f"WHERE c.relkind IN ('r', 'p', 'f') AND n.nspname IN {within} ORDER BY 2, 3" + ): + oid, schema, table, kind, persistence, rls, force, select, filtered = row + if kind == "p": + # A partitioned table holds no rows of its own; its partitions + # are read. Its definition is part of the database's code. + parents.append([schema, table]) + continue + name = _pg_name(str(schema), str(table), default) + tables[int(oid)] = (name, str(schema), [kind, persistence, rls, force]) + d = out[name] = _Def( + [], [], None, {}, {}, "", relation=_pg_relation(str(schema), str(table)) + ) + if kind == "f": + d.unreadable = "a foreign table: its rows live outside this database" + elif not select: + d.unreadable = "the read account has no SELECT on it" + elif filtered: + d.unreadable = "row-level security hides rows from the read account" + parts: dict[str, list[Any]] = {n: list(meta) for n, _, meta in tables.values()} + for ( + oid, + column, + typ, + mod, + ctype, + notnull, + cdefault, + identity, + generated, + coll, + ) in s.rows( + "SELECT a.attrelid, a.attname, a.atttypid, a.atttypmod, " + "format_type(a.atttypid, a.atttypmod), " + "a.attnotnull, pg_get_expr(ad.adbin, ad.adrelid), a.attidentity, a.attgenerated, " + "co.collname FROM pg_attribute a JOIN pg_class c ON c.oid = a.attrelid " + "JOIN pg_namespace n ON n.oid = c.relnamespace " + "LEFT JOIN pg_attrdef ad ON ad.adrelid = a.attrelid AND ad.adnum = a.attnum " + "LEFT JOIN pg_collation co ON co.oid = a.attcollation " + "WHERE a.attnum > 0 AND NOT a.attisdropped AND c.relkind IN ('r', 'f') " + f"AND n.nspname IN {within} ORDER BY a.attrelid, a.attnum" + ): + if int(oid) not in tables: + continue + name = tables[int(oid)][0] + d = out[name] + d.columns.append(str(column)) + d.defaults[str(column)] = _default_text(cdefault, "postgresql") + try: + d.encode[str(column)] = _pg_rule(types, int(typ), int(mod)) + except ValueError as exc: + d.encode[str(column)] = "" + d.unreadable = d.unreadable or f"column {column}: {exc}" + parts[name].append( + [column, ctype, notnull, cdefault, identity, generated, coll] + ) + for oid, column in s.rows( + "SELECT i.indrelid, a.attname FROM pg_index i " + "CROSS JOIN LATERAL unnest(i.indkey::int2[]) WITH ORDINALITY AS k(attnum, ord) " + "JOIN pg_attribute a ON a.attrelid = i.indrelid AND a.attnum = k.attnum " + "WHERE i.indisprimary ORDER BY i.indrelid, k.ord" + ): + if int(oid) in tables: + out[tables[int(oid)][0]].primary_key.append(str(column)) + for oid, con, kind, definition in s.rows( + "SELECT conrelid, conname, contype, pg_get_constraintdef(oid) " + "FROM pg_constraint WHERE conrelid <> 0 ORDER BY 1, 2" + ): + if int(oid) in tables: + parts[tables[int(oid)][0]].append([con, kind, definition]) + for name, d in out.items(): + d.schema = _digest(parts[name]) + code = [ + parents, + s.rows( + "SELECT n.nspname, c.relname, t.tgname, t.tgenabled, pg_get_triggerdef(t.oid) " + "FROM pg_trigger t JOIN pg_class c ON c.oid = t.tgrelid " + "JOIN pg_namespace n ON n.oid = c.relnamespace " + f"WHERE NOT t.tgisinternal AND n.nspname IN {within} ORDER BY 1, 2, 3" + ), + s.rows( + "SELECT n.nspname, c.relname, c.relkind, pg_get_viewdef(c.oid) " + "FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace " + f"WHERE c.relkind IN ('v', 'm') AND n.nspname IN {within} ORDER BY 1, 2" + ), + s.rows( + "SELECT n.nspname, p.proname, pg_get_function_identity_arguments(p.oid), " + "p.prokind, p.prosrc, p.probin, p.proconfig::text, p.prosecdef, l.lanname " + "FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace " + "JOIN pg_language l ON l.oid = p.prolang " + f"WHERE n.nspname IN {within} ORDER BY 1, 2, 3" + ), + s.rows( + "SELECT schemaname, tablename, rulename, definition FROM pg_rules " + f"WHERE schemaname IN {within} ORDER BY 1, 2, 3" + ), + s.rows( + "SELECT schemaname, tablename, policyname, permissive, roles::text, cmd, " + f"qual, with_check FROM pg_policies WHERE schemaname IN {within} " + "ORDER BY 1, 2, 3" + ), + s.rows( + "SELECT evtname, evtevent, evtenabled, evtfoid::regproc::text, evttags::text " + "FROM pg_event_trigger ORDER BY 1" + ), + s.rows("SELECT extname, extversion FROM pg_extension ORDER BY 1"), + s.rows( + "SELECT n.nspname, t.typname, t.typtype, " + "format_type(t.typbasetype, t.typtypmod), t.typnotnull, t.typdefault, " + "(SELECT string_agg(e.enumlabel, ',' ORDER BY e.enumsortorder) " + "FROM pg_enum e WHERE e.enumtypid = t.oid), " + "(SELECT string_agg(pg_get_constraintdef(c.oid), ';' ORDER BY c.conname) " + "FROM pg_constraint c WHERE c.contypid = t.oid) " + "FROM pg_type t JOIN pg_namespace n ON n.oid = t.typnamespace " + f"WHERE t.typtype IN ('d', 'e', 'r') AND n.nspname IN {within} " + "ORDER BY 1, 2" + ), + ] + return out, _digest(code) + + +def _pg_counters( + defs: dict[str, _Def], sequences: Mapping[str, _PgSequence] +) -> tuple[dict[str, int], list[_PgSequence]]: + """Each table's counter (the sequence its key, or else its first + numbered column, owns), and every other sequence, which is read as a + one-row table of its own.""" + + owned: dict[str, list[_PgSequence]] = {} + for q in sequences.values(): + if q.table in defs and q.column in defs[q.table].columns: + owned.setdefault(q.table, []).append(q) + counters: dict[str, int] = {} + used: set[str] = set() + for table, qs in owned.items(): + d = defs[table] + qs.sort(key=lambda q: d.columns.index(q.column or "")) + pick = next((q for q in qs if d.primary_key == [q.column]), qs[0]) + if pick.increment <= 0: + continue # a descending sequence is read as its own table + d.auto_column = pick.column + if d.primary_key == [pick.column]: + d.generated_key = pick.column + used.add(pick.name) + if pick.last is not None: + counters[table] = pick.last + else: + d.unreadable = d.unreadable or ( + f"the read account has no SELECT on its sequence {pick.name}" + ) + return counters, [q for n, q in sequences.items() if n not in used] + + +def _pg_sequence_def(q: _PgSequence) -> _Def: + """A sequence no table counts by, as a one-row table: a value drawn from + it is a change, like any other row.""" + + columns = ["sequence", "last_value", "is_called"] + d = _Def( + columns, + ["sequence"], + None, + dict.fromkeys(columns), + {}, + _digest(["sequence", *q.definition]), + relation=q.relation, + fixed=[], + ) + if q.raw is None: + d.unreadable = "a sequence the read account has no SELECT on" + else: + d.fixed = [{"sequence": q.name, "last_value": q.raw, "is_called": q.is_called}] + return d + + +#: The name the frame and the allow-list use for PostgreSQL's large objects. +LARGE_OBJECTS = "pg_largeobject" + + +def _pg_large_objects() -> _Def: + """Large objects as one table: each object's oid, owner, privileges and + the SHA-256 of its content. They live in ``pg_catalog``, which is + otherwise not read, and a column that holds an object's oid keeps it when + the content changes (``lo_put``). The content is read only when there + are objects; one the account can't read is refused.""" + + columns = ["oid", "owner", "acl", "sha256"] + return _Def( + columns, + ["oid"], + None, + dict.fromkeys(columns), + dict.fromkeys(columns, "text"), + _digest(["large objects", *columns]), + relation="pg_catalog.pg_largeobject_metadata", + select=( + "SELECT m.oid::text, m.lomowner::regrole::text, m.lomacl::text, " + "encode(sha256(lo_get(m.oid)), 'hex') FROM pg_largeobject_metadata m" + ), + ) + + +def _read_rows_pg(s: _Session, d: _Def) -> list[dict[str, Any]]: + if d.select: + try: + rows = s.rows(d.select) + except Exception as exc: # noqa: BLE001 - e.g. no SELECT on an object + raise SnapshotError( + f"large objects can't be read ({exc}); grant the read account " + f"SELECT on them, or allow-list {LARGE_OBJECTS} with mode skip " + "and a reason" + ) from exc + return [{c: _pg_value("text", v) for c, v in zip(d.columns, r)} for r in rows] + cells = ", ".join(_pg_cell(c, d.encode[c]) for c in d.columns) or "NULL" + return [ + {c: _pg_value(d.encode[c], v) for c, v in zip(d.columns, r)} + for r in s.rows(f"SELECT {cells} FROM ONLY {d.relation}") + ] + + +def key_text(primary_key: Sequence[str], row: Mapping[str, Any]) -> str: + """A keyed row's key as text: a JSON array of its key columns' + canonical values, so (``x|y``, ``z``) and (``x``, ``y|z``) differ.""" + + return json.dumps([canonical(row.get(c)) for c in primary_key]) + + def _row_key( d: _Def, row: Mapping[str, Any], seen: dict[str, int], skip: Sequence[str] = () ) -> str: if d.primary_key: - return "|".join(canonical(row.get(c)) for c in d.primary_key) + return key_text(d.primary_key, row) # No key: the row's contents, without the columns a rule lets move (a # counter row stays the same row when its counter moves). ignored = set(skip) @@ -851,7 +1501,9 @@ def take_snapshot( ignore: Table -> columns left out of each row's contents MAC. ref_columns: Columns attribution and side records read (a record key, a link field): their cell MACs are kept in every table. - schemas: Schemas the read must see (MySQL); refused if one isn't. + schemas: Schemas the read must see; refused if one isn't. Every + visible schema is read either way. On PostgreSQL a schema is left + out only by a ``.*`` entry in ``modes`` with ``skip``. optional_modes: Like ``modes``, for tables an app profile names that this database may not have (a profile covers several versions). ``ignore`` may name ``*`` for every table. @@ -903,18 +1555,54 @@ def _snapshot( dialect = s.dialect counters: dict[str, int] = {} visible = True + sequences: dict[str, _PgSequence] = {} + scope: list[str] = [] + default_schema = "" if dialect == "mysql": # Before the snapshot starts: a row committed after this read is in # the snapshot, so the counter never runs ahead of what it shows. counters = _mysql_counters(s) visible = _sees_triggers(s) + elif dialect == "postgresql": + skipped_schemas = sorted( + t[:-2] for t, m in modes.items() if t.endswith(".*") and m == "skip" + ) + scope, default_schema = _pg_scope(s, schemas, skipped_schemas) + sequences = _pg_sequences(s, scope, default_schema) # before, as above s.begin() - snap = Snapshot( - dialect=dialect, - clock=_clock_text(s.rows("SELECT CURRENT_TIMESTAMP")[0][0]), - code_visible=visible, - ) - defs, snap.objects = _definitions(s) + if dialect == "postgresql": + # The first statement takes the snapshot; its start is the clock. + clock, isolation, read_only = s.rows( + "SELECT statement_timestamp()::text, " + "current_setting('transaction_isolation'), " + "current_setting('transaction_read_only')" + )[0] + if (isolation, read_only) != ("repeatable read", "on"): + raise SnapshotError( + f"the read ran as {isolation}, read_only={read_only}, not one " + "read-only repeatable-read transaction" + ) + else: + clock = s.rows("SELECT CURRENT_TIMESTAMP")[0][0] + snap = Snapshot(dialect=dialect, clock=_clock_text(clock), code_visible=visible) + if dialect == "postgresql": + defs, snap.objects = _pg_definitions(s, scope, default_schema) + counters, others = _pg_counters(defs, sequences) + for q in others: + if q.name in defs: + raise SnapshotError(f"sequence {q.name!r} has the name of a table") + defs[q.name] = _pg_sequence_def(q) + if LARGE_OBJECTS in defs: + # A table of that name would be replaced, and so never read. + raise SnapshotError( + f"a table is named {LARGE_OBJECTS!r}, which the snapshot uses" + ) + defs[LARGE_OBJECTS] = _pg_large_objects() + snap.schemas = list(scope) + snap.schemas_skipped = skipped_schemas + modes = {t: m for t, m in modes.items() if not t.endswith(".*")} + else: + defs, snap.objects = _definitions(s) if dialect == "mysql": default = str(s.rows("SELECT DATABASE()")[0][0]) seen = {default} | {t.split(".", 1)[0] for t in defs if "." in t} @@ -925,7 +1613,7 @@ def _snapshot( ) snap.schemas = sorted(seen) else: - if any( + if dialect == "sqlite" and any( n == "sqlite_sequence" for (n,) in s.rows("SELECT name FROM sqlite_master WHERE type = 'table'") ): @@ -945,17 +1633,30 @@ def _snapshot( for t in sorted(t for t in defs if modes.get(t) == "skip"): d = defs[t] snap.tables[t] = Table("skip", d.schema, d.columns, d.primary_key, None, {}) + refused = sorted(f"{t} ({defs[t].unreadable})" for t in names if defs[t].unreadable) + if refused: + raise SnapshotError( + "can't read exactly: " + + "; ".join(refused) + + ". Skip a table only by allow-listing it with mode skip and a reason" + ) if not names: return snap - counts = { - str(t): int(n) - for t, n in s.rows( - " UNION ALL ".join( - f"SELECT {_literal(t)}, COUNT(*) FROM {_quote(dialect, t)}" - for t in names + + def rel(t: str) -> str: + return defs[t].relation or _quote(dialect, t) + + stored = [t for t in names if defs[t].fixed is None] + counts = {t: len(defs[t].fixed or []) for t in names if defs[t].fixed is not None} + if stored: + counts.update( + (str(t), int(n)) + for t, n in s.rows( + " UNION ALL ".join( + f"SELECT {_literal(t)}, COUNT(*) FROM {rel(t)}" for t in stored + ) ) ) - } numbered = [t for t in names if defs[t].auto_column] maxes = ( { @@ -963,7 +1664,7 @@ def _snapshot( for t, m in s.rows( " UNION ALL ".join( f"SELECT {_literal(t)}, MAX({_quote(dialect, defs[t].auto_column or '')}) " - f"FROM {_quote(dialect, t)}" + f"FROM {rel(t)}" for t in numbered ) ) @@ -1002,7 +1703,15 @@ def _snapshot( table.cell_columns = list(d.columns) skip = [*ignore.get(t, ()), *ignore.get("*", ())] seen_keys: dict[str, int] = {} - for row in _read_rows(s, t, d) if table.count else []: + if d.fixed is not None: + read = d.fixed + elif not table.count: + read = [] + elif dialect == "postgresql": + read = _read_rows_pg(s, d) + else: + read = _read_rows(s, t, d) + for row in read: text = _row_key(d, row, seen_keys, skip) k = row_key_hash(key, text, t) if links is not None and d.primary_key and mode == "compare": @@ -1055,13 +1764,18 @@ def unread_reasons( ): reasons.append("existing rows changed or removed") if new.counter is not None: - # sqlite_sequence holds the last number used; AUTO_INCREMENT the next. + # sqlite_sequence and a PostgreSQL sequence hold the last number + # used; AUTO_INCREMENT the next. step = 1 if dialect == "mysql" else 0 expected = max( old.counter or 0, 0 if new.max_auto is None else new.max_auto + step ) if new.counter > expected: reasons.append("a row was written and removed (counter moved)") + if old.counter is not None and new.counter < old.counter: + # A PostgreSQL sequence can be set back (setval); later inserts + # would then reuse numbers. + reasons.append("the counter went back") return reasons diff --git a/packages/flow/openadapt_flow/runtime/effects/frame.py b/packages/flow/openadapt_flow/runtime/effects/frame.py index cc386f1f46..aa41a4e0c4 100644 --- a/packages/flow/openadapt_flow/runtime/effects/frame.py +++ b/packages/flow/openadapt_flow/runtime/effects/frame.py @@ -256,7 +256,8 @@ class AllowedTable(BaseModel): table: str #: ``append_only`` (the default): rows may be added, existing rows must #: not change, and the table needs a generated key. ``skip``: not read at - #: all, which needs a ``reason``. + #: all, which needs a ``reason``. On PostgreSQL ``.*`` names a + #: whole schema, which may only be skipped. mode: Literal["append_only", "skip"] = "append_only" reason: Optional[str] = None @@ -264,6 +265,10 @@ class AllowedTable(BaseModel): def _skip_needs_reason(self) -> "AllowedTable": if self.mode == "skip" and not (self.reason or "").strip(): raise ValueError(f"skipping table {self.table!r} needs a reason") + if self.table.endswith(".*") and self.mode != "skip": + raise ValueError( + f"schema {self.table!r} can only be skipped, with a reason" + ) return self @@ -286,7 +291,7 @@ class FrameDatabase(BaseModel): model_config = ConfigDict(extra="forbid") - dialect: Literal["sqlite", "mysql"] + dialect: Literal["sqlite", "mysql", "postgresql"] #: The read-only connection: ``sqlite_database``, or ``sql_driver`` with #: ``sql_connect_args`` and ``sql_password_env``, as in a ``kind: sql`` #: verifier (no query: every table is read). @@ -307,8 +312,11 @@ class FrameDatabase(BaseModel): #: Over this many rows a table keeps, per row, its key and contents MACs #: and cell MACs only for the columns attribution and side records read. max_rows: int = Field(default=2000, ge=1) - #: Schemas the read account must see (MySQL); the snapshot refuses when - #: one isn't visible, so a schema the app writes can't be left out. + #: Schemas the read account must see; the snapshot refuses when one isn't + #: visible, so a schema the app writes can't be left out. Every other + #: visible schema is read too. On PostgreSQL a schema the account can't + #: see is refused unless ``allow_tables`` skips ``.*`` with a + #: reason. schemas: list[str] = Field(default_factory=list) #: An app profile (``openadapt_flow.app_profiles``): the app's own #: bookkeeping, versioned and shared with rollback and the RL grader. Its @@ -448,6 +456,12 @@ class FrameSpec(BaseModel): #: Fields of a pre-existing target record the save may change. Unset: #: the fields the effects name. target_fields: Optional[list[str]] = None + #: Surface -> the fields a declared row of that surface may change + #: (a database table counts as the surface ``database.views`` maps it + #: to). Set: it replaces ``target_fields`` for every declared row, and a + #: surface it doesn't list may change nothing, so a row touched beside + #: the write can't take the target's fields. + target_fields_by_surface: Optional[dict[str, list[str]]] = None attribution: Optional[FrameAttribution] = None #: Rows the save adds beside each new declared record (``FrameSideRecord``). side_records: list[FrameSideRecord] = Field(default_factory=list) @@ -465,6 +479,8 @@ def _compact(self, handler: Any) -> dict[str, Any]: data: dict[str, Any] = handler(self) if data.get("versions") is None: data.pop("versions", None) + if data.get("target_fields_by_surface") is None: + data.pop("target_fields_by_surface", None) return data @model_validator(mode="after") @@ -831,6 +847,13 @@ def judge_frame( surface_ignore |= set(attribution.server_set) written = declared_fields(effects) may_change = set(spec.target_fields) if spec.target_fields is not None else written + if spec.target_fields_by_surface is not None: + # Every field any declared row may change, for the attribution check. + may_change = ( + set().union(*spec.target_fields_by_surface.values()) + if (spec.target_fields_by_surface) + else set() + ) if attribution is not None and attribution.kind == "record_field": # A field this run writes (or any run parameter names) is not evidence # of who changed a record: refuse the attribution, fail closed. @@ -1315,6 +1338,15 @@ def attribute( concurrent: list[FrameChange] = [] added_by_others: set[tuple[str, str]] = set() views_of = dict(database.views) if database is not None else {} + + def may_change_for(name: str) -> set[str]: + by_surface = spec.target_fields_by_surface + if by_surface is None: + return may_change + if name in by_surface: + return set(by_surface[name]) + return set(by_surface.get(views_of.get(name, ""), [])) + declared_keys: set[str] = set() #: What an audit entry names for each compared record (``ref_of``). refs: dict[tuple[str, str], tuple[Any, Optional[str]]] = {} @@ -1534,12 +1566,13 @@ def partner( if after is None: continue # a removed target: the delta check's case fields = changed(name, before, after) # type: ignore[arg-type] - stray = [f for f in fields if f not in may_change] + row_may = may_change_for(name) + stray = [f for f in fields if f not in row_may] if other_on_target(name, key, before, after): ref, field = ref_of(name, key, before, after) if ( fields - and not (set(fields) & may_change) + and not (set(fields) & row_may) and attribution is not None and attribution.write_events and attribution.table_field diff --git a/packages/flow/openadapt_flow/runtime/effects/late_write.py b/packages/flow/openadapt_flow/runtime/effects/late_write.py index 8b749879dc..080e4e982f 100644 --- a/packages/flow/openadapt_flow/runtime/effects/late_write.py +++ b/packages/flow/openadapt_flow/runtime/effects/late_write.py @@ -261,7 +261,7 @@ def row( from openadapt_flow.runtime.effects.db_snapshot import ( KEY, ROW, - canonical, + key_text, row_hash, row_key_hash, ) @@ -272,7 +272,7 @@ def row( old = self.baseline.tables.get(table) if after is not None: row = {c: v for c, v in after.items() if c not in (KEY, ROW)} - text = "|".join(canonical(row.get(c)) for c in now.primary_key) + text = key_text(now.primary_key, row) k = row_key_hash(self.key, text, table) stored = now.rows.get(k) if stored is None or stored[0] != row_hash(self.key, table, row, skip): diff --git a/packages/flow/openadapt_flow/runtime/effects/versions.py b/packages/flow/openadapt_flow/runtime/effects/versions.py index 01e5a91bb4..1658ee11f4 100644 --- a/packages/flow/openadapt_flow/runtime/effects/versions.py +++ b/packages/flow/openadapt_flow/runtime/effects/versions.py @@ -37,7 +37,7 @@ from dataclasses import dataclass, field from typing import Any, Literal, Optional -from pydantic import BaseModel, ConfigDict, Field, model_validator +from pydantic import BaseModel, ConfigDict, Field, model_serializer, model_validator from openadapt_flow.runtime.effects.effect import ValueExpr @@ -76,6 +76,9 @@ class FrameVersions(BaseModel): #: The target record's field (in the contract's read set) that equals #: ``record_field``. Unset: ``frame.identity_field``, else ``id``. target_field: Optional[str] = None + #: Only records matched on this surface are targets (the main record's + #: surface). Unset: every record a confirmed write effect matched. + target_surface: Optional[str] = None #: The version row's version id (``meta.versionId``, ``res_ver``, an #: audit entry's id). version_field: str = Field(min_length=1) @@ -99,6 +102,15 @@ def _coerce_self(cls, data: Any) -> Any: ] return data + @model_serializer(mode="wrap") + def _compact(self, handler: Any) -> dict[str, Any]: + # Unset, ``target_surface`` is left out, so a contract that doesn't + # use it keeps its digests. + data: dict[str, Any] = handler(self) + if data.get("target_surface") is None: + data.pop("target_surface", None) + return data + @model_validator(mode="after") def _shape(self) -> "FrameVersions": if bool(self.author_field) != bool(self.self_actors): diff --git a/packages/flow/openadapt_flow/verifier.py b/packages/flow/openadapt_flow/verifier.py index aaed95f6c3..7547ea85a7 100644 --- a/packages/flow/openadapt_flow/verifier.py +++ b/packages/flow/openadapt_flow/verifier.py @@ -2401,8 +2401,8 @@ def capture_database( from openadapt_flow.verify_run import _db_connect - if dialect not in ("sqlite", "mysql"): - raise VerifierError(f"dialect {dialect!r} is not sqlite or mysql") + if dialect not in ("sqlite", "mysql", "postgresql"): + raise VerifierError(f"dialect {dialect!r} is not sqlite, mysql or postgresql") connect = read if callable(read) else _db_connect(read) extra: dict[str, str] = {} if profile is not None: diff --git a/packages/flow/openadapt_flow/verify_run.py b/packages/flow/openadapt_flow/verify_run.py index 1b0aaffc47..2d4ad30797 100644 --- a/packages/flow/openadapt_flow/verify_run.py +++ b/packages/flow/openadapt_flow/verify_run.py @@ -496,10 +496,29 @@ def _judge_versions( # change, never by the target field's value after the write: a target # field the write changed would make an old record look created. identity = cfg.frame.identity_field or "id" # type: ignore[union-attr] - old: dict[str, dict[str, Any]] = {} - for r in before.records: - if r.get(identity) not in (None, ""): - old.setdefault(str(r.get(identity)), dict(r)) + from openadapt_flow.runtime.effects.surfaces import surface_of + + def index(rows: Sequence[Mapping[str, Any]]) -> dict[str, dict[str, Any]]: + out: dict[str, dict[str, Any]] = {} + for r in rows: + if r.get(identity) not in (None, ""): + out.setdefault(str(r.get(identity)), dict(r)) + return out + + def before_rows(effect: Effect) -> Optional[dict[str, dict[str, Any]]]: + """The records read before the write where this effect looks: its + surface's own read, else the verifier's records. None: no read says + which records were there (then no target is taken as new).""" + + surface = surface_of(effect) + if surface: + sub = (before.detail or {}).get(surface) + if not isinstance(sub, Mapping) or not sub.get("reachable"): + return None + rows = sub.get("records") + return index(rows) if isinstance(rows, list) else None + return index(before.records) if before.reachable else None + ignored = set(cfg.frame.ignore_fields) # type: ignore[union-attr] targets: list[tuple[Optional[str], bool, bool]] = [] for effect, verdict in zip(effects, verdicts): @@ -509,13 +528,20 @@ def _judge_versions( continue # a guard: "nothing here" if not verdict.confirmed: continue + if ( + spec.target_surface is not None + and surface_of(effect) != spec.target_surface + ): + continue # a row beside the target, not the target + old = before_rows(effect) for record in verdict.matched_records: value = record.get(field_name) text = None if value in (None, "") else str(value) key = record.get(identity) - was = None if key in (None, "") else old.get(str(key)) - # No identity to look up: never assume the record is new. - existed = was is not None or key in (None, "") + was = None if key in (None, "") or old is None else old.get(str(key)) + # A record is new only when the read before the write shows it + # wasn't there. No identity, or no read to look in: never new. + existed = was is not None or key in (None, "") or old is None # Changed in the window: a field read before differs now (a # sealed baseline marks a changed field, so it differs too). changed = existed and ( @@ -906,6 +932,11 @@ def _database_frame( 1 for t in new.tables.values() if t.non_transactional ), "schemas": list(new.schemas), + **( + {"schemas_skipped": list(new.schemas_skipped)} + if new.schemas_skipped + else {} + ), "allow_list_sha256": database.allow_list_sha256(), "settle_attempts": used, "settle_attempts_allowed": attempts, diff --git a/packages/flow/tests/test_db_snapshot_postgres.py b/packages/flow/tests/test_db_snapshot_postgres.py new file mode 100644 index 0000000000..c01c3736d7 --- /dev/null +++ b/packages/flow/tests/test_db_snapshot_postgres.py @@ -0,0 +1,743 @@ +"""The whole-database snapshot on PostgreSQL: every column type read exactly. + +The first-contact study ran Redmine on PostgreSQL, where ``frame.database`` +had no dialect, so a write to a table the contract doesn't read, or a write +after the verdict, passed. These tests hold the PostgreSQL dialect to the +MySQL one's bar: one read-only repeatable-read transaction, every column +type read by a declared rule, a one-step change to any column shown as that +column and nothing else, and a clear refusal for what can't be read exactly. + +The rule tests need no database. The live tests run only with +``OPENADAPT_SNAPSHOT_POSTGRES`` set to a libpq DSN for an account that may +create and drop a scratch database and a role, for example:: + + OPENADAPT_SNAPSHOT_POSTGRES='host=127.0.0.1 port=5432 user=postgres + password=... dbname=postgres' \\ + python -m pytest tests/test_db_snapshot_postgres.py -v +""" + +from __future__ import annotations + +import os +import secrets +from decimal import Decimal +from pathlib import Path +from typing import Any, Iterator + +import pytest + +from openadapt_flow.runtime.effects import db_snapshot +from openadapt_flow.runtime.effects.db_snapshot import ( + CLOCK_DEFAULT, + SnapshotError, + StoredNumber, + canonical, + diff, + is_default, + take_snapshot, +) + +# -- rules: no database --------------------------------------------------------- + + +def _types(**extra: db_snapshot._PgType) -> dict[int, db_snapshot._PgType]: + T = db_snapshot._PgType + base = { + 1: T("int4", "pg_catalog", "b", "N", 0, 0), + 2: T("numeric", "pg_catalog", "b", "N", 0, 0), + 3: T("money", "pg_catalog", "b", "N", 0, 0), + 4: T("_int4", "pg_catalog", "b", "A", 1, 0), + 5: T("_money", "pg_catalog", "b", "A", 3, 0), + 6: T("int4range", "pg_catalog", "r", "R", 0, 0, subtype=1), + 7: T("mood", "public", "e", "E", 0, 0), + 8: T("posint", "public", "d", "N", 0, 1), + 9: T("addr", "public", "c", "C", 0, 0), + 10: T("citext", "public", "b", "S", 0, 0), + 11: T("regclass", "pg_catalog", "b", "N", 0, 0), + 12: T("price", "public", "d", "N", 0, 2, typmod=655366), + 13: T("free", "public", "d", "N", 0, 2), + 14: T("bytea", "pg_catalog", "b", "U", 0, 0), + } + return {**base, **extra} + + +@pytest.mark.parametrize( + "oid, typmod, rule", + [ + (1, -1, "int"), + (2, 655366, "numeric"), + (2, -1, "numeric_free"), + (3, -1, "money"), + (4, -1, "text"), + (6, -1, "text"), + (7, -1, "text"), + (8, -1, "int"), + (12, -1, "numeric"), + (13, -1, "numeric_free"), + (14, -1, "bytea"), + ], +) +def test_each_type_has_a_declared_rule(oid: int, typmod: int, rule: str) -> None: + assert db_snapshot._pg_rule(_types(), oid, typmod) == rule + + +@pytest.mark.parametrize( + "oid, message", + [ + (5, "money"), + (9, "public.addr"), + (10, "public.citext"), + (11, "pg_catalog.regclass"), + (99, "not in the catalog"), + ], +) +def test_a_type_without_a_rule_is_refused(oid: int, message: str) -> None: + with pytest.raises(ValueError, match=message): + db_snapshot._pg_rule(_types(), oid) + + +@pytest.mark.parametrize( + "expr, stored", + [ + (None, None), + ("NULL::character varying", None), + ("'open'::character varying", "open"), + ("'O''Brien'::text", "O'Brien"), + ("''::character varying", ""), + ("0", "0"), + ("(-1)", "-1"), + ("'1.50'::numeric", "1.50"), + ("false", "false"), + ("now()", CLOCK_DEFAULT), + ("CURRENT_TIMESTAMP", CLOCK_DEFAULT), + ("('now'::text)::date", CLOCK_DEFAULT), + ("nextval('issues_id_seq'::regclass)", "nextval('issues_id_seq'::regclass)"), + ("gen_random_uuid()", "gen_random_uuid()"), + ], +) +def test_a_default_is_read_only_from_a_literal(expr: Any, stored: Any) -> None: + assert db_snapshot._default_text(expr, "postgresql") == stored + + +def test_a_numeric_without_scale_keeps_it_and_equals_its_value() -> None: + a, b = ( + db_snapshot._pg_value("numeric_free", "1.50"), + db_snapshot._pg_value("numeric_free", "1.5"), + ) + assert isinstance(a, StoredNumber) and a == b == Decimal("1.5") + assert str(a) == "1.5" + assert canonical(a) != canonical(b) + assert is_default(a, "1.5") and is_default( + db_snapshot._pg_value("numeric", "0.00"), "0" + ) + + +@pytest.mark.parametrize( + "rule, text, value", + [ + ("int", "-9223372036854775808", -9223372036854775808), + ("numeric", "12.50", Decimal("12.5")), + ("numeric", "NaN", Decimal("NaN")), + ("float4", "0.1", Decimal("0.1")), + ("float4", "0.100000001", Decimal("0.1")), + ("float8", "0.30000000000000004", Decimal("0.30000000000000004")), + ("float8", "-Infinity", Decimal("-Infinity")), + ("money", "12.30", Decimal("12.3")), + ("bool", "true", True), + ("bytea", "00ff", b"\x00\xff"), + ("text", "P1D", "P1D"), + ], +) +def test_a_value_is_read_by_its_rule(rule: str, text: str, value: Any) -> None: + got = db_snapshot._pg_value(rule, text) + assert canonical(got) == canonical(value) + + +def test_a_boolean_that_isnt_true_or_false_is_refused() -> None: + with pytest.raises(SnapshotError): + db_snapshot._pg_value("bool", "t") + + +# -- a real PostgreSQL ----------------------------------------------------------- + +_DSN = os.environ.get("OPENADAPT_SNAPSHOT_POSTGRES") +live = pytest.mark.skipif( + not _DSN, reason="set OPENADAPT_SNAPSHOT_POSTGRES to a libpq DSN" +) +KEY = b"k" * 32 + +#: A reader whose session prints every type differently from the pinned one: +#: a zone with daylight saving, floats with 6 digits, bytes escaped, SQL +#: dates, Postgres intervals. A pinned read is the same through it. +HOSTILE = ( + "-c TimeZone=America/New_York -c extra_float_digits=0 -c bytea_output=escape " + "-c DateStyle=SQL,DMY -c IntervalStyle=postgres" +) + +#: One column per type, each with a value, and a second value one smallest +#: step away (or differing only where a lossy read would hide it). +COLUMNS: list[tuple[str, str, str, str]] = [ + ("c_int2", "smallint", "-32768", "-32767"), + ("c_int8", "bigint", "9223372036854775807", "9223372036854775806"), + ( + "c_numeric", + "numeric(40,20)", + "12345678901234567890.01234567890123456789", + "12345678901234567890.01234567890123456788", + ), + ("c_numeric_scale", "numeric", "1.5", "1.50"), + ("c_numeric_nan", "numeric", "0", "'NaN'"), + ("c_real", "real", "0.1", "0.10000001"), + ("c_double", "double precision", "1.0000000000000002e300", "1e300"), + ("c_double_zero", "double precision", "0", "'-0'"), + ("c_money", "money", "12.34", "12.35"), + ("c_bool", "boolean", "true", "false"), + ("c_bytea", "bytea", "'\\x00ff'", "'\\x00fe'"), + ("c_char", "char(4)", "'ab'", "'ac'"), + ("c_varchar", "varchar(20)", "'a '", "'a'"), + ("c_text", "text", "'x'", "'X'"), + ( + "c_uuid", + "uuid", + "'00000000-0000-0000-0000-000000000001'", + "'00000000-0000-0000-0000-000000000002'", + ), + ("c_inet", "inet", "'10.0.0.1'", "'10.0.0.1/24'"), + ("c_cidr", "cidr", "'10.0.0.0/24'", "'10.0.0.0/25'"), + ("c_macaddr", "macaddr", "'08:00:2b:01:02:03'", "'08:00:2b:01:02:04'"), + ("c_date", "date", "'2026-01-02'", "'2026-01-03'"), + ("c_time", "time(6)", "'03:04:05.000001'", "'03:04:05'"), + ("c_timetz", "timetz", "'03:04:05+01'", "'03:04:05+02'"), + ( + "c_timestamp", + "timestamp(6)", + "'2026-01-02 03:04:05.6'", + "'2026-01-02 03:04:05.600001'", + ), + ( + "c_timestamptz", + "timestamptz", + "'2026-11-01 05:30:00+00'", + "'2026-11-01 06:30:00+00'", + ), + ("c_interval", "interval", "'1 day'", "'24 hours'"), + ("c_bit", "bit(5)", "B'10101'", "B'10100'"), + ("c_varbit", "varbit(8)", "B'1'", "B'10'"), + ("c_json", "json", "'{\"b\": 1.10}'", "'{\"b\": 1.1}'"), + ("c_jsonb", "jsonb", "'{\"b\": 1.10}'", "'{\"b\": 1.1}'"), + ("c_xml", "xml", "'1'", "'2'"), + ("c_tsvector", "tsvector", "'a:1 b:2'", "'a:1 b:3'"), + ("c_tsquery", "tsquery", "'a & b'", "'a | b'"), + ("c_point", "point", "'(1,2)'", "'(1,2.0000000000000004)'"), + ("c_line", "line", "'{1,2,3}'", "'{1,2,4}'"), + ("c_lseg", "lseg", "'[(0,0),(1,1)]'", "'[(0,0),(1,2)]'"), + ("c_box", "box", "'(1,1),(0,0)'", "'(2,1),(0,0)'"), + ("c_path", "path", "'[(0,0),(1,1)]'", "'((0,0),(1,1))'"), + ("c_polygon", "polygon", "'((0,0),(1,1),(1,0))'", "'((0,0),(1,1),(2,0))'"), + ("c_circle", "circle", "'<(0,0),1>'", "'<(0,0),2>'"), + ("c_int_array", "int[]", "'{1,2}'", "'{2,1}'"), + ("c_float_array", "float8[]", "'{0.1}'", "'{0.30000000000000004}'"), + ("c_real_array", "real[]", "'{0.1}'", "'{0.10000001}'"), + ("c_text_array", "text[]", "'{\"a b\"}'", "'{a,b}'"), + ("c_bytea_array", "bytea[]", "ARRAY['\\x00'::bytea]", "ARRAY['\\x0000'::bytea]"), + ("c_int4range", "int4range", "'[1,5)'", "'[1,6)'"), + ( + "c_tstzrange", + "tstzrange", + "'[2026-01-01 00:00+00,2026-01-02 00:00+00)'", + "'[2026-01-01 00:00+00,2026-01-02 00:00:00.000001+00)'", + ), + ("c_multirange", "int4multirange", "'{[1,3)}'", "'{[1,3),[4,5)}'"), + ("c_enum", "mood", "'x'", "'y'"), + ("c_domain", "posint", "1", "2"), +] + + +class Pg: + def __init__(self, dsn: str) -> None: + import psycopg2 + + self.psycopg2 = psycopg2 + self.dsn = dsn + self.name = "openadapt_snapshot_pg_" + secrets.token_hex(4) + self.reader = "openadapt_snapshot_ro_" + secrets.token_hex(4) + self.password = secrets.token_hex(16) + + def admin(self, dbname: str | None = None) -> Any: + extra = {} if dbname is None else {"dbname": dbname} + con = self.psycopg2.connect(self.dsn, **extra) + con.autocommit = True + return con + + def sql(self, *statements: str) -> None: + con = self.admin(self.name) + with con.cursor() as cur: + for statement in statements: + cur.execute(statement) + con.close() + + def connect(self, options: str = "", reader: bool = False) -> Any: + args: dict[str, Any] = {"dbname": self.name} + if options: + args["options"] = options + if reader: + args.update(user=self.reader, password=self.password) + return lambda: self.psycopg2.connect(self.dsn, **args) + + def snap(self, before: Any = None, options: str = "", **kw: Any) -> Any: + return take_snapshot( + self.connect(options, kw.pop("reader", False)), + "postgresql", + key=KEY, + before=before, + **kw, + ) + + +@pytest.fixture +def pg() -> Iterator[Pg]: + db = Pg(_DSN or "") + con = db.admin() + with con.cursor() as cur: + cur.execute(f"CREATE DATABASE {db.name}") + cur.execute(f"CREATE ROLE {db.reader} LOGIN PASSWORD '{db.password}'") + cols = ", ".join(f"{c} {t}" for c, t, _, _ in COLUMNS) + names = ", ".join(c for c, *_ in COLUMNS) + values = ", ".join(v for _, _, v, _ in COLUMNS) + db.sql( + "CREATE TYPE mood AS ENUM ('x', 'y')", + "CREATE DOMAIN posint AS integer CHECK (VALUE > 0)", + f"CREATE TABLE t (id serial PRIMARY KEY, {cols})", + f"INSERT INTO t ({names}) VALUES ({values})", + f"INSERT INTO t ({names}) VALUES ({values})", + "INSERT INTO t (id) VALUES (DEFAULT)", + "CREATE TABLE pairs (a int, b text)", + "INSERT INTO pairs VALUES (1, 'x'), (1, 'x'), (2, 'y')", + f"GRANT USAGE ON SCHEMA public TO {db.reader}", + f"GRANT SELECT ON ALL TABLES IN SCHEMA public TO {db.reader}", + f"GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO {db.reader}", + ) + try: + yield db + finally: + with con.cursor() as cur: + cur.execute(f"DROP DATABASE {db.name} WITH (FORCE)") + cur.execute(f"DROP ROLE {db.reader}") + con.close() + + +@live +def test_every_type_is_read_and_a_reread_is_equal(pg: Pg) -> None: + a = pg.snap() + assert a.tables["t"].count == 3 and a.schemas == ["public"] + assert a.tables["t"].counter == 3 and a.tables["t"].generated_key == "id" + assert diff(a, pg.snap(before=a), key=KEY) == {} + + +@live +def test_a_reader_session_set_otherwise_reads_the_same(pg: Pg) -> None: + a = pg.snap() + assert diff(a, pg.snap(before=a, options=HOSTILE), key=KEY) == {} + + +@live +@pytest.mark.parametrize( + "column, _type, _old, new", COLUMNS, ids=[c[0] for c in COLUMNS] +) +def test_a_one_step_change_to_any_column_shows( + pg: Pg, column: str, _type: str, _old: str, new: str +) -> None: + a = pg.snap(options=HOSTILE) + pg.sql(f"UPDATE t SET {column} = {new} WHERE id = 1") + b = pg.snap(before=a, options=HOSTILE) + moved = diff(a, b, key=KEY) + assert list(moved) == ["t"] + assert (moved["t"].changed, moved["t"].changed_columns) == (1, [column]) + assert b.tables["t"].values, "the changed row's plain values are kept" + + +@live +@pytest.mark.parametrize( + "column, _type, old, _new", COLUMNS, ids=[c[0] for c in COLUMNS] +) +def test_a_null_and_a_value_differ( + pg: Pg, column: str, _type: str, old: str, _new: str +) -> None: + a = pg.snap() + pg.sql(f"UPDATE t SET {column} = {old} WHERE id = 3") + moved = diff(a, pg.snap(before=a), key=KEY) + assert (moved["t"].changed, moved["t"].changed_columns) == (1, [column]) + + +@live +def test_plain_values_read_as_people_write_them(pg: Pg) -> None: + a = pg.snap() + pg.sql("UPDATE t SET c_int2 = 7 WHERE id = 1") + row = next(iter(pg.snap(before=a).tables["t"].values.values())) + assert str(row["c_real"]) == "0.1" and str(row["c_numeric_scale"]) == "1.5" + assert str(row["c_money"]) == "12.34" and row["c_bool"] is True + assert row["c_bytea"] == b"\x00\xff" and row["c_interval"] == "P1D" + assert row["c_timestamptz"] == "2026-11-01 05:30:00+00" + + +@live +def test_a_table_without_a_key_counts_duplicate_rows(pg: Pg) -> None: + a = pg.snap() + assert a.tables["pairs"].count == 3 and not a.tables["pairs"].primary_key + pg.sql("INSERT INTO pairs VALUES (1, 'x')") + b = pg.snap(before=a) + assert ( + diff(a, b, key=KEY)["pairs"].added, + diff(a, b, key=KEY)["pairs"].removed, + ) == (1, 0) + pg.sql( + "DELETE FROM pairs WHERE ctid IN (SELECT ctid FROM pairs WHERE a = 1 LIMIT 2)" + ) + moved = diff(a, pg.snap(before=a), key=KEY)["pairs"] + assert (moved.added, moved.removed) == (0, 1) + + +@live +def test_a_row_written_and_removed_moves_the_counter(pg: Pg) -> None: + a = pg.snap() + pg.sql("INSERT INTO t (id) VALUES (DEFAULT)", "DELETE FROM t WHERE id = 4") + moved = diff(a, pg.snap(before=a), key=KEY) + assert moved["t"].unread == ["a row was written and removed (counter moved)"] + + +@live +def test_a_sequence_no_table_owns_is_compared_unless_skipped(pg: Pg) -> None: + pg.sql("CREATE SEQUENCE invoice_no", f"GRANT SELECT ON invoice_no TO {pg.reader}") + a = pg.snap(reader=True) + pg.sql("SELECT nextval('invoice_no')") + moved = diff(a, pg.snap(before=a, reader=True), key=KEY) + assert list(moved) == ["invoice_no"] and moved["invoice_no"].changed == 1 + skip = {"invoice_no": "skip"} + a = pg.snap(reader=True, modes=skip) + pg.sql("SELECT nextval('invoice_no')") + assert diff(a, pg.snap(before=a, reader=True, modes=skip), key=KEY) == {} + + +@live +def test_a_column_without_a_rule_is_refused_unless_skipped(pg: Pg) -> None: + pg.sql( + "CREATE TYPE addr AS (street text, n int)", + "CREATE TABLE places (id int PRIMARY KEY, a addr, m money[])", + ) + with pytest.raises( + SnapshotError, match=r"places \(column a: no exact reading rule" + ): + pg.snap() + a = pg.snap(modes={"places": "skip"}) + assert a.tables["places"].mode == "skip" + + +@live +def test_what_the_reader_cant_see_is_refused(pg: Pg) -> None: + pg.sql("CREATE TABLE secret (id int PRIMARY KEY)") + with pytest.raises( + SnapshotError, + match="secret .no SELECT|secret \\(the read account has no SELECT", + ): + pg.snap(reader=True) + pg.sql( + f"GRANT SELECT ON secret TO {pg.reader}", + "CREATE SCHEMA hidden", + "CREATE TABLE hidden.h (id int PRIMARY KEY)", + ) + with pytest.raises( + SnapshotError, match="can't see schema.s. hidden.*'.\\*'" + ): + pg.snap(reader=True) + # Naming schemas doesn't narrow the read (review #1673, B1): a hidden + # schema is still refused. + with pytest.raises(SnapshotError, match="can't see schema.s. hidden"): + pg.snap(reader=True, schemas=["public"]) + with pytest.raises(SnapshotError, match="can't see schema.s.: hidden"): + pg.snap(reader=True, schemas=["public", "hidden"]) + # Only the allow-list leaves a schema out. + a = pg.snap(reader=True, modes={"hidden.*": "skip"}) + assert a.schemas == ["public"] and a.schemas_skipped == ["hidden"] + assert "hidden.h" not in a.tables + with pytest.raises(SnapshotError, match="allow-listed schemas not in the database"): + pg.snap(reader=True, modes={"nosuch.*": "skip"}) + + +@live +def test_named_schemas_still_read_every_visible_schema(pg: Pg) -> None: + # Review #1673, B1: with schemas=["public"], a stray row in another + # schema and a rewritten trigger function in a third both show. + pg.sql( + "CREATE SCHEMA other", + "CREATE TABLE other.note (id serial PRIMARY KEY, body text)", + "CREATE SCHEMA util", + "CREATE FUNCTION util.f() RETURNS trigger LANGUAGE plpgsql AS 'BEGIN RETURN NEW; END'", + "CREATE TRIGGER tr BEFORE INSERT ON pairs FOR EACH ROW EXECUTE FUNCTION util.f()", + ) + a = pg.snap(schemas=["public"]) + assert a.schemas == ["other", "public", "util"] + pg.sql("INSERT INTO other.note (body) VALUES ('stray')") + assert ( + diff(a, pg.snap(before=a, schemas=["public"]), key=KEY)["other.note"].added == 1 + ) + b = pg.snap(schemas=["public"]) + pg.sql( + "CREATE OR REPLACE FUNCTION util.f() RETURNS trigger LANGUAGE plpgsql " + "AS 'BEGIN NEW.b := ''x''; RETURN NEW; END'" + ) + assert "(database code)" in diff(b, pg.snap(before=b, schemas=["public"]), key=KEY) + + +def test_sqlite_composite_keys_that_join_alike_stay_apart(tmp_path: Path) -> None: + # Review #1673, S1, on every dialect: the key is a JSON array. + import sqlite3 + + path = tmp_path / "kv.sqlite" + con = sqlite3.connect(path) + con.executescript( + "CREATE TABLE kv (a TEXT, b TEXT, PRIMARY KEY (a, b));" + "INSERT INTO kv VALUES ('x|y', 'z'), ('x', 'y|z');" + ) + con.commit() + + def connect() -> Any: + return sqlite3.connect(f"file:{path}?mode=ro", uri=True) + + a = take_snapshot(connect, "sqlite", key=KEY) + assert len(a.tables["kv"].rows) == 2 + con.execute("DELETE FROM kv WHERE a = 'x|y'") + con.commit() + assert ( + diff(a, take_snapshot(connect, "sqlite", key=KEY, before=a), key=KEY)[ + "kv" + ].removed + == 1 + ) + + +def test_a_schema_entry_in_the_allow_list_can_only_be_skipped() -> None: + from openadapt_flow.runtime.effects.frame import AllowedTable + + AllowedTable(table="audit.*", mode="skip", reason="the DBA's own audit trail") + with pytest.raises(ValueError, match="can only be skipped"): + AllowedTable(table="audit.*") + + +@live +def test_composite_keys_that_join_alike_stay_apart(pg: Pg) -> None: + # Review #1673, S1: ('x|y', 'z') and ('x', 'y|z') are two rows. + pg.sql( + "CREATE TABLE kv (a text, b text, PRIMARY KEY (a, b))", + "INSERT INTO kv VALUES ('x|y', 'z'), ('x', 'y|z')", + ) + a = pg.snap() + assert len(a.tables["kv"].rows) == 2 + pg.sql("DELETE FROM kv WHERE a = 'x|y'") + assert diff(a, pg.snap(before=a), key=KEY)["kv"].removed == 1 + + +@live +def test_large_objects_are_read_by_content_or_skipped(pg: Pg) -> None: + # Review #1673, S2: lo_put keeps the oid but changes the content. + pg.sql("SELECT lo_from_bytea(424242, 'scan v1')") + a = pg.snap() + assert a.tables["pg_largeobject"].count == 1 + pg.sql("SELECT lo_put(424242, 0, 'SCAN')") + moved = diff(a, pg.snap(before=a), key=KEY) + assert list(moved) == ["pg_largeobject"] + assert moved["pg_largeobject"].changed_columns == ["sha256"] + b = pg.snap() + pg.sql("SELECT lo_unlink(424242)") + assert diff(b, pg.snap(before=b), key=KEY)["pg_largeobject"].removed == 1 + pg.sql("SELECT lo_from_bytea(424243, 'x')") + with pytest.raises(SnapshotError, match="large objects can't be read"): + pg.snap(reader=True) + skip = {"pg_largeobject": "skip"} + c = pg.snap(reader=True, modes=skip) + pg.sql("SELECT lo_put(424243, 0, 'y')") + assert diff(c, pg.snap(before=c, reader=True, modes=skip), key=KEY) == {} + + +@live +def test_a_counter_set_back_shows(pg: Pg) -> None: + # Review #1673, N1: setval back lets later inserts reuse ids. + a = pg.snap() + pg.sql("SELECT setval('t_id_seq', 1)") + moved = diff(a, pg.snap(before=a), key=KEY) + assert moved["t"].unread == ["the counter went back"] + + +@live +def test_row_level_security_that_hides_rows_is_refused(pg: Pg) -> None: + pg.sql( + "ALTER TABLE pairs ENABLE ROW LEVEL SECURITY", + f"CREATE POLICY only_one ON pairs FOR SELECT TO {pg.reader} USING (a = 1)", + ) + with pytest.raises(SnapshotError, match="row-level security"): + pg.snap(reader=True) + + +@live +def test_another_schema_is_read_and_named_with_it(pg: Pg) -> None: + pg.sql( + "CREATE SCHEMA billing", + "CREATE TABLE billing.invoice (id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, n int)", + "INSERT INTO billing.invoice (n) VALUES (1)", + ) + a = pg.snap() + assert a.schemas == ["billing", "public"] + assert a.tables["billing.invoice"].counter == 1 + pg.sql("INSERT INTO billing.invoice (n) VALUES (2)") + assert diff(a, pg.snap(before=a), key=KEY)["billing.invoice"].added == 1 + + +@live +def test_a_partitioned_table_is_read_once_through_its_partitions(pg: Pg) -> None: + pg.sql( + "CREATE TABLE ev (id int, at date) PARTITION BY RANGE (at)", + "CREATE TABLE ev_2026 PARTITION OF ev FOR VALUES FROM ('2026-01-01') TO ('2027-01-01')", + "INSERT INTO ev VALUES (1, '2026-05-01')", + ) + a = pg.snap() + assert "ev" not in a.tables and a.tables["ev_2026"].count == 1 + + +@live +def test_a_trigger_planted_in_the_window_shows(pg: Pg) -> None: + a = pg.snap() + pg.sql( + "CREATE FUNCTION f() RETURNS trigger LANGUAGE plpgsql AS 'BEGIN RETURN NEW; END'", + "CREATE TRIGGER tr BEFORE INSERT ON pairs FOR EACH ROW EXECUTE FUNCTION f()", + ) + b = pg.snap(before=a) + assert "(database code)" in diff(a, b, key=KEY) + pg.sql("ALTER TABLE pairs DISABLE TRIGGER tr") + assert "(database code)" in diff(b, pg.snap(before=b), key=KEY) + + +@live +@pytest.mark.parametrize( + "change", + [ + "ALTER TYPE mood ADD VALUE 'z'", + "ALTER DOMAIN posint ADD CONSTRAINT small CHECK (VALUE < 1000)", + "CREATE RULE r AS ON INSERT TO pairs DO INSTEAD NOTHING", + "CREATE VIEW v AS SELECT a FROM pairs", + ], +) +def test_a_change_to_types_rules_or_views_shows(pg: Pg, change: str) -> None: + a = pg.snap() + pg.sql(change) + assert "(database code)" in diff(a, pg.snap(before=a), key=KEY) + + +@live +def test_a_column_default_is_read_from_the_catalog(pg: Pg) -> None: + pg.sql( + "CREATE TABLE d (id serial PRIMARY KEY, s varchar(9) DEFAULT 'open', " + "n numeric(6,2) DEFAULT 0, at timestamptz DEFAULT now(), f boolean DEFAULT false)" + ) + defaults = pg.snap().tables["d"].defaults + assert defaults == { + "id": db_snapshot.ANY_VALUE, + "s": "open", + "n": "0", + "at": CLOCK_DEFAULT, + "f": "false", + } + + +# -- the frame and its recheck, end to end ---------------------------------------- + + +@live +def test_the_frame_refutes_a_late_write_and_an_extra_record_on_postgresql( + pg: Pg, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + from psycopg2.extensions import parse_dsn + + from openadapt_flow import verifier + from openadapt_flow.verifier import ( + Actor, + Contract, + begin_write, + recheck, + verify_write, + ) + + monkeypatch.setenv("OPENADAPT_LEDGER_DIR", str(tmp_path / "ledger")) + monkeypatch.setenv("OPENADAPT_SEAL_HOME", str(tmp_path / "seal")) + monkeypatch.setattr(verifier, "USED_DIR", tmp_path / "used") + monkeypatch.setattr(verifier, "seconds_until", lambda due: 0.0) + monkeypatch.setenv("PG_READER_PW", pg.password) + pg.sql( + "CREATE TABLE contact (id serial PRIMARY KEY, name text, email text)", + "CREATE TABLE note (id serial PRIMARY KEY, body text)", + f"GRANT SELECT ON ALL TABLES IN SCHEMA public TO {pg.reader}", + f"GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO {pg.reader}", + "CREATE SCHEMA other", + "CREATE TABLE other.note (id serial PRIMARY KEY, body text)", + f"GRANT USAGE ON SCHEMA other TO {pg.reader}", + f"GRANT SELECT ON ALL TABLES IN SCHEMA other TO {pg.reader}", + f"GRANT SELECT ON ALL SEQUENCES IN SCHEMA other TO {pg.reader}", + ) + dsn = parse_dsn(pg.dsn) + read = { + "sql_driver": "psycopg2", + "sql_connect_args": { + "host": dsn.get("host", "127.0.0.1"), + "port": int(dsn.get("port", 5432)), + "user": pg.reader, + "dbname": pg.name, + }, + "sql_password_env": "PG_READER_PW", + } + contract = Contract.from_dict( + { + "schema_version": "openadapt.verify/v1", + "verifier": {"kind": "sql", **read, "sql_query": "SELECT * FROM contact"}, + "record": { + "name": "contact", + "match": ["email"], + "fields": {"name": "name", "email": "email"}, + }, + "frame": { + "identity_field": "id", + "database": { + "dialect": "postgresql", + "read": read, + "views": {"contact": ""}, + "settle_s": 0, + "recheck_after_s": 60, + # Must be visible; every other visible schema is read too. + "schemas": ["public"], + }, + }, + } + ) + bot = Actor(kind="agent", id="bot") + + def save(n: int, *stray: str) -> Any: + params = {"name": "Ada", "email": f"ada{n}@example.test"} + before = begin_write(contract, params, actor=bot) + pg.sql( + f"INSERT INTO contact (name, email) VALUES ('Ada', 'ada{n}@example.test')", + *stray, + ) + return params, verify_write(before) + + params, ok = save(1) + assert ok.verified, ok.reasons + assert recheck(contract, ok, params=params, actor=bot).outcome == "CONFIRMED" + + params, extra = save(2, "INSERT INTO note (body) VALUES ('stray')") + assert not extra.verified and extra.outcome == "REFUTED", extra.reasons + + # Review #1673, B1: a stray row in a schema the contract doesn't name. + params, other = save(4, "INSERT INTO other.note (body) VALUES ('stray')") + assert not other.verified and other.outcome == "REFUTED", other.reasons + + params, late = save(3) + assert late.verified, late.reasons + pg.sql("UPDATE contact SET name = 'Eve' WHERE email = 'ada3@example.test'") + assert recheck(contract, late, params=params, actor=bot).outcome == "REFUTED" diff --git a/packages/flow/tests/test_generate_verify_contract.py b/packages/flow/tests/test_generate_verify_contract.py new file mode 100644 index 0000000000..3806f18152 --- /dev/null +++ b/packages/flow/tests/test_generate_verify_contract.py @@ -0,0 +1,459 @@ +"""The contract generator on a real PostgreSQL: demonstrations in, contracts out. + +A throwaway PostgreSQL (``initdb``) holds a small issue tracker shaped like +Redmine: names behind keys with no declared foreign keys, a person's name in +two columns, notes in a journal table, custom fields in an +entity-attribute-value table, look-alike projects with the same issue +subject. The test records two demonstration writes per kind, lets the +generator propose contracts, then runs correct writes and planted faults +through ``begin_write`` and ``verify_write``. Every correct write must be +CONFIRMED and every fault stopped. Skipped when PostgreSQL isn't installed. +Records are made up. +""" + +from __future__ import annotations + +import json +import os +import shutil +import socket +import subprocess +import tempfile +import time +from pathlib import Path +from typing import Any, Callable + +import pytest +import yaml + +psycopg2 = pytest.importorskip("psycopg2") + +from openadapt_flow.generate import verify_contract as gen # noqa: E402 + +INITDB = shutil.which("initdb") or "/opt/homebrew/bin/initdb" +PG_CTL = shutil.which("pg_ctl") or "/opt/homebrew/bin/pg_ctl" + +SCHEMA = """ +CREATE TABLE projects (id serial PRIMARY KEY, name text NOT NULL); +CREATE TABLE users (id serial PRIMARY KEY, login text, firstname text, lastname text); +CREATE TABLE statuses (id serial PRIMARY KEY, name text); +CREATE TABLE issues ( + id serial PRIMARY KEY, project_id integer NOT NULL, subject text NOT NULL, + status_id integer NOT NULL, assigned_to_id integer, due_date date, + done_ratio integer NOT NULL DEFAULT 0, is_private boolean NOT NULL DEFAULT false, + author_id integer NOT NULL, created_on timestamp, updated_on timestamp, + lock_version integer NOT NULL DEFAULT 0); +CREATE TABLE journals (id serial PRIMARY KEY, journalized_id integer, + journalized_type text, user_id integer, notes text DEFAULT '', created_on timestamp); +CREATE TABLE journal_details (id serial PRIMARY KEY, journal_id integer, + prop_key text, old_value text, value text); +CREATE TABLE custom_values (id serial PRIMARY KEY, customized_type text, + customized_id integer, custom_field_id integer, value text); +CREATE TABLE settings (id serial PRIMARY KEY, name text, value text); +INSERT INTO projects (name) VALUES ('Billing Portal'), ('Billing Portal Legacy'), ('Help Desk'); +INSERT INTO users (login, firstname, lastname) VALUES ('bot', 'Ops', 'Bot'), + ('jpark', 'Jon', 'Park'), ('roconnor', 'Renée', 'O''Connor'), ('roconner', 'Renee', 'O''Conner'); +INSERT INTO statuses (name) VALUES ('New'), ('In Progress'), ('Resolved'); +INSERT INTO issues (project_id, subject, status_id, author_id, created_on, updated_on) VALUES + (1, 'Refund button missing', 1, 2, '2026-01-01', '2026-01-01'), + (2, 'Refund button missing', 1, 2, '2026-01-01', '2026-01-01'), + (3, 'Printer jams', 1, 2, '2026-01-01', '2026-01-01'), + (3, 'VPN drops', 2, 2, '2026-01-01', '2026-01-01'); +INSERT INTO settings (name, value) VALUES ('ui_theme', 'classic'); +CREATE ROLE oa_reader LOGIN PASSWORD 'reader-pw'; +GRANT SELECT ON ALL TABLES IN SCHEMA public TO oa_reader; +GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO oa_reader; +""" + +BOT = 1 + + +def _free_port() -> int: + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + return int(s.getsockname()[1]) + + +@pytest.fixture(scope="module") +def pg() -> Any: + if not (os.path.exists(INITDB) and os.path.exists(PG_CTL)): + pytest.skip("PostgreSQL (initdb, pg_ctl) is not installed") + root = Path(tempfile.mkdtemp(prefix="oa-gen-pg-")) + port = _free_port() + env = {**os.environ, "LC_ALL": "C"} + subprocess.run([INITDB, "-D", str(root / "data"), "-U", "admin", "--auth=trust", "-E", "UTF8", "--locale=C"], + check=True, capture_output=True, env=env) # fmt: skip + subprocess.run( + [PG_CTL, "-D", str(root / "data"), "-l", str(root / "log"), "-w", "start", "-o", + f"-p {port} -c unix_socket_directories='' -c listen_addresses=127.0.0.1"], + check=True, capture_output=True, env=env, + ) # fmt: skip + try: + yield {"root": root, "port": port} + finally: + subprocess.run([PG_CTL, "-D", str(root / "data"), "-m", "immediate", "stop"], + capture_output=True, env=env) # fmt: skip + shutil.rmtree(root, ignore_errors=True) + + +class App: + """The tracker's own writes, as its API would make them.""" + + def __init__(self, port: int, db: str) -> None: + self.port, self.db = port, db + + def con(self) -> Any: + c = psycopg2.connect( + host="127.0.0.1", port=self.port, user="admin", dbname=self.db + ) + c.autocommit = True + return c + + def q(self, sql: str, args: tuple = ()) -> list[tuple]: + with self.con() as c, c.cursor() as cur: + cur.execute(sql, args) + return cur.fetchall() if cur.description else [] + + def one(self, sql: str, args: tuple = ()) -> Any: + rows = self.q(sql, args) + assert len(rows) == 1, (sql, rows) + return rows[0][0] + + def user(self, name: str) -> int: + first, last = name.split(" ", 1) + return self.one( + "SELECT id FROM users WHERE firstname=%s AND lastname=%s", (first, last) + ) + + def create_issue(self, v: dict[str, Any], *, project_id: Any = None) -> int: + pid = project_id or self.one( + "SELECT id FROM projects WHERE name=%s", (v["project"],) + ) + sid = self.one("SELECT id FROM statuses WHERE name=%s", (v["status"],)) + uid = self.user(v["assignee"]) if v.get("assignee") else None + iid = self.one( + "INSERT INTO issues (project_id, subject, status_id, assigned_to_id, due_date, " + "author_id, created_on, updated_on) VALUES (%s,%s,%s,%s,%s,%s,now(),now()) RETURNING id", + (pid, v["subject"], sid, uid, v.get("due_date"), BOT), + ) + if v.get("customer_ref") is not None: + self.q("INSERT INTO custom_values (customized_type, customized_id, custom_field_id, value) " + "VALUES ('Issue', %s, 1, %s)", (iid, v["customer_ref"])) # fmt: skip + return iid + + def find(self, project: str, subject: str) -> int: + return self.one( + "SELECT i.id FROM issues i JOIN projects p ON p.id=i.project_id " + "WHERE p.name=%s AND i.subject=%s", (project, subject)) # fmt: skip + + def update_issue(self, iid: int, v: dict[str, Any], *, user: int = BOT) -> None: + sets, details = [], [] + if v.get("status") is not None: + sets.append( + ( + "status_id", + self.one("SELECT id FROM statuses WHERE name=%s", (v["status"],)), + ) + ) + if v.get("assignee") is not None: + sets.append(("assigned_to_id", self.user(v["assignee"]))) + if v.get("due_date") is not None: + sets.append(("due_date", v["due_date"])) + if v.get("done_ratio") is not None: + sets.append(("done_ratio", v["done_ratio"])) + for col, val in sets: + old = self.one(f"SELECT {col}::text FROM issues WHERE id=%s", (iid,)) + self.q(f"UPDATE issues SET {col}=%s WHERE id=%s", (val, iid)) + details.append((col, old, str(val))) + self.q( + "UPDATE issues SET updated_on=now(), lock_version=lock_version+1 WHERE id=%s", + (iid,), + ) + jid = self.one( + "INSERT INTO journals (journalized_id, journalized_type, user_id, notes, created_on) " + "VALUES (%s,'Issue',%s,%s,now()) RETURNING id", (iid, user, v.get("notes") or "")) # fmt: skip + for col, old, new in details: + self.q("INSERT INTO journal_details (journal_id, prop_key, old_value, value) " + "VALUES (%s,%s,%s,%s)", (jid, col, old, new)) # fmt: skip + + +KINDS = { + "kinds": [ + { + "kind": "create_issue", + "parameters": [ + "project", + "subject", + "status", + "assignee", + "due_date", + "customer_ref", + ], + "examples": [ + { + "values": { + "project": "Billing Portal", + "subject": "Tax rounding", + "status": "New", + "assignee": "Jon Park", + "due_date": "2026-11-20", + "customer_ref": None, + } + }, + { + "values": { + "project": "Help Desk", + "subject": "Onboard Zoë", + "status": "New", + "assignee": "Renée O'Connor", + "due_date": None, + "customer_ref": "007341", + } + }, + ], + }, + { + "kind": "update_issue", + "parameters": [ + "project", + "subject", + "status", + "assignee", + "due_date", + "done_ratio", + "notes", + ], + "examples": [ + { + "values": { + "project": "Help Desk", + "subject": "Printer jams", + "status": "In Progress", + "assignee": "Jon Park", + "due_date": None, + "done_ratio": None, + "notes": None, + } + }, + { + "values": { + "project": "Billing Portal", + "subject": "Refund button missing", + "status": None, + "assignee": None, + "due_date": None, + "done_ratio": 30, + "notes": "Customer confirmed the refund.", + } + }, + ], + }, + ] +} + + +@pytest.fixture(scope="module") +def generated(pg: Any) -> dict[str, Any]: + root, port = pg["root"], pg["port"] + admin = psycopg2.connect( + host="127.0.0.1", port=port, user="admin", dbname="postgres" + ) + admin.autocommit = True + admin.cursor().execute("CREATE DATABASE tracker") + admin.close() + app = App(port, "tracker") + with app.con() as c, c.cursor() as cur: + cur.execute(SCHEMA) + os.environ["OA_GEN_TEST_PW"] = "reader-pw" + work = root / "work" + work.mkdir() + read = {"sql_driver": "psycopg2", + "sql_connect_args": {"host": "127.0.0.1", "port": port, "user": "oa_reader", "dbname": "tracker"}, + "sql_password_env": "OA_GEN_TEST_PW"} # fmt: skip + (work / "read.yaml").write_text(yaml.safe_dump(read)) + (work / "kinds.yaml").write_text(yaml.safe_dump(KINDS, allow_unicode=True)) + common = ["--read", str(work / "read.yaml")] + assert gen.main(["schema", *common]) == 0 + + def demo(kind: str, values: dict[str, Any], write: Callable[[], Any]) -> None: + assert gen.main(["snapshot", *common, "--out", str(work / "before.json")]) == 0 + write() + assert gen.main(["record", *common, "--before", str(work / "before.json"), "--kind", kind, + "--values", json.dumps(values), "--demos", str(work / "demos")]) == 0 # fmt: skip + + for ex in KINDS["kinds"][0]["examples"]: + demo("create_issue", ex["values"], lambda v=ex["values"]: app.create_issue(v)) + for ex in KINDS["kinds"][1]["examples"]: + v = ex["values"] + demo( + "update_issue", + v, + lambda v=v: app.update_issue(app.find(v["project"], v["subject"]), v), + ) + argv = ["propose", *common, "--kinds", str(work / "kinds.yaml"), "--demos", str(work / "demos"), + "--out", str(work / "out"), "--actor-param", f"bot_user_id={BOT}"] # fmt: skip + assert gen.main(argv) == 0 + report = json.loads((work / "out" / "report.json").read_text()) + answers = {q["key"]: q["proposal"] for q in report["questions"]} + # The person takes the proposal (each unchecked column named) and names + # journal_details a log. + proposal = answers["bookkeeping"] + answers["bookkeeping"] = { + **(proposal if isinstance(proposal, dict) else {"accept": True}), + "logs_add": ["journal_details"], + } + (work / "answers.yaml").write_text(yaml.safe_dump(answers)) + assert gen.main([*argv, "--answers", str(work / "answers.yaml")]) == 0 + report = json.loads((work / "out" / "report.json").read_text()) + if os.environ.get("OA_GEN_TEST_KEEP"): + shutil.copytree(work, os.environ["OA_GEN_TEST_KEEP"], dirs_exist_ok=True) + return {"app": app, "out": work / "out", "report": report, "read": read} + + +def _run( + generated: dict[str, Any], + kind: str, + values: dict[str, Any], + write: Callable[[], Any], +) -> str: + from openadapt_flow.verifier import Actor, begin_write, verify_write + + contract, params = gen.choose( + generated["out"] / "manifest.yaml", kind, values, {"bot_user_id": str(BOT)} + ) + with tempfile.TemporaryDirectory() as tmp: + pending = begin_write(contract, params, actor=Actor("agent", "ops-bot"), ledger=None, + write_key=f"t-{time.time_ns()}", workdir=tmp) # fmt: skip + write() + result = verify_write(pending) + return ( + getattr(result.outcome, "value", str(result.outcome)) + + " " + + "; ".join(result.reasons)[:400] + ) + + +def test_questions_and_contracts(generated: dict[str, Any]) -> None: + report = generated["report"] + keys = [q["key"] for q in report["questions"]] + assert len(keys) <= 3, keys + kinds = report["kinds"] + assert kinds["create_issue"]["mode"] == "create" + assert kinds["update_issue"]["mode"] == "update" + assert kinds["create_issue"]["emitted"] > 0, ( + kinds["create_issue"]["refused"], + kinds["create_issue"]["rejected"][:1], + ) + # Four change fields the examples give (status, assignee, done ratio, + # notes), and the due date that create_issue's demonstrations placed on + # the same table: every non-empty subset gets a contract. + assert kinds["update_issue"]["placed_from_another_kind"] == ["due_date"] + # lock_version steps by one per save: the update's contracts read it as + # the target's versions (a lost update shows there). It is preferred to + # the journals, which an issue saved before them doesn't have. + assert kinds["update_issue"]["versions"]["kind"] == "step" + assert kinds["update_issue"]["versions"]["column"] == "lock_version" + assert kinds["update_issue"]["emitted"] == 31, json.dumps( + kinds["update_issue"], indent=1 + ) + assert "journal_details" in kinds["update_issue"]["log_candidates"] + + +def test_a_correct_create_is_confirmed(generated: dict[str, Any]) -> None: + app = generated["app"] + v = {"project": "Billing Portal", "subject": "Dark mode", "status": "New", + "assignee": "Renée O'Connor", "due_date": "2026-12-01", "customer_ref": "0099"} # fmt: skip + assert _run(generated, "create_issue", v, lambda: app.create_issue(v)).startswith( + "CONFIRMED" + ) + + +def test_found_existing_with_no_write_is_confirmed(generated: dict[str, Any]) -> None: + app = generated["app"] + v = {"project": "Help Desk", "subject": "Make sure", "status": "New", + "assignee": "Jon Park", "due_date": None, "customer_ref": None} # fmt: skip + app.create_issue(v) + assert _run(generated, "create_issue", v, lambda: None).startswith("CONFIRMED") + + +def test_an_undemonstrated_update_combination_is_confirmed( + generated: dict[str, Any], +) -> None: + app = generated["app"] + v = {"project": "Help Desk", "subject": "VPN drops", "status": "Resolved", "assignee": None, + "due_date": None, "done_ratio": 80, "notes": "Fixed the tunnel."} # fmt: skip + iid = app.find("Help Desk", "VPN drops") + assert _run( + generated, "update_issue", v, lambda: app.update_issue(iid, v) + ).startswith("CONFIRMED") + + +def test_an_update_already_in_place_with_no_write_is_confirmed( + generated: dict[str, Any], +) -> None: + app = generated["app"] + v = {"project": "Help Desk", "subject": "VPN drops", "status": None, "assignee": None, + "due_date": None, "done_ratio": 80, "notes": None} # fmt: skip + app.q( + "UPDATE issues SET done_ratio=80 WHERE id=%s", + (app.find("Help Desk", "VPN drops"),), + ) + assert _run(generated, "update_issue", v, lambda: None).startswith("CONFIRMED") + # The same task with a stray write on the issue is stopped. + stray = _run(generated, "update_issue", v, + lambda: app.q("UPDATE issues SET is_private=true WHERE subject='VPN drops'")) # fmt: skip + app.q("UPDATE issues SET is_private=false WHERE subject='VPN drops'") + assert not stray.startswith("CONFIRMED"), stray + + +CREATE ={"project": "Billing Portal", "subject": "Export CSV", "status": "New", + "assignee": "Jon Park", "due_date": "2026-12-05", "customer_ref": "0042"} # fmt: skip +UPDATE = {"project": "Billing Portal", "subject": "Refund button missing", "status": "Resolved", + "assignee": None, "due_date": None, "done_ratio": None, "notes": "Done."} # fmt: skip + + +def _faults(app: App) -> dict[str, tuple[str, dict[str, Any], Callable[[], Any]]]: + def legacy() -> int: + return app.find("Billing Portal Legacy", "Refund button missing") + + def target() -> int: + return app.find("Billing Portal", "Refund button missing") + + return { + "wrong value": ("create_issue", CREATE, lambda: app.create_issue({**CREATE, "assignee": "Renee O'Conner"})), + "wrong project": ("create_issue", CREATE, lambda: app.create_issue(CREATE, project_id=2)), + "duplicate": ("create_issue", CREATE, lambda: (app.create_issue(CREATE), app.create_issue(CREATE))), + "nothing written": ("create_issue", CREATE, lambda: None), + "missing custom field": ("create_issue", CREATE, lambda: app.create_issue({**CREATE, "customer_ref": None})), + "extra issue": ("create_issue", CREATE, lambda: (app.create_issue(CREATE), app.create_issue({**CREATE, "subject": "Follow-up"}))), + "setting changed": ("create_issue", CREATE, lambda: (app.create_issue(CREATE), app.q("UPDATE settings SET value='dark'"))), + "look-alike target": ("update_issue", UPDATE, lambda: app.update_issue(legacy(), UPDATE)), + "both targets": ("update_issue", UPDATE, lambda: (app.update_issue(target(), UPDATE), app.update_issue(legacy(), {"status": "Resolved"}))), + "note missing": ("update_issue", UPDATE, lambda: app.update_issue(target(), {**UPDATE, "notes": None})), + "collateral field": ("update_issue", UPDATE, lambda: (app.update_issue(target(), UPDATE), app.q("UPDATE issues SET is_private=true WHERE id=%s", (target(),)))), + "collateral delete": ("update_issue", UPDATE, lambda: (app.update_issue(target(), UPDATE), app.q("DELETE FROM issues WHERE subject='Printer jams'"))), + "note by someone else": ("update_issue", UPDATE, lambda: app.update_issue(target(), UPDATE, user=2)), + } # fmt: skip + + +@pytest.mark.parametrize("name", list(_faults(App(0, "")).keys())) +def test_each_fault_is_stopped(generated: dict[str, Any], name: str) -> None: + app = generated["app"] + kind, values, write = _faults(app)[name] + with app.con() as c, c.cursor() as cur: # each fault starts from the same rows + cur.execute("BEGIN") + snapshot = {t: cur.execute(f"SELECT * FROM {t}") or cur.fetchall() + for t in ("issues", "journals", "journal_details", "custom_values", "settings")} # fmt: skip + cur.execute("COMMIT") + try: + outcome = _run(generated, kind, values, write) + print(f"FAULT {name}: {outcome}") + assert not outcome.startswith("CONFIRMED"), (name, outcome) + finally: + with app.con() as c, c.cursor() as cur: + for t, rows in snapshot.items(): + cur.execute(f"DELETE FROM {t}") + for r in rows: + cur.execute( + f"INSERT INTO {t} VALUES ({', '.join(['%s'] * len(r))})", r + ) diff --git a/packages/flow/tests/test_generate_verify_contract_attacks.py b/packages/flow/tests/test_generate_verify_contract_attacks.py new file mode 100644 index 0000000000..9188445071 --- /dev/null +++ b/packages/flow/tests/test_generate_verify_contract_attacks.py @@ -0,0 +1,257 @@ +# ruff: noqa: F811 (pytest fixtures imported from the generator test, then used as arguments) +"""The adversarial review's attacks on generated contracts (#1672 round 1). + +Each test is one way a wrong write could have passed, from the review, run +on the same throwaway PostgreSQL as ``test_generate_verify_contract``. Every +wrong write must now stop, and a correct update of an issue that already has +history must be CONFIRMED. Records are made up. +""" + +from __future__ import annotations + +import json +import os +from typing import Any + +import pytest +import yaml + +psycopg2 = pytest.importorskip("psycopg2") + +from openadapt_flow.generate import verify_contract as gen # noqa: E402 +from tests.test_generate_verify_contract import ( # noqa: E402,F401 + BOT, + CREATE, + KINDS, + SCHEMA, + App, + _run, + generated, + pg, +) + + +def _fresh_issue(app: App, subject: str) -> int: + return app.create_issue( + {"project": "Help Desk", "subject": subject, "status": "New", + "assignee": None, "due_date": None, "customer_ref": None} + ) # fmt: skip + + +def test_a1_a_number_written_with_the_same_digits_is_stopped( + generated: dict[str, Any], +) -> None: + """The custom field is stored as given, so it is compared exactly; the + phone rule applies only to a value a person confirmed as a phone.""" + + app = generated["app"] + for asked, written in (("1500.75", "15007.5"), ("2026-1104", "202-611-04")): + v = {**CREATE, "subject": f"A1 {asked}", "customer_ref": asked} + out = _run(generated, "create_issue", v, + lambda v=v, w=written: app.create_issue({**v, "customer_ref": w})) # fmt: skip + assert not out.startswith("CONFIRMED"), out + + +def test_a2_a_display_name_two_users_share_is_stopped( + generated: dict[str, Any], +) -> None: + app = generated["app"] + other = app.one( + "INSERT INTO users (login, firstname, lastname) VALUES ('jpark2','Jon','Park') RETURNING id" + ) + try: + v = {**CREATE, "subject": "A2 shared name", "assignee": "Jon Park"} + + def write() -> None: + iid = app.create_issue({**v, "assignee": None}) + app.q("UPDATE issues SET assigned_to_id=%s WHERE id=%s", (other, iid)) + + out = _run(generated, "create_issue", v, write) + assert not out.startswith("CONFIRMED"), out + finally: + app.q("UPDATE issues SET assigned_to_id=NULL WHERE assigned_to_id=%s", (other,)) + app.q("DELETE FROM users WHERE id=%s", (other,)) + + +def test_a3_a_first_and_last_name_split_two_ways_is_stopped( + generated: dict[str, Any], +) -> None: + app = generated["app"] + a = app.one( + "INSERT INTO users (login, firstname, lastname) VALUES ('mal','Mary Ann','Lee') RETURNING id" + ) + b = app.one( + "INSERT INTO users (login, firstname, lastname) VALUES ('mal2','Mary','Ann Lee') RETURNING id" + ) + try: + v = {**CREATE, "subject": "A3 split", "assignee": "Mary Ann Lee"} + + def write() -> None: + iid = app.create_issue({**v, "assignee": None}) + app.q("UPDATE issues SET assigned_to_id=%s WHERE id=%s", (b, iid)) + + out = _run(generated, "create_issue", v, write) + assert not out.startswith("CONFIRMED"), out + finally: + app.q( + "UPDATE issues SET assigned_to_id=NULL WHERE assigned_to_id IN (%s,%s)", + (a, b), + ) + app.q("DELETE FROM users WHERE id IN (%s,%s)", (a, b)) + + +def test_a4_a_rewrite_of_a_log_table_on_postgresql_is_stopped( + generated: dict[str, Any], +) -> None: + app = generated["app"] + iid = _fresh_issue(app, "A4 fresh") + v = {"project": "Help Desk", "subject": "A4 fresh", "status": "Resolved", "assignee": None, + "due_date": None, "done_ratio": None, "notes": None} # fmt: skip + app.q( + "INSERT INTO journal_details (journal_id, prop_key, old_value, value) VALUES (999,'status_id','1','2')" + ) + + def write() -> None: + app.update_issue(iid, v) + app.q("UPDATE journal_details SET value='999' WHERE journal_id=999") + + out = _run(generated, "update_issue", v, write) + assert not out.startswith("CONFIRMED"), out + + +def test_a5_a_correct_update_of_an_issue_with_history_is_confirmed( + generated: dict[str, Any], +) -> None: + app = generated["app"] + iid = _fresh_issue(app, "A5 history") + first = {"project": "Help Desk", "subject": "A5 history", "status": "In Progress", "assignee": None, + "due_date": None, "done_ratio": None, "notes": None} # fmt: skip + app.update_issue( + iid, first, user=2 + ) # earlier history, by a colleague, before the window + v = {**first, "status": "Resolved"} + out = _run(generated, "update_issue", v, lambda: app.update_issue(iid, v)) + assert out.startswith("CONFIRMED"), out + + +# -- an app that touches a counter row on each create, and soft-deletes ------------- + +SCHEMA_B = ( + SCHEMA.replace( + "lock_version integer NOT NULL DEFAULT 0);", + "lock_version integer NOT NULL DEFAULT 0, deleted smallint NOT NULL DEFAULT 0);", + ) + .replace( + "INSERT INTO settings (name, value) VALUES ('ui_theme', 'classic');", + "INSERT INTO settings (name, value) VALUES ('ui_theme', 'classic'), ('last_issue', '0');", + ) + .replace("CREATE ROLE oa_reader LOGIN PASSWORD 'reader-pw';", "") +) + + +class AppB(App): + def create_issue(self, v: dict[str, Any], *, project_id: Any = None) -> int: + iid = super().create_issue(v, project_id=project_id) + self.q("UPDATE settings SET value=%s WHERE name='last_issue'", (str(iid),)) + return iid + + +@pytest.fixture(scope="module") +def gen_b(pg: Any) -> dict[str, Any]: + root, port = pg["root"], pg["port"] + admin = psycopg2.connect( + host="127.0.0.1", port=port, user="admin", dbname="postgres" + ) + admin.autocommit = True + admin.cursor().execute("CREATE DATABASE trackerb") + admin.close() + app = AppB(port, "trackerb") + with app.con() as c, c.cursor() as cur: + cur.execute( + "DO $$BEGIN IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname='oa_reader') " + "THEN CREATE ROLE oa_reader LOGIN PASSWORD 'reader-pw'; END IF; END$$" + ) + cur.execute(SCHEMA_B) + os.environ["OA_GEN_TEST_PW"] = "reader-pw" + work = root / "workb" + work.mkdir() + read = {"sql_driver": "psycopg2", + "sql_connect_args": {"host": "127.0.0.1", "port": port, "user": "oa_reader", "dbname": "trackerb"}, + "sql_password_env": "OA_GEN_TEST_PW"} # fmt: skip + (work / "read.yaml").write_text(yaml.safe_dump(read)) + kinds = {"kinds": [KINDS["kinds"][0]]} + (work / "kinds.yaml").write_text(yaml.safe_dump(kinds, allow_unicode=True)) + common = ["--read", str(work / "read.yaml")] + for ex in kinds["kinds"][0]["examples"]: + assert gen.main(["snapshot", *common, "--out", str(work / "before.json")]) == 0 + app.create_issue(ex["values"]) + assert gen.main(["record", *common, "--before", str(work / "before.json"), "--kind", "create_issue", + "--values", json.dumps(ex["values"]), "--demos", str(work / "demos")]) == 0 # fmt: skip + argv = ["propose", *common, "--kinds", str(work / "kinds.yaml"), "--demos", str(work / "demos"), + "--out", str(work / "out"), "--actor-param", f"bot_user_id={BOT}"] # fmt: skip + assert gen.main(argv) == 0 + report = json.loads((work / "out" / "report.json").read_text()) + answers = {q["key"]: q["proposal"] for q in report["questions"]} + (work / "answers.yaml").write_text(yaml.safe_dump(answers)) + assert gen.main([*argv, "--answers", str(work / "answers.yaml")]) == 0 + report = json.loads((work / "out" / "report.json").read_text()) + return {"app": app, "out": work / "out", "report": report, "read": read} + + +def test_b0_the_counter_row_is_excused_by_key_and_a_correct_create_passes( + gen_b: dict[str, Any], +) -> None: + rows = gen_b["report"]["kinds"]["create_issue"]["touched_rows"] + assert [(r["table"], r["columns"]) for r in rows] == [ + ("settings", {"value": "counter"}) + ] + app = gen_b["app"] + v = {**CREATE, "subject": "B0 correct"} + out = _run(gen_b, "create_issue", v, lambda: app.create_issue(v)) + assert out.startswith("CONFIRMED"), out + + +def test_b1_another_setting_changed_is_stopped(gen_b: dict[str, Any]) -> None: + app = gen_b["app"] + v = {**CREATE, "subject": "B1"} + try: + out = _run(gen_b, "create_issue", v, lambda: ( + app.create_issue(v), app.q("UPDATE settings SET value='dark' WHERE name='ui_theme'"))) # fmt: skip + finally: + app.q("UPDATE settings SET value='classic' WHERE name='ui_theme'") + assert not out.startswith("CONFIRMED"), out + + +def test_b2_another_records_custom_field_changed_is_stopped( + gen_b: dict[str, Any], +) -> None: + app = gen_b["app"] + victim = app.create_issue({**CREATE, "subject": "victim", "customer_ref": "777"}) + v = {**CREATE, "subject": "B2"} + out = _run(gen_b, "create_issue", v, lambda: ( + app.create_issue(v), + app.q("UPDATE custom_values SET value='HIJACKED' WHERE customized_id=%s", (victim,)))) # fmt: skip + assert not out.startswith("CONFIRMED"), out + + +def test_b3_a_trashed_record_with_the_values_is_not_found( + gen_b: dict[str, Any], +) -> None: + app = gen_b["app"] + v = {**CREATE, "subject": "B3 trashed"} + iid = app.create_issue(v) + app.q("UPDATE issues SET deleted=1 WHERE id=%s", (iid,)) + out = _run(gen_b, "create_issue", v, lambda: None) + assert not out.startswith("CONFIRMED"), out + + +def test_b4_a_create_that_writes_deleted_is_stopped(gen_b: dict[str, Any]) -> None: + app = gen_b["app"] + v = {**CREATE, "subject": "B4"} + + def write() -> None: + iid = app.create_issue(v) + app.q("UPDATE issues SET deleted=1 WHERE id=%s", (iid,)) + + out = _run(gen_b, "create_issue", v, write) + assert not out.startswith("CONFIRMED"), out diff --git a/packages/flow/tests/test_generate_verify_contract_attacks_r2.py b/packages/flow/tests/test_generate_verify_contract_attacks_r2.py new file mode 100644 index 0000000000..d51e4a5df6 --- /dev/null +++ b/packages/flow/tests/test_generate_verify_contract_attacks_r2.py @@ -0,0 +1,424 @@ +# ruff: noqa: F811 (pytest fixtures imported from the generator test, then used as arguments) +"""The adversarial review's round-2 attacks on generated contracts (#1672). + +An issue tracker that audits one row per changed field and touches the +issue's project on every save, on a throwaway PostgreSQL. Every wrong write +must stop; the correct writes beside them must be CONFIRMED. Also: the +bookkeeping answer can name only what its question offered, and the +PostgreSQL read-only check counts inherited and non-DML privileges. Records +are made up. +""" + +from __future__ import annotations + +import json +import os +from typing import Any + +import pytest +import yaml + +psycopg2 = pytest.importorskip("psycopg2") + +from openadapt_flow.generate import verify_contract as gen # noqa: E402 +from tests.test_generate_verify_contract import ( # noqa: E402,F401 + BOT, + SCHEMA, + App, + _run, + pg, +) + +SCHEMA_V = ( + SCHEMA.replace( + "CREATE TABLE projects (id serial PRIMARY KEY, name text NOT NULL);", + "CREATE TABLE projects (id serial PRIMARY KEY, name text NOT NULL, " + "description text, updated_on timestamp);", + ) + .replace( + # No version column: the versions come from the audit trail. + "lock_version integer NOT NULL DEFAULT 0);", + "description text, parent_id integer);", + ) + .replace( + "CREATE TABLE settings", + "CREATE TABLE audits (id serial PRIMARY KEY, issue_id integer, user_id integer, " + "action text, field_name text, old_value text, new_value text);\nCREATE TABLE settings", + ) + .replace("CREATE ROLE oa_reader LOGIN PASSWORD 'reader-pw';", "") +) + + +class AppV(App): + """Audits one row per changed field; touches the project on every save + (a parent ``touch``); writes no journals and has no version column.""" + + def create_issue(self, v: dict[str, Any], *, project_id: Any = None) -> int: + pid = project_id or self.one( + "SELECT id FROM projects WHERE name=%s", (v["project"],) + ) + sid = self.one("SELECT id FROM statuses WHERE name=%s", (v["status"],)) + parent = None + if v.get("parent"): + parent = v.get("parent_id") or self.one( + "SELECT i.id FROM issues i WHERE i.subject=%s ORDER BY id LIMIT 1", + (v["parent"],), + ) + iid = self.one( + "INSERT INTO issues (project_id, subject, status_id, author_id, created_on, updated_on, parent_id) " + "VALUES (%s,%s,%s,%s,now(),now(),%s) RETURNING id", + (pid, v["subject"], sid, BOT, parent), + ) + self.q( + "INSERT INTO audits (issue_id, user_id, action) VALUES (%s,%s,'create')", + (iid, BOT), + ) + self.q("UPDATE projects SET updated_on=now() WHERE id=%s", (pid,)) + return iid # fmt: skip + + def save(self, iid: int, sets: dict[str, Any], *, user: int = BOT) -> None: + for col, val in sets.items(): + if col == "status": + col, val = ( + "status_id", + self.one("SELECT id FROM statuses WHERE name=%s", (val,)), + ) + old = self.one(f"SELECT {col}::text FROM issues WHERE id=%s", (iid,)) + self.q(f"UPDATE issues SET {col}=%s WHERE id=%s", (val, iid)) + self.q( + "INSERT INTO audits (issue_id, user_id, action, field_name, old_value, new_value) " + "VALUES (%s,%s,'update',%s,%s,%s)", + (iid, user, col, old, None if val is None else f"v{val}"), + ) + self.q("UPDATE issues SET updated_on=now() WHERE id=%s", (iid,)) + self.q("UPDATE projects SET updated_on=now() WHERE id=(SELECT project_id FROM issues WHERE id=%s)", + (iid,)) # fmt: skip + + +KINDS_V = { + "kinds": [ + {"kind": "create_issue", "parameters": ["project", "subject", "status", "parent"], + "examples": [ + {"values": {"project": "Help Desk", "subject": "Toner low", "status": "New", "parent": "Printer jams"}}, + {"values": {"project": "Billing Portal", "subject": "VPN cert", "status": "New", "parent": "VPN drops"}}, + ]}, + {"kind": "set_status", "parameters": ["project", "subject", "status"], + "examples": [ + {"values": {"project": "Help Desk", "subject": "Printer jams", "status": "In Progress"}}, + {"values": {"project": "Help Desk", "subject": "VPN drops", "status": "Resolved"}}, + ]}, + {"kind": "edit_description", "parameters": ["project", "subject", "description"], + "examples": [ + {"values": {"project": "Help Desk", "subject": "VPN drops", "description": "Drops every hour"}}, + {"values": {"project": "Billing Portal", "subject": "Refund button missing", + "description": "Only on Safari"}}, + ]}, + ] +} # fmt: skip + + +def _propose(work: Any, argv: list[str], answers: Any) -> int: + (work / "answers.yaml").write_text(yaml.safe_dump(answers)) + return gen.main([*argv, "--answers", str(work / "answers.yaml")]) + + +@pytest.fixture(scope="module") +def gv(pg: Any) -> dict[str, Any]: + root, port = pg["root"], pg["port"] + admin = psycopg2.connect( + host="127.0.0.1", port=port, user="admin", dbname="postgres" + ) + admin.autocommit = True + admin.cursor().execute("CREATE DATABASE trackerv") + admin.close() + app = AppV(port, "trackerv") + with app.con() as c, c.cursor() as cur: + cur.execute( + "DO $$BEGIN IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname='oa_reader') " + "THEN CREATE ROLE oa_reader LOGIN PASSWORD 'reader-pw'; END IF; END$$" + ) + cur.execute(SCHEMA_V) + os.environ["OA_GEN_TEST_PW"] = "reader-pw" + work = root / "workv" + work.mkdir() + read = {"sql_driver": "psycopg2", + "sql_connect_args": {"host": "127.0.0.1", "port": port, "user": "oa_reader", "dbname": "trackerv"}, + "sql_password_env": "OA_GEN_TEST_PW"} # fmt: skip + (work / "read.yaml").write_text(yaml.safe_dump(read)) + (work / "kinds.yaml").write_text(yaml.safe_dump(KINDS_V, allow_unicode=True)) + common = ["--read", str(work / "read.yaml")] + + def demo(kind: str, values: dict[str, Any], write: Any) -> None: + assert gen.main(["snapshot", *common, "--out", str(work / "before.json")]) == 0 + write() + assert gen.main(["record", *common, "--before", str(work / "before.json"), "--kind", kind, + "--values", json.dumps(values), "--demos", str(work / "demos")]) == 0 # fmt: skip + + for k in KINDS_V["kinds"]: + for ex in k["examples"]: + v = ex["values"] + if k["kind"] == "create_issue": + demo("create_issue", v, lambda v=v: app.create_issue(v)) + else: + sets = {p: v[p] for p in k["parameters"][2:] if v.get(p) is not None} + demo( + k["kind"], + v, + lambda v=v, s=sets: app.save( + app.find(v["project"], v["subject"]), s + ), + ) + argv = ["propose", *common, "--kinds", str(work / "kinds.yaml"), "--demos", str(work / "demos"), + "--out", str(work / "out"), "--actor-param", f"bot_user_id={BOT}"] # fmt: skip + assert gen.main(argv) == 0 + report = json.loads((work / "out" / "report.json").read_text()) + answers = {q["key"]: q["proposal"] for q in report["questions"]} + assert _propose(work, argv, answers) == 0 + report = json.loads((work / "out" / "report.json").read_text()) + return {"app": app, "out": work / "out", "report": report, "read": read, + "work": work, "argv": argv, "answers": answers} # fmt: skip + + +def _fresh(app: AppV, subject: str) -> int: + return app.create_issue( + {"project": "Help Desk", "subject": subject, "status": "New"} + ) + + +def test_versions_read_write_events_of_creates_and_updates(gv: dict[str, Any]) -> None: + v = gv["report"]["kinds"]["edit_description"]["versions"] + assert v["kind"] == "audit" and v["table"] == "audits", v + # No demonstration logged a code on a record it didn't change, so no + # code is shown to be a read: every entry counts as a save. A column + # that varies with the fields saved (field_name) is never a filter. + assert v["where"] == {}, v + + +def test_correct_status_update_confirmed(gv: dict[str, Any]) -> None: + app = gv["app"] + iid = _fresh(app, "V1 control") + v = {"project": "Help Desk", "subject": "V1 control", "status": "Resolved"} + out = _run(gv, "set_status", v, lambda: app.save(iid, {"status": "Resolved"})) + assert out.startswith("CONFIRMED"), out + + +def test_correct_description_edit_confirmed(gv: dict[str, Any]) -> None: + app = gv["app"] + iid = _fresh(app, "T1 control") + v = {"project": "Help Desk", "subject": "T1 control", "description": "Fixed"} + out = _run( + gv, "edit_description", v, lambda: app.save(iid, {"description": "Fixed"}) + ) + assert out.startswith("CONFIRMED"), out + + +def test_lost_update_of_another_field_stops(gv: dict[str, Any]) -> None: + """Another user sets the due date; the agent's stale save puts it back.""" + app = gv["app"] + iid = _fresh(app, "V4 lost update") + v = {"project": "Help Desk", "subject": "V4 lost update", "description": "Fixed"} + + def write() -> None: + app.save(iid, {"due_date": "2026-12-31"}, user=2) + app.save(iid, {"description": "Fixed", "due_date": None}) + + out = _run(gv, "edit_description", v, write) + assert not out.startswith("CONFIRMED"), out + + +def test_lost_update_of_the_same_field_stops(gv: dict[str, Any]) -> None: + app = gv["app"] + iid = _fresh(app, "V5") + v = {"project": "Help Desk", "subject": "V5", "description": "Fixed"} + + def write() -> None: + app.save(iid, {"description": "Theirs"}, user=2) + app.save(iid, {"description": "Fixed"}) + + out = _run(gv, "edit_description", v, write) + assert not out.startswith("CONFIRMED"), out + + +def test_target_without_a_baseline_version_stops(gv: dict[str, Any]) -> None: + """An issue with no audit row at all (written outside the app) can't be + shown to be new: with no version before the write it stops as missing.""" + app = gv["app"] + iid = _fresh(app, "No history") + app.q("DELETE FROM audits WHERE issue_id=%s", (iid,)) + v = {"project": "Help Desk", "subject": "No history", "description": "Fixed"} + out = _run( + gv, "edit_description", v, lambda: app.save(iid, {"description": "Fixed"}) + ) + assert not out.startswith("CONFIRMED"), out + + +def test_touched_parent_row_cant_take_the_targets_field(gv: dict[str, Any]) -> None: + app = gv["app"] + iid = _fresh(app, "T2 leak") + v = {"project": "Help Desk", "subject": "T2 leak", "description": "Fixed"} + + def write() -> None: + app.save(iid, {"description": "Fixed"}) + app.q( + "UPDATE projects SET description='Overwritten by the agent' WHERE name='Help Desk'" + ) + + try: + out = _run(gv, "edit_description", v, write) + finally: + app.q("UPDATE projects SET description=NULL WHERE name='Help Desk'") + assert not out.startswith("CONFIRMED"), out + + +def test_target_cant_take_the_touched_rows_field(gv: dict[str, Any]) -> None: + """A column nothing excuses on the target (is_private) stops, beside a + correct save.""" + app = gv["app"] + iid = _fresh(app, "T4") + v = {"project": "Help Desk", "subject": "T4", "description": "Fixed"} + + def write() -> None: + app.save(iid, {"description": "Fixed"}) + app.q("UPDATE issues SET is_private=true WHERE id=%s", (iid,)) + + out = _run(gv, "edit_description", v, write) + assert not out.startswith("CONFIRMED"), out + + +def test_correct_create_with_parent_confirmed(gv: dict[str, Any]) -> None: + app = gv["app"] + v = { + "project": "Help Desk", + "subject": "P0 child", + "status": "New", + "parent": "VPN drops", + } + out = _run(gv, "create_issue", v, lambda: app.create_issue(v)) + assert out.startswith("CONFIRMED"), out + + +def test_parent_named_by_a_subject_two_issues_share_stops(gv: dict[str, Any]) -> None: + """'Refund button missing' names two issues: a key back to the main table + must name exactly one row too.""" + app = gv["app"] + legacy = app.find("Billing Portal Legacy", "Refund button missing") + v = { + "project": "Help Desk", + "subject": "P1 child", + "status": "New", + "parent": "Refund button missing", + } + out = _run( + gv, "create_issue", v, lambda: app.create_issue({**v, "parent_id": legacy}) + ) + assert not out.startswith("CONFIRMED"), out + + +# -- answers name only what the question offered -------------------------------- + + +def _offered(gv: dict[str, Any]) -> Any: + return dict(gv["answers"]) + + +def test_answer_with_an_unknown_key_is_refused(gv: dict[str, Any], capsys: Any) -> None: + answers = _offered(gv) + answers["bookkeeping"] = {"accept": True, "also_excuse": {"issues": ["subject"]}} + assert _propose(gv["work"], gv["argv"], answers) != 0 + assert "unknown answer keys" in capsys.readouterr().err + + +def test_logs_add_of_an_unlisted_table_is_refused( + gv: dict[str, Any], capsys: Any +) -> None: + answers = _offered(gv) + answers["bookkeeping"] = {"accept": True, "logs_add": ["issues"]} + assert _propose(gv["work"], gv["argv"], answers) != 0 + assert "logs_add names tables" in capsys.readouterr().err + + +def test_columns_add_of_an_unlisted_column_is_refused( + gv: dict[str, Any], capsys: Any +) -> None: + answers = _offered(gv) + answers["bookkeeping"] = {"accept": True, "columns_add": {"issues": ["subject"]}} + assert _propose(gv["work"], gv["argv"], answers) != 0 + assert "columns_add names" in capsys.readouterr().err + + +def test_a_mapping_without_accept_takes_no_proposed_column(gv: dict[str, Any]) -> None: + """``{logs_add: [...]}`` alone doesn't accept the proposed bookkeeping: + the save's updated_on then isn't excused and a correct update stops.""" + answers = _offered(gv) + answers["bookkeeping"] = {"logs_add": ["audits"]} + assert _propose(gv["work"], gv["argv"], answers) == 0 + try: + app = gv["app"] + iid = _fresh(app, "No accept") + v = {"project": "Help Desk", "subject": "No accept", "status": "Resolved"} + out = _run(gv, "set_status", v, lambda: app.save(iid, {"status": "Resolved"})) + assert not out.startswith("CONFIRMED"), out + finally: + assert _propose(gv["work"], gv["argv"], gv["answers"]) == 0 + + +def test_unnamed_unchecked_columns_refuse_the_kind(gv: dict[str, Any]) -> None: + """A new row's time stamp named nowhere: no contract, so the write stops.""" + answers = _offered(gv) + proposal = answers["bookkeeping"] + assert isinstance(proposal, dict) and proposal.get("unchecked"), proposal + answers["bookkeeping"] = {"accept": True} + try: + assert _propose(gv["work"], gv["argv"], answers) == 0 + report = json.loads((gv["out"] / "report.json").read_text()) + refused = " ".join(report["kinds"]["create_issue"]["refused"]) + assert "bookkeeping.unchecked" in refused, report["kinds"]["create_issue"] + app = gv["app"] + v = { + "project": "Help Desk", + "subject": "No contract", + "status": "New", + "parent": "VPN drops", + } + with pytest.raises(gen.GenerateError): + _run(gv, "create_issue", v, lambda: app.create_issue(v)) + finally: + assert _propose(gv["work"], gv["argv"], gv["answers"]) == 0 + + +# -- PostgreSQL read-only check ---------------------------------------------------- + + +def test_pg_read_only_check_counts_inherited_and_non_dml_privileges(pg: Any) -> None: + port = pg["port"] + admin = psycopg2.connect( + host="127.0.0.1", port=port, user="admin", dbname="postgres" + ) + admin.autocommit = True + admin.cursor().execute("CREATE DATABASE rotest") + admin.close() + a = psycopg2.connect(host="127.0.0.1", port=port, user="admin", dbname="rotest") + a.autocommit = True + a.cursor().execute( + "CREATE TABLE t (id serial primary key, v text); CREATE ROLE writers; " + "GRANT INSERT, UPDATE, DELETE ON t TO writers; " + "CREATE ROLE ro1 LOGIN; GRANT SELECT ON t TO ro1; GRANT writers TO ro1; " + "CREATE ROLE ro2 LOGIN; GRANT SELECT, TRIGGER ON t TO ro2; GRANT CREATE ON SCHEMA public TO ro2; " + "CREATE ROLE ro3 LOGIN; GRANT SELECT ON t TO ro3; GRANT SELECT ON t_id_seq TO ro3;" + ) # fmt: skip + a.close() + + def check(role: str) -> list[str]: + return gen.check_read_only( + lambda: psycopg2.connect( + host="127.0.0.1", port=port, user=role, dbname="rotest" + ), + "postgresql", + ) + + ro1 = check("ro1") + assert any("INSERT on table public.t" in p for p in ro1), ro1 + ro2 = check("ro2") + assert any("TRIGGER on table public.t" in p for p in ro2), ro2 + assert any("CREATE on schema public" in p for p in ro2), ro2 + assert check("ro3") == [] diff --git a/packages/flow/tests/test_generate_verify_contract_attacks_r3.py b/packages/flow/tests/test_generate_verify_contract_attacks_r3.py new file mode 100644 index 0000000000..8d54af31f0 --- /dev/null +++ b/packages/flow/tests/test_generate_verify_contract_attacks_r3.py @@ -0,0 +1,320 @@ +# ruff: noqa: F811 (pytest fixtures imported from the generator test, then used as arguments) +"""The adversarial review's round-3 attacks on generated contracts (#1672). + +The round-2 tracker with no version column, so updates read their versions +from the audit trail, on a throwaway PostgreSQL. The person's answers are +the proposal plus ``logs_add: [audits]``. Every wrong write must stop; the +correct writes beside them must be CONFIRMED. Also: the PostgreSQL read-only +check counts roles reachable with SET ROLE and column grants. Records are +made up. +""" + +from __future__ import annotations + +import json +import os +from typing import Any + +import pytest +import yaml + +psycopg2 = pytest.importorskip("psycopg2") + +from openadapt_flow.generate import verify_contract as gen # noqa: E402 +from tests.test_generate_verify_contract import BOT, _run, pg # noqa: E402,F401 +from tests.test_generate_verify_contract_attacks_r2 import ( # noqa: E402 + KINDS_V, + SCHEMA_V, + AppV, +) + + +class AppA(AppV): + """Logs each save under the given event code (a bulk edit, say).""" + + def save( + self, iid: int, sets: dict[str, Any], *, user: int = BOT, action: str = "update" + ) -> None: + for col, val in sets.items(): + if col == "status": + col, val = ( + "status_id", + self.one("SELECT id FROM statuses WHERE name=%s", (val,)), + ) + old = self.one(f"SELECT {col}::text FROM issues WHERE id=%s", (iid,)) + self.q(f"UPDATE issues SET {col}=%s WHERE id=%s", (val, iid)) + self.q( + "INSERT INTO audits (issue_id, user_id, action, field_name, old_value, new_value) " + "VALUES (%s,%s,%s,%s,%s,%s)", + (iid, user, action, col, old, None if val is None else f"v{val}"), + ) + self.q("UPDATE issues SET updated_on=now() WHERE id=%s", (iid,)) + self.q("UPDATE projects SET updated_on=now() WHERE id=(SELECT project_id FROM issues WHERE id=%s)", + (iid,)) # fmt: skip + + +@pytest.fixture(scope="module") +def ga(pg: Any) -> dict[str, Any]: + root, port = pg["root"], pg["port"] + admin = psycopg2.connect( + host="127.0.0.1", port=port, user="admin", dbname="postgres" + ) + admin.autocommit = True + admin.cursor().execute("CREATE DATABASE tracker3") + admin.close() + app = AppA(port, "tracker3") + with app.con() as c, c.cursor() as cur: + cur.execute( + "DO $$BEGIN IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname='oa_reader') " + "THEN CREATE ROLE oa_reader LOGIN PASSWORD 'reader-pw'; END IF; END$$" + ) + cur.execute(SCHEMA_V) + os.environ["OA_GEN_TEST_PW"] = "reader-pw" + work = root / "work3" + work.mkdir() + read = {"sql_driver": "psycopg2", + "sql_connect_args": {"host": "127.0.0.1", "port": port, "user": "oa_reader", "dbname": "tracker3"}, + "sql_password_env": "OA_GEN_TEST_PW"} # fmt: skip + (work / "read.yaml").write_text(yaml.safe_dump(read)) + (work / "kinds.yaml").write_text(yaml.safe_dump(KINDS_V, allow_unicode=True)) + common = ["--read", str(work / "read.yaml")] + + def demo(kind: str, values: dict[str, Any], write: Any) -> None: + assert gen.main(["snapshot", *common, "--out", str(work / "before.json")]) == 0 + write() + assert gen.main(["record", *common, "--before", str(work / "before.json"), "--kind", kind, + "--values", json.dumps(values), "--demos", str(work / "demos")]) == 0 # fmt: skip + + for k in KINDS_V["kinds"]: + for ex in k["examples"]: + v = ex["values"] + if k["kind"] == "create_issue": + demo("create_issue", v, lambda v=v: app.create_issue(v)) + else: + sets = {p: v[p] for p in k["parameters"][2:] if v.get(p) is not None} + demo( + k["kind"], + v, + lambda v=v, s=sets: app.save( + app.find(v["project"], v["subject"]), s + ), + ) + argv = ["propose", *common, "--kinds", str(work / "kinds.yaml"), "--demos", str(work / "demos"), + "--out", str(work / "out"), "--actor-param", f"bot_user_id={BOT}"] # fmt: skip + assert gen.main(argv) == 0 + report = json.loads((work / "out" / "report.json").read_text()) + answers = {q["key"]: q["proposal"] for q in report["questions"]} + bk = answers["bookkeeping"] + bk = {"accept": True} if bk == "accept" else dict(bk) + bk["logs_add"] = ["audits"] + answers["bookkeeping"] = bk + (work / "answers.yaml").write_text(yaml.safe_dump(answers)) + assert gen.main([*argv, "--answers", str(work / "answers.yaml")]) == 0 + report = json.loads((work / "out" / "report.json").read_text()) + return {"app": app, "out": work / "out", "report": report, "read": read, + "answers": answers} # fmt: skip + + +def _fresh(app: AppA, subject: str) -> int: + return app.create_issue( + {"project": "Help Desk", "subject": subject, "status": "New"} + ) + + +# -- SF1: free text every demonstration left empty stays empty ------------------- + + +def test_proposal_leaves_free_text_out(ga: dict[str, Any]) -> None: + unchecked = ga["answers"]["bookkeeping"].get("unchecked", {}) + assert "description" not in unchecked.get("issues", []), unchecked + + +def test_correct_create_confirmed(ga: dict[str, Any]) -> None: + app = ga["app"] + v = { + "project": "Help Desk", + "subject": "P0 child", + "status": "New", + "parent": "VPN drops", + } + out = _run(ga, "create_issue", v, lambda: app.create_issue(v)) + assert out.startswith("CONFIRMED"), out + + +def test_create_that_also_fills_an_empty_description_stops(ga: dict[str, Any]) -> None: + app = ga["app"] + v = { + "project": "Help Desk", + "subject": "A1 child", + "status": "New", + "parent": "VPN drops", + } + + def write() -> None: + iid = app.create_issue(v) + app.q( + "UPDATE issues SET description='Refund approved, wire to acct 99' WHERE id=%s", + (iid,), + ) + + out = _run(ga, "create_issue", v, write) + assert not out.startswith("CONFIRMED"), out + + +# -- SF6.1: the audit table's own key never names the record --------------------- + + +def test_versions_come_from_the_audit_trail_for_every_update_kind( + ga: dict[str, Any], +) -> None: + for kind in ("set_status", "edit_description"): + v = ga["report"]["kinds"][kind]["versions"] + assert v and v["table"] == "audits" and v["record_column"] == "issue_id", ( + kind, + v, + ) + + +def test_lost_update_of_another_field_stops(ga: dict[str, Any]) -> None: + app = ga["app"] + iid = _fresh(app, "AS1") + v = {"project": "Help Desk", "subject": "AS1", "status": "Resolved"} + + def write() -> None: + app.save(iid, {"due_date": "2026-12-31"}, user=2) + app.save(iid, {"status": "Resolved", "due_date": None}) + + out = _run(ga, "set_status", v, write) + assert not out.startswith("CONFIRMED"), out + + +def test_lost_update_of_the_same_field_stops(ga: dict[str, Any]) -> None: + app = ga["app"] + iid = _fresh(app, "AS2") + v = {"project": "Help Desk", "subject": "AS2", "status": "Resolved"} + + def write() -> None: + app.save(iid, {"status": "In Progress"}, user=2) + app.save(iid, {"status": "Resolved"}) + + out = _run(ga, "set_status", v, write) + assert not out.startswith("CONFIRMED"), out + + +# -- SF3: an event code no demonstration showed counts as a save ----------------- + + +def test_no_code_is_shown_to_be_a_read_so_nothing_is_filtered( + ga: dict[str, Any], +) -> None: + v = ga["report"]["kinds"]["edit_description"]["versions"] + assert v["where"] == {}, v + + +def test_correct_description_edit_confirmed(ga: dict[str, Any]) -> None: + app = ga["app"] + iid = _fresh(app, "A2 ctrl") + v = {"project": "Help Desk", "subject": "A2 ctrl", "description": "Fixed"} + out = _run( + ga, "edit_description", v, lambda: app.save(iid, {"description": "Fixed"}) + ) + assert out.startswith("CONFIRMED"), out + + +def test_lost_update_under_an_unseen_code_stops(ga: dict[str, Any]) -> None: + app = ga["app"] + iid = _fresh(app, "A2C1") + v = {"project": "Help Desk", "subject": "A2C1", "description": "Fixed"} + + def write() -> None: + app.save(iid, {"due_date": "2026-12-31"}, user=2, action="bulk_update") + app.save(iid, {"description": "Fixed", "due_date": None}) + + out = _run(ga, "edit_description", v, write) + assert not out.startswith("CONFIRMED"), out + + +def test_same_field_lost_update_under_an_unseen_code_stops(ga: dict[str, Any]) -> None: + app = ga["app"] + iid = _fresh(app, "A2C2") + v = {"project": "Help Desk", "subject": "A2C2", "description": "Fixed"} + + def write() -> None: + app.save(iid, {"description": "Theirs"}, user=2, action="bulk_update") + app.save(iid, {"description": "Fixed"}) + + out = _run(ga, "edit_description", v, write) + assert not out.startswith("CONFIRMED"), out + + +def test_versions_filter_drops_only_a_code_shown_to_be_a_read( + ga: dict[str, Any], +) -> None: + """A read code is one the demonstrations logged on a record they didn't + change; the read keeps NULL codes and every other code.""" + frame = gen._versions_frame( + gen.SqlBuilder(_schema(ga)), + _plan_with_where({"action": {"not_in": ["read"]}}), + _settings(ga), + ) + assert frame is not None + sql = frame["read"]["sql_query"] + assert "IS NULL OR" in sql and "NOT IN ('read')" in sql, sql + + +def _schema(ga: dict[str, Any]) -> Any: + from openadapt_flow.generate import sql_store as store + + return store.read_schema(store.connector(ga["read"]), "postgresql") + + +def _plan_with_where(where: dict[str, Any]) -> Any: + plan = gen.KindPlan("edit_description") + plan.anchor = "issues" + plan.versions = {"kind": "audit", "table": "audits", "record_column": "issue_id", + "author_column": "user_id", "actor_param": "bot_user_id", "where": where} # fmt: skip + return plan + + +def _settings(ga: dict[str, Any]) -> Any: + return gen.Settings( + dict(ga["read"]), {"bot_user_id": str(BOT)}, 300, "test-environment" + ) + + +# -- SF4: PostgreSQL roles reachable with SET ROLE, and column grants ------------- + + +def test_pg_read_only_check_counts_set_role_and_column_grants(pg: Any) -> None: + port = pg["port"] + admin = psycopg2.connect( + host="127.0.0.1", port=port, user="admin", dbname="postgres" + ) + admin.autocommit = True + admin.cursor().execute("CREATE DATABASE rotest3") + admin.close() + a = psycopg2.connect(host="127.0.0.1", port=port, user="admin", dbname="rotest3") + a.autocommit = True + a.cursor().execute( + "REVOKE CREATE ON SCHEMA public FROM PUBLIC; CREATE TABLE t (id serial primary key, v text); " + "CREATE ROLE w3; GRANT INSERT, UPDATE ON t TO w3; " + "CREATE ROLE ro3 LOGIN NOINHERIT; GRANT SELECT ON t TO ro3; GRANT w3 TO ro3; " + "CREATE ROLE ro4 LOGIN; GRANT SELECT ON t TO ro4; GRANT UPDATE (v) ON t TO ro4; " + "CREATE ROLE ro6 LOGIN; GRANT SELECT ON t TO ro6; " + "REVOKE CREATE ON DATABASE rotest3 FROM PUBLIC; REVOKE TEMP ON DATABASE rotest3 FROM PUBLIC" + ) # fmt: skip + a.close() + + def check(role: str) -> list[str]: + return gen.check_read_only( + lambda: psycopg2.connect( + host="127.0.0.1", port=port, user=role, dbname="rotest3" + ), + "postgresql", + ) + + ro3 = check("ro3") + assert any("(as w3) has INSERT on table public.t" in p for p in ro3), ro3 + ro4 = check("ro4") + assert any("has UPDATE on table public.t" in p for p in ro4), ro4 + assert check("ro6") == []