[abrichr]

Stacked on #1678 (shadow mode). Large: about 2,700 lines of its own, 1,230 of them tests.

The problem

Many apps write to their own database on every request: the acting user's last_login, an API token's last_seen, a first-login property row for the acting user. The whole-database frame sees those rows move inside every write's window, nothing in the contract explains them, and it refutes every write, correct or not. In the held-out retest, three of four apps had every correct write refuted this way.

The change: learn, then enforce

openadapt-flow learn (also python -m openadapt_flow.learn and openadapt learn):

• collect: runs shadow mode (SQL) with a hook on Shadow.on_checked. Each verifier call adds one line to episodes.jsonl: the kind, the outcome, which parts held, and the frame's moved rows by table and column name. No record values.
• propose: writes proposal.yaml and a readable proposal.md with a digest.
• approve: one command by a person writes the excuses into the contracts as frame.learned, recording the proposal's digest, who approved and when. All contracts of a kind or none. An edited proposal is refused.

Fail closed at every step:

1. Only checks whose declared effects held and whose frame was refuted by database rows alone teach. Every other check is ignored and counted by reason in the proposal.
2. A moved row can be excused only as target (a row the write declares or adds) or actor_row (the users table's row keyed by the actor's id, or a row whose user column holds it; the user columns come from the schema's foreign keys or a person's --user-column). Any other row is listed as "stays a stop", however often it moves.
3. An excuse is {table, columns, change: changed|added, relation, actor_column}. Exactly the columns seen to move. Never a whole table, a wildcard, any row, or a removal. Proposed only when seen in at least K of every N learnable checks of that kind (defaults 3 of 5; each column must meet the same bar). Why: five is the smallest sample in which a majority still allows two checks where a throttled column (a last-used time written at most once a minute) didn't move, and a one-off isn't proposed. The count isn't the safety boundary; the relation is.
4. Enforcement (frame.learned in runtime/effects/learned.py and frame.py): at each check and recheck the relation is derived again from this run's actor and declared rows. A target excuse applies only to rows this write declares; an actor_row excuse only when the actor column holds this run's actor id before and after; the moved columns must all be in the excuse. A row in a table over max_rows, whose baseline didn't keep every column, is never excused. Anything else is collateral, as before. The receipt names the excuses used and the proposal digest. Contracts without the key keep their digests.

The verifier now also reports, for a refuted frame, each collateral row by names only (evidence.frame.moves: table, change, columns, whether it's the write's own row, which columns hold the actor's id). The learner reads only that.

Tests

tests/test_learned_excuses.py, on SQLite, and on MariaDB and PostgreSQL with OPENADAPT_SNAPSHOT_MYSQL / OPENADAPT_SNAPSHOT_POSTGRES:

• before/after on a real write: refuted by the bookkeeping, confirmed once learned;
• a learned actor_row excuse never covers another user's row or token, another column of the user's row, a row moved to the actor, a removal, a row added for another user, an added row of a kind only changes were learned for, a column a large table didn't compare, or the same moves under another run's actor;
• a learned target excuse never covers a look-alike row, another column of the target, or the excused column on a second row (the retest's fail-open, where a column excused for the target was excused on every row);
• learn, propose and approve end to end on real writes, user columns from the live schema's foreign keys on MariaDB and PostgreSQL;
• the learner's rules: rows that are neither target nor actor stay a stop when seen in every check, K of N, columns under the bar left out, every ignored check counted, an edited proposal refused, approval all-or-nothing with --reject;
• collecting through the real shadow proxy's hook;
• the CLI.

A mutation check (scratch, not committed): letting any row count as the actor's fails 6 tests; letting any column count fails 2; letting a row count as the target fails 2.

Real-app evidence

Running.

Docs

packages/flow/docs/LEARNED_EXCUSES.md (new), pointers in VERIFY_RUN.md and SHADOW_MODE.md, the command map and CAPABILITIES.

Note on the diff

This branch merges main into #1678's branch to get the PostgreSQL frame (#1673), so until #1678 is updated from main the diff against it also shows #1672 and #1673.