[abrichr]

What changed

A verified write can now be checked again later. Before this change, a write that the app made after the verdict went unnoticed.

• frame.database.recheck_after_s is a new optional setting with no default. When it's unset, behavior and every contract digest stay the same. When it's set and the run is verified:
  • The receipt has recheck_due_at, in whole-second UTC. It is never early: the post-save read time is truncated to the second, so one second is added before rounding up. A verified receipt that carries it is provisional until the recheck runs.
  • evidence.frame.recheck names what the recheck compares and binds a salted digest of the settled post-save state.
  • WriteResult.recheck holds that state privately. It contains the post-save snapshot as keyed MACs, the audit entries seen then as keyed hashes, the run's bound baseline, what the run's frame reported as collateral (row key MACs and entry hashes) and the salt. As with the baseline since verify-run: the baseline holds the read set as keyed MACs, not values #1599, no values are kept. RecheckState.save writes the state with mode 0600.
  • verify_write records each state it issues under the verifier's state directory (rechecks/, named by the state's digest). A state's salt and digests can be recomputed by whoever writes the file, so recheck uses only a state with that record. An unissued state being rechecked gives UNCERTAIN; an unissued excuse raises VerifierError.
• openadapt_flow.verifier.recheck(contract, write, *, params, actor, excused_by=(), wait=False) -> RecheckResult reads the whole database at or after the due time. It reads twice, settle_s apart, as after the save. It judges every change since the post-save read with the frame's own rules: profile bookkeeping, audit attribution of other actors, and declared side records. The outcome is CONFIRMED, REFUTED (late) or UNCERTAIN. status maps these to verified, refuted and unconfirmed.
• excused_by takes later writes by the same actor under the same contract, of any outcome. A later write accounts only for its own window, the time between its baseline and its post-save read, where its own frame judged every change:
  • A row that changed in that window and still holds what its post-save read saw is the later write's. The match is by key and contents MAC under that write's key. A row that matches its selector but lands after its window is judged like any other, and so is a row changed again since. This covers a delayed duplicate and a save that landed late.
  • A removal counts when that window recorded it, matched by a per-host row link MAC that is kept only when recheck_after_s is set. Tables without a primary key are never excused this way.
  • When the later write was not verified and the change is one its frame reported as collateral, or its delta failed, the change may be what refuted it, and the harm may be the earlier write's. The recheck then gives UNCERTAIN, not CONFIRMED.
  • Anything else unexplained refutes the earlier write.
• Fail-closed:
  • A database that can't be read or is still moving gives UNCERTAIN.
  • So does a state that doesn't match its receipt, the contract, the actor or the params. The salt commitment, the baseline digest, the post-state digest, the frame digest, the executor ref and each salted parameter digest are all checked.
  • An excuse that doesn't qualify raises VerifierError. Examples are another actor, a write that started before the post-save read, or a loaded state without its params.
• A recheck before its due time raises RecheckNotDue (with due_at and wait_s). With wait=True, it sleeps until the due time instead.

Code layout: the state's shape, the due time and the later-write window (LaterWindow) are in a new module, runtime/effects/late_write.py, next to the frame. The read and the judgment reuse judge_run_frame in verify_run.py, which gets a database_only mode and a later hook. db_snapshot.py can keep row link MACs. verifier.py adds only a thin entry point.

The second commit fixes a fail-open the review found. In the first version, a later write's declared effects were excused with no time bound, so a duplicate matching its selector, or a save of its that landed after its window, passed the earlier write's recheck. Probes P1 to P4 from the review are now regression tests.

Docs: docs/VERIFIER_API.md has a new "Writes that land after the result" section and the narrowed scope line; docs/VERIFY_RUN.md has a new "Writes that land later" section, frame.py documents the field next to settle_s, and CHANGELOG and CAPABILITIES.md are updated.

• frame.database.settle_attempts is a new setting (default 1, as before). It lets the reads after a save, and at a recheck, retry up to that many times, settle_s apart, until two reads in a row match. A live OpenEMR moves on its own, for example when a UI session polls, so a single pair of reads often differed and the recheck came back UNCERTAIN.
  • Two reads also match when they differ only by rows added to append-only tables, which the frame excuses anyway. An edited row of such a table, or any other difference, still counts as movement.
  • The earlier read of the matching pair is the one judged, and evidence.frame.database.settle_attempts records how many reads it took.
  • When no pair matches within settle_attempts * settle_s, the frame is not checked, as before.

Why

verify_write and the frame check a write once: two whole-database reads settle_s apart, right after the save. A collateral write that lands later is missed, such as a background job, a trigger-driven integration or a delayed duplicate side record. A VERIFIED write can turn out wrong a minute later.

On top of the one verifier API (#1643)

#1643 merged the standalone verifier API from #1628. This branch merges main and ports the recheck onto it:

• WriteResult (CONFIRMED/REFUTED/UNCERTAIN) carries recheck and recheck_due_at, and to_dict() includes recheck_due_at.
• verify_write(pending) saves the state as recheck.json in the write's folder, beside baseline.json and effect-receipt.json, and records it as issued under the verifier's state directory. That directory follows the same private-directory rules as snapshot issuance. It is kept apart from the ledger's own directory, so ledger=None and memory ledgers work too.
• The deprecated capture/verify_write(contract, before, ...) form schedules a recheck the same way.
• When a recheck is due, scope.not_covered names its due time in place of the generic delayed-writes line, and says the result is provisional until the recheck runs.
• The recheck tests now use begin_write/verify_write. One test covers the deprecated form.
• Reconciling an attempt (_reconcile calls _judge) also writes a recheck state for that run.

Behavior is otherwise what was reviewed: a later write excuses only its own window, a possibly-refuting change gives UNCERTAIN, issuance is required, the due time is never early, and removals match by link MAC. The only change made by porting is the vocabulary: reason text now says "the earlier CONFIRMED" instead of "the earlier VERIFIED".

How it was tested

• tests/test_verifier_recheck.py has 51 focused tests on SQLite through the public API:
  • due time in the receipt, the evidence and the state, which is 0600 and holds no values;
  • nothing changes when the setting is unset, including digest stability;
  • an unverified write is refused;
  • a recheck before the due time is refused, and wait sleeps until it;
  • late collateral of 5 kinds refutes;
  • a later verified write's own rows are excused, but only its own;
  • P1: a duplicate matching a later write's selector after its window refutes;
  • P2: a later write whose save landed after its window excuses nothing;
  • P3: the earlier write's harm inside a refuted later write's window gives UNCERTAIN;
  • P4: a removal that a refuted later window reported gives UNCERTAIN, while a removal a verified later window explained is excused and a removal no window recorded refutes;
  • what refuted a later write leaves the earlier one UNCERTAIN, while a new change still refutes;
  • a state not issued under the state directory gives UNCERTAIN, and as an excuse it raises;
  • the due time is never early;
  • a recheck that settles on a later read gives the normal verdict, one that never settles gives UNCERTAIN, and a pair that differs only by appended log rows counts as settled;
  • a row changed again after the later write is not excused;
  • another actor's attributed write is excused, and the bot's own unattributed write is not;
  • bookkeeping is excused;
  • an unreadable database and a still-moving database each give UNCERTAIN;
  • the saved state round-trips;
  • 6 kinds of tampered state give UNCERTAIN, as do another contract, another actor or other params;
  • an excuse that started early or came from another actor is refused.
• These existing test files pass: test_verifier_api.py (public API pins updated), test_write_ledger.py, test_verifier_snapshot_api.py, test_verify_write.py, the launcher's tests/test_verify_command.py, test_durable_authority_v13.py, test_frame_database.py, test_verify_run.py, test_frame_conditions.py, test_app_profile.py, test_frame_links.py, test_frame_secret.py, test_guarded_save.py, test_fault_gate.py, test_jit_reuse_binding.py and test_jit_repair_admission.py.
• ruff check and ruff format pass on the changed files. mypy (CI config) reports no errors in the touched modules; the remaining errors are in other modules and were already there.

What the live test needs

tests/test_late_write_recheck_live_openemr.py is skipped unless OPENEMR_LATE_WRITE_LIVE=1. It has not been run yet. It needs:

• the local OpenEMR fixture up and bootstrapped;
• a DB-API driver for MariaDB (pymysql by default);
• a read-only database account the test can reach, with SELECT on openemr and TRIGGER.

It has two tests:

1. It saves a patient through the REST API and verifies it. Then it changes another chart with direct SQL. The recheck must give REFUTED.
2. A clean control: after the first save, only a second verified save happens. The recheck, with that save as excused_by, must give CONFIRMED.

OPENEMR_LATE_WRITE_LIVE=1 OPENEMR_DB_USER=<read-only user> OPENEMR_DB_PASSWORD=<password> \
OPENEMR_DB_HOST=127.0.0.1 OPENEMR_DB_PORT=<port> PYTHONPATH=../types:. \
  python -m pytest tests/test_late_write_recheck_live_openemr.py -v -p no:cacheprovider

The test's new_row_fields for OpenEMR's REST patient insert is a starting guess. If the first save isn't verified, the assertion prints the frame evidence. OPENEMR_LATE_WRITE_FRAME_JSON can add fields without editing the test. The test's recheck_after_s (2 s, or OPENEMR_LATE_WRITE_RECHECK_S) is a test value only. No app gets a recommended value.