Skip to content

benchmark(lifecycle): the scripted person signs its approvals with a fixture-only key - #1565

Merged
abrichr merged 3 commits into
mainfrom
bench/soak-signed-approvals
Oct 25, 2026
Merged

abrichr merged 3 commits into
mainfrom
bench/soak-signed-approvals

Conversation

@abrichr

@abrichr abrichr commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

This PR makes the lifecycle soak's scripted approvals count the way a person's do: each one is signed by a fixture-only approver key that belongs to the run. It follows OpenAdaptAI/OpenAdapt#1513, which has merged.

Why

Since #1486, a person's approval counts only when a key on the signed approver roster signs it. The harness still wrote unsigned answers that claimed a cloud_aal2 sign-in. In the final soak (#1513, production-2026-10-25-final), the one approval request, on item 26 after the relabel repair, was refused (auth_method_unverified). The gate stopped before saving (gate_error), and the item was entered by hand.

What changes

  • Per-run approver (benchmark/lifecycle_proof/approver.py):
    • a throwaway Ed25519 key, generated under the run directory (approver-keys/) with a random passphrase kept beside it, owner-only;
    • a one-entry roster that the key roots;
    • a deployment.yaml that pins approvers.root_public_key and the key directory. It is read only after its bytes match the recorded SHA-256 (pin_deployment_bytes, the same check as --config-sha256). The run gate uses the same config.
  • Pinning: every approval store the proof or the soak creates takes its pin only from that config (config_only=True), never from OPENADAPT_APPROVER_ROOT.
  • Signing: ScriptedPerson (the proof) and OperatorPerson (the soak) sign every answer with that key. If an answer is refused, the ledger records person_answer_refused with its code before the gate stops. Before, the item just ended as a bare gate_error.
  • Ledger: a fixture_approver row with the public root and the config digest only. The key and passphrase never leave the run directory, which is never committed.

No real person's key is used, and no product code changes.

Tests

tests/test_lifecycle_fixture_approver.py:

  • the old unsigned scripted answer is refused (auth_method_unverified), and so is an unsigned answer with no sign-in claim;
  • the fixture approver's signed answer passes answer_refusal against the config-pinned store, and the store reports it approved;
  • ScriptedPerson signs; with no approver, it logs person_answer_refused and the request stays waiting;
  • an edited deployment.yaml is refused;
  • reopening the run gets the same root and digest. The ledger fields carry no key material, and the passphrase file is owner-only.

Locally: those tests, test_lifecycle_benchmark.py and test_human_approval_verified.py (31 passed); ruff, ruff format and mypy are clean.

Not covered

  • No real-app run yet. No soak has run with this change, and these tests don't drive OpenEMR. The next soak's admission and first approval will exercise it.
  • Self-repair approvals. SelfRepair opens its own repair-approval store without a pin (openadapt_flow/repair/self_repair.py). A self-repair that needs a person's approval would still be refused, failing closed. The final soak never reached that path. Fixing it means SelfRepair taking a pinned store, which is product code and out of scope here.

🤖 Generated with Claude Code

abrichr and others added 2 commits October 25, 2026 10:00
…fixture-only key

Since #1486 a person's approval counts only when a key on the signed
approver roster signs it. The soak's scripted person still wrote unsigned
answers claiming a cloud sign-in, so the final soak's one approval was
refused (auth_method_unverified) and the gate stopped before saving.

Each run now gets its own approver: a throwaway key generated under the
run directory, a one-entry roster it roots, and a deployment config that
pins the root (approvers.root_public_key) and key directory, read only when
its bytes match their SHA-256. Approval stores take their pin only from
that config, never from OPENADAPT_APPROVER_ROOT. The ledger records the
public root and the config digest; the key and passphrase stay in the run
directory. A refused answer is logged with its code before it stops the
gate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@abrichr
abrichr enabled auto-merge October 25, 2026 15:32
@abrichr
abrichr merged commit 00de155 into main Oct 25, 2026
37 checks passed
@abrichr
abrichr deleted the bench/soak-signed-approvals branch October 25, 2026 16:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant