diff --git a/docs/build-plan.md b/docs/build-plan.md index deb888b..9ff621e 100644 --- a/docs/build-plan.md +++ b/docs/build-plan.md @@ -244,7 +244,8 @@ Scope (owner, 2026-11-05, spec P6-5): the review targets are the 4 pilot repos; - [ ] Keep the supported repo inventory current. A repo joins or leaves through a PR to this repo. A new target: immutable subject first, a test PR right after, then a run on each open PR. - [x] Owner: Phase 6 also applies the org security configurations. Done and read back 2026-11-05: the configuration for public repos is enforced on the active public repos and is the default for new public repos; the dependency configuration is enforced and is the default for new private and internal repos. One planned change to the dependency configuration is skipped by owner decision (spec P6-5). Details are kept in the owner's records. - [x] Eval environment split, code part (spec P6-4): `eval.yml` uses `eval` on `main` and `eval-test` on any other ref, chosen by an exact, case-sensitive shell test in a select job with no permissions, whose one output also sets `EVAL_ENVIRONMENT` (an Actions expression compares without case, so `MAIN` would have chosen `eval`); `evals.py` holds the federation IDs per environment and fails closed on another value and on an environment whose rule is not set; the workflow unit test allows the two names only in `eval.yml`. -- [ ] Owner: create the `eval-test` environment on `.github` (required reviewer the owner, "prevent self-review" on, admin bypass as `eval`), and its federation rule on the test workspace (as the eval rule, with the exact claim `environment: "eval-test"`). Then set the rule ID in `evals.py` in a PR, and test it as V27: an approved `eval-test` run on a branch gets a token; a run with no environment, or in another environment, gets none. Until then branch evals fail closed ("eval-test federation rule not configured"). +- [x] Owner: create the `eval-test` federation rule on the test workspace: `eval-test-federation-rule`, created 2026-11-06 as a copy of the `eval` rule (same issuer, subject prefix, owner, repository and event claims, scope `workspace:developer`, service account, workspace and 600 s lifetime) with the exact claim `environment: "eval-test"`. Its ID is set in `evals.py` (`fdrl_01WkhnTsBLVnYBHgAfxkugnc`). +- [ ] Test the `eval-test` rule as V27: an approved `eval-test` run on a branch gets a token; a run with no environment, or in another environment, gets none. The fail-closed path ("eval-test federation rule not configured") stays in the code for an unset entry, and a unit test covers it. - [x] Release item: summary comment format (spec P6-1): one row per cause, the repo setting label for a Dependency graph that is off, plain paths, no empty token table, and the informational unsigned-commit row with the signing guide link. Display only; the gate does not change. Shipped in 364058d (PR #26), released 2026-11-04. - [ ] Release item: comment fixes from the first live run (spec P6-3): the TIP callout names the signing step when the unsigned-commit row shows, "commit" for a one-commit PR, no setting label on a negated message, and an error body cut off mid-read is treated as empty. Display only; the gate does not change. The same release carries the harden-runner v2.22.3 and setup-uv v10.3.3 pins (audit mode stays; `releases.astral.sh` added to the setup-uv jobs' allowlists). Ships as in "Each release". diff --git a/docs/claude-review-design.md b/docs/claude-review-design.md index 91fed6e..55b8e1d 100644 --- a/docs/claude-review-design.md +++ b/docs/claude-review-design.md @@ -690,7 +690,7 @@ token lifetime: 10 minutes History: phases 3 and 4 used a two-rule flow (a new rule R2 per release at the placeholder SHA, its ID in the candidate, two Console edits, then delete R1; V26, P0-15). Every phase 4 release instead edited R1 in place at the merge SHA (cc710fb, cf19216, 96f4966), and P4-9 makes that the spec. The trade-off: no overlap window in which runs at the old SHA still get tokens, and a rollback is one more rule edit instead of a switch back to the old rule. - **Only analyze can mint a token.** Collect and publish have no `id-token` permission. This is how "the GitHub-writing job has no Anthropic credential" holds, because the federation rule cannot tell jobs in the same workflow apart. -- **Fixture eval credential.** `eval.yml` runs on `workflow_dispatch`, so the review federation rule rejects its token. Evals use their own federation rules, one per eval environment (P6-4): the evals job uses the environment `eval` for a run on `refs/heads/main` and `eval-test` for any other ref, and sets `EVAL_ENVIRONMENT` to the same value. A select job with no permissions makes the choice with an exact shell test (`[ "$GITHUB_REF" = "refs/heads/main" ]`) and exports it as its one output; the evals job takes that output (`${{ needs.select-environment.outputs.environment }}`) for both. The choice is not an Actions expression: expressions compare strings without case, so `github.ref == 'refs/heads/main'` is also true for a branch named `MAIN` or `Main` (review AC, I1). If the select job fails, the evals job does not run. `evals.py` holds the IDs per environment (separate from the production IDs in `analyze.py`): the same organization; for each environment its own rule and a service account (`eval-test` uses the `eval` service account until the owner chooses another). The runner accepts only `eval` and `eval-test` in `EVAL_ENVIRONMENT`; any other value, or an environment whose rule ID is not set, fails closed before any token is minted ("eval-test federation rule not configured" until the owner creates that rule). The `eval` rule is described below; the `eval-test` rule is the same with the exact claim `environment: "eval-test"`. It is bound to the test workspace `claude-review-p0-test`, which is not production and has its own spend limit (P2-1). It uses pattern mode, the subject prefix `repo:OperationCode@10479364/*`, the scope `workspace:developer` and the test workspace only. The scratch rules of phase 3 (V26 c) are on the same workspace, so evals and those tests share its spend limit: when it is spent, evals and those tests stop, and production reviews do not. Exact claims: `repository_owner_id`, `repository_id` of `.github`, `event_name: "workflow_dispatch"` and `environment: "eval"` ([OIDC](https://docs.github.com/en/actions/reference/security/oidc)). In a dispatch run `job_workflow_ref` ends in the branch ref (V3), and evals run on any branch, so an exact claim cannot name `eval.yml`. V27 passed on 2026-10-27 (UTC), on the throwaway branch `v27-probe`, which is never merged. P1 (approved `workflow_dispatch` run, job in environment `eval`): token issued (HTTP 200, req_011CgS2vx6h7nwH6wi3dAFNz, 03:30:02Z). N1 (no environment): 401 (req_011CgS2ticG9pbbKTffkfMeK, 03:29:32Z). N2 (approval rejected): the job never started and made no exchange. N3 (`push` event, approved): 401 (req_011CgS2r8BXRmeR8EPtPd9si, 03:28:57Z). N4 (`push` in claude-review-test, no environment): 401 (req_011CgS34fZVr9eZEVx8z8xZw, 03:31:47Z); the job has no environment and ran on `push`, so the missing environment or the `push` event alone explains the 401, and the `repository_id` check is not tested separately. The dispatch run with its own `eval` environment in claude-review-test was not run. U1 (unit tests, `test_workflows.py -k eval`, on a clean tree): pass. An independent check of the run logs agreed with these results and found no secret. Every deny returns the same 401 body, so the owner read the reasons in the Console federation authentication history on 2026-10-27 (all rows: issuer `github-actions`, rule `eval-federation-rule`, service account `claude-review-p0`; times in UTC). 03:27:59Z Failure `match_claim_value_mismatch` (the no-environment job in the push run). 03:28:57Z Failure `match_claim_value_mismatch` (N3, `push` event). 03:29:32Z Failure `match_claim_absent` (N1, no environment). 03:30:03Z Success (P1, approved dispatch; the probe log has 03:30:02Z, so the two clocks differ by one second). 03:30:34Z Failure `match_claim_absent` (the no-environment job in the rejected-approval run; the Console showed no row for N2 itself, the rejected job, as expected). 03:31:47Z Failure `match_claim_value_mismatch` (N4, other repo). The list view does not name the claim and does not show the scope of the P1 token: claim name and P1 scope not shown in the list view. The next sentence is inference from the probe token claims, not read from the Console: each failure code fits the expected cause, an absent `environment` claim or a mismatched `event_name` or `repository` claim, but the list view names no claim. The no-environment job in the push run got `match_claim_value_mismatch`, not `match_claim_absent`; this fits the `event_name` check failing first (also inference). +- **Fixture eval credential.** `eval.yml` runs on `workflow_dispatch`, so the review federation rule rejects its token. Evals use their own federation rules, one per eval environment (P6-4): the evals job uses the environment `eval` for a run on `refs/heads/main` and `eval-test` for any other ref, and sets `EVAL_ENVIRONMENT` to the same value. A select job with no permissions makes the choice with an exact shell test (`[ "$GITHUB_REF" = "refs/heads/main" ]`) and exports it as its one output; the evals job takes that output (`${{ needs.select-environment.outputs.environment }}`) for both. The choice is not an Actions expression: expressions compare strings without case, so `github.ref == 'refs/heads/main'` is also true for a branch named `MAIN` or `Main` (review AC, I1). If the select job fails, the evals job does not run. `evals.py` holds the IDs per environment (separate from the production IDs in `analyze.py`): the same organization; for each environment its own rule and a service account (`eval-test` uses the `eval` service account until the owner chooses another). The runner accepts only `eval` and `eval-test` in `EVAL_ENVIRONMENT`; any other value, or an environment whose rule ID is not set, fails closed before any token is minted ("eval-test federation rule not configured"; no entry is unset now, and a unit test unsets one to cover the path). The `eval` rule is described below; the `eval-test` rule, `eval-test-federation-rule` (created by the owner on 2026-11-06, ID in `evals.py`), is a copy of it with the exact claim `environment: "eval-test"`. It is bound to the test workspace `claude-review-p0-test`, which is not production and has its own spend limit (P2-1). It uses pattern mode, the subject prefix `repo:OperationCode@10479364/*`, the scope `workspace:developer` and the test workspace only. The scratch rules of phase 3 (V26 c) are on the same workspace, so evals and those tests share its spend limit: when it is spent, evals and those tests stop, and production reviews do not. Exact claims: `repository_owner_id`, `repository_id` of `.github`, `event_name: "workflow_dispatch"` and `environment: "eval"` ([OIDC](https://docs.github.com/en/actions/reference/security/oidc)). In a dispatch run `job_workflow_ref` ends in the branch ref (V3), and evals run on any branch, so an exact claim cannot name `eval.yml`. V27 passed on 2026-10-27 (UTC), on the throwaway branch `v27-probe`, which is never merged. P1 (approved `workflow_dispatch` run, job in environment `eval`): token issued (HTTP 200, req_011CgS2vx6h7nwH6wi3dAFNz, 03:30:02Z). N1 (no environment): 401 (req_011CgS2ticG9pbbKTffkfMeK, 03:29:32Z). N2 (approval rejected): the job never started and made no exchange. N3 (`push` event, approved): 401 (req_011CgS2r8BXRmeR8EPtPd9si, 03:28:57Z). N4 (`push` in claude-review-test, no environment): 401 (req_011CgS34fZVr9eZEVx8z8xZw, 03:31:47Z); the job has no environment and ran on `push`, so the missing environment or the `push` event alone explains the 401, and the `repository_id` check is not tested separately. The dispatch run with its own `eval` environment in claude-review-test was not run. U1 (unit tests, `test_workflows.py -k eval`, on a clean tree): pass. An independent check of the run logs agreed with these results and found no secret. Every deny returns the same 401 body, so the owner read the reasons in the Console federation authentication history on 2026-10-27 (all rows: issuer `github-actions`, rule `eval-federation-rule`, service account `claude-review-p0`; times in UTC). 03:27:59Z Failure `match_claim_value_mismatch` (the no-environment job in the push run). 03:28:57Z Failure `match_claim_value_mismatch` (N3, `push` event). 03:29:32Z Failure `match_claim_absent` (N1, no environment). 03:30:03Z Success (P1, approved dispatch; the probe log has 03:30:02Z, so the two clocks differ by one second). 03:30:34Z Failure `match_claim_absent` (the no-environment job in the rejected-approval run; the Console showed no row for N2 itself, the rejected job, as expected). 03:31:47Z Failure `match_claim_value_mismatch` (N4, other repo). The list view does not name the claim and does not show the scope of the P1 token: claim name and P1 scope not shown in the list view. The next sentence is inference from the probe token claims, not read from the Console: each failure code fits the expected cause, an absent `environment` claim or a mismatched `event_name` or `repository` claim, but the list view names no claim. The no-environment job in the push run got `match_claim_value_mismatch`, not `match_claim_absent`; this fits the `event_name` check failing first (also inference). - **Claim boundary.** Each eval rule accepts a dispatch run of any workflow file in `.github` whose job uses its environment, not only `eval.yml`. That only `eval.yml` uses `eval` and `eval-test` is a repo convention; the rules cannot enforce it. Three controls hold the credential separation: the `eval` branch policy (only `main`, which holds reviewed code, can deploy to it), the owner's approval of each `eval-test` run, and a workflow unit test. The unit test reads every workflow file in `.github` and fails if a workflow other than `eval.yml` uses `eval` or `eval-test`, if `eval.yml` lacks the select job (no permissions, no environment, the one output, the fixed exact shell test), if a job in `eval.yml` with an environment does not need that job, uses anything but its output, or sets `EVAL_ENVIRONMENT` to anything else, or if such a job sets no explicit job-level `permissions` or has any `write` permission other than `id-token: write`. Environment names are not case sensitive, so the test treats `EVAL` as `eval`, and in any other workflow an environment that is not a literal string (an expression) counts as a use. Job-level `permissions` is required on every job that uses `eval`, even when the workflow level sets it, and it holds exactly `contents: read` and `id-token: write`. `eval.yml` triggers on `workflow_dispatch` only. The test handles the string forms: `write-all` at either level fails the test, and `read-all` or a missing job-level block also fails it, because the job must state `contents: read` and `id-token: write` itself. The eval job has `contents: read` and `id-token: write` only, so the job that holds the Anthropic credential has no GitHub write permission. - **Approval (P6-4).** Two environments, chosen by the ref: @@ -875,7 +875,7 @@ The tasks, owner actions and exit criteria for each step are in [docs/build-plan - [ ] Protect `.github` main: CODEOWNERS review, 1 approval, no force push, no deletion. CODEOWNERS is the Contributors team, which needs write access to the repo. Org admins may bypass through a pull request only. - [ ] Also on `.github` main: dismiss stale approvals on push, and require the `test.yml` check (the job `unit-tests`). - [ ] Anthropic Console: dedicated workspace with a hard monthly spend limit, service account, federation rule as in "Auth and cost" (pattern mode, ID-form subject prefix, scope `workspace:developer`, the review workspace only, the placeholder SHA until phase 3; V26). Done 2026-10-27: workspace `claude-review`, service account `claude-review`, rule R1 `claude-review-prod-r1` created at the placeholder SHA and accepted by the Console (V26 a partial); no read-back is recorded, so this item stays open. -- [ ] Anthropic Console, eval part: the eval federation rule (V27) on the test workspace `claude-review-p0-test`, separate from the review workspace (P2-1). GitHub environments on `.github` (P6-4): `eval` with a main-only branch policy and no required reviewer (since 2026-11-04); `eval-test` with the owner as required reviewer and "prevent self-review" on, for every other ref. [ ] An `eval-test` federation rule like the eval rule, with the exact claim `environment: "eval-test"`, and its ID in `evals.py`. No API key. +- [ ] Anthropic Console, eval part: the eval federation rule (V27) on the test workspace `claude-review-p0-test`, separate from the review workspace (P2-1). GitHub environments on `.github` (P6-4): `eval` with a main-only branch policy and no required reviewer (since 2026-11-04); `eval-test` with the owner as required reviewer and "prevent self-review" on, for every other ref. [x] The `eval-test` federation rule `eval-test-federation-rule` (2026-11-06), like the eval rule, with the exact claim `environment: "eval-test"`, and its ID in `evals.py`. [ ] Test it as V27. No API key. - [ ] Org Actions setting: default `GITHUB_TOKEN` permissions read-only. - [ ] Org setting: require 2FA for all members. A stolen maintainer account bypasses everything in this design. - [ ] Enable Dependabot alerts with npm malware detection on active repos ([changelog](https://github.blog/changelog/2026-03-17-dependabot-now-detects-malware-in-npm-dependencies/)). @@ -1083,5 +1083,5 @@ P6: phase 6, org-wide enforcement, from 2026-11-04 (owner and coordinator). Wher - P6-1. Summary comment format (from the PR #23 comment review and P5-9). Five display changes in the sticky summary; the gate does not change. (1) One row per cause: the dependency-review job result and step outcome reasons are one row, passes that failed for the same reason share one row, in the findings table and in the record's reasons table, which still lists every detail. (2) Repo setting rows: a dependency-review failure is labeled as a repo setting (Dependency graph off) that only a repo admin can fix only on positive evidence from GitHub's error message (P6-2), and the author is not told to fix it. (3) File paths and code-made reason details show as plain text in code spans, not as backslash-escaped prose, with the same cleaning, redaction and reference breaking. (4) The token usage table collapses to one line when no pass recorded any count. (5) An informational row counts the PR's commits that are not verified-signed, with a link to GitHub's commit signing guide; no row at zero. Details in "Summary comment". The commit list and the dependency probe are read by the gate job (no Anthropic credential) for display only: they are not gate inputs, never reach Claude, and cannot change the result, a reason, the waiver line or the exit code. Unit tests check each item and that the gate result is the same with and without them. - P6-2. Review of P6-1 (three Important findings, fixed before release). (1) The unsigned-commit row could push a summary near the 64,000-character limit into the fail-closed overflow notice, so unsigned commits could fail a passing run: the summary is now rendered without the row first, the row stays only when it changes no cut, and every read error gives no row. (2) A 403 on the dependency comparison alone does not prove the Dependency graph is off (private repo without Advanced Security, fork, permission, secondary rate limit): the setting label now needs GitHub's error message to say the graph is disabled; otherwise the row stays neutral. (3) A file name with a backslash before a pipe could end a table cell early in some parsers: a backslash run before a pipe is now split from it with a zero-width space. Unit tests cover the near-limit case, read errors, 14 probe answers that are not evidence, and hostile names (backslash, pipe, backtick, newline, bidi) under two table-splitting rules. - P6-3. Comment fixes from the first live run at `364058d` (claude-review-test PR 4, 2026-11-04; display only, the gate does not change). (1) The TIP callout said "Nothing to do for this review" while the unsigned-commit row said the merge box blocks: with the row, the TIP now names the signing step. (2) The row said "1 of 1 commits ... is": it now says "commit" when the PR has one commit. (3) The Dependency graph setting match ignored negation and could join two clauses: it now needs one narrow affirmative phrase, "dependency graph", an optional "for this repository", one verb ("is", "was", "has been"), an optional "currently", then "disabled", "not enabled" or "turned off", with nothing else between them (review AC, I2: "is neither disabled nor turned off" passed the first fix). (4) The error-body read in `collect.default_transport` did not catch `http.client.IncompleteRead`: it now catches `http.client.HTTPException` like the other read errors; the body is empty and the status still fails the caller. (5) "Never approves" shows once in the pass comment: already true at `364058d` (a unit test checks it), no change. -- P6-4. Eval environment split (owner and coordinator, 2026-11-04). Release evals run on `main`, which holds only reviewed code, so they need no approval; evals of other refs can run unreviewed code with the eval credential, so they need the owner's approval. Step 1 (owner, 2026-11-04): the `eval` environment got a main-only branch policy and lost its required reviewer and "prevent self-review"; admin bypass stays on. Until step 2, branch evals are refused by that policy. Step 2 (this change): `eval.yml` picks `eval` on `refs/heads/main` and `eval-test` on any other ref, with an exact shell test in a select job with no permissions (an Actions expression compares without case, review AC I1), and passes the choice to `evals.py` in `EVAL_ENVIRONMENT`; `evals.py` holds the federation IDs per environment and fails closed on any other value and on an environment whose rule is not set. Open owner steps: create the `eval-test` environment (required reviewer the owner, "prevent self-review" on), create its federation rule on the test workspace with the exact claim `environment: "eval-test"`, then set the rule ID in `evals.py` (a PR) and test it as V27 (an approved run gets a token, a run in another environment or with no environment gets none). +- P6-4. Eval environment split (owner and coordinator, 2026-11-04). Release evals run on `main`, which holds only reviewed code, so they need no approval; evals of other refs can run unreviewed code with the eval credential, so they need the owner's approval. Step 1 (owner, 2026-11-04): the `eval` environment got a main-only branch policy and lost its required reviewer and "prevent self-review"; admin bypass stays on. Until step 2, branch evals are refused by that policy. Step 2 (this change): `eval.yml` picks `eval` on `refs/heads/main` and `eval-test` on any other ref, with an exact shell test in a select job with no permissions (an Actions expression compares without case, review AC I1), and passes the choice to `evals.py` in `EVAL_ENVIRONMENT`; `evals.py` holds the federation IDs per environment and fails closed on any other value and on an environment whose rule is not set. Step 3 (owner, 2026-11-06): the owner created the federation rule `eval-test-federation-rule` on the test workspace as a copy of the `eval` rule (same issuer, subject prefix, owner, repository and event claims, scope `workspace:developer`, service account, workspace and 600 s lifetime) with the exact claim `environment: "eval-test"`; its ID is set in `evals.py`. Open: test it as V27 (an approved run gets a token, a run in another environment or with no environment gets none). - P6-5. Phase 6 scope (owner, 2026-11-05). (1) Phase 4 exit evidence: 5 real PRs reviewed (was 10; owner and Dependabot PRs count), 2 weekly metric reports, and 7 days with no unexplained false-positive hard block, run failure or maintainer bypass without a reason. (2) `START_HERE` is not a review target: it will be archived after its content moves into `.github` (issue #27 in this repo). The review targets stay the 4 pilot repos. (3) The bypass switch is deferred to follow-up work: the review ruleset keeps the pilot bypass, and the review stays an advisory signal for maintainers, not a hard gate, until someone asks the owner for a gate. The name change to `claude-review` waits for that switch. (4) Code security configurations: the configuration for public repos (enforced on the active public repos, default for new public repos) and the dependency configuration (enforced, default for new private and internal repos) are live and read back; one planned change to the dependency configuration is skipped by owner decision. Details are kept in the owner's records. (5) The phase 6 final acceptance test checks this shape (build plan, phase 6 exit); the gate-shape checks (org-admin-only bypass, the renamed ruleset, a live check that a Maintain user cannot bypass) move to the follow-up. diff --git a/review/src/oc_review/evals.py b/review/src/oc_review/evals.py index a6bf12d..27b51ba 100644 --- a/review/src/oc_review/evals.py +++ b/review/src/oc_review/evals.py @@ -81,11 +81,11 @@ class EvalRule: EVAL_FEDERATION: Mapping[str, EvalRule] = MappingProxyType({ # `eval`: runs on main only (reviewed code), no approval. EVAL_ENVIRONMENT_MAIN: EvalRule("fdrl_01BL4n1gQzkrroCmDZoXUePm", "svac_017eR3ohB6rP5MEAsda6U7X6"), - # `eval-test`: any other ref, owner approval with prevent self-review. The - # rule is not created yet (owner step); set its ID here when it exists. - # The service account is the `eval` one; the owner may give this rule - # its own service account and change it here. - EVAL_ENVIRONMENT_TEST: EvalRule(None, "svac_017eR3ohB6rP5MEAsda6U7X6"), + # `eval-test`: any other ref, owner approval with prevent self-review. + # Rule `eval-test-federation-rule` (created 2026-11-06): a copy of the + # `eval` rule with the claim `environment: eval-test`. It uses the + # `eval` service account. + EVAL_ENVIRONMENT_TEST: EvalRule("fdrl_01WkhnTsBLVnYBHgAfxkugnc", "svac_017eR3ohB6rP5MEAsda6U7X6"), }) EVAL_REPOSITORY = "OperationCode/eval-fixtures" diff --git a/review/tests/unit/test_evals.py b/review/tests/unit/test_evals.py index 3c2075b..b5dcf12 100644 --- a/review/tests/unit/test_evals.py +++ b/review/tests/unit/test_evals.py @@ -241,9 +241,10 @@ def test_eval_environments_are_exactly_eval_and_eval_test(): assert evals.EVAL_ENVIRONMENT_VAR == "EVAL_ENVIRONMENT" -def test_eval_test_rule_is_unset_and_shares_the_eval_service_account(): - # The owner has not created the eval-test rule yet; the owner may change its service account. - assert evals.EVAL_FEDERATION["eval-test"].rule_id is None +def test_eval_test_rule_is_set_and_shares_the_eval_service_account(): + # Rule `eval-test-federation-rule`, created 2026-11-06; the owner may change its service account. + assert evals.EVAL_FEDERATION["eval-test"].rule_id == "fdrl_01WkhnTsBLVnYBHgAfxkugnc" + assert evals.EVAL_FEDERATION["eval-test"].rule_id != evals.EVAL_FEDERATION["eval"].rule_id assert evals.EVAL_FEDERATION["eval-test"].service_account_id == evals.EVAL_FEDERATION["eval"].service_account_id @@ -254,7 +255,10 @@ def test_every_set_eval_rule_id_has_the_real_id_form(): assert analyze.ids_valid(ids) and not analyze.ids_are_placeholders(ids), name -def test_eval_test_without_a_rule_fails_closed(tmp_path, capsys): +def test_eval_test_without_a_rule_fails_closed(tmp_path, capsys, monkeypatch): + # Keeps the fail-closed path covered: the entry is set in the code, so unset it here. + unset = evals.EvalRule(None, evals.EVAL_FEDERATION["eval-test"].service_account_id) + monkeypatch.setattr(evals, "EVAL_FEDERATION", {**evals.EVAL_FEDERATION, "eval-test": unset}) client, oidc_calls = FakeClient(), [] assert main_in(tmp_path, "eval-test", client=client, oidc_calls=oidc_calls) == 1 assert "eval-test federation rule not configured" in capsys.readouterr().err