Repository navigation
FIX Make scorer batch evaluation eligibility explicit - #4013
Open
hannahwestra25 wants to merge 4 commits into
Open
hannahwestra25 wants to merge 4 commits into
hannahwestra25 wants to merge 4 commits into
Conversation
Exclude the raw Decision scorer from automatic batch evaluation while keeping its threshold wrapper eligible. Preserve strict failures for unconfigured candidates and empty selections, and document the generic batch-only policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the changes that make batch-evaluation eligibility explicit.
No open issues 🎉
What changed in this PR
Following #3832, this PR makes batch-evaluation eligibility explicit in PyRIT without changing metric semantics or scorer availability for normal use.
Changes:
- Adds
skip_batch_evaluationtagging to exclude the raw Decision scorer while keeping its threshold wrapper eligible. - Makes empty eligible selections fail while preserving failures for missing mappings and evaluation errors.
- Adds regression tests and updates batch-evaluation documentation.
| File | Description |
|---|---|
tests/unit/setup/test_scorer_initializer.py |
Verifies Decision scorer mappings and exclusion tagging. |
tests/unit/build_scripts/test_evaluate_scorers.py |
Tests wrapper evaluation, failure handling, and empty selections. |
pyrit/setup/initializers/scorers.py |
Defines the exclusion tag and applies it to the raw Decision scorer. |
doc/contributing/10_release_process.md |
Documents batch eligibility and failure behavior for releases. |
doc/code/scoring/4_scorer_metrics.py |
Updates batch-evaluation guidance. |
doc/code/scoring/4_scorer_metrics.ipynb |
Synchronizes notebook documentation with the paired Python file. |
build_scripts/evaluate_scorers.py |
Filters excluded scorers, reports skips, and rejects empty selections. |
Review effort: Balanced
Rank F1 only with known positive complete coverage. Preserve configured fallback choices and tags with explicit diagnostics, and document the selection boundary without changing persisted metrics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep batch exclusion, complete-coverage ranking, and configured fallback guidance concise in paired notebook sources. Preserve code cells and saved outputs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
hannahwestra25
enabled auto-merge
November 6, 2026 19:07
Roman Lutz (romanlutz)
approved these changes
Nov 6, 2026
| from pyrit.registry import ScorerRegistry | ||
| from pyrit.setup import IN_MEMORY, initialize_pyrit_async | ||
| from pyrit.setup.initializers import ScorerInitializer, TargetInitializer | ||
| from pyrit.setup.initializers.scorers import ScorerInitializerTags |
Contributor
There was a problem hiding this comment.
I'd hope we can shorten this import
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follow-up to #3832. Skip unconfigured raw Decision in batch benchmarks; retain its threshold wrapper. Partial scores can mislead rankings: rank objective F1 only with known, positive, complete coverage; preserve and diagnose configured fallbacks. Metrics remain unchanged.
Follow-up: paired/shared-subset comparisons using stable per-example response-objective IDs and retained outcomes, alongside coverage. Aggregates cannot reconstruct them; small/nonrepresentative intersections need an explicit coverage policy. Richer abstention accounting and backfills remain deferred.
Tests and Documentation
189 tests passed; one pre-existing ty-warning failure. Ruff/docs passed; typing warning-only. Notebook synchronized, unexecuted; outputs preserved. Benchmarks not rerun.