Repository navigation
FIX: Surface scorer evaluation coverage in scenario progress GUI (#3325) - #4000
Open
Yogyabhilash wants to merge 2 commits into
Open
Yogyabhilash wants to merge 2 commits into
Yogyabhilash wants to merge 2 commits into
Conversation
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain for the change exposing scorer evaluation coverage alongside accuracy.
No open issues 🎉
What changed in this PR
Addresses #3325 by carrying scorer evaluation coverage through the scenario progress API to the GUI, so users can distinguish accuracy from evaluation completeness.
Changes:
- Adds optional evaluated and total response counts to backend models and frontend types.
- Displays coverage metrics and a caveat when responses without verdicts were excluded from accuracy.
- Adds backend tests for count propagation and compatibility with legacy records.
| File | Description |
|---|---|
tests/unit/backend/test_scenario_progress_read_model.py |
Tests coverage propagation and legacy record loading. |
pyrit/models/scenario_progress.py |
Adds optional, non-negative coverage counts. |
pyrit/backend/services/scenario_progress_read_model.py |
Forwards coverage counts from recorded evaluation metrics. |
frontend/src/utils/scorerPresentation.ts |
Adds coverage labels and descriptions. |
frontend/src/types/index.ts |
Adds nullable coverage counts to the metrics interface. |
frontend/src/components/AttackResults/ObjectiveScorerDetails.tsx |
Displays a coverage caveat for incomplete evaluations. |
Review effort: Balanced
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #3325.
In the scenario progress GUI and backend read model, scorer evaluation metrics (
ScenarioObjectiveScorerMetrics) omittednum_responses(evaluated count) andnum_input_responses(total count), even though:ObjectiveScorerMetricsalready records both fields during evaluation.doc/code/scoring/4_scorer_metrics.py) explicitly instructs users to comparenum_responsesagainstnum_input_responsesto verify coverage, because responses that do not yield a verdict are excluded from accuracy calculations.Without these fields, the scenario progress GUI displays evaluated metrics without indicating coverage or incomplete evaluations (e.g., displaying 100% accuracy on a run where only a fraction of prompts had verdicts, with no indication that responses without verdicts were excluded).
Changes:
pyrit/models/scenario_progress.py: Added optional non-negative integer fieldsnum_responsesandnum_input_responsestoScenarioObjectiveScorerMetrics(defaults toNonefor backward compatibility with older/legacy records).pyrit/backend/services/scenario_progress_read_model.py: Forwardednum_responsesandnum_input_responsesfromofficial_metricswhen buildingScenarioObjectiveScorerMetrics.frontend/src/types/index.ts: UpdatedScenarioObjectiveScorerMetricsinterface to includenum_responses?: number | nullandnum_input_responses?: number | null.frontend/src/utils/scorerPresentation.ts: Added presentation labels (Evaluated responses,Total responses) and metric descriptions.frontend/src/components/AttackResults/ObjectiveScorerDetails.tsx: Rendered coverage context hint whennum_responses < num_input_responsesexplaining that responses without a verdict were excluded from accuracy.Tests and Documentation
tests/unit/backend/test_scenario_progress_read_model.py:test_progress_snapshot_propagates_scorer_coverage_metrics: Verifies thatnum_responsesandnum_input_responsesare propagated from scorer evaluation files to the progress snapshot.test_load_objective_metrics_reads_coverage_and_legacy_records: Verifies that records with coverage metrics are read correctly and legacy JSONL records lackingnum_input_responsescontinue to load safely without error.pytest tests/unit/backend/test_scenario_progress_read_model.py(60 passed).tests/unit/backend(3506 passed, 4 skipped).ruff check pyrit tests/unit/backend/test_scenario_progress_read_model.py(all checks passed).ruff format --check pyrit tests/unit/backend/test_scenario_progress_read_model.py(3 files already formatted).uv run --frozen --extra all -m ty check(all checks passed)..pypairs were modified.Checklist
ruff check) and formatting (ruff format) verifiedty check) verifiedAI assistance disclosure: Prepared with an AI coding agent under direction; verified, tested, and reviewed locally.