Skip to content

FIX: Surface scorer evaluation coverage in scenario progress GUI (#3325) - #4000

Open
Yogyabhilash wants to merge 2 commits into
microsoft:mainfrom
Yogyabhilash:auto-fix-3325
Open

Yogyabhilash wants to merge 2 commits into
microsoft:mainfrom
Yogyabhilash:auto-fix-3325

Conversation

@Yogyabhilash

@Yogyabhilash Yogyabhilash commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Description

Fixes #3325.

In the scenario progress GUI and backend read model, scorer evaluation metrics (ScenarioObjectiveScorerMetrics) omitted num_responses (evaluated count) and num_input_responses (total count), even though:

  1. ObjectiveScorerMetrics already records both fields during evaluation.
  2. The scorer evaluation documentation (doc/code/scoring/4_scorer_metrics.py) explicitly instructs users to compare num_responses against num_input_responses to verify coverage, because responses that do not yield a verdict are excluded from accuracy calculations.

Without these fields, the scenario progress GUI displays evaluated metrics without indicating coverage or incomplete evaluations (e.g., displaying 100% accuracy on a run where only a fraction of prompts had verdicts, with no indication that responses without verdicts were excluded).

Changes:

  • pyrit/models/scenario_progress.py: Added optional non-negative integer fields num_responses and num_input_responses to ScenarioObjectiveScorerMetrics (defaults to None for backward compatibility with older/legacy records).
  • pyrit/backend/services/scenario_progress_read_model.py: Forwarded num_responses and num_input_responses from official_metrics when building ScenarioObjectiveScorerMetrics.
  • frontend/src/types/index.ts: Updated ScenarioObjectiveScorerMetrics interface to include num_responses?: number | null and num_input_responses?: number | null.
  • frontend/src/utils/scorerPresentation.ts: Added presentation labels (Evaluated responses, Total responses) and metric descriptions.
  • frontend/src/components/AttackResults/ObjectiveScorerDetails.tsx: Rendered coverage context hint when num_responses < num_input_responses explaining that responses without a verdict were excluded from accuracy.

Tests and Documentation

  • Added unit tests in tests/unit/backend/test_scenario_progress_read_model.py:
    • test_progress_snapshot_propagates_scorer_coverage_metrics: Verifies that num_responses and num_input_responses are propagated from scorer evaluation files to the progress snapshot.
    • test_load_objective_metrics_reads_coverage_and_legacy_records: Verifies that records with coverage metrics are read correctly and legacy JSONL records lacking num_input_responses continue to load safely without error.
  • Verified test suite:
    • pytest tests/unit/backend/test_scenario_progress_read_model.py (60 passed).
    • Full backend test suite tests/unit/backend (3506 passed, 4 skipped).
  • Verified code quality:
    • ruff check pyrit tests/unit/backend/test_scenario_progress_read_model.py (all checks passed).
    • ruff format --check pyrit tests/unit/backend/test_scenario_progress_read_model.py (3 files already formatted).
    • uv run --frozen --extra all -m ty check (all checks passed).
  • JupyText: Not applicable as no documentation notebooks or tutorial .py pairs were modified.

Checklist

AI assistance disclosure: Prepared with an AI coding agent under direction; verified, tested, and reviewed locally.

Copilot AI balanced review requested due to automatic review settings November 6, 2026 09:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the change exposing scorer evaluation coverage alongside accuracy.

No open issues 🎉

What changed in this PR

Addresses #3325 by carrying scorer evaluation coverage through the scenario progress API to the GUI, so users can distinguish accuracy from evaluation completeness.

Changes:

  • Adds optional evaluated and total response counts to backend models and frontend types.
  • Displays coverage metrics and a caveat when responses without verdicts were excluded from accuracy.
  • Adds backend tests for count propagation and compatibility with legacy records.
File Description
tests/​unit/​backend/​test_scenario_progress_read_model.py Tests coverage propagation and legacy record loading.
pyrit/​models/​scenario_progress.py Adds optional, non-negative coverage counts.
pyrit/​backend/​services/​scenario_progress_read_model.py Forwards coverage counts from recorded evaluation metrics.
frontend/​src/​utils/​scorerPresentation.ts Adds coverage labels and descriptions.
frontend/​src/​types/​index.ts Adds nullable coverage counts to the metrics interface.
frontend/​src/​components/​AttackResults/​ObjectiveScorerDetails.tsx Displays a coverage caveat for incomplete evaluations.

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scorer-eval GUI drops the coverage fields the docs say to compare

3 participants