Skip to content

FEAT: Backend APIs for scan results and running scan attacks - #4030

Open
Richard Lundeen (richlundeen) wants to merge 3 commits into
microsoft:mainfrom
richlundeen:richlundeen-scanner-stack-1-backend-apis
Open

Richard Lundeen (richlundeen) wants to merge 3 commits into
microsoft:mainfrom
richlundeen:richlundeen-scanner-stack-1-backend-apis

Conversation

@richlundeen

@richlundeen Richard Lundeen (richlundeen) commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Description

Layer 1 of 3 for the scan results redesign: backend APIs so Chat can be the single conversation view for both finished and still-running scan attacks, and so scan results can show linked findings and score details without fabricating data.

  • Running attacks are readable. GET /api/attacks/{id}, /metadata, /conversations, and /messages (paged and unpaged) now resolve an attack that an active scan is still running, instead of 404. Summary and metadata responses set in_progress: true, outcome undetermined, and execution_read_only: true; conversation/message responses do not carry the running-state flags. The current conversation is main; other objective conversations so far are pruned references. A saved result always wins, so the same IDs resolve to the saved result once the attack finishes.
  • Clean boundary. AttackService reads through a small read-only LiveAttackLookup port (pyrit/backend/services/live_attack_reader.py) that ScenarioRunService.find_live_attack provides; the singleton factory wires it. The saved-conversation reader takes an injectable reference source, so live and saved paging share one code path, and AttackConversationPage.from_references owns the reference ordering/revision for both.
  • Provenance. Attack summaries and metadata add scenario_result_id (the owning scan, from attribution_parent_id).
  • Findings per attack. GET /api/attacks/{id}/findings lists findings across Operations that link the attack (finding + operation_name + evidence_id), backed by MemoryInterface.get_findings_for_attack_result_async and a new ix_FindingEvidenceEntries_attack_result_id index (Alembic b5d7f9a1c3e6, a plain UUID-key index valid on SQLite and SQL Server).
  • Live conversations. ScenarioActiveConversation.conversation_ids lists every objective conversation so far, alongside conversation_id.
  • Score details. Scan progress results and hierarchy nodes already carried scorer name, type, value, and rationale; ScenarioProgressScore.score_category adds the missing category. Official scorer evaluation metrics remain available separately under progress.summary.objective_scorer.metrics.
  • Frontend contract only. Matching types plus attacksApi.getAttackFindings; no UI changes in this layer.

Stack: this PR (backend APIs) → #4031 (frontend foundations) → #4032 (scan results redesign + live Chat).

API walkthrough

These are illustrative, selected-field contract fragments, not captured production responses or measured scorer performance. UUID placeholders identify related records.

Change Request / data What it enables
Live reads and provenance GET /api/attacks/{id}/metadata The same attack ID works during execution and after saving; the metadata identifies its owning scan and read-only state.
Bounded conversation navigation GET /api/attacks/{id}/conversations?limit=100 and /messages?conversation_id={cid}&limit=100 Current and earlier objective conversations share the saved-reader paging path; adversarial/preparation conversations stay excluded. Scan progress supplies both the current conversation_id and all objective conversation_ids.
Findings across Operations GET /api/attacks/{id}/findings Each finding includes operation_name and the evidence_id used to detach the link. Results are ordered by severity, then newest; the migration indexes the attack-result evidence lookup on SQLite and SQL Server.
Score explanation progress.results[].score and hierarchy node score The frontend can show the actual score value, scorer, category and rationale rather than constructing a synthetic score.
Live metadata → saved result, without changing the attack URL
{
  "attack_result_id": "<attack UUID>",
  "scenario_result_id": "<scan UUID>",
  "in_progress": true,
  "outcome": "undetermined",
  "execution_read_only": true
}

Once the saved result exists it wins over live lookup. Writes still require a saved result. Before the first objective conversation, the read remains 404. Read running-state flags from summary/metadata, not conversation/message responses. Restart reference paging if the live-to-saved transition changes its revision.

Findings identify both the finding and its evidence link
{
  "attack_result_id": "<attack UUID>",
  "items": [{
    "id": "<finding UUID>",
    "operation_id": "<Operation UUID>",
    "operation_name": "<current Operation name>",
    "evidence_id": "<evidence-link UUID>",
    "title": "<stored finding title>",
    "severity": "<stored severity>"
  }]
}

No linked findings, including for an unsaved attack, yields items: []; a non-UUID attack ID returns 422.

Attack scores are not scorer evaluation-quality metrics

A per-attack score fragment, using a unit-test fixture rather than a measured evaluation:

{
  "scorer_name": "TestScorer",
  "score_type": "true_false",
  "status": "complete",
  "score_value": "true",
  "score_rationale": "The objective was achieved.",
  "score_category": ["jailbreak"]
}

That is one attack's score, not the scorer's accuracy or the scan's success rate. Existing official evaluation metrics remain at progress.summary.objective_scorer.metrics: accuracy, accuracy standard error, F1, precision, recall, and average score time. They are loaded only for matching evaluation identity; no matching evaluation means null metrics, not invented zeros. The scorer registry API already exposes metrics and a separate evaluation_identifier for comparison/disclosure in the frontend layers. This backend layer removes no inspected score or official-evaluation fields.

- GET /api/attacks/{id}/findings lists findings linked to an attack across Operations,
  backed by a new FindingEvidenceEntries.attack_result_id index (Alembic b5d7f9a1c3e6).
- Attack read APIs (summary, metadata, conversations, messages) resolve scan attacks that
  are still running through a read-only LiveAttackLookup provided by ScenarioRunService.
  Saved results always win; summaries/metadata expose in_progress and scenario_result_id.
- ScenarioActiveConversation.conversation_ids lists every objective conversation so far.
- ScenarioProgressScore carries score_category for progress results and hierarchy nodes.
- Frontend types and attacksApi.getAttackFindings for later layers.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 7, 2026 01:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the backend and API contract changes.

No open issues 🎉

What changed in this PR

Adds the backend foundation for viewing running and completed scan attacks in Chat, with linked findings and richer score details. This layer updates API contracts without changing the UI.

Changes:

  • Makes running scan attacks readable through a read-only lookup, with saved results taking precedence.
  • Adds attack-linked findings retrieval and a supporting database index.
  • Exposes scan provenance, objective conversation IDs, and score categories through backend and frontend contracts.
File Description
tests/​unit/​memory/​test_score_participation_migration.py Allows a newer migration while preserving the single-head check.
tests/​unit/​memory/​test_findings.py Tests cross-operation findings lookup and ordering.
tests/​unit/​memory/​test_finding_evidence_attack_index_migration.py Tests index migration, rollback, and SQL Server compilation.
tests/​unit/​memory/​memory_interface/​test_interface_scenario_progress.py Verifies score categories in progress results.
tests/​unit/​backend/​test_scenario_resume.py Tests live attack reads transitioning to saved results.
tests/​unit/​backend/​test_live_attack_reads.py Covers live lookup, read APIs, and linked findings.
pyrit/​models/​scenario_progress.py Adds score categories and objective conversation IDs.
pyrit/​memory/​memory_models.py Declares the attack-result evidence index.
pyrit/​memory/​memory_interface.py Adds findings lookup, shared reference paging, and score categories.
pyrit/​memory/​alembic/​versions/​b5d7f9a1c3e6_index_finding_evidence_attack_result.py Adds the evidence lookup index migration.
pyrit/​backend/​services/​scenario_run_service.py Provides live attack lookup and conversation IDs.
pyrit/​backend/​services/​saved_conversation_read_service.py Accepts an injectable conversation reference reader.
pyrit/​backend/​services/​operation_service.py Builds linked findings responses with operation names.
pyrit/​backend/​services/​live_attack_reader.py Defines the read-only live attack projection.
pyrit/​backend/​services/​attack_service.py Resolves live attacks when no saved result exists.
pyrit/​backend/​routes/​attacks.py Exposes the attack findings endpoint.
pyrit/​backend/​README.md Documents live reads and linked findings contracts.
pyrit/​backend/​models/​operations.py Defines attack-linked findings response models.
pyrit/​backend/​models/​attacks.py Adds running-state and scan provenance fields.
pyrit/​backend/​mappers/​attack_mappers.py Maps running-state and provenance into summaries.
frontend/​src/​types/​index.ts Mirrors the expanded backend contracts.
frontend/​src/​services/​api.ts Adds the attack findings API client method.
frontend/​src/​services/​api.test.ts Tests findings requests and error propagation.

Review effort: Balanced

Integrate main 25417ff without rewriting the existing stack history.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Limit running-state guarantees to attack summary and metadata reads without expanding conversation/message schemas or removing score details.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; it does not authorize merge.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants