Repository navigation
FEAT: Backend APIs for scan results and running scan attacks - #4030
Open
Richard Lundeen (richlundeen) wants to merge 3 commits into
Open
Richard Lundeen (richlundeen) wants to merge 3 commits into
Richard Lundeen (richlundeen) wants to merge 3 commits into
Conversation
- GET /api/attacks/{id}/findings lists findings linked to an attack across Operations,
backed by a new FindingEvidenceEntries.attack_result_id index (Alembic b5d7f9a1c3e6).
- Attack read APIs (summary, metadata, conversations, messages) resolve scan attacks that
are still running through a read-only LiveAttackLookup provided by ScenarioRunService.
Saved results always win; summaries/metadata expose in_progress and scenario_result_id.
- ScenarioActiveConversation.conversation_ids lists every objective conversation so far.
- ScenarioProgressScore carries score_category for progress results and hierarchy nodes.
- Frontend types and attacksApi.getAttackFindings for later layers.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues were identified for the backend and API contract changes.
No open issues 🎉
What changed in this PR
Adds the backend foundation for viewing running and completed scan attacks in Chat, with linked findings and richer score details. This layer updates API contracts without changing the UI.
Changes:
- Makes running scan attacks readable through a read-only lookup, with saved results taking precedence.
- Adds attack-linked findings retrieval and a supporting database index.
- Exposes scan provenance, objective conversation IDs, and score categories through backend and frontend contracts.
| File | Description |
|---|---|
tests/unit/memory/test_score_participation_migration.py |
Allows a newer migration while preserving the single-head check. |
tests/unit/memory/test_findings.py |
Tests cross-operation findings lookup and ordering. |
tests/unit/memory/test_finding_evidence_attack_index_migration.py |
Tests index migration, rollback, and SQL Server compilation. |
tests/unit/memory/memory_interface/test_interface_scenario_progress.py |
Verifies score categories in progress results. |
tests/unit/backend/test_scenario_resume.py |
Tests live attack reads transitioning to saved results. |
tests/unit/backend/test_live_attack_reads.py |
Covers live lookup, read APIs, and linked findings. |
pyrit/models/scenario_progress.py |
Adds score categories and objective conversation IDs. |
pyrit/memory/memory_models.py |
Declares the attack-result evidence index. |
pyrit/memory/memory_interface.py |
Adds findings lookup, shared reference paging, and score categories. |
pyrit/memory/alembic/versions/b5d7f9a1c3e6_index_finding_evidence_attack_result.py |
Adds the evidence lookup index migration. |
pyrit/backend/services/scenario_run_service.py |
Provides live attack lookup and conversation IDs. |
pyrit/backend/services/saved_conversation_read_service.py |
Accepts an injectable conversation reference reader. |
pyrit/backend/services/operation_service.py |
Builds linked findings responses with operation names. |
pyrit/backend/services/live_attack_reader.py |
Defines the read-only live attack projection. |
pyrit/backend/services/attack_service.py |
Resolves live attacks when no saved result exists. |
pyrit/backend/routes/attacks.py |
Exposes the attack findings endpoint. |
pyrit/backend/README.md |
Documents live reads and linked findings contracts. |
pyrit/backend/models/operations.py |
Defines attack-linked findings response models. |
pyrit/backend/models/attacks.py |
Adds running-state and scan provenance fields. |
pyrit/backend/mappers/attack_mappers.py |
Maps running-state and provenance into summaries. |
frontend/src/types/index.ts |
Mirrors the expanded backend contracts. |
frontend/src/services/api.ts |
Adds the attack findings API client method. |
frontend/src/services/api.test.ts |
Tests findings requests and error propagation. |
Review effort: Balanced
This was referenced Nov 7, 2026
Integrate main 25417ff without rewriting the existing stack history. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Limit running-state guarantees to attack summary and metadata reads without expanding conversation/message schemas or removing score details. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
left a comment
Contributor
Author
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; it does not authorize merge.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Layer 1 of 3 for the scan results redesign: backend APIs so Chat can be the single conversation view for both finished and still-running scan attacks, and so scan results can show linked findings and score details without fabricating data.
GET /api/attacks/{id},/metadata,/conversations, and/messages(paged and unpaged) now resolve an attack that an active scan is still running, instead of 404. Summary and metadata responses setin_progress: true, outcomeundetermined, andexecution_read_only: true; conversation/message responses do not carry the running-state flags. The current conversation is main; other objective conversations so far are pruned references. A saved result always wins, so the same IDs resolve to the saved result once the attack finishes.AttackServicereads through a small read-onlyLiveAttackLookupport (pyrit/backend/services/live_attack_reader.py) thatScenarioRunService.find_live_attackprovides; the singleton factory wires it. The saved-conversation reader takes an injectable reference source, so live and saved paging share one code path, andAttackConversationPage.from_referencesowns the reference ordering/revision for both.scenario_result_id(the owning scan, fromattribution_parent_id).GET /api/attacks/{id}/findingslists findings across Operations that link the attack (finding +operation_name+evidence_id), backed byMemoryInterface.get_findings_for_attack_result_asyncand a newix_FindingEvidenceEntries_attack_result_idindex (Alembicb5d7f9a1c3e6, a plain UUID-key index valid on SQLite and SQL Server).ScenarioActiveConversation.conversation_idslists every objective conversation so far, alongsideconversation_id.ScenarioProgressScore.score_categoryadds the missing category. Official scorer evaluation metrics remain available separately underprogress.summary.objective_scorer.metrics.attacksApi.getAttackFindings; no UI changes in this layer.Stack: this PR (backend APIs) → #4031 (frontend foundations) → #4032 (scan results redesign + live Chat).
API walkthrough
These are illustrative, selected-field contract fragments, not captured production responses or measured scorer performance. UUID placeholders identify related records.
GET /api/attacks/{id}/metadataGET /api/attacks/{id}/conversations?limit=100and/messages?conversation_id={cid}&limit=100conversation_idand all objectiveconversation_ids.GET /api/attacks/{id}/findingsoperation_nameand theevidence_idused to detach the link. Results are ordered by severity, then newest; the migration indexes the attack-result evidence lookup on SQLite and SQL Server.progress.results[].scoreand hierarchy nodescoreLive metadata → saved result, without changing the attack URL
{ "attack_result_id": "<attack UUID>", "scenario_result_id": "<scan UUID>", "in_progress": true, "outcome": "undetermined", "execution_read_only": true }Once the saved result exists it wins over live lookup. Writes still require a saved result. Before the first objective conversation, the read remains 404. Read running-state flags from summary/metadata, not conversation/message responses. Restart reference paging if the live-to-saved transition changes its revision.
Findings identify both the finding and its evidence link
{ "attack_result_id": "<attack UUID>", "items": [{ "id": "<finding UUID>", "operation_id": "<Operation UUID>", "operation_name": "<current Operation name>", "evidence_id": "<evidence-link UUID>", "title": "<stored finding title>", "severity": "<stored severity>" }] }No linked findings, including for an unsaved attack, yields
items: []; a non-UUID attack ID returns 422.Attack scores are not scorer evaluation-quality metrics
A per-attack score fragment, using a unit-test fixture rather than a measured evaluation:
{ "scorer_name": "TestScorer", "score_type": "true_false", "status": "complete", "score_value": "true", "score_rationale": "The objective was achieved.", "score_category": ["jailbreak"] }That is one attack's score, not the scorer's accuracy or the scan's success rate. Existing official evaluation metrics remain at
progress.summary.objective_scorer.metrics: accuracy, accuracy standard error, F1, precision, recall, and average score time. They are loaded only for matching evaluation identity; no matching evaluation means null metrics, not invented zeros. The scorer registry API already exposesmetricsand a separateevaluation_identifierfor comparison/disclosure in the frontend layers. This backend layer removes no inspected score or official-evaluation fields.