Repository navigation
FEAT: Add jailbreak_multi_turn delivery to Jailbreak scenario - #3657
Absarahmedofficial wants to merge 5 commits into
Conversation
|
Thanks for the detailed review. All three points are addressed in the follow-up commit. The multi-turn description now states that it requires native multi-turn support, and the Jailbreak overview lists it as a third delivery method. For the objective handoff, the delivery now sets an opt-in flag on its technique seed group so the objective is appended as the final user prompt during seed composition; MultiPromptSendingAttack itself is unchanged, and a mixed seed group test covers a context prompt followed by the objective. The run-size estimator now gives jailbreak_multi_turn the same capability treatment as the system-prompt delivery, including the ValueError when it is the only selected technique and the target cannot run multi-turn. |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve objective-only and static-user inputs while rejecting authored non-user and simulated history through the shared compatibility policy. Strengthen sequence, immutability, and final-score regressions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Checked the three fixes described in #3657 (comment) against 95a90be; the description, ordinary-context objective handoff, and target-aware estimate are addressed. The additional history-input repair preserves those fixes and the opt-in/default-mode behavior, while explicitly rejecting authored non-user or simulated history through the existing compatibility policy. |
There was a problem hiding this comment.
🟡 Changes recommended
With objective appending enabled and the default preserve placement, an objective-only group can reuse sequence 0, merging the objective into the first technique message instead of creating a separate final user turn.
1 open issue
What changed in this PR
Adds opt-in jailbreak_multi_turn delivery to PyRIT’s Jailbreak scenario, sending template framing before the objective in the same conversation and scoring only the final response.
Changes:
- Adds multi-turn delivery with target-capability checks and run-size estimates.
- Extends technique seed groups with objective appending and static-user-prompt requirements.
- Adds tests for delivery order, scoring, compatibility, and reusable seed inputs.
| File | Description |
|---|---|
tests/unit/scenario/airt/test_jailbreak.py |
Tests multi-turn delivery, scoring, compatibility, and estimates. |
tests/unit/models/test_seed_group.py |
Tests objective handoff, multipart ordering, and input preservation. |
tests/unit/models/test_seed_group_requirements.py |
Tests static-user-prompt restrictions. |
pyrit/scenario/scenarios/airt/jailbreak.py |
Adds the opt-in delivery technique and construction logic. |
pyrit/scenario/core/attack_technique.py |
Includes objective-appending behavior in technique identity. |
pyrit/models/seeds/seed_group_requirements.py |
Adds user-prompt-only validation. |
pyrit/models/seeds/attack_technique_seed_group.py |
Adds the objective-appending option. |
pyrit/models/seeds/attack_seed_group.py |
Appends an objective prompt during technique merging. |
Review effort: Balanced
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
The sequence collision described here and in the inline comment is fixed in 6b6d3cd. Objective generation now happens after composition/normalization and accounts for all static prompts and complete simulation reservations, making the objective a separate final user turn under preserve or prepend. The shared default-false behavior and authored multipart groups remain unchanged. In reply to #3657 (review) |
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; it does not authorize merge.

Description
Many persona-based jailbreak templates (including long-context persona framing prompts) are designed as a two-step user handshake rather than a single-turn inline wrapper:
"Tailwind, hawk, snap, ready to build.").When evaluated with single-turn
prompt_sending, inline{{ prompt }}injection can disrupt the handshake structure, or the scorer evaluates Turn 1's handshake acknowledgment before any objective has been requested. Nativejailbreak_system_promptdelivery avoids inline injection, but requires targets that expose a writable system prompt.This PR adds an opt-in delivery technique,
jailbreak_multi_turn, to theJailbreakscenario:MultiPromptSendingAttack.role="user",sequence=0): Sends the rendered jailbreak framing (get_jailbreak_system_prompt()) and records the target's handshake response without scoring it.role="user",sequence=1): Sends that run's objective as a follow-up user turn in the same conversation and scores only the final response.MultiPromptSendingAttackParameters.from_seed_group_asyncso that when a general-technique seed group (is_general_technique=True) is merged into an objective-onlyAttackSeedGroup, the objective is appended as the final user turn.Tests
uv run pytest tests/unit/scenario/airt/test_jailbreak.py tests/unit/executor/attack/multi_turn/test_multi_prompt_sending.py