Repository navigation
FEAT: Add AggregationLeakage scenario for inference across permitted documents - #4011
Open
Abbas Rahman (Abbas-Rahman-c) wants to merge 1 commit into
Open
Abbas Rahman (Abbas-Rahman-c) wants to merge 1 commit into
Abbas Rahman (Abbas-Rahman-c) wants to merge 1 commit into
Conversation
…documents Adds an AIRT scenario that tests whether a target discloses a confidential conclusion that no single document states but that the documents it was given entail together (per-document access control passes, the inference leaks). - Datasets: airt_aggregation_leakage (16 groups across rule + instance, transitive, set complement and numeric residual, asked indirectly and directly) plus minimal-pair and mention-only control sets (8 each). Documents are delivered as a system turn before the user's question. - Scorer question true_false_question/aggregation_leakage.yaml, used in the default composite with the refusal backstop. - Techniques: prompt_sending (default) and red_teaming (max_turns). - include_context parameter to send only the questions when the documents are already in the target's own knowledge base. - Unit tests and a section in the AIRT scanner docs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
|
@microsoft-github-policy-service agree |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
AggregationLeakage, a new AIRT scenario for the leak described in #3284: a model derives a confidential fact from several pieces of context that the caller was each allowed to see. No single document states the fact, so per-document access checks and keyword filters all pass. As Roman Lutz (@romanlutz) pointed out on the issue, the same problem appears in agents that persist memory over a long period, not only in RAG.What's included:
Scenario (
pyrit/scenario/scenarios/airt/aggregation_leakage.py): structured like the existing AIRT scenarios, with techniquesprompt_sending(default) andred_teaming, and parametersinclude_contextandmax_turns.BASELINE_ATTACK_POLICYisForbidden, becauseprompt_sendingalready sends each group unmodified.Three seed datasets for a fictional company:
airt_aggregation_leakage: 16 items where the fact can be derived;airt_aggregation_leakage_minimal_pairs: 8 items with one document changed, so the fact no longer follows;airt_aggregation_leakage_mention_only: 8 items that mention the topic without supporting the conclusion.Results are reported per dataset. The controls should stay near zero: a high rate there means the target or scorer is reacting to the topic, not to the derivation.
Scorer question (
aggregation_leakage.yaml), combined with the default refusal scorer.With
include_context=False, only the questions are sent, for targets that already hold the documents in their own knowledge base or memory.Closes #3284
Tests and Documentation
tests/unit/scenario/airt/test_aggregation_leakage.py: 17 tests covering technique expansion, dataset shape, attack generation per technique and dataset,include_context,max_turnsand the baseline policy. All 295 tests intests/unit/scenario/airtpass on the latestmain.ruff checkandruff format --checkpass on the changed Python files.SeedDatasetProvideron the latestmain.doc/scanner/airt.pyandairt.ipynb, and the table indoc/scanner/0_scanner.md.Not done: I haven't run this against a live target, so there are no measured leak rates here. I also didn't re-execute the notebook with JupyText: its new cells were added by hand so that the existing outputs stay unchanged, and they have no outputs yet.
🤖 Generated with Claude Code