Repository navigation
FIX: Fingerprint target judgment adaptation in replay contract (#3753) - #3845
Open
Hrushikesh Ramilla (Hrushikesh-ramilla) wants to merge 1 commit into
Conversation
Copilot started reviewing on behalf of
Hrushikesh Ramilla (Hrushikesh-ramilla)
November 2, 2026 23:35
Hrushikesh Ramilla (Hrushikesh-ramilla)
force-pushed
the
fix/judgment-replay-normalizer-adaptation-3753
branch
from
November 2, 2026 23:38
64ea1ad to
bb53c11
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Replay fingerprints omit the wrapped normalizer’s prompt-affecting settings, allowing replay to accept judgments acquired with different prompts.
1 open issue
What changed in this PR
This PR addresses #3753 by including target-side normalizer configuration in PyRIT’s judgment replay contract, preventing reuse when schema instructions or adaptation differ.
Changes:
- Adds parameter-aware normalizer projections to replay fingerprints.
- Passes targets into scorer replay validation while preserving legacy observation compatibility.
- Adds regression tests for template changes, native versus adapted schemas, and SQLite reopen behavior.
| File | Description |
|---|---|
tests/unit/score/test_judgment_observation.py |
Adds adaptation and legacy replay regression tests. |
pyrit/score/true_false/shieldgemma_scorer.py |
Passes target into replay validation. |
pyrit/score/true_false/self_ask_true_false_scorer.py |
Passes target into replay validation. |
pyrit/score/true_false/self_ask_refusal_scorer.py |
Passes target into replay validation. |
pyrit/score/true_false/self_ask_general_true_false_scorer.py |
Passes target into replay validation. |
pyrit/score/true_false/self_ask_category_scorer.py |
Passes target into replay validation. |
pyrit/score/true_false/llamaguard_scorer.py |
Passes target into replay validation. |
pyrit/score/llm_scoring.py |
Fingerprints target adaptation and handles legacy contracts. |
pyrit/score/float_scale/self_ask_scale_scorer.py |
Passes target into replay validation. |
pyrit/score/float_scale/self_ask_likert_scorer.py |
Passes target into replay validation. |
pyrit/score/float_scale/self_ask_general_float_scale_scorer.py |
Passes target into replay validation. |
pyrit/score/float_scale/insecure_code_scorer.py |
Passes target into replay validation. |
pyrit/score/float_scale/decision_scorer.py |
Passes target into replay validation. |
pyrit/prompt_target/common/target_configuration.py |
Projects normalizer types and parameters. |
pyrit/message_normalizer/json_schema_normalizer.py |
Exposes schema instruction template parameters. |
pyrit/message_normalizer/history_squash_normalizer.py |
Exposes history count and child formatter identity. |
pyrit/message_normalizer/generic_system_squash.py |
Exposes an empty parameter projection. |
pyrit/message_normalizer/chat_message_normalizer.py |
Exposes role and system-message settings. |
Review effort: Balanced
Comment on lines
+71
to
+73
| params["message_normalizer"] = self._message_normalizer.get_identifier().model_dump( | ||
| mode="json", exclude={"pyrit_version", "hash", "eval_hash"} | ||
| ) |
Hrushikesh Ramilla (Hrushikesh-ramilla)
force-pushed
the
fix/judgment-replay-normalizer-adaptation-3753
branch
2 times, most recently
from
November 3, 2026 00:25
a7798f3 to
15a2349
Compare
…soft#3753) Fingerprint effective target-side judgment adaptation in the stored replay contract to prevent replaying observations across targets with changed schema instruction templates. Maintain explicit compatibility for legacy observations stored before target adaptation fingerprinting. Signed-off-by: Hrushikesh Ramilla <ramillahrushikesh@gmail.com>
Hrushikesh Ramilla (Hrushikesh-ramilla)
force-pushed
the
fix/judgment-replay-normalizer-adaptation-3753
branch
from
November 6, 2026 16:00
15a2349 to
299468d
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #3753.
When replaying a target-backed judgment observation, previously only the response handler, category, and judgment identity were fingerprinted in the stored replay contract. If an otherwise identical target had a modified
JsonSchemaNormalizerschema-instructions template (or differing normalizer adaptation), the replay check accepted the observation without calling the target, even though the changed instructions alter the judge's effective behavior.This change:
TargetConfiguration.project_pipeline()and includes target adaptation in_contract_fingerprint/_replay_contract_fingerprint.Tests and Documentation
Added comprehensive unit tests in
tests/unit/score/test_judgment_observation.py:test_judgment_replay_rejects_changed_schema_normalizer_template_asynctest_judgment_replay_rejects_native_vs_adapted_json_schema_asynctest_judgment_replay_schema_template_variant_verified_after_sqlite_reopen_asynctest_judgment_replay_legacy_observation_compatibility_asyncAll 86 tests in
test_judgment_observation.pyand existing tests acrossprompt_targetandmessage_normalizerpass cleanly.