Repository navigation
[BREAKING] FIX: Do not score write-only target deliveries as attack responses - #3757
Merged
Richard Lundeen (richlundeen) merged 19 commits intoNov 17, 2026
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Write-only first-level branches can all be pruned, leaving an UNDETERMINED result with an empty conversation ID and risking misclassification of the outgoing request as the last response.
1 open issue
What changed in this PR
This PR addresses #3701 by distinguishing write-only deliveries from target responses, preventing exported requests from producing false attack success results.
Changes:
- Adds an explicit write-only delivery marker and guards attack and backend response scoring.
- Preserves side-effect scoring for eligible scorers while handling unanswered turns appropriately.
- Adds regression coverage and updates notebook output.
| File | Description |
|---|---|
| tests/unit/score/test_scorer_condition_contracts.py | Tests leaf evidence types through scorer wrappers. |
| tests/unit/score/test_message_trace_scoring.py | Tests composite file and trace scoring without responses. |
| tests/unit/score/test_file_write_scorer.py | Tests write-only file effects and stored results. |
| tests/unit/prompt_normalizer/test_prompt_normalizer.py | Tests delivery detection independently of message role. |
| tests/unit/executor/attack/test_write_only_objective_target.py | Adds cross-attack write-only regression coverage. |
| tests/unit/executor/attack/single_turn/test_prompt_sending.py | Updates scoring-call expectations. |
| tests/unit/executor/attack/multi_turn/test_tree_of_attacks.py | Tests unanswered branch handling. |
| tests/unit/executor/attack/multi_turn/test_multi_prompt_sending.py | Updates final-response scoring expectations. |
| tests/unit/executor/attack/multi_turn/test_crescendo.py | Tests rejection of write-only deliveries. |
| tests/unit/executor/attack/core/test_attack_scoring.py | Tests side-effect-only scorer selection. |
| tests/unit/backend/test_chat_scoring.py | Tests role-independent delivery scoring guards. |
| pyrit/score/scorer.py | Exposes supported evidence types per leaf. |
| pyrit/prompt_normalizer/prompt_normalizer.py | Marks returned requests as write-only deliveries. |
| pyrit/executor/attack/single_turn/prompt_sending.py | Separates delivery evidence from response scoring. |
| pyrit/executor/attack/multi_turn/tree_of_attacks.py | Records unanswered branches and supports undetermined results. |
| pyrit/executor/attack/multi_turn/red_teaming.py | Rejects deliveries without required responses. |
| pyrit/executor/attack/multi_turn/multi_prompt_sending.py | Judges the final send without substituting earlier responses. |
| pyrit/executor/attack/multi_turn/crescendo.py | Rejects deliveries without required responses. |
| pyrit/executor/attack/multi_turn/chunked_request.py | Excludes delivered requests from combined response scoring. |
| pyrit/executor/attack/core/attack_scoring.py | Filters scoring to eligible side-effect scorers. |
| pyrit/backend/services/message_send_service.py | Prevents chat deliveries from being scored as responses. |
| doc/code/memory/5_advanced_memory.ipynb | Updates displayed undetermined-outcome reasons. |
Review effort: Balanced
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
approved these changes
Nov 16, 2026
Richard Lundeen (richlundeen)
left a comment
Contributor
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.
…ault Replace the _WriteOnlyDelivery marker with the supports_responses capability as the single contract: - PromptTarget._SUPPORTS_RESPONSES masks supports_responses to False through the existing capability masking (defaults, custom configurations, apply_capabilities, and saved capability scopes). TextTarget and AzureBlobStorageTarget set it; the blob URL is a receipt, not a reply. - AttackStrategy requires RESPONSES by default. PromptSendingAttack and its subclasses opt out, as does SequentialAttack, whose children validate themselves. TAP, PAIR, red teaming, Crescendo, multi-prompt, chunked, and barge-in now reject write-only targets at construction, so their runtime write-only handling is removed. - Scorable types declare RECORDS_SIDE_EFFECTS, and Scorer.judges_side_effects replaces get_leaf_scorable_types and the attack-layer type set. - The normalizer and backend ask the target instead of checking a marker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep delivery anchors separate from target responses when publishing chat evaluations, and retain inherited file-effect criteria. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Compare canonical target-conversation anchors without assigning a synthetic response, and preserve newer verdicts under reverse completion. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use a strict newer-than comparison for inclusive memory queries, matching existing response recency semantics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
approved these changes
Nov 17, 2026
Richard Lundeen (richlundeen)
left a comment
Contributor
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Storage spaces out tied multipart piece timestamps without changing in-hand pieces, so on a coarse clock an equal-time delivery in another conversation looked newer and current write-only verdicts were skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ends The count=2 send evaluates both conversations concurrently, so the newest delivery's score can be created first; delivery-ordered publication stays covered by the deterministic ordering tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Description
Fixes #3701. A write-only target such as
TextTargetreturns no response, soPromptNormalizerhands back the sent request, andPromptSendingAttackscored that request as if it were the target's reply. A scorer matching the objective then reported SUCCESS with no response at all. Targets now declare whether they reply, and attacks ask the target before scoring.supports_responsescapability (defaultTrue). A target class that never replies sets_SUPPORTS_RESPONSES = False, which keeps the capability False under every configuration, including custom configurations and saved capability snapshots.TextTargetandAzureBlobStorageTargetdeclare it; the blob URL is a delivery receipt, not a reply.supports_responsesby default and reject a write-only target at construction.PromptSendingAttackand its subclasses (ManyShotJailbreakAttack,SkeletonKeyAttack) accept write-only targets, as doesSequentialAttack, whose child attacks validate their own targets.PromptSendingAttacknever scores the returned message as a response. Scorers that judge side effects (FileWriteScorer,OtelToolCallScorer, and composites of them) still judge the turn; evidence types declareRECORDS_SIDE_EFFECTS, andScorer.judges_side_effects()checks every leaf. With no such scorer, the result isUNDETERMINEDwith no last response.PromptNormalizerreturns the sent request only for a target that declares no responses; any other target that returns nothing gets an empty error response. The backend chat send also skips scoring for write-only targets.Breaking changes
RedTeamingAttack,TreeOfAttacksWithPruningAttack,PAIRAttack,CrescendoAttack,MultiPromptSendingAttack,ChunkedRequestAttack, andBargeInAttacknow raise at construction forTextTarget,AzureBlobStorageTarget, or any target declaringsupports_responses=False. These attacks need the target's replies; usePromptSendingAttackfor write-only targets._SUPPORTS_RESPONSES = Falseon the target class, orsupports_responses=Falsein its configuration.TARGET_REQUIREMENTSshould keepCapabilityName.RESPONSESunless they support write-only targets.co-written by richlundeen