Skip to content

FIX: Count jailbroken responses once per response in FuzzerGenerator (#4230) - #4317

Closed
Hrushikesh Ramilla (Hrushikesh-ramilla) wants to merge 4 commits into
microsoft:mainfrom
Hrushikesh-ramilla:fix/fuzzer-per-response-jailbreak-count-4230
Closed

Hrushikesh Ramilla (Hrushikesh-ramilla) wants to merge 4 commits into
microsoft:mainfrom
Hrushikesh-ramilla:fix/fuzzer-per-response-jailbreak-count-4230

Conversation

@Hrushikesh-ramilla

Copy link
Copy Markdown
Contributor

Fixes #4230.

Description

When multi-score evaluators (such as AzureContentFilterScorer, which returns scores across multiple harm categories per response) were used with FuzzerGenerator, _process_scoring_results incremented jailbreak_count and appended the response's conversation_id for every passing score category. Consequently:

  • A single response triggering multiple harm categories was counted as multiple jailbreaks, prematurely terminating fuzzing before target_jailbreak_goal_count distinct jailbroken responses were obtained.
  • Duplicate conversation IDs were recorded in FuzzerResult.jailbreak_conversation_ids.
  • In _step_async, _update_mcts_rewards was passed num_queries=len(scores) rather than the number of responses, measuring the fraction of score categories passed rather than the jailbreak success rate across the question batch.

This PR fixes these issues:

  • Groups passing scores by response piece (message_piece_id), attributing at most one jailbreak verdict per target response.
  • Records each jailbroken response's conversation_id exactly once in context.jailbreak_conversation_ids.
  • Updates the MCTS reward denominator in _step_async to use the number of response queries (len(responses)), ensuring the reward represents the proportion of target responses that achieved a jailbreak (bounded in [0, 1]).
  • Added unit and end-to-end regression tests verifying deduplicated conversation tracking and correct query-limit termination with multi-category scorers.

…icrosoft#4230)

Ensure FuzzerGenerator._process_scoring_results attributes jailbreak verdicts per target response rather than per passing score category, preventing duplicate conversation IDs and premature goal completion when multi-score evaluators are used. Update MCTS reward calculation to divide jailbreaks by the count of target responses.

Signed-off-by: hrushikeshramilla <ramillahrushikesh@gmail.com>
Copilot AI balanced review requested due to automatic review settings November 15, 2026 14:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A nullable conversation ID causes a type error in the required ty check, and the execution test lacks assertions to catch an incorrect MCTS reward calculation.

2 open issues
What changed in this PR

Fixes #4230 by making FuzzerGenerator count successful responses rather than passing score categories, preventing premature goal completion with multi-category scorers.

Changes:

  • Deduplicates jailbreak counts and conversation tracking by response piece.
  • Calculates MCTS rewards using the number of responses rather than scores.
  • Adds unit regressions for multi-category scoring, missing conversation IDs, and goal-based termination.
File Description
tests/​unit/​executor/​promptgen/​fuzzer/​test_fuzzer.py Adds response-level counting and execution regression tests.
pyrit/​executor/​promptgen/​fuzzer/​fuzzer.py Deduplicates successful responses and corrects the reward denominator.

Review effort: Balanced

for piece in response_pieces:
if str(piece.id) in jailbroken_piece_ids:
jailbreak_count += 1
context.jailbreak_conversation_ids.append(piece.conversation_id)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Narrowed piece.conversation_id explicitly with a None check before appending to context.jailbreak_conversation_ids, ensuring the attribute type is properly narrowed for type checkers.

Comment on lines +754 to +767
with (
patch.object(target, "_send_prompt_to_target_async", new_callable=AsyncMock, side_effect=respond_async),
patch.object(
converter,
"convert_async",
new_callable=AsyncMock,
return_value=ConverterResult(output_text="{{ prompt }}", output_type="text"),
),
):
result = await generator.execute_async(
prompts=["objective"],
prompt_templates=["{{ prompt }}"],
max_query_limit=10,
)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added verification wrapping _mcts_explorer.update_rewards to assert that each successful query's MCTS reward is recorded at the true response-level success rate (reward == 1.0) across all queries.

…uzzer execution test

Signed-off-by: hrushikeshramilla <ramillahrushikesh@gmail.com>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix for multi-category scorers inflating jailbreak_count makes sense, and the end-to-end test asserting reward == 1.0 is a good regression guard. One question about units:

  1. Per piece vs. per response. _process_scoring_results now counts per entry in response_pieces, while _update_mcts_rewards is given num_queries=len(responses). If response_pieces is flattened from responses, a single message with two jailbroken pieces would be counted twice, and reward = jailbreak_count / num_queries could exceed 1, which the comment says it must not. The title says "once per response". Should the counting group by message (or conversation) instead? I haven't seen how response_pieces is built, so I may be missing a guarantee of one piece per response.
  2. Tests and docs. Please add a multi-piece message case to the tests. The docstring should also state that any passing score on a response counts it as a jailbreak, since that is a semantic choice for multi-category scorers.

…multi-piece test

Signed-off-by: hrushikeshramilla <ramillahrushikesh@gmail.com>
@Hrushikesh-ramilla

Copy link
Copy Markdown
Contributor Author

Thank you Varun Poojari (@Varunpoojari) for the thorough review and great catch on the per-piece vs. per-response distinction!

Updated in commit d474d9287:

  1. Per-response grouping: Grouped jailbreak attribution at the Message level (any(str(piece.id) in jailbroken_piece_ids for piece in response.message_pieces)). Regardless of whether a response contains a single piece or multiple pieces with passing scores, the response contributes at most 1 to jailbreak_count and records its conversation_id once, keeping reward = jailbreak_count / num_queries strictly bounded in [0, 1].
  2. Docstring: Documented in _process_scoring_results that any passing score on a response piece attributes a jailbreak to that response, counting at most once per response message even if multiple categories or pieces pass.
  3. Multi-piece regression test: Added test_process_scoring_results_multi_piece_response_counts_once verifying that a multi-piece response where multiple pieces receive passing scores is counted as a single jailbreak. All 53 unit tests pass cleanly.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick update. I read the diff at d474d92 and both of my earlier points look addressed:

  1. Per-response counting. Grouping by response.message_pieces and checking any(str(piece.id) in jailbroken_piece_ids ...) means a multi-piece message adds at most 1 to jailbreak_count, so jailbreak_count / len(responses) stays within [0, 1]. The new test_process_scoring_results_multi_piece_response_counts_once covers exactly this case.
  2. Docstring. The "any passing score attributes a jailbreak to the response" semantics are now stated in _process_scoring_results.

One small optional point, from the diff only, so I may be missing a guarantee elsewhere: the conversation_id is None check inside the first loop now only validates the scored piece's id, while the id that is recorded comes from response.message_pieces[0] in the second loop. If pieces in one message always share a conversation_id this is moot. Otherwise it may be simpler to do the None check only where the id is recorded, or to record the id of the jailbroken piece. Also, a score whose message_piece_id matches no response piece would raise "Response piece has no conversation_id", which could be misleading for that case, though that behaviour predates this PR.

I haven't run the tests locally.

…idate where recorded

Signed-off-by: hrushikeshramilla <ramillahrushikesh@gmail.com>
@Hrushikesh-ramilla

Copy link
Copy Markdown
Contributor Author

Thank you Varun Poojari (@Varunpoojari)! Adopted your suggestion in commit d6a08a9:

  1. Jailbroken piece conversation ID: Recorded conversation_id directly from the matching jailbroken piece (jailbroken_pieces[0].conversation_id) rather than arbitrarily picking
    esponse.message_pieces[0].
  2. Simplified validation: Removed the redundant conversation_by_piece_id map and narrowed/validated conversation_id is None solely where recorded, eliminating the misleading error path for scores without a matching response piece.

All 53 unit tests in est_fuzzer.py and formatting/lint checks pass cleanly.

@siddhant-rajhans

Copy link
Copy Markdown
Contributor

This is the same fix as #4231, which has been open for #4230 since Nov 12: each jailbroken response counts once, its conversation id is recorded once, and the MCTS reward is divided by the number of responses.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: FuzzerGenerator counts one jailbroken response once per passing score

4 participants