Skip to content

FIX: Count each jailbroken response once in FuzzerGenerator - #4231

Merged
hannahwestra25 merged 1 commit into
microsoft:mainfrom
siddhant-rajhans:fix/fuzzer-count-jailbroken-responses
Nov 20, 2026
Merged

hannahwestra25 merged 1 commit into
microsoft:mainfrom
siddhant-rajhans:fix/fuzzer-count-jailbroken-responses

Conversation

@siddhant-rajhans

@siddhant-rajhans Siddhant Rajhans (siddhant-rajhans) commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Description

Fixes #4230. FuzzerGenerator counted a jailbreak for every passing score. A scorer that returns several scores per response, such as AzureContentFilterScorer with one per harm category, made one jailbroken response count several times, listed its conversation several times in jailbreak_conversation_ids, and could stop at target_jailbreak_goal_count before that many responses were jailbroken.

  • _process_scoring_results counts a response once when any of its scores passes and records its conversation once. Attribution by the scored piece id (FIX: attribute fuzzer jailbreak scores through the scored piece id #3300) is unchanged.
  • The MCTS reward is now jailbroken responses over responses. FIX: bound the Fuzzer MCTS reward by the scored query count #3628 kept it in [0, 1] by dividing by the number of scores; with each response counted once, dividing by the number of responses keeps the same bound and makes the reward the template's jailbreak rate, as in GPTFuzzer's MCTS-Explore (succ_num / (len(questions) * len(prompt_nodes)), one verdict per response). A response that gets no score now counts as a response that was not jailbroken, where before it was left out of the rate.

The repro from #4230 (goal of 2 jailbreaks, two of four categories above the threshold):

  • main: queries: 1, jailbreak conversations: 2 (1 distinct)
  • this PR: queries: 2, jailbreak conversations: 2 (2 distinct)

Copilot AI balanced review requested due to automatic review settings November 12, 2026 04:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

A scorer can return several scores for one response, for example
AzureContentFilterScorer with one score per harm category. The fuzzer
counted every passing score as a jailbreak, so one response could be
counted and listed several times, and target_jailbreak_goal_count could
be reached before that many responses were jailbroken. A response now
counts once when any of its scores passes, and the MCTS reward is the
jailbroken share of the template's responses, as in GPTFuzzer.

Signed-off-by: Siddhant Rajhans <51508945+siddhant-rajhans@users.noreply.github.com>

@hannahwestra25 hannahwestra25 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for contributing!

@siddhant-rajhans

Copy link
Copy Markdown
Contributor Author

Thanks for the review! The two red checks look unrelated to this change. main-job (windows-latest, 3.13, pyrit, dev) failed in test_websocket_copilot_target.py::TestConnectAndSend::test_connect_and_send_overall_timeout_during_stream, a timing test. Build latest's only error is the link check on https://genai.owasp.org/llmrisk/llm02-insecure-output-handling/ in 1_true_false_scorers.ipynb, which returned 403 to the runner. I'm happy to update the branch if a fresh run would help.

@hannahwestra25
hannahwestra25 added this pull request to the merge queue Nov 20, 2026
Merged via the queue into microsoft:main with commit b866293 Nov 20, 2026
97 of 101 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: FuzzerGenerator counts one jailbroken response once per passing score

3 participants