Repository navigation
TEST: Cover Fuzzer MCTS search score, stop conditions and score normalization - #4320
Open
Lanre Shittu (Shizoqua) wants to merge 1 commit into
Open
Lanre Shittu (Shizoqua) wants to merge 1 commit into
Lanre Shittu (Shizoqua) wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain for the regression-test additions covering FuzzerGenerator and _MCTSExplorer edge cases.
No open issues 🎉
What changed in this PR
Adds focused regression tests for FuzzerGenerator and _MCTSExplorer, addressing #4319 without production changes. The tests remain separate from the jailbreak-counting work in #4231.
Changes:
- Tests finite step-zero search scores and ordering by node visits and rewards.
- Covers query-limit boundaries and the jailbreak-goal stop condition.
- Verifies score normalization, clamping, and warnings for unexpected types.
| File | Description |
|---|---|
tests/unit/executor/promptgen/fuzzer/test_fuzzer_search_edge_cases.py |
Adds 15 test cases covering search scores, stopping conditions, and score normalization. |
Review effort: Balanced
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Closes #4319.
Adds unit tests for
FuzzerGeneratorand_MCTSExplorerlogic that had no direct coverage:_calculate_uct_score: step 0 gives the same finite score as step 1 instead of takinglog(0), a less visited node scores higher than an often visited one with equal rewards, and a more rewarded node scores higher with equal visits._should_stop_generation: generation stops when the next batch would exceedmax_query_limit(but not when it fits exactly) and whentarget_jailbreak_goal_countis reached._normalize_score_to_float: booleans map to 1.0 and 0.0, numbers are clamped to [0, 1], and unexpected types are treated as 0.0 with a warning.The tests live in a new file so they stay independent of the open jailbreak-counting change in #4231. No production code changes.
Tests and Documentation
Added
tests/unit/executor/promptgen/fuzzer/test_fuzzer_search_edge_cases.py(15 tests). No docs or notebooks changed, so JupyText was not needed.