Repository navigation
FEAT: Add DecisionOptionOrderConverter - #4311
Open
Dev3 ^~^ (shashank03-dev) wants to merge 3 commits into
Open
Dev3 ^~^ (shashank03-dev) wants to merge 3 commits into
Dev3 ^~^ (shashank03-dev) wants to merge 3 commits into
Conversation
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The review identifies no active issues with the decision-option ordering converter change.
No open issues 🎉
What changed in this PR
Adds DecisionOptionOrderConverter to PyRIT’s converter framework as a focused step toward decision-model robustness testing in #3246, split from #3250.
Changes:
- Reverses or reproducibly shuffles choice options while preserving state, option definitions, and other question types.
- Exposes the converter through lazy package exports and adds usage documentation.
- Adds tests for validation, deterministic shuffling, registry resolution, and compatibility with
PromptNormalizerand a mocked decision SDK.
| File | Description |
|---|---|
tests/unit/converter/test_seeded_converter_determinism.py |
Includes the new converter in shared determinism tests. |
tests/unit/converter/test_decision_option_order_converter.py |
Tests reordering, validation, identifiers, and integration paths. |
pyrit/converter/decision_option_order_converter.py |
Implements reverse and shuffle modes for choice options. |
pyrit/converter/__init__.py |
Adds lazy export and static typing support. |
doc/code/converters/1_text_to_text_converters.py |
Adds a decision-request reordering example. |
doc/code/converters/1_text_to_text_converters.ipynb |
Adds the matching notebook example and saved output. |
doc/code/converters/0_converters.ipynb |
Includes the converter in the modality table. |
Review effort: Balanced
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to #3246, part 2 of the split proposed in #3250.
Adds
DecisionOptionOrderConverter, which reverses or shuffles the options of eachchoicequestion in a decision request and leaves the state and other questions unchanged. Option order shouldn't change a decision, so a changed answer is a flip. The state converter from #3250 isn't included yet; I asked there whether it should become a general JSON-field converter.Tests: 38,981 passed, 30 skipped, 3 xfailed; lint and full type checks pass; diff coverage 100%. Tests cover seeded determinism, shuffles never returning the original order, the
PromptNormalizerpath, a converted request sent throughLiteLLMDecisionTargetto a mocked SDK, and registry resolution oforder.