Skip to content

FEAT: Add DecisionOptionOrderConverter - #4311

Open
Dev3 ^~^ (shashank03-dev) wants to merge 3 commits into
microsoft:mainfrom
shashank03-dev:feat/decision-option-order-converter
Open

Dev3 ^~^ (shashank03-dev) wants to merge 3 commits into
microsoft:mainfrom
shashank03-dev:feat/decision-option-order-converter

Conversation

@shashank03-dev

Copy link
Copy Markdown
Contributor

Related to #3246, part 2 of the split proposed in #3250.

Adds DecisionOptionOrderConverter, which reverses or shuffles the options of each choice question in a decision request and leaves the state and other questions unchanged. Option order shouldn't change a decision, so a changed answer is a flip. The state converter from #3250 isn't included yet; I asked there whether it should become a general JSON-field converter.

Tests: 38,981 passed, 30 skipped, 3 xfailed; lint and full type checks pass; diff coverage 100%. Tests cover seeded determinism, shuffles never returning the original order, the PromptNormalizer path, a converted request sent through LiteLLMDecisionTarget to a mocked SDK, and registry resolution of order.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The review identifies no active issues with the decision-option ordering converter change.

No open issues 🎉

What changed in this PR

Adds DecisionOptionOrderConverter to PyRIT’s converter framework as a focused step toward decision-model robustness testing in #3246, split from #3250.

Changes:

  • Reverses or reproducibly shuffles choice options while preserving state, option definitions, and other question types.
  • Exposes the converter through lazy package exports and adds usage documentation.
  • Adds tests for validation, deterministic shuffling, registry resolution, and compatibility with PromptNormalizer and a mocked decision SDK.
File Description
tests/​unit/​converter/​test_seeded_converter_determinism.py Includes the new converter in shared determinism tests.
tests/​unit/​converter/​test_decision_option_order_converter.py Tests reordering, validation, identifiers, and integration paths.
pyrit/​converter/​decision_option_order_converter.py Implements reverse and shuffle modes for choice options.
pyrit/​converter/​__init__.py Adds lazy export and static typing support.
doc/​code/​converters/​1_text_to_text_converters.py Adds a decision-request reordering example.
doc/​code/​converters/​1_text_to_text_converters.ipynb Adds the matching notebook example and saved output.
doc/​code/​converters/​0_converters.ipynb Includes the converter in the modality table.

Review effort: Balanced

@hannahwestra25 hannahwestra25 self-assigned this Nov 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants