Repository navigation
FEAT: Add disemvowel Encoding technique via SearchReplaceConverter - #4449
Roman Lutz (romanlutz) wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The converter assigns the same parameter identifier to vowel configurations that produce different results because iterable entries are not normalized into individual characters.
1 open issue
What changed in this PR
Adds DisemvowelConverter to PyRIT’s text transformation tools and integrates it into the garak Encoding scenario as an ALL-only technique.
Changes:
- Implements configurable, case-insensitive vowel removal with component identifiers and lazy public exports.
- Registers the technique and updates scenario and identifier test data.
- Adds converter unit tests and a Python documentation example.
| File | Description |
|---|---|
tests/unit/scenario/garak/test_encoding.py |
Updates technique counts and ALL-only expectations. |
tests/unit/registry/converter_identifier_samples.py |
Adds custom vowel samples for identifier testing. |
tests/unit/converter/test_disemvowel_converter.py |
Tests conversion, validation, and identifiers. |
pyrit/scenario/scenarios/garak/encoding.py |
Registers the disemvowel technique and converter. |
pyrit/converter/disemvowel_converter.py |
Implements configurable vowel removal. |
pyrit/converter/__init__.py |
Exposes the converter through lazy exports. |
doc/code/converters/1_text_to_text_converters.py |
Adds a disemvowel usage example. |
Review effort: Balanced
| Raises: | ||
| ValueError: If ``vowels`` is empty. | ||
| """ | ||
| vowel_chars = {char.lower() for char in vowels} |
There was a problem hiding this comment.
[P2] Remove the vowels option. I agree with this comment, and the problem is wider than lists of whole vowel strings. vowels=['ae', 'i'] records the identifier 'aei' but removes only i and I. vowels=[''] passes the empty check and removes nothing. A generator is consumed in init, so DisemvowelConverter(**converter.get_reconstruction_parameters()) raises ValueError. No code in this PR passes vowels, so instead of adding normalization, fix the vowels to a, e, i, o, u; SearchReplaceConverter already covers other character sets. If you keep the option, accept only str, reject the empty string, and build the removal set and the identifier from the same characters.
There was a problem hiding this comment.
Withdrawn: this earlier suggestion to drop the named converter in favor of configuring an existing one no longer applies. Named converters are welcome; see the current review.
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Full source review at 5787ebb. The named DisemvowelConverter is welcome. The comments below are about how it is built, its option surface, and the docs and tests that go with it. A reply in the existing vowels thread covers the identifier problem.
| from pyrit.models import ComponentIdentifier, PromptDataType | ||
|
|
||
|
|
||
| class DisemvowelConverter(Converter): |
There was a problem hiding this comment.
[P2] Build DisemvowelConverter on SearchReplaceConverter. Keep the named class, but do not reimplement the filter: this class writes its own character loop, input check and identifier, and SearchReplaceConverter already does all three. With the vowels option removed (see the other thread), a subclass whose init calls super().init(pattern='[aeiouAEIOU]', replace='') gives the same output for every Unicode character, including Turkish dotted and dotless I and accented text, and still reconstructs with no parameters. Remove the custom convert_async, _build_identifier and the empty-set check, so later fixes to SearchReplaceConverter also reach this class, and remove the DisemvowelConverter.vowels entry in tests/unit/registry/converter_identifier_samples.py. Do not use re.IGNORECASE: it also deletes U+0130 and U+0131. Also shorten the class docstring to a summary, the 'Hello World' -> 'Hll Wrld' example, and one line that y is kept.
| print("ROT13:", await ROT13Converter().convert_async(prompt=prompt)) # type: ignore | ||
| print("Base64:", await Base64Converter().convert_async(prompt=prompt)) # type: ignore | ||
| print("Base2048:", await Base2048Converter().convert_async(prompt=prompt)) # type: ignore | ||
| print("Disemvowel:", await DisemvowelConverter().convert_async(prompt=prompt)) # type: ignore |
There was a problem hiding this comment.
[P2] Update the notebook pair. This PR adds the DisemvowelConverter import and example to the .py file only, so 1_text_to_text_converters.ipynb, which the docs site shows, has no example and the pair is out of sync. Add the same import and print line, with the output, to the .ipynb, or regenerate it with jupytext --execute. Also re-run the modality table cell in 0_converters.ipynb, because converters.instructions.md requires new converters in that table (it currently also misses BifidConverter).
| LeetSpeak = ("leet_speak", {"default"}) | ||
| AsciiSmuggler = ("ascii_smuggler", set[str]()) | ||
| Bifid = ("bifid", set[str]()) | ||
| Disemvowel = ("disemvowel", set[str]()) |
There was a problem hiding this comment.
[P3] Update the garak scanner docs for the new technique. doc/scanner/garak.py and doc/scanner/garak.ipynb still say "18 encodings" and the list ends at Bifid, so users do not see disemvowel. Change the count to 19 and add Disemvowel to the list in both files, keeping the .py and .ipynb in sync. Also add Bifid and Disemvowel to the ALL-only list in the comment above this enum ("(Braille, Ecoji, Zalgo, AsciiSmuggler)").
There was a problem hiding this comment.
[P3] This still applies at 8f51435. The current count is now 20, not the previously suggested 19: ColumnarTransposition has also merged. Both scanner documentation files still say 18 and omit ColumnarTransposition and Disemvowel. The ALL-only comment also omits Bifid, ColumnarTransposition, and Disemvowel.
| all_names = {t.value for t in EncodingTechnique.get_all_techniques()} | ||
| assert default_names < all_names | ||
| assert len(all_names) == 18 | ||
| assert len(all_names) == 19 |
There was a problem hiding this comment.
[P3] Add a test that links each technique to a converter variant. If the disemvowel row in _converter_variants is deleted or misspelled, all tests still pass and a disemvowel run builds no attacks. Add one test that each non-aggregate EncodingTechnique has at least one variant row and each row names a real technique. This also covers the other ALL-only techniques.
There was a problem hiding this comment.
[P3] The new focused test still does not cover this wiring: it builds a separate SearchReplaceConverter. Removing the actual disemvowel variant only in memory leaves all 68 targeted tests passing. Please exercise the selected Disemvowel technique and its converted raw/decode prompts. One correction to the earlier wording: if that row is missing, selected-only initialization raises IncompatibleTechniqueError in Scenario.initialize_async; an ALL run can still silently omit the technique while running the others.
| from pyrit.converter import ConverterResult, DisemvowelConverter | ||
|
|
||
|
|
||
| async def test_disemvowel_converter_basic(): |
There was a problem hiding this comment.
[P3] Parametrize the default-conversion tests. Nine tests build DisemvowelConverter() and differ only in input and expected text. Replace them with one parametrized test. Also add match= to both pytest.raises(ValueError) calls, so an unrelated ValueError does not pass, and make the empty-vowels test synchronous because it awaits nothing.
5787ebb to
328026f
Compare
|
Reworked per your suggestion: dropped |
Per review (reuse and compose before adding): keep the disemvowel technique but implement it with SearchReplaceConverter(pattern='[AEIOUaeiou]', replace='') instead of a dedicated DisemvowelConverter class. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
328026f to
ed882ec
Compare
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.
[P3] inline:4638732233: Supported F4 remains open. doc/scanner/garak.py:139-141 and garak.ipynb:306-308 still state 18 encodings and end at Bifid. encoding.py:115-133 and the independently rerun actual enum establish 20, including ColumnarTransposition and Disemvowel. encoding.py:112-113's exhaustive ALL-only comment also omits Bifid, ColumnarTransposition, and Disemvowel. The earlier requested 19 is stale; update both scanner formats to 20 and update or simplify the comment. The PR extends this stale catalog to another technique. Documentation/catalog discoverability polish is P3, not a current executable workflow defect.
[P3] inline:4638732238: Supported F5 remains open. test_disemvowel_technique.py:9 builds a separate converter, while test_encoding.py:254-267 checks only enum membership. Reused both exact-head receipts showing 68 targeted passes normally and 68 after process-local removal of the production variant. Independently reran missing-row initialization: selected-only selection raises IncompatibleTechniqueError at scenario.py:1750; ALL still initializes 105 attacks with none for disemvowel. Independently reran actual-row construction: the present row correctly builds five raw/decode attacks. Add a public selected-technique behavior test, not necessarily a private table assertion. This is small regression-sensitivity coverage for currently working configuration-only code, not risky changed control flow or a present runtime failure; retain P3.
[P3] inline:4645599151: Supported useful extension of F4/root inline:4638732233, not a new independent defect. It correctly updates the earlier suggested count from 19 to 20 after ColumnarTransposition merged. Both scanner formats still say 18 and omit both additions; the ALL-only comment omits the three named techniques. Same unresolved P3 docs/catalog correction, retained under both collected IDs without inventing a local finding.
[P3] inline:4645599500: Supported useful extension of F5/root inline:4638732238. It evaluates the new focused test with the supplied independently verified 68-pass missing-row mutation and correctly qualifies the old outcome. This assessor independently reran the actual-row and missing-row probes: the present configuration works; hypothetical removal raises for selected-only but ALL succeeds with 105 other attacks and no disemvowel. The public selected-technique regression-test request remains a P3 improvement. This is an evidence-based extension, not duplicate agreement or a claim that the current row is absent.
[P3] local:F4: Supported, unfixed scanner-catalog cause, identical to inline:4638732233 and its posted extension inline:4645599151. doc/scanner/garak.py:139-141 and garak.ipynb:306-308 still state 18 encodings and end at Bifid; encoding.py:115-133 has 20 concrete techniques, including ColumnarTransposition and Disemvowel, independently confirmed by the prior offline actual-row probe. The ALL-only comment at encoding.py:112-113 omits Bifid, ColumnarTransposition, and Disemvowel. Update the synchronized scanner docs to 20 and update or simplify the comment. The earlier requested 19 is stale. This PR extends the pre-existing catalog omission to a new technique. It is P3 documentation/discoverability polish without a present executable failure. The reply is evidence of reporting, not of repair.
[P3] local:F5: Supported, unfixed scenario-wiring test-sensitivity cause, identical to inline:4638732238 and posted extension inline:4645599500. test_disemvowel_technique.py:9 constructs its own SearchReplaceConverter; test_encoding.py:254-267 checks enum membership rather than the production row. Exact-head full-review and independent-verifier receipts both recorded 68 targeted passes before and after a process-local removal of that row. This assessor independently reran the actual-row probe (five correct raw/decode attacks) and missing-row probe (selected-only IncompatibleTechniqueError at scenario.py:1750; ALL still initializes 105 attacks without disemvowel). A public selected-technique behavior test would detect the hypothetical regression. The current row works, and no risky changed control flow lacks coverage, so this is a P3 regression-sensitivity improvement rather than a current product failure. Posting the conditional correction did not fix the test gap.

Adds the disemvowel Encoding technique by configuring an existing converter rather than adding a new class (per reuse and compose before adding):
SearchReplaceConverter(pattern='[AEIOUaeiou]', replace=''). Keeps the named technique and a focused test.Disemvowel — remove all vowels. Source: the "disemvoweling" comment-moderation technique (popularized on the Making Light blog).
gap-1 converter series: #4441 #4442 #4443 #4444 #4445 #4446 #4447 #4448 #4449 #4450 #4451 #4452 #4454 #4455 #4456 #4457 #4458
🤖 Generated with Claude Code