Repository navigation
FEAT: Add ubbi_dubbi Encoding technique via SearchReplaceConverter - #4448
Roman Lutz (romanlutz) wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The converter’s regex can stall the event loop through quadratic backtracking on punctuation-heavy inputs and leaves newline-containing tokens unencoded.
1 open issue
What changed in this PR
Adds a configurable Ubbi Dubbi encoder to PyRIT’s converter library and makes it available in the Garak Encoding scenario.
Changes:
- Implements vowel-group encoding with configurable vowels and word selection.
- Exports the converter and registers
ubbi_dubbiwithout changing the scenario’s default technique set. - Adds unit coverage, identifier samples, and a Python documentation example.
| File | Description |
|---|---|
tests/unit/scenario/garak/test_encoding.py |
Updates technique counts and ALL-only expectations. |
tests/unit/registry/converter_identifier_samples.py |
Adds vowel configurations for identifier checks. |
tests/unit/converter/test_ubbi_dubbi_converter.py |
Tests encoding, capitalization, punctuation, and validation. |
pyrit/scenario/scenarios/garak/encoding.py |
Registers the Ubbi Dubbi technique and converter variant. |
pyrit/converter/ubbi_dubbi_converter.py |
Implements the configurable word-level converter. |
pyrit/converter/__init__.py |
Adds lazy export and typing support. |
doc/code/converters/1_text_to_text_converters.py |
Adds an import and usage example. |
Review effort: Balanced
| # Splits a whitespace token into (leading non-letters, alphabetic-ish core, trailing non-letters). | ||
| # The core is non-greedy so leading/trailing punctuation is peeled off while interior characters | ||
| # (e.g. the apostrophe in "don't") stay inside the core and are passed through untouched. | ||
| _TOKEN_PATTERN = re.compile(r"^([^A-Za-z]*)(.*?)([^A-Za-z]*)$") |
There was a problem hiding this comment.
[P2] Also remove the silent fallback
I agree, and I found more. The comment on line 111 says that the pattern always matches, but it does not match a token with a line break. Lines 112-113 then return the plaintext word with no error, and # pragma: no cover hides this branch from coverage. CRLF has the same problem: line one\r\nline two becomes lubinube one\r\nline twubo, so the two words at each line break stay plaintext. On head, 'a' + '!' * n + 'b' takes about 0.9 s at n=10,000 and 14 s at n=40,000 on the event loop. Unspaced CJK text between ASCII letters is a realistic trigger: about 0.9 s at 10,000 characters and 3.7 s at 20,000. The preferred fix is to build the class on SearchReplaceConverter (see the comment on line 20), which removes this regex. If you keep the token parser, replace _TOKEN_PATTERN with a scan for the first and last ASCII letter, delete the guard and the pragma, and add tests for hello\nworld -> hubellubo\nwuborld and for a long non-letter run between two letters.
There was a problem hiding this comment.
Withdrawn: this earlier suggestion to drop the named converter in favor of configuring an existing one no longer applies. Named converters are welcome; see the current review.
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Implementation review at dd1a23d. The named UbbiDubbiConverter and its ALL-only ubbi_dubbi technique are welcome; these comments are about how the class is built (build it on SearchReplaceConverter, or one regex substitution on WordLevelConverter), the line-break and backtracking defect in the existing Copilot thread, the unused vowels option, and docs and test conventions.
| _DEFAULT_VOWELS = "aeiou" | ||
|
|
||
|
|
||
| class UbbiDubbiConverter(WordLevelConverter): |
There was a problem hiding this comment.
[P2] Build the converter on SearchReplaceConverter
Keep the named class, but do not reimplement the substitution. The class adds a token regex, a vowel loop, and a casing layer (lines 13 and 80-129), but SearchReplaceConverter already expresses this as one regex substitution. The casing layer lowercases each mixed-case word, so the output does not decode to the input: iPhone becomes ubiphubonube and PyRIT becomes Pyrubit. In garak.encoding, 13 of the 17 default garak_web_html_js seeds do not decode back to the seed. For example, String.fromCharCode decodes to string.fromcharcode, and the base64 in the data: URL seed is lowercased, so it decodes to different bytes, not the script. The default DecodingScorer ignores case, so scores do not change, but the converter loses information that a letter-preserving cipher does not need to lose. Make the class a subclass of SearchReplaceConverter with a no-argument __init__ that calls super().__init__(pattern=r"([aeiou]+)", replace=r"ub\1", regex_flags=re.IGNORECASE). I tested this form: all 17 seeds round-trip, hello\nworld becomes hubellubo\nwuborld, and the identifier records the pattern, replacement, and flags. The output keeps the source case and inserts a lowercase ub (Apple becomes ubApplube). Then remove _TOKEN_PATTERN, _DEFAULT_VOWELS, _transform_core, the casing block, _build_identifier, and their tests, and use SelectiveTextConverter for word selection. If you must keep built-in word selection, keep WordLevelConverter and make convert_word_async one compiled-regex substitution that keeps the source letters, like ROT13Converter's one-line convert_word_async.
| def __init__( | ||
| self, | ||
| *, | ||
| vowels: str = _DEFAULT_VOWELS, |
There was a problem hiding this comment.
[P2] Remove the vowels option
No caller sets vowels: the Encoding scenario and the doc example use the default, and only tests set it. The framework says not to add options for hypothetical needs, and a public parameter needs a deprecation to remove later. The option also does not do what the docstring says. The punctuation regex treats only ASCII letters as letters, so a non-ASCII vowel at a word edge never gets ub: UbbiDubbiConverter(vowels="aeiouéè") leaves été unchanged. Remove vowels, _DEFAULT_VOWELS, the empty-vowels ValueError, the _build_identifier override, the UbbiDubbiConverter.vowels identifier sample, and the four vowel tests. The SearchReplaceConverter change on line 20 also removes this option.
| print("ROT13:", await ROT13Converter().convert_async(prompt=prompt)) # type: ignore | ||
| print("Base64:", await Base64Converter().convert_async(prompt=prompt)) # type: ignore | ||
| print("Base2048:", await Base2048Converter().convert_async(prompt=prompt)) # type: ignore | ||
| print("Ubbi Dubbi:", await UbbiDubbiConverter().convert_async(prompt=prompt)) # type: ignore |
There was a problem hiding this comment.
[P2] Update the .ipynb pair of this notebook
This change adds the UbbiDubbiConverter import and print line to the .py file only. 1_text_to_text_converters.ipynb has no Ubbi Dubbi text, and the docs site shows the .ipynb. The framework says that each percent-format .py file stays in sync with its .ipynb pair, and docs.instructions.md says never to edit only one file of the pair. Add the same import and print line to the .ipynb code cell, as the BifidConverter PR (#4309) did.
| LeetSpeak = ("leet_speak", {"default"}) | ||
| AsciiSmuggler = ("ascii_smuggler", set[str]()) | ||
| Bifid = ("bifid", set[str]()) | ||
| UbbiDubbi = ("ubbi_dubbi", set[str]()) |
There was a problem hiding this comment.
[P3] List ubbi_dubbi in the garak scanner guide
This line adds the ubbi_dubbi technique, but doc/scanner/garak.py and garak.ipynb still say "Available techniques (18 encodings)" and end the list at Bifid. A reader of the scanner guide does not find ubbi_dubbi as a --techniques value. Add UbbiDubbi to the list in both files, and set the count to match the enum after you rebase. Main already has ColumnarTransposition (#4327), which is also missing from the list, so the count is 20 if no other new technique merges first. The BifidConverter PR (#4309) updated both files in the same way.
| from pyrit.converter.toxic_sentence_generator_converter import ToxicSentenceGeneratorConverter | ||
| from pyrit.converter.translation_converter import TranslationConverter | ||
| from pyrit.converter.transparency_attack_converter import TransparencyAttackConverter | ||
| from pyrit.converter.ubbi_dubbi_converter import UbbiDubbiConverter |
There was a problem hiding this comment.
[P3] Add the converter to the modality table output
converters.instructions.md says that a new converter must update the modality table in doc/code/converters/0_converters.ipynb. The .py file builds the table from get_converter_modalities(), but the saved .ipynb output has no row for UbbiDubbiConverter, so the docs site does not show it. Add the UbbiDubbiConverter row to the saved table output in 0_converters.ipynb, as the FullwidthConverter PR (#4185) did. Running jupytext --execute on this notebook also works, but it needs OpenAI credentials because later cells call OpenAIChatTarget. BifidConverter (and ColumnarTranspositionConverter after you rebase) are missing too. Other open PRs in this series change the same output, so one update after the series merges is also acceptable if you say so in the PR.
| from pyrit.converter import UbbiDubbiConverter | ||
|
|
||
|
|
||
| @pytest.mark.asyncio |
There was a problem hiding this comment.
[P3] Remove the asyncio marks and parametrize the tests
test.instructions.md says not to add @pytest.mark.asyncio, because asyncio_mode = "auto" is set for the project. This file adds the mark 18 times. Also, 14 tests have the same shape: make a converter, convert one input, and compare the output. The framework says to parametrize instead of copying tests. Remove the marks, put the input and output cases in one @pytest.mark.parametrize table, and keep the error and identifier tests separate.
dd1a23d to
3f0ec48
Compare
|
Reworked per your suggestion: dropped |
3f0ec48 to
3f809d9
Compare
Per review (reuse and compose before adding): keep the ubbi_dubbi technique but implement it with SearchReplaceConverter instead of a dedicated UbbiDubbiConverter class. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
3f809d9 to
028ba68
Compare
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Follow-up review at 3365164. The rework to one SearchReplaceConverter row in the Encoding scenario removes the causes of my earlier comments on the converter class, its docs, and its tests. One new P3 is on the new test. The earlier P3 about the garak scanner guide (r4638761720) still applies: doc/scanner/garak.py and garak.ipynb still list 18 encodings that end at Bifid.
|
|
||
| async def test_ubbi_dubbi_technique_inserts_ub_before_vowels(): | ||
| """The ubbi_dubbi Encoding technique (a SearchReplaceConverter config) prefixes vowel runs with 'ub'.""" | ||
| converter = SearchReplaceConverter(pattern=r"([aeiouAEIOU]+)", replace=r"ub\1") |
There was a problem hiding this comment.
[P3] Test the registered ubbi_dubbi row
This test makes its own SearchReplaceConverter with a copy of the regex. Thus, it tests SearchReplaceConverter, not the new Encoding technique. If you delete the ubbi_dubbi row in encoding.py, or change its pattern so that it matches nothing, all tests still pass. I patched the row to pattern="unmatchable", and the 33 tests in the two changed files passed. Get the converter from Encoding instead, for example the ubbi_dubbi row from Encoding._converter_variants(), or the converter stack of the selected EncodingTechnique.UbbiDubbi attack as the base64 tests in test_encoding.py do. Then assert its output. Put the case in test_encoding.py with a small @pytest.mark.parametrize table, for example speak -> spubeak, queue -> qubueue, and HELLO -> HubELLubO, and delete this separate file.
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.
[P3] inline:4638761720: R5: doc/scanner/garak.py:139-141 and garak.ipynb:306-308 still say 'Available techniques (18 encodings) ... AsciiSmuggler, Bifid' and omit ubbi_dubbi, which this PR adds (ALL now has 20 techniques). The PR changes no docs. Supported, unfixed, P3 docs discoverability (operators can still select ubbi_dubbi by name); feedback/feedback.json R5. Thread is current and unresolved; follow-up review 5996808015 restates it.
[P3] inline:4645621954: C1: tests/unit/scenario/garak/test_ubbi_dubbi_technique.py imports only SearchReplaceConverter and copies the regex, so it cannot fail if the registered encoding.py:324 row is deleted or broken (reviewer mutation: 33 passed with the row patched to 'unmatchable'). The live row is currently correct (speak -> spubeak). Supported, unfixed, P3 test-quality improvement; feedback/feedback.json C1.

Adds the ubbi_dubbi Encoding technique via an existing converter (reuse before adding):
SearchReplaceConverter(pattern=r'([aeiouAEIOU]+)', replace=r'ub\1')(prefixes each vowel run withub, source case preserved). Keeps the named technique and a focused test.Ubbi Dubbi — a language game. Source: popularized by PBS's Zoom (1972).
gap-1 converter series: #4441 #4442 #4443 #4444 #4445 #4446 #4447 #4448 #4449 #4450 #4451 #4452 #4454 #4455 #4456 #4457 #4458
🤖 Generated with Claude Code