Repository navigation
FIX: Keep combining marks inside words in InsertPunctuationConverter - #4475
Open
Saquib Jawed (saquibjawedbit) wants to merge 1 commit into
Open
Saquib Jawed (saquibjawedbit) wants to merge 1 commit into
Saquib Jawed (saquibjawedbit) wants to merge 1 commit into
Conversation
Between-words mode tokenized words with \w+, which stops at combining accents and Indic vowel signs. Each mark became its own non-word token, so punctuation landed inside a word (decomposed "café" -> "cafe!" + accent, "नमस्ते" -> "नमस!्त!े"). Match marks as part of the preceding word.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #4474. In its default between-words mode,
InsertPunctuationConvertersplits the prompt withre.findall(r"\w+|[^\w\s]|\s", ...). Combining marks (Unicode categoryM*: decomposed accents, Indic vowel signs) are not\w, so each one became its own "punctuation" token: punctuation could be inserted between a letter and its mark (decomposedcafé→cafe!́,नमस्ते→नमस!्त!े) and the word count was wrong. Same class of fix as #4390 forSATAMaskingConverter.\w[\w\p{M}]*with theregexpackage (already a PyRIT dependency), so marks stay with the word they belong to.tests/unit/converterpasses (3264 passed, 52 skipped) and pre-commit passes on the changed files.