Skip to content

FIX: Keep combining marks inside words in InsertPunctuationConverter - #4475

Open
Saquib Jawed (saquibjawedbit) wants to merge 1 commit into
microsoft:mainfrom
saquibjawedbit:fix/punctuation-combining-marks
Open

Saquib Jawed (saquibjawedbit) wants to merge 1 commit into
microsoft:mainfrom
saquibjawedbit:fix/punctuation-combining-marks

Conversation

@saquibjawedbit

Copy link
Copy Markdown
Contributor

Description

Fixes #4474. In its default between-words mode, InsertPunctuationConverter splits the prompt with re.findall(r"\w+|[^\w\s]|\s", ...). Combining marks (Unicode category M*: decomposed accents, Indic vowel signs) are not \w, so each one became its own "punctuation" token: punctuation could be inserted between a letter and its mark (decomposed café → cafe!́, नमस्ते → नमस!्त!े) and the word count was wrong. Same class of fix as #4390 for SATAMaskingConverter.

  • Match words as \w[\w\p{M}]* with the regex package (already a PyRIT dependency), so marks stay with the word they belong to.
  • Added a test covering decomposed Latin and Devanagari across several random seeds; it fails on main (6 of 10 cases) and passes with the fix. tests/unit/converter passes (3264 passed, 52 skipped) and pre-commit passes on the changed files.
  • The within-words mode inserts at character positions by design and is unchanged.

Between-words mode tokenized words with \w+, which stops at combining
accents and Indic vowel signs. Each mark became its own non-word token,
so punctuation landed inside a word (decomposed "café" -> "cafe!" + accent,
"नमस्ते" -> "नमस!्त!े"). Match marks as part of the preceding word.
Copilot AI balanced review requested due to automatic review settings November 20, 2026 15:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: InsertPunctuationConverter inserts punctuation inside words with accents or Indic vowel signs

2 participants