Fix case repair splitting a decomposed (NFD) word at its accent - #584
Merged
Merged
Conversation
…ase repair
A name typed decomposed (NFD) was split at each combining mark, since
_WORD's \w matches no category-M character: 'josé garcía' repaired to
'José GarcíA'. Every clause now reads a mark as part of its letter:
- _sub_words runs a _WORD match on through the marks after it
- the Mac/Mc clause is decided on the composed spelling and applied to
the word as written ('macée' gave 'MacÉe' composed, 'Macée' NFD)
- _apply_mask's split-off-initial test and _letter_run_ge2 read a
letter's neighbours past marks (_beside), two shapes the issue
expected to need no change
The output keeps the form it was typed in. A mark with no letter
before it heads no word. As a side effect 'ǰ' (which upper-cases to
J + combining caron) is now a fixpoint under a second forced pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rules.md#R4 gains the statement and two NFD example lines (the expected values spell the mark as ́, so the form is visible and an editor that NFC-normalizes the file fails the example); corpus_rules.jsonl is regenerated to carry them. decisions.md#R4 records the choice, the declined NFC-compose option, the mask shapes measured against the issue's claim, and retires the 2026-09-24 SPLITTING boundary. Release log and usage.rst's decomposed-text section follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docs review findings. The hyphen clause's _DOTTED_INITIAL and the unclassified-text fallback's _INITIAL matched the raw word, so a decomposed 'й.' (й being the one default conjunction that decomposes) read as the conjunction where its composed twin reads as an initial: 'ivan petrov-й.-sidorov' repaired to 'Petrov-й.-Sidorov' NFD against 'Petrov-Й.-Sidorov'. Both now match the NFC spelling. decisions.md#R4's #542 bullet quoted outputs from a splitter-only intermediate as pre-fix behavior; it now gives both, says the 'é.x.' mask row agreed before the fix, drops the false "no corpus name is NFD", and records initials() dropping a decomposed accent as out of reach. _lexicon's reason for storing mask values composed is updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Second docs review: the R4 bullet and capitalized()'s docstring said every clause reads a mark as part of its letter, while the three shape clauses ask about the composed spelling instead; both now name the two mechanisms and the one case where they differ. The clause count and the gate's compared surfaces are corrected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #584 +/- ##
=======================================
Coverage 98.97% 98.97%
=======================================
Files 45 45
Lines 4181 4210 +29
=======================================
+ Hits 4138 4167 +29
Misses 43 43 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #542.
A name typed decomposed (NFD: macOS file names, some databases) writes
íasifollowed by a combining accent (U+0301). Case repair's word regex(\w|\.)+matches no combining mark, so the accent split the word and the letters after it were repaired as a separate word.josé garcíacame out asJosé GarcíA. The parse was already right; only thecapitalized()/capitalize()view was wrong.The fix (
nameparser/_render.py)This is the issue's option 1. Option 2, composing the text to NFC before repair, was declined: it hands back text in a form the writer didn't use, which rules.md#R4 forbids and
test_case_repair_changes_case_and_nothing_elsewould report. A decomposed word now repairs the way its composed spelling does, and the output keeps the form it was typed in._sub_wordsreplaces_WORD.sub: each match runs on through the combining marks after it, tested withunicodedata.category(ch)[0] == "M". A hand-written list of combining-mark ranges would drift; it would already miss Adlam. A mark with no letter before it starts no word of its own._apply_mask's split-off-initial check and_letter_run_ge2, now look past marks via_beside. The issue said these needed no change. Measured, that was wrong: two mask shapes still disagreed once only the splitter was fixed.macéewasMacÉecomposed againstMacéEdecomposed before this change.й.read as the conjunctionйwhere its composed spelling is an initial:ivan petrov-й.-sidorov.й.case.ǰlimit is gone: a second forced pass now leavesJ̌ounchanged instead of givingJ̌O, because the mark stays with its letter.Verification
The differential gate can't see this, because
capitalized()is not a compared surface. Coverage instead:test_a_decomposed_name_repairs_as_its_composed_twinre-encodes every corpus and case-table text that has accents, then compares its repair with the composed text's on both surfaces, plain and forced._render.py. A live control test re-breaks the code on purpose to show the walk can fail.test_render.pycovers the shapes no corpus name reaches: Mac, the two mask shapes, the hyphenatedй., the spliced-inй., theǰrepeat-pass case, and a leading mark. Each was confirmed to fail without the change; the mask rows were also checked against a version that ignores marks.́, so it's visible that the output stayed decomposed, and an editor that silently converts the file to NFC would make them fail.corpus_rules.jsonlis regenerated to carry them.й.gap, and before-fix outputs that had been copied from an intermediate state. The second found three wording errors. All are fixed.Out of scope, recorded in decisions.md#R4
initials()takes a token's first character, so for a decomposed initial it drops the accent:émile zolaNFD givese. z.. That's a rules.md#R3 view, not case repair, so it gets its own issue.🤖 Generated with Claude Code