diff --git a/AGENTS.md b/AGENTS.md index 152eeee44..d75e32279 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,7 +12,7 @@ After a review round, **review the fix commit too** — second-round passes on # **Check whether the PR description still needs updating before merging.** A review that moves behavior or retracts a measurement leaves it stale, and a stale body reads as authoritative. -**When a defect is found that an earlier check would have caught, write the check down where the next person will meet it, in the same PR.** A review finding, a failing sweep or a regression that arrived because nobody thought to test a shape is a lesson about the repository, not only about the change, and it is lost when the PR merges unless it lands somewhere a later session reads before making the same mistake. Ask, before merging: what should I have checked, and where would I have looked? Then put it there — a missed input shape in the sweeps (comma shapes, then `name_order`, under Architecture; then spellings, in `nameparser/AGENTS.md`; each earned this way); a trap in a module's behavior under Gotchas; a design-review miss in docs/design/AGENTS.md's axes; a measurement or test-design trap in mechanisms.md's Verification shapes or Field notes; a reusable design move in mechanisms.md proper. Name the PR or issue that earned it and the input that showed it, so the rule can be checked against its case. #610 is the shape: its sweep test found that the maiden take missed a shape-only numeral, and the spelling sweep in `nameparser/AGENTS.md` records the class (`VI` beside `V` and `III`). This is how the file accumulates expertise rather than history — prefer amending an existing paragraph to adding a new one, and delete a lesson the code has since made impossible. +**When a defect is found that an earlier check would have caught, write the check down where the next person will meet it, in the same PR.** A review finding, a failing sweep or a regression that arrived because nobody thought to test a shape is a lesson about the repository, not only about the change, and it is lost when the PR merges unless it lands somewhere a later session reads before making the same mistake. Ask, before merging: what should I have checked, and where would I have looked? Then put it there — a missed input shape in the sweeps in `nameparser/AGENTS.md` (comma shapes, then `name_order`, then spellings, then the encoding, each earned this way); a trap in a module's behavior under Gotchas; a design-review miss in docs/design/AGENTS.md's axes; a measurement or test-design trap in mechanisms.md's Verification shapes or Field notes; a reusable design move in mechanisms.md proper. Name the PR or issue that earned it and the input that showed it, so the rule can be checked against its case. #610 is the shape: its sweep test found that the maiden take missed a shape-only numeral, and the spelling sweep in `nameparser/AGENTS.md` records the class (`VI` beside `V` and `III`). This is how the file accumulates expertise rather than history — prefer amending an existing paragraph to adding a new one, and delete a lesson the code has since made impossible. **A scripted multi-edit that asserts as it goes discards everything when a late pattern misses.** Assert every pattern before applying any, then verify the edit landed — three times in one session a docs batch failed its last assertion, wrote nothing, and reported success, invisible because prose edits fail no test. And replace by an anchored match, never across a whole file: a history line quotes the wrong spelling on purpose, and #909's `sed` turning `segment_scripts=()` into `frozenset()` also rewrote #337's record of what shipped wrong, until the design-docs review found it. @@ -193,11 +193,11 @@ Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely o **A constant's membership is a question you may reopen.** Proposing that a word be ADDED, REMOVED or MOVED between vocabulary sets is ordinary design work — a shipped entry is not evidence that anyone judged it. `SUFFIX_ACRONYMS` arrived in `af5bdab` as a bulk Wikipedia import never reviewed against surname collisions: 572 of its 577 alphabetic entries leave `family` empty in `"John "` against five ambiguous-gated exceptions (recomputed 2026-09-07; the fifth is `ba`; 568 of 575 against seven, measured 2026-09-25 after #540), and `sa`, `se` and `om` are borne as surnames (measured 2026-08-23; all three since decided, `sa` removed from the vocabulary on 2026-10-25, #857), as were `rai`, `cha`, `ba` and `mc` before `decisions/suffix-acronym-collisions` decided all four — `rai` and `cha` removed, `ba` marked ambiguous, `mc` left alone as no borne name at all. The same entry marked `meng` and `lac` ambiguous on 2026-09-25 (#540). When a fix starts to look like new machinery, check the vocabulary first. Criterion: `decisions/vocabulary-collisions`, with #360's positional qualifier. -**Sweep the forms a change can reach: comma shapes, then `name_order`.** No comma, a FULL name before the comma, a ONE-WORD name before the comma and NOTHING before the comma are four paths, and the last two are the misses — #429, #430 and #432 are all the one-word path disagreeing with the full-name path on inputs the full-name path reads correctly, and #613's PR review found `, PhD` and `, Jr.` (an empty surname field, as CSV data writes it) read as given and title by a decision that returned early on an empty head, after two review rounds whose every grid had a name before the comma. Put an empty head and a nickname-only head (`(Bob), PhD`) in every comma grid. Count the one-word path the way C3 counts, which is neither tokens nor pieces: a particle run and the one word it attaches to are one word (#575), a connective join and a bound given-name pair count as their words (#613, so `Ortega y Gasset, Dr.` is a whole name), and a suffix beside the surname counts as no name word at all. #603's review found the suffix half (`Vega y Lopez Jr., PhD Jones` lost its family where `Vega y Lopez, PhD Jones` kept it); #613 then reversed the connective half, and both now read given 'Vega', middle 'y', family 'Lopez' — so test a particle surname, a connective surname and a surname with a suffix, each before a comma. For orders the sweep already exists (`tests/v2/test_cases.py` runs every row under all three) but carries one assertion, R2's family partition, so it checks nothing a new change moves; coverage there has been vacuous before (PR #394's review found the suite passed with `name_order` discarded from grouping). Parse your change's names in each comma shape and each order, and read the ones you did not predict. And read a generated grid's moves by SHAPE, not by corpus membership: #614's first design passed the suite, the five gates and a fingerprint of every corpus name, because its grid moves were sorted into 'corpus name moved' and 'garbage', and the review found the moves that mattered among the 'garbage' -- a bare salutation (`Mr. and Mrs.` lost its connective to the family name) and a name with one trailing word (`Freiherr von Berg X.Y.Z.`). Pull out the head-plus-one-word slice of the grid and a salutation with nothing after it, and read every move there. Two more were the second review's (#614): no grid put TWO words that are both a title and a particle at the head (`Freiherr St van Berg MA`, where an index into the pieces was applied to a merged view of them -- `TITLES ∩ particles_ambiguous` is `{freiherr, st}`, so put both in every head list, each in its bare and its period-marked spelling -- #637's grid had `St` and `Freiherr` alone and missed that `St.` and `Freiherr.` are declined by the bound join and then attached by P6), and the grids compared ambiguity KINDS, so a report telling the caller the wrong field (`Freiherr von Berg Ma`, 'read as a given name' of a word in the family) passed every pin. #701's review added one: put a title/suffix dual (`Ms`, `Sr.`, `Cpl`) SECOND in a title run, behind an ordinary title. A stop that asks only "is this word suffix vocabulary" passes every row that opens with the dual and fails `Rev. Sr. Smith, Mary`; it was found by comparing `T Smith, John` against `T John Smith` over every title word, and a comma-versus-comma-less comparison is the cheap check for any rule that moves a word between the two slots. Compare each report's `detail` against the field its tokens land in -- since #626 `test_a_report_names_the_field_its_word_lands_in` (tests/v2/test_cases.py) does it over every non-locale case row under all three orders (a row carrying its own policy only as declared), for the phrasings its `_FIELD_CLAIMS` lists -- so a row with the shape is the test only where the report's wording is listed: a new emitter's wording goes in that list, and segment's report on a part past the second comma names no field (#629). #626's own first grid then missed an emitter for want of one comma shape (`, Ma Dr.`: an empty head, an ambiguous word, a trailing title, which H1 moves to the family): a trailing title belongs in every empty-head comma grid too. And #620's: an exclusion list in code being replaced is a set of rules, not noise -- #614's merged copy excluded every word the trailing read weighs from its units, the first #620 prototype dropped the list as a symptom of the copy, and its review found the list had been keeping positional counting's rule that a member in front counts as a word (`Freiherr von Berg MA X.Y.Z.` lost its suffix). Before deleting one, mutate each entry out and read what moves. The converse holds for replacing a hand copy WITH the shared predicate: it is a behavior change wherever the two disagreed, and #735's /simplify shipped one as "no reading moves" because the suite and the snapshot were its only check (`St Sir de Mesnil` family-first, a token test replaced by P1's piece test); diff the two predicates' answers over a grid before calling the swap a refactor. +**Sweep the forms a change can reach: comma shapes, then `name_order`.** No comma, a FULL name before the comma, a ONE-WORD name before the comma and NOTHING before the comma are four paths, and the last two are the misses. For orders, `tests/v2/test_cases.py` runs every row under all three but asserts only R2's family partition there, so it checks nothing a new change moves: parse your change's names in each comma shape AND each order yourself. The grid each needs (an empty head and a nickname-only head, a particle, connective and suffixed surname before the comma, two title-particles at the head, a dual second in a title run, a trailing title after an empty head), how to read a grid's moves, and the predicate and exclusion-list traps are in `nameparser/AGENTS.md`, beside the spelling sweep. **Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes, so when a change touches a word class, test one spelling from each path. The classes where a path was missed (roman numerals, credentials and titles, a separator's spacing, a full stop inside a word, capitals between name words, adjacent pairs), each with the PR that found it, are in `nameparser/AGENTS.md`, which loads when a session reads anything under `nameparser/`, the vocabulary TOML included. -**And the encoding: a letter class written `[^\W\d_]` or tested with `isalpha()` is blind to a combining mark.** Python's `\w` and `isalpha()` do not match Mn, so a letter written as base plus accent (NFD) ends a word early; the vocabulary path is immune because `_normalize` NFC-folds, the shape path is not. #652: `Dr. inž. Janez Novak` read `inž.` as the given name when the ž was decomposed, and `Janez Novak, ŽŠČ` reversed, in two families that share the blind spot — the letter-shape regexes (`period_abbreviation`, `_DOTTED`, `_CHUNKED`, `_PAIRED_INITIALS`), now spelled with `_vocab._LETTER` (a letter plus the general combining blocks; `test_regex_sync` compares the two `_MARKS` strings and pins `period_abbreviation` against config's copy), and the caps predicates, which NFC-compose a non-ASCII word before `isalpha()` (`is_caps_word`, behind C-level `isalpha()`/`is_normalized` tests, so already-composed text pays no frame — measured 2026-10-12, `Smith, José` was 193 frames before and after; the first version called it for every non-ASCII word and cost one frame each, and no guard saw it because the reference name is ASCII). A width test must count letters, not code points (`composed_len`): admitting a decomposed word to the caps set put `John Smith, MÁ` on the pair exception's wrong side, and the review's grid found it where the rows (all three letters long) did not — put a two-letter word in every caps grid. The initial shape (`_INITIAL`, `\w\.`, v1-verbatim and public) is not widened but asked of the COMPOSED word, in `is_initial_shaped` and in `_render`'s two copies; and the single-letter connective tests count composed letters, since a decomposed `й` is two code points (#665, where `É. y É. Smith` and `John й Smith` read differently decomposed). Compose the WHOLE predicate, not one half: `is_initial` is a shape ANDed with the #320 script veto, and composing the shape alone made a decomposed `씨.` an initial, since decomposed hangul is conjoining jamo outside every script range (#665's first version; `in_initialless_script` composes too now). Not covered, deliberately: the marks only some scripts write (Indic vowel signs, Arabic and Hebrew points), declined in decisions/nfd-shapes because widening the class would move readings nobody had measured; rules/H2's Accepted clause records the one consequence it names, that a period-marked abbreviation in Bengali or Devanagari never reads as a title by shape. The guard is `test_a_decomposed_name_renders_as_its_composed_twin` over the CLDR corpus plus `test_a_decomposed_abbreviation_reads_as_its_composed_twin` (rows × three orders × EVERYWHERE); the corpus guard reaches a shape only if some corpus name does, which is how the caps shape was missed until review, so when adding a shape rule add a row there. Four of #652's rows and two of #665's pass with every fix reverted (measured) and are parity pins, and the test compares ambiguity kinds and `initials()` too, since #665's `John й Smith` differed in its report; `_DOTTED` has no observable input and is pinned by a direct call. +**And the encoding: a letter class written `[^\W\d_]` or tested with `isalpha()` is blind to a combining mark.** Python's `\w` and `isalpha()` do not match Mn, so a letter written as base plus accent (NFD) ends a word early; the vocabulary path is immune because `_normalize` NFC-folds, the shape path is not. Which predicates were blind (#652, #665), how each was fixed, what is deliberately left uncovered and the guards to add a row to are in `nameparser/AGENTS.md`, beside the other sweeps. ### Vocabulary (`nameparser/vocab/`) and the configuration layer (`nameparser/config/`) diff --git a/docs/design/decisions/A1-ambiguity-report.md b/docs/design/decisions/A1-ambiguity-report.md index bd2a00ff1..50a22f79f 100644 --- a/docs/design/decisions/A1-ambiguity-report.md +++ b/docs/design/decisions/A1-ambiguity-report.md @@ -20,7 +20,7 @@ Open: none in this file. MEASURED: over the review's 43,925-name grid under eight configurations, 9,806 rows on 1,637 distinct names lose exactly one `conjunction-or-initial` and nothing else — no report of any other kind moves and none is gained. On the deduped corpus-union-cases population the only name affected is `JOHN QUINCY SMITH I`, a row this round added, so the ledger's three-name `fix(#397) the Catalan link reports a connective-or-initial in a one-case name` rule is untouched: all three of its names role the letter MIDDLE and all three still report. The invariant is `tests/v2/test_properties.py::test_no_report_contradicts_the_reading_beside_it`, which states it twice — no connective-or-initial on a token roled SUFFIX or TITLE, and no token named by two reports whose details assert different readings — and fails on 8,204 rows at `dc3bdf9c`, 0 here. - 2026-10-08 #626 (its literal wording for the comma's credential reading superseded in part by #793 below) — A REPORT NAMES THE FIELD ITS WORD LANDS IN. Six of assign's reports worded the field from the role the word held at assign (`read as a given name`), and rules in post_rules still move a word after that: H1 moves the name behind a title to the family, P1's family-first fold takes the particle run and one unit and lays the rest out by the order, P6's attachment takes a trailing name word into the family at the end of a name read family-first, and under opt-in policies the patronymic rotations (O1, O2) and `middle_as_family`'s fold (O3) move one too. So `Attorney General of Minnesota` reported 'General of Minnesota' "read as a given name by convention" with the unit in the family (since 2.3.0, #491's join clause); `Kim Min Do` under FAMILY_FIRST told the caller 'Do' was "read as a middle name" with 'Do' in the family, and `de Kim Ma` under FAMILY_FIRST said 'Ma' was a middle name with 'Ma' the given name (both this cycle, since #289 made the Title-case word a name word); `, Ma Dr.` said 'Ma' after the comma was "read as the given name" with 'Ma' in the family (this cycle, since #289 added that comma report; 2.3.0 reads the same fields and reports nothing); and `Van Ivan Petrovich` under the East Slavic patronymic rule said the leading 'Van' was "read as a given name" after O1 rotated it into the family (at 2.3.0 too). #614's second review had fixed one instance of the class (`Freiherr von Berg Ma`) by moving its report to the stage that knows the outcome. DECIDED: the DECISION stays at its site and the WORDING of the field moves to assemble, the one place that knows final roles (the same reasoning as the 2026-09-20 withdrawal above). An emitter whose detail names a field a later rule could change writes the sentence up to the field and hands the rest as `PendingAmbiguity.field_tail` (`""` where the field ends the sentence); assemble joins them around `FIELD_READING` of the first referent's final role. All six such emitters in assign use it -- H4's all-suffix name, title-or-name's join clause, O5's lone word, the trailing peel's picks, the leading ambiguous particle and the comma report on the word after a family comma -- including those right on every measured input, so no later rule can make them wrong. A report in a name emptied for want of content is withdrawn rather than shipped half-worded (no emitter reaches that today). A placeholder in the detail was declined: token text can carry any characters, and a name containing the placeholder would corrupt its own report. Reports worded where the field is settled keep their literal wording: a comma's credential reading, the suffix run's and the numeral fork's, H4's "the name" and P6's own "joins the family". The comma report's given-side wording moves from "the given name" to "a given name" with the rest. - MISSED, then caught in review: the first draft converted five emitters and its grid had no empty-head comma with a trailing title, the shape AGENTS.md's comma sweep already asks for; four of the review's five readers found the sixth emitter on `, Ma Dr.`. The draft also named only H1 and P6 as movers; P1's fold was found by stepping the stages, and the review of the fix found the opt-in movers O1 to O3, which no grid over the three orders reaches -- reverting the leading-particle emitter passed the whole suite until rows under those policies pinned it. + MISSED, then caught in review: the first draft converted five emitters and its grid had no empty-head comma with a trailing title, the shape nameparser/AGENTS.md's comma sweep already asks for; four of the review's five readers found the sixth emitter on `, Ma Dr.`. The draft also named only H1 and P6 as movers; P1's fold was found by stepping the stages, and the review of the fix found the opt-in movers O1 to O3, which no grid over the three orders reaches -- reverting the leading-particle emitter passed the whole suite until rows under those policies pinned it. MEASURED 2026-10-08, each report's field phrase compared against its tokens' final roles. RECIPE: texts are the 1,536 names of `tools/differential/corpus*.jsonl` and 347 `tests/v2/cases.py` texts (the table at 0c54662b, the parent of #614's first commit, less the corpus names; today's table adds rows this entry's review wrote for the shapes the grid missed, so it reads more wrong claims on master, 6 more as measured on the review), plus every head in {'John', 'John Smith', 'John Q. Smith', 'Mary Ann Smith', 'John van Smith', 'Juan de la Vega', 'John van', 'anh van', 'Jan Freiherr von Berg', 'Freiherr von Berg', 'Freiherr St', 'St', 'Dr. St', 'Freiherr', 'Dr.', 'Mr. and Mrs.', 'Attorney General of', 'abdul salam', 'Juan y Garcia', 'Jane Doe nee van der Berg', 'Dr. John Smith', 'JOHN SMITH', 'john smith', 'Smith', 'de Mesnil', 'Dr. de Mesnil', 'Kim Min', 'Kim Min Jun', 'Ortega y Gasset', 'van', 'Do', ''} followed by one or two words (repetition allowed) from the list [PhD, MA, Ma, DO, Do, do, vd, mc, Jr, III, V, V., VI, X.Y.Z., Ph. D., Esq., Prof., Sir, Ed, van, and, Jones, ba, John, Smith, Minnesota, de, St, Freiherr], space-joined and stripped; 20,000 draws after Python's `random.seed(626)`, each `random.choice(heads)`, then `random.randint(3, 4)` words each `random.choice(words)`, space-joined and stripped, both lists in the order written here; then, unstripped, `{head}, {word}`, `Smith, {head} {word}` with any double space replaced by one, and `{word}, {head}` for every head and word, and `, {w1} {w2}` and `(Bob), {w1} {w2}` for every ordered pair of words; the empty text dropped and duplicates removed -- 53,517 texts; each parsed under the three orders, so the opt-in movers are outside these figures; a claim is a match of `(reads? as|joins) ()` for the phrases `_FIELD_CLAIMS` in tests/v2/test_cases.py lists plus "the given name", and it is wrong when its tokens' roles are not among the fields the phrase names. On master (26cdb891) 2,725 of 89,092 claims were wrong, in three emitters: title-or-name's join clause (1,575), the trailing peel's picks (1,114) and the comma report (36); on the tree, 0. The comparator is master, which words the field at assign; the tree words it at assemble. Fingerprint against master over the differential corpora, master's case-table names and a seeded comma grid (13,090 names) under six policies (the three orders, `lenient_comma_suffixes=False`, a `/` suffix delimiter and `CapsSuffixes.OFF`): no role, kind, initials or rendering moves; the details that change are 6,395 comma reports on 1,086 names reworded from "the given name" to "a given name", 85 on 15 names now naming the family they land in, and 40 title-or-name join reports on 10 names now naming the family. Pinned by `test_a_report_names_the_field_its_word_lands_in` in tests/v2/test_cases.py, over every case row under all three orders except the locale rows (a row with its own policy only as declared), whose recorded control is 14 failures on master, and by `test_the_field_sweep_sees_the_claims_it_checks`, which records how many claims that sweep checks; the opt-in movers by the rows `Van Ivan Petrovich` (East Slavic), `Van Ali Veli oglu` (Turkic) and `Do Bishop Do` (FAMILY_FIRST with `middle_as_family`), which the sweep reads as declared. - 2026-10-10 #629 — A REPORT ON WORDS IN TWO FIELDS NAMES NEITHER. Segment's `comma-structure` report on a part past the second comma said the part was "consumed as suffix best-effort", true through 2.3.0, when the whole part read as suffix. Since #603 (rules/C2) a title word there reads as a title, so `John Smith, Jr., Dr. Bart` was told 'Dr.' was a suffix with 'Dr.' in the title (this cycle, never shipped). diff --git a/docs/design/decisions/C13-credential-opening-the-part-after-the-comma.md b/docs/design/decisions/C13-credential-opening-the-part-after-the-comma.md index 60950b967..69b04c6c9 100644 --- a/docs/design/decisions/C13-credential-opening-the-part-after-the-comma.md +++ b/docs/design/decisions/C13-credential-opening-the-part-after-the-comma.md @@ -17,7 +17,7 @@ Open: none in this file. THE ONE-WORD PATH, DECIDED (Derek): the whole part is suffixes. `Doe, PhD Jones` reads family 'Doe', suffix 'PhD Jones', no given name, as `Smith, PhD` already reads. DECLINED: reading the credential as a suffix and the next name word as the given name (`Smith, PhD Jones` → given 'Jones'), proposed by analogy with a title at the head of the given part. Asked whether that is a real format, it is not -- nobody writes `Family, Credential Given` on purpose -- and the one realistic input it served, a misplaced generation (`Smith, Jr. John` for `Smith Jr., John`), is garbage in, garbage out. One reading for both comma paths was the deciding ground: the one-word path disagreeing with the full-name path is the shape #429, #430 and #432 each found. Accepted cost: `Smith, Jr. John` loses 'John' to the suffix, where 2.3 read given 'John' with 'Jr.' misfiled as a title. A TITLE/SUFFIX DUAL OPENS NOTHING (Derek). A word in both vocabularies (`md`, `ms`, `sr`, the ranks `lt`, `cpl` and the rest) is the listing form's title in front of a given name, `Smith, Ms Jane`, and a credential behind it opens nothing either (`Smith, Ms PhD Jones` keeps given 'PhD', as `Smith, MD PhD Ma` keeps given 'Ma' under #544). DECLINED after building it: a dual opening the part behind two or more words before the comma. Derek first chose it for `John Smith, MD Jones`; the reviews found what the count cannot see -- a surname of two words (`García Márquez, Ms Gabriela`, `Smith Jones, Sr. Maria`), a title in front of a one-word surname (`Dr. Smith, Ms Jane`, the count taking the title as a word), and spaced initials (`García Márquez, Ms G. J.`) each lost the given name -- and with the word explained (a dual is a word in both lists, decided by position), Derek chose that duals never open. So `John Smith, MD Jones` and the `MD - PhD - FACS` delimiter rows keep their 2.3 reading. An ambiguous-class member opens nothing either (`John Smith, Ma Jones` keeps given 'Ma'), nor does a suffix word S2's run start refuses (`John Smith, vd Jones`). STRICT MODE: the veto refuses an initial-shaped word the suffix READING, not the run. `Smith, PSM I.` under `lenient_comma_suffixes=False` reads suffix 'PSM I.' (2.3: given 'PSM', suffix 'I.'), reporting the absorbed 'I.', which matches #602's no-comma run (`John Smith PhD V.` reports; its given-part run does not, a disagreement #602 left and this does not touch). - THE SPELLING GAP THIS FOUND, FIXED HERE. `starts_a_credential_run` refused any piece longer than one token, so the split credential `Ph. D.` group merges started neither #602's run nor this one while `PhD` started both: `Smith, John Ph. D. Jones` read middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones'. The merged piece now starts the run, the "suffix" piece tag being the Ph./D. merge's alone; `trailing_candidates` had already admitted it, so its pre-check and the predicate now agree. The no-comma spelling `John Smith Ph. D. Jones` still reads family 'Jones': the trailing peel's walk never holds the merged piece, which is a separate path and is left (AGENTS.md's spelling sweep). + THE SPELLING GAP THIS FOUND, FIXED HERE. `starts_a_credential_run` refused any piece longer than one token, so the split credential `Ph. D.` group merges started neither #602's run nor this one while `PhD` started both: `Smith, John Ph. D. Jones` read middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones'. The merged piece now starts the run, the "suffix" piece tag being the Ph./D. merge's alone; `trailing_candidates` had already admitted it, so its pre-check and the predicate now agree. The no-comma spelling `John Smith Ph. D. Jones` still reads family 'Jones': the trailing peel's walk never holds the merged piece, which is a separate path and is left (nameparser/AGENTS.md's spelling sweep). C2, WIDENED (Derek): a word of the title vocabulary in a part past the second comma that is not also suffix vocabulary is a title -- the part stands behind a comma, the line H5 draws for reading a title from the end of a name -- and a piece holding a suffix word stays a suffix, a dual and a connective join that took one (`PhD - and MD`) alike. `Eric H. Holder, Jr., Attorney General` reads title 'Attorney General', and a part of title and suffix words is recognized rather than flagged. The flag's test is asked word by word against the title vocabulary, a period-joined word through `period_joined_vocab` as classify tags it (`John Smith, Jr., Lt.Gov.`, which the code review found still flagged), so `Secretary of State` reads its title but keeps `comma-structure`: knowing that group's title chain takes the connective would be `segment` modelling `group`, the sign #611 is about. Every release through 2.3.0 read these as suffixes, 1.4.0 included (`Eric H. Holder, Jr., Attorney General` suffix 'Jr., Attorney General', `Smith, John, Prof.` suffix 'Prof.', measured with the released wheels 2026-10-04); a first draft of this entry said 1.4.0 read them as titles, inferring it from the 1.4.0 run reporting no unexplained diff, when the reason was the broad `fix(comma-family)` rule there claiming them, which #603's two rules now precede by declaration. Accepted consequence: a noble rank there reads as a title, `John Smith, Jr., Freiherr von Richthofen` → title 'Freiherr', suffix 'Jr., von Richthofen'. WHAT MOVES, measured 2026-10-04 against 97d36e0d (PR #615): four of the 1,515 corpus names, every one intended (`Steven Hardman, RN - CRNA` now reads given 'Steven', suffix 'RN - CRNA', and `John, Smith, Dr.`, `Andrew Perkins, Jr., Col. (Ret)` and `1 & 2, 3 4 5, Mr.` read their tail title); and 589 of 98,630 generated texts under three policies, 581 of them a part a credential that is not also a title opens or a title past the second comma, one the two-name-piece count reading a connective-joined piece beside a suffix word as the family (`Jr. i Ó VI Attorney, Ph.`), and the other seven read by hand. Frames are unchanged on the reference band and plain names, and `John Smith, PhD Jones` pays 338 → 361. - 2026-10-19 — SPLIT FROM decisions/C8 with the rule. The #603 entry above was written under decisions/C1 while this rule was part of rules/C1, moved to decisions/C8 with the listing form's clause in #728, and moved here unchanged when rules/C8 split; where it says rules/C1, C1 or C8, it means the rule as it then stood, which held this one. Its C2, WIDENED paragraph decides rules/C2 (a title word past the second comma) and stays rules/C2's. diff --git a/docs/design/decisions/H5-trailing-title-run.md b/docs/design/decisions/H5-trailing-title-run.md index e74aba293..aa8243c8b 100644 --- a/docs/design/decisions/H5-trailing-title-run.md +++ b/docs/design/decisions/H5-trailing-title-run.md @@ -44,5 +44,5 @@ Open: none in this file. DECIDED (Derek): the narrower bound. Declined: the issue's proposal, reading the given part's run once ahead of the chain and splitting it off as the main segment's is (#614). It would have assign place the given part off group's read, a second reader beside the walk's own predicates, for a class the bound alone closes: the transparency measurement below finds no text the narrower fix leaves broken that the read would mend. MEASURED 2026-10-10 against 4d5631c2 (#634's merge), py3.11, `nameparser.__file__` asserted on each side. A given-part grid -- `{family}, {head} {middle} {tail}`, and the same with `, Jr.` appended, for family in `Smith`, `van der Berg`, `Ortega y Gasset`, (empty), `(Bob)`; head in `John`, `Dr. John`, `Freiherr`, `abdul`, `Freiherr St`, `J.`; middle in (none), `van`, `van der`, `de la`, `van Berg`, `van der Berg`, `Mc Mc`, `von und zu`, `Berg`, `Berg van`, `do`, `St`, `y Lopez`; tail in (none), `Prof.`, `Dr.`, `Prof. Dr.`, `Prof`, `Sir`, `Prof. PhD`, `PhD Prof.`, `Jr. Prof.`, `Prof. Jr.`, `MA`, `Ma`, `VI`, `V`, `X.Y.Z.`, `Ph. D.`, `Esq.`, `née Jones`, `Prof. Jones`, `Xyz.`, `King.`, `Prof. V`, `MA Prof.`, `Lt.Gov.` -- 18,480 texts under the three orders and `lenient_comma_suffixes=False`, comparing the seven fields and every report: 17,340 of the 73,920 parses move a role and none a report alone, all but 500 parses (125 texts) putting a trailing period-marked title into the title field (PR #635); H5's transparency over the grid's 5,390 default-order texts ending in a trailing title breaks on 392 against master's 3,048, and on no text master kept (352 H1's move behind an empty head, 40 P5's post-comma join taking the title, fixed by #637); 500 parses (125 texts) put a title in front of a numeral with a third comma part into the middle, garbage; the S2 stress grid's comma half (#614's recipe) and #624's title-particle grid move 56 and 24 of the same class, #625's particle grid, every corpus name and case text, and #624's comma-tail grid none. Frames: a family-comma name with a particle pays 2 with no trailing run, 4 with a suffix, and the moved shapes 38 to 73 in a sample, rare shapes no guard sees, and one without a particle pays less (`Smith, John Quincy` 221 → 213, `Doe, John MA` 254 → 227), which `test_a_given_part_with_no_particle_skips_the_chain_bound` holds; recorded negative control: with the old bound, 8 tests fail. - THE REVIEW (2026-10-10). The bound is `tail_reading`'s run, read as the comma-less name reads it, and the walk reads the given part by its own predicates, so where the two disagree on a word behind a title the chain stops for a title the walk does not take, or cannot cross one to reach a particle P6 takes: a title in front of a shape-only numeral (`Smith, John Mc Prof. VI` → suffix 'Mc', middle 'Prof. VI'; master middle 'Mc Prof. VI'; the walk never reads 'VI' as a suffix after a comma), in front of a numeral with a third comma part (the 500 above), in the middle of the given part in front of a particle (`Smith, John van Prof. Mc` → title 'Prof.', middle 'van', family 'Mc Smith'), and a period-marked title-particle ending it, which P6 attaches (`Smith, John van MA St.` → suffix 'MA', family 'St. Smith'). Every such shape is garbage, most read as master already reads the text without the title (`Smith, John Mc VI` suffix 'Mc'), and none is a reading a well-formed name loses: over about 1.6 million generated rows under every order, the opt-in policies, one-case writing and caller lexicons, and over a 134,720-pair transparency grid (`Smith, ` and `, ` heads; one to three words from `John`, `van`, `der`, `von und zu`, `Berg`, `Ma`, `MA`, `V`, `VI`, `Jr.`, `PhD`, `Ph. D.`, `Mc`, `St.`, `St`, `Freiherr`, `X.Y.Z.`, `Esq.`, `abdul`, `Prof.`; a trailing ` Prof.`, with and without `, Jr.` after it; the three orders and `lenient_comma_suffixes=False`; transparent where given, middle, family, nickname and maiden equal the title-less text's and the title gains exactly ` Prof.`; master broken on 53,384 pairs, the tree on 9,734), no text transparent to its title on master is broken now. (The first review named `Smith, John Ph. D. St. Prof.` as one, its oracle reading the direction backwards: master gives suffix 'Ph. D. St. Prof.' against the title-less 'family St. Smith', and the tree is transparent.) Recorded, not fixed (Derek): garbage, and narrowing the bound would add a condition for it. The review also found the bound's particle gate held by no test, two of its narrowings undoing readings master gives -- a scan from the name's start rather than the chain's lead (`Smith, Freiherr von Berg Prof.`) and a test of single-token particles alone (`Smith, John von und zu Berg Prof.`) -- now rows of their own, with an empty-head row (`, Jan van Prof.`, AGENTS.md's comma sweep), and the frame saving now held as above. Each of the three mutations fails a test. The re-ask after the chain (`left < tail`) still decides readings, though its documented case (`Freiherr von Berg Ma`) takes the read-once path since #614: removed, it moves 400 parses of the given-part grid on master and here alike, a title-particle head before a particle run and a dotted acronym (`Smith, Freiherr van Berg X.Y.Z.` → given 'van Berg X.Y.Z.', as 1.4.0 reads it; without it suffix 'X.Y.Z.'). It decides a main segment's run with a name piece inside it too (`Freiherr van Berg Dr. X.Y.Z.` → family 'van Berg Dr. X.Y.Z.', as master and 2.3.0 read it; without it given 'van Berg Dr.', family 'X.Y.Z.', 1.4.0's reading), a garbage shape no row holds. The first review reported it changing nothing over a grid with no such shape, and #638 was filed on that; the fix review found it live and held by no test, and the row `the_chain_runs_again_where_its_merges_free_the_acronym` (parity) holds it now. Left for follow-ups, two gaps outside this change: a part past the second comma reads a trailing title behind a particle into the suffix (`John Smith, Jr., van der Berg Prof.`, where `John Smith, Jr., Prof.` reads title 'Prof.'; #636), and P5's post-comma join taking a title (the 40 above; #637). + THE REVIEW (2026-10-10). The bound is `tail_reading`'s run, read as the comma-less name reads it, and the walk reads the given part by its own predicates, so where the two disagree on a word behind a title the chain stops for a title the walk does not take, or cannot cross one to reach a particle P6 takes: a title in front of a shape-only numeral (`Smith, John Mc Prof. VI` → suffix 'Mc', middle 'Prof. VI'; master middle 'Mc Prof. VI'; the walk never reads 'VI' as a suffix after a comma), in front of a numeral with a third comma part (the 500 above), in the middle of the given part in front of a particle (`Smith, John van Prof. Mc` → title 'Prof.', middle 'van', family 'Mc Smith'), and a period-marked title-particle ending it, which P6 attaches (`Smith, John van MA St.` → suffix 'MA', family 'St. Smith'). Every such shape is garbage, most read as master already reads the text without the title (`Smith, John Mc VI` suffix 'Mc'), and none is a reading a well-formed name loses: over about 1.6 million generated rows under every order, the opt-in policies, one-case writing and caller lexicons, and over a 134,720-pair transparency grid (`Smith, ` and `, ` heads; one to three words from `John`, `van`, `der`, `von und zu`, `Berg`, `Ma`, `MA`, `V`, `VI`, `Jr.`, `PhD`, `Ph. D.`, `Mc`, `St.`, `St`, `Freiherr`, `X.Y.Z.`, `Esq.`, `abdul`, `Prof.`; a trailing ` Prof.`, with and without `, Jr.` after it; the three orders and `lenient_comma_suffixes=False`; transparent where given, middle, family, nickname and maiden equal the title-less text's and the title gains exactly ` Prof.`; master broken on 53,384 pairs, the tree on 9,734), no text transparent to its title on master is broken now. (The first review named `Smith, John Ph. D. St. Prof.` as one, its oracle reading the direction backwards: master gives suffix 'Ph. D. St. Prof.' against the title-less 'family St. Smith', and the tree is transparent.) Recorded, not fixed (Derek): garbage, and narrowing the bound would add a condition for it. The review also found the bound's particle gate held by no test, two of its narrowings undoing readings master gives -- a scan from the name's start rather than the chain's lead (`Smith, Freiherr von Berg Prof.`) and a test of single-token particles alone (`Smith, John von und zu Berg Prof.`) -- now rows of their own, with an empty-head row (`, Jan van Prof.`, nameparser/AGENTS.md's comma sweep), and the frame saving now held as above. Each of the three mutations fails a test. The re-ask after the chain (`left < tail`) still decides readings, though its documented case (`Freiherr von Berg Ma`) takes the read-once path since #614: removed, it moves 400 parses of the given-part grid on master and here alike, a title-particle head before a particle run and a dotted acronym (`Smith, Freiherr van Berg X.Y.Z.` → given 'van Berg X.Y.Z.', as 1.4.0 reads it; without it suffix 'X.Y.Z.'). It decides a main segment's run with a name piece inside it too (`Freiherr van Berg Dr. X.Y.Z.` → family 'van Berg Dr. X.Y.Z.', as master and 2.3.0 read it; without it given 'van Berg Dr.', family 'X.Y.Z.', 1.4.0's reading), a garbage shape no row holds. The first review reported it changing nothing over a grid with no such shape, and #638 was filed on that; the fix review found it live and held by no test, and the row `the_chain_runs_again_where_its_merges_free_the_acronym` (parity) holds it now. Left for follow-ups, two gaps outside this change: a part past the second comma reads a trailing title behind a particle into the suffix (`John Smith, Jr., van der Berg Prof.`, where `John Smith, Jr., Prof.` reads title 'Prof.'; #636), and P5's post-comma join taking a title (the 40 above; #637). diff --git a/docs/design/decisions/S2-suffix-word-ends-a-name-and-never-opens-the-string.md b/docs/design/decisions/S2-suffix-word-ends-a-name-and-never-opens-the-string.md index 3c9cc437a..6c97443a2 100644 --- a/docs/design/decisions/S2-suffix-word-ends-a-name-and-never-opens-the-string.md +++ b/docs/design/decisions/S2-suffix-word-ends-a-name-and-never-opens-the-string.md @@ -23,7 +23,7 @@ Open: none in this file. - 2026-10-07 (Derek), #614 (its counting view, a merged copy of the pieces, superseded by #620, below, and the view's name start inside the titles by #624, decisions/P2) — S2'S TRAILING RUN IS READ ONCE, BETWEEN P3'S CONNECTIVE JOINS AND THE PARTICLE CHAIN, AND BINDS. The issue's diagnosis held on the day: the words-to-spare count is taken over pieces and group's joins make pieces, so group re-asked the peel before and after its own joins -- the particle chain's stop and its re-ask and rollback, the bound-given reserve's two views (P5) -- and assign then read the run again. Now the run of a main segment (no comma, or the part before a suffix comma) of three or more pieces is read once, after the Ph. D. merge, the maiden take and P3's connective joins, which do not depend on the run and make the units it counts, and before the particle chain and the bound-given join, which do; the run is split off, the chain and the join group the name in front of it, and the run is put back unchanged (`_pieces.read_trailing_run`). The chain's stop is the end of what is left and the reserve counts the name pieces the read left; assign places and reports the run off the read (`ParseState.tail_reads`, `_assign._placed`). The read COUNTS a particle run and the plain name words after it as the one unit the chain will make of them (`_counting_view`, a copy; rules/P3's own count, mechanisms.md#UNIT-PARTITION), the words both particles and suffix vocabulary straight behind a particle among them, while anything the read weighs stays a word of its own -- suffix vocabulary, the ambiguous class and its shape-only members, roman numerals by shape, initials, period-marked titles. STALE IN THE ISSUE, measured before designing: segment no longer decides the comma or emits S2 reports (#613 moved C1 to `_comma.decide`); assign predicts P1's fold but no longer P6, which it decides since #613; the two comma restart inputs (`John Smith, PhD vd DO`, `PhD DO DO`) no longer reach the chain, decide binding that part first; `credential_run` already applied in one place; and #613's `name_word_count` reads particle and suffix facts at P1's reach and ignores connective and bound-given joins, so S2 could not share it as it stood. The issue's 'licensed by neighbour', read literally, moves about 25 inputs (`JOHN SMITH MA`, six S3 example lines) and still five read as scoped to the duals, so it was not taken; the counting view's particle run is the one part of it the design needed. - THE FIRST DESIGN AND THE REVIEW THAT REVERSED IT. Derek decided first that the read WINS OVER EVERY JOIN, connectives included -- prototyped both ways in a scratch copy -- and that implementation passed the suite, the gates and the fingerprint. #614's PR review then found it broke what the corpus never tests: a salutation alone read its connective as the family name (`Mr. and Mrs.` gave title 'Mr. Mrs.', family 'and' -- the H5 chain took 'Mrs.' before P3 could join), a joined leading salutation counted as three words (`Mr. and Mrs. Jr` gave family 'and', `Mr. and Mrs. Do` a spurious report), and the read counted a particle chain's or a connective join's words one by one where rules/P3 says a joined part is one name word wherever another rule counts (`Freiherr von Berg X.Y.Z.` and `Juan y Garcia X.Y.Z.` took the acronym as a credential against every release, while `Freiherr Berg X.Y.Z.` kept it the family). DECIDED (Derek, 2026-10-07): connectives join first, and the read counts the chain's units -- which returns every one of those to master's reading, and most of the first design's 'tail beats connectives' class with them. A second-round finding the counting view itself made: folding the plain words after a particle into its unit swallowed a shape-only roman numeral (`Juan de la Vega VI` read family 'de la Vega VI'), the #610 class AGENTS.md's spelling sweep records, so a word the numeral fork reads by shape stays out of the unit. + THE FIRST DESIGN AND THE REVIEW THAT REVERSED IT. Derek decided first that the read WINS OVER EVERY JOIN, connectives included -- prototyped both ways in a scratch copy -- and that implementation passed the suite, the gates and the fingerprint. #614's PR review then found it broke what the corpus never tests: a salutation alone read its connective as the family name (`Mr. and Mrs.` gave title 'Mr. Mrs.', family 'and' -- the H5 chain took 'Mrs.' before P3 could join), a joined leading salutation counted as three words (`Mr. and Mrs. Jr` gave family 'and', `Mr. and Mrs. Do` a spurious report), and the read counted a particle chain's or a connective join's words one by one where rules/P3 says a joined part is one name word wherever another rule counts (`Freiherr von Berg X.Y.Z.` and `Juan y Garcia X.Y.Z.` took the acronym as a credential against every release, while `Freiherr Berg X.Y.Z.` kept it the family). DECIDED (Derek, 2026-10-07): connectives join first, and the read counts the chain's units -- which returns every one of those to master's reading, and most of the first design's 'tail beats connectives' class with them. A second-round finding the counting view itself made: folding the plain words after a particle into its unit swallowed a shape-only roman numeral (`Juan de la Vega VI` read family 'de la Vega VI'), the #610 class nameparser/AGENTS.md's spelling sweep records, so a word the numeral fork reads by shape stays out of the unit. DECIDED BEFORE THE REVERSAL AND STANDING: (1) the read wins over the particle chain and the bound-given join; (2) the consequences below, accepted as a class. WHAT MOVES. One realistic reading: a trailing period-marked title behind a particle chain is the title, `John van der Berg Prof.` reading title 'Prof.', family 'van der Berg' (every release 1.4.0-2.3.0 and 0c54662b: family 'van der Berg Prof.'), which retires the cost rules/H5 recorded as accepted; the six heads of the stress grid below that end in a particle chain move the same way with `Dr.` or `Prof.` behind them. The garbage `Freiherr von Berg Dr. and Ed. Prof.` reads title 'Freiherr von Berg Dr. and Ed.', family 'Prof.' (0c54662b family 'von Berg Dr. and Ed. Prof.'): the title is read off before the chain, which then joins a piece carrying the joined title and reads as a leading title, so assign reads the name again. P5 no longer re-reads the joined view: `abdul J. V` and `Sir abdul J. V` read given 'abdul J.', family 'V' -- 1.4.0's reading restored; 2.0 and 2.1 read suffix 'V', 2.2 and 2.3 kept 'J.' a middle name -- the initial in front suppressing the numeral fork in the one reading taken, three name words standing and the join leaving two; and `Sheik abdul Ma` reports the pick the read weighed (decisions/P5, this date). Reports: a pick the particle chain takes into the name is reported by the chain, in its own words, and group takes it out of the read's picks so assign does not report it again (#614's /simplify: assign first skipped a pick it found already reported, inferring group's decision from group's output), so `Jan vd Ma`, `John de Ma`, `John van der Berg Ma` and `Nguyen Van Lac` report as on 0c54662b. The second review found why it must be the chain: assign words a pick by the role it holds before post_rules, and under a leading title-particle that is the given role the fold then moves, so with assign reporting `Freiherr von Berg Ma` told the caller 'read as a given name' of a word in the family. The chain also reports a member the read never weighed, one it takes short of the run, its scan ending at the next particle (`Freiherr von Berg Ed Do` reports 'Ed'). One report moves for the better: a pick the run keeps is reported as the suffix it reads as, `Juan y Garcia Mc Do MD van Ed` reporting 'Ed' 'read as a suffix' where 0c54662b said the particle chain took it into the name, of a word in the suffix. THE SECOND REVIEW (bfa798fb, 2026-10-07) found four defects in the counting view and the report hand-off, every one on a shape the corpus, the gates and the first stress grid held no instance of, and each now has a case row that dies without its fix. (1) The read started at the end of the leading titles counted in PIECES, applied to the view: a title that is also a particle opens a unit inside the titles, so the index landed past the name and `Freiherr St van Berg MA` read family 'St van Berg MA' (0c54662b suffix 'MA'); the view now returns where the name starts in its own indices -- the first unit opened at or before the titles' end, since a unit opened inside them makes a name of them (SUPERSEDED IN PART by #624, decisions/P2, 2026-10-10: the chain now opens no unit inside the titles) even where it ends short (`Freiherr Freiherr Prof do` lost its report on 'do' until that second half). (2) Assign reported the chain's picks, in the wrong field's words, above. (3) The view made units INSIDE #602's credential run, which hid a title word from the run's title test: `John Smith MD van Secretary Jones` lost title 'Secretary' to the suffix. Past a run starter a unit is now the bare particle run, as the chain's own merge leaves it. (4) A `title` piece-tag exclusion in `_counted_as_one` made garbage differ from 0c54662b (`Freiherr von Mr. and Mrs. X.Y.Z.`) and guarded nothing (3) did not; removed rather than pinned. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 9ca507316..7912396c4 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -198,7 +198,7 @@ CLDR's person-name data shows why. Its surname-first renderings put the title be | `Brühl Dr. Ada` | family `Brühl`, given `Dr.` | | `Brühl Prof. Dr. Ada Cornelia` | family `Brühl`, given `Prof.`, middle `Dr. Ada Cornelia` | -The plan is to keep order a few permutations and make shapes first-class, each order accepting a listed set of them. The shape inventory (#469) is the start, and it doubles as the test grid that AGENTS.md's comma-shape and `name_order` sweeps describe. The related comma bug, a title before a family comma, was fixed in 2.x ([rules/H6](rules/H6-title-before-a-family-comma.md), a title before a family comma). +The plan is to keep order a few permutations and make shapes first-class, each order accepting a listed set of them. The shape inventory (#469) is the start, and it doubles as the test grid that nameparser/AGENTS.md's comma-shape and `name_order` sweeps describe. The related comma bug, a title before a family comma, was fixed in 2.x ([rules/H6](rules/H6-title-before-a-family-comma.md), a title before a family comma). ### The rules a pack can choose diff --git a/nameparser/AGENTS.md b/nameparser/AGENTS.md index b56209e2d..c61a039c6 100644 --- a/nameparser/AGENTS.md +++ b/nameparser/AGENTS.md @@ -1,6 +1,36 @@ # Working in nameparser/ -Loaded when a session reads or edits anything under `nameparser/`: the pipeline, the vocabulary TOML, the config shim. The root AGENTS.md carries the sweeps every behavior change runs (comma shapes, then `name_order`, under Architecture) and the 2.0 conventions; `nameparser/_pipeline/AGENTS.md` carries the ambiguity-report rules. This file holds the third sweep, which a vocabulary-only change needs as much as a code change does. +Loaded when a session reads or edits anything under `nameparser/`: the pipeline, the vocabulary TOML, the config shim. It holds the sweeps every behavior change runs before it ships — comma shapes, then `name_order`, then spellings, then the encoding — here rather than in the root AGENTS.md because they bind every change to parsing behavior, and the spelling sweep binds a vocabulary-only change as much as a code change. The encoding sweep is the exception there: the vocabulary path NFC-folds, so a TOML-only change does not need it. The root AGENTS.md carries the 2.0 conventions; `nameparser/_pipeline/AGENTS.md` carries the ambiguity-report rules. + +## Sweep the forms a change can reach: comma shapes, then `name_order` + +### Four comma shapes + +No comma, a FULL name before the comma, a ONE-WORD name before the comma and NOTHING before the comma are four paths, and the last two are the misses — #429, #430 and #432 are all the one-word path disagreeing with the full-name path on inputs the full-name path reads correctly, and #613's PR review found `, PhD` and `, Jr.` (an empty surname field, as CSV data writes it) read as given and title by a decision that returned early on an empty head, after two review rounds whose every grid had a name before the comma. Put an empty head and a nickname-only head (`(Bob), PhD`) in every comma grid. + +### Counting the one-word path + +Count the one-word path the way C3 counts, which is neither tokens nor pieces: a particle run and the one word it attaches to are one word (#575), a connective join and a bound given-name pair count as their words (#613, so `Ortega y Gasset, Dr.` is a whole name), and a suffix beside the surname counts as no name word at all. #603's review found the suffix half (`Vega y Lopez Jr., PhD Jones` lost its family where `Vega y Lopez, PhD Jones` kept it); #613 then reversed the connective half, and both now read given 'Vega', middle 'y', family 'Lopez' — so test a particle surname, a connective surname and a surname with a suffix, each before a comma. + +### Orders + +For orders the sweep already exists (`tests/v2/test_cases.py` runs every row under all three) but carries one assertion, R2's family partition, so it checks nothing a new change moves; coverage there has been vacuous before (PR #394's review found the suite passed with `name_order` discarded from grouping). Parse your change's names in each comma shape and each order, and read the ones you did not predict. + +### Reading a grid's moves + +Read a generated grid's moves by SHAPE, not by corpus membership: #614's first design passed the suite, the five gates and a fingerprint of every corpus name, because its grid moves were sorted into 'corpus name moved' and 'garbage', and the review found the moves that mattered among the 'garbage' -- a bare salutation (`Mr. and Mrs.` lost its connective to the family name) and a name with one trailing word (`Freiherr von Berg X.Y.Z.`). Pull out the head-plus-one-word slice of the grid and a salutation with nothing after it, and read every move there. + +### Heads and title runs + +#614's second review found two gaps. The first was a shape: no grid put TWO words that are both a title and a particle at the head (`Freiherr St van Berg MA`, where an index into the pieces was applied to a merged view of them -- `TITLES ∩ particles_ambiguous` is `{freiherr, st}`, so put both in every head list, each in its bare and its period-marked spelling -- #637's grid had `St` and `Freiherr` alone and missed that `St.` and `Freiherr.` are declined by the bound join and then attached by P6). #701's review added one: put a title/suffix dual (`Ms`, `Sr.`, `Cpl`) SECOND in a title run, behind an ordinary title. A stop that asks only "is this word suffix vocabulary" passes every row that opens with the dual and fails `Rev. Sr. Smith, Mary`; it was found by comparing `T Smith, John` against `T John Smith` over every title word, and a comma-versus-comma-less comparison is the cheap check for any rule that moves a word between the two slots. + +### A report's field + +The second was in what the grids compared: they compared ambiguity KINDS, so a report telling the caller the wrong field (`Freiherr von Berg Ma`, 'read as a given name' of a word in the family) passed every pin (#614). So compare each report's `detail` against the field its tokens land in -- since #626 `test_a_report_names_the_field_its_word_lands_in` (tests/v2/test_cases.py) does it over every non-locale case row under all three orders (a row carrying its own policy only as declared), for the phrasings its `_FIELD_CLAIMS` lists -- so a row with the shape is the test only where the report's wording is listed: a new emitter's wording goes in that list, and segment's report on a part past the second comma names no field (#629). #626's own first grid then missed an emitter for want of one comma shape (`, Ma Dr.`: an empty head, an ambiguous word, a trailing title, which H1 moves to the family): a trailing title belongs in every empty-head comma grid too. + +### Exclusion lists and shared predicates + +#620 found that an exclusion list in code being replaced is a set of rules, not noise -- #614's merged copy excluded every word the trailing read weighs from its units, the first #620 prototype dropped the list as a symptom of the copy, and its review found the list had been keeping positional counting's rule that a member in front counts as a word (`Freiherr von Berg MA X.Y.Z.` lost its suffix). Before deleting one, mutate each entry out and read what moves. The converse holds for replacing a hand copy WITH the shared predicate: it is a behavior change wherever the two disagreed, and #735's /simplify shipped one as "no reading moves" because the suite and the snapshot were its only check (`St Sir de Mesnil` family-first, a token test replaced by P1's piece test); diff the two predicates' answers over a grid before calling the swap a refactor. ## Sweep the spellings: a class reached by vocabulary is also reached by SHAPE @@ -41,3 +71,27 @@ Capitals between two name words have the same second reading: some writers put t ### Adjacent pairs of words A predicate over ADJACENT PAIRS of words meets two shapes no grid of one word holds: an overlapping pair and a joining word beside an initial. #752's split-credential test (phase 2e) read `Dr. Ph.` in `Jane Doe Dr. Ph. D.` as the listed `drph`, leaving `D.` a word of its own (found by the maiden take's candidate grid), and a particle beside a surname written short as the listed `ded` (`DE D JUNIOR`, found only in tse-donors' readings), so put a title, a particle and a connective beside each pair spelling, and a pair overlapping another, in the grid. + +## Sweep the encoding: a letter class written `[^\W\d_]` or tested with `isalpha()` is blind to a combining mark + +Python's `\w` and `isalpha()` do not match Mn, so a letter written as base plus accent (NFD) ends a word early; the vocabulary path is immune because `_normalize` NFC-folds, the shape path is not. + +### The two families #652 found + +#652: `Dr. inž. Janez Novak` read `inž.` as the given name when the ž was decomposed, and `Janez Novak, ŽŠČ` reversed, in two families that share the blind spot — the letter-shape regexes (`period_abbreviation`, `_DOTTED`, `_CHUNKED`, `_PAIRED_INITIALS`), now spelled with `_vocab._LETTER` (a letter plus the general combining blocks; `test_regex_sync` compares the two `_MARKS` strings and pins `period_abbreviation` against config's copy), and the caps predicates, which NFC-compose a non-ASCII word before `isalpha()` (`is_caps_word`, behind C-level `isalpha()`/`is_normalized` tests, so already-composed text pays no frame — measured 2026-10-12, `Smith, José` was 193 frames before and after; the first version called it for every non-ASCII word and cost one frame each, and no guard saw it because the reference name is ASCII). + +### Widths and the initial shape + +A width test must count letters, not code points (`composed_len`): admitting a decomposed word to the caps set put `John Smith, MÁ` on the pair exception's wrong side, and the review's grid found it where the rows (all three letters long) did not — put a two-letter word in every caps grid. The initial shape (`_INITIAL`, `\w\.`, v1-verbatim and public) is not widened but asked of the COMPOSED word, in `is_initial_shaped` and in `_render`'s two copies; and the single-letter connective tests count composed letters, since a decomposed `й` is two code points (#665, where `É. y É. Smith` and `John й Smith` read differently decomposed). + +### Compose the WHOLE predicate + +Compose the WHOLE predicate, not one half: `is_initial` is a shape ANDed with the #320 script veto, and composing the shape alone made a decomposed `씨.` an initial, since decomposed hangul is conjoining jamo outside every script range (#665's first version; `in_initialless_script` composes too now). + +### What is deliberately not covered + +Not covered, deliberately: the marks only some scripts write (Indic vowel signs, Arabic and Hebrew points), declined in decisions/nfd-shapes because widening the class would move readings nobody had measured; rules/H2's Accepted clause records the one consequence it names, that a period-marked abbreviation in Bengali or Devanagari never reads as a title by shape. + +### The guards + +The guard is `test_a_decomposed_name_renders_as_its_composed_twin` over the CLDR corpus plus `test_a_decomposed_abbreviation_reads_as_its_composed_twin` (rows × three orders × EVERYWHERE); the corpus guard reaches a shape only if some corpus name does, which is how the caps shape was missed until review, so when adding a shape rule add a row there. Four of #652's rows and two of #665's pass with every fix reverted (measured) and are parity pins, and the test compares ambiguity kinds and `initials()` too, since #665's `John й Smith` differed in its report; `_DOTTED` has no observable input and is pinned by a direct call. diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 2ee2a95b2..5096e9b96 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -915,7 +915,7 @@ def _check_cjk_shape_purity(self) -> None: Case("one_word_head_two_capital_credential", "Smith, DM", {"family": "Smith", "suffix": "DM"}, classification="fix(#797)", - notes="the one-word-head comma path (AGENTS.md's comma sweep): " + notes="the one-word-head comma path (nameparser/AGENTS.md's comma sweep): " "a listed two-capital credential no longer reads as " "paired initials (given 'DM'), as MS and MA do not"), Case("empty_head_joined_credential", ", FNP-C", @@ -994,7 +994,7 @@ def _check_cjk_shape_purity(self) -> None: Case("empty_head_si_is_a_given_name", ", Si", {"given": "Si"}, ambiguities=("suffix-or-name",), - notes="AGENTS.md's empty-head comma grid: with no name before " + notes="nameparser/AGENTS.md's empty-head comma grid: with no name before " "the comma the part is the given name, as 1.4.0 read it " "('first Si')"), Case("empty_head_caps_si_is_a_given_name", ", SI", @@ -2850,7 +2850,7 @@ def _check_cjk_shape_purity(self) -> None: Case("an_elided_particle_beside_a_suffix", "Ivoi, Paul d', Jr.", {"given": "Paul", "family": "d' Ivoi", "suffix": "Jr."}, classification="fix(#700)", - notes="AGENTS.md's comma sweep: a suffix in its own part leaves " + notes="nameparser/AGENTS.md's comma sweep: a suffix in its own part leaves " "the given part's attachment alone. 1.4.0 middle \"d'\""), Case("an_elided_particle_beside_a_glued_comma", "Ivoi, Paul d',Jr.", @@ -4321,7 +4321,7 @@ def _check_cjk_shape_purity(self) -> None: {"title": "Prof.", "suffix": "van der Berg"}, ambiguities=("comma-structure",), classification="fix(#636)", - notes="the empty head (AGENTS.md's comma sweep): the part past " + notes="the empty head (nameparser/AGENTS.md's comma sweep): the part past " "the second comma makes no chain, so the title word is a " "title (rules/C2, #636); 1.4.0 and 2.3.0 read suffix " "'van der Berg Prof.'"), @@ -6534,7 +6534,7 @@ def _check_cjk_shape_purity(self) -> None: {"title": "Prof.", "given": "Jan", "family": "van"}, ambiguities=("particle-or-given",), classification="fix(#623)", - notes="the empty head (AGENTS.md's comma sweep): the bound " + notes="the empty head (nameparser/AGENTS.md's comma sweep): the bound " "holds there too, and P6's attachment, read in post_rules " "behind an empty family part, takes 'van' as the family; " "1.4.0 and 2.3.0 read middle 'van Prof.'"), @@ -8680,7 +8680,7 @@ def _check_cjk_shape_purity(self) -> None: Case("semicolon_glued_after_an_empty_head", ", MD; PhD", {"suffix": "MD; PhD"}, classification="fix(semicolon-suffix-delimiter)", - notes="the empty head AGENTS.md's comma sweep names, read as " + notes="the empty head nameparser/AGENTS.md's comma sweep names, read as " "', MD PhD' is. 1.4.0 through 2.5.0 read given 'MD;'"), Case("semicolon_glued_after_a_nickname_head", "(Bob), MD;", {"nickname": "Bob", "suffix": "MD;"}, @@ -10662,7 +10662,7 @@ def _check_cjk_shape_purity(self) -> None: notes="#626's review: with nothing before the comma, the comma " "report on 'Ma' was worded 'the given name' at assign, " "and H1 then moved it to the family behind the trailing " - "title. The empty-head comma shape AGENTS.md's sweep " + "title. The empty-head comma shape nameparser/AGENTS.md's sweep " "paragraph asks for; 1.4.0 read first 'Ma', suffix 'Dr.'"), Case("lone_joined_unit_carrying_collision_set_title_vocabulary", "Smith and Pope", {"given": "Smith and Pope"}, diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index a8c4af83e..05f23dee4 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -1978,7 +1978,7 @@ def test_the_catalan_recipe_unmarks_the_link_for_one_case_names() -> None: "Janez Novak, univ. dipl. inž.", "Novak, inž. Janez", "Smith, inž. Ž.Š.", - # the comma shapes AGENTS.md asks of every comma grid + # the comma shapes nameparser/AGENTS.md asks of every comma grid ", inž.", "(Bob), inž.", # paired initials (rules/C5) diff --git a/tests/v2/test_properties.py b/tests/v2/test_properties.py index 6b34d02f9..366c269f7 100644 --- a/tests/v2/test_properties.py +++ b/tests/v2/test_properties.py @@ -3288,7 +3288,7 @@ def test_a_semicolon_standing_alone_in_a_tail_reads_as_its_comma_twin() -> None: # as a comma glued there would" -- so a semicolon glued to a word after # the first comma reads as the same text with a space in its place, the # glued word kept in its field with its semicolon. Two inputs (docs/ -# design/AGENTS.md axis 11), over every comma shape AGENTS.md's sweep +# design/AGENTS.md axis 11), over every comma shape nameparser/AGENTS.md's sweep # names (a whole name, one word, an empty head, a nickname head, a # suffix before the comma, a particle, a connective and a title-particle # head), a second comma part or none, and all three orders, with a word