diff --git a/docs/design/decisions/generational-words.md b/docs/design/decisions/generational-words.md index f61f561d5..e6961192b 100644 --- a/docs/design/decisions/generational-words.md +++ b/docs/design/decisions/generational-words.md @@ -10,7 +10,7 @@ Closes #661. 31 spelled-out generational words join SUFFIX_WORDS, and one predic - A refused word carries `excluded = { generation = … }` with its reason in the file of the language whose reading is refused (`uv run python tools/vocab/query.py --excluded generation`) — 2026-10-14 #661; 2026-10-14 #679. - The words ship spaced only, `二世` excepted, so a hyphen-glued Jr. is no suffix: `Теодор Рузвелт-младши` reads family 'Рузвелт-младши' — 2026-10-14 #661. - `二世` is a generation word that peels spaced and glued, the one exception to spaced-only shipping: `陳怡君二世` reads family '陳怡君', suffix '二世', and its `ja.toml` and `zh.toml` entries carry `issue = 671` and `attached` with `issue = 707` — 2026-10-21 #762. -- The generational words whose collisions are with OTHER languages written in the same script (fr `père`; the Latin abbreviations `ml.` and `st.`) ship in no default, that being the case a locale pack exists for: `Jiří Dienstbier ml.` reads family 'ml.', and no cs pack exists; cs `st.` is recorded as a reading off by default, for that pack — 2026-10-14 #679; 2026-10-25 #752. +- The generational words whose collisions are with OTHER languages written in the same script (fr `père`; the Latin abbreviations `ml.` and `st.` in cs, hr, sk and sl) ship in no default, that being the case a language choice exists for: each is a reading written `default = false` in its language's file, so `Jiří Dienstbier ml.` reads family 'ml.' by default and suffix 'ml.' with `cs` chosen — 2026-10-14 #679; 2026-10-25 #752; 2026-10-29 #753, phase 3c. - fr `fils` is a generation reading gated by its family reading, its case and the count (rules/S11): `Alexandre Dumas fils` reads suffix 'fils' and `Marie Fils` family 'Fils', both reported — 2026-10-25 #752. Elsewhere: the run start that lets any unambiguous suffix word of two or more letters begin the credential run, whatever its script (`Antonio Pavlović mlađi Jones` reads suffix 'mlađi Jones', as does `Pieter Bruegel stariji Jones` with 'stariji Jones'), is decisions/S8's 2026-10-14 (Derek), #679 entry, which retracted this file's #602 run-start bullet; the glued peel of 二世 is decisions/W2's 2026-10-17 #707 entry. @@ -38,3 +38,5 @@ Open: #684 (locale packs for `Jiří Dienstbier ml.` and `st.`, the generational - **2026-10-23 #843 (Derek) — JÚNIOR SHIPS; THE AGNOMENS WAIT.** "Suffix, Júnior only for now." `júnior` joins `pt.toml` as a generation reading with `same_as = "jr"`: the standard Portuguese spelling, which `José Silva Júnior` read as family 'Júnior', middle 'Silva' in 1.4.0 and 2.5.0 (checked against both wheels) while `José Silva Junior` read suffix 'Junior'. Its collisions are `junior`'s, already accepted: a given name (Wikidata 37+ as Júnior, 43,182+ US SSA births as Junior), where the suffix claim does not act (`Júnior Silva` keeps given 'Júnior'), and family bearers (Wikidata 49+, US Census 2,146 as JUNIOR), which a suffix word costs its surname wherever it ends a name (`Pedro Júnior` reads given 'Pedro', suffix 'Júnior', as `Pedro Junior` does), counts from `tools/name_usage.py` on 2026-10-23. In the local corpus `tse-donors` (fetched 2026-10-22), 1,071 of its 854,531 records write it anywhere in the name (`tools/name_usage.py`), 991 distinct filed names end in it, and 983 records end in it, in either case, where the tax registry's name for the same donor ends `JUNIOR` (last words compared after `str.upper()`). The accented spelling reads as the unaccented one in all 120 pairs of a grid of 20 shapes (comma shapes, an empty head, a title, a particle, a trailing title, a nickname, a maiden clause, a credential and a name word behind it, both one-case spellings) under all three orders, composed and decomposed, comparing fields, ambiguity kinds and `initials()`; with `júnior` removed from the lexicon (`Lexicon.default().remove(suffix_words={'júnior'})`), 108 of the 120 pairs differ, and the 12 that do not hold Júnior where no suffix reading reaches it (`Júnior Silva`, and `Júnior, José`, before a family comma). Measured 2026-10-23 on `tse-donors` against the tree before this change (`snapshot.py --local tse-donors`, baseline recorded with `--tree`): 993 of 848,351 filed names move, 990 ending in Júnior and gaining it as the suffix out of the family name (3 of them two-word names, `RONALDO JÚNIOR`, left with no family as `RONALDO JUNIOR` is), and 3 with Júnior inside the name, where it starts a credential run (decisions/S8) that takes the surnames behind it, `JORGE ANTÔNIO JÚNIOR BARBOSA SILVA` reading family 'ANTÔNIO', suffix 'JÚNIOR BARBOSA SILVA' as the JUNIOR spelling already did; and `local_corpora.py pairs tse-donors --agreement` stops listing JÚNIOR / JUNIOR (866 accents-only pairs read with two surnames before, none after). The agnomens (distinct filed names ending in them: `Filho` 10,979, `Neto` 8,439, `Sobrinho` 955, `Netto` 432) stay family names for now. They are legally part of the surname, and `Neto` is a borne surname and given name alone (Wikidata 276+ family, 12+ given), which a suffix reading would leave with no family, as `Pedro Junior` has none. No exclusion records them, because nothing was refused: the question is open on #843. - **2026-10-25 #752 — THE AGNOMENS AND `fils` ARE GATED GENERATION READINGS (phase 2h).** SUPERSEDES: 2026-10-23 #843's agnomen half, which kept the agnomens family names, and 2026-10-14 #679's refusal of `fils` as pack-shaped. The 2026-10-23 #752 entry in decisions/vocabulary-model gave each generation reading a gate by its kind, and the record, with its measurements, is decisions/S11. `fils` collides inside French, a generation word and a French surname, so no French pack could settle it, which is the premise of #684 the design reversed: its `generation` reading now stands beside its `family` reading, and the gate reads the name around it. The agnomens are not part of the surname: an agnomen marks its bearer against a relative of the same name, as Júnior does, and its name readings (dated counts in `pt.toml` and `pt-BR.toml`, `netto` there being the spelling of Brazil's older orthography) keep a two-word name's family. cs `st.` keeps no exclusion: it is a `generation` reading written `default = false`, recorded for a Czech pack, since nothing in its writing tells it from Saint; hr, sk and sl keep theirs, as do `ml.` and `père`. + +- **2026-10-29 #753, phase 3c — `ml.`, `st.` AND `père` ARE READINGS OFF BY DEFAULT.** The exclusions #679 recorded as pack-shaped are `generation` readings written `default = false` (cs `ml`, hr, sk and sl `ml` and `st`, fr `père`; cs `st` since phase 2h), read when their language is chosen, and the default does not move. The record, with what each choice moves, is decisions/vocabulary-model's 2026-10-29 #753, phase 3c entry. diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 05e4cd050..eb79a9e71 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -12,6 +12,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - A patronymic field is to come in 3.0, as a view that names its words without emptying `family`, and the particle kinds already draw the `clan`/`patronymic` line for it: a patronymic is followed by a parent's name, a clan prefix by a family name the lineage shares — 2026-10-15/16; 2026-10-19 #749; 2026-10-28 #754 (Derek). - A language pack is a choice of languages on `Locale.languages` selecting which readings compile, a tag choosing its own file and its ancestors', and a reading index, loaded on its first lookup, carries the languages to run time; strict waits for 3i, and third-party language files are deferred — 2026-10-27 #753 (Derek); 2026-10-29 #753, phase 3a (Derek); 2026-10-29 #753, phase 3b. - A choice is matched without regard to case, a tag whose file does not exist selects its ancestors' (`pt-PT` selects `pt`), `mul` and `und` are no language to choose, and a reading turned on also prefers, so `de`'s generation `i` drops `ca`'s and `pl`'s conjunction under a `de` choice; a choice is one call's: one chained through `base=` applies on top of the base's rather than unioning with it, and a choice drops the caller's own base entries it removes — 2026-10-29 #753, phase 3b; 2026-10-29 #753, phase 3b, chained choices (Derek). +- A reading right in its own language and wrong only as a default, because another language written in the same script collides with it, is written `default = false` in its file, never excluded, so that after the packs an exclusion means wrong everywhere: cs `st` since phase 2h, and cs `ml`, hr, sk and sl `ml` and `st` and fr `père` since 3c, the rest in their packs' PRs — 2026-10-27 #753 (Derek); 2026-10-29 #753, phase 3c. - A name reading written `default = false` derives no marker in the default and counts for no check guarding the default's compile; a particle whose word's only `given` readings are off is refused until a pack needs one — 2026-10-29 #753, phase 3a. - A caller configures vocabulary by reading, `Lexicon` deriving the sets and gates the build derives, and the compiled sets become internal in 3.0 — 2026-10-28 #754, the `Lexicon` surface (Derek). - A `Locale` may carry CLDR-shaped render patterns, display and sorting, a later addition whose shape is agreed; how a call reaches one is open — 2026-10-28 #754, render patterns. @@ -312,3 +313,10 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 1. **Chained choices compose one after the other.** `parser_for(EN, base=parser_for(PT))` reads `Maria do Carmo` family `Carmo` and `John Smith DO` family `DO`: `en` drops `pt`'s particle and `pt` had dropped `en`'s credential. The way to choose both is one call, `parser_for(PT, EN)`, which keeps both readings (`do Carmo`, suffix `DO`). 2. **A choice drops what it removes from the base, whoever put it there.** `parser_for(EN, base=Parser(lexicon=Lexicon.empty().add(particles={'do', 'van'})))` keeps only `van`: the delta removes `do` from `particles`, the caller's listing included, and leaves every entry it does not touch. Weighed against: (b) a `Parser.languages` field recording the choice, so a base's choice unions with the packs' (a new public name, safe for pickles, `Parser`'s guard honoring `ADDED_IN`), deferred rather than declined; (c) a `Lexicon` field recording it, which every older `Lexicon`, `Parser` and pack pickle would refuse, `Lexicon`'s guard not honoring `ADDED_IN`. Open: (b), with the configure-by-reading method names (the 2026-10-28 #754 `Lexicon` surface entry), and with it whether `Reading.languages` follows the choice (the phase 3b entry above). + +- **2026-10-29 #753, phase 3c — THE EIGHT PACK-SHAPED EXCLUSIONS ARE READINGS OFF BY DEFAULT: cs `ml`, hr, sk and sl `ml` and `st`, fr `père`; THE DEFAULT DOES NOT MOVE.** Carries out the 3c half of item 9 of the 2026-10-27 #753 entry, and 3c of its item 11. + - **The split.** Every exclusion whose reason opened "pack-shaped" refused a `generation` reading that is right in its language and wrong as a default only because another language written in Latin script collides with it: `ml.` against the initials `M. L.`, `st.` against Saint, `père` outside French. Each is now a `generation` reading written `default = false` in its own file, with the issue list it had and #753, an `expands` naming the full word, and a `note` giving the collision, the shape cs `st` has had since 2026-10-25 (phase 2h). The other exclusions a pack turns on (`pan`, `roz`, `nic`, `shri`, `sig`, `jaun`, `bt`, `bay`, `sb`) stay exclusions until the PR of the pack that turns each on; every other exclusion is wrong everywhere and stays. Recompute: `grep -l 'pack-shaped' nameparser/vocab/*.toml` lists no file, which tests/v2/test_locales.py's `test_no_exclusion_is_pack_shaped_any_more` holds. + - **The default does not move.** The build reports `_compiled.py` current, only `_index.py` gaining the eight rows (11 readings written `default = false`, from 3), and tests/v2/test_snapshot.py passes with no re-record. + - **What choosing each language now does,** measured 2026-10-29 on `b27dadd5` plus this change with `snapshot.py --languages TAG`, entries moved (names held only by `corpus_cldr.jsonl` among them): `cs`, `sk`, `sl` and `hr` 14 each (4), where `cs` moved 11 (2) and the other three none before; `fr` 0, no committed name holding `père`. The three new entries under each read `ml.` as the suffix: the rules corpus's `Jiří Dienstbier ml.`, and two CLDR renderings with `ml.` before a comma of credentials (`g. Janez Peter Novak ml., univ. dipl. inž.`). The other eleven are `st`, and each language's turning on its reading also prefers it (the phase 3b entry's turning-on bullet), so en's and fr's title and particle readings of `st` leave: `St Clair` reads given `St`, family `Clair`, where the default reads title `St`. That cost reached `cs` with 3b; it reaches `sk`, `sl` and `hr` here, and settling it, whether a Saint reading should outlast a choice that holds `st` as a generation, is 3f's, which turns those packs on. tests/v2/test_locales.py's `test_a_chosen_language_reads_its_off_generation_and_the_default_does_not` parses every generation reading off by default of more than one letter (`Anna Novak ml.`, with and without the period) under its language and the default; a one-letter word (es `h`, de `i`) is left out, the initial veto taking it written with its period (`Anna Novak h.` reads family `h.` under `es`, while a bare `h` reads suffix `h`). + - **Open for 3f: a turned-on `ml` reads a lone comma part as the suffix, silently.** `ml` has no name reading, so under a choice it is an ungated suffix word: `Novak, ML` reads suffix `ML` with nothing reported, where the default reads given `ML`, and `Novak ml., Anna` family `Novak`, suffix `ml.`; the gated `st` reports `SUFFIX_OR_NAME`, and fr `Jean Père` reports `GIVEN_OR_FAMILY` (suffix `Père`). The initial pairs `Anna M. L. Novak` and `Anna ML Novak` do not move. Pinned in tests/v2/test_locales.py; whether `ml` takes a name reading or a gate is 3f's. So is the one-letter gap: es `h` and de `i` turned on stay inert written with a period (`Juan Pérez H.` keeps family `H.` under `es`, `Wilhelm Karl I.` family `I.` under `de`), a strict expected failure there. + - Not touched: no `Lexicon` or `Parser` field, no code. The vocabulary README's example of an exclusion is now `duke`, wrong everywhere, where it was `père`. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 4490b88d8..344c16d40 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -26,7 +26,7 @@ Four phases, in a fixed order: |---|---|---|---| | 1 | The per-word TOML format, compiled to today's sets | no | shipped in 2.6 (#737, #744); #749–#751 gave the words in `und.toml` their languages, leaving two particles of unrecorded origin | | 2 | Ambiguity derived from clashing readings; the ambiguity gate extended to every reading kind | yes, the first phase that does | #752, design agreed 2026-10-23; 2a, the two name-clash markers derived, landed without moving a parse (decisions/vocabulary-model, 2026-10-25) | -| 3 | Language packs that prefer or restrict their language | yes, for callers who choose one | #753, for 3.0, needs 1 and 2; design agreed 2026-10-28, 3g's connective open; 3a, the reading index and the shared compiler, landed without moving a parse (decisions/vocabulary-model, 2026-10-29) | +| 3 | Language packs that prefer or restrict their language | yes, for callers who choose one | #753, for 3.0, needs 1 and 2; design agreed 2026-10-28, 3g's connective open; 3a, the reading index and the shared compiler, 3b, the choice on `Locale.languages`, and 3c, the pack-shaped exclusions as readings off by default, landed without moving a default parse (decisions/vocabulary-model, 2026-10-29) | | 4 | Order and shape; the rules a pack can choose; the output model | yes | #754, 3.0 work; its output model agreed 2026-10-28, ahead of or alongside phase 3's mechanism | Two commitments hold through all four. **The default loads every language and keeps working for names of any language, as today**; choosing a language narrows it (decision 5 in decisions/vocabulary-model). And **languages own vocabulary and defaults, the core owns structure**: a pack configures rules the core implements, and never ships code (mechanisms.md#LOCALE-PACKS-PURE-DATA). @@ -164,7 +164,7 @@ Each sub-phase is one PR, measured on its own base: the committed snapshot read ## Phase 3: packs that prefer or restrict -Status: **agreed** (proposed 2026-10-27, #753; Derek accepted every item on 2026-10-28, six and then the other six); **3a landed 2026-10-29**, the reading index, the compiler moved into the package and `default = false` on name readings (decisions/vocabulary-model's 2026-10-29 entry, which records what 3a decided and measured, and the index's lazy load, which Derek chose on 2026-10-29 over loading it with `Lexicon.default()`; "Where the provenance lives at runtime" below describes it). **3b landed 2026-10-29**, the choice: `Locale.languages`, its tag selection, turning on, prefer and the shape clash of `may_be_initial`, the derived deviation surface, `snapshot.py --languages`, and mechanisms.md#LOCALE-PACKS-PURE-DATA amended (decisions/vocabulary-model's 2026-10-29 phase 3b entry, which records what it decided, measured and left open). The record is decisions/vocabulary-model's 2026-10-27 #753 entry; this section describes its items. One design inside the work order is still open: 3g's connective (below), which accepting the work order did not decide. Phase 3 is 3.0 work (re-planned 2026-10-28): 2.8 is the last 2.x feature release, and 3.0 ships phases 3 and 4 with the facade's removal. Packs stay opt-in, so a caller who chooses no language parses as the default does, and nothing in this phase moves the default's parses. One standing text is amended with it, in 3b's PR, which builds the selection: mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack adds, and selects among the default's readings by language, its deviations derived for a vocabulary pack). decisions/3-0-reevaluations' line that removing base vocabulary through a pack is part of 3.0's pack design is carried out rather than moved. +Status: **agreed** (proposed 2026-10-27, #753; Derek accepted every item on 2026-10-28, six and then the other six); **3a landed 2026-10-29**, the reading index, the compiler moved into the package and `default = false` on name readings (decisions/vocabulary-model's 2026-10-29 entry, which records what 3a decided and measured, and the index's lazy load, which Derek chose on 2026-10-29 over loading it with `Lexicon.default()`; "Where the provenance lives at runtime" below describes it). **3b landed 2026-10-29**, the choice: `Locale.languages`, its tag selection, turning on, prefer and the shape clash of `may_be_initial`, the derived deviation surface, `snapshot.py --languages`, and mechanisms.md#LOCALE-PACKS-PURE-DATA amended (decisions/vocabulary-model's 2026-10-29 phase 3b entry, which records what it decided, measured and left open). **3c landed 2026-10-29**, the eight pack-shaped exclusions (cs `ml`, hr, sk and sl `st` and `ml`, fr `père`) written as `generation` readings `default = false`, the default's compile unchanged (decisions/vocabulary-model's 2026-10-29 phase 3c entry, which records what each choice now moves). The record is decisions/vocabulary-model's 2026-10-27 #753 entry; this section describes its items. One design inside the work order is still open: 3g's connective (below), which accepting the work order did not decide. Phase 3 is 3.0 work (re-planned 2026-10-28): 2.8 is the last 2.x feature release, and 3.0 ships phases 3 and 4 with the facade's removal. Packs stay opt-in, so a caller who chooses no language parses as the default does, and nothing in this phase moves the default's parses. One standing text is amended with it, in 3b's PR, which builds the selection: mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack adds, and selects among the default's readings by language, its deviations derived for a vocabulary pack). decisions/3-0-reevaluations' line that removing base vocabulary through a pack is part of 3.0's pack design is carried out rather than moved. **Phase 4's shape questions came first.** The output model decides what a pack can express: a patronymic field, a second surname, credentials apart from generation suffixes (#326), and what `middle` means (#704). A pack reading of a kind with no field to land in is a reading nothing can surface, so those questions were to be settled before or with phase 3's mechanism, by the same shape-before-value rule, and the output model was agreed on 2026-10-28 (decisions/vocabulary-model's 2026-10-28 #754 entry): the seven roles stay, and `generation`, `credentials`, `patronymic` and `surname2` are views read from `Token.reading` or rules/O6's tag. Of the work order below, 3a (the reading index and the shared compiler) and 3c (the pack-shaped exclusions converted) are independent of the output model: the index carries whatever kinds the files hold, and 3c moves data between two encodings that compile alike. 3b's selection semantics are independent too, and its API's dependency, what `Locale` and `Policy` look like once the facade is gone, is answered (below). The packs depend on the output model in part, each on the phase 4 sub-phase that builds its view: `cs`'s generations and `pt-BR`'s agnomens on 4b's `generation` (3f and 3h), `es` and `pt` on 4d's `surname2` and 4g's compound given name (3g); `ne`'s readings do not. diff --git a/docs/release_log.rst b/docs/release_log.rst index 93a2db718..072b8bdeb 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -6,7 +6,7 @@ Release Log **Additions** - - **Add Locale.languages, a choice of languages for the default vocabulary.** ``parser_for(Locale(code="pt", lexicon=Lexicon.empty(), languages=frozenset({"pt"}))).parse("ANA SILVA E SOUZA")`` gives family ``SILVA E SOUZA``, where the default reads family ``SOUZA``, the ``E`` an initial: a chosen language's reading of a word it has a vocabulary reading of wins, and the other languages' vocabulary readings of that word leave, while its readings as a name and the readings shared by every language stay, so under ``pt`` ``John Smith DO`` reads family ``DO``. A pack choosing no language parses as before. See :doc:`locales`. :issue:`753` + - **Add Locale.languages, a choice of languages for the default vocabulary.** ``parser_for(Locale(code="pt", lexicon=Lexicon.empty(), languages=frozenset({"pt"}))).parse("ANA SILVA E SOUZA")`` gives family ``SILVA E SOUZA``, where the default reads family ``SOUZA``, the ``E`` an initial: a chosen language's reading of a word it has a vocabulary reading of wins, and the other languages' vocabulary readings of that word leave, while its readings as a name and the readings shared by every language stay, so under ``pt`` ``John Smith DO`` reads family ``DO``. A generation word another language collides with is read only under its own language: ``Jiří Dienstbier ml.`` gives suffix ``ml.`` under ``cs``, and the Czech, Slovak, Slovenian and Croatian ``ml.`` and ``st.`` and the French ``père`` read this way. A pack choosing no language parses as before. See :doc:`locales`. :issue:`753` - **Add Token.reading, the reading the parse took for each word.** ``parse("John Smith PhD Jones").tokens[-1].reading`` is ``Reading('credential' as 'suffix', 'position')``, the reading 2.7's ``Ambiguity.alternative`` gave only for the other side of a fork, and ``ReadingKind`` gains ``patronymic``. The field is not compared, and a ``ParsedName`` pickled by 2.x loads with no readings. :issue:`754` - **Add Reading.languages, the language files that list a word the way it was read.** ``parse("Dr. Juan de la Vega III").tokens[2].reading.languages`` is ``('es', 'fr', 'it', 'nl', 'pt')`` for the particle ``de``, and ``Ambiguity.alternative`` names its files too; a reading by shape or position names none. It is not compared, and a ``Reading`` pickled by 2.7 or 2.8 loads with none. The first parse in a process that meets a listed word loads the reading index once: about 1 ms, or about 16 ms where Python writes no bytecode cache. :issue:`754`, :issue:`753` diff --git a/nameparser/config/_index.py b/nameparser/config/_index.py index 9b022b4bb..c864eedcf 100644 --- a/nameparser/config/_index.py +++ b/nameparser/config/_index.py @@ -1066,6 +1066,7 @@ ('misses', (('en', 'salutation', (('after_given', True),)),)), ('missionary', (('en', 'occupation', ()),)), ('mister', (('en', 'salutation', ()), ('en', 'family', ()))), + ('ml', (('cs', 'generation', (('default', False),)), ('hr', 'generation', (('default', False),)), ('sk', 'generation', (('default', False),)), ('sl', 'generation', (('default', False),)))), ('mladšia', (('sk', 'generation', ()),)), ('mladší', (('cs', 'generation', ()), ('sk', 'generation', ()))), ('mlajša', (('sl', 'generation', ()),)), @@ -1280,6 +1281,7 @@ ('pursuivant', (('en', 'office', ()),)), ('pv2', (('en', 'military', ()),)), ('pvt', (('en', 'military', ()),)), + ('père', (('fr', 'generation', (('default', False),)),)), ('qam', (('en', 'credential', ()),)), ('qc', (('en', 'credential', ()),)), ('qcsw', (('en', 'credential', ()),)), @@ -1445,7 +1447,7 @@ ('sscp', (('en', 'credential', ()),)), ('ssg', (('en', 'military', ()),)), ('ssgt', (('en', 'military', ()),)), - ('st', (('cs', 'generation', (('default', False),)), ('en', 'religious', ()), ('en', 'saint', ()), ('en', 'given', ()), ('fr', 'religious', ()), ('fr', 'saint', ()))), + ('st', (('cs', 'generation', (('default', False),)), ('en', 'religious', ()), ('en', 'saint', ()), ('en', 'given', ()), ('fr', 'religious', ()), ('fr', 'saint', ()), ('hr', 'generation', (('default', False),)), ('sk', 'generation', (('default', False),)), ('sl', 'generation', (('default', False),)))), ('staff', (('en', 'modifier', ()),)), ('starejša', (('sl', 'generation', ()),)), ('starejši', (('sl', 'generation', ()),)), diff --git a/nameparser/vocab/AGENTS.md b/nameparser/vocab/AGENTS.md index 29a306430..c044bc5d5 100644 --- a/nameparser/vocab/AGENTS.md +++ b/nameparser/vocab/AGENTS.md @@ -20,7 +20,7 @@ Check the predicates that test a member's SPELLING rather than its membership: t ## Step 4: record the vetting on the word -Record what the vetting found ON the word, not in a comment or a decisions sentence: `note`, `issue`, a `given`/`family` reading with dated counts, and for a word you decided to keep out an `excluded` entry with its reason, so the next contributor who adds it meets the reason as a build error (decisions/vocabulary-model). A name reading is not only provenance on a word that has a credential, particle or title reading: it IS the clash, and derives the marker (an ambiguous acronym, a particle that may be given, since 2026-10-25, a generation or agnomen that may be a name, rules/S11, since phase 2h the same day, so given-name counts recorded on `junior` would gate it, and a title that may be a given name, rules/H9, since phase 2i), so counts kept as evidence AGAINST marking such a word go in the clashing reading's `note` instead, as `ld`'s do. +Record what the vetting found ON the word, not in a comment or a decisions sentence: `note`, `issue`, a `given`/`family` reading with dated counts, and for a word you decided to keep out an `excluded` entry with its reason, so the next contributor who adds it meets the reason as a build error (decisions/vocabulary-model) — unless the reading is right in its own language and wrong only as a default, because another language collides with it, which is a reading written `default = false` instead, for a language choice to turn on (3.0's phase 3c, cs `ml`). A name reading is not only provenance on a word that has a credential, particle or title reading: it IS the clash, and derives the marker (an ambiguous acronym, a particle that may be given, since 2026-10-25, a generation or agnomen that may be a name, rules/S11, since phase 2h the same day, so given-name counts recorded on `junior` would gate it, and a title that may be a given name, rules/H9, since phase 2i), so counts kept as evidence AGAINST marking such a word go in the clashing reading's `note` instead, as `ld`'s do. ## Step 6: the user docs' doctests diff --git a/nameparser/vocab/README.md b/nameparser/vocab/README.md index d5fdab493..14141de1d 100644 --- a/nameparser/vocab/README.md +++ b/nameparser/vocab/README.md @@ -54,12 +54,13 @@ note = "Vietnamese Đỗ leads a great many surnames" ```toml [[word]] -text = "père" -[word.excluded] -generation = { reason = "pack-shaped: collides outside French", issue = [679, 684] } +text = "duke" +given = { us_ssa = 11595, wikidata = 151, as_of = 2026-10-19, note = "male births only" } +family = { us_census_2010 = 35053, wikidata = 315, as_of = 2026-10-19 } +excluded = { nobility = "a common given name (Duke Kahanamoku), and a surname besides" } ``` -`père` ("father", the French word for the Elder) is deliberately not a suffix. It has no vocabulary reading at all, only an exclusion refusing the `generation` reading with the reason. Anyone who adds `generation = {}` to `père` in any file gets a build error that quotes that reason. Its son, `fils`, was refused the same way until 2026-10-25; it now has the `generation` reading beside its `family` reading, which gates it by the name around it ([derived markers](#derived-markers)), so `Marie Fils` keeps her family name. +`duke` is deliberately not a title. It has no vocabulary reading at all, only its two name readings and an exclusion refusing the `nobility` reading with the reason. Anyone who adds `nobility = {}` to `duke` in any file gets a build error that quotes that reason. An exclusion means the reading is wrong wherever it is read. A reading that is right in its own language and wrong only as a default is written `default = false` instead ([Properties](#properties)), as `père` ("father", the French word for the Elder) is in `fr.toml`, so that choosing French turns it on; it was an exclusion until 3.0's phase 3c. Its son, `fils`, was an exclusion too until 2026-10-25, and is now on by default: its `generation` reading stands beside its `family` reading, which gates it by the name around it ([derived markers](#derived-markers)), so `Marie Fils` keeps her family name. ```toml [[word]] @@ -82,7 +83,7 @@ Two rules about the `text`: Every file but `und.toml` lists its words in one order, and the build refuses a file that does not (check 10), naming the entry to move and where it belongs: -1. **By kind**, in the order of the table under [Kinds](#kinds), which lists the categories in turn and each category's kinds together. A word with several readings sorts under its first kind in that order. A word whose only vocabulary reading is a refusal (`père`, `duke`) sorts under the kind it refuses, a refused category counting as its first kind, and a name reading places a word only when it has nothing else. +1. **By kind**, in the order of the table under [Kinds](#kinds), which lists the categories in turn and each category's kinds together. A word with several readings sorts under its first kind in that order. A word whose only vocabulary reading is a refusal (`duke`) sorts under the kind it refuses, a refused category counting as its first kind, and a name reading places a word only when it has nothing else. 2. **Then by the word it is a spelling of**: a reading's `same_as`, or the word's own text when it has none. A head comes first and its spellings follow it, so `jr` is followed by `jnr` and `junior`, and `sr` by `snr`. A spelling that has spellings of its own heads a group inside its head's: in `ar.toml`, `el-` follows the `a…-` spellings of `al-` with its own assimilated forms after it, and `ul-` comes last. Only the file's own entries are followed, so a word that is a head in one file and a spelling in another is placed in each file by that file alone. 3. **Then by the word's text.** @@ -228,12 +229,12 @@ A name reading may stand without a count, as a particle kept possibly given unde ## Exclusions -An exclusion records that a reading was considered and refused, with the reason, so that the next person to propose it meets the reason as a build error instead of rediscovering the evidence. +An exclusion records that a reading was considered and refused, with the reason, so that the next person to propose it meets the reason as a build error instead of rediscovering the evidence. It means the reading is wrong in every language: a reading right in its own language and wrong only as a default, because another language collides with it, is written `default = false` instead ([Properties](#properties)), for a choice of that language to turn on, as cs `ml` is. - **A refused reading** is a word-level table: `[word.excluded.generation]` with a `reason`, or `excluded = { generation = "the reason" }`. It refuses that kind in every file, and also every other kind that compiles into the same set, so a generic `suffix = {}` cannot put back a refused `generation`. A category name refuses the whole category (`excluded = { suffix = "..." }`). A `name` kind refuses only itself. - **A refused property** sits on a reading the word keeps: `[word.honorific.excluded]` with `attached = "the reason"`. It refuses any value of that property other than the one it takes unwritten, on any reading of the category; a refused `cased`, which belongs to the word, on any reading of the word. -The value is a reason string, or a table with `reason` and optionally `issue`, `decided` and `note`. The exclusion goes in the file of the language whose reading it refuses. An entry may hold exclusions and no vocabulary reading, as `père` does. An entry with no reading and no exclusion is refused as inert, and one holding only a name reading is a warning, which the test suite makes an error, unless that reading is a `given` reading carrying `same_as`, which compiles. +The value is a reason string, or a table with `reason` and optionally `issue`, `decided` and `note`. The exclusion goes in the file of the language whose reading it refuses. An entry may hold exclusions and no vocabulary reading, as `duke` does. An entry with no reading and no exclusion is refused as inert, and one holding only a name reading is a warning, which the test suite makes an error, unless that reading is a `given` reading carrying `same_as`, which compiles. To see what has been refused, and why: `uv run python tools/vocab/query.py --excluded generation`. diff --git a/nameparser/vocab/cs.toml b/nameparser/vocab/cs.toml index 110b386ba..c40f11e15 100644 --- a/nameparser/vocab/cs.toml +++ b/nameparser/vocab/cs.toml @@ -19,9 +19,7 @@ salutation = { issue = 606, gender = "f", after_given = true } [[word]] text = "ml" -[word.excluded.generation] -reason = "pack-shaped: real and standard, but collides with ML initials in other languages written in Latin script" -issue = [661, 679, 684] +generation = { default = false, expands = "mladší, the Younger: Jiří Dienstbier ml.", issue = [661, 679, 684, 753], note = "real and standard, but ML is a pair of initials in other languages written in Latin script, and nothing in its writing tells the two apart, so no default reads it; a Czech pack turns it on" } [[word]] text = "mladší" diff --git a/nameparser/vocab/fr.toml b/nameparser/vocab/fr.toml index 48750301d..581661971 100644 --- a/nameparser/vocab/fr.toml +++ b/nameparser/vocab/fr.toml @@ -102,8 +102,7 @@ family = { us_census_2010 = 767, wikidata = 5, as_of = 2026-10-16 } [[word]] text = "père" -[word.excluded] -generation = { reason = "pack-shaped: collides outside French", issue = [679, 684] } +generation = { default = false, expands = "father, the Elder: Alexandre Dumas père", issue = [679, 684, 753], note = "real and standard, but collides outside French, so no default reads it; a French pack turns it on" } [[word]] text = "d'" diff --git a/nameparser/vocab/hr.toml b/nameparser/vocab/hr.toml index 33fbae343..f32983974 100644 --- a/nameparser/vocab/hr.toml +++ b/nameparser/vocab/hr.toml @@ -14,9 +14,7 @@ salutation = { issue = 606, gender = "f", after_given = true } [[word]] text = "ml" -[word.excluded.generation] -reason = "pack-shaped: real and standard, but collides with ML initials in other languages written in Latin script" -issue = [661, 679, 684] +generation = { default = false, expands = "mlađi, the Younger", issue = [661, 679, 684, 753], note = "real and standard, but ML is a pair of initials in other languages written in Latin script, and nothing in its writing tells the two apart, so no default reads it; a Croatian pack turns it on" } [[word]] text = "mlađa" @@ -28,9 +26,7 @@ generation = { issue = 661 } [[word]] text = "st" -[word.excluded.generation] -reason = "pack-shaped: real and standard, but St. is Saint in other languages written in Latin script" -issue = [661, 679, 684] +generation = { default = false, expands = "stariji, the Elder", issue = [661, 679, 684, 753], note = "real and standard, but St. is Saint in other languages written in Latin script, and nothing in its writing tells the two apart, so no default reads it; a Croatian pack turns it on" } [[word]] text = "starija" diff --git a/nameparser/vocab/sk.toml b/nameparser/vocab/sk.toml index 33b27a145..0c4dbf4dd 100644 --- a/nameparser/vocab/sk.toml +++ b/nameparser/vocab/sk.toml @@ -2,9 +2,7 @@ schema = 1 [[word]] text = "ml" -[word.excluded.generation] -reason = "pack-shaped: real and standard, but collides with ML initials in other languages written in Latin script" -issue = [661, 679, 684] +generation = { default = false, expands = "mladší, the Younger", issue = [661, 679, 684, 753], note = "real and standard, but ML is a pair of initials in other languages written in Latin script, and nothing in its writing tells the two apart, so no default reads it; a Slovak pack turns it on" } [[word]] text = "mladšia" @@ -16,9 +14,7 @@ generation = { issue = 661 } [[word]] text = "st" -[word.excluded.generation] -reason = "pack-shaped: real and standard, but St. is Saint in other languages written in Latin script" -issue = [661, 679, 684] +generation = { default = false, expands = "starší, the Elder", issue = [661, 679, 684, 753], note = "real and standard, but St. is Saint in other languages written in Latin script, and nothing in its writing tells the two apart, so no default reads it; a Slovak pack turns it on" } [[word]] text = "staršia" diff --git a/nameparser/vocab/sl.toml b/nameparser/vocab/sl.toml index 43716d568..acb024f6f 100644 --- a/nameparser/vocab/sl.toml +++ b/nameparser/vocab/sl.toml @@ -2,9 +2,7 @@ schema = 1 [[word]] text = "ml" -[word.excluded.generation] -reason = "pack-shaped: real and standard, but collides with ML initials in other languages written in Latin script" -issue = [661, 679, 684] +generation = { default = false, expands = "mlajši, the Younger", issue = [661, 679, 684, 753], note = "real and standard, but ML is a pair of initials in other languages written in Latin script, and nothing in its writing tells the two apart, so no default reads it; a Slovenian pack turns it on" } [[word]] text = "mlajša" @@ -16,9 +14,7 @@ generation = { issue = 679 } [[word]] text = "st" -[word.excluded.generation] -reason = "pack-shaped: real and standard, but St. is Saint in other languages written in Latin script" -issue = [661, 679, 684] +generation = { default = false, expands = "starejši, the Elder", issue = [661, 679, 684, 753], note = "real and standard, but St. is Saint in other languages written in Latin script, and nothing in its writing tells the two apart, so no default reads it; a Slovenian pack turns it on" } [[word]] text = "starejša" diff --git a/tests/v2/test_locales.py b/tests/v2/test_locales.py index 02b784d41..91e8f0d1b 100644 --- a/tests/v2/test_locales.py +++ b/tests/v2/test_locales.py @@ -7,11 +7,15 @@ import sys import types from collections.abc import Callable, Iterable +from pathlib import Path from types import ModuleType import pytest -from nameparser import Locale, Parser, locales, parse, parser_for +from nameparser import ( + Locale, ParsedName, Parser, locales, parse, parser_for, +) +from nameparser import _reading_index as _ri from nameparser._lexicon import _VOCAB_FIELDS, Lexicon from nameparser._policy import ( FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, UNSET, PatronymicRule, Policy, @@ -1144,6 +1148,104 @@ def test_the_choice_gate_fails_when_its_surface_misses_a_word() -> None: _default_corpus()) +# -- 3c: the pack-shaped exclusions as readings off by default -- +# +# A generation word whose collision is with OTHER languages written in +# the same script (cs/hr/sk/sl `ml` and `st`, fr `père`) is a reading +# written `default = false` in its own file, read by no default and +# read once its language is chosen (decisions/vocabulary-model's +# 2026-10-29 #753, phase 3c entry). The sweep is every such +# reading in the index, so a later one joins it unlisted; a one-letter +# word (es `h`, de `i`) is left out, the initial veto +# (`_vocab.is_suffix_strict`) taking it written with its period: +# `Anna Novak h.` reads family `h.` under es, a bare `h` suffix `h`. +_OFF_GENERATIONS = sorted( + (r.text, r.tag) for r in _ri.index() + if r.kind == "generation" and not r.on_by_default and len(r.text) > 1) + + +def test_the_off_generation_sweep_reaches_the_converted_words() -> None: + # an empty parametrize is a skip, not a failure + slavic = {"cs", "hr", "sk", "sl"} + assert ({("ml", t) for t in slavic} | {("st", t) for t in slavic} + | {("père", "fr")}) <= set(_OFF_GENERATIONS) + + +@pytest.mark.parametrize("text, tag", _OFF_GENERATIONS, + ids=lambda v: str(v)) +@pytest.mark.parametrize("spelling", ["{}.", "{}"]) +def test_a_chosen_language_reads_its_off_generation_and_the_default_does_not( + text: str, tag: str, spelling: str) -> None: + word = spelling.format(text) + name = f"Anna Novak {word}" + chosen = parser_for(Locale(code="choice", lexicon=Lexicon.empty(), + languages=frozenset({tag}))) + assert parse(name).suffix == "" + assert (chosen.parse(name).suffix, chosen.parse(name).family) == ( + word, "Novak") + + +def _fields(name: ParsedName) -> tuple[object, ...]: + return (name.given, name.middle, name.family, name.suffix, + tuple(a.kind.name for a in name.ambiguities)) + + +# Today's readings of the comma and initial-pair shapes, pinned (given, +# middle, family, suffix, ambiguity kinds), default first. `ml` has no +# name reading, so under a choice it is an ungated suffix word and its +# comma readings take it with nothing reported (`Novak, ML` suffix +# 'ML', where the default reads given 'ML'), a question recorded for +# 3f in decisions/vocabulary-model's 2026-10-29 #753, phase 3c entry; +# `père`'s family reading gates it and reports the fork. +@pytest.mark.parametrize("tag, name, default, chosen", [ + ("cs", "Novak, ML", ("ML", "", "Novak", "", ()), + ("", "", "Novak", "ML", ())), + ("cs", "Novak, Anna ml.", ("Anna", "ml.", "Novak", "", ()), + ("Anna", "", "Novak", "ml.", ())), + ("cs", "Novak ml., Anna", ("Anna", "", "Novak ml.", "", ()), + ("Anna", "", "Novak", "ml.", ())), + ("cs", "Anna M. L. Novak", ("Anna", "M. L.", "Novak", "", ()), + ("Anna", "M. L.", "Novak", "", ())), + ("cs", "Anna ML Novak", ("Anna", "ML", "Novak", "", ()), + ("Anna", "ML", "Novak", "", ())), + ("fr", "Jean Père", ("Jean", "", "Père", "", ()), + ("Jean", "", "", "Père", ("GIVEN_OR_FAMILY",))), +]) +def test_an_off_generations_readings_in_other_shapes_are_pinned( + tag: str, name: str, default: tuple[object, ...], + chosen: tuple[object, ...]) -> None: + choice = parser_for(Locale(code="choice", lexicon=Lexicon.empty(), + languages=frozenset({tag}))) + assert (_fields(parse(name)), _fields(choice.parse(name))) == ( + default, chosen) + + +@pytest.mark.xfail(strict=True, reason=( + "a one-letter generation turned on stays inert written with its " + "period: the initial veto takes it before the suffix set is read, " + "so the choice changes nothing for it (3c's one-letter gap)")) +@pytest.mark.parametrize("tag, name, suffix", [ + ("es", "Juan Pérez H.", "H."), + ("de", "Wilhelm Karl I.", "I."), +]) +def test_a_chosen_one_letter_generation_reads_with_its_period( + tag: str, name: str, suffix: str) -> None: + choice = parser_for(Locale(code="choice", lexicon=Lexicon.empty(), + languages=frozenset({tag}))) + assert choice.parse(name).suffix == suffix + + +def test_no_exclusion_is_pack_shaped_any_more() -> None: + # after 3c an exclusion means wrong everywhere until a pack's PR + # converts its own (the RFC's Phase 3 table): a reason opening + # "pack-shaped" is a reading a pack turns on, so it is written as + # one. The recompute is decisions/vocabulary-model's grep. + vocab = Path(__file__).resolve().parents[2] / "nameparser" / "vocab" + found = [f.name for f in sorted(vocab.glob("*.toml")) + if "pack-shaped" in f.read_text(encoding="utf-8")] + assert vocab.is_dir() and found == [] + + # -- #269: non-Latin default vocabulary (Cyrillic, Greek, Arabic, # Hebrew, Devanagari, Bengali) -- # diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index f52a6a5f9..efbc2ad68 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -2528,10 +2528,10 @@ def languages(parser: Parser, name: str, text: str) -> tuple[str, ...]: def test_a_choice_that_changes_nothing_leaves_the_lexicon_itself() -> None: - # fr.toml holds no reading the default leaves off or another file's + # sv.toml holds no reading the default leaves off or another file's # reading clashes with, so choosing it changes no compiled set import nameparser._parser as parser_mod from nameparser._reading_index import changed_words, selected - assert changed_words(selected(["fr"])) == frozenset() + assert changed_words(selected(["sv"])) == frozenset() mine = Lexicon.default().add(titles={"zzq"}) - assert parser_mod._apply_choice(mine, frozenset({"fr"})) is mine + assert parser_mod._apply_choice(mine, frozenset({"sv"})) is mine diff --git a/tests/v2/test_vocab.py b/tests/v2/test_vocab.py index b24f26b9e..2179f62db 100644 --- a/tests/v2/test_vocab.py +++ b/tests/v2/test_vocab.py @@ -151,8 +151,8 @@ def test_an_exclusion_in_one_file_refuses_the_reading_in_another( files = dict(shipped) # appended to an ordered file, out of place: the reason still comes # first, check 10 running last - files["en.toml"] += '\n[[word]]\ntext = "père"\ngeneration = {}\n' - with pytest.raises(build.VocabError, match="collides outside French"): + files["fr.toml"] += '\n[[word]]\ntext = "duke"\nnobility = {}\n' + with pytest.raises(build.VocabError, match="Duke Kahanamoku"): _compile(files)