diff --git a/AGENTS.md b/AGENTS.md index 329539b7..bc923cbc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -206,7 +206,7 @@ Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely o ### Vocabulary (`nameparser/vocab/`) and the configuration layer (`nameparser/config/`) -**The default vocabulary is written per word, in TOML, and compiled.** `nameparser/vocab/.toml` holds one file per BCP 47 language tag, each `[[word]]` entry a `text` and its readings keyed by kind. What a file means, how an entry is built, the kinds and properties with the set each compiles into, exclusions, bearer counts and the steps for adding a word are in `nameparser/vocab/README.md`, the reference for anyone editing the TOML; `tests/v2/test_vocab.py` holds its kind, property, derived-marker and reserved-name tables to the names `build.py` accepts and the sets it compiles, so read it rather than restating it here. The parser never reads the TOML: `tools/vocab/build.py` runs the build checks that replaced the modules' import-time asserts (its docstring numbers them), turns every reading into an index row with its file's tag, and writes the rows as the committed reading index `nameparser/config/_index.py` (#753, phase 3a) and, compiled, as the committed `nameparser/config/_compiled.py`, frozenset literals one entry per line; the build never reads `_index.py` back. A file a shipped pack reads may also hold a `[rules]` table, each key a `Policy` field (`patronymic_rules`, `name_order`, `two_family_names`, `two_given_names`), which the build checks against its `RULE_KEYS` and `PACK_RULES` and writes as `nameparser/config/_rules.py`; `_locale.rules_patch(tag)` turns a file's table into the `PolicyPatch` RU and TR_AZ carry, from `ru.toml` and `tr.toml` (#754, 4f; decisions/vocabulary-model's 2026-10-29 4f entry). The compiler is `nameparser/_reading_index.py`'s `compile_readings`, with the kind and property tables it reads (`CATEGORY`, `BEHAVIORAL`, `GATED_WORDS`), so the build and a choice of languages compile alike: `Locale.languages` (3.0, phase 3b) is a choice of BCP 47 tags, which `parser_for` unions across its packs and applies as a delta against the default's compile, `choose` there selecting the readings (a chosen file's `default = false` ones on; on a word a chosen file holds a vocabulary reading of, other files' vocabulary readings off, save `mul.toml`'s and `und.toml`'s; a chosen conjunction's `may_be_initial` dropped), and `changed_words` deriving a choice's deviation surface, which tests/v2/test_locales.py's gate reads (decisions/vocabulary-model, 2026-10-29 3b); the build keeps the checks and loads that module by path, as it loads `_lexicon.py`, and `tests/v2/test_reading_index.py` holds that the index compiled with no choice is `_compiled.py` exactly. `readings_of(text)` there is the run-time reader of the index, every reading of a word with its file's tag, loaded on first call; `languages_of(text, categories)` is the reader a parse calls, from assemble, for `Reading.languages` (3.0), so the first parse that reads a word from its `Lexicon`'s vocabulary loads the index. Each `config/` module re-exports its constants from `_compiled.py` (`TITLES = _compiled.TITLES`) under its autodoc docstring, so the public names and the docs did not move. `tests/v2/test_vocab.py` fails when `_compiled.py` is stale, and `tests/v2/test_reading_index.py` when the index is, each naming the refresh command, and `test_vocab.py` gives every build check a mutation. The model, the phasing and what was declined: decisions/vocabulary-model. +**The default vocabulary is written per word, in TOML, and compiled.** `nameparser/vocab/.toml` holds one file per BCP 47 language tag, each `[[word]]` entry a `text` and its readings keyed by kind. What a file means, how an entry is built, the kinds and properties with the set each compiles into, exclusions, bearer counts and the steps for adding a word are in `nameparser/vocab/README.md`, the reference for anyone editing the TOML; `tests/v2/test_vocab.py` holds its kind, property, derived-marker and reserved-name tables to the names `build.py` accepts and the sets it compiles, so read it rather than restating it here. The parser never reads the TOML: `tools/vocab/build.py` runs the build checks that replaced the modules' import-time asserts (its docstring numbers them), turns every reading into an index row with its file's tag, and writes the rows as the committed reading index `nameparser/config/_index.py` (#753, phase 3a) and, compiled, as the committed `nameparser/config/_compiled.py`, frozenset literals one entry per line; the build never reads `_index.py` back. A file a shipped pack reads may also hold a `[rules]` table, each key a `Policy` field (`patronymic_rules`, `name_order`, `two_family_names`, `two_given_names`), which the build checks against its `RULE_KEYS` and `PACK_RULES` and writes as `nameparser/config/_rules.py`; `_locale.rules_patch(tag)` turns a file's table into the `PolicyPatch` a pack carries, RU's and TR_AZ's from `ru.toml` and `tr.toml`, and the `two_family_names` of ES, CS and SK from `es.toml`, `cs.toml` and `sk.toml` (#754, 4f; decisions/vocabulary-model's 2026-10-29 4f entry). The compiler is `nameparser/_reading_index.py`'s `compile_readings`, with the kind and property tables it reads (`CATEGORY`, `BEHAVIORAL`, `GATED_WORDS`), so the build and a choice of languages compile alike: `Locale.languages` (3.0, phase 3b) is a choice of BCP 47 tags, which `parser_for` unions across its packs and applies as a delta against the default's compile, `choose` there selecting the readings (a chosen file's `default = false` ones on; on a word a chosen file holds a vocabulary reading of, other files' vocabulary readings off, save `mul.toml`'s and `und.toml`'s; a chosen conjunction's `may_be_initial` dropped), and `changed_words` deriving a choice's deviation surface, which tests/v2/test_locales.py's gate reads (decisions/vocabulary-model, 2026-10-29 3b); the build keeps the checks and loads that module by path, as it loads `_lexicon.py`, and `tests/v2/test_reading_index.py` holds that the index compiled with no choice is `_compiled.py` exactly. `readings_of(text)` there is the run-time reader of the index, every reading of a word with its file's tag, loaded on first call; `languages_of(text, categories)` is the reader a parse calls, from assemble, for `Reading.languages` (3.0), so the first parse that reads a word from its `Lexicon`'s vocabulary loads the index. Each `config/` module re-exports its constants from `_compiled.py` (`TITLES = _compiled.TITLES`) under its autodoc docstring, so the public names and the docs did not move. `tests/v2/test_vocab.py` fails when `_compiled.py` is stale, and `tests/v2/test_reading_index.py` when the index is, each naming the refresh command, and `test_vocab.py` gives every build check a mutation. The model, the phasing and what was declined: decisions/vocabulary-model. The `config/` SETS are frozen since 2.2 (#293): there is no `.add()`/`.remove()` on any of them, so a default word list is changed at runtime by configuring a `Lexicon`, never by editing the constant, and the shipped default is changed by editing the TOML and rebuilding. **Neither dict was frozen, and `CAPITALIZATION_EXCEPTIONS` is not covered by anything else either.** `REGEXES` is a compiled-pattern table rather than vocabulary and was never in #293's scope; `CAPITALIZATION_EXCEPTIONS` is vocabulary-shaped and is a decided, in-scope exemption. So the hazard the freeze closes is still live for it: `CAPITALIZATION_EXCEPTIONS['dclinpsy'] = 'DClinPsy'` after the first parse misses the cached `Lexicon.default()`. Same advice — configure the object (`dataclasses.replace(lexicon, capitalization_exceptions=...)`). `tests/v2/test_contracts.py::test_every_vocabulary_constant_is_frozen` names both dicts as explicit exemptions rather than letting its `isinstance` filter drop them. diff --git a/docs/design/decisions/O6-two-family-names.md b/docs/design/decisions/O6-two-family-names.md index 2a15f225..2a335186 100644 --- a/docs/design/decisions/O6-two-family-names.md +++ b/docs/design/decisions/O6-two-family-names.md @@ -4,6 +4,7 @@ - Two family names are an opt-in `Policy` switch, `two_family_names`, off by default: the convention is Spain's and most of Latin America's but not Argentina's, and given-first it is the same string shape as a name with a middle name, so no default can read it — 2026-10-24 #754. - `locales.ES` turns the switch on, carrying the Spanish given-name list beside it, one pack for Spain and Latin America alike, Argentina included; `locales.PT` leaves it off until a Portuguese given-name list exists, decided in chat — 2026-10-30 #753, phase 3g. +- `locales.CS` and `locales.SK` turn the switch on with no given-name list, a net gain in every real population measured (CLDR's fictional renderings lose), two given names the recorded cost; `locales.SL` and `locales.HR` leave it off, decided in chat — 2026-10-30 #753, phase 3f amended. - Both family names stay in `family` through 2.x; a second-surname field is the RFC's Phase 4 output model (docs/design/vocabulary-model-rfc.md, "The rules a pack can choose"), and a pack may later switch this rule on — 2026-10-24 #754; landed as the `surname2` view, `family` keeping both, 2026-10-29 #754, sub-phase 4d. - The family's words are counted as P1's fold counts them, a connective join and a hyphen counting as their words, so a family already holding two keeps what it has; the word taken must count as one by the same count and be no initial written alone — 2026-10-24 #754. - A name its script (W4) reads in an order other than the declared `name_order` is left alone; where the two agree it is read like any other — 2026-10-24 #754. @@ -151,3 +152,14 @@ Open: hyphenated surnames block the switch (`Andrés Sánchez-Cid Torres` keeps | PSC | Cuban | 41 | 36.6% | 75.6% | 78.0% | 24 / 7 | 1 / 0 | Portuguese, the same recipe, the switch alone under a `pt` choice (no list exists): ORCID PT 71.5% → 83.2% (4,296 fixed, 2,034 broken), BR 49.5% → 74.4% (7,004 / 2,268), Angola 75.9% → 57.3%, Mozambique 85.2% → 50.2%; PSC Portuguese 61.1% → 56.1% (1,215 / 1,478), Brazilian 53.4% → 55.6% (606 / 564). With INE's Spanish list standing in for a Portuguese one, an experiment only, PSC Portuguese reads 60.7% and ORCID PT 88.3%, Angola and Mozambique still below the default. So `locales.PT` waits for a list. The candidate is the given-name and surname counts that Receita Federal's open CNPJ register of company partners yields (`CaiqueDourado/nomes-e-sobrenomes-brasil`, CC BY 4.0, 46,840 given names and 27,047 surnames, each word's first-position and later-position counts); IBGE's Census 2022 "Nomes no Brasil" gives the same shape over the whole population but states no licence, and Portugal publishes no frequency list (IRN's list of admitted given names has no counts). Measured 2026-10-30 on `5c3acb56` plus this change, Python 3.11, ORCID's extract of 2026-10-27 capped at 20,000 records a country and PSC's fetch recorded on #849. Recompute: `local_agreement.read` per record of `local_corpora.load(source)` with `FIELDED[source]`, records with both boxes, grouped by `group_of`, under `Parser()`, `Parser(policy=Policy(two_family_names=True))` and `parser_for(locales.ES)`, each with `_title_words` of its own lexicon, every record counted, a record PSC's title set-aside would drop counting as a disagreement, so PSC's populations are the #959 entry's plus its set-aside records (Spanish 6,101 here against 6,089 there); a record is fixed where it agrees only under the pack, broken where it agrees only under the comparator. For `pt`, `parser_for(Locale(code="t", lexicon=Lexicon.empty(), policy=PolicyPatch(two_family_names=True), languages=frozenset({"pt"})))`, and for the experiment its lexicon `Lexicon(given_names=GIVEN_NAMES)`. + +- 2026-10-30 #753, phase 3f amended — `locales.CS` AND `locales.SK` TURN THE SWITCH ON, WITH NO GIVEN-NAME LIST. A Czech or Slovak woman who keeps her surname beside her husband's writes the two without a hyphen (`Alena Pešová Řehořová`), the shape the default reads as a middle name. Decided in chat on the measurement in decisions/vocabulary-model's entry of this date, which holds the recipe. Records agreeing in every field, before → after, the pack at `bfb91163` the before of each row but the second, which is the default's (on ORCID the two read alike): + + | Population | Records | Before → after | Fixed | Broken | + |---|---:|---|---:|---:| + | cz-candidates, the Czech election registers' boxes | 170,181 | 96.97% → 97.47% | 1,295 | 442 | + | cz-candidates, against the default | 170,181 | 96.96% → 97.47% | 1,318 | 440 | + | ORCID, country CZ | 20,000 | 94.99% → 96.41% | 635 | 352 | + | ORCID, country SK | 12,962 | 94.84% → 96.40% | 347 | 144 | + + Every broken record, in all three, has two or more words in the given box, read as a given name and a family name: no population-wide Czech or Slovak given-name list is published to veto the word, the register's frequencies having been withdrawn after 2016's; ČSÚ's top 100 newborn given names a year (CC BY 4.0) is a list, and the open lead. The circular ceiling such a list could reach, and the sources looked for, are in the vocabulary-model entry. diff --git a/docs/design/decisions/deviates-registry.md b/docs/design/decisions/deviates-registry.md index 9eabf393..c73d7ac1 100644 --- a/docs/design/decisions/deviates-registry.md +++ b/docs/design/decisions/deviates-registry.md @@ -2,7 +2,7 @@ ## Standing -- A locale pack declares its deviations from the defaults as a module-level `DEVIATES` callable, acceptance metadata kept off `Locale`, whose value equality and pickling a callable field would break; `test_registry_is_the_pack_contract` fails unless every registered policy or segmentation pack ships `DEVIATES`, and every registered pack a rotator list, which is what lets mechanisms.md#LOCALE-PACKS-PURE-DATA promise packs no code paths of their own. A field on `Locale` is to be revisited only if a runtime consumer of the declaration, such as an `explain()`, appears, and none has — 2026-07-18. A vocabulary pack's deviations are derivable instead, from the reading index (`_reading_index.changed_words`), which tests/v2/test_locales.py's choice gate reads; the gate ors the derived surface into every registered pack with `languages`, so a hand `DEVIATES` declares only what a fragment or a patch beside a choice changes; the contract test refuses a `DEVIATES` on a pack that is nothing but a choice of languages (`locales.NE`, the first, and `CS`, `SK`, `SL` and `HR`), and a choice pack whose choice changes no word, and requires one of every other pack — 2026-10-29 #753, phase 3b; 2026-10-30 #753, phase 3d; 2026-10-30 #753, phase 3f. +- A locale pack declares its deviations from the defaults as a module-level `DEVIATES` callable, acceptance metadata kept off `Locale`, whose value equality and pickling a callable field would break; `test_registry_is_the_pack_contract` fails unless every registered policy or segmentation pack ships `DEVIATES`, and every registered pack a rotator list, which is what lets mechanisms.md#LOCALE-PACKS-PURE-DATA promise packs no code paths of their own. A field on `Locale` is to be revisited only if a runtime consumer of the declaration, such as an `explain()`, appears, and none has — 2026-07-18. A vocabulary pack's deviations are derivable instead, from the reading index (`_reading_index.changed_words`), which tests/v2/test_locales.py's choice gate reads; the gate ors the derived surface into every registered pack with `languages`, so a hand `DEVIATES` declares only what a fragment or a patch beside a choice changes; the contract test refuses a `DEVIATES` on a pack that is nothing but a choice of languages (`locales.NE`, the first, and `SL` and `HR`), and a choice pack whose choice changes no word, and requires one of every other pack, `CS` and `SK` among them since their patch — 2026-10-29 #753, phase 3b; 2026-10-30 #753, phase 3d; 2026-10-30 #753, phase 3f; 2026-10-30 #753, phase 3f amended. Open: none in this file. @@ -12,3 +12,4 @@ Open: none in this file. - 2026-10-29 #753, phase 3b — A VOCABULARY PACK'S DEVIATIONS ARE DERIVED, NOT DECLARED. `Locale.languages` selects among the default's readings, so the words a choice can move are computable: `_reading_index.changed_words(chosen)` is every word joining or leaving a compiled constant under the choice, and tests/v2/test_locales.py's `test_non_interference_each_choice` requires the default's reading for every corpus name holding none of them, with a negative control (`test_the_choice_gate_fails_when_its_surface_misses_a_word`). No pack ships in 3b, so the gate runs over choices of the files' tags; the registry contract test still asks every REGISTERED pack for `DEVIATES` and a rotator list, and learns to read a derived surface for a pack with `languages` in the PR that registers the first one (3d). The policy and segmentation packs (JA, RU, TR_AZ, ZH) keep theirs, decisions/vocabulary-model's 2026-10-27 #753 entry, item 7. - 2026-10-30 #753, phase 3d — THE CONTRACT READS A DERIVED SURFACE. `locales.NE` registers the first pack whose content is a choice of languages and nothing else (`Locale.lexicon` empty, `Locale.policy` the empty patch). tests/v2/test_locales.py's `_deviates(code)` gives the gate, the rotator test and the combined gate every pack's predicate: the surface derived from its `languages` through `changed_words` and the 3b choice gate's `_surface_deviates`, or'd with its `DEVIATES`, so a pack with a fragment or a patch beside a choice needs to declare only that part (`test_a_pack_with_a_choice_and_a_patch_declares_only_the_patch`; nothing refuses a hand copy of the choice's words there); `test_registry_is_the_pack_contract` refuses a `DEVIATES` on a pack that is nothing but its choice, so a hand copy cannot drift beside the derived one, and still requires one of every other pack. Its rotator list stays required, as the gate's positive side. decisions/vocabulary-model's 2026-10-30 phase 3d entry, item 2. - 2026-10-30 #753, phase 3f — A CHOICE PACK MUST CHANGE A WORD. `locales.CS`, `SK`, `SL` and `HR` register as choice packs under phase 3d's contract, with no `DEVIATES`. `test_registry_is_the_pack_contract` adds one refusal: a choice pack whose choice changes no word (an empty `changed_words`), since its gate would pass on nothing; and `test_choice_rotators_cover_every_changed_word` holds each choice pack's rotator list to a name per changed word, so the gate's positive side reaches every word. The combined gate reads the surface of the union of every registered pack's languages, since `parser_for` applies one choice, not one per pack, or'd with every pack's hand `DEVIATES` whatever the pack, since a mixed pack carries both. decisions/vocabulary-model's 2026-10-30 phase 3f entry. +- 2026-10-30 #753, phase 3f amended — A CHOICE PACK THAT GAINS A PATCH DECLARES THE PATCH'S PART, AND THE PACKS SHARING A PATCH SHARE ONE DECLARATION. `locales.CS` and `locales.SK` gain rules/O6's patch (`two_family_names`), so they leave the choice packs and ship `DEVIATES`, as `locales.ES` does, the contract test requiring it of every pack that is not a choice alone. Their declaration is the one `ES` already made for the same patch: three or more name words, or three or more runs of letters. It moved from `locales/es.py` into `_locale.deviates_by_two_family_names`, which all three packs' `DEVIATES` return, so the three cannot drift apart (`test_cs_and_sk_pack_contents` holds them to it). Each pack's title and generation rotators are names the patch alone leaves as the default reads them (`MUDr Novák`, `Novák PhDr.`, `Jana Nováková (roz. Svobodová)`), since `test_a_choice_word_rotator_is_moved_by_the_choice_alone` refuses one the patch moves; the patch has rotators of its own. decisions/vocabulary-model's entry of this date. diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 6e3c91ba..78cb61f5 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -354,6 +354,21 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **Measured** 2026-10-30 on `40675146` plus this change, Python 3.11.17. Committed corpora, `snapshot.py --languages TAG`, entries moved (held only by `corpus_cldr.jsonl` among them): `cs` 17 (6), `sk` 15 (4), `sl` 15 (4), `hr` 15 (4), `pl` 11 (2). Fifteen of each are `st` and `ml` moves, 3c's fourteen and rules/S11's new `[cs]` example `Jiří Dienstbier st.`, which this change adds to the rules corpus; cs and pl add `Pan Tomasz Nowak` and `Pan Nowak Tomasz Jan` (under `FAMILY_FIRST`), both now reading title `Pan`; pl's other 9 are its conjunction `i`, which choosing pl already moved. The CLDR locale view, newly recorded for the four packs in `cldr_agreement_locales.json`: whole renderings agreeing cs 14/24, sk 15/24, sl 16/24, hr 18/24, default and pack alike; the pack moves two sl renderings and no other, each `ml.` from family to suffix (`g. Janez Peter Novak ml., univ. dipl. inž.`, family `Novak`), both still disagreeing elsewhere. Local corpora against the base's default readings (`snapshot.py --local SOURCE --base --languages TAG`), counted by the first changed word each moved name holds: `ctgov-officials` cs 178 (st 69, mudr 44, pan 17, drsc 15 and 2 `Dr. Sc.`, phdr 12, rndr 7, pharmdr 6, ml 3, and one each of paeddr, mddr, mvdr), sk 161 (the same less pan), sl and hr 72 (st 69, ml 3); `orcid-researchers` cs 245 (st 124, pan 46, phdr 16, ml 14, mudr 11, judr 8, drsc 6, mga 5, paeddr 5, rndr 4, roz 3, and one each of thdr, pharmdr, bca), sk 191, sl and hr 138 (st 124, ml 14). Read by kind: the titles and `drsc` are fixes wherever they move a field (`Denisa Wilkes, MUDr` stops reversing, `Christian Salmon, DrSc` keeps its given name, `Prof. MUDr.` reads one title run); every `pan` move is a Chinese name read with a title (`PAN DAI` → title `PAN`), no Czech salutation among them; `roz` fixes one (`Lenka Šenkárová (roz. Veverková)` → maiden `Veverková`) and costs two given names `Roz` (`ELLA ROZ PERRY` → maiden `PERRY`); `st` is 3c's (Saint names, and `PhD St.`, a PhD student, read as the generation). The fielded scores, `local_agreement.read` over the records of the four countries: ORCID's 56,808 records from CZ, SK, SI and HR agree 52,239 under the default and 52,240 under cs and sk (`dr.sc. Sara Soldo`, HR, fixed), unchanged under sl and hr; PSC's 3,578 Czech, Slovak, Slovenian and Croatian filers agree 3,505 under every pack, cs and sk fixing `Mudr Jozef Stec` and moving `Mr Judr Branislav Fridich`, whose filer typed `Judr` in the forename box. The same run over `psc-individuals`: cs 416 (st 395, pan 16, mudr 2, and one each of judr, mvdr and roz, `Miss Naila Roz Roz` → maiden `Roz`), sk 399, sl 395 (all st; hr not run, its choice changing the same two words the same way); `tse-donors` cs 9 (st 5, roz 2, pan 2, `PAN SHISHEN` → title `PAN`); `ukec-individuals` cs 9 (st 8, and `Victor MGA Killearn`, only its forced case repair moving, `MGA` → `MgA`). Recompute: the commands above, and `uv run python -c "from nameparser import _reading_index as ri; print(sorted(ri.changed_words(ri.selected(['cs']))))"` for each tag's words. - **Found, not changed:** a gated generation behind a particle run after a family comma stays in the given part unreported, `Novák, Freiherr von Berg st.` reading given `von Berg st.` under these packs as the default reads `Dumas, Freiherr von Berg fils` (rules/S11); the default reading of the same `st.` is stranger, family `st. Novák`. Pinned in the 3f table. And with one word before a family comma, `Novák, st.` reads title `st.` by its period shape (rules/H2) under all four packs, reporting suffix-or-name with the generation as the alternative though the reading taken is a title, as the default reads `Dumas, fils.`; the ungated `Novák, ml.` reads suffix `ml.`. `Novák st., Jan` keeps family `Novák st.`, as the default keeps `Dumas fils, Alexandre`: the gated word before a family comma is a name word. Both pinned in the 3f table. - **Open:** `pan`, kept because the work order names it: the corpora show its cost (81 Chinese names across ctgov, ORCID, PSC and TSE read with the title) and no gain, the shape that excluded `bc`, and as a title on its own it also takes a lone given name after a family comma (`Smith, Pan` reads title `Pan` with no given name, a bare `Pan` title `Pan`), so whether it stays in the Czech pack is a question for the reviewer. Whether the uncontested titles (`mudr`, `judr`, `phdr`, `rndr` and the rest, no name reading and no clash found) belong in the default rather than a pack is a default change, measured on ctgov, outside phase 3's rule that nothing moves the default. +- **2026-10-30 #753, phase 3f amended — `locales.CS` AND `locales.SK` READ TWO FAMILY NAMES: cs.toml AND sk.toml TURN ON `two_family_names` (rules/O6) IN A `[rules]` TABLE, WITH NO GIVEN-NAME LIST; `SL` AND `HR` STAY CHOICES ALONE; THE DEFAULT DOES NOT MOVE.** Decided in chat on 2026-10-30, on the measurement below. Amends two claims of the phase 3f entry for these two packs: "a pack is then nothing but its tag", and "Each module ships no `DEVIATES`". + - **What changes.** cs.toml and sk.toml each gain `[rules]` `two_family_names = true`, compiled into the pack's policy patch (`PACK_RULES` in tools/vocab/build.py gains `cs` and `sk`). Each pack becomes a choice with a patch beside it, as `locales.ES` is, and declares `DEVIATES` for the patch's part only. That declaration is rules/O6's, `_locale.deviates_by_two_family_names`, now one predicate for the es, cs and sk packs (decisions/deviates-registry's entry of this date). The married double surname written without a hyphen stays whole: `Alena Pešová Řehořová` gives family `Pešová Řehořová`, where the default gives middle `Pešová`. + - **Measured** 2026-10-30 on `bfb91163` plus this change, Python 3.11.17, records agreeing in every field, from these runs, each against a checkout of `bfb91163` (`--base`): `local_agreement.py cz-candidates --base` (the default, matching the base), `cz-candidates --locale cs --base` and `cz-candidates --locale sk --base` (each pack now, its moves against the pack at `bfb91163`), `orcid-researchers --base`, `orcid-researchers --locale cs --base` and `orcid-researchers --locale sk --base` (each report's CZ and SK rows). Three figures no report prints are read with `local_agreement.read` and `FIELDED` over the records under two parsers: the ORCID per-country moves, the cz-candidates moves against the default, and that the cs and sk packs read the ORCID CZ and SK records alike. + - cz-candidates (fetched 2026-10-30; data kv2022 2026-03-28, ps2025 2025-10-05; 170,181 records): default 165,002 (96.96%); `locales.CS` before this change 165,027 (96.97%); now 165,880 (97.47%), 1,295 records now agreeing and 442 now disagreeing against the pack before, 1,318 and 440 against the default (+878); `locales.SK` now 165,878. + - orcid-researchers (data 2025-10-17), country CZ, 20,000 records at the corpus's cap: 18,999 (94.99%) under the default and the pack before alike, now 19,282 (96.41%), +635 and −352. + - orcid-researchers, country SK, 12,962 records: 12,293 (94.84%), now 12,496 (96.40%), +347 and −144. On ORCID the cs and sk packs read alike: they differ on none of the CZ or SK records, measured. + - The default's readings do not move: both corpora's `--base` checks without a locale match the base, and the committed snapshot is unchanged. + - **The cost**, the same in all three populations: every record that now disagrees has a given box of two or more words, read as a given name and a family name (`Petr Pavel Novák` gives family `Pavel Novák`). cz-candidates: 442 of its 473 such records. ORCID CZ: of the 620 such records with a family box, 396 agree before and 220 now. ORCID SK: 166 of 240 before, 45 now. Such a name is about sixteen times as common on ORCID (4.4% of CZ records) as on the registers (0.28%); both sources' given boxes hold the middle names too, so why is not measured. A family comma keeps both readings, rules/O6 standing down there (`Novák, Petr Pavel`). The CLDR locale view, fictional renderings, moves the other way: whole renderings agreeing, cs 14/24 → 12/24 and sk 15/24 → 11/24, its native double surnames gained and its given-given-surname renderings lost, re-recorded in `cldr_agreement_locales.json`. And a birth marker the chosen language does not list, standing between two surnames, is taken into the family: `Jana Nováková rod. Svobodová` reads family `rod. Svobodová` under both packs (the default reads middle `Nováková rod.`), as does `rodená`, and cs's `roz.` under `SK`; under `CS` the listed `roz.` routes the maiden name as before, and the Czech `rozená`, listed on by default, does under both. Slovak `rod.` and `rodená` are birth markers in no file, Open below. + - **Open: a Czech and Slovak given-name list,** which would keep a listed given name out of the family as the Spanish list does (rules/O6's veto). Its ceiling, measured with a circular list (every word of each population's own given boxes): cz-candidates 166,298 (97.72%), ORCID CZ 19,574 (97.87%), ORCID SK 12,624 (97.39%). No population-wide source is published: + - The Ministry of the Interior published given-name and surname frequencies from the population register, the last for 2016, and withdrew them; its page says it is not authorized to publish statistics (archiv.mv.gov.cz/clanek/cetnost-jmen-a-prijmeni.aspx, read 2026-10-30). + - ČSÚ holds no register of names (its press release of 2005-04-21, csu.gov.cz/cesky_statisticky_urad_nema_pristup_k_prijmenim_obyvatel). It publishes the top 100 given names of newborns a year, as XLSX for 2022 to 2025 (csu.gov.cz/nejoblibenejsi_detska_jmena), under CC BY 4.0 by its site terms: a different generation from the adults these packs read, and how many of the two-given-name records it covers is not measured. + - The Spanish INE list is no stand-in: carried beside the pack, it moved 186 cz-candidates records, 134 to agreeing, through the international names it holds (`Jan`, `Marie`, `Anna`). + - **Open: the Slovak birth markers** `rod.` and `rodená`, which no file lists, so the switch reads them into the family (above); a `birth_marker` reading in sk.toml is vocabulary work, measured on its own. + - **Left alone:** `two_given_names` (rules/O7) stays off, not measured on these populations. `SL` and `HR` read one family name, no population of theirs measured. + - **Two packs setting a scalar to the same value no longer warn.** `parser_for` warned whenever two packs set one scalar field, even to the same value; with `cs`, `sk` and `es` all turning `two_family_names` on, the most natural stack for these packs, `parser_for(locales.CS, locales.SK)`, would have warned, failing a caller running with warnings as errors, though nothing was overridden (found in this PR's review). It warns now only where the later value differs, AGENTS.md's "invariants guard harm, not no-ops": a pack agreeing with another changes no parse. tests/v2/test_parser.py pins both sides; docs/locales.rst's composition list says so. - **2026-10-30 #753, phase 3h — THE `ga`, `gd` AND `pt-BR` PACKS: THE IRISH CLAN FORMS AND `sb` TURN ON; `F` AND `N` STAY OUT, MEASURED; THE DEFAULT DOES NOT MOVE.** Carries out 3h of item 11 of the 2026-10-27 #753 entry, and for `nic` and `sb` the rest of item 9. - **What each pack turns on.** `locales.GA` chooses `ga`, `locales.GD` `gd` and `locales.PT_BR` `pt-BR` (code `pt_br`, its choice selecting `pt` too, item 6 of the phase 3b entry), each with an empty lexicon fragment and no policy patch. ga.toml: `nic` (daughter of), its exclusion converted, with a `given` reading (US SSA 77, Wikidata 173) and a `family` one (US Census 108), so it is a possibly-given particle and a leading `Nic` stays given, reported (rules/P1); `mhic` and `uí`, the married forms of `mac` and `ó`, `may_be_given = false`; `mic`, the genitive of `mac`, with `given` (US SSA 26) and `family` readings (ORCID's family box: Romania 4, Ireland none). Each clan reading is `default = false` and `same_as` its head where it has one. gd.toml: `nic` alone. pt-BR.toml: `sb`, its exclusion converted to an `agnomen` reading `default = false`, `same_as = "sobrinho"`, with a declared `family` reading as its siblings have, and `cased = "SB"`, so the S.B. degree's capitals survive `capitalized(force=True)` under the pack, the recasing that was half of its exclusion's reason; the mask is the word's, so Sobrinho's spelling recases to it too (`José Silva Sb.` force-repairs to `SB.`). Irish `nic`, `mhic` and `mic` carry their title case as `cased` too, as `ní` and `uí` do: a particle reading without one is lowercased by case repair (rules/R4), so `MÁIRE NIC GIOLLA PHÁDRAIG` repaired to `Máire nic Giolla Phádraig` under the pack, where the default gives `Nic` (found in review). Counts as of 2026-10-30 from `tools/name_usage.py nic mic mhic uí --boxes`. - **`mhic` stays out of gd.toml.** Written in Scottish Gaelic it mostly links a patronymic chain, `Iain mac Dhòmhnaill mhic Sheumais`, which the default reads family `mac Dhòmhnaill mhic Sheumais` only because `mhic` is unlisted; listed, it starts a second particle run and the chain splits at it, middle `mac Dhòmhnaill`, family `mhic Sheumais`, as the default already splits `Iain mac Dhòmhnaill mac Sheumais`. The Irish married form (`Máire Mhic Gabhann`) is the word's modern written use, so ga.toml keeps it and accepts the chain's split (`Seán mac Seáin mhic Thomáis`, pinned nowhere and moving no corpus name). diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 3e60c3f9..119db5b1 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -301,7 +301,7 @@ A language file will hold a `[rules]` table that turns on rules implemented in t | married-name ending (hu) | `Kovács Jánosné` | given `Kovács`, family `Jánosné` | | digraph initials (cs) | `Petr Chmelař` | initials `P. C.` | -The surname span has shipped ahead of the packs, as an opt-in `Policy` field rather than a pack, since the convention is not one language's ([rules/O6](rules/O6-two-family-names.md), two family names, #875 and #878; decisions/O6 has why it is not `locales.ES`). A pack can later switch it on in its `[rules]`, as it will a default order. +The surname span has shipped ahead of the packs, as an opt-in `Policy` field rather than a pack, since the convention is not one language's ([rules/O6](rules/O6-two-family-names.md), two family names, #875 and #878; decisions/O6 has why it is not `locales.ES`). A pack can later switch it on in its `[rules]`, as it will a default order; `locales.ES` does since phase 3g, and `locales.CS` and `locales.SK` since phase 3f's amendment (decisions/O6). **A language file's `[rules]` compiles into its shipped `Locale`'s `policy` patch**, at build time as the vocabulary does, so a rule a pack turns on is a `Policy` field (`two_family_names`, `patronymic_rules`, and the new rule kinds as new fields) and `Locale` gains no field for rules. RU and TR_AZ become that: their `PolicyPatch` compiled from `ru.toml`'s and `tr.toml`'s `[rules]` instead of written by hand. Built in 4f: the build writes each file's table as tuples in `nameparser/config/_rules.py`, which `_locale.rules_patch` builds into the pack's patch when the pack is imported; each key is the `Policy` field of its name, a file turns a rule on and never supplies what it matches, and `DEVIATES` stays hand-written, held by a test to the rules the patch turns on (decisions/vocabulary-model's 2026-10-29 sub-phase 4f entry; `nameparser/vocab/README.md`'s "Rules" has the keys). The compound given name is measured in the fielded corpora: in ORCID and PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy whose given-names box holds three or more words with a particle inside (the `Maria do Carmo` shape) and whose family box is filled, 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records read with the given box's words pulled into the family (2026-10-28, master `c8357e74`). Like the surname span, it is a rule a caller or a pack turns on, since `Juan de la Vega` is the same string shape with the opposite answer. diff --git a/docs/locales.rst b/docs/locales.rst index 634cb5e6..e737036a 100644 --- a/docs/locales.rst +++ b/docs/locales.rst @@ -127,9 +127,11 @@ How each part combines: declared set-valued in :class:`~nameparser.PolicyPatch`: ``patronymic_rules``, ``segment_scripts`` and the delimiter fields. - **Every other, scalar field is later-wins.** If two packs with - different codes set the same scalar field — even to the same value — - the later one wins and a ``UserWarning`` is raised; a pack overrides - the base parser's value silently. + different codes set the same scalar field to different values, the + later one wins and a ``UserWarning`` is raised; two that set it to the + same value agree, so ``parser_for(locales.CS, locales.SK)``, which + both turn ``two_family_names`` on, is silent. A pack overrides the base + parser's value silently. Choosing languages ~~~~~~~~~~~~~~~~~~ @@ -248,8 +250,8 @@ before the name, run together in their own capitalization (``MUDr.``, words: ``ml.`` is also a pair of initials, ``st.`` is Saint in English and French, and ``Pan`` and ``Roz`` are names in other languages. So they wait for a pack, and each of the -four languages is a pack of its own, which chooses that one language -and nothing else: +four languages is a pack of its own, which chooses that one language; +the Czech and Slovak packs also read two family names (below): .. list-table:: :header-rows: 1 @@ -261,10 +263,12 @@ and nothing else: ``JUDr.``, ``PhDr.``, ``RNDr.``, ``PaedDr.``, ``PharmDr.``, ``ThDr.``, ``Mgr.``, ``MgA.``, ``BcA.``; ``DrSc.`` after the name; ``Pan`` (Mr.); ``roz.``, which routes a maiden - name as ``rozená`` does + name as ``rozená`` does; two family names + (``two_family_names``) * - ``locales.SK`` - ``ml.``, ``st.``; the same titles but ``MgA.`` and ``BcA.``; - ``ArtD.`` and ``DrSc.`` after the name + ``ArtD.`` and ``DrSc.`` after the name; two family names + (``two_family_names``) * - ``locales.SL``, ``locales.HR`` - ``ml.``, ``st.`` @@ -295,6 +299,58 @@ reading is on by default, whatever the parser chose: ``ml.`` above names none, and ``Mgr.`` names ``en`` and ``fr``, whose Monsignor reading the Czech choice dropped for its own. +A Czech or Slovak woman who keeps her own surname beside her +husband's writes the two without a hyphen, ``Alena Pešová Řehořová``, +which the default reads as a middle name and a family name. The two +packs read two family names (``two_family_names``, see +:ref:`two-family-names`), so the double surname stays whole. Two given +names pay for it: with no Czech or Slovak given-name list to keep a +given name out of the family (none is published for the whole +population), ``Petr Pavel Novák`` reads family ``Pavel Novák``. A family +comma keeps both as the default reads them: + +.. doctest:: + + >>> czech.parse("Alena Pešová Řehořová").family + 'Pešová Řehořová' + >>> czech.parse("Petr Pavel Novák").family + 'Pavel Novák' + >>> czech.parse("Novák, Petr Pavel").middle + 'Pavel' + +It gains more names than it costs. Among ORCID's researchers, who split +their own names into a given and a family box, the names agreeing in +both rise from 95.0% to 96.4% in Czechia and from 94.8% to 96.4% in +Slovakia; among the 170,181 records of the 2022 municipal and 2025 +Chamber elections' candidates, each a distinct set of the boxes the +Czech registers file a name in, from 97.0% to 97.5% (measured +2026-10-30). To keep the packs' vocabulary and +turn the switch off, rebuild the policy: + +.. doctest:: + + >>> import dataclasses + >>> one_family = Parser(lexicon=czech.lexicon, policy=dataclasses.replace( + ... czech.policy, two_family_names=False)) + >>> one_family.parse("Petr Pavel Novák").family + 'Novák' + +Stacked with another pack, the switch reads that pack's names too. +With ``locales.RU`` a patronymic written after the given name joins the +family (``Ivan Petrovich Sidorov`` → family ``Petrovich Sidorov``, where +``locales.RU`` alone reads the patronymic ``Petrovich``), as it does +stacked with ``locales.ES``. With ``locales.ES``, the Spanish given-name +list is applied to Czech and Slovak names and keeps the given names it +holds out of the family, many of them names borne in both countries +(``Jan``, ``Marie``, ``Anna``): over the Czech registers it moved 186 +records against ``locales.CS`` alone, 134 of them to agreeing (measured +2026-10-30). + +.. doctest:: + + >>> parser_for(locales.CS, locales.RU).parse("Ivan Petrovich Sidorov").family + 'Petrovich Sidorov' + Choosing the language costs the other readings of these words. ``Pan Wei`` reads title ``Pan``, ``Rosalind Roz Smith`` takes ``Smith`` for a maiden name, and ``st`` loses its Saint readings, so ``St Clair @@ -344,12 +400,17 @@ Shipped packs title ``Adhyapak`` (professor), read as titles where the default reads a name word (``Smt Mamata Banerjee`` → title ``Smt``). See `Romanized Indic titles`_. - * - ``cs``, ``sk``, ``sl``, ``hr`` - - Czech, Slovak, Slovenian and Croatian vocabulary — a choice of - that one language, which turns on the generations ``ml.`` and - ``st.``, and in Czech and Slovak the academic titles written - before the name and the degrees written after it (see - :ref:`west-south-slavic-packs`). + * - ``cs``, ``sk`` + - Czech and Slovak — a choice of that one language, which turns on + the generations ``ml.`` and ``st.``, the academic titles written + before the name and the degrees written after it, and reads two + family names (``Alena Pešová Řehořová`` → family ``Pešová + Řehořová``), so two given names read as a given name and a family + name (see :ref:`west-south-slavic-packs`). + * - ``sl``, ``hr`` + - Slovenian and Croatian vocabulary — a choice of that one + language, which turns on the generations ``ml.`` and ``st.`` + (see :ref:`west-south-slavic-packs`). * - ``es`` - Spanish — chooses ``es``, which turns on the courtesy, academic and professional titles that lead a Spanish name (``Lic.``, @@ -429,14 +490,15 @@ Shipped packs Japanese name read with it can split after its first character (``安藤一郎`` → family ``安``), so use ``ja`` for Japanese data. -``bn``, ``cs``, ``ga``, ``gd``, ``hi``, ``hr``, ``ne``, ``pt``, ``pt_br``, ``sk`` and ``sl`` are each a choice of languages +``bn``, ``ga``, ``gd``, ``hi``, ``hr``, ``ne``, ``pt``, ``pt_br`` and ``sl`` are each a choice of languages and nothing else: the readings each turns on are the default vocabulary's own, written in its language's file (``ne.toml`` for Nepali) and off until a caller chooses the language. ``ja``, ``ru`` and ``tr_az`` are policy-only — they carry no vocabulary of their own. ``es`` is a choice with a list and one policy field beside it: it turns ``two_family_names`` on and carries the given names that switch leaves -where they stand. ``zh`` is both halves at once: a surname list, plus the +where they stand. ``cs`` and ``sk`` are a choice with that one policy +field beside it and no list. ``zh`` is both halves at once: a surname list, plus the one policy field that turns segmentation on for the script it covers. See :ref:`config-containers` for how that split (language vocabulary vs. behavior) is drawn, and `Contributing a pack to nameparser`_ for which half a @@ -758,7 +820,7 @@ by ``tests/v2/test_locales.py``: docstring explains how the pack-contract test enforces this. A pack's choice of languages is never declared: its deviations are derived from the words the choice changes, so a pack that is nothing - but a choice (``ne``, ``cs``, ``ga``) declares no ``DEVIATES`` — the + but a choice (``ne``, ``sl``, ``ga``) declares no ``DEVIATES`` — the contract test refuses one, and refuses a choice that changes no word — and one with a fragment or a patch beside its choice declares only what those change. diff --git a/docs/release_log.rst b/docs/release_log.rst index f599b495..567886a0 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -17,7 +17,7 @@ Release Log - **Add Lexicon.given_names, given names two_family_names leaves out of the family.** ``Parser(policy=Policy(two_family_names=True), lexicon=Lexicon.default().add(given_names={"luis"})).parse("José Luis Pérez")`` gives middle ``Luis``, family ``Pérez``, where the switch alone, in 2.8.0 as now, gives family ``Luis Pérez``. Empty by default, so the switch alone reads as before; an entry without accents matches the word with them. :issue:`959` - **Add Token.reading, the reading the parse took for each word.** ``parse("John Smith PhD Jones").tokens[-1].reading`` is ``Reading('credential' as 'suffix', 'position')``, the reading 2.7's ``Ambiguity.alternative`` gave only for the other side of a fork, and ``ReadingKind`` gains ``patronymic``. The field is not compared, and a ``ParsedName`` pickled by 2.x loads with no readings. :issue:`754` - **Add Reading.languages, the language files that list a word the way it was read.** ``parse("Dr. Juan de la Vega III").tokens[2].reading.languages`` is ``('es', 'fr', 'it', 'nl', 'pt')`` for the particle ``de``, and ``Ambiguity.alternative`` names its files too; a reading by shape or position names none. It is not compared, and a ``Reading`` pickled by 2.7 or 2.8 loads with none. The first parse in a process that meets a listed word loads the reading index once: about 1 ms, or about 16 ms where Python writes no bytecode cache. :issue:`754`, :issue:`753` - - **Add Czech, Slovak, Slovenian and Croatian packs, locales.CS, SK, SL and HR.** ``parser_for(locales.CS).parse("Jan Novák, MUDr")`` gives title ``MUDr``, given ``Jan``, family ``Novák``, where 1.4.0 through 2.8.0 gave given ``MUDr``, family ``Jan Novák``. Each pack chooses its one language and turns on words the default does not read: ``ml.`` and ``st.`` after the name in all four (``Jiří Dienstbier ml.``), the Czech and Slovak academic titles and ``DrSc.``, and Czech ``Pan`` and ``roz.``. Choosing the language drops ``st``'s Saint readings (``St Clair Smith`` reads given ``St``), which choosing ``fr`` as well keeps. :issue:`753` + - **Add Czech, Slovak, Slovenian and Croatian packs, locales.CS, SK, SL and HR.** ``parser_for(locales.CS).parse("Jan Novák, MUDr")`` gives title ``MUDr``, given ``Jan``, family ``Novák``, where 1.4.0 through 2.8.0 gave given ``MUDr``, family ``Jan Novák``. Each pack chooses its one language and turns on words the default does not read: ``ml.`` and ``st.`` after the name in all four (``Jiří Dienstbier ml.``), the Czech and Slovak academic titles and ``DrSc.``, and Czech ``Pan`` and ``roz.``. Choosing the language drops ``st``'s Saint readings (``St Clair Smith`` reads given ``St``), which choosing ``fr`` as well keeps. ``CS`` and ``SK`` also read two family names, so ``Alena Pešová Řehořová`` gives family ``Pešová Řehořová`` and two given names read as a given name and a family name (``Petr Pavel Novák`` gives family ``Pavel Novák``); a family comma keeps both. :issue:`753` - **Add Hindi and Bengali packs, locales.HI and locales.BN, for the romanized Indic titles.** ``parser_for(locales.HI).parse("Smt Nirmala Sitharaman")`` gives title ``Smt``, given ``Nirmala``, where 1.4.0 through 2.8.0 gave given ``Smt``, middle ``Nirmala``. ``HI`` reads ``Smt``, ``Shrimati`` and ``Shriyut`` as titles and ``BN`` ``Smt``, ``Shrimati`` and ``Adhyapak``; ``Shri`` and ``Sushri`` stay name words under both, since each opens given names (``Shri Kant``, ``Sushri Sangita``). See :doc:`locales`. :issue:`345`, :issue:`753` **Breaking Changes** @@ -27,6 +27,7 @@ Release Log **Behavior Changes** - **Render a name in the order it was read in, keeping its written separators.** ``str(parse("김민준"))`` gives ``'김민준'``, where 2.1.0 through 2.8.0 gave ``'민준 김'``; a name read family-first renders family-first, and a comma before a suffix stays (``John Smith, Jr.``, ``Smith, Jr.``), while a family comma between name words never does (``Smith, John`` gives ``'John Smith'``). A spec you pass to ``render()`` renders as written, so 2.x's default spec keeps given-first output, and a name pickled by 2.x renders as it did. ``initials()`` gives nothing for a word in Han, Hangul or kana. The new ``ParsedName.order`` records the order; ``render()``'s spec now defaults to ``None``. See :doc:`migrate`. :issue:`754` + - **Stop warning when two locale packs set a policy field to the same value.** ``parser_for(locales.CS, locales.SK)``, which both turn ``two_family_names`` on, is silent, as is any pair of packs agreeing on a value, however each spells it, where 2.0.0 through 2.8.0 warned that the later pack overrides the field. Packs setting different values still warn, the later winning. :issue:`753` - **Fix Policy(name_order=...) accepting a set.** ``Policy(name_order=frozenset(FAMILY_FIRST))`` raises ``TypeError``, where 2.0.0 through 2.8.0 accepted it as an order under some hash seeds and raised ``ValueError`` under others; pass the tuple. :issue:`754` * 2.8.0 - October 28, 2026 diff --git a/nameparser/_locale.py b/nameparser/_locale.py index aaab6065..0dabccbf 100644 --- a/nameparser/_locale.py +++ b/nameparser/_locale.py @@ -16,6 +16,7 @@ import dataclasses import re +import unicodedata from collections.abc import Callable, Iterable, Mapping from dataclasses import dataclass, field from typing import Any @@ -198,3 +199,39 @@ def rules_patch(tag: str) -> PolicyPatch: # Known mypy limitation with **dict-unpacked construction; see the # full explanation at Lexicon._edit in _lexicon.py. return PolicyPatch(**values) # type: ignore[arg-type] + + +def deviates_by_two_family_names(name: str) -> bool: + """The DEVIATES declaration of a pack whose patch turns on + ``Policy.two_family_names`` (rules/O6), the es, cs and sk packs, and + of the given-name list es carries beside it, which acts only where + O6 does. O6 acts only on a name of three or more name words, so a + name declares that holds three or more words, commas read as spaces, + or three or more units counting each run of letters and each word + holding no letter: the tokenizer may split one word into two runs + (a zero-width space, a Hangul name segmented), and a word of no + letter (``1``, ``-``) is a name word all the same. Titles, suffixes + and a family comma count here though O6 reads none of them, which + over-declares, the safe direction.""" + pieces = name.replace(",", " ").split() + letterless = sum(1 for piece in pieces if not any(ch.isalpha() for ch in piece)) + return len(pieces) >= 3 or _letter_runs(name) + letterless >= 3 + + +def _letter_runs(text: str) -> int: + """The runs of letters in `text`, whatever separates one from the + next: a hyphen, a zero-width space or an interpunct, which the + tokenizer may read as separators (rules/T1, T2, T3). A run of Han, + hangul or kana counts twice, since segmentation may divide it into a + family and a given name (rules/W1). A letter written with a combining + mark splits a run, which over-declares.""" + runs, inside = 0, False + for ch in text: + letter = ch.isalpha() + if letter and not inside: + runs += 2 if unicodedata.name(ch, "").startswith(_CJK) else 1 + inside = letter + return runs + + +_CJK = ("CJK", "HANGUL", "HIRAGANA", "KATAKANA") diff --git a/nameparser/_parser.py b/nameparser/_parser.py index 601e4fb1..41c7a819 100644 --- a/nameparser/_parser.py +++ b/nameparser/_parser.py @@ -363,7 +363,8 @@ def parser_for(*locales: Locale, base: Parser | None = None, pack are wrapped with that pack's identity (rule D2) -- PolicyPatch validates lazily, so with stacked packs the raw error would otherwise point at nothing. Two packs setting the same SCALAR - field is a declared conflict: UserWarning, later wins. + field to different values is a declared conflict: UserWarning, later + wins; two setting it to the same value agree, and nothing is warned. A ``segmenter`` is passed straight through to the built Parser -- ``parser_for(locales.JA, segmenter=locales.ja_segmenter())`` is how @@ -416,20 +417,16 @@ def parser_for(*locales: Locale, base: Parser | None = None, raise type(exc)( f"while applying the languages chosen, " f"{', '.join(sorted(languages))}: {exc}") from exc - scalar_setters: dict[str, str] = {} + # each scalar field's setter so far and the value it left in the + # policy: a later pack leaving another value overrides it, which is + # warned; one leaving the same value (cs and sk's two_family_names) + # agrees. Compared as the applied Policy holds it, canonical, so a + # mapping written in another key order (script_orders) is the value + # it is, not the spelling it was given in. + scalar_setters: dict[str, tuple[str, object]] = {} + scalars = [f.name for f in dataclasses.fields(PolicyPatch) + if f.metadata.get("compose") != "union"] for loc in locales: - for f in dataclasses.fields(PolicyPatch): - if f.metadata.get("compose") == "union": - continue - if getattr(loc.policy, f.name) is UNSET: - continue - if f.name in scalar_setters and scalar_setters[f.name] != loc.code: - warnings.warn( - f"locale {loc.code!r} overrides scalar policy field " - f"{f.name!r} already set by locale " - f"{scalar_setters[f.name]!r}; later wins", - UserWarning, stacklevel=2) - scalar_setters[f.name] = loc.code try: lexicon = lexicon | loc.lexicon policy = apply_patch(policy, loc.policy) @@ -439,6 +436,18 @@ def parser_for(*locales: Locale, base: Parser | None = None, # a subclass with extra mandatory args would break this rewrap raise type(exc)( f"while applying locale {loc.code!r}: {exc}") from exc + for name in scalars: + if getattr(loc.policy, name) is UNSET: + continue + value = getattr(policy, name) + if name in scalar_setters: + code, held = scalar_setters[name] + if code != loc.code and held != value: + warnings.warn( + f"locale {loc.code!r} overrides scalar policy field " + f"{name!r} already set by locale {code!r}; later wins", + UserWarning, stacklevel=2) + scalar_setters[name] = (loc.code, value) # rules/D1: "constructing a parser that activates division for # scripts with no covering surnames and no segmenter warns at # construction, naming the dead scripts and each way out" diff --git a/nameparser/config/_rules.py b/nameparser/config/_rules.py index f49dd400..c005b1b6 100644 --- a/nameparser/config/_rules.py +++ b/nameparser/config/_rules.py @@ -11,7 +11,9 @@ RULES: tuple[tuple[str, tuple[tuple[str, object], ...]], ...] = ( + ('cs', (('two_family_names', True),)), ('es', (('two_family_names', True),)), ('ru', (('patronymic_rules', ('east-slavic',)),)), + ('sk', (('two_family_names', True),)), ('tr', (('patronymic_rules', ('turkic',)),)), ) diff --git a/nameparser/locales/cs.py b/nameparser/locales/cs.py index ed50ca0b..c14cae6c 100644 --- a/nameparser/locales/cs.py +++ b/nameparser/locales/cs.py @@ -1,5 +1,5 @@ -"""The Czech locale pack: a choice of languages, ``cs``, and nothing else --- no lexicon fragment and no policy patch (3.0, phase 3f, #753; +"""The Czech locale pack: a choice of languages, ``cs``, and a policy +patch, no lexicon fragment (3.0, phase 3f, #753; mechanisms.md#LOCALE-PACKS-PURE-DATA). Choosing it turns on nameparser/vocab/cs.toml's readings written ``default = false``: the generations ``ml.`` (mladší) and ``st.`` (starší), the salutation @@ -12,19 +12,42 @@ ``fr`` too, whose only other word the choice changes is ``père``, as decisions/vocabulary-model's 2026-10-30 phase 3f entry records). +It reads two family names (``Policy.two_family_names``, rules/O6), its +policy patch compiled from cs.toml's [rules] table by +tools/vocab/build.py, so the married double surname written without a +hyphen stays whole: ``Alena Pešová Řehořová`` gives family ``Pešová +Řehořová``, where the default gives the first of them as a middle name. +Two given names pay for it, with no given-name list to keep them out of +the family: ``Petr Pavel Novák`` gives family ``Pavel Novák`` +(decisions/vocabulary-model's 2026-10-30 phase 3f amended entry, which +records the cost and the list that could lift it). + Data sources: the readings' own entries in nameparser/vocab/cs.toml, each with its issue and the vetting on its note; nothing is listed here. -Deviations: derived, not declared. The words a choice of ``cs`` -can move are ``_reading_index.changed_words``, which the -non-interference gate in tests/v2/test_locales.py reads -(decisions/deviates-registry), so this module carries no DEVIATES. +Declared deviations (mechanisms.md#LOCALE-PACKS-PURE-DATA): the choice's +are derived, not declared -- the words whose compiled readings it +changes (``_reading_index.changed_words``), which the non-interference +gate in tests/v2/test_locales.py ors in -- so DEVIATES below declares +only what the patch changes. """ from __future__ import annotations from nameparser._lexicon import Lexicon -from nameparser._locale import Locale +from nameparser._locale import Locale, deviates_by_two_family_names, rules_patch + +CS = Locale( + code="cs", + lexicon=Lexicon.empty(), + # nameparser/vocab/cs.toml's [rules], compiled by tools/vocab/build.py + policy=rules_patch("cs"), + languages=frozenset({"cs"}), +) + -CS = Locale(code="cs", lexicon=Lexicon.empty(), - languages=frozenset({"cs"})) +def DEVIATES(name: str) -> bool: + """True when this pack's patch may parse `name` differently from the + default parser: rules/O6's declaration, shared with the es pack + (_locale.deviates_by_two_family_names).""" + return deviates_by_two_family_names(name) diff --git a/nameparser/locales/es.py b/nameparser/locales/es.py index c6c19edd..8088ab7b 100644 --- a/nameparser/locales/es.py +++ b/nameparser/locales/es.py @@ -26,10 +26,8 @@ """ from __future__ import annotations -import unicodedata - from nameparser._lexicon import Lexicon -from nameparser._locale import Locale, rules_patch +from nameparser._locale import Locale, deviates_by_two_family_names, rules_patch from nameparser.locales._es_given_names import GIVEN_NAMES ES = Locale( @@ -43,32 +41,8 @@ def DEVIATES(name: str) -> bool: """True when this pack's patch or fragment may parse `name` - differently from the default parser: rules/O6 acts only on a name - of three or more name words, and the given-name list only where O6 - acts, so a name of three or more words, commas read as spaces, or of - three or more runs of letters declares. Titles, suffixes and a - family comma count here though O6 reads none of them, which - over-declares, the safe direction.""" - pieces = name.replace(",", " ").split() - return len(pieces) >= 3 or _letter_runs(name) >= 3 - - -def _letter_runs(text: str) -> int: - """The runs of letters in `text`, whatever separates one from the - next: a hyphen, a zero-width space or an interpunct, which the - tokenizer may read as separators (rules/T1, T2, T3). A run of Han, - hangul or kana counts twice, since segmentation may divide it into a - family and a given name (rules/W1). A letter written with a combining - mark splits a run, which over-declares. Written with str methods - rather than a pattern, since tests/v2/test_locales.py reads a - module-level pattern as a marker regex to sweep.""" - runs, inside = 0, False - for ch in text: - letter = ch.isalpha() - if letter and not inside: - runs += 2 if unicodedata.name(ch, "").startswith(_CJK) else 1 - inside = letter - return runs - - -_CJK = ("CJK", "HANGUL", "HIRAGANA", "KATAKANA") + differently from the default parser: rules/O6's declaration, which + the cs and sk packs share (_locale.deviates_by_two_family_names); + the given-name list acts only where O6 does, so it declares + nothing more.""" + return deviates_by_two_family_names(name) diff --git a/nameparser/locales/sk.py b/nameparser/locales/sk.py index f929911f..073173a2 100644 --- a/nameparser/locales/sk.py +++ b/nameparser/locales/sk.py @@ -1,5 +1,5 @@ -"""The Slovak locale pack: a choice of languages, ``sk``, and nothing else --- no lexicon fragment and no policy patch (3.0, phase 3f, #753; +"""The Slovak locale pack: a choice of languages, ``sk``, and a policy +patch, no lexicon fragment (3.0, phase 3f, #753; mechanisms.md#LOCALE-PACKS-PURE-DATA). Choosing it turns on nameparser/vocab/sk.toml's readings written ``default = false``: the generations ``ml.`` and ``st.``, the academic titles written before the @@ -11,19 +11,42 @@ ``fr`` too, whose only other word the choice changes is ``père``, as decisions/vocabulary-model's 2026-10-30 phase 3f entry records). +It reads two family names (``Policy.two_family_names``, rules/O6), its +policy patch compiled from sk.toml's [rules] table by +tools/vocab/build.py, so the married double surname written without a +hyphen stays whole: ``Mária Nováková Kováčová`` gives family ``Nováková +Kováčová``, where the default gives the first of them as a middle name. +Two given names pay for it, with no given-name list to keep them out of +the family: ``Ján Peter Mrkva`` gives family ``Peter Mrkva`` +(decisions/vocabulary-model's 2026-10-30 phase 3f amended entry, which +records the cost and the list that could lift it). + Data sources: the readings' own entries in nameparser/vocab/sk.toml, each with its issue and the vetting on its note; nothing is listed here. -Deviations: derived, not declared. The words a choice of ``sk`` -can move are ``_reading_index.changed_words``, which the -non-interference gate in tests/v2/test_locales.py reads -(decisions/deviates-registry), so this module carries no DEVIATES. +Declared deviations (mechanisms.md#LOCALE-PACKS-PURE-DATA): the choice's +are derived, not declared -- the words whose compiled readings it +changes (``_reading_index.changed_words``), which the non-interference +gate in tests/v2/test_locales.py ors in -- so DEVIATES below declares +only what the patch changes. """ from __future__ import annotations from nameparser._lexicon import Lexicon -from nameparser._locale import Locale +from nameparser._locale import Locale, deviates_by_two_family_names, rules_patch + +SK = Locale( + code="sk", + lexicon=Lexicon.empty(), + # nameparser/vocab/sk.toml's [rules], compiled by tools/vocab/build.py + policy=rules_patch("sk"), + languages=frozenset({"sk"}), +) + -SK = Locale(code="sk", lexicon=Lexicon.empty(), - languages=frozenset({"sk"})) +def DEVIATES(name: str) -> bool: + """True when this pack's patch may parse `name` differently from the + default parser: rules/O6's declaration, shared with the es pack + (_locale.deviates_by_two_family_names).""" + return deviates_by_two_family_names(name) diff --git a/nameparser/vocab/cs.toml b/nameparser/vocab/cs.toml index 91021433..bff662cb 100644 --- a/nameparser/vocab/cs.toml +++ b/nameparser/vocab/cs.toml @@ -1,5 +1,11 @@ schema = 1 +# locales.CS reads its names with two family names (rules/O6), the +# married double surname written without a hyphen (decisions/vocabulary-model, +# the 2026-10-30 phase 3f amended entry) +[rules] +two_family_names = true + [[word]] text = "pan" salutation = { default = false, issue = [606, 753], gender = "m", note = "Czech and Polish Mr., off by default because Pan is the Chinese surname 潘, which leads (US Census 2010 12,664 bearers; 285 of ClinicalTrials.gov's 536,981 officials write the word and 488 of ORCID's 1,831,533 records, data 2026-10-21 and 2025-10-17, nearly all of them the surname): under a Czech or Polish choice Pan Wei reads title 'Pan'. The accented paní ships" } diff --git a/nameparser/vocab/sk.toml b/nameparser/vocab/sk.toml index a766b891..b4f2dc42 100644 --- a/nameparser/vocab/sk.toml +++ b/nameparser/vocab/sk.toml @@ -1,5 +1,11 @@ schema = 1 +# locales.SK reads its names with two family names (rules/O6), the +# married double surname written without a hyphen (decisions/vocabulary-model, +# the 2026-10-30 phase 3f amended entry) +[rules] +two_family_names = true + [[word]] text = "judr" academic = { default = false, cased = "JUDr", issue = [753, 848], expands = "doktor práv, Juris utriusque doctor" } diff --git a/tests/v2/test_locales.py b/tests/v2/test_locales.py index cc441e12..8f549841 100644 --- a/tests/v2/test_locales.py +++ b/tests/v2/test_locales.py @@ -231,6 +231,30 @@ def test_es_and_pt_pack_contents() -> None: assert not _es.DEVIATES("Pérez García") +def test_cs_and_sk_pack_contents() -> None: + # 3f, amended 2026-10-30: cs and sk are their choice plus rules/O6's + # patch, no fragment; sl and hr stay nothing but their choice + from nameparser._locale import deviates_by_two_family_names + from nameparser.locales import cs as _cs, es as _es, sk as _sk + for code, pack in (("cs", locales.CS), ("sk", locales.SK)): + assert pack == Locale(code=code, lexicon=Lexicon.empty(), + policy=PolicyPatch(two_family_names=True), + languages=frozenset({code})), code + for code in ("sl", "hr"): + assert _is_vocabulary_pack(locales.get(code)), code + # the patch's declaration is rules/O6's, one predicate for the three + # packs that turn O6 on, so their declarations cannot drift apart + for module in (_cs, _sk, _es): + for name in ("Alena Pešová Řehořová", "Novák, Petr Pavel", + "Mária-\u200bJana Kováčová", "Jan Novák", "Novák"): + assert module.DEVIATES(name) == deviates_by_two_family_names(name) + assert _cs.DEVIATES("Petr Pavel Novák") and not _cs.DEVIATES("Pešová Řehořová") + # a word the tokenizer splits beside a word of no letter is three + # name words (the #973 review: these moved undeclared under all three) + for name in ("김민준 1", "Jan\u200bPavel -"): + assert deviates_by_two_family_names(name), name + + def test_es_and_pt_read_a_leading_title_and_leave_don_a_given_name() -> None: es, pt = _require_packed("es"), _require_packed("pt") for text, title in (("Lic Juan Pérez", "Lic"), ("LIC JUAN PEREZ", "LIC"), @@ -1123,39 +1147,50 @@ def _default_corpus() -> list[str]: # The choice packs (3f): one name per word of the derived surface, each # in a shape the default reads differently, so the gate's positive side # reaches every word the choice changes (held by -# test_choice_rotators_cover_every_changed_word below). +# test_choice_rotators_cover_every_changed_word below). cs and sk carry +# rules/O6's patch too, so each word's rotator is a name the patch alone +# leaves as the default reads it -- two name words, or a marker run in +# brackets -- (held by test_a_choice_word_rotator_is_moved_by_the_choice_alone), +# and the patch has rotators of its own. _ROTATORS["cs"] = [ - "BcA Tomáš Krejčí", # bca + "BcA Krejčí", # bca "Jan Novák, JUDr", # judr, bare after a comma - "MDDr Eva Dvořáková", # mddr - "MgA Petra Malá", # mga - "MUDr Jan Novák", # mudr - "MVDr Petr Svoboda", # mvdr - "PaedDr Jana Králová", # paeddr - "PharmDr Jana Králová", # pharmdr - "Jan Novák PhDr.", # phdr, dotted at the end + "MDDr Dvořáková", # mddr + "MgA Malá", # mga + "MUDr Novák", # mudr + "MVDr Svoboda", # mvdr + "PaedDr Králová", # paeddr + "PharmDr Králová", # pharmdr + "Novák PhDr.", # phdr, dotted at the end "Novák, RNDr", # rndr, after a one-word head - "ThDr Jan Hus", # thdr + "ThDr Hus", # thdr "Jan Novák, DrSc", # drsc, a credential after a comma - "Pan Jan Novák", # pan - "Jana Nováková roz. Svobodová", # roz, the birth marker - "Jiří Dienstbier ml.", # ml + "Pan Novák", # pan + "Jana Nováková (roz. Svobodová)", # roz, the birth marker, bracketed + "Dienstbier ml.", # ml "Jiří Dienstbier st.", # st + "Alena Pešová Řehořová", # rules/O6, the patch + "Ing. Alena Pešová Řehořová", # rules/O6 behind a title + "김민준 1", # rules/O6 over a segmented name and a word of no letter + "Jan\u200bPavel -", # rules/O6 over a zero-width split and a lone mark ] _ROTATORS["sk"] = [ "Jozef Mrkva, ArtD.", # artd, a credential after a comma "Jozef Mrkva, JUDr", # judr - "MDDr Eva Kováčová", # mddr - "MUDr Jozef Mrkva", # mudr - "MVDr Peter Horváth", # mvdr - "PaedDr Jana Kováčová", # paeddr - "PharmDr Jana Kováčová", # pharmdr - "Jozef Mrkva PhDr.", # phdr + "MDDr Kováčová", # mddr + "MUDr Mrkva", # mudr + "MVDr Horváth", # mvdr + "PaedDr Kováčová", # paeddr + "PharmDr Kováčová", # pharmdr + "Mrkva PhDr.", # phdr "Mrkva, RNDr", # rndr - "ThDr Ján Chryzostom", # thdr + "ThDr Chryzostom", # thdr "Jozef Mrkva, DrSc", # drsc - "Ján Mrkva ml.", # ml + "Mrkva ml.", # ml "Ján Mrkva st.", # st + "Mária Nováková Kováčová", # rules/O6, the patch + "김민준 1", # rules/O6 over a segmented name and a word of no letter + "Jan\u200bPavel -", # rules/O6 over a zero-width split and a lone mark ] _ROTATORS["sl"] = [ "Janez Novak ml.", # ml @@ -1280,6 +1315,8 @@ def test_a_choice_word_rotator_is_moved_by_the_choice_alone(code: str) -> None: "Doña María Pérez", # doña "Gabriel García Márquez", # rules/O6, the patch "María José García López", # rules/O6 behind a two-word given part + "김민준 1", # rules/O6 over a segmented name and a word of no letter + "Jan\u200bPavel -", # rules/O6 over a zero-width split and a lone mark ] _ROTATORS["pt"] = [ "Doutor João Silva", # doutor @@ -1437,6 +1474,8 @@ def test_non_interference_all_packs_combined() -> None: all_rotators = [n for code in sorted(_ROTATORS) if code in _PACKED for n in _ROTATORS[code]] corpus = _default_corpus() + all_rotators + # cs, es and sk each set two_family_names to the same value, which + # agrees and warns nothing (test_parser.py pins both sides) if _JA_AVAILABLE: packed = parser_for( *(locales.get(code) for code in sorted(_PACKS)), @@ -1774,10 +1813,40 @@ def test_no_exclusion_is_pack_shaped_any_more() -> None: ("", "", "", "Rosalind", "", "Smith", ())), ("cs", None, "St Clair Smith", ("St", "Clair", "", "Smith", "", "", ()), - ("", "St", "Clair", "Smith", "", "", ())), + ("", "St", "", "Clair Smith", "", "", ())), ("cs+fr", None, "St Clair Smith", ("St", "Clair", "", "Smith", "", "", ()), ("St", "Clair", "", "Smith", "", "", ())), + # the patch, two family names (rules/O6): a married double + # surname written without a hyphen stays whole, behind a title + # too and under a family-first order, and the cost, two given + # names, pays for it; a family comma, where O6 stands down, + # keeps both as the default reads them + # (decisions/vocabulary-model's 2026-10-30 phase 3f amended entry) + ("cs", None, "Alena Pešová Řehořová", + ("", "Alena", "Pešová", "Řehořová", "", "", ()), + ("", "Alena", "", "Pešová Řehořová", "", "", ())), + ("cs", None, "Ing. Alena Pešová Řehořová", + ("Ing.", "Alena", "Pešová", "Řehořová", "", "", ()), + ("Ing.", "Alena", "", "Pešová Řehořová", "", "", ())), + ("cs", "FAMILY_FIRST", "Pešová Řehořová Alena", + ("", "Řehořová", "Alena", "Pešová", "", "", ()), + ("", "Alena", "", "Pešová Řehořová", "", "", ())), + ("sk", None, "Mária Nováková Kováčová", + ("", "Mária", "Nováková", "Kováčová", "", "", ()), + ("", "Mária", "", "Nováková Kováčová", "", "", ())), + ("cs", None, "Petr Pavel Novák", + ("", "Petr", "Pavel", "Novák", "", "", ()), + ("", "Petr", "", "Pavel Novák", "", "", ())), + ("sk", None, "Ján Peter Mrkva", + ("", "Ján", "Peter", "Mrkva", "", "", ()), + ("", "Ján", "", "Peter Mrkva", "", "", ())), + ("cs", None, "Pešová Řehořová, Alena", + ("", "Alena", "", "Pešová Řehořová", "", "", ()), + ("", "Alena", "", "Pešová Řehořová", "", "", ())), + ("cs", None, "Novák, Petr Pavel", + ("", "Petr", "Pavel", "Novák", "", "", ()), + ("", "Petr", "Pavel", "Novák", "", "", ())), # left alone: a bare title ending the given part is a name word # (rules/H5), the title Mgr was one already, and BC, excluded for # the US credential it spells, stays one (cs.toml's bc) diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 95e34031..3896923e 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -7,7 +7,8 @@ import pytest from nameparser import ( - Lexicon, Locale, Parser, Policy, PolicyPatch, locales, parse, parser_for, + Lexicon, Locale, Parser, Policy, PolicyPatch, Script, locales, parse, + parser_for, ) from nameparser._policy import ( CapsSuffixes, FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, GIVEN_FIRST, @@ -133,6 +134,64 @@ def test_parser_for_warns_on_scalar_conflict() -> None: assert p.policy.strip_emoji is True # later wins +# Each scalar PolicyPatch field, with two values of it that differ, and +# for a mapping-shaped field a respelling of the first in another key +# order: the same value written differently. Held to the patch's own +# field list below, so a new scalar field cannot escape the two tests. +_HAN_FIRST = {Script.HAN: FAMILY_FIRST, Script.HANGUL: GIVEN_FIRST} +_SCALARS: dict[str, tuple[object, object, object]] = { + "name_order": (GIVEN_FIRST, FAMILY_FIRST, GIVEN_FIRST), + "script_orders": (_HAN_FIRST, {Script.HAN: GIVEN_FIRST}, + dict(reversed(list(_HAN_FIRST.items())))), + "unlisted_caps_suffixes": (CapsSuffixes.OFF, CapsSuffixes.AFTER_COMMA, + CapsSuffixes.OFF), + **{flag: (False, True, False) for flag in ( + "middle_as_family", "lenient_comma_suffixes", "strip_emoji", + "strip_bidi", "unlisted_dotted_suffixes", "two_family_names", + "two_given_names")}, +} + + +def test_the_scalar_table_is_every_scalar_patch_field() -> None: + assert set(_SCALARS) == {f.name for f in dataclasses.fields(PolicyPatch) + if f.metadata.get("compose") != "union"} + + +def _pack(code: str, field: str, value: object) -> Locale: + return Locale(code=code, lexicon=Lexicon.empty(), + policy=PolicyPatch(**{field: value})) # type: ignore[arg-type] + + +@pytest.mark.parametrize("field", sorted(_SCALARS)) +def test_parser_for_warns_where_two_packs_set_a_scalar_differently(field: str) -> None: + first, other, _ = _SCALARS[field] + with pytest.warns(UserWarning, match=f"'bb' overrides scalar policy field {field!r} " + f"already set by locale 'aa'; later wins"): + stacked = parser_for(_pack("aa", field, first), _pack("bb", field, other)) + # later wins: the field reads as the second pack alone sets it + assert getattr(stacked.policy, field) == getattr( + parser_for(_pack("bb", field, other)).policy, field) + + +@pytest.mark.parametrize("field", sorted(_SCALARS)) +def test_parser_for_is_silent_where_two_packs_set_a_scalar_alike(field: str) -> None: + # two packs agreeing on a value override nothing, whatever spelling of + # it each writes (filterwarnings = error makes any warning a failure); + # the warning is for a value that differs, test above + first, _, respelled = _SCALARS[field] + for second in (first, respelled): + stacked = parser_for(_pack("aa", field, first), _pack("bb", field, second)) + assert stacked.policy == parser_for(_pack("aa", field, first)).policy + + +def test_parser_for_is_silent_over_the_shipped_packs_that_agree() -> None: + # cs, sk and es all turn two_family_names on: their stacks, in either + # order, must not fail a caller running with warnings as errors + for stack in ((locales.CS, locales.SK), (locales.SK, locales.CS), + (locales.ES, locales.CS, locales.SK)): + assert parser_for(*stack).policy.two_family_names is True + + def test_matches_component_wise_case_insensitive() -> None: pn = parse("John Smith") assert pn.matches("JOHN SMITH") diff --git a/tests/v2/test_vocab.py b/tests/v2/test_vocab.py index f6133c21..a9d18e6d 100644 --- a/tests/v2/test_vocab.py +++ b/tests/v2/test_vocab.py @@ -176,7 +176,7 @@ def test_a_reading_off_by_default_compiles_to_nothing() -> None: "mul.toml": _w('[[word]]\ntext = "zzq"\ngeneration = {}'), "de.toml": _w('[[word]]\ntext = "zzq"\ncredential = { default = false }'), "hr.toml": _w('[[word]]\ntext = "zzr"\nexcluded = { generation = "x" }'), - "cs.toml": _w('[[word]]\ntext = "zzr"\ngeneration = { default = false }'), + "pl.toml": _w('[[word]]\ntext = "zzr"\ngeneration = { default = false }'), }) assert "zzq" in both["SUFFIX_WORDS"] and "zzq" not in both["SUFFIX_ACRONYMS"] assert not any("zzr" in v for v in both.values()) @@ -418,8 +418,8 @@ def _w(body: str) -> str: # a title kind's exclusion refuses every other title kind, and the # title category every kind: one title kind made both impossible # before #750 - ("excluded-title-kind-sidestepped-by-a-sibling-kind", {"cs.toml": _w('[[word]]\ntext = "pan"\nexcluded = { salutation = "the Chinese surname Pan" }'), "en.toml": _w('[[word]]\ntext = "pan"\noffice = {}')}, "the Chinese surname Pan"), - ("excluded-title-kind-sidestepped-by-the-generic-kind", {"cs.toml": _w('[[word]]\ntext = "pan"\nexcluded = { salutation = "the Chinese surname Pan" }'), "und.toml": _w('[[word]]\ntext = "pan"\ntitle = {}')}, "the Chinese surname Pan"), + ("excluded-title-kind-sidestepped-by-a-sibling-kind", {"pl.toml": _w('[[word]]\ntext = "pan"\nexcluded = { salutation = "the Chinese surname Pan" }'), "en.toml": _w('[[word]]\ntext = "pan"\noffice = {}')}, "the Chinese surname Pan"), + ("excluded-title-kind-sidestepped-by-the-generic-kind", {"pl.toml": _w('[[word]]\ntext = "pan"\nexcluded = { salutation = "the Chinese surname Pan" }'), "und.toml": _w('[[word]]\ntext = "pan"\ntitle = {}')}, "the Chinese surname Pan"), # a refused property is refused on every kind of its category, with # the recorded reason: keyed on the kind, check 5 would still refuse # this, but with its own message instead of the reason @@ -545,7 +545,7 @@ def test_every_mutation_starts_from_a_pool_the_checks_accept() -> None: "de.toml": _w('[[word]]\ntext = "mister"\noffice = { same_as = "mr" }')}, # a title kind's exclusion refuses the title set, not the word: # its particle reading stays free - {"cs.toml": _w('[[word]]\ntext = "pan"\nexcluded = { salutation = "the Chinese surname Pan" }'), + {"pl.toml": _w('[[word]]\ntext = "pan"\nexcluded = { salutation = "the Chinese surname Pan" }'), "und.toml": _w('[[word]]\ntext = "pan"\nparticle = { may_be_given = false }')}, # one title in a different kind per file, agreeing on addresses {"en.toml": _w('[[word]]\ntext = "sir"\nnobility = { addresses = "given" }'), @@ -1332,5 +1332,5 @@ def test_the_pack_roster_names_shipped_packs_their_files_compile_into() -> None: assert set(build.PACK_RULES.values()) <= set(locales._REGISTRY) for tag, attr in build.PACK_RULES.items(): assert getattr(locales, attr).policy == rules_patch(tag), attr - with pytest.raises(KeyError, match="no vocabulary file 'en' holds a \\[rules\\] table; the files that do: es, ru, tr"): + with pytest.raises(KeyError, match="no vocabulary file 'en' holds a \\[rules\\] table; the files that do: cs, es, ru, sk, tr"): rules_patch("en") diff --git a/tools/differential/cldr_agreement_locales.json b/tools/differential/cldr_agreement_locales.json index 9db09fd3..32f439db 100644 --- a/tools/differential/cldr_agreement_locales.json +++ b/tools/differential/cldr_agreement_locales.json @@ -328,23 +328,19 @@ "cs/foreignFull/surnameFirst": { "family": { "expected": "von Brühl González Domingo", - "got": "von" + "got": "von Brühl" }, "family_base": { "expected": "Brühl González Domingo", - "got": "von" - }, - "family_particles": { - "expected": "von", - "got": "" + "got": "Brühl" }, "given_names": { "expected": "Ada Cornelia César Martín", - "got": "Brühl González Domingo Ada Cornelia César Martín" + "got": "González Domingo Ada Cornelia César Martín" }, "surname2": { "expected": "González Domingo", - "got": "" + "got": "Brühl" }, "text": "Prof. Dr. von Brühl González Domingo Ada Cornelia César Martín, M.D. DDS." }, @@ -370,24 +366,27 @@ }, "text": "Jack" }, - "cs/nativeFull/givenFirst": { + "cs/foreignGGS/givenFirst": { "family": { - "expected": "Machová Ondřejová", - "got": "Ondřejová" - }, - "family_base": { - "expected": "Machová Ondřejová", - "got": "Ondřejová" + "expected": "Stöber", + "got": "Hamish Stöber" }, "given_names": { - "expected": "Alexandra Zuzana", - "got": "Alexandra Zuzana Machová" + "expected": "Zäzilia Hamish", + "got": "Zäzilia" }, - "surname2": { - "expected": "Ondřejová", - "got": "" + "text": "Zäzilia Hamish Stöber" + }, + "cs/foreignGGS/surnameFirst": { + "family": { + "expected": "Stöber", + "got": "Stöber Zäzilia" }, - "text": "paní Alexandra Zuzana Machová Ondřejová, Ph.D." + "given_names": { + "expected": "Zäzilia Hamish", + "got": "Hamish" + }, + "text": "Stöber Zäzilia Hamish" }, "cs/nativeFull/sorting": { "given_names": { @@ -400,25 +399,6 @@ }, "text": "Machová, Alexandra Zuzana (paní, Ph.D.)" }, - "cs/nativeFull/surnameFirst": { - "family": { - "expected": "Machová Ondřejová", - "got": "Machová" - }, - "family_base": { - "expected": "Machová Ondřejová", - "got": "Machová" - }, - "given_names": { - "expected": "Alexandra Zuzana", - "got": "Ondřejová Alexandra Zuzana" - }, - "surname2": { - "expected": "Ondřejová", - "got": "" - }, - "text": "paní Machová Ondřejová Alexandra Zuzana, Ph.D." - }, "cs/nativeG/sorting": { "family": { "expected": "", @@ -441,6 +421,28 @@ }, "text": "Jiří" }, + "cs/nativeGGS/givenFirst": { + "family": { + "expected": "Svoboda", + "got": "Václav Svoboda" + }, + "given_names": { + "expected": "Josef Václav", + "got": "Josef" + }, + "text": "Josef Václav Svoboda" + }, + "cs/nativeGGS/surnameFirst": { + "family": { + "expected": "Svoboda", + "got": "Svoboda Josef" + }, + "given_names": { + "expected": "Josef Václav", + "got": "Václav" + }, + "text": "Svoboda Josef Václav" + }, "es/foreignFull/sorting": { "surname2": { "expected": "Jones", @@ -1633,23 +1635,19 @@ "sk/foreignFull/surnameFirst": { "family": { "expected": "von Kant Nováková Predná", - "got": "von" + "got": "von Kant" }, "family_base": { "expected": "Kant Nováková Predná", - "got": "von" - }, - "family_particles": { - "expected": "von", - "got": "" + "got": "Kant" }, "given_names": { "expected": "Anna Karolína Eva Kristína", - "got": "Kant Nováková Predná Anna Karolína Eva Kristína" + "got": "Nováková Predná Anna Karolína Eva Kristína" }, "surname2": { "expected": "Nováková Predná", - "got": "" + "got": "Kant" }, "text": "Mgr. Ing. von Kant Nováková Predná Anna Karolína Eva Kristína, CSc." }, @@ -1675,10 +1673,40 @@ }, "text": "Jack" }, + "sk/foreignGGS/givenFirst": { + "family": { + "expected": "Stöber", + "got": "Hamish Stöber" + }, + "given_names": { + "expected": "Zäzilia Hamish", + "got": "Zäzilia" + }, + "text": "Zäzilia Hamish Stöber" + }, + "sk/foreignGGS/surnameFirst": { + "family": { + "expected": "Stöber", + "got": "Stöber Zäzilia" + }, + "given_names": { + "expected": "Zäzilia Hamish", + "got": "Hamish" + }, + "text": "Stöber Zäzilia Hamish" + }, "sk/nativeFull/givenFirst": { + "family": { + "expected": "Macáková", + "got": "Zuzana Macáková" + }, + "family_base": { + "expected": "Macáková", + "got": "Zuzana Macáková" + }, "given_names": { "expected": "Alexandra Zuzana", - "got": "pani Alexandra Zuzana" + "got": "pani Alexandra" }, "text": "pani Alexandra Zuzana Macáková PhD", "title": { @@ -1689,15 +1717,11 @@ "sk/nativeFull/surnameFirst": { "family": { "expected": "Macáková", - "got": "pani" + "got": "pani Macáková" }, "family_base": { "expected": "Macáková", - "got": "pani" - }, - "given_names": { - "expected": "Alexandra Zuzana", - "got": "Macáková Alexandra Zuzana" + "got": "pani Macáková" }, "text": "pani Macáková Alexandra Zuzana, PhD", "title": { @@ -1727,6 +1751,28 @@ }, "text": "Jozef" }, + "sk/nativeGGS/givenFirst": { + "family": { + "expected": "Slovák", + "got": "Matej Slovák" + }, + "given_names": { + "expected": "Ján Matej", + "got": "Ján" + }, + "text": "Ján Matej Slovák" + }, + "sk/nativeGGS/surnameFirst": { + "family": { + "expected": "Slovák", + "got": "Slovák Ján" + }, + "given_names": { + "expected": "Ján Matej", + "got": "Matej" + }, + "text": "Slovák Ján Matej" + }, "sl/foreignFull/givenFirst": { "suffix": { "expected": "ml. dr. med., dr. dent. med.", diff --git a/tools/vocab/build.py b/tools/vocab/build.py index 34f6d4a5..12e60ab0 100644 --- a/tools/vocab/build.py +++ b/tools/vocab/build.py @@ -304,7 +304,8 @@ def _alnum(text: str) -> str: #: (check_pack_rules); tests/v2/test_locales.py holds the roster to the #: packs. TR_AZ reads tr.toml's: no az.toml exists, and the pack's rule #: is one rule for both languages. -PACK_RULES: dict[str, str] = {"ru": "RU", "tr": "TR_AZ", "es": "ES"} +PACK_RULES: dict[str, str] = {"ru": "RU", "tr": "TR_AZ", "es": "ES", "cs": "CS", + "sk": "SK"} class VocabError(ValueError):