diff --git a/README.rst b/README.rst index 16b97e2a..e625c5cd 100644 --- a/README.rst +++ b/README.rst @@ -82,7 +82,7 @@ Quick Start Example >>> russian.parse("Сидоров Иван Петрович").family 'Сидоров' >>> locales.available() - ('ja', 'ru', 'tr_az', 'zh') + ('ja', 'ne', 'ru', 'tr_az', 'zh') Learn more ---------- diff --git a/docs/design/decisions/deviates-registry.md b/docs/design/decisions/deviates-registry.md index fa75fbe4..f899e3cd 100644 --- a/docs/design/decisions/deviates-registry.md +++ b/docs/design/decisions/deviates-registry.md @@ -2,7 +2,7 @@ ## Standing -- A locale pack declares its deviations from the defaults as a module-level `DEVIATES` callable, acceptance metadata kept off `Locale`, whose value equality and pickling a callable field would break; `test_registry_is_the_pack_contract` fails unless every registered pack ships `DEVIATES` and a rotator list, which is what lets mechanisms.md#LOCALE-PACKS-PURE-DATA promise packs no code paths of their own. A field on `Locale` is to be revisited only if a runtime consumer of the declaration, such as an `explain()`, appears, and none has — 2026-07-18. A vocabulary pack's deviations are derivable instead, from the reading index (`_reading_index.changed_words`), which tests/v2/test_locales.py's choice gate reads; the contract test still asks every registered pack for `DEVIATES` until the PR registering the first pack whose content is a choice of languages (3d) teaches it to accept the derived surface — 2026-10-29 #753, phase 3b. +- A locale pack declares its deviations from the defaults as a module-level `DEVIATES` callable, acceptance metadata kept off `Locale`, whose value equality and pickling a callable field would break; `test_registry_is_the_pack_contract` fails unless every registered pack ships `DEVIATES` and a rotator list, which is what lets mechanisms.md#LOCALE-PACKS-PURE-DATA promise packs no code paths of their own. A field on `Locale` is to be revisited only if a runtime consumer of the declaration, such as an `explain()`, appears, and none has — 2026-07-18. A vocabulary pack's deviations are derivable instead, from the reading index (`_reading_index.changed_words`), which tests/v2/test_locales.py's choice gate reads; the gate ors the derived surface into every registered pack with `languages`, so a hand `DEVIATES` declares only what a fragment or a patch beside a choice changes; the contract test refuses a `DEVIATES` on a pack that is nothing but a choice of languages (`locales.NE`, the first) and requires one of every other pack — 2026-10-29 #753, phase 3b; 2026-10-30 #753, phase 3d. Open: none in this file. @@ -10,3 +10,4 @@ Open: none in this file. - 2026-07-18 (d4aaafa; the DEVIATES design note) — a `deviates` field ON Locale was rejected: callables break value equality and pickle-by-reference, and DEVIATES is acceptance metadata, not runtime behavior (revisit only if a runtime consumer like explain() appears). The REGISTRY became the contract instead: test_registry_is_the_pack_contract fails structurally unless every registered pack ships DEVIATES and its rotator list. This is why LOCALE-PACKS-PURE-DATA can promise "no code paths of their own" and mean it. - 2026-10-29 #753, phase 3b — A VOCABULARY PACK'S DEVIATIONS ARE DERIVED, NOT DECLARED. `Locale.languages` selects among the default's readings, so the words a choice can move are computable: `_reading_index.changed_words(chosen)` is every word joining or leaving a compiled constant under the choice, and tests/v2/test_locales.py's `test_non_interference_each_choice` requires the default's reading for every corpus name holding none of them, with a negative control (`test_the_choice_gate_fails_when_its_surface_misses_a_word`). No pack ships in 3b, so the gate runs over choices of the files' tags; the registry contract test still asks every REGISTERED pack for `DEVIATES` and a rotator list, and learns to read a derived surface for a pack with `languages` in the PR that registers the first one (3d). The policy and segmentation packs (JA, RU, TR_AZ, ZH) keep theirs, decisions/vocabulary-model's 2026-10-27 #753 entry, item 7. +- 2026-10-30 #753, phase 3d — THE CONTRACT READS A DERIVED SURFACE. `locales.NE` registers the first pack whose content is a choice of languages and nothing else (`Locale.lexicon` empty, `Locale.policy` the empty patch). tests/v2/test_locales.py's `_deviates(code)` gives the gate, the rotator test and the combined gate every pack's predicate: the surface derived from its `languages` through `changed_words` and the 3b choice gate's `_surface_deviates`, or'd with its `DEVIATES`, so a pack with a fragment or a patch beside a choice needs to declare only that part (`test_a_pack_with_a_choice_and_a_patch_declares_only_the_patch`; nothing refuses a hand copy of the choice's words there); `test_registry_is_the_pack_contract` refuses a `DEVIATES` on a pack that is nothing but its choice, so a hand copy cannot drift beside the derived one, and still requires one of every other pack. Its rotator list stays required, as the gate's positive side. decisions/vocabulary-model's 2026-10-30 phase 3d entry, item 2. diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 48732546..09c1233e 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -333,3 +333,12 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 7. **ZH and JA keep their hand-written patches**, reversing in part the same clause of item 10 for ZH's surname list, which stays a Python list in its pack rather than becoming `[rules]` data. Both turn on `segment_scripts`, which the table could express as a set of script names, but neither turn-on stands alone: ZH's segmentation reads its surname list, a Python list outside the per-word model, and JA's reads the caller's segmenter, the one code hook, so a `[rules]` table would split each pack's behavior across a file and a module without making either pure data. ZH's moves when its surname list has a home in the vocabulary; JA's stays as it is, its turn-on serving the segmenter, the one code hook a pack keeps. - **Measured unchanged** 2026-10-29 on `23aacddc` plus this change, Python 3.11: `locales.RU` and `locales.TR_AZ` compare equal to the `Locale`s written by hand before, their rules `PatronymicRule` members (tests/v2/test_locales.py's `test_rules_packs_equal_their_hand_written_patches`); pickles of both, and of `parser_for(RU)`, `parser_for(TR_AZ)` and `parser_for(RU, TR_AZ)`, made under the 2.8.0 wheel and under `23aacddc`, load in this tree equal to its own and parse alike, and under `PYTHONHASHSEED=0` this tree's pickles of all five are byte-identical to `23aacddc`'s; the full suite, the snapshot and the locale agreement records (`cldr_agreement_locales.json`, `wikidata_native_agreement_locales.json`) do not move. Import, `nameparser` already imported in each case (medians of 25 fresh processes, each `python -c "import time, nameparser; t = time.perf_counter(); import nameparser.locales as L; L.RU; print(time.perf_counter() - t)"`, dropping `; L.RU` for the first figure): `import nameparser.locales` 0.23 ms before and 0.24 after; the first access of `locales.RU` or `locales.TR_AZ` 0.50 ms before and 0.69 after, which the review measured as 0.27 → 0.46 ms and 0.31 → 0.45 ms on other runs, about 0.2 ms each time, the import of `config/_rules.py`; packs still load lazily. A cold `import nameparser` is about 47 ms, unchanged. The default parse runs no new code, `rules_patch` being called once per pack at its import. - **Open:** #754, deriving `DEVIATES` from the rules (item 6), with 4g; #753, ZH's turn-on (item 7). +- **2026-10-30 #753, phase 3d — THE FIRST LANGUAGE PACK, `locales.NE`: A CHOICE OF `ne` TURNS ON `given` READINGS OF `श्री`, `guru` AND `गुरु`, SO EACH IS A TITLE ONLY INSIDE A TITLE PHRASE AND ELSEWHERE A NAME WORD PLACED BY POSITION; THE DEFAULT DOES NOT MOVE.** Carries out 3d of item 11 of the 2026-10-27 #753 entry, the route the RFC's "Name readings in a pack" measured: a name reading turned on derives the marker any name reading derives (`TITLES_AMBIGUOUS`, rules/H9), and decision 15 holds inside the pack. + 1. **The readings are in `ne.toml`, each `given = { default = false, note = … }`.** A chosen tag selects its own file and its ancestors' (the phase 3b entry), and `ne` has none, so `languages=("ne",)` reaches only `ne.toml`; a `given` reading in `hi.toml` or `bn.toml` would be turned on by a Hindi or Bengali choice instead, which is 3e's to weigh. The title readings stay where they were (`श्री` in `hi.toml` and `mr.toml`, `guru` in `bn.toml`, `hi.toml` and `pa.toml`, `गुरु` in `hi.toml`), and a name reading never prefers (`choose`'s "Name readings never leave and never settle anything"), so the choice keeps them and the pair derives the marker. `hi.toml` is untouched: its `श्री` note, which says a `given` reading there would make the title one only inside a title phrase, stays true of the default. A count on each reading is in its `note`, there being no count key for a corpus of pairs. + 2. **`locales.NE` is nothing but its choice** (`Locale(code="ne", lexicon=Lexicon.empty(), languages=frozenset({"ne"}))`), registered in `locales/__init__.py`, and it **declares no `DEVIATES`**: mechanisms.md#LOCALE-PACKS-PURE-DATA says a vocabulary pack's deviations are derived, and a pack module may not import `_reading_index` (layering). tests/v2/test_locales.py's `_deviates` ors the surface derived from `changed_words` (`guru`, `गुरु`, `श्री` for NE, pinned by `test_ne_pack_contents`, through the 3b gate's `_surface_deviates`) into the predicate of EVERY pack with `languages`, so a hand `DEVIATES` needs to declare only what a fragment or a patch beside a choice changes, the choice's words no longer needing a hand copy (nothing refuses one that restates them) (`test_a_pack_with_a_choice_and_a_patch_declares_only_the_patch`); the registry contract refuses a `DEVIATES` on a pack that is nothing but its choice and requires one of every other pack. `snapshot.py --languages` and `alike --languages` read the choice alone, which is the whole of NE but not of a pack with a fragment or a patch beside its choice; both say so where they are described. NE's rotators cover each word in each script, at the head, behind a listed title and after a family comma. This is the pattern 3e to 3h copy where a pack is a choice alone. + 3. **`local_corpora.py alike` takes `--languages`**, as `snapshot.py` has since 3b: the flags under the choice, against the default's flags (the tree's own, or `--base`'s), so the RFC's "Measuring a pack" pair measure is one command. A choice is read by `_parse(languages)`, which `surname` and `_read` now ask, and which joins `_ALIKE_CODE`, so the change re-keys every kept `alike` base once. + - **The default does not move.** `build.py` reports `_compiled.py` current, only `_index.py` gaining the three rows; tests/v2/test_snapshot.py passes with no re-record. + - **What `ne` moves,** measured 2026-10-30 on `40675146` plus this change, Python 3.11. Committed corpora (`snapshot.py --languages ne`): 6 entries, 4 of them held only by `corpus_cldr.jsonl`. Those four are CLDR's `hi`, `kok` and `mr` renderings opening with the salutation (`श्री ललित कुमार शर्मा एम॰डी॰`, CLDR title `श्री`), where `श्री` leaves the title for a name word placed by position: given in that one, family in the other three (`श्री शर्मा ललित कुमार, एम॰डी॰`, a family-first rendering, → family `श्री शर्मा ललित कुमार`), one of which, an `mr` rendering whose title was `श्री. एम.`, keeps `एम.`. The choice reads Hindi, Konkani and Marathi names as Nepali, the cost the user docs state. The other two are `corpus_issues.jsonl`'s `Guru Gobind Singh` and `Guru Nanak`, read given `Guru` where the Sikh Gurus' `Guru` is the title; decisions/indic-honorifics' accepted cost (`Guru Dutt` reads title `Guru`) is lifted under the choice and paid the other way. CLDR agreement per pack (`cldr_agreement.py`, `ne` mapped to NE in `LOCALE_FILES` and the `ne-np` stratum in `NATIVE_LABELS`): `ne` 15/24 with and without the pack, Wikidata's `ne-np` 10/10 both ways, neither file holding one of the words; the record gains NE's rows, unchanged. nepal-candidates (`snapshot.py --local nepal-candidates --base --languages ne`): 296 of its names move, all by the word leaving the title for the given name in the default order (a lone word after it going from given to family: `गुरु चौधरी`), 209 opening `श्री`, 42 `Guru`, 41 `गुरु`, 1 `GURU`, 1 `श्री.`, and 2 the glued `सु.श्री`; under either family-first order the word is placed as the family name instead (`श्री राम बहादुर थापा` → family `श्री`, given `राम` under `FAMILY_FIRST`, given `थापा` and middle `राम बहादुर` under `FAMILY_FIRST_GIVEN_LAST`). Pairs (`local_corpora.py alike nepal-candidates --languages ne`): 125,833 of 126,981 read alike under the default, 126,040 under `ne`, 211 now alike and 4 now apart. The 211 all hold `श्री` (4 `गुरु` too), the romanization writing it as a name word (Shree 191, Shri 6, Sri 2); the `Guru` and `गुरु` pairs move together and stay alike. Of the 4 apart, 3 romanize `श्री` as `Mr.` (`Mr. Bal Bahadur Singh`), which the romanization reads as the salutation and the pack does not, and 1 is the defect below. The RFC's 2026-10-27 figures (123,440 → 123,636, 198 fixed, 2 broken) were measured on `cfe8bf92` by roles per word; this measure is `alike`'s structure, on a later tree. + - **Not a rule change.** rules/H9's When already names "a title word that is also borne as a given name", and the choice adds three words to that class without moving its limits: `His Holiness Guru Prasad` keeps title `His Holiness Guru` (a style partner), `Swami Guru Prasad` reads title `Swami`, given `Guru` (H9's Accepted, a religious title behind any title but a style), and `Guru. Prasad Sharma` keeps title `Guru.` by rules/H2's shape, while `श्री. कृष्ण श्रेष्ठ` reads given `श्री.`, H2's shape never reading a Devanagari word (its Accepted clause on abugidas). Pinned in tests/v2/test_locales.py. + - **Open: a period-joined token whose only title chunk the choice makes possibly given reads as a name word, in both directions.** `_vocab.period_joined_vocab` makes a token a title where ANY chunk is in `titles`, and under `ne` `श्री` has moved to `titles_ambiguous`. That costs `सु.श्री.` (सुश्री, Ms., written with its periods), title under the default and given under `ne`, 2 nepal-candidates names; and it fixes a given name glued behind the word, `श्री.राम थापा`, title `श्री.राम` under the default and given `श्री.राम` under `ne`. Whether a possibly-given chunk counts there is open; a fix would move the default for every word of `TITLES_AMBIGUOUS`, and its grid holds both `सु.श्री.` and `श्री.X`. + - **Recompute:** the three commands above from a checkout of this change; the CLDR record with `cldr_agreement.py`; the word counts with `tools/name_usage.py guru गुरु श्री`. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index a47bea2f..47378679 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -164,7 +164,7 @@ Each sub-phase is one PR, measured on its own base: the committed snapshot read ## Phase 3: packs that prefer or restrict -Status: **agreed** (proposed 2026-10-27, #753; Derek accepted every item on 2026-10-28, six and then the other six); **3a landed 2026-10-29**, the reading index, the compiler moved into the package and `default = false` on name readings (decisions/vocabulary-model's 2026-10-29 entry, which records what 3a decided and measured, and the index's lazy load, which Derek chose on 2026-10-29 over loading it with `Lexicon.default()`; "Where the provenance lives at runtime" below describes it). **3b landed 2026-10-29**, the choice: `Locale.languages`, its tag selection, turning on, prefer and the shape clash of `may_be_initial`, the derived deviation surface, `snapshot.py --languages`, and mechanisms.md#LOCALE-PACKS-PURE-DATA amended (decisions/vocabulary-model's 2026-10-29 phase 3b entry, which records what it decided, measured and left open). **3c landed 2026-10-29**, the eight pack-shaped exclusions (cs `ml`, hr, sk and sl `st` and `ml`, fr `père`) written as `generation` readings `default = false`, the default's compile unchanged (decisions/vocabulary-model's 2026-10-29 phase 3c entry, which records what each choice now moves). The record is decisions/vocabulary-model's 2026-10-27 #753 entry; this section describes its items. One design inside the work order is still open: 3g's connective (below), which accepting the work order did not decide. Phase 3 is 3.0 work (re-planned 2026-10-28): 2.8 is the last 2.x feature release, and 3.0 ships phases 3 and 4 with the facade's removal. Packs stay opt-in, so a caller who chooses no language parses as the default does, and nothing in this phase moves the default's parses. One standing text is amended with it, in 3b's PR, which builds the selection: mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack adds, and selects among the default's readings by language, its deviations derived for a vocabulary pack). decisions/3-0-reevaluations' line that removing base vocabulary through a pack is part of 3.0's pack design is carried out rather than moved. +Status: **agreed** (proposed 2026-10-27, #753; Derek accepted every item on 2026-10-28, six and then the other six); **3a landed 2026-10-29**, the reading index, the compiler moved into the package and `default = false` on name readings (decisions/vocabulary-model's 2026-10-29 entry, which records what 3a decided and measured, and the index's lazy load, which Derek chose on 2026-10-29 over loading it with `Lexicon.default()`; "Where the provenance lives at runtime" below describes it). **3b landed 2026-10-29**, the choice: `Locale.languages`, its tag selection, turning on, prefer and the shape clash of `may_be_initial`, the derived deviation surface, `snapshot.py --languages`, and mechanisms.md#LOCALE-PACKS-PURE-DATA amended (decisions/vocabulary-model's 2026-10-29 phase 3b entry, which records what it decided, measured and left open). **3c landed 2026-10-29**, the eight pack-shaped exclusions (cs `ml`, hr, sk and sl `st` and `ml`, fr `père`) written as `generation` readings `default = false`, the default's compile unchanged (decisions/vocabulary-model's 2026-10-29 phase 3c entry, which records what each choice now moves). **3d landed 2026-10-30**, the first language pack, `locales.NE`, a choice of `ne` turning on `given` readings of `श्री`, `guru` and `गुरु` written `default = false` in `ne.toml`, its deviations derived rather than declared, and `local_corpora.py alike --languages` for the pair measure (decisions/vocabulary-model's 2026-10-30 phase 3d entry, which records what it moves and the one defect left open). The record is decisions/vocabulary-model's 2026-10-27 #753 entry; this section describes its items. One design inside the work order is still open: 3g's connective (below), which accepting the work order did not decide. Phase 3 is 3.0 work (re-planned 2026-10-28): 2.8 is the last 2.x feature release, and 3.0 ships phases 3 and 4 with the facade's removal. Packs stay opt-in, so a caller who chooses no language parses as the default does, and nothing in this phase moves the default's parses. One standing text is amended with it, in 3b's PR, which builds the selection: mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack adds, and selects among the default's readings by language, its deviations derived for a vocabulary pack). decisions/3-0-reevaluations' line that removing base vocabulary through a pack is part of 3.0's pack design is carried out rather than moved. **Phase 4's shape questions came first.** The output model decides what a pack can express: a patronymic field, a second surname, credentials apart from generation suffixes (#326), and what `middle` means (#704). A pack reading of a kind with no field to land in is a reading nothing can surface, so those questions were to be settled before or with phase 3's mechanism, by the same shape-before-value rule, and the output model was agreed on 2026-10-28 (decisions/vocabulary-model's 2026-10-28 #754 entry): the seven roles stay, and `generation`, `credentials`, `patronymic` and `surname2` are views read from `Token.reading` or rules/O6's tag. Of the work order below, 3a (the reading index and the shared compiler) and 3c (the pack-shaped exclusions converted) are independent of the output model: the index carries whatever kinds the files hold, and 3c moves data between two encodings that compile alike. 3b's selection semantics are independent too, and its API's dependency, what `Locale` and `Policy` look like once the facade is gone, is answered (below). The packs depend on the output model in part, each on the phase 4 sub-phase that builds its view: `cs`'s generations and `pt-BR`'s agnomens on 4b's `generation` (3f and 3h), `es` and `pt` on 4d's `surname2` and 4g's compound given name (3g); `ne`'s readings do not. diff --git a/docs/locales.rst b/docs/locales.rst index 72525ff1..d853536c 100644 --- a/docs/locales.rst +++ b/docs/locales.rst @@ -243,7 +243,7 @@ look one up dynamically by its lowercase code with .. doctest:: >>> locales.available() - ('ja', 'ru', 'tr_az', 'zh') + ('ja', 'ne', 'ru', 'tr_az', 'zh') >>> locales.get("ru") is locales.RU True @@ -268,6 +268,13 @@ Shipped packs (``山田太郎`` → family ``山田``, given ``太郎``). The pack ships no surname list, because no list divides a kanji name, and no order: a Japanese name already reads family-first by default. + * - ``ne`` + - Nepali given names — chooses the language ``ne``, which reads + ``श्री``, ``गुरु`` and ``Guru`` as name words, placed by position, + where the default reads them as titles (``श्री कृष्ण श्रेष्ठ`` → + given ``श्री`` in the default order, where the default gives + title ``श्री``). Each stays a title inside a title phrase + (``His Holiness Guru Prasad``). * - ``ru`` - East Slavic patronymic order — detects the formal family/given/patronymic shape (three name words, the last ending @@ -295,16 +302,43 @@ Shipped packs (``安藤一郎`` → family ``安``), so use ``ja`` for Japanese data. ``ja``, ``ru`` and ``tr_az`` are policy-only — they carry no vocabulary -of their own. ``zh`` is both halves at once: a surname list, plus the +of their own. ``ne`` is a choice of languages and nothing else: the +readings it turns on are in the default vocabulary's ``ne.toml``, off +until a caller chooses Nepali. ``zh`` is both halves at once: a surname list, plus the one policy field that turns segmentation on for the script it covers. See :ref:`config-containers` for how that split (language vocabulary vs. behavior) is drawn, and `Contributing a pack to nameparser`_ for which half a new naming rule belongs in. +Under ``ne`` a name opening with one of its words reads it as a name +word, placed by position as any other is, a listed title in front of it +included: the given name in the default order, the family name under a +family-first one. + +.. doctest:: + + >>> ne = parser_for(locales.NE) + >>> parse("Guru Prasad Sharma").title + 'Guru' + >>> guru = ne.parse("Guru Prasad Sharma") + >>> guru.title, guru.given, guru.family + ('', 'Guru', 'Sharma') + >>> ne.parse("Mr. Guru Sharma").given + 'Guru' + >>> from nameparser import FAMILY_FIRST, Parser, Policy + >>> family_first = Parser(policy=Policy(name_order=FAMILY_FIRST)) + >>> parser_for(locales.NE, base=family_first).parse("Guru Prasad Sharma").family + 'Guru' + +The choice reads Hindi and Marathi names too, where ``श्री`` is +usually the salutation (Mr.), so use it for Nepali data. + .. warning:: - A pack declares a name *shape*, not a language, and it cannot tell - whose name it is looking at. Any surname that happens to end in a + A pack declares a name *shape*, not a language (save one that + chooses languages, such as ``ne``, which names a language yet acts + on every name holding one of its words, whatever language that name + is in). Either way, it cannot tell whose name it is looking at. Any surname that happens to end in a patronymic suffix matches the East Slavic rule, including names that are not East Slavic at all: @@ -458,7 +492,12 @@ by ``tests/v2/test_locales.py``: scope is a *script* builds the predicate with ``_script_matcher`` from ``nameparser/_policy.py``, the way ``zh`` and ``ja`` do — never by compiling its own character ranges: the factory's - docstring explains how the pack-contract test enforces this. + docstring explains how the pack-contract test enforces this. A + pack's choice of languages is never declared: its deviations are + derived from the words the choice changes, so a pack that is nothing + but a choice (``ne``) declares no ``DEVIATES``, and one with a + fragment or a patch beside its choice declares only what those + change. #. Add a rotator list to ``tests/v2/test_locales.py``. Every pack needs one, but what it has to contain follows from how the pack declares its scope: @@ -472,6 +511,9 @@ by ``tests/v2/test_locales.py``: the same weight by hand — the unspaced names the pack must split, one per shape of the vocabulary it ships: single surname, compound surname, and any spelling variant it means to cover. + - **By a choice of languages** (``ne``): a name for each word the + choice changes, in each script it is written in and each position + where the default reads it otherwise. - **With no vocabulary at all**: list the shapes its *segmenter* must divide instead, and mark the rotator tests to skip when the optional dependency is absent, so the contract tests still run diff --git a/docs/release_log.rst b/docs/release_log.rst index ee9f4fa7..c6ac573f 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -7,6 +7,7 @@ Release Log **Additions** - **Add Locale.languages, a choice of languages for the default vocabulary.** ``parser_for(Locale(code="pt", lexicon=Lexicon.empty(), languages=frozenset({"pt"}))).parse("ANA SILVA E SOUZA")`` gives family ``SILVA E SOUZA``, where the default reads family ``SOUZA``, the ``E`` an initial: a chosen language's reading of a word it has a vocabulary reading of wins, and the other languages' vocabulary readings of that word leave, while its readings as a name and the readings shared by every language stay, so under ``pt`` ``John Smith DO`` reads family ``DO``. A generation word another language collides with is read only under its own language: ``Jiří Dienstbier ml.`` gives suffix ``ml.`` under ``cs``, and the Czech, Slovak, Slovenian and Croatian ``ml.`` and ``st.`` and the French ``père`` read this way. A pack choosing no language parses as before. See :doc:`locales`. :issue:`753` + - **Add locales.NE, a Nepali pack, the first that is a choice of languages.** ``parser_for(locales.NE).parse("Guru Prasad Sharma")`` gives given ``Guru``, middle ``Prasad``, where the default parser gives title ``Guru``, given ``Prasad``; ``श्री`` and ``गुरु`` read the same way, each a name word placed by position (the family name under a family-first order) and still a title inside a title phrase (``His Holiness Guru Prasad``). It reads Hindi and Marathi names too, where ``श्री`` is usually the salutation, so use it for Nepali data. :issue:`753` - **Add ParsedName.generation and ParsedName.credentials, the suffix split by kind.** ``parse("Smith, John, Jr., MD, Esq.")`` gives generation ``Jr.`` and credentials ``MD, Esq.``, where 1.4.0 through 2.8.0 gave only suffix ``Jr., MD, Esq.``, which is unchanged. They are views like ``family_base``: render keys, not fields of ``as_dict()``. An honorific after a name (``Esq.``, ``さん``) is a credential, and a suffix written by ``replace()``, or in a name unpickled from 2.x, is all credentials. ``Token.reading`` reads ``Jr.`` and ``Sr.`` as a generation, and ``Ambiguity.alternative`` of a lone ``Jr`` names a generation where 2.7.0 and 2.8.0 named a post-nominal. :issue:`326`, :issue:`754` - **Add ParsedName.patronymic, the words naming the bearer's parent.** ``parse("Mahathir bin Mohamad")`` gives patronymic ``bin Mohamad`` beside family ``bin Mohamad``, which 1.4.0 through 2.8.0 gave alone and which is unchanged; under ``locales.RU``, ``Ivan Ivanovich Ivanov`` gives patronymic ``Ivanovich`` and keeps it as the middle name, and under ``locales.TR_AZ`` a marker such as ``oglu`` reads with the name before it. A view like ``family_base``, not a field of ``as_dict()``, read only from a patronymic particle (``bin``, ``binti``, ``binte``, ``ibn``, ``bint``, ``bar``, ``bat`` and their Arabic and Hebrew forms) or a rule a locale turns on, so ``Anwar Ibrahim`` has none, and a name written by ``replace()``, or unpickled from 2.x, has none either. ``Ambiguity.alternative`` of a leading ``bin`` read as a given name names a patronymic where 2.7.0 and 2.8.0 named a particle. :issue:`754` - **Add ParsedName.surname2, the second of two family names.** ``Parser(policy=Policy(two_family_names=True)).parse("Gabriel García Márquez")`` gives surname2 ``Márquez`` beside family ``García Márquez``, which 2.8.0 gave alone and which is unchanged. A view like ``family_base``, not a field of ``as_dict()``, filled only where ``two_family_names`` reads the family as two, so the default, a family comma (``Lloyd Webber, Andrew``) and a hyphenated family have none, and so does a name unpickled from 2.x or one whose family ``replace()`` rewrote. ``Reading.second_surname`` marks its words; it is not compared. :issue:`754` diff --git a/nameparser/config/_index.py b/nameparser/config/_index.py index c864eedc..2e4db339 100644 --- a/nameparser/config/_index.py +++ b/nameparser/config/_index.py @@ -816,7 +816,7 @@ ('gräfin', (('de', 'nobility', ()),)), ('gsmieee', (('en', 'credential', ()),)), ('guitarist', (('en', 'occupation', ()),)), - ('guru', (('bn', 'religious', (('addresses', 'given'),)), ('hi', 'religious', (('addresses', 'given'),)), ('pa', 'religious', (('addresses', 'given'),)))), + ('guru', (('bn', 'religious', (('addresses', 'given'),)), ('hi', 'religious', (('addresses', 'given'),)), ('ne', 'given', (('default', False),)), ('pa', 'religious', (('addresses', 'given'),)))), ('gyani', (('pa', 'religious', ()), ('pa', 'given', ()))), ('gysgt', (('en', 'military', ()),)), ('h', (('es', 'generation', (('default', False),)),)), @@ -1674,7 +1674,7 @@ ('عبد', (('ar', 'bound_given', ()),)), ('مهندس', (('ar', 'academic', (('addresses', 'given'),)),)), ('و', (('ar', 'conjunction', ()),)), - ('गुरु', (('hi', 'religious', (('addresses', 'given'),)),)), + ('गुरु', (('hi', 'religious', (('addresses', 'given'),)), ('ne', 'given', (('default', False),)))), ('जनाब', (('hi', 'salutation', (('after_given', True),)),)), ('जी', (('hi', 'honorific', ()), ('mr', 'honorific', ()))), ('जुनियर', (('hi', 'generation', ()),)), @@ -1697,7 +1697,7 @@ ('महाराज', (('hi', 'courtesy', ()),)), ('महाराजा', (('hi', 'nobility', ()), ('mr', 'nobility', ()))), ('मौलाना', (('hi', 'religious', ()),)), - ('श्री', (('hi', 'salutation', ()), ('mr', 'salutation', ()))), + ('श्री', (('hi', 'salutation', ()), ('mr', 'salutation', ()), ('ne', 'given', (('default', False),)))), ('श्रीमती', (('hi', 'salutation', (('after_given', True),)), ('mr', 'salutation', (('after_given', True),)))), ('श्रीमान', (('hi', 'salutation', (('after_given', True),)),)), ('श्रीयुत', (('hi', 'salutation', (('after_given', True),)),)), diff --git a/nameparser/locales/__init__.py b/nameparser/locales/__init__.py index 284b251d..40d20e51 100644 --- a/nameparser/locales/__init__.py +++ b/nameparser/locales/__init__.py @@ -24,6 +24,7 @@ #: lowercase module names; attribute constants are uppercase. _REGISTRY = { "JA": ("nameparser.locales.ja", "JA"), + "NE": ("nameparser.locales.ne", "NE"), "RU": ("nameparser.locales.ru", "RU"), "TR_AZ": ("nameparser.locales.tr_az", "TR_AZ"), "ZH": ("nameparser.locales.zh", "ZH"), diff --git a/nameparser/locales/ne.py b/nameparser/locales/ne.py new file mode 100644 index 00000000..ea15273e --- /dev/null +++ b/nameparser/locales/ne.py @@ -0,0 +1,30 @@ +"""The Nepali locale pack: vocabulary-only -- it chooses the language +``ne``, which turns on the readings nameparser/vocab/ne.toml writes +``default = false``: the given-name readings of ``श्री``, ``guru`` and +``गुरु``, words the default reads as titles (decisions/vocabulary-model, +the 2026-10-30 phase 3d entry). Each is then a title only inside a +title phrase (rules/H9), so it reads as the given name at the head of a +name: ``श्री कृष्ण श्रेष्ठ`` gives given ``श्री`` where the default +gives title ``श्री``. + +Data sources: nepal-candidates, Nepal's 2079 BS local election +candidates, each name written in Devanagari and romanized +(tools/differential/local_nepal.py); the counts are on the readings in +ne.toml. The pack carries no lexicon fragment and no policy patch. + +Declared deviations (mechanisms.md#LOCALE-PACKS-PURE-DATA): a +vocabulary pack's deviations are derived, not declared -- the words +whose compiled readings its choice changes +(``_reading_index.changed_words``), which the non-interference gate in +tests/v2/test_locales.py reads -- so this module defines no DEVIATES. +""" +from __future__ import annotations + +from nameparser._lexicon import Lexicon +from nameparser._locale import Locale + +NE = Locale( + code="ne", + lexicon=Lexicon.empty(), + languages=frozenset({"ne"}), +) diff --git a/nameparser/vocab/ne.toml b/nameparser/vocab/ne.toml index b19ca7fc..1c68b2db 100644 --- a/nameparser/vocab/ne.toml +++ b/nameparser/vocab/ne.toml @@ -9,3 +9,15 @@ text = "जेआर" [word.excluded.generation] reason = "rare, and also JR" issue = 679 + +[[word]] +text = "guru" +given = { default = false, note = "a Nepali given name as well as a title (hi.toml, bn.toml, pa.toml): nepal-candidates holds 45 pairs whose romanization writes it (Guru 44, GURU 1), 44 distinct romanized names, 43 of them opening with it, and under a Nepali choice those 43 (Guru 42, GURU 1) read it as a name word, the default reading it as the title (tools/name_usage.py and snapshot.py --local nepal-candidates --languages ne, 2026-10-30); off by default, where a title is the commoner reading of the word, and turned on by a Nepali choice, which makes the title one only inside a title phrase (rules/H9)" } + +[[word]] +text = "गुरु" +given = { default = false, note = "the Devanagari spelling of guru, a Nepali given name as well as a title (hi.toml): nepal-candidates holds 45 pairs writing it, 41 of them read as the given name under a Nepali choice where the default reads the title (2026-10-30); off by default and turned on by a Nepali choice, as guru is" } + +[[word]] +text = "श्री" +given = { default = false, note = "a Nepali given-name element (Shree, Shri) as well as the salutation (hi.toml, mr.toml): nepal-candidates holds 220 pairs writing it (श्री 219, श्री. 1), 211 of which go from reading apart to reading alike under a Nepali choice, the romanization writing it as a name word (Shree in 191 of them, Shri in 6) (local_corpora.py alike nepal-candidates --languages ne, 2026-10-30); Wikidata 1 given (2026-10-24); off by default, where the salutation is the commoner reading, and turned on by a Nepali choice, which makes the title one only inside a title phrase (rules/H9)" } diff --git a/tests/v2/test_local_corpora.py b/tests/v2/test_local_corpora.py index 6a8c55a0..974198c2 100644 --- a/tests/v2/test_local_corpora.py +++ b/tests/v2/test_local_corpora.py @@ -4353,6 +4353,34 @@ def _alike_kept(cache: Path) -> list[Path]: return sorted((cache / "local" / _NEPAL / local.ALIKE_KEPT).glob("*.json")) +#: A pair the default reads apart and a Nepali choice alike: the +#: romanization writes Shree as a name word, the Devanagari श्री a title +#: until `ne` turns its given reading on (locales.NE, 3.0's phase 3d). +_SHREE = {"name": "Shree Ram Thapa", "devanagari": "श्री राम थापा"} + + +def test_alike_reads_a_pair_under_a_choice_of_languages() -> None: + nepal = local.Local(_NEPAL, {}, [*_NEPAL_RECORDS, _SHREE]) + assert local.alike_flags(nepal)["3"] is False + assert local.alike_flags(nepal, languages=("ne",)) == { + "0": True, "1": True, "2": False, "3": True} + + +def test_alike_under_languages_counts_the_moves_against_the_default( + monkeypatch: pytest.MonkeyPatch, _own_cache: Path, + capsys: pytest.CaptureFixture[str]) -> None: + _nepal(monkeypatch, [*_NEPAL_RECORDS, _SHREE]) + assert local.main(["alike", _NEPAL, "--languages", "ne"]) == 1 + out = capsys.readouterr().out + assert "under the choice ne" in out + assert "3 of 4 pairs read alike (2 in the base)" in out + assert "1 pairs now alike, 0 now apart" in out + with pytest.raises(SystemExit): + local.main(["alike", _NEPAL, "--languages", "ne", "--out", "x.json"]) + with pytest.raises(SystemExit): + local.main(["alike", _NEPAL, "--languages", " , "]) + + def test_alike_against_a_planted_base_reports_moves_and_none_across_a_copy( monkeypatch: pytest.MonkeyPatch, _own_cache: Path, tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None: diff --git a/tests/v2/test_local_jrc.py b/tests/v2/test_local_jrc.py index df5fd92f..5fd18f10 100644 --- a/tests/v2/test_local_jrc.py +++ b/tests/v2/test_local_jrc.py @@ -520,7 +520,7 @@ def test_the_groups_command_does_not_hide_a_parser_error_as_damage( _lenient(monkeypatch) local.fetch(_JRC) - def broken(name: str) -> None: + def broken(name: str, languages: tuple[str, ...] = ()) -> None: raise ValueError("a parser bug") monkeypatch.setattr(local, "_read", broken) with pytest.raises(ValueError, match="a parser bug"): diff --git a/tests/v2/test_locales.py b/tests/v2/test_locales.py index 771647f1..c35162c6 100644 --- a/tests/v2/test_locales.py +++ b/tests/v2/test_locales.py @@ -19,7 +19,7 @@ from nameparser._lexicon import _VOCAB_FIELDS, Lexicon from nameparser._policy import ( FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, UNSET, PatronymicRule, Policy, - Script, _SCRIPT_RANGES, + PolicyPatch, Script, _SCRIPT_RANGES, ) from nameparser._types import AmbiguityKind from nameparser.locales import ja as _ja @@ -103,8 +103,8 @@ def test_locales_module_attribute_access() -> None: def test_locales_get_and_available() -> None: assert locales.get("ru") is locales.RU - assert set(locales.available()) == {"ja", "ru", "tr_az", "zh"} - with pytest.raises(KeyError, match="ja, ru, tr_az, zh"): + assert set(locales.available()) == {"ja", "ne", "ru", "tr_az", "zh"} + with pytest.raises(KeyError, match="ja, ne, ru, tr_az, zh"): locales.get("xx") # the segmenter FACTORY is not a locale: it is an ordinary function # on the package, so it must not appear among the codes @@ -119,6 +119,32 @@ def test_tr_az_pack_contents() -> None: assert locales.TR_AZ.lexicon == Lexicon.empty() +def test_ne_pack_contents() -> None: + # 3d: the first vocabulary pack, nothing but its choice; what the + # choice changes is exactly the three given readings ne.toml writes + # `default = false`, each also a title in another file + from nameparser._reading_index import changed_words, selected + assert locales.NE == Locale(code="ne", lexicon=Lexicon.empty(), + languages=frozenset({"ne"})) + assert changed_words(selected(locales.NE.languages)) == { + "guru", "गुरु", "श्री"} + + +def test_ne_reads_the_word_as_given_at_the_head_and_title_in_a_phrase() -> None: + ne = _require_packed("ne") + assert parse("श्री कृष्ण श्रेष्ठ").title == "श्री" + assert (ne.parse("श्री कृष्ण श्रेष्ठ").given, + ne.parse("श्री कृष्ण श्रेष्ठ").title) == ("श्री", "") + # behind a listed title the word no longer joins the title run + assert ne.parse("Mr. Guru Sharma").as_dict() == parse( + "Mr. Guru Sharma").as_dict() | { + "title": "Mr.", "given": "Guru", "family": "Sharma"} + # the title reading stands: inside a title phrase (rules/H9) and + # written with its period at the head (rules/H2's shape), still a title + assert ne.parse("His Holiness Guru Prasad").title == "His Holiness Guru" + assert ne.parse("Guru. Prasad Sharma").title == "Guru." + + def test_zh_pack_contents() -> None: assert locales.ZH.code == "zh" # the pack activates Han segmentation and ships vocabulary -- @@ -947,6 +973,18 @@ def _default_corpus() -> list[str]: "諸葛亮", # traditional-script compound "萧红", # the 肖/萧 merger's second simplified spelling ] +# ne is a vocabulary pack: its rotators cover each word it turns on, in +# each script it is written in, at the head of the name, behind a +# listed title and after a family comma -- the positions where the +# default reads the word as a title and the pack as the given name. +_ROTATORS["ne"] = [ + "श्री कृष्ण श्रेष्ठ", # श्री at the head + "श्रेष्ठ, श्री कृष्ण", # श्री after a family comma + "गुरु प्रसाद शर्मा", # गुरु at the head + "Guru Prasad Sharma", # guru at the head + "Mr. Guru Sharma", # guru behind a listed title + "Sharma, Guru Prasad", # guru after a family comma +] # ja has no marker regexes and no vocabulary either: its rotators cover # the shapes only the pack PLUS its segmenter changes. Every row is # UNSPACED on purpose -- a spaced kana-licensed name (高橋 みなみ) now @@ -999,6 +1037,16 @@ def test_registry_is_the_pack_contract() -> None: # than a reviewer remembering to extend the gate by hand. assert set(_PACKS) == set(locales.available()) for code, module in _PACKS.items(): + # a choice's surface is derived (_deviates ors it in), so a pack + # that is nothing but its choice declares nothing, while a + # fragment or a patch beside a choice deviates where no derived + # word stands and declares that part, and only that part + pack = locales.get(code) + if _is_vocabulary_pack(pack): + assert not hasattr(module, "DEVIATES"), ( + f"vocabulary pack {code!r} declares DEVIATES by hand; its " + f"surface is derived from its choice") + continue assert callable(getattr(module, "DEVIATES", None)), ( f"pack {code!r} does not declare DEVIATES") # _MARKER_REGEX_PACKS selects on module-level patterns, so a @@ -1062,7 +1110,7 @@ def test_rotator_deviates_and_declares(code: str, name: str) -> None: # per-name proof behind the gate's declared-count floor below packed = _require_packed(code) assert packed.parse(name).as_dict() != _default_parse(name) - assert _PACKS[code].DEVIATES(name) + assert _deviates(code)(name) @pytest.mark.parametrize("code", sorted(_PACKS)) @@ -1070,7 +1118,7 @@ def test_non_interference_each_pack(code: str) -> None: packed = _require_packed(code) corpus = _default_corpus() + _ROTATORS[code] declared = _assert_non_interference( - packed, _PACKS[code].DEVIATES, corpus) + packed, _deviates(code), corpus) # the positive side: every synthetic rotator (plus the corpus's own # in-scope names) must actually flow through the declared branch assert declared >= len(_ROTATORS[code]) @@ -1098,7 +1146,7 @@ def test_non_interference_all_packs_combined() -> None: *(locales.get(code) for code in sorted(_PACKS))) declared = _assert_non_interference( packed, - lambda n: any(m.DEVIATES(n) for m in _PACKS.values()), + lambda n: any(_deviates(code)(n) for code in _PACKS), corpus) assert declared >= len(all_rotators) @@ -1108,8 +1156,9 @@ def test_non_interference_all_packs_combined() -> None: # A vocabulary choice's deviation surface is DERIVED, not declared: # _reading_index.changed_words gives the words whose compiled readings # the choice changes, and a name holding none of them must parse as the -# default does (the RFC's "Measuring a pack"). No language pack ships -# before 3d, so the gate runs over choices of the shipped files' tags +# default does (the RFC's "Measuring a pack"). The shipped vocabulary +# packs (NE, since 3d) run through the registry's gate above, its +# surface derived by _deviates; this gate runs over choices of the shipped files' tags # that move something, each standing for a kind of effect: pt prefer # and the shape clash of `e`, en prefer over several languages, cs and # de turning a reading on (de's `i`, against ca's and pl's conjunction, @@ -1144,6 +1193,52 @@ def deviates(name: str) -> bool: return deviates +def _is_vocabulary_pack(pack: Locale) -> bool: + """Whether `pack` is nothing but its choice of languages, so that its + deviation surface is derived (mechanisms.md#LOCALE-PACKS-PURE-DATA).""" + return (bool(pack.languages) and pack.lexicon == Lexicon.empty() + and pack.policy == PolicyPatch()) + + +@functools.cache +def _deviates(code: str) -> Callable[[str], bool]: + """The registered pack `code`'s deviation predicate: the surface + derived from its choice of languages, where it makes one, or'd with + its DEVIATES, which declares only what the rest of the pack (its + fragment, its patch) changes, so no pack restates the choice's words + by hand.""" + return _pack_deviates(locales.get(code), _PACKS[code]) + + +def _pack_deviates(pack: Locale, module: ModuleType) -> Callable[[str], bool]: + """_deviates for `pack`, whose module is `module`.""" + from nameparser._reading_index import changed_words, selected + parts: list[Callable[[str], bool]] = [] + if pack.languages: + parts.append(_surface_deviates(changed_words(selected(pack.languages)))) + hand = getattr(module, "DEVIATES", None) + if hand is not None: + parts.append(hand) + return lambda name: any(part(name) for part in parts) + + +def test_a_pack_with_a_choice_and_a_patch_declares_only_the_patch() -> None: + # the hand DEVIATES of a pack that also chooses languages covers + # only what its patch changes: the choice's words are or'd in, so + # the hand copy 3b removed is no longer needed + mixed = Locale(code="mixed", lexicon=Lexicon.empty(), + policy=PolicyPatch(patronymic_rules=frozenset( + {PatronymicRule.EAST_SLAVIC})), + languages=frozenset({"ne"})) + module = types.SimpleNamespace(DEVIATES=_ru.DEVIATES) + deviates = _pack_deviates(mixed, module) # type: ignore[arg-type] + assert deviates("Guru Prasad Sharma") # the choice's, derived + assert deviates("Sidorov Ivan Petrovich") # the patch's, declared + assert not deviates("John Smith") + assert not _is_vocabulary_pack(mixed) + assert _is_vocabulary_pack(locales.NE) + + @pytest.mark.parametrize("tags", _CHOICES, ids="+".join) def test_non_interference_each_choice(tags: tuple[str, ...]) -> None: from nameparser._reading_index import changed_words, selected diff --git a/tests/v2/test_wikidata_native_corpus.py b/tests/v2/test_wikidata_native_corpus.py index fbcb656a..719dc86a 100644 --- a/tests/v2/test_wikidata_native_corpus.py +++ b/tests/v2/test_wikidata_native_corpus.py @@ -467,10 +467,11 @@ def test_every_committed_text_is_in_its_strata_script() -> None: row["text"], native.LANGUAGES[_STRATA[row["label"]].language].script)] -def test_every_han_and_kana_stratum_is_scored_under_its_pack() -> None: - # Both ways: a zh or ja stratum missing from the scorer's NATIVE_LABELS - # would never be parsed with its pack, and nothing else would notice. - packs = {"zh": "ZH", "ja": "JA"} +def test_every_stratum_of_a_packs_language_is_scored_under_its_pack() -> None: + # Both ways: a zh, ja or ne stratum missing from the scorer's + # NATIVE_LABELS would never be parsed with its pack, and nothing else + # would notice. ne joined with locales.NE (3.0's phase 3d). + packs = {"zh": "ZH", "ja": "JA", "ne": "NE"} assert agreement.NATIVE_LABELS == {s.label: packs[s.language] for s in native.STRATA if s.language in packs} diff --git a/tools/differential/README.md b/tools/differential/README.md index d2048955..a69f9595 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -333,7 +333,7 @@ uv run python tools/differential/local_corpora.py groups jrc-names On the fetch of 2026-10-24 (data dated 2024-03-15), 93.7% of the 218,822 persons with two or more spellings read alike in every spelling, and the run took 50s (11.8s in workers, 2026-10-27). Within a script the disagreement is mostly the writing, not the parse. Katakana / Katakana reads alike for 0.2% of its 4,289 persons because the JRC writes a transliterated name three ways: unspaced and undivided, which the parser cannot divide and reads as a given name alone (4,347 of the 13,039 Katakana spellings), divided by `・` with no space (4,337), and spaced (4,355), the last two read given then family. Han / Han, 2.2% of 2,420, holds unspaced Han, read as one family word, beside spaced Han, read family first, and Han divided by `·`, read in its written order. Arabic / Arabic reads alike for 67.8% of 3,329 persons, and the Arabic comma is most of why: each of the 736 persons with a spelling holding one differs, of the 1,072 that differ, since the comma moves the family to the front. Across scripts read in the same order most persons read alike (Cyrillic / Latin 99.3% of 20,837, Arabic / Latin 98.2% of 5,347, Greek / Latin 98.6% of 915); Hangul against a script read given first reads alike for 3% or fewer in every such row `--top 40` lists (Hangul / Latin 2.0% of 1,247), its family standing first, while Han / Hangul, both read family first, is 87.1% of 1,002. The title words read in some spellings only start `sheikh` 50, `bard` 33, `shaikh` 27, `sir` 23, `executive` 22 and include `mahdi` 17, a common given name; the suffix words `jr` 62, `sa` 38, `iii` 27, `ii` 27. -`alike` reads the spelling views as one flag per person, for a source whose records pair spellings (`ALIKE_SOURCES`: `jrc-names`' persons, `nepal-candidates`' romanized and Devanagari pairs, `tse-donors`' filed and registry pairs): whether the person's spellings read alike, as `groups` compares them for a person, the two readings' structure for a Nepali pair, and one shared family last word, as `pairs --agreement` counts it, for a filed pair that differs. With `--base` it prints how many persons (`jrc-names`) or pairs now read alike and how many now apart, the counts phase 2j quoted for each batch. It counts flips only: a person whose spellings all read differently and still alike is no move here, though the local snapshot shows it. The base's flags are kept under `alike/` beside the corpus, keyed like the baselines (the fetch, the base's `nameparser/` files, the interpreter's minor version, and the view's code, `alike_fingerprint`), read in a child process where none is kept, and published under their key only once whole and once the base's digest, taken again, is unchanged. A key holds no name: a person is its entity or its record's place in `raw.jsonl`. +`alike` reads the spelling views as one flag per person, for a source whose records pair spellings (`ALIKE_SOURCES`: `jrc-names`' persons, `nepal-candidates`' romanized and Devanagari pairs, `tse-donors`' filed and registry pairs): whether the person's spellings read alike, as `groups` compares them for a person, the two readings' structure for a Nepali pair, and one shared family last word, as `pairs --agreement` counts it, for a filed pair that differs. With `--base` it prints how many persons (`jrc-names`) or pairs now read alike and how many now apart, the counts phase 2j quoted for each batch. It counts flips only: a person whose spellings all read differently and still alike is no move here, though the local snapshot shows it. The base's flags are kept under `alike/` beside the corpus, keyed like the baselines (the fetch, the base's `nameparser/` files, the interpreter's minor version, and the view's code, `alike_fingerprint`), read in a child process where none is kept, and published under their key only once whole and once the base's digest, taken again, is unchanged. A key holds no name: a person is its entity or its record's place in `raw.jsonl`. With `--languages ne` it reads the tree under that choice of languages (`Locale.languages`) and counts the persons or pairs the choice moves against the default's flags, the tree's own or `--base`'s, as `snapshot.py --languages` does for the names: the pair measure a language pack reads. Both read the choice alone, never a registered pack's lexicon fragment or policy patch, which is the whole of a pack only where the pack is nothing but its choice, as `locales.NE` is, the RFC's "Measuring a pack" (`locales.NE`, 2026-10-30: 211 nepal-candidates pairs now alike, 4 now apart). ``` uv run python tools/differential/local_corpora.py alike jrc-names --base ../master-checkout @@ -550,7 +550,7 @@ data can. and say why in the commit. Run it without `--record` for per-role agreement counts. Nothing is special-cased: an unnatural rendering or a modelling difference (CLDR's `さん` title is a suffix here) simply stays a recorded disagreement, which makes the record a measured worklist as well as a guard. -A second view measures the locale packs (#672). `LOCALE_FILES` in `cldr_agreement.py` maps each CLDR file to the pack for its language (`zh`, `zh_Hant`, `yue`, `yue_Hans` → `ZH`; `ja` → `JA`; `ru` → `RU`; `tr`, `az` → `TR_AZ`); those renderings are parsed with `parser_for(pack)`, over the same `FAMILY_FIRST` base where CLDR's order is surnameFirst, and their disagreements go to `cldr_agreement_locales.json`, ratcheted per pack by the same test. The native-script Wikidata source's Han and kana strata are mapped by `NATIVE_LABELS` (`zh-cn`, `zh-tw` → `ZH`; `ja-jp` → `JA`), since their file is a QID that names no language, and the view parses them the same way into `wikidata_native_agreement_locales.json`. Each source's pack is read from its own rendering key (`locale_sources()`: CLDR's file, the native source's stratum), so a recorded rendering that is gone is still checked by its pack. The summary prints, per pack, how many of its renderings agree in every role under the default parser and under the pack, so a pack PR shows what it moved on its own language. A new pack adds its files to `LOCALE_FILES`, or its name to `NO_CLDR`; the test fails on a pack in neither. The modelling differences stay recorded here too: a pack is not meant to agree with CLDR where the models differ by design. +A second view measures the locale packs (#672). `LOCALE_FILES` in `cldr_agreement.py` maps each CLDR file to the pack for its language (`zh`, `zh_Hant`, `yue`, `yue_Hans` → `ZH`; `ja` → `JA`; `ne` → `NE`; `ru` → `RU`; `tr`, `az` → `TR_AZ`); those renderings are parsed with `parser_for(pack)`, over the same `FAMILY_FIRST` base where CLDR's order is surnameFirst, and their disagreements go to `cldr_agreement_locales.json`, ratcheted per pack by the same test. The native-script Wikidata source's strata of a pack's language are mapped by `NATIVE_LABELS` (`zh-cn`, `zh-tw` → `ZH`; `ja-jp` → `JA`; `ne-np` → `NE`), since their file is a QID that names no language, and the view parses them the same way into `wikidata_native_agreement_locales.json`. Each source's pack is read from its own rendering key (`locale_sources()`: CLDR's file, the native source's stratum), so a recorded rendering that is gone is still checked by its pack. The summary prints, per pack, how many of its renderings agree in every role under the default parser and under the pack, so a pack PR shows what it moved on its own language. A new pack adds its files to `LOCALE_FILES`, or its name to `NO_CLDR`; the test fails on a pack in neither. The modelling differences stay recorded here too: a pack is not meant to agree with CLDR where the models differ by design. **The GND source.** `build_gnd_corpus.py` writes the GND's name parts in CLDR's field vocabulary (`forename` → `given`, `prefix` → `surname-prefix` with a one-word `surname` as `surname-core`, any other family → `surname` whole, `counting` → `generation`, and a `personalName` -- a person entered under the whole name, with no surname -- as `given` alone, so the family is expected empty), so `cldr_agreement.py` compares `gnd_fields.jsonl` by the same rules and records its disagreements in `gnd_agreement.json`, ratcheted by the same test. The locale view does not read it: no pack covers its names. A composed rendering says so (`composed: true`); where a particle is glued to the surname (`d'Arenberg`, `al-Ǧaʿfarī`) the composed line states the glued word as a whole `surname`, since the comparison matches fields word by word, and the index form, where the particle is a word of its own, keeps the split. A family with no `prefix`, or with a surname of several words, is stated whole for the same kind of reason: GND does not split a particle inside a surname (`Vélez de Guevara`, `Zuylen van Nijevelt`), so what it leaves unsplit means "not stated", not "no particle", and only `family` is compared there. Raw lobid responses are cached outside the tree, in a `gnd/` directory under `$NAMEPARSER_TOOLS_CACHE` (default `~/.cache/nameparser-tools`); `--refresh` fetches them again. The build stops before writing anything when a stratum falls below its floor, a query matches nothing, or a pool pages short, so a lobid change fails it rather than shrinking the corpus; `--record` prints, per source, how many changes of each kind it accepts. diff --git a/tools/differential/cldr_agreement.py b/tools/differential/cldr_agreement.py index 6d18d0d7..28348f69 100644 --- a/tools/differential/cldr_agreement.py +++ b/tools/differential/cldr_agreement.py @@ -37,7 +37,7 @@ A second view does the same with each locale pack (#672): the renderings of every CLDR file in LOCALE_FILES, and of every native-script Wikidata -stratum in NATIVE_LABELS (its Han and kana names), are parsed +stratum in NATIVE_LABELS (its Han, kana and Nepali names), are parsed with parser_for(pack), and their disagreements are recorded in cldr_agreement_locales.json and wikidata_native_agreement_locales.json and ratcheted the same way, so a pack change shows what it moved on its @@ -141,12 +141,12 @@ #: keyed like the default one. A new pack adds its files here, or its #: name to NO_CLDR; test_cldr_agreement.py fails on a pack in neither. LOCALE_FILES = {"zh": "ZH", "zh_Hant": "ZH", "yue": "ZH", "yue_Hans": "ZH", - "ja": "JA", "ru": "RU", "tr": "TR_AZ", "az": "TR_AZ"} + "ja": "JA", "ne": "NE", "ru": "RU", "tr": "TR_AZ", "az": "TR_AZ"} #: Packs with no CLDR file to measure them on. NO_CLDR: frozenset[str] = frozenset() #: The locale view of the native-script Wikidata source: stratum label -> #: the pack its renderings are parsed with (its QID file names no language). -NATIVE_LABELS = {"zh-cn": "ZH", "zh-tw": "ZH", "ja-jp": "JA"} +NATIVE_LABELS = {"zh-cn": "ZH", "zh-tw": "ZH", "ja-jp": "JA", "ne-np": "NE"} #: Packs measured only where an optional extra is installed: pack -> the #: module the extra provides. _EXTRAS = {"JA": "namedivider"} diff --git a/tools/differential/cldr_agreement_locales.json b/tools/differential/cldr_agreement_locales.json index 0d93f87c..a68dbdbf 100644 --- a/tools/differential/cldr_agreement_locales.json +++ b/tools/differential/cldr_agreement_locales.json @@ -305,6 +305,141 @@ }, "text": "一郎安藤" }, + "ne/foreignFull/givenFirst": { + "family": { + "expected": "भोन ब्रुहल", + "got": "डिडिएस" + }, + "family_base": { + "expected": "ब्रुहल", + "got": "डिडिएस" + }, + "family_particles": { + "expected": "भोन", + "got": "" + }, + "given_names": { + "expected": "एडा कर्नेलिया सिजर मार्टिन", + "got": "एडा कर्नेलिया सिजर मार्टिन भोन ब्रुहल एमडी" + }, + "suffix": { + "expected": "एमडी डिडिएस", + "got": "" + }, + "text": "एडा कर्नेलिया सिजर मार्टिन भोन ब्रुहल एमडी डिडिएस" + }, + "ne/foreignFull/sorting": { + "family": { + "expected": "ब्रुहल भोन", + "got": "ब्रुहल" + }, + "family_particles": { + "expected": "भोन", + "got": "" + }, + "given_names": { + "expected": "एडा कर्नेलिया सिजर मार्टिन", + "got": "एडा कर्नेलिया सिजर मार्टिन भोन" + }, + "text": "ब्रुहल, एडा कर्नेलिया सिजर मार्टिन भोन" + }, + "ne/foreignFull/surnameFirst": { + "family": { + "expected": "भोन ब्रुहल", + "got": "भोन" + }, + "family_base": { + "expected": "ब्रुहल", + "got": "भोन" + }, + "family_particles": { + "expected": "भोन", + "got": "" + }, + "given_names": { + "expected": "एडा कर्नेलिया सिजर मार्टिन", + "got": "ब्रुहल एडा कर्नेलिया सिजर मार्टिन एमडी डिडिएस" + }, + "suffix": { + "expected": "एमडी डिडिएस", + "got": "" + }, + "text": "भोन ब्रुहल एडा कर्नेलिया सिजर मार्टिन एमडी डिडिएस" + }, + "ne/foreignG/sorting": { + "family": { + "expected": "", + "got": "सिनबाद" + }, + "given_names": { + "expected": "सिनबाद", + "got": "" + }, + "text": "सिनबाद" + }, + "ne/foreignG/surnameFirst": { + "family": { + "expected": "", + "got": "सिनबाद" + }, + "given_names": { + "expected": "सिनबाद", + "got": "" + }, + "text": "सिनबाद" + }, + "ne/nativeFull/givenFirst": { + "family": { + "expected": "वुस्टर", + "got": "एमपी" + }, + "family_base": { + "expected": "वुस्टर", + "got": "एमपी" + }, + "given_names": { + "expected": "बर्ट्राम विलबरफोर्स हेनरी रोबर्ट", + "got": "बर्ट्राम विलबरफोर्स हेनरी रोबर्ट वुस्टर" + }, + "suffix": { + "expected": "एमपी", + "got": "" + }, + "text": "बर्ट्राम विलबरफोर्स हेनरी रोबर्ट वुस्टर एमपी" + }, + "ne/nativeFull/surnameFirst": { + "given_names": { + "expected": "बर्ट्राम विलबरफोर्स हेनरी रोबर्ट", + "got": "बर्ट्राम विलबरफोर्स हेनरी रोबर्ट एमपी" + }, + "suffix": { + "expected": "एमपी", + "got": "" + }, + "text": "वुस्टर बर्ट्राम विलबरफोर्स हेनरी रोबर्ट एमपी" + }, + "ne/nativeG/sorting": { + "family": { + "expected": "", + "got": "सुन्दर" + }, + "given_names": { + "expected": "सुन्दर", + "got": "" + }, + "text": "सुन्दर" + }, + "ne/nativeG/surnameFirst": { + "family": { + "expected": "", + "got": "सुन्दर" + }, + "given_names": { + "expected": "सुन्दर", + "got": "" + }, + "text": "सुन्दर" + }, "ru/foreignFull/givenFirst": { "family": { "expected": "фон Брюль", diff --git a/tools/differential/local_corpora.py b/tools/differential/local_corpora.py index 9e365b10..4acb188f 100644 --- a/tools/differential/local_corpora.py +++ b/tools/differential/local_corpora.py @@ -24,7 +24,11 @@ flags are kept under alike/ beside the corpus, keyed like snapshot.py's baselines by the fetch, a hash of the base's nameparser/ files, the interpreter's minor version and the view's code (alike_fingerprint), -and read in a child process where none is kept. +and read in a child process where none is kept. `alike --languages ne` +reads the tree under a choice of languages (Locale.languages) and counts +the persons it moves against the default's flags, the tree's own or +--base's, as snapshot.py --languages does, and like it reads the choice +alone, not a registered pack's fragment or patch beside it. `fetch` writes three files into local// under $NAMEPARSER_TOOLS_CACHE (default ~/.cache/nameparser-tools), and @@ -544,34 +548,44 @@ def total(self) -> int: return sum(self.shapes.values()) -def surname(name: str) -> str: +@functools.cache +def _parse(languages: tuple[str, ...]) -> Callable[[str], Any]: + """The parse a view reads with: parse() itself, or a parser under the + choice of `languages` (Locale.languages), built once per process.""" + from nameparser import Lexicon, Locale, parse, parser_for + if not languages: + return parse + return parser_for(Locale(code="choice", lexicon=Lexicon.empty(), + languages=frozenset(languages))).parse + + +def surname(name: str, languages: tuple[str, ...] = ()) -> str: """The last word of the family name parse() reads in `name`, in capitals without accents, or "" where it reads none: the word two spellings of one person should share unless the last surname itself is what one leaves out (a particle or a middle name left out, a middle name shortened, accents or case changed). Two readings with no surname do not share one.""" - from nameparser import parse - - family = _pair_words(parse(name).family) + family = _pair_words(_parse(languages)(name).family) return family[-1] if family else "" -def _surnames(names: list[str]) -> tuple[str, list[str]]: +def _surnames(names: list[str], languages: tuple[str, ...] = () + ) -> tuple[str, list[str]]: """The nameparser this process imported, and surname() of each name: a unit of work for _parsed.""" import nameparser - return nameparser.__file__, [surname(n) for n in names] + return nameparser.__file__, [surname(n, languages) for n in names] -def _parsed(chunk: Callable[[list[str]], tuple[str, list[Any]]], names: Iterable[str], - jobs: int) -> dict[str, Any]: - """chunk's answer for each distinct name, read in `jobs` worker - processes where there are enough (snapshot.parallel).""" +def _parsed(chunk: Callable[..., tuple[str, list[Any]]], names: Iterable[str], + jobs: int, *args: object) -> dict[str, Any]: + """chunk(part, *args)'s answer for each distinct name, read in `jobs` + worker processes where there are enough (snapshot.parallel).""" from snapshot import parallel distinct = list(dict.fromkeys(names)) - return dict(zip(distinct, parallel(chunk, distinct, jobs, module="local_corpora"), - strict=True)) + return dict(zip(distinct, parallel(chunk, distinct, jobs, *args, + module="local_corpora"), strict=True)) def pair_counts(local: Local, *, agreement: bool = False, jobs: int = 1) -> PairCounts: @@ -655,10 +669,8 @@ class _Read(NamedTuple): suffix: str -def _read(name: str) -> _Read: - from nameparser import parse - - parsed = parse(name) +def _read(name: str, languages: tuple[str, ...] = ()) -> _Read: + parsed = _parse(languages)(name) roles = [str(t.role) for t in parsed.tokens if str(t.role) in _NAME_ROLES] family = [i for i, role in enumerate(roles) if role == "family"] at: FamilyAt @@ -756,11 +768,12 @@ def _by_entity(local: Local) -> dict[str, list[tuple[str, str]]]: return by_entity -def _reads(names: list[str]) -> tuple[str, list[_Read]]: +def _reads(names: list[str], languages: tuple[str, ...] = () + ) -> tuple[str, list[_Read]]: """The nameparser this process imported, and _read of each name: a unit of work for _parsed.""" import nameparser - return nameparser.__file__, [_read(n) for n in names] + return nameparser.__file__, [_read(n, languages) for n in names] def group_counts(local: Local, *, jobs: int = 1) -> GroupCounts: @@ -871,7 +884,8 @@ def alike_kind(local: Local) -> str: raise Ungroupable(f"{local.source}'s records pair no two spellings of one person") -def alike_flags(local: Local, *, jobs: int = 1) -> dict[str, bool]: +def alike_flags(local: Local, *, jobs: int = 1, + languages: tuple[str, ...] = ()) -> dict[str, bool]: """Whether each person's spellings read alike, keyed by the entity or by the record's place in raw.jsonl, so that a key names no one. A person spelled two or more ways (alike_kind "entity") reads alike @@ -879,20 +893,24 @@ def alike_flags(local: Local, *, jobs: int = 1) -> dict[str, bool]: pair where the two readings are one; a filed and reference pair whose spellings differ where both read one surname, the same (surname(), as pair_counts' agreement counts it). Each distinct name is parsed once, - in `jobs` worker processes.""" + in `jobs` worker processes, under the choice of `languages` where + one is given.""" kind = alike_kind(local) if kind == "entity": groups = {e: [n for _, n in v] for e, v in _by_entity(local).items() if len(v) > 1} - read = _parsed(_reads, (n for names in groups.values() for n in names), jobs) + read = _parsed(_reads, (n for names in groups.values() for n in names), jobs, + languages) return {e: len({read[n].reading for n in names}) == 1 for e, names in groups.items()} if kind == "devanagari": pairs = {str(i): (r["name"], r["devanagari"]) for i, r in enumerate(local.records) if r["devanagari"].strip()} - read = _parsed(_reads, (n for pair in pairs.values() for n in pair), jobs) + read = _parsed(_reads, (n for pair in pairs.values() for n in pair), jobs, + languages) return {i: read[a].reading == read[b].reading for i, (a, b) in pairs.items()} pairs = {str(i): (r["name"], r["registry"]) for i, r in enumerate(local.records) if pair_shape(r["name"], r["registry"]) not in ("the same", "no reference name")} - family = _parsed(_surnames, (n for pair in pairs.values() for n in pair), jobs) + family = _parsed(_surnames, (n for pair in pairs.values() for n in pair), jobs, + languages) return {i: family[a] == family[b] != "" for i, (a, b) in pairs.items()} @@ -900,7 +918,7 @@ def alike_flags(local: Local, *, jobs: int = 1) -> dict[str, bool]: #: with _ALIKE_CONSTANTS, the record cut (read_records and #: corpus_build.jsonl_lines, which place a record at its index) and the #: fields of the readings compared. -_ALIKE_CODE = ("alike_kind", "alike_flags", "_by_entity", "_reads", "_read", "reading", +_ALIKE_CODE = ("alike_kind", "alike_flags", "_by_entity", "_reads", "_read", "_parse", "reading", "_surnames", "_parsed", "surname", "_pair_words", "_bare", "_unmarked", "pair_shape", "_in_order") _ALIKE_CONSTANTS = ("ALIKE_UNITS", "PAIR_MISSPELLED", "_PAIR_WORD", "_NAME_ROLES") @@ -1390,10 +1408,13 @@ def _positive(text: str) -> int: def _alike_command(local: Local, args: argparse.Namespace) -> int: """`alike`: the tree's flags (this checkout's, or --tree's, checked to be the package imported), against --base's where one is named; - exits 1 where a person moved, as the other base checks do.""" + under --languages, the tree's flags under that choice, against the + default's (--base's, or the tree's own); exits 1 where a person + moved, as the other base checks do.""" import snapshot root = Path(args.tree).resolve() if args.tree else snapshot.REPO_ROOT - base = snapshot.base_root(args.base, root) if args.base else None + languages = args.languages + base = snapshot.base_root(args.base, root, bool(languages)) if args.base else None try: alike_kind(local) old = base_flags(base, local, args.jobs) if base is not None else None @@ -1403,18 +1424,31 @@ def _alike_command(local: Local, args: argparse.Namespace) -> int: sys.path.insert(0, str(root)) import nameparser snapshot.check_source(nameparser.__file__, root) - flags = alike_flags(local, jobs=args.jobs) + if languages and old is None: + old = alike_flags(local, jobs=args.jobs) + flags = alike_flags(local, jobs=args.jobs, languages=languages) snapshot.check_source(nameparser.__file__, root) if args.out: write_flags(args.out, local, flags) return 0 print(local.header()) if base is not None: - print(snapshot.head_line(root, base)) + print(snapshot.head_line(root, base, languages)) + if languages: + print(f"under the choice {', '.join(languages)}, against the default's " + f"flags\n") print("\n".join(format_alike(local, flags, old))) return int(old is not None and any(flags[k] != old[k] for k in flags)) +def _tags(value: str) -> tuple[str, ...]: + """--languages' comma-separated tags, at least one.""" + tags = tuple(t.strip() for t in value.split(",") if t.strip()) + if not tags: + raise argparse.ArgumentTypeError("needs at least one tag") + return tags + + def main(argv: list[str] | None = None) -> int: ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) sub = ap.add_subparsers(dest="command", required=True) @@ -1449,6 +1483,10 @@ def main(argv: list[str] | None = None) -> int: alike.add_argument("--tree", metavar="PATH", help="read with another checkout") alike.add_argument("--out", type=Path, metavar="PATH", help="write the tree's flags here (what --base's child does)") + alike.add_argument("--languages", type=_tags, default=(), metavar="TAGS", + help="read under a choice of languages, comma-separated " + "BCP 47 tags (ne,hi), and count the persons it moves " + "against the default's flags") for command in (pair, group, alike): command.add_argument("--jobs", type=_positive, metavar="N", default=os.cpu_count() or 1, @@ -1456,6 +1494,9 @@ def main(argv: list[str] | None = None) -> int: args = ap.parse_args(argv) if args.command == "alike" and args.base and args.out: ap.error("--base compares; --out writes the tree's flags, for a child") + if args.command == "alike" and args.languages and args.out: + ap.error("--languages reads a choice against the default's flags; " + "--out writes the default's, for a child") if args.command == "fetch": local = None if not args.refresh: diff --git a/tools/differential/snapshot.py b/tools/differential/snapshot.py index 2191255f..040c6524 100644 --- a/tools/differential/snapshot.py +++ b/tools/differential/snapshot.py @@ -50,6 +50,9 @@ readings, so with no other flag it prints every committed corpus name the choice moves, and with --local SOURCE --base PATH every local one, grouped by changed field as for the default; a choice records nothing. +It reads the choice alone: a registered pack's lexicon fragment or +policy patch beside its choice is not applied, so a pack holding either +is measured whole only by parsing with parser_for(pack). --sample N reads a fixed 1-in-N sample of a local corpus, chosen by a hash of the name so a sample recorded from one tree and checked from another holds the same names, into a