Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely o

### Vocabulary (`nameparser/vocab/`) and the configuration layer (`nameparser/config/`)

**The default vocabulary is written per word, in TOML, and compiled.** `nameparser/vocab/<tag>.toml` holds one file per BCP 47 language tag, each `[[word]]` entry a `text` and its readings keyed by kind. What a file means, how an entry is built, the kinds and properties with the set each compiles into, exclusions, bearer counts and the steps for adding a word are in `nameparser/vocab/README.md`, the reference for anyone editing the TOML; `tests/v2/test_vocab.py` holds its kind, property, derived-marker and reserved-name tables to the names `build.py` accepts and the sets it compiles, so read it rather than restating it here. The parser never reads the TOML: `tools/vocab/build.py` runs the build checks that replaced the modules' import-time asserts (its docstring numbers them), turns every reading into an index row with its file's tag, and writes the rows as the committed reading index `nameparser/config/_index.py` (#753, phase 3a) and, compiled, as the committed `nameparser/config/_compiled.py`, frozenset literals one entry per line; the build never reads `_index.py` back. A file a shipped pack reads may also hold a `[rules]` table, each key a `Policy` field (`patronymic_rules`, `name_order`, `two_family_names`, `two_given_names`), which the build checks against its `RULE_KEYS` and `PACK_RULES` and writes as `nameparser/config/_rules.py`; `_locale.rules_patch(tag)` turns a file's table into the `PolicyPatch` RU and TR_AZ carry, from `ru.toml` and `tr.toml` (#754, 4f; decisions/vocabulary-model's 2026-10-29 4f entry). The compiler is `nameparser/_reading_index.py`'s `compile_readings`, with the kind and property tables it reads (`CATEGORY`, `BEHAVIORAL`, `GATED_WORDS`), so the build and a choice of languages compile alike: `Locale.languages` (3.0, phase 3b) is a choice of BCP 47 tags, which `parser_for` unions across its packs and applies as a delta against the default's compile, `choose` there selecting the readings (a chosen file's `default = false` ones on; on a word a chosen file holds a vocabulary reading of, other files' vocabulary readings off, save `mul.toml`'s and `und.toml`'s; a chosen conjunction's `may_be_initial` dropped), and `changed_words` deriving a choice's deviation surface, which tests/v2/test_locales.py's gate reads (decisions/vocabulary-model, 2026-10-29 3b); the build keeps the checks and loads that module by path, as it loads `_lexicon.py`, and `tests/v2/test_reading_index.py` holds that the index compiled with no choice is `_compiled.py` exactly. `readings_of(text)` there is the run-time reader of the index, every reading of a word with its file's tag, loaded on first call; `languages_of(text, categories)` is the reader a parse calls, from assemble, for `Reading.languages` (3.0), so the first parse that reads a word from its `Lexicon`'s vocabulary loads the index. Each `config/` module re-exports its constants from `_compiled.py` (`TITLES = _compiled.TITLES`) under its autodoc docstring, so the public names and the docs did not move. `tests/v2/test_vocab.py` fails when `_compiled.py` is stale, and `tests/v2/test_reading_index.py` when the index is, each naming the refresh command, and `test_vocab.py` gives every build check a mutation. The model, the phasing and what was declined: decisions/vocabulary-model.
**The default vocabulary is written per word, in TOML, and compiled.** `nameparser/vocab/<tag>.toml` holds one file per BCP 47 language tag, each `[[word]]` entry a `text` and its readings keyed by kind. What a file means, how an entry is built, the kinds and properties with the set each compiles into, exclusions, bearer counts and the steps for adding a word are in `nameparser/vocab/README.md`, the reference for anyone editing the TOML; `tests/v2/test_vocab.py` holds its kind, property, derived-marker and reserved-name tables to the names `build.py` accepts and the sets it compiles, so read it rather than restating it here. The parser never reads the TOML: `tools/vocab/build.py` runs the build checks that replaced the modules' import-time asserts (its docstring numbers them), turns every reading into an index row with its file's tag, and writes the rows as the committed reading index `nameparser/config/_index.py` (#753, phase 3a) and, compiled, as the committed `nameparser/config/_compiled.py`, frozenset literals one entry per line; the build never reads `_index.py` back. A file a shipped pack reads may also hold a `[rules]` table, each key a `Policy` field (`patronymic_rules`, `name_order`, `two_family_names`, `two_given_names`), which the build checks against its `RULE_KEYS` and `PACK_RULES` and writes as `nameparser/config/_rules.py`; `_locale.rules_patch(tag)` turns a file's table into the `PolicyPatch` a pack carries, RU's and TR_AZ's from `ru.toml` and `tr.toml`, and the `two_family_names` of ES, CS and SK from `es.toml`, `cs.toml` and `sk.toml` (#754, 4f; decisions/vocabulary-model's 2026-10-29 4f entry). The compiler is `nameparser/_reading_index.py`'s `compile_readings`, with the kind and property tables it reads (`CATEGORY`, `BEHAVIORAL`, `GATED_WORDS`), so the build and a choice of languages compile alike: `Locale.languages` (3.0, phase 3b) is a choice of BCP 47 tags, which `parser_for` unions across its packs and applies as a delta against the default's compile, `choose` there selecting the readings (a chosen file's `default = false` ones on; on a word a chosen file holds a vocabulary reading of, other files' vocabulary readings off, save `mul.toml`'s and `und.toml`'s; a chosen conjunction's `may_be_initial` dropped), and `changed_words` deriving a choice's deviation surface, which tests/v2/test_locales.py's gate reads (decisions/vocabulary-model, 2026-10-29 3b); the build keeps the checks and loads that module by path, as it loads `_lexicon.py`, and `tests/v2/test_reading_index.py` holds that the index compiled with no choice is `_compiled.py` exactly. `readings_of(text)` there is the run-time reader of the index, every reading of a word with its file's tag, loaded on first call; `languages_of(text, categories)` is the reader a parse calls, from assemble, for `Reading.languages` (3.0), so the first parse that reads a word from its `Lexicon`'s vocabulary loads the index. Each `config/` module re-exports its constants from `_compiled.py` (`TITLES = _compiled.TITLES`) under its autodoc docstring, so the public names and the docs did not move. `tests/v2/test_vocab.py` fails when `_compiled.py` is stale, and `tests/v2/test_reading_index.py` when the index is, each naming the refresh command, and `test_vocab.py` gives every build check a mutation. The model, the phasing and what was declined: decisions/vocabulary-model.

The `config/` SETS are frozen since 2.2 (#293): there is no `.add()`/`.remove()` on any of them, so a default word list is changed at runtime by configuring a `Lexicon`, never by editing the constant, and the shipped default is changed by editing the TOML and rebuilding. **Neither dict was frozen, and `CAPITALIZATION_EXCEPTIONS` is not covered by anything else either.** `REGEXES` is a compiled-pattern table rather than vocabulary and was never in #293's scope; `CAPITALIZATION_EXCEPTIONS` is vocabulary-shaped and is a decided, in-scope exemption. So the hazard the freeze closes is still live for it: `CAPITALIZATION_EXCEPTIONS['dclinpsy'] = 'DClinPsy'` after the first parse misses the cached `Lexicon.default()`. Same advice — configure the object (`dataclasses.replace(lexicon, capitalization_exceptions=...)`). `tests/v2/test_contracts.py::test_every_vocabulary_constant_is_frozen` names both dicts as explicit exemptions rather than letting its `isinstance` filter drop them.

Expand Down
12 changes: 12 additions & 0 deletions docs/design/decisions/O6-two-family-names.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

- Two family names are an opt-in `Policy` switch, `two_family_names`, off by default: the convention is Spain's and most of Latin America's but not Argentina's, and given-first it is the same string shape as a name with a middle name, so no default can read it — 2026-10-24 #754.
- `locales.ES` turns the switch on, carrying the Spanish given-name list beside it, one pack for Spain and Latin America alike, Argentina included; `locales.PT` leaves it off until a Portuguese given-name list exists, decided in chat — 2026-10-30 #753, phase 3g.
- `locales.CS` and `locales.SK` turn the switch on with no given-name list, a net gain in every real population measured (CLDR's fictional renderings lose), two given names the recorded cost; `locales.SL` and `locales.HR` leave it off, decided in chat — 2026-10-30 #753, phase 3f amended.
- Both family names stay in `family` through 2.x; a second-surname field is the RFC's Phase 4 output model (docs/design/vocabulary-model-rfc.md, "The rules a pack can choose"), and a pack may later switch this rule on — 2026-10-24 #754; landed as the `surname2` view, `family` keeping both, 2026-10-29 #754, sub-phase 4d.
- The family's words are counted as P1's fold counts them, a connective join and a hyphen counting as their words, so a family already holding two keeps what it has; the word taken must count as one by the same count and be no initial written alone — 2026-10-24 #754.
- A name its script (W4) reads in an order other than the declared `name_order` is left alone; where the two agree it is read like any other — 2026-10-24 #754.
Expand Down Expand Up @@ -151,3 +152,14 @@ Open: hyphenated surnames block the switch (`Andrés Sánchez-Cid Torres` keeps
| PSC | Cuban | 41 | 36.6% | 75.6% | 78.0% | 24 / 7 | 1 / 0 |

Portuguese, the same recipe, the switch alone under a `pt` choice (no list exists): ORCID PT 71.5% → 83.2% (4,296 fixed, 2,034 broken), BR 49.5% → 74.4% (7,004 / 2,268), Angola 75.9% → 57.3%, Mozambique 85.2% → 50.2%; PSC Portuguese 61.1% → 56.1% (1,215 / 1,478), Brazilian 53.4% → 55.6% (606 / 564). With INE's Spanish list standing in for a Portuguese one, an experiment only, PSC Portuguese reads 60.7% and ORCID PT 88.3%, Angola and Mozambique still below the default. So `locales.PT` waits for a list. The candidate is the given-name and surname counts that Receita Federal's open CNPJ register of company partners yields (`CaiqueDourado/nomes-e-sobrenomes-brasil`, CC BY 4.0, 46,840 given names and 27,047 surnames, each word's first-position and later-position counts); IBGE's Census 2022 "Nomes no Brasil" gives the same shape over the whole population but states no licence, and Portugal publishes no frequency list (IRN's list of admitted given names has no counts). Measured 2026-10-30 on `5c3acb56` plus this change, Python 3.11, ORCID's extract of 2026-10-27 capped at 20,000 records a country and PSC's fetch recorded on #849. Recompute: `local_agreement.read` per record of `local_corpora.load(source)` with `FIELDED[source]`, records with both boxes, grouped by `group_of`, under `Parser()`, `Parser(policy=Policy(two_family_names=True))` and `parser_for(locales.ES)`, each with `_title_words` of its own lexicon, every record counted, a record PSC's title set-aside would drop counting as a disagreement, so PSC's populations are the #959 entry's plus its set-aside records (Spanish 6,101 here against 6,089 there); a record is fixed where it agrees only under the pack, broken where it agrees only under the comparator. For `pt`, `parser_for(Locale(code="t", lexicon=Lexicon.empty(), policy=PolicyPatch(two_family_names=True), languages=frozenset({"pt"})))`, and for the experiment its lexicon `Lexicon(given_names=GIVEN_NAMES)`.

- 2026-10-30 #753, phase 3f amended — `locales.CS` AND `locales.SK` TURN THE SWITCH ON, WITH NO GIVEN-NAME LIST. A Czech or Slovak woman who keeps her surname beside her husband's writes the two without a hyphen (`Alena Pešová Řehořová`), the shape the default reads as a middle name. Decided in chat on the measurement in decisions/vocabulary-model's entry of this date, which holds the recipe. Records agreeing in every field, before → after, the pack at `bfb91163` the before of each row but the second, which is the default's (on ORCID the two read alike):

| Population | Records | Before → after | Fixed | Broken |
|---|---:|---|---:|---:|
| cz-candidates, the Czech election registers' boxes | 170,181 | 96.97% → 97.47% | 1,295 | 442 |
| cz-candidates, against the default | 170,181 | 96.96% → 97.47% | 1,318 | 440 |
| ORCID, country CZ | 20,000 | 94.99% → 96.41% | 635 | 352 |
| ORCID, country SK | 12,962 | 94.84% → 96.40% | 347 | 144 |

Every broken record, in all three, has two or more words in the given box, read as a given name and a family name: no population-wide Czech or Slovak given-name list is published to veto the word, the register's frequencies having been withdrawn after 2016's; ČSÚ's top 100 newborn given names a year (CC BY 4.0) is a list, and the open lead. The circular ceiling such a list could reach, and the sources looked for, are in the vocabulary-model entry.
3 changes: 2 additions & 1 deletion docs/design/decisions/deviates-registry.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Standing

- A locale pack declares its deviations from the defaults as a module-level `DEVIATES` callable, acceptance metadata kept off `Locale`, whose value equality and pickling a callable field would break; `test_registry_is_the_pack_contract` fails unless every registered policy or segmentation pack ships `DEVIATES`, and every registered pack a rotator list, which is what lets mechanisms.md#LOCALE-PACKS-PURE-DATA promise packs no code paths of their own. A field on `Locale` is to be revisited only if a runtime consumer of the declaration, such as an `explain()`, appears, and none has — 2026-07-18. A vocabulary pack's deviations are derivable instead, from the reading index (`_reading_index.changed_words`), which tests/v2/test_locales.py's choice gate reads; the gate ors the derived surface into every registered pack with `languages`, so a hand `DEVIATES` declares only what a fragment or a patch beside a choice changes; the contract test refuses a `DEVIATES` on a pack that is nothing but a choice of languages (`locales.NE`, the first, and `CS`, `SK`, `SL` and `HR`), and a choice pack whose choice changes no word, and requires one of every other pack — 2026-10-29 #753, phase 3b; 2026-10-30 #753, phase 3d; 2026-10-30 #753, phase 3f.
- A locale pack declares its deviations from the defaults as a module-level `DEVIATES` callable, acceptance metadata kept off `Locale`, whose value equality and pickling a callable field would break; `test_registry_is_the_pack_contract` fails unless every registered policy or segmentation pack ships `DEVIATES`, and every registered pack a rotator list, which is what lets mechanisms.md#LOCALE-PACKS-PURE-DATA promise packs no code paths of their own. A field on `Locale` is to be revisited only if a runtime consumer of the declaration, such as an `explain()`, appears, and none has — 2026-07-18. A vocabulary pack's deviations are derivable instead, from the reading index (`_reading_index.changed_words`), which tests/v2/test_locales.py's choice gate reads; the gate ors the derived surface into every registered pack with `languages`, so a hand `DEVIATES` declares only what a fragment or a patch beside a choice changes; the contract test refuses a `DEVIATES` on a pack that is nothing but a choice of languages (`locales.NE`, the first, and `SL` and `HR`), and a choice pack whose choice changes no word, and requires one of every other pack, `CS` and `SK` among them since their patch — 2026-10-29 #753, phase 3b; 2026-10-30 #753, phase 3d; 2026-10-30 #753, phase 3f; 2026-10-30 #753, phase 3f amended.

Open: none in this file.

Expand All @@ -12,3 +12,4 @@ Open: none in this file.
- 2026-10-29 #753, phase 3b — A VOCABULARY PACK'S DEVIATIONS ARE DERIVED, NOT DECLARED. `Locale.languages` selects among the default's readings, so the words a choice can move are computable: `_reading_index.changed_words(chosen)` is every word joining or leaving a compiled constant under the choice, and tests/v2/test_locales.py's `test_non_interference_each_choice` requires the default's reading for every corpus name holding none of them, with a negative control (`test_the_choice_gate_fails_when_its_surface_misses_a_word`). No pack ships in 3b, so the gate runs over choices of the files' tags; the registry contract test still asks every REGISTERED pack for `DEVIATES` and a rotator list, and learns to read a derived surface for a pack with `languages` in the PR that registers the first one (3d). The policy and segmentation packs (JA, RU, TR_AZ, ZH) keep theirs, decisions/vocabulary-model's 2026-10-27 #753 entry, item 7.
- 2026-10-30 #753, phase 3d — THE CONTRACT READS A DERIVED SURFACE. `locales.NE` registers the first pack whose content is a choice of languages and nothing else (`Locale.lexicon` empty, `Locale.policy` the empty patch). tests/v2/test_locales.py's `_deviates(code)` gives the gate, the rotator test and the combined gate every pack's predicate: the surface derived from its `languages` through `changed_words` and the 3b choice gate's `_surface_deviates`, or'd with its `DEVIATES`, so a pack with a fragment or a patch beside a choice needs to declare only that part (`test_a_pack_with_a_choice_and_a_patch_declares_only_the_patch`; nothing refuses a hand copy of the choice's words there); `test_registry_is_the_pack_contract` refuses a `DEVIATES` on a pack that is nothing but its choice, so a hand copy cannot drift beside the derived one, and still requires one of every other pack. Its rotator list stays required, as the gate's positive side. decisions/vocabulary-model's 2026-10-30 phase 3d entry, item 2.
- 2026-10-30 #753, phase 3f — A CHOICE PACK MUST CHANGE A WORD. `locales.CS`, `SK`, `SL` and `HR` register as choice packs under phase 3d's contract, with no `DEVIATES`. `test_registry_is_the_pack_contract` adds one refusal: a choice pack whose choice changes no word (an empty `changed_words`), since its gate would pass on nothing; and `test_choice_rotators_cover_every_changed_word` holds each choice pack's rotator list to a name per changed word, so the gate's positive side reaches every word. The combined gate reads the surface of the union of every registered pack's languages, since `parser_for` applies one choice, not one per pack, or'd with every pack's hand `DEVIATES` whatever the pack, since a mixed pack carries both. decisions/vocabulary-model's 2026-10-30 phase 3f entry.
- 2026-10-30 #753, phase 3f amended — A CHOICE PACK THAT GAINS A PATCH DECLARES THE PATCH'S PART, AND THE PACKS SHARING A PATCH SHARE ONE DECLARATION. `locales.CS` and `locales.SK` gain rules/O6's patch (`two_family_names`), so they leave the choice packs and ship `DEVIATES`, as `locales.ES` does, the contract test requiring it of every pack that is not a choice alone. Their declaration is the one `ES` already made for the same patch: three or more name words, or three or more runs of letters. It moved from `locales/es.py` into `_locale.deviates_by_two_family_names`, which all three packs' `DEVIATES` return, so the three cannot drift apart (`test_cs_and_sk_pack_contents` holds them to it). Each pack's title and generation rotators are names the patch alone leaves as the default reads them (`MUDr Novák`, `Novák PhDr.`, `Jana Nováková (roz. Svobodová)`), since `test_a_choice_word_rotator_is_moved_by_the_choice_alone` refuses one the patch moves; the patch has rotators of its own. decisions/vocabulary-model's entry of this date.
Loading
Loading