diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 40ab5db63..3a6bc5b0f 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -15,10 +15,12 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - Of the romanized titles a pack's phase lists, one whose native spelling is a default title turns on unless, counted in persons and pooled across every population measured, names open with it more often than it opens one as the title; a spelling outside the phase's list (`Shreemati`, `Srimati`) stays open until a phase writes it as a reading: `hi` turns on `smt`, `shrimati` and `shriyut`, `bn` `smt`, `shrimati` and `adhyapak` (`shriyut` with no bearer anywhere, `adhyapak` with family-name bearers only, every one where the title claim, acting at the head, does not reach), while `shri`, opening given names in every population measured, stays excluded, and `sushri`, an Odia given name, waits with no reading written for data strict can read since 3j, a `default = false` salutation in hi.toml with the Odia name declared in or.toml (2026-10-31 #753, phase 3j), a plain choice being made for data that holds those names — 2026-10-30 #753, phase 3e. - A language pack is a choice of languages on `Locale.languages` selecting which readings compile, a tag choosing its own file and its ancestors', and a reading index, loaded on its first lookup, carries the languages to run time; strict is file-strict, the chosen files and native-script vocabulary only, `mul.toml` and `und.toml` not among them, so it needs no 3i; third-party language files are deferred — 2026-10-27 #753 (Derek); 2026-10-29 #753, phase 3a (Derek); 2026-10-29 #753, phase 3b; 2026-10-30 #753, file-strict. - A strict choice is `parser_for(..., strict=True)` over the whole choice, applied as a delta as a plain choice is; a strict call whose packs choose no language raises; a strict choice accepts a tag whose file holds no reading, which a plain one refuses as changing nothing; a bare choice has no constructor, no `locales.choose`; and the words strict ungates are derived against the plain choice of the same tags and held to a reviewed set by a test — 2026-10-31 #753, phase 3j. -- A choice is matched without regard to case, a tag whose file does not exist selects its ancestors' (`pt-PT` selects `pt`), `mul` and `und` are no language to choose, and a reading turned on also prefers, so `de`'s generation `i` drops `ca`'s and `pl`'s conjunction under a `de` choice; a choice is one call's: one chained through `base=` applies on top of the base's rather than unioning with it, and a choice drops the caller's own base entries it removes — 2026-10-29 #753, phase 3b; 2026-10-29 #753, phase 3b, chained choices (Derek). +- A choice is matched without regard to case, a tag whose file does not exist selects its ancestors' (`pt-PT` selects `pt`), `mul` and `und` are no language to choose, and a reading turned on also prefers, so `de`'s generation `i` drops `ca`'s and `pl`'s conjunction under a `de` choice; the `Lexicon` is to record its choice, so that a choice chained through `base=` unions with the base's, and a caller's own edits are to apply after the choice, plain or strict, in the order written (4i-c; until it lands a choice is one call's and drops the caller's base entries it removes) — 2026-10-29 #753, phase 3b; 2026-11-02 #754, the `Lexicon` surface (Derek). - A reading right in its own language and wrong only as a default, because another language written in the same script collides with it, is written `default = false` in its file, never excluded, so that after the packs an exclusion means wrong everywhere: cs `st` since phase 2h, and cs `ml`, hr, sk and sl `ml` and `st` and fr `père` since 3c, the rest in their packs' PRs — 2026-10-27 #753 (Derek); 2026-10-29 #753, phase 3c. - A name reading written `default = false` derives no marker in the default and counts for no check guarding the default's compile; a particle whose word's only `given` readings are off is refused until a pack needs one — 2026-10-29 #753, phase 3a. - A caller configures vocabulary by reading, `Lexicon` deriving the sets and gates the build derives, and the compiled sets become internal in 3.0 — 2026-10-28 #754, the `Lexicon` surface (Derek). +- The configure-by-reading API: `add(*words, kind, **properties)` and `remove(*words, kind=None)`, the views reading a word's kind from the lexicon, `readings_of` returning a public `LexiconReading` and a repr counting readings by kind, a five-part state (base, ordered edit log, choice, lists, masks) compared structurally, a clash a later choice exposes raised at `parser_for` naming the choice, the choice recorded on the `Lexicon` with chained choices unioning, the default's base lazy, a 2.x pickle translated onto the nearer named base and a per-word log, old spellings refused with a per-word translation, built in 4i-a to 4i-d with no default parse moving — 2026-11-02 #754, the `Lexicon` surface (Derek). +- Configured by reading, an `add` of a title reading that a `given` reading keeps a title only inside a phrase is to warn, naming `remove(w, kind="given")`, save one that only updates a held title reading's properties — 2026-11-01 #754, the title warning (Derek). - A `Locale` may carry CLDR-shaped render patterns, display and sorting, a later addition whose shape is agreed; how a call reaches one is open — 2026-10-28 #754, render patterns. - 3.0's output model keeps the seven roles and adds every new field as a view, read from `Token.reading`, the reading each token took, or for `surname2` from rules/O6's tag: `generation` and `credentials` partitioning `suffix`, `patronymic`, and `surname2` where rules/O6 read two family names; `middle` and `surnames` keep their meaning, a compound given name being a rule a caller or pack turns on; the default `str()` renders a name in the order it was read in with its written separators in a normalized form, through `render(spec: str | None = None)`, `None` being that default; a language file's `[rules]` compile into its `Locale`'s policy patch — 2026-10-28 #754 (Derek). - `surname2` is read from O6's mark as `Token.reading` publishes it, `Reading.second_surname`, a field not compared, and not from a published tag, which would make an O6 parse unequal to its 2.x pickle (decisions/R10) — 2026-10-29 #754, sub-phase 4d. @@ -55,7 +57,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part Not standing: the plans the classifications carried out, phase 1's mechanical conversion of the unclassified categories into `und.toml` (2026-10-18 #737) and the shared readings left in place until their categories were classified (2026-10-19 #749), both done by 2026-10-20 #751, the conjunctions; and #747's reason for exempting `und.toml`, its entries keeping the Python lists' order for its vetting comments to point along (2026-10-19 #747), overtaken once it held two entries and no such comment. -Open: #913 (the title words phase 2j left, read as titles though borne as names), #753 (phase 3, language packs; which mul.toml words move so that strict `en` can shed them, and the 16 words strict ungates, each read for a name reading in the chosen file; whether a `Parser` records its choice, so that chained choices union, `Parser.languages`, decided with the configure-by-reading method names), #754 (phase 4, order, shapes and the output model). +Open: #913 (the title words phase 2j left, read as titles though borne as names), #753 (phase 3, language packs; which mul.toml words move so that strict `en` can shed them, and the 16 words strict ungates, each read for a name reading in the chosen file), #754 (phase 4, order, shapes and the output model; the `Lexicon` surface's build, 4i-a to 4i-d). ## Log @@ -314,7 +316,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **A choice chained through `base=` is applied on top of the base's, not unioned with it**, decided in the next entry (option (a) of three weighed there). A `Parser` records no choice, so `parser_for(EN, base=parser_for(PT))` applies `en`'s delta to a lexicon `pt` already changed: `Maria do Carmo` reads family `Carmo` and `John Smith DO` family `DO`, where `parser_for(PT, EN)` keeps both readings (`do Carmo`, `Smith`). Item 1's order-independence holds within one call. - **`Reading.languages` does not follow a choice: it names the files holding the reading on by default, whatever choice built the parser.** SUPERSEDES: the 4a follow-up entry's expectation, in its item 1, that 3b changes `languages_of`'s filter on `on_by_default`. Measured on this branch merged with that follow-up: under `pt`, the one-case `E` of `ANA SILVA E SOUZA` reads `conjunction` with languages `('it', 'pt')`, though prefer dropped `it`'s reading from the pool; under `cs`, the `st.` of `Jiří Dienstbier st.` reads `generation` with languages `()`, though `cs.toml` holds the reading the choice turned on. Kept, for two reasons: the follow-up decided the field names the default's files whatever `Lexicon` the parse used, a caller's own `St` in `suffix_words` naming none for the same `cs` reading; and a parse cannot know its choice, a `Parser` recording none (the chained-choices entry below), so a field that followed the choice needs the `Parser.languages` left open there. The cost is that a reading a choice turns on names no file and a language a choice dropped is still named; tests/v2/test_parser.py's `test_reading_languages_name_the_default_files_under_a_choice` pins both, so a change with `Parser.languages` moves them deliberately. - Not touched: `Lexicon` and `Parser` gain no field; `_locale.py` imports nothing new beyond `_types.ADDED_IN` and `_policy`'s string-and-mapping guard; `_parser.py` imports `_reading_index` at call time, and tests/v2/test_layering.py's table says so. 4f's `[rules]` compiled into a `Locale`'s policy patch needs no change to this shape: a pack's policy patch and its `languages` are separate fields. Strict (item 4) is not built. -- **2026-10-29 #753, phase 3b, chained choices (Derek) — ONLY ONE CALL'S CHOICE COUNTS: A CHOICE CHAINED THROUGH `base=` APPLIES ON TOP OF THE BASE'S, AND A CHOICE DROPS THE CALLER'S OWN BASE ENTRIES THAT IT REMOVES.** Derek decided in chat on 2026-10-29, choosing option (a) of the three the phase 3b entry above weighed. `parser_for` unions the `languages` of the packs passed to it and applies that one choice as a delta against the default's compile; a `Parser` records no choice, so a base built by an earlier `parser_for` carries its choice only as the lexicon it changed. Two consequences, both pinned by tests in tests/v2/test_parser.py so that a change to them is deliberate: +- **2026-10-29 #753, phase 3b, chained choices (Derek) — SUPERSEDED by 2026-11-02 #754, the `Lexicon` surface (to take effect with 4i-c) — ONLY ONE CALL'S CHOICE COUNTS: A CHOICE CHAINED THROUGH `base=` APPLIES ON TOP OF THE BASE'S, AND A CHOICE DROPS THE CALLER'S OWN BASE ENTRIES THAT IT REMOVES.** Derek decided in chat on 2026-10-29, choosing option (a) of the three the phase 3b entry above weighed. `parser_for` unions the `languages` of the packs passed to it and applies that one choice as a delta against the default's compile; a `Parser` records no choice, so a base built by an earlier `parser_for` carries its choice only as the lexicon it changed. Two consequences, both pinned by tests in tests/v2/test_parser.py so that a change to them is deliberate: 1. **Chained choices compose one after the other.** `parser_for(EN, base=parser_for(PT))` reads `Maria do Carmo` family `Carmo` and `John Smith DO` family `DO`: `en` drops `pt`'s particle and `pt` had dropped `en`'s credential. The way to choose both is one call, `parser_for(PT, EN)`, which keeps both readings (`do Carmo`, suffix `DO`). 2. **A choice drops what it removes from the base, whoever put it there.** `parser_for(EN, base=Parser(lexicon=Lexicon.empty().add(particles={'do', 'van'})))` keeps only `van`: the delta removes `do` from `particles`, the caller's listing included, and leaves every entry it does not touch. Weighed against: (b) a `Parser.languages` field recording the choice, so a base's choice unions with the packs' (a new public name, safe for pickles, `Parser`'s guard honoring `ADDED_IN`), deferred rather than declined; (c) a `Lexicon` field recording it, which every older `Lexicon`, `Parser` and pack pickle would refuse, `Lexicon`'s guard not honoring `ADDED_IN` (an objection removed on 2026-10-30, when that guard learned the marker, #960, decisions/A1). Open: (b), with the configure-by-reading method names (the 2026-10-28 #754 `Lexicon` surface entry), and with it whether `Reading.languages` follows the choice (the phase 3b entry above). @@ -447,7 +449,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **3i's evidence, kept** from the RFC paragraph this entry retires, should 3i be wanted again: all measured 2026-10-28 on master `c8357e74`, not on this entry's tree: of the 227 name readings, 165 words carried one; `do`'s family box led by Brazil in ORCID (none of Brazil's 7,227 `do` alone, 2,629 opening with it and 4,598 holding it inside) but by share Vietnam (1.90% against 0.81%; PatentsView 1.78% and 0.64%), Vietnam having 618 boxes of `do` alone and Korea 151; `kim`'s family box 17.67% of ORCID's Korean holders and 13.30% of PatentsView's, its given box 0.93% of PatentsView's Danish inventors. Recompute: `uv run python tools/name_usage.py do kim --no-wikidata --boxes`, and the whole-box split from the cached ORCID extract's `family` and `country` keys. - **Candidate next steps, not taken here:** Vietnamese degree prefixes as vi.toml readings, each checked for collisions and bearers per nameparser/vocab/AGENTS.md first; `ông` and `bà` stay excluded, Ông being a Vietnamese surname, a clash within Vietnamese strict does not settle; 3h's `nl` prepositions and 3e's `sushri` as data with the clashing name readings declared in ko.toml, vi.toml and or.toml, gated under the plain choice and not under strict, though the plain `nl` choice still reads `Park In Hwan` family `In Hwan` and `Nguyen To Nga` family `To Nga`, item 4's recorded cost, the marker acting at the head only (re-run 2026-10-31), which a follow-up must settle before shipping them; `_reading_index`'s `UNCHOOSABLE` comment and `selected`'s refusal text, which say no choice removes a reading of `mul` or `und`, updated with 3j; the 16 ungated words read one by one. - **Recompute:** a strict choice of the tags `T` is `compile_readings(choose(rows, T))` from `nameparser._reading_index`, `rows` the `index()` rows whose tag is in `T` or whose text has a letter and no letter whose `unicodedata.name` contains `LATIN`, applied to `Lexicon.default()` as `_parser._apply_choice` applies `_choice_delta` (the delta taken against `compile_choice(frozenset())`, the markers of a dropped base entry removed), and a pack's own patch and fragment applied by `parser_for(dataclasses.replace(pack, languages=frozenset()), base=Parser(lexicon=that))`. Count a word's share of the moves as the moves whose name, split at spaces and commas with edge periods dropped and casefolded, holds it. Score ORCID with `local_agreement.read(record, local_agreement.FIELDED["orcid-researchers"], parser, local_agreement._title_words(parser.lexicon))` over the cache's `raw.jsonl` records of each country, and PSC the same with `"psc-individuals"` and the `nationality` key; the title-word filter is the given box's first word, split at spaces and periods and casefolded, in {dr, prof, professor, mr, mrs, ms, miss, dra, ing, mgr, assoc, assist, er, engr, drs, ir}. The ungated words: for each choosable tag, the words in a marker (`particles_ambiguous`, `suffix_acronyms_ambiguous`, `suffix_words_ambiguous`, `titles_ambiguous`) of the plain choice of that tag (`parser_for(Locale(code="x", lexicon=Lexicon.empty(), languages={tag})).lexicon`) that stay in the base set of the strict lexicon and leave the marker, or join `titles`; compared with `Lexicon.default()`'s markers instead, it misses the readings a choice turns on (`st` under cs, hr, sk and sl, `nic` under gd) and finds 15 words. The scripts are in no tree: they are a few dozen lines each from this recipe. -- **2026-10-31 #753, phase 3j — FILE-STRICT IS BUILT: `parser_for(..., strict=True)` COMPILES ONLY THE CHOSEN FILES AND NATIVE-SCRIPT VOCABULARY; A STRICT CALL CHOOSING NO LANGUAGE RAISES; NO `locales.choose`; THE WORDS STRICT UNGATES ARE DERIVED AND HELD BY A TEST; A FILE HOLDING NO READING IS A STRICT CHOICE ONLY; THE DEFAULT DOES NOT MOVE.** SUPERSEDES: in the 2026-10-30 #753, file-strict entry, its verdict's "the rest of its API is proposed, not decided, and nothing is built"; its "Proposed, not decided" bullet, every item decided as proposed save `locales.choose(*tags)`, which is left out; and the files-with-no-readings bullet's "strict `vi` needs vi.toml readings first". Carries out 3j of item 11 of the 2026-10-27 #753 entry, with the definition of the 2026-10-30 entry. The four API points below were decided on 2026-10-31; the rest of the 2026-10-30 entry's proposed bullet is built as proposed (no `Locale.strict` field; the chosen files' `default = false` readings turned on and no other file's; patches and fragments applied after the strict compile, untouched; `Lexicon.given_names` and `Lexicon.surnames` outside it; packs stacked as a union; a caller's listing of a dropped word dropped by the delta). Built on master `8438ccbb`. +- **2026-10-31 #753, phase 3j (its built clause that a caller's listing of a dropped word is dropped by the delta superseded by 2026-11-02 #754, the `Lexicon` surface, to take effect with 4i-c) — FILE-STRICT IS BUILT: `parser_for(..., strict=True)` COMPILES ONLY THE CHOSEN FILES AND NATIVE-SCRIPT VOCABULARY; A STRICT CALL CHOOSING NO LANGUAGE RAISES; NO `locales.choose`; THE WORDS STRICT UNGATES ARE DERIVED AND HELD BY A TEST; A FILE HOLDING NO READING IS A STRICT CHOICE ONLY; THE DEFAULT DOES NOT MOVE.** SUPERSEDES: in the 2026-10-30 #753, file-strict entry, its verdict's "the rest of its API is proposed, not decided, and nothing is built"; its "Proposed, not decided" bullet, every item decided as proposed save `locales.choose(*tags)`, which is left out; and the files-with-no-readings bullet's "strict `vi` needs vi.toml readings first". Carries out 3j of item 11 of the 2026-10-27 #753 entry, with the definition of the 2026-10-30 entry. The four API points below were decided on 2026-10-31; the rest of the 2026-10-30 entry's proposed bullet is built as proposed (no `Locale.strict` field; the chosen files' `default = false` readings turned on and no other file's; patches and fragments applied after the strict compile, untouched; `Lexicon.given_names` and `Lexicon.surnames` outside it; packs stacked as a union; a caller's listing of a dropped word dropped by the delta). Built on master `8438ccbb`. 1. **A strict call that chooses no language raises `ValueError`.** The message names the packs given and offers a pack that chooses a language, the first pack's code where a strict choice accepts it as one (`parser_for(locales.ZH, strict=True)` offers `Locale(code='zh', lexicon=Lexicon.empty(), languages=frozenset({'zh'}))`), else `en`; tests/v2/test_parser.py runs the offered code. 2. **No public `locales.choose`.** A bare choice is a `Locale` with an empty lexicon and the tags, the spelling docs/locales.rst teaches. 3. **The words strict ungates are derived**, `_reading_index.strict_ungated`, each strict choice against the plain choice of the same tags and never against the default's markers, which miss a reading the choice turns on; tests/v2/test_reading_index.py holds every one-tag strict choice to a per-tag table, which is the 2026-10-30 entry's 16 words and 27 word-and-tag pairs over 13 tags exactly, so a word that joins or leaves is a reviewed diff. @@ -458,3 +460,29 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **Measured: the 2026-10-30 entry's ORCID table re-run on the built code**, its recipe with `parser_for(pack, strict=True)` (and a bare `en` pack beside it for strict + en) in place of the simulation, on this phase's branch over `8438ccbb`, Python 3.11: every cell matches, 14 countries by four columns. - **Measured: the worked examples on the built code**, default order, against the default: under strict `ko`, `Kim Do Hyun` reads middle `Do`, family `Hyun` (default family `Do Hyun`, pt's particle), `Kim Bin Hyun` middle `Bin` (ar's and ms's particle), `Ra Min Jun` given `Ra` (de's academic title), `Kim Min Ra` family `Ra` (en's credential), back to suffix `Ra` under strict `ko` + `en`; strict `ja` reads `Gen Hoshino` given `Gen` (default title), strict `zh` `De Xu` given `De`, family `Xu` (default family `De Xu`). Every example the 2026-10-30 entry gives parses on the built code as it says. - The default does not move: tests/v2/test_snapshot.py passes unchanged, a plain choice compiling as before. +- **2026-11-01 #754, the `Lexicon` surface proposed (item 1's title warning decided by 2026-11-01 #754, the title warning; the rest decided as recommended by 2026-11-02 #754, the `Lexicon` surface) — `add` AND `remove` TAKE WORDS AND A READING KIND, `readings_of` ANSWERS QUERIES, THE CONSTRUCTOR TAKES ONLY THE LISTS AND MASKS, THE `Lexicon` RECORDS ITS CHOICE SO CHAINED CHOICES UNION, AND A 2.x PICKLE LOADS WHERE ITS SETS TRANSLATE BACK; PROPOSED, NOT DECIDED, AND NOTHING IS BUILT.** Proposes the API the 2026-10-28 #754 `Lexicon` surface entry left open (its method names) and the chained-choices entry's open option, for the maintainer to decide before sub-phase 4i starts. The arguments and the options weighed are the RFC's Phase 4 section "The `Lexicon` surface: configuring by reading"; this entry records what is proposed and what was measured. + 1. **Configuring.** `Lexicon.add(*words, kind, **properties)` and `Lexicon.remove(*words, kind=None)`, `kind` a vocabulary kind as the files write it (the five categories with a kind of their own name, `title`, `suffix`, `particle`, `conjunction` and `bound_given`, taking it on `add`; any category, `name` and `relation` included, matching every kind of it on `remove`), the properties the category's behavioral ones (`default` and `cased` refused), a property given its unwritten value removed, and `add` of a kind the word holds updating that reading in every file. Derived markers follow through `compile_readings` at construction. An `add` of a title reading, save one that only updates a held title reading's properties, warns where, in the compile under no choice, the word's `given` reading makes it a title only inside a phrase, naming `remove(w, kind="given")` (`add("dean", kind="title")`); no other edit warns for changing nothing at construction, since a later choice can make it matter (`add("do", kind="preposition")` under `en`). The `generation`, `credentials` and `patronymic` views read a listed word's kind from the lexicon's compile rather than the shipped index, rules/R8 and rules/R9 amended in 4i-b, so a kind a caller states reaches them. Weighed against a positional `add(word, kind)`, a reading object, and new method names beside an `add` kept to refuse. + 2. **Queries.** `Lexicon.readings_of(word)`, a tuple of a new public `LexiconReading` (text, kind, category, properties, languages) under the lexicon's choice; the repr counts readings by kind against the nearer named lexicon. Weighed against publishing `IndexReading` and against a predicate per category. + 3. **What stays public.** `surnames`, `given_names` and `capitalization_exceptions` as the constructor's only fields, `empty()`, `default()` and `|`; a frozen, slotted dataclass whose state is a base, an ordered log of the caller's edits, a choice, the lists and the masks, equal when all five are, the log compared as written, each edit stored frozen (structural, so equality survives a later choice; a base by reference or empty compared by name, loading nothing, a snapshot by its rows), the first three constructor parameters under underscore names so `dataclasses.replace` carries them, validated as `parser_for` validates a choice (a private route beside the 2026-10-31 #753, phase 3j entry's item 2, "No public `locales.choose`. A bare choice is a `Locale` with an empty lexicon and the tags", which binds the public spelling and so is not excepted); a compile in `parser_for` refusing what construction refuses, wrapped with the choice (raising recommended; letting the caller's reading win, dropping the clashing shipped reading, offered as the alternative); `|` concatenating the logs, later wins, with `parser_for` joining the packs' fragments in pack order, a later pack winning as in the policy fold, and the caller's lexicon after them, and refusing two different choices or non-empty bases; the `nameparser.config` constants kept as read-only data. + 4. **The choice.** Recorded on the `Lexicon` (read-only `Lexicon.languages` and `Lexicon.strict`, `Parser.languages` and `Parser.strict` reading through), set publicly only by `parser_for` (the underscore constructor parameters carrying it through `replace` and pickle); the default's base every row of the reading index, its readings off by default included, which a choice turns on, held by reference and loaded only when a lexicon needs its rows, so the default keeps the 2026-10-29 lazy load (an eager base offered as the alternative); chained choices union, a caller's edits apply after the choice in the order written, never baked into the base, a strict mismatch in a chained call raises, and `_apply_choice`'s delta retires. If decided it SUPERSEDES both items of the 2026-10-29 #753, phase 3b, chained choices entry, and under strict the 2026-10-31 #753, phase 3j entry's clause that a caller's listing of a dropped word is dropped by the delta. Weighed against recording nothing and against a `Parser` field. Whether `Reading.languages` follows the choice is left to an optional 4i-d. + 5. **Pickle.** A `Lexicon` pickles its five parts, its base as a snapshot of the rows; a 2.x `Lexicon` state is translated as the nearer named base, `default` or `empty`, and a log replacing each word whose 2.x memberships differ from that base's compile (removing it first only where the base holds it, so the 2.8 fixture's `dean` lexicon equals a fresh `Lexicon.empty().add("dean", kind="title")`), so a 2.8 pack fragment stays empty-based, and loads where the compile gives its sets back (a word in both `titles` and `titles_ambiguous`, which 2.x reads as a title, compared as in `titles` alone), else is refused naming the fields that differ. Whether `Lexicon`'s guard learns `ADDED_IN` needs no work: it has admitted the marker since 2026-10-30 (decisions/A1's #959 prerequisite entry, PR #960). + 6. **Migration.** The three sites a third-party caller meets (`add`/`remove` with a set-field keyword, `Lexicon(...)` and `dataclasses.replace` with one, an attribute read) each refuse with code that runs, translated for the words named, by one translation shared with the pickle load. + 7. **Work order.** 4i-a (the pipeline reads the sets through one internal object), 4i-b (the break, with the tests, docs and migrate.rst's table), 4i-c (the choice recorded), 4i-d optional (`Reading.languages` follows the choice), none moving a default parse. + - **Measured: the recipes.** On master `a3b1f3e1`, Python 3.11.17, 2026-11-01, a throwaway prototype compiling a lexicon's readings with `compile_readings` over `choose(index(), chosen)`, the whole index with its readings off by default, the caller's edits applied after the choice, folded into today's `Lexicon` through `_lexicon._vocab_fields`: eleven 2.x recipes and their proposed spellings parse every field alike (the RFC section's table: `Dean Smith`, `John Smith LEED AP`, `Professor Plum`, `Guru Prasad Sharma`, `Ram Bahadur DC`, `Van Johnson`, `ANA SILVA E SOUZA`, `Mohamad Ali Khan`, `Jane Smith formerly Jones`, `山田さん`, `Jane Doe Cha`); `add("dc", kind="given")` puts `dc` in the ambiguous acronyms; a configured lexicon compiles in 2.7 ms (mean of 20). The chained call `parser_for(EN, base=parser_for(PT))` reads `Maria do Carmo` family `Carmo` and `John Smith DO` family `DO` on the tree, and `do Carmo` and suffix `DO` with the choice recorded; `parser_for(EN, base=Parser(lexicon=Lexicon.empty().add(particles={"do", "van"})))` keeps `van` alone on the tree and both on the prototype. Recompute: build each pair of lexicons and compare `Parser(lexicon=...).parse(name)` role by role. + - **Measured: the 2.x pickle translation.** Each set field maps to the readings that compile it (`titles_ambiguous` a title and a `given` reading, `particles_ambiguous` a particle and a `given` reading, `suffix_acronyms_ambiguous` a credential and a `family` reading, `title_phrases` its first word's `attaches`, a title reading of a `modifier` or `style` word folded into that reading, and a word `titles` also holds left out of `titles_ambiguous`'s translation); `compile_readings` over the translation gives back every set of `Lexicon.default()`, of `Lexicon.default().add(titles={"dean", "chancellor"})` with its overlap compared as `titles` alone (the two spellings parse 48 names over both words in four cases and twelve shapes alike, every field and ambiguity), and of both lexicons in tests/v2/data/lexicon_v28.pickle (`Lexicon.empty().add(titles={"dean"})` and `locales.RU`'s empty fragment), and does not for `Lexicon.empty().add(particles={"xo"}, particles_ambiguous={"xo"}, suffix_acronyms={"xo"})`, whose translated `given` reading also marks the acronym ambiguous. A pickle of the default as readings, every index row with its tag grouped by word as nameparser/config/_index.py writes them (2,261 rows, 2,190 on by default), is 33,122 bytes against 27,204 for today's sets, protocol 4. The first draft of this entry counted the on rows alone as the default's base, which a review on 2026-11-01 showed would undo every pack that turns a reading on (`es`'s titles, `ne`'s gate on `guru`, `ga`'s clan particles, `cs`'s `ml` and `st`), and claimed a round trip the `dean` recipe failed until the overlap rule above. + - **Measured: the callers,** on `a3b1f3e1`, 2026-11-01. A configuring call is a call to `Lexicon`, `add`, `remove` or `replace` with a set field as a keyword, found by walking the syntax tree of each committed `.py` file (generated `nameparser/config/_*.py` left out) and by matching the code lines (`>>>`, `...`, indented) of each `.rst` and `.md` file; a read is an attribute load of a set field; `surnames` and `given_names` are left out of both, item 3 keeping them and `ParsedName` having properties of both names. Configuring calls: tests 204 in 26 files (107 `add`, 22 `remove`, 56 `Lexicon(...)`, 19 `replace`), user docs 29 in 4 (`customize.rst`, `locales.rst`, `migrate.rst`, and `release_log.rst`, history that stays), library 1 (`_default_lexicon`), docs/design 1, locale packs none. Reads: library 111 in 12 files (the parser's own, which 4i-a moves to the internal sets), tests 157 in 16, user docs 25 in 3, tools 17 in 4, 16 of them `Lexicon` reads in `tools/name_usage.py`, `tools/es_given_names.py` and `tools/differential/local_agreement.py` and one a counter in `tools/differential/local_corpora.py`. The first draft counted `surnames` in, giving tests 215 calls and 180 reads. `capitalization_exceptions` by keyword, which stays: 47 test calls and 1 library call. + - **Checked:** a frozen, slotted dataclass built with `init=False` and a hand-written `__init__` refuses `Lexicon(titles=...)` and `dataclasses.replace(lexicon, titles=...)` with its own message, `replace` calling that `__init__`, and `__getattr__` refuses `lexicon.titles` with its own (2026-11-01, a throwaway class). + - **Revised after an independent review on 2026-11-01** of head `c5263dc8`, each finding measured again here on `a3b1f3e1`: a 2.x state translated onto an empty base diverged from a fresh default under a later choice (`parser_for(PT, base=loaded)` read `John Smith DO` suffix `DO`, `parser_for(PT)` family `DO`), so it is translated as the default plus a log, and 2.8.0's pickled default, which differs from the tree's compile in 23 words, every one a suffix reading 3.0 added, round-trips that way exactly; when a removal applies was unstated, and the two orders differ (`remove("do", kind="credential")` chosen `en` reads `Maria do Carmo` middle `do` applied after the choice, as 2.x's delta reads it, and family `do Carmo` baked into the base); `dataclasses.replace` drops fields built `init=False`, so the base, log and choice are underscore constructor parameters, a hand-written `__init__` taking them shown to survive `replace` and still refuse `titles=`; the default holding its rows would have reversed the 3a lazy load (loading and building every row costs about 2 ms cached, 18 to 20 ms uncached, in this checkout); the views ignored a stated kind (on the tree, `add(suffix_words={"secundus"})` reads `Marcus Tullius Secundus` credentials `Secundus`, generation `''`); `add("dean", kind="title")` changed nothing in silence (0 of 45 shapes against 27 of 45 for 2.x `add(titles={"dean"})`, the review's grid); equality and `|` left a choice unstated; the strict half of the supersession was unmarked; and `name` and `relation` are categories with no kind of their name. A second review, of that revision, found its translation onto the default base would make a loaded 2.8 pack fragment (built from `Lexicon.empty()`) remove some 1,800 words through `|`, its no-op warning fire on edits a later choice makes matter, and its base equality unstated; the translation now takes the nearer named base, the warning is narrowed to a title made phrase-only by a `given` reading under no choice, and bases compare by name or, a snapshot, by rows. + - **Revised after the second independent review** (2026-11-01, of `c5263dc8..911d97a0`): normalizing the log made lexicons that parse differently equal (`sir` with `addresses` removed and then `attaches` added, against the second edit alone), so logs compare as written; validation was claimed eager though a later choice can turn on a reading an edit clashes with (`add("st", kind="credential")` under `cs`), so `parser_for` compiles again and refuses with the choice named; the underscore parameters reach a choice outside `parser_for`, so they are validated (first marked an exception to 3j, a wording the third review below reverses); the title warning fired on property updates (`prince`), so a property update of a held title reading is exempt; a paragraph on loading a 2.x pickle of a chosen parser described a pickle no 2.x release can write (choosing languages arrived in 3.0), and is replaced by that fact, and the first wording of the exemption dropped `add("dean", kind="office")`, which warns again (found by review of the revision); `bound_given` is a fifth category with a kind of its name; and `|` put a pack's edits after the caller's, so `parser_for` joins the fragment under the caller's lexicon. + - **Revised after the third independent review** (2026-11-01, of `911d97a0..f5b4e014`): the phrase quoted as 3j's, "a bare choice has no constructor", is the Standing line's restatement, not the 2026-10-31 #753, phase 3j entry's text, whose item 2 is now quoted; that item binds the public spelling of a bare choice, so the underscore parameters are a private route beside it and no exception to it, and item 4 says the choice is set publicly only by `parser_for`; `parser_for` joins the packs' fragments in pack order, a later pack winning as the policy fold has it, before the caller's lexicon; edits are stored frozen; and the clash at `parser_for` is offered as raise (recommended) or caller-wins. + - Not touched: no code, no test, no rule; `Lexicon`, `Parser` and `Locale` keep their 2.x shape until 4i-b. +- **2026-11-01 #754, the title warning (Derek) — AN `add` OF A TITLE READING THAT A `given` READING KEEPS A TITLE ONLY INSIDE A PHRASE WARNS, NAMING `remove(w, kind="given")`; DECIDED AHEAD OF THE REST OF THE `Lexicon` SURFACE, WHICH STAYS PROPOSED.** Decided on 2026-11-01, on item 1 of the 2026-11-01 #754, the `Lexicon` surface proposed entry. Under the configure-by-reading API, when it is built (4i-b), an `add` of a title reading warns at construction (`UserWarning`) where, in the compile under no choice, the word's `given` reading makes it a title only inside a phrase (rules/H9), the warning naming `remove(w, kind="given")`: the trap of the 1.4 recipe `add(titles={"dean"})`, whose literal translations `add("dean", kind="title")` and `add("dean", kind="office")` move no parse (0 of 45 shapes, against 27 of 45 for the 2.x spelling, measured by an independent review on 2026-11-01). It fires where the `add` creates a title reading or re-adds a held one with no property changed, and not where it only updates the properties of a title reading the word holds (`add("prince", kind="nobility", attaches=[...])`), the scope the review proposed with the decision. The reference is the compile under no choice, stated because a caller's edits apply after any later choice; nothing else of the proposal is decided by this entry. +- **2026-11-02 #754, the `Lexicon` surface (Derek) — THE CONFIGURE-BY-READING API IS DECIDED AS RECOMMENDED; CALLER-WINS AT `parser_for`, AN EAGER DEFAULT BASE AND REFUSING EVERY 2.x PICKLE ARE DECLINED; NOTHING IS BUILT.** Decides the 2026-11-01 #754, the `Lexicon` surface proposed entry, each item as it recommends; its arguments are the RFC's Phase 4 section "The `Lexicon` surface: configuring by reading", and its measurements that entry. SUPERSEDES: both items of the 2026-10-29 #753, phase 3b, chained choices entry, and the 2026-10-31 #753, phase 3j entry's built clause that a caller's listing of a dropped word is dropped by the delta (each effective when 4i-c lands). + 1. **Configuring.** `Lexicon.add(*words, kind, **properties)` and `Lexicon.remove(*words, kind=None)`, kinds and properties as the language files spell them; derived markers through `compile_readings`; the `generation`, `credentials` and `patronymic` views reading a listed word's kind from the lexicon, rules/R8 and rules/R9 amended when 4i-b builds it. The title warning was decided on 2026-11-01 (2026-11-01 #754, the title warning). + 2. **Queries.** `Lexicon.readings_of(word)` returns a public `LexiconReading`; the repr counts readings by kind against the nearer named lexicon. + 3. **State.** Five parts, a base, an ordered log of the caller's edits, the choice, the lists and the masks, compared structurally with the log as written and each edit stored frozen; the base, log and choice constructor parameters under underscore names, validated as `parser_for` validates a choice; `|` later-wins, `parser_for` joining the packs' fragments in pack order and the caller's lexicon last. An edit that clashes with a reading a later choice turns on raises at `parser_for`, naming the choice and the edit. + 4. **The choice.** The `Lexicon` records its languages and strictness, set publicly only by `parser_for`; chained choices union; the caller's edits apply after the choice, in the order written; the default's base stays lazy, keeping the 2026-10-29 lazy load (the 2026-10-29 #753, phase 3a entry). + 5. **Pickle.** Option (b): a `Lexicon` pickles its five parts, its base a snapshot; a 2.x state translates onto the nearer named base and a per-word log, loading where the compile gives its sets back and refused otherwise, naming the fields. + 6. **Migration.** The old spellings (`add`/`remove` with a set keyword, `Lexicon(...)` and `dataclasses.replace` with one, a set attribute read) fail loudly with a per-word translation, the one the pickle load uses. + 7. **Work order.** 4i-a, the pipeline reading the sets through one internal object; 4i-b, the break; 4i-c, the recorded choice; 4i-d, `Reading.languages` following the choice, optional; none moves a default parse. + - Declined: letting the caller's reading win where it clashes with a reading a later choice turns on (the clash is refused at construction, and one rule at both points is the one a caller can predict); an eager default base, holding the index rows in `Lexicon.default()` (it would reverse the 2026-10-29 lazy load for every caller, about 2 ms cached and 18 to 20 ms uncached on the first use); refusing every 2.x `Lexicon` state, pickle option (a) (the default and the 2.8 fixtures translate back exactly, and the commonest customization, `Lexicon.default().add(titles={"dean"})`, does with a word in both `titles` and `titles_ambiguous` compared as in `titles` alone, the two spellings parsing alike). The alternatives the proposal weighed for its other items, each recorded there with its reason, are declined with them: a positional `add(word, kind)`, a reading object, new method names, publishing `IndexReading`, a predicate per category, recording no choice, a `Parser` field for it, and pickling the base by reference. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 742d3759a..a59aebd23 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -27,7 +27,7 @@ Four phases, in a fixed order: | 1 | The per-word TOML format, compiled to today's sets | no | shipped in 2.6 (#737, #744); #749–#751 gave the words in `und.toml` their languages, leaving two particles of unrecorded origin | | 2 | Ambiguity derived from clashing readings; the ambiguity gate extended to every reading kind | yes, the first phase that does | #752, design agreed 2026-10-23; 2a, the two name-clash markers derived, landed without moving a parse (decisions/vocabulary-model, 2026-10-25) | | 3 | Language packs that prefer or restrict their language | yes, for callers who choose one | #753, for 3.0, needs 1 and 2; design agreed 2026-10-28, 3g's connective declined 2026-10-30; 3a, the reading index and the shared compiler, 3b, the choice on `Locale.languages`, and 3c, the pack-shaped exclusions as readings off by default, landed without moving a default parse (decisions/vocabulary-model, 2026-10-29); strict redefined on 2026-10-30 as file-strict, the chosen files and native-script vocabulary only, which needs no 3i, the definition measured on 2026-10-31 (decisions/vocabulary-model's 2026-10-30 #753, file-strict entry); 3j, file-strict, landed 2026-10-31 as `parser_for(..., strict=True)`, its API decided that day (decisions/vocabulary-model's 2026-10-31 #753, phase 3j entry) | -| 4 | Order and shape; the rules a pack can choose; the output model | yes | #754, 3.0 work; its output model agreed 2026-10-28, ahead of or alongside phase 3's mechanism; 4a to 4d, the reading and the views `generation`, `credentials`, `patronymic` and `surname2`, landed without moving a parse, 4e, the render defaults and `ParsedName.order`, moved the default `str()`, and 4f, `[rules]` compiled into a pack's policy patch, moved no parse (decisions/vocabulary-model, decisions/R6 and decisions/R8 to R10, 2026-10-29) | +| 4 | Order and shape; the rules a pack can choose; the output model | yes | #754, 3.0 work; its output model agreed 2026-10-28, ahead of or alongside phase 3's mechanism; 4a to 4d, the reading and the views `generation`, `credentials`, `patronymic` and `surname2`, landed without moving a parse, 4e, the render defaults and `ParsedName.order`, moved the default `str()`, and 4f, `[rules]` compiled into a pack's policy patch, moved no parse (decisions/vocabulary-model, decisions/R6 and decisions/R8 to R10, 2026-10-29); 4i, the `Lexicon` surface, decided 2026-11-02 and not yet built (decisions/vocabulary-model's 2026-11-02 #754, the `Lexicon` surface entry) | Two commitments hold through all four. **The default loads every language and keeps working for names of any language, as today**; choosing a language narrows it (decision 5 in decisions/vocabulary-model). And **languages own vocabulary and defaults, the core owns structure**: a pack configures rules the core implements, and never ships code (mechanisms.md#LOCALE-PACKS-PURE-DATA). @@ -306,11 +306,11 @@ Measured 2026-10-27 on `cfe8bf92`: the pool holds 2,140 readings over 1,797 word Two cheaper-looking forms do not work. Per-language deltas compiled at build time do not compose: under prefer `{pt, en}`, `do` keeps both its particle and its credential, but Portuguese's delta removes the credential and English's removes the particle. And precompiling every combination of languages is combinatorial. -A choice is applied as a delta against the default's compile, so it composes with a caller's own `Lexicon`: `parser_for(locales.NE, base=Parser(lexicon=custom))` keeps a title the caller added, and a `Locale`'s own `lexicon` fragment is still unioned on after the choice. The delta removes what the choice drops whoever put it in the base: as built in 3b, `parser_for(EN, base=Parser(lexicon=Lexicon.empty().add(particles={'do', 'van'})))` keeps only `van`, since `en` drops `pt`'s particle `do`, the caller's listing of it included. And a choice is one call's (decided 2026-10-29, decisions/vocabulary-model's chained-choices entry): a `Parser` records no choice, so `parser_for(EN, base=parser_for(PT))` applies `en`'s after `pt`'s, reading `Maria do Carmo` family `Carmo` and `John Smith DO` family `DO`, and `parser_for(PT, EN)` is how both are chosen, keeping both readings. Whether a `Parser` records its choice, so that chained choices union, is decided with the configure-by-reading method names. +A choice is applied as a delta against the default's compile, so it composes with a caller's own `Lexicon`: `parser_for(locales.NE, base=Parser(lexicon=custom))` keeps a title the caller added, and a `Locale`'s own `lexicon` fragment is still unioned on after the choice. The delta removes what the choice drops whoever put it in the base: as built in 3b, `parser_for(EN, base=Parser(lexicon=Lexicon.empty().add(particles={'do', 'van'})))` keeps only `van`, since `en` drops `pt`'s particle `do`, the caller's listing of it included. And a choice is one call's (decided 2026-10-29, decisions/vocabulary-model's chained-choices entry): a `Parser` records no choice, so `parser_for(EN, base=parser_for(PT))` applies `en`'s after `pt`'s, reading `Maria do Carmo` family `Carmo` and `John Smith DO` family `DO`, and `parser_for(PT, EN)` is how both are chosen, keeping both readings. Whether a `Parser` records its choice, so that chained choices union, is decided with the configure-by-reading method names. Decided on 2026-11-02, to be built in 4i-c: the `Lexicon` records it, so chained choices union and a caller's own readings apply after the choice (Phase 4's "The `Lexicon` surface: configuring by reading", item 4). ### The API and tags -This rested on `Locale` and `Policy` keeping their 2.x shape after the facade's removal, and the 2026-10-28 3.0 freeze answers it: both keep it, with additions. A choice is carried by `Locale`, which gains a `languages` field, a set of BCP 47 tags. `parser_for` unions the `languages` of the packs it is given, so the choice is order-independent while their policy patches still fold in order. Strict is a property of the whole choice, not of one pack, and an argument: `parser_for(locales.NE, locales.HI, strict=True)` (file-strict, its API in "Strict: the chosen files and nothing else" above). The shipped language packs are `Locale` constants like the four today (`locales.NE`, `locales.get("ne")`), each with an empty lexicon fragment (what a fragment holds once 3.0 configures vocabulary by reading is decisions/vocabulary-model's the 2026-10-28 #754 entry on the `Lexicon` surface); a code keeps `Locale`'s `[a-z0-9_]+`, so `pt-BR` is `locales.PT_BR`. +This rested on `Locale` and `Policy` keeping their 2.x shape after the facade's removal, and the 2026-10-28 3.0 freeze answers it: both keep it, with additions. A choice is carried by `Locale`, which gains a `languages` field, a set of BCP 47 tags. `parser_for` unions the `languages` of the packs it is given, so the choice is order-independent while their policy patches still fold in order. Strict is a property of the whole choice, not of one pack, and an argument: `parser_for(locales.NE, locales.HI, strict=True)` (file-strict, its API in "Strict: the chosen files and nothing else" above). The shipped language packs are `Locale` constants like the four today (`locales.NE`, `locales.get("ne")`), each with an empty lexicon fragment (what a fragment holds once 3.0 configures vocabulary by reading is decisions/vocabulary-model's the 2026-10-28 #754 entry on the `Lexicon` surface; decided on 2026-11-02, the lists outside the per-word model and the case masks only, which is all a shipped fragment holds already); a code keeps `Locale`'s `[a-z0-9_]+`, so `pt-BR` is `locales.PT_BR`. A chosen tag selects its own file and its ancestors', never its descendants': `pt-BR` selects `pt-BR.toml` and `pt.toml`, and `pt` selects only `pt.toml`, so a caller in Portugal does not turn on the Brazilian agnomens (#843 left the single-letter `F` and `N` to a `pt-BR` pack). No macrolanguage table is read: Malay and Indonesian are separate tags, though BCP 47 counts Indonesian within the Malay macrolanguage, and a caller with names in both chooses both. The Czech, Slovak, Slovenian and Croatian files each hold the words they share, so each language is a pack of its own. Script subtags need no matching rule yet, since no file carries one. @@ -461,6 +461,109 @@ How a call reaches a pack's pattern is open, and each answer costs something: `P **Gender**, recorded only where a word gives a definitive clue (decision 9), is not part of this design: the vocabulary records it on readings (`bint`'s `gender = "f"`), and `Token.reading` could carry it later as one more addition. +### The `Lexicon` surface: configuring by reading + +Status: **decided 2026-11-02 as recommended below (decisions/vocabulary-model's 2026-11-02 #754, the `Lexicon` surface entry), item 1's title warning on 2026-11-01 (decisions/vocabulary-model's 2026-11-01 #754, the title warning entry); nothing is built.** What is decided is the direction (2026-10-28): a caller configures vocabulary by reading, `Lexicon` derives the sets and gates the build derives, and the compiled sets become internal (on 2026-11-01, 20 of `Lexicon`'s 23 public fields: the 19 of `_lexicon._COMPILED_FIELDS` and the derived `particles_ambiguous`; item 3 keeps the other three), with no 2.x warning (2026-10-29). This section proposes the API that direction leaves open, each question with the options weighed and one recommendation; decisions/vocabulary-model's 2026-11-01 #754, the `Lexicon` surface proposed entry records the proposal and its measurements, and decisions/vocabulary-model's 2026-11-02 #754, the `Lexicon` surface entry the decision, each recommendation accepted and the alternatives below declined. Where this section says "proposed" or "recommended", read "decided"; the wording is kept as the record of what was weighed. Every parse below was run on master `a3b1f3e1` (Python 3.11.17) through a throwaway prototype that compiles a lexicon's readings with the shipped `compile_readings` over `choose(index(), chosen)`, the whole index with its readings off by default, as item 4 has the default hold them, the caller's edits applied after the choice, and folds the result into today's `Lexicon`, so the real parser reads it; the prototype is not committed. + +**1. Configuring: `add` and `remove` take words and a reading kind.** Proposed: + +```python +Lexicon.add(*words: str, kind: str, **properties) -> Lexicon +Lexicon.remove(*words: str, kind: str | None = None) -> Lexicon +``` + +- `kind` is a vocabulary kind as the language files write it, a key of `_reading_index.CATEGORY` (`title`, `academic`, `salutation`, `credential`, `generation`, `preposition`, `conjunction`, `bound_given`, `birth_marker`, `given`, `family`, ...), keyword-only and required on `add`. Five categories have a kind of their own name (`title`, `suffix`, `particle`, `conjunction`, `bound_given`), so `add(..., kind="title")` adds a title not yet classified, as `title = {}` does in a file. On `remove` the argument may also be a category, a value of `CATEGORY`, matching every kind of it: `remove("dean", kind="title")` drops dean's `office` reading, `kind="name"` its `given` and `family` readings, `kind="relation"` a birth marker. `name` and `relation` are categories with no kind of their name, so `add` refuses them, naming the kinds. `remove` with no kind drops every reading of the word, which then leaves the vocabulary. +- `properties` are the behavioral properties of that kind's category, spelled as the files spell them (`addresses`, `after_given`, `attaches`, `acronym`, `attached`, `may_be_initial`, and `same_as` on a `given` reading). A property given its unwritten value (`False`, `None`) is removed. `default` is refused (a caller's reading is on: it is the caller's choice), and so is `cased`, a case mask belonging to `capitalization_exceptions` (item 3). +- `add` of a kind the word already holds updates that reading's properties in every file that holds it, so `add("e", kind="conjunction", may_be_initial=False)` settles `e`'s one-case fork for Italian and Portuguese alike; otherwise it adds the caller's own reading, which names no file. +- **Decided 2026-11-01:** an `add` of a title reading warns (`UserWarning`, at construction) unless it only updates the properties of a title reading the word holds, when, in the compile under no choice, the word's `given` reading makes it a title only inside a phrase (rules/H9), naming `remove(w, kind="given")`. That is the 1.4 recipe's trap: `add("dean", kind="title")` and `add("dean", kind="office")` read 0 of 45 shapes differently from the default, where 2.x `add(titles={"dean"})` reads 27 of 45 differently (measured by an independent review of this proposal (2026-11-01)). The reference is stated because edits apply after a later choice (item 4), which can make an edit matter that changes nothing at construction: a strict choice of a file not holding `dean`'s `given` reading makes the same `add` a title everywhere, and `add("do", kind="preposition")`, which changes no compile under no choice, keeps `do` a particle under `en`, where plain `en` reads `Maria do Carmo` middle `do`. So, the title case below aside, no other edit warns for changing nothing at construction, nor does a `remove` of a reading not held (`remove("st", kind="generation")` matters under `cs`, which turns that reading on); the first draft of this bullet warned on both, and review found the two cases above. A warning, not a raise, by root AGENTS.md's construction-time `UserWarning`: a caller who wants the title gated is served, and filters it. It fires where the `add` creates a title reading (`add("dean", kind="title")`, a kind `dean` lacks) and where it re-adds one the word holds with no property changed (`add("dean", kind="office")`, the same trap spelled with the word's own kind); a property update of a held title reading is silent (`add("prince", kind="nobility", attaches=[...])`), where the warning's fix would make `prince` a title everywhere, which that caller never asked for (the property-update exemption from the second independent review (2026-11-01), and the re-add kept in by review of that fix). +- **The views read the lexicon's kinds.** The `generation` and `credentials` views (rules/R8) and `patronymic` (rules/R9) read a listed word's kind from the shipped index, not from the `Lexicon`, which until now could not state one: on the tree `Lexicon.default().add(suffix_words={"secundus"})` reads `Marcus Tullius Secundus` generation `''`, credentials `Secundus` (measured 2026-11-01). Once a caller states the kind, a view ignoring it is wrong in silence: on the first draft of this proposal `add("secundus", kind="generation")` still read credentials `Secundus`, and removing `bin`'s patronymic reading still read patronymic `bin Mohamad` (an independent review of this proposal (2026-11-01)). Proposed: the compile also gives each listed suffix word's kind and the patronymic particles, as internal sets the views read in place of the index, and 4i-b amends rules/R8 and rules/R9 to say the kind is the lexicon's. On the default it is the index's by the compile invariant, so no default parse moves. Weighed against stating it as a limit: the kind is the one thing a reading says that a set could not. +- Validation runs at every compile: at construction, under the lexicon's own choice, and again in `parser_for` under the choice it records, since the caller's edits apply after a choice and a choice can turn on a reading an edit then clashes with. What the build refuses in a file is refused at either point, raised with the fix; at `parser_for` the error is wrapped with the choice (rules/D2) and names the edit. `Lexicon.default().add("st", kind="credential")` compiles, and `parser_for(locales.CS, base=Parser(lexicon=that))` raises, `cs` turning on `st`'s generation reading beside the caller's credential, and names `remove("st", kind="generation")` written before the add (found by the second independent review (2026-11-01)). Two ways to meet that clash were weighed, and (a) was decided on 2026-11-02: (a) raise, since construction refuses the same clash and one rule at both points is the one a caller can predict; (b), declined, let the caller's reading win, dropping the shipped suffix reading of another kind it clashes with, which is item 4's caller-outranks-a-pack rule carried into the compile. Three of the build's checks reach a caller: a word holding two suffix kinds (check 4; `add("dc", kind="generation")` on `dc`'s credential raises, naming `remove("dc", kind="credential")`), a phrase written as any reading but a birth marker or a title addressing the given name (check 8), and a bound given name that is also a particle with no `given` reading (check 9). The 2.x construction guards dissolve: the four subset checks of `_SUBSET_FIELDS` and the two gate-bypass checks guard relations between sets that a compile from readings holds by construction, a marker existing only beside its base reading. + +Derived markers follow by the shared compiler, at construction: `Lexicon.default().add("dc", kind="given")` compiles `dc` into the ambiguous acronyms, as the build would for a `given` reading written in a file (prototype, 2026-11-01). A configured lexicon pays one compile, 2.7 ms on the prototype (mean of 20); `Lexicon.default()` pays none, its sets being `nameparser/config/_compiled.py`, which tests/v2/test_reading_index.py already holds equal to the compile of no choice. + +Each 2.x recipe below parses the same through the proposed call (prototype, 2026-11-01; recompute by building both lexicons and comparing `Parser(lexicon=...).parse(name)` field by field): + +| Name | 2.x | 3.0, proposed | Reads | +|---|---|---|---| +| `Dean Smith` | `add(titles={"dean"})` | `remove("dean", kind="given")` | title `Dean` | +| `John Smith LEED AP` | `add(suffix_acronyms={"leed", "ap"})` | `add("leed", "ap", kind="credential")` | suffix `LEED AP` | +| `Professor Plum` | `remove(titles={"professor"})` | `remove("professor", kind="title")` | given `Professor` | +| `Guru Prasad Sharma` | `remove(titles={"guru"}).add(titles_ambiguous={"guru"})` | `add("guru", kind="given")` | given `Guru` | +| `Ram Bahadur DC` | `remove(suffix_acronyms={"dc"})` | `remove("dc", kind="credential")` | family `DC` | +| `Van Johnson` | `remove(particles_ambiguous={"van"})` | `remove("van", kind="given")` | family `Van Johnson` | +| `ANA SILVA E SOUZA` | `remove(conjunctions_ambiguous={"e"})` | `add("e", kind="conjunction", may_be_initial=False)` | family `SILVA E SOUZA` | +| `Mohamad Ali Khan` | `add(bound_given_names={"mohamad"})` | `add("mohamad", kind="bound_given")` | given `Mohamad Ali` | +| `Jane Smith formerly Jones` | `add(maiden_markers={"formerly"})` | `add("formerly", kind="birth_marker")` | maiden `Jones` | +| `山田さん` | `remove(honorific_tails={"さん"})` | `add("さん", kind="honorific", attached=None)` | family `山田さん` | +| `Jane Doe Cha` | `add(suffix_acronyms={"cha"})` | `add("cha", kind="credential")` | suffix `Cha` | + +The first row is the point of the change. The 1.4 recipe `add(titles={"dean"})` meant "a title everywhere"; since rules/H9 it is spelled in sets as membership of `titles` beside `titles_ambiguous`, which no caller can be expected to know. In readings it says why: `dean` is a title already, and its `given` reading is what keeps it one only inside a phrase. + +Options weighed for the signature: + +- (a) **kind keyword-only, words variadic**, recommended: one call adds several words of one kind, a bare string is one word, and a misplaced kind cannot pass for a word. +- (b) `add(word, kind)` positional: the commonest call is one word, but two strings in either order both type-check, and several words need a loop or a str-or-iterable argument, the shape the validation conventions refuse. +- (c) a reading object, `add(LexiconReading("dean", "title"))`: one more public type to construct for every call, and `Reading`, the type that exists, is the parse's (`ReadingKind`, coarser than a vocabulary kind: every title kind reads `TITLE`), so it cannot carry `academic` or `preposition`. +- (d) new names (`with_reading`, `without_reading`) beside an `add` kept only to raise: the method a 2.x caller reaches for would exist to refuse. Reusing `add` and `remove` lets the new signature catch the old spelling exactly, every 2.x call being keywords named for a set field (item 6). + +What the reading terms cannot say that the sets could, found by translating every set: a caller's multi-word `given_name_titles` key (`"sir and dame"`), a phrase matched against a run of titles, has no reading; the shipped vocabulary holds none (a multi-word entry warns in 2.x). Recommended: drop it, a run being matched by its last word since #489. A single title phrase is removed through its first word's `attaches`, which may name title words as well as kinds (`add("chief", kind="nobility", attaches=["military"])` keeps `Chief Sergeant` and drops `Chief Justice`, with every other office phrase `chief` makes: prototype, 2026-11-01), coarser than `remove(title_phrases={"chief justice"})` but the reading the phrase is made from; naming the kept office words in `attaches` beside `military` restores them. + +**2. Queries.** Proposed: `Lexicon.readings_of(word) -> tuple[LexiconReading, ...]`, the readings this lexicon holds for the word under its choice (item 4), one per kind and properties, each with `languages`, the files holding it, sorted, empty for a caller's own reading; `LexiconReading(text, kind, category, properties, languages)` is a new frozen, slotted, picklable type. "Is X a title?" is `any(r.category == "title" for r in lexicon.readings_of(x))`; "does the parse read X as a title here?" stays the parse's question (`Token.reading`), since a title reading beside a `given` one is a title only inside a phrase. Weighed against: publishing `_reading_index.IndexReading`, which describes one file's row of the shipped index and does not hash or pickle (its `props` is a read-only view); and per-kind predicates (`is_title(word)`), one method per category, where the category is one attribute of a reading. A flag saying a reading is gated by a name reading (`dean`'s title, `dc`'s credential after item 1's call) is left as a later addition: the derived marker is the compile's business, and a caller who wants to know asks the parse. + +The repr counts readings by kind against the nearer of the two named lexicons, never naming words, as today's counts entries by field: `Lexicon(default + given: -1)` for the `dean` row above, `Lexicon(empty + credential: +2; surnames: +1)`, and the choice where there is one (`; languages en pt`). It stays bounded, the kinds being a closed set (33 keys of `CATEGORY` on 2026-11-01) and the list fields three. + +**3. What stays public on `Lexicon`.** Proposed: `Lexicon` stays a frozen, slotted dataclass, immutable and hashable. Its state is five parts: a base (item 4), the ordered log of the caller's edits, the choice it was compiled under, the lists and the masks. A base is one of three: the running release's index by reference (`default`), none (`empty`), or a snapshot of rows that a pickle from another release carries. Two by-reference or empty bases compare by name, loading nothing, so `==` and `hash` on a default-based lexicon stay off the index; a snapshot compares by its rows, and a loaded snapshot equal to the running release's rows becomes the by-reference base at load (item 5), so a pickle round-trip within a release is equal. Two lexicons are equal when all five parts are, the log compared as written, edit by edit, with no normalization (each edit stored frozen, its words normalized and its properties a sorted tuple of pairs, a list value such as `attaches` a sorted tuple, as the index writes a row, so the log hashes and the `Lexicon` stays immutable), so `parser_for(PT).lexicon != parser_for(EN).lexicon` and their parsers hash apart. Equality is structural rather than "the same compiled readings" because choices chain: two lexicons whose readings agree under one choice can disagree under the next, and an equality a later `parser_for` could break is none. Its accepted cost: two logs that compile alike can be unequal, a log that undoes itself (`add(w, kind="title")`, then `remove(w, kind="title")`) among them. A normalization was drafted, a later edit of one word and kind replacing an earlier one, and dropped: an `add` of a kind the word holds merges properties, and a `remove` before an `add` is not the `add` alone, so it made lexicons that parse differently equal (`add("sir", kind="nobility", addresses=None)` then `add("sir", kind="nobility", attaches=["military"])` against the second edit alone: `Sir Smith` reads family `Smith` under the first and given `Smith` under the second; the second independent review (2026-11-01)). The base, the log and the choice are constructor parameters under four underscore names (`_base`, `_edits`, `_languages`, `_strict`), which are not public API, so that `dataclasses.replace(lexicon, surnames=...)` (and `copy.replace` on 3.13 and later) carries them across. They are validated as `parser_for` validates a choice (each tag selecting a file, strict refused with no language), since `Lexicon(_languages=...)` and `replace(lexicon, _languages=...)` reach them; that is a private route to a choice outside `parser_for`, a second route beside the 2026-10-31 #753, phase 3j entry's item 2 ("No public `locales.choose`. A bare choice is a `Locale` with an empty lexicon and the tags, the spelling docs/locales.rst teaches."), which the file's Standing line restates as "a bare choice has no constructor, no `locales.choose`". That decision is about the public spelling of a bare choice, and the underscore parameters are no public spelling, so this is no exception to it, but it is a route the maintainer should know of; dropping it would need another way for `replace` and pickle to carry the choice (raised by the second independent review, 2026-11-01): a field built `init=False` is not copied by `replace`, which would return the lexicon with no readings and no choice (found by an independent review of this proposal (2026-11-01) on a 3.11 frozen, slotted class; a hand-written `__init__` taking the underscore parameters was checked the same day to survive `replace` and still refuse `titles=`). Its public parameters are only what is outside the per-word model, which every shipped pack's fragment already uses alone (`locales.ZH` its `surnames`, `locales.ES` its `given_names`, every other pack `Lexicon.empty()`): + +- `surnames` and `given_names`, lists outside the per-word model (decisions/vocabulary-model's Standing line on name readings), stay fields, changed through `|` with a fragment (`Lexicon.default() | Lexicon(surnames={"김민"})`, the form the packs use) or `dataclasses.replace`. Writing them as `family` and `given` readings was weighed and set aside: a name reading gates vocabulary, while these lists divide unspaced hangul and pair Spanish given names, and mixing the two would make every surname a gate. +- `capitalization_exceptions` stays the pair field it is, with `capitalization_exceptions_map`: a case mask repairs the rendering and is no reading clash, so it is configured as today, and `cased` is refused on `add`. The default's masks are still the compile of the files' `cased`. +- `Lexicon.empty()`, `Lexicon.default()` and `|` stay. `a | b` unions the lists, takes the masks right-biased as today, concatenates the logs, `a`'s first, and takes the base that is not empty and the choice of the side that has one; two different non-empty bases, or two different choices or strictnesses, raise `ValueError`, naming `parser_for` as the way to combine choices. Later wins, as for the masks, so `b`'s edits apply after `a`'s. `parser_for` joins the packs' fragments in the order the packs are given, a later pack's edits and masks after an earlier one's, as their policy patches fold with the later winning, and the caller's lexicon after them all: the base is the caller's, and its log is the fragments' logs in pack order followed by the caller's, so a caller's own statement outranks every pack's (item 4) and a later pack outranks an earlier one. Today's `lexicon | loc.lexicon` puts each fragment after the caller's lexicon, harmless while every fragment is empty or holds lists only (ZH's `surnames`, ES's `given_names`), and 4i-c changes it. A fragment records no choice, so the join keeps the base's. + +The `nameparser.config` constants stay public and frozen: they are the default's compile, read-only, and a reader of them configures nothing. Hiding them is a separate break with no gain this release; their docs gain a pointer to `readings_of`, and `Lexicon`'s field docs, which name them, go with the fields. + +**4. The choice: recorded on the `Lexicon`, so chained choices union.** Proposed: a `Lexicon` records the languages and strictness it was compiled under, read-only (`Lexicon.languages`, `Lexicon.strict`, with `Parser.languages` and `Parser.strict` reading through to them), set publicly only by `parser_for`, the underscore constructor parameters (item 3) carrying it through `replace` and pickle, so a bare choice keeps the one public spelling 3j decided, a `Locale` with an empty lexicon and the tags (decisions/vocabulary-model's 2026-10-31 #753, phase 3j entry, item 2). A lexicon is compiled from the shipped readings its base holds, the choice applied by `choose`, and the caller's own readings and removals applied after it, in the order the caller wrote them: the lexicon keeps its base and that ordered log apart, and never bakes an edit into the base. The difference shows under a choice: `Lexicon.default().remove("do", kind="credential")` chosen `en` reads `Maria do Carmo` middle `do`, the removal applied after `en` preferred its credential over `pt`'s particle, which is how 2.x's delta reads it on the tree; baked into the base it would read family `do Carmo`, `en` then holding no reading of `do` to prefer (measured 2026-11-01, raised by an independent review of this proposal (2026-11-01)). The default's base is every row of the reading index, the 71 written `default = false` among them (2,261 rows, 2,190 on by default, measured 2026-11-01), since a choice turns those on: a base of the on rows alone would silently undo every pack that turns a reading on, `es`'s titles, `ne`'s gate on `guru`, `ga`'s clan particles and `cs`'s `ml` and `st` among them (found by review on 2026-11-01, `compile_readings(choose(on_rows, {tag}))` against `choose(index(), {tag})`). That base is held by reference and loaded lazily: `Lexicon.default()` keeps the 2026-10-29 decision that the index stays out of it (decisions/vocabulary-model's 2026-10-29 #753, phase 3a entry), its sets read from `_compiled.py` as today, and the rows load only when a lexicon needs them, at its first edit, choice, `readings_of` or pickle, so `parse("John Smith")` on the default still never loads the index. What a lexicon that needs them pays: loading the index and building every row, about 2 ms with bytecode cached and 18 to 20 ms without (measured 2026-11-01 in this checkout). Weighed against holding the rows in the default eagerly, which is simpler but supersedes the decided lazy load for every caller: declined 2026-11-02, the base staying lazy. So: + +- `parser_for(EN, base=parser_for(PT))` chooses `pt` and `en` together and reads as `parser_for(PT, EN)` does: `Maria do Carmo` family `do Carmo`, `John Smith DO` suffix `DO`, where today the chained call reads family `Carmo` and family `DO` (both measured 2026-11-01, the proposed reading on the prototype). SUPERSEDES, if decided, item 1 of the 2026-10-29 chained-choices entry. +- A caller's reading survives a choice, whichever was written first: `parser_for(EN, base=Parser(lexicon=Lexicon.empty().add("do", "van", kind="preposition")))` keeps both particles, where today the delta drops `do`. SUPERSEDES, if decided, that entry's item 2, and under strict the 2026-10-31 #753, phase 3j entry's "a caller's listing of a dropped word dropped by the delta": a base adding `dr` and `zzz` keeps `zzz` alone under strict `vi` today and both under this proposal. Keeping 3j's behavior for strict alone was weighed and not proposed, since it would make the order of writing matter again for one flag. With the choice recorded there is no order of writing left to appeal to, and a caller's own statement about a word outranking a pack's preference extends to a caller's reading what the 2026-10-28 `Lexicon` surface entry says of a caller's removal, that decision 15 binds the default and the packs, not a caller's data. +- A base built from `Lexicon.empty()` holds no shipped reading, so a choice selects nothing in it; today the delta joins the chosen files' `default = false` readings even there. Strict is the spelling for "only these languages' vocabulary". +- A chained call whose `strict` differs from the base's recorded one raises `ValueError`, offering both spellings; raising now keeps a later rule for mixing them an addition. +- `_parser._apply_choice` and `_choice_delta` retire: the choice is no longer a delta against the default's compile but a filter over the base's own readings. + +Weighed against: (a) recording nothing, as decided on 2026-10-29 for a lexicon of sets, which could not recompile from what it held, so a choice could only be a delta; and (b) a `Parser.languages` field, the 3b session's recommendation, which leaves `parser.lexicon` holding the compiled choice beside a record of it, so the record and what it describes could disagree once a caller replaces the lexicon. The objection to a `Lexicon` field, that its own pickle guard refused an added field, was removed on 2026-10-30 when the guard learned `ADDED_IN` (#960, decisions/A1). Whether `Reading.languages` then follows the choice is left to its own sub-phase (4i-d): the parse can know its choice once the lexicon records it, which answers the 3b entry's reason for not following it, but its memo is keyed by the word alone today and would have to take the lexicon's readings into its key. + +**5. Pickle.** Whether `Lexicon`'s guard learns `ADDED_IN` needs no work here: it has admitted the marker since 2026-10-30 (decisions/A1's #959 prerequisite entry, PR #960). Proposed: a `Lexicon` pickles its five parts, its base as a snapshot of the rows, as today's pickle holds its sets whole, with its ordered log, choice, lists and masks; at load a snapshot equal to the running release's index becomes the by-reference base again (item 3's base equality), and one from another release stays a snapshot, parsing as it did where it was pickled; a pickle of the default holds its base, all 2,261 index rows with their tags, grouped by word as nameparser/config/_index.py writes them (33,122 bytes against today's 27,204 for the default's sets, protocol 4, measured 2026-11-01). Weighed against pickling the base by reference with the caller's edits, which is smaller but makes a pickled `Parser()` load the loading release's vocabulary in silence, the drift decisions/A1's 2026-10-30 entry asks to be avoided or recorded. + +A 2.8 pickle holds the compiled sets, which cannot be mapped back to the files they came from. Two options: (a) refuse every 2.x `Lexicon` state, naming the fix (rebuild the lexicon in 3.0); (b) translate the sets as the nearer of the two named bases and a log, nearer as the repr counts it (the fewest entries differing): each word whose 2.x memberships differ from that base's compile is replaced in the log (`remove(word)` where the base holds the word, then the readings that compile its memberships, a word the base does not hold taking the adds alone), so that the 2.8 fixture's `Lexicon.empty().add(titles={"dean"})` loads with the one-entry log `[add("dean", kind="title")]` and equals a fresh `Lexicon.empty().add("dean", kind="title")`, which tests/v2/test_lexicon.py's 2.8.0 pickle test asserts in its new spelling; a loaded lexicon reads the shipped readings, a later choice and strict included, everywhere but on those words. A 2.x lexicon built from `Lexicon.empty()`, every 2.8 pack fragment among them, so stays empty-based, and `parser_for`'s `lexicon | loc.lexicon` keeps the base's vocabulary as today; translated onto the default base, its log would remove some 1,800 words and `|` would carry the removals into every parser built with the pack, while the default's readings off by default, in no 2.x set and so never in the log, would be left for a later choice to turn on (both found by review of the revision). Translated onto an empty base whatever it was built from, as first drafted, a loaded default diverged from a fresh one under a later choice (`parser_for(PT, base=loaded)` read `John Smith DO` suffix `DO` where `parser_for(PT)` reads family `DO`), strict was inert on it, and it equaled no default (an independent review of this proposal (2026-11-01)). Each set maps to the readings that compile it (`titles_ambiguous` to a title and a `given` reading, save a word `titles` also holds, which 2.x reads as a title wherever it stands and which is translated as a title alone; `particles_ambiguous` to a particle and a `given` reading; `title_phrases` to its first word's `attaches`), load where the compile gives the pickled sets back, a word in both `titles` and `titles_ambiguous` compared as in `titles` alone, and refuse otherwise, naming the fields that differ. Decided 2026-11-02: (b); (a) declined. The overlap is the commonest 2.x customization, `Lexicon.default().add(titles={"dean"})`, docs/customize.rst's first example, and the compile cannot give it back as two sets, since it keeps `titles` and `titles_ambiguous` apart by construction; the two spellings parse alike (48 names over `dean` and `chancellor` in four cases and twelve shapes, every field and ambiguity, measured 2026-11-01), so comparing it as the one set loses nothing. Measured 2026-11-01: `Lexicon.default()` pickled by the released 2.8.0 wheel differs from the tree's default compile in 23 words, every one a suffix reading 3.0 added (20 to the acronyms, 3 to the words), and loads as the default with those 23 words replaced, the compile giving its sets back exactly, so it parses as it did under 2.8 and is unequal to the 3.0 default. `Lexicon.default()`'s sets round-trip, so does `Lexicon.default().add(titles={"dean", "chancellor"})` under that comparison, and so do both lexicons in tests/v2/data/lexicon_v28.pickle (`Lexicon.empty().add(titles={"dean"})` and `locales.RU`'s empty fragment); a lexicon whose `particles_ambiguous` names a word its `suffix_acronyms` also holds does not, the translated `given` reading also marking the acronym ambiguous, and is refused. A 2.8 pickle that stops loading is one holding a `Lexicon` that does not round-trip: `Lexicon` itself, `Parser` and `Locale`, the three public types holding one; `ParsedName`, `Token`, `Reading`, `Ambiguity` and `Policy` hold none and are untouched. No 2.x pickle carries a choice, since choosing languages arrived in 3.0 (phase 3b), so the translation has none to recover; a 3.0 pickle carries its choice whole. A word the translation replaced reads as the caller's own readings, naming no file in `readings_of`, which is what its pickle knows; every other word keeps its files. + +**6. Migration.** Measured 2026-11-01 on `a3b1f3e1`: a call that configures a set field is a call to `Lexicon`, `add`, `remove` or `replace` with one as a keyword, found by walking each committed Python file's syntax tree, and by matching code lines (`>>>`, `...` and indented ones) in the `.rst` and `.md` files; a read is an attribute load of one. `surnames` and `given_names` are left out of both: item 3 keeps them, and `ParsedName` has properties of both names: + +| Where | Configuring calls | Reads | +|---|---|---| +| library, outside the packs | 1 (`_default_lexicon`) | 111 in 12 files, the parser's own, which move to the internal sets | +| locale packs | none (`locales.ZH`'s `surnames` and `locales.ES`'s `given_names` are fields that stay) | none | +| tools | none | 17 in 4 files, 16 of them `Lexicon` reads in `tools/name_usage.py`, `tools/es_given_names.py` and `tools/differential/local_agreement.py`; the other, in `tools/differential/local_corpora.py`, reads a counter named `titles` | +| tests | 204 in 26 files (107 `add`, 22 `remove`, 56 `Lexicon(...)`, 19 `replace`) | 157 in 16 files | +| user docs | 29 in 4 files (`customize.rst`, `locales.rst`, `migrate.rst`, and `release_log.rst`, whose entries are history and stay) | 25 in 3 files | +| docs/design | 1 | 1 | + +`capitalization_exceptions` is passed by keyword in 47 more test calls and 1 library call; it stays a field, so they need no change. + +A third-party caller meets the change at one of three sites, each refusing with code that runs, built for the words the caller named: + +- `add(...)` or `remove(...)` with a set field as a keyword raises `TypeError` translating each word: `Lexicon.default().add(titles={"dean"})` names `.remove('dean', kind='given')`, since `dean` holds a title reading already; `add(titles={"provost"})` names `.add('provost', kind='title')`. The translation is the one the 2.x pickle load uses (item 5), so it is written once. +- `Lexicon(titles=...)` and `dataclasses.replace(lexicon, titles=...)` raise `TypeError` from a hand-written `__init__` (the class decorated `init=False`, its fields still init fields, so `replace` passes them all to it, item 3), with the same translation. +- Reading `lexicon.titles` raises `AttributeError` from `__getattr__`, naming `readings_of` and, for the parser's view, `Token.reading`. + +All three were checked on a frozen, slotted throwaway dataclass on 2026-11-01. The `docs/migrate.rst` table, one row per 2.x field, is the table under item 1 widened to every field: `titles` → `add(w, kind="title")` (and `remove(w, kind="given")` where the word may be a given name); `given_name_titles` → `add(w, kind="title", addresses="given")`; `after_given_titles` → `add(w, kind="salutation", after_given=True)`; `titles_ambiguous` → `add(w, kind="title")` and `add(w, kind="given")`; `title_modifiers` → `kind="modifier"`; `title_styles` → `kind="style"`; `title_phrases` → `attaches` on the first word; `suffix_acronyms` → `kind="credential"`; `suffix_acronyms_ambiguous` → a credential and a `family` reading; `suffix_words` → `kind="suffix"` (or `generation`); `suffix_words_ambiguous` → a `generation` and a `family` reading; `agnomens` → `kind="agnomen"` and a `family` reading; `honorific_tails` → `attached="after"` on the suffix reading; `particles` → `kind="particle"`; `particles_ambiguous` → a particle and a `given` reading, so `remove(particles_ambiguous=...)` is `remove(w, kind="given")`; `conjunctions` → `kind="conjunction"`; `conjunctions_ambiguous` → `may_be_initial=True`; `bound_given_names` → `kind="bound_given"`; `given_name_abbreviations` → `add(w, kind="given", same_as="")`; `maiden_markers` → `kind="birth_marker"`, a phrase whole (`"z domu"`); `surnames`, `given_names` and `capitalization_exceptions` unchanged. The 1.4 lineage (`c.titles.add("dean")`, `Constants.titles` in the migrate.rst config map) points at the same rows. + +**7. The work order.** Sub-phase 4i, three PRs and one optional, each on its own base, none moving a default parse: the committed snapshot stays byte-identical through each, and the default's sets stay `_compiled.py`. + +| Sub-phase | What | Needs | +|---|---|---| +| 4i-a | The pipeline reads the compiled sets through one internal object, not `Lexicon`'s public fields; the three tools move to it or to the index. No public change; a frozenset test stays a C-level attribute read, so the frame count holds (`_CALL_BASELINE`) | — | +| 4i-b | The break: `Lexicon` holds readings, `add`/`remove` by kind, `readings_of` and `LexiconReading`, the narrowed constructor, the three refusals with their translation, the per-kind repr, the warning on a title reading made phrase-only by a `given` reading, the views reading the lexicon's kinds with rules/R8 and rules/R9 amended, the lazy base, the 2.x pickle translation; the tests and docs moved, `docs/migrate.rst`'s table, a release-log bullet | 4i-a | +| 4i-c | The choice recorded on the `Lexicon`: chained choices union, a caller's readings applied after the choice, the strict mismatch raised, `|` refusing two choices, a pack's fragment joined under the caller's lexicon, the compile and its refusals at `parser_for`, the underscore choice parameters validated, `_apply_choice` retired; the chained-choices entry and 3j's dropped-listing clause superseded | 4i-b | +| 4i-d | Optional: `Reading.languages` follows the choice | 4i-c | + ### The order of work Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's are. The views and the reading move no field of any parse, so the committed snapshot holds still through 4a to 4d; the snapshot records the new views where they are non-empty, as it records the case views, so a reviewer sees what each fills. @@ -475,6 +578,7 @@ Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's ar | 4f | `[rules]` compiled into a `Locale`'s policy patch; RU and TR_AZ rebuilt from their files. Landed (decisions/vocabulary-model's 2026-10-29 sub-phase 4f entry) | 3b | | 4g | the compound given name, a rule kind (#704): built 2026-10-30, `Policy.two_given_names` (rules/O7) | 4f | | 4h | the shape inventory per order (#469), starting with the title between a family and a given name under `FAMILY_FIRST`. Landed as an inventory, no reading moved: the arrangements each order is written in, listed with what each reads (decisions/vocabulary-model's 2026-10-30 #754, sub-phase 4h entry); the title between, and bare after, under a family-first order stays a name word, #938 declined on a survey of real names, which write the title in front (decisions/H5's 2026-10-30 #938 entry) | — | +| 4i | the `Lexicon` surface: configuring by reading, the compiled sets internal, and the choice recorded on the `Lexicon`. Decided 2026-11-02, to be built in four PRs (4i-a to 4i-d, "The `Lexicon` surface: configuring by reading" above) | 3b, 3j | **Interleaved with phase 3** (its sub-phases 3a to 3j, above): 4a can start beside 3a, its `languages` waiting for 3a's index; 4d needs nothing of phase 3, and 4b and 4c need 3a's index only for a listed word's kind, a suffix word's and a particle's; they unblock its packs, `cs`'s generations (3f) and `pt-BR`'s agnomens (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; 4f lands with or after 3b, since both decide what a shipped `Locale` holds; an `is` pack, which no phase 3 sub-phase lists, needs 3b, 4c and 4f. 3j, file-strict, needs 3b alone (3i, which name-strict would have read, is set aside), and 4e and 4h stand apart from both. So the order is 3a and 4a, then 4b to 4d with 3b and 3c (4b after 3a), then 4f, then the packs with 4g, with 4e at any point and 3j at any point after 3b. @@ -482,7 +586,7 @@ Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's ar A shape question, decided 2026-10-28: -- **Whether `Lexicon` keeps a public field for each compiled set**, or answers word, kind and property queries from phase 3's reading index and keeps its sets internal. `dataclasses.fields(Lexicon)` gives 22 public fields on 2026-10-28, 11 of them added since 2.0.0's 11 (`surnames`, `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles`, `suffix_words_ambiguous`, `agnomens`, `given_name_abbreviations`, `titles_ambiguous`, `title_modifiers`, `title_styles`, `title_phrases`), each a new parse behavior's set, and a rule keyed on a title kind or on one language's reading of a kind (phase 3's design, "a property a language's own reading should take only under its pack") would add more. Keeping the fields: each is frozen API, shown in the bounded repr (`Lexicon(default + titles: +1)`), and pickled, where a new field breaks every older `Lexicon` pickle, and with it every pickled `Parser` and locale pack, unless it carries the `ADDED_IN` marker and a default, which `Lexicon`'s own `__setstate__` admits since 2026-10-30 as the guard the other types share does (decisions/A1, #959). Turning them internal: a break that has to land in 3.0, where a query added beside the fields is an addition that can come later; a caller who configures vocabulary by the field today (`Lexicon.add(titles=...)`, `dataclasses.replace`) needs another way in, for words that carry no kind and no language; and phase 3's reading index, which a `Lexicon` answering queries would read, is loaded on its first lookup (decided lazy on 2026-10-29 rather than loaded with `Lexicon.default()`; about 0.6 ms with its bytecode cached, by phase 3's timing above). Since 4a's follow-up filled `Reading.languages` (2026-10-29), the first parse that reads a word from its `Lexicon`'s vocabulary makes one, so answering queries from it adds no load beyond that first lookup. The parser need not pay either way: a frozenset test per word costs no frame and an internal cache keeps that path, while a query method called per word would add one per token, the cost the frame budget counts as scaling with the input. **Derek decided on 2026-10-28 that a caller configures vocabulary by reading and the compiled sets become internal** (decisions/vocabulary-model's 2026-10-28 #754 entry on the `Lexicon` surface). A caller states what a word may be in the language files' terms, a `given` reading on `guru`, no credential reading on `dc`, and `Lexicon` derives the sets and gates the build derives, so making `Guru` a possible given name needs no knowledge of rules/H9's sets, where today it is `remove(titles={'guru'}).add(titles_ambiguous={'guru'})`, and `add(given_name_titles=...)` or `add(suffix_acronyms_ambiguous=...)`, the plausible wrong guesses, do something else. A reading is also what a caller can know about a word and what a pack contribution states. Removing a reading from a caller's own lexicon may settle a clash; decision 15 binds the default and the packs, not a caller's choice about their own data. It breaks the frozen `Lexicon` (decisions/3-0-reevaluations); the method names and whether a caller's own language file can join the pool later (phase 3 defers it) are open, and the 2.x warning the freeze asks for was waived on 2026-10-29, no 2.x feature release being left to carry it (decisions/3-0-reevaluations, THE LEXICON WARNING). +- **Whether `Lexicon` keeps a public field for each compiled set**, or answers word, kind and property queries from phase 3's reading index and keeps its sets internal. `dataclasses.fields(Lexicon)` gives 22 public fields on 2026-10-28, 11 of them added since 2.0.0's 11 (`surnames`, `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles`, `suffix_words_ambiguous`, `agnomens`, `given_name_abbreviations`, `titles_ambiguous`, `title_modifiers`, `title_styles`, `title_phrases`), each a new parse behavior's set, and a rule keyed on a title kind or on one language's reading of a kind (phase 3's design, "a property a language's own reading should take only under its pack") would add more. Keeping the fields: each is frozen API, shown in the bounded repr (`Lexicon(default + titles: +1)`), and pickled, where a new field breaks every older `Lexicon` pickle, and with it every pickled `Parser` and locale pack, unless it carries the `ADDED_IN` marker and a default, which `Lexicon`'s own `__setstate__` admits since 2026-10-30 as the guard the other types share does (decisions/A1, #959). Turning them internal: a break that has to land in 3.0, where a query added beside the fields is an addition that can come later; a caller who configures vocabulary by the field today (`Lexicon.add(titles=...)`, `dataclasses.replace`) needs another way in, for words that carry no kind and no language; and phase 3's reading index, which a `Lexicon` answering queries would read, is loaded on its first lookup (decided lazy on 2026-10-29 rather than loaded with `Lexicon.default()`; about 0.6 ms with its bytecode cached, by phase 3's timing above). Since 4a's follow-up filled `Reading.languages` (2026-10-29), the first parse that reads a word from its `Lexicon`'s vocabulary makes one, so answering queries from it adds no load beyond that first lookup. The parser need not pay either way: a frozenset test per word costs no frame and an internal cache keeps that path, while a query method called per word would add one per token, the cost the frame budget counts as scaling with the input. **Derek decided on 2026-10-28 that a caller configures vocabulary by reading and the compiled sets become internal** (decisions/vocabulary-model's 2026-10-28 #754 entry on the `Lexicon` surface). A caller states what a word may be in the language files' terms, a `given` reading on `guru`, no credential reading on `dc`, and `Lexicon` derives the sets and gates the build derives, so making `Guru` a possible given name needs no knowledge of rules/H9's sets, where today it is `remove(titles={'guru'}).add(titles_ambiguous={'guru'})`, and `add(given_name_titles=...)` or `add(suffix_acronyms_ambiguous=...)`, the plausible wrong guesses, do something else. A reading is also what a caller can know about a word and what a pack contribution states. Removing a reading from a caller's own lexicon may settle a clash; decision 15 binds the default and the packs, not a caller's choice about their own data. It breaks the frozen `Lexicon` (decisions/3-0-reevaluations); it left open the method names, since decided (2026-11-02, "The `Lexicon` surface: configuring by reading" above), and whether a caller's own language file can join the pool later (phase 3 defers it), which is still open; and the 2.x warning the freeze asks for was waived on 2026-10-29, no 2.x feature release being left to carry it (decisions/3-0-reevaluations, THE LEXICON WARNING). Value questions inside the shapes above: