Reading.languages names the files on in the parse's own Lexicon (#754, 4i-d)

docs/design/vocabulary-model-rfc.md

@@ -463,7 +463,7 @@ How a call reaches a pack's pattern is open, and each answer costs something: `P
 
 ### The `Lexicon` surface: configuring by reading
 
-Status: **decided 2026-11-02 as recommended below (decisions/vocabulary-model's 2026-11-02 #754, the `Lexicon` surface entry), item 1's title warning on 2026-11-01 (decisions/vocabulary-model's 2026-11-01 #754, the title warning entry); 4i-a, 4i-b and 4i-c are built (decisions/vocabulary-model's 2026-11-02 #754, sub-phase 4i-a, sub-phase 4i-b and sub-phase 4i-c entries), the optional 4i-d is not.** What is decided is the direction (2026-10-28): a caller configures vocabulary by reading, `Lexicon` derives the sets and gates the build derives, and the compiled sets become internal (on 2026-11-01, 20 of `Lexicon`'s 23 public fields: the 19 of `_lexicon._COMPILED_FIELDS` and the derived `particles_ambiguous`; item 3 keeps the other three), with no 2.x warning (2026-10-29). This section proposes the API that direction leaves open, each question with the options weighed and one recommendation; decisions/vocabulary-model's 2026-11-01 #754, the `Lexicon` surface proposed entry records the proposal and its measurements, and decisions/vocabulary-model's 2026-11-02 #754, the `Lexicon` surface entry the decision, each recommendation accepted and the alternatives below declined. Where this section says "proposed" or "recommended", read "decided"; the wording is kept as the record of what was weighed. Every parse below was run on master `a3b1f3e1` (Python 3.11.17) through a throwaway prototype that compiles a lexicon's readings with the shipped `compile_readings` over `choose(index(), chosen)`, the whole index with its readings off by default, as item 4 has the default hold them, the caller's edits applied after the choice, and folds the result into today's `Lexicon`, so the real parser reads it; the prototype is not committed.
+Status: **decided 2026-11-02 as recommended below (decisions/vocabulary-model's 2026-11-02 #754, the `Lexicon` surface entry), item 1's title warning on 2026-11-01 (decisions/vocabulary-model's 2026-11-01 #754, the title warning entry); 4i-a, 4i-b, 4i-c and the optional 4i-d are built (decisions/vocabulary-model's 2026-11-02 #754, sub-phase 4i-a, sub-phase 4i-b, sub-phase 4i-c and sub-phase 4i-d entries).** What is decided is the direction (2026-10-28): a caller configures vocabulary by reading, `Lexicon` derives the sets and gates the build derives, and the compiled sets become internal (on 2026-11-01, 20 of `Lexicon`'s 23 public fields: the 19 of `_lexicon._COMPILED_FIELDS` and the derived `particles_ambiguous`; item 3 keeps the other three), with no 2.x warning (2026-10-29). This section proposes the API that direction leaves open, each question with the options weighed and one recommendation; decisions/vocabulary-model's 2026-11-01 #754, the `Lexicon` surface proposed entry records the proposal and its measurements, and decisions/vocabulary-model's 2026-11-02 #754, the `Lexicon` surface entry the decision, each recommendation accepted and the alternatives below declined. Where this section says "proposed" or "recommended", read "decided"; the wording is kept as the record of what was weighed. Every parse below was run on master `a3b1f3e1` (Python 3.11.17) through a throwaway prototype that compiles a lexicon's readings with the shipped `compile_readings` over `choose(index(), chosen)`, the whole index with its readings off by default, as item 4 has the default hold them, the caller's edits applied after the choice, and folds the result into today's `Lexicon`, so the real parser reads it; the prototype is not committed.
 
 **1. Configuring: `add` and `remove` take words and a reading kind.** Proposed:
 
@@ -528,7 +528,7 @@ The `nameparser.config` constants stay public and frozen: they are the default's
 - A chained call whose `strict` differs from the base's recorded one raises `ValueError`, offering both spellings; raising now keeps a later rule for mixing them an addition.
 - `_parser._apply_choice` and `_choice_delta` retire: the choice is no longer a delta against the default's compile but a filter over the base's own readings.
 
-Weighed against: (a) recording nothing, as decided on 2026-10-29 for a lexicon of sets, which could not recompile from what it held, so a choice could only be a delta; and (b) a `Parser.languages` field, the 3b session's recommendation, which leaves `parser.lexicon` holding the compiled choice beside a record of it, so the record and what it describes could disagree once a caller replaces the lexicon. The objection to a `Lexicon` field, that its own pickle guard refused an added field, was removed on 2026-10-30 when the guard learned `ADDED_IN` (#960, decisions/A1). Whether `Reading.languages` then follows the choice is left to its own sub-phase (4i-d): the parse can know its choice once the lexicon records it, which answers the 3b entry's reason for not following it, but its memo is keyed by the word alone today and would have to take the lexicon's readings into its key.
+Weighed against: (a) recording nothing, as decided on 2026-10-29 for a lexicon of sets, which could not recompile from what it held, so a choice could only be a delta; and (b) a `Parser.languages` field, the 3b session's recommendation, which leaves `parser.lexicon` holding the compiled choice beside a record of it, so the record and what it describes could disagree once a caller replaces the lexicon. The objection to a `Lexicon` field, that its own pickle guard refused an added field, was removed on 2026-10-30 when the guard learned `ADDED_IN` (#960, decisions/A1). Whether `Reading.languages` then follows the choice is left to its own sub-phase (4i-d): the parse can know its choice once the lexicon records it, which answers the 3b entry's reason for not following it, but its memo is keyed by the word alone today and would have to take the lexicon's readings into its key. 4i-d built it on 2026-11-02: the field names the files whose reading is on in the parse's own lexicon, and the memo takes a per-lexicon key (decisions/vocabulary-model's 2026-11-02 #754, sub-phase 4i-d entry).
 
 **5. Pickle.** Whether `Lexicon`'s guard learns `ADDED_IN` needs no work here: it has admitted the marker since 2026-10-30 (decisions/A1's #959 prerequisite entry, PR #960). Proposed: a `Lexicon` pickles its five parts, its base as a snapshot of the rows, as today's pickle holds its sets whole, with its ordered log, choice, lists and masks; at load a snapshot equal to the running release's index becomes the by-reference base again (item 3's base equality), and one from another release stays a snapshot, parsing as it did where it was pickled; a pickle of the default holds its base, all 2,261 index rows with their tags, grouped by word as nameparser/config/_index.py writes them (33,122 bytes against today's 27,204 for the default's sets, protocol 4, measured 2026-11-01). Weighed against pickling the base by reference with the caller's edits, which is smaller but makes a pickled `Parser()` load the loading release's vocabulary in silence, the drift decisions/A1's 2026-10-30 entry asks to be avoided or recorded.
 
@@ -562,7 +562,7 @@ All three were checked on a frozen, slotted throwaway dataclass on 2026-11-01. T
 | 4i-a | The pipeline reads the compiled sets through one internal object, not `Lexicon`'s public fields; the three tools move to it or to the index. No public change; a frozenset test stays a C-level attribute read, so the frame count holds (`_CALL_BASELINE`). Landed 2026-11-02 (decisions/vocabulary-model's 2026-11-02 #754, sub-phase 4i-a entry) | — |
 | 4i-b | The break: `Lexicon` holds readings, `add`/`remove` by kind, `readings_of` and `LexiconReading`, the narrowed constructor, the three refusals with their translation, the per-kind repr, the warning on a title reading made phrase-only by a `given` reading, the views reading the lexicon's kinds with rules/R8 and rules/R9 amended, the lazy base, the 2.x pickle translation; the tests and docs moved, `docs/migrate.rst`'s table, a release-log bullet. Landed 2026-11-02 (decisions/vocabulary-model's 2026-11-02 #754, sub-phase 4i-b entry) | 4i-a |
 | 4i-c | The choice recorded on the `Lexicon`: chained choices union, a caller's readings applied after the choice, the strict mismatch raised, `|` refusing two choices, a pack's fragment joined under the caller's lexicon, the compile and its refusals at `parser_for`, the underscore choice parameters validated, `_apply_choice` retired; the chained-choices entry and 3j's dropped-listing clause superseded. Landed 2026-11-02 (decisions/vocabulary-model's 2026-11-02 #754, sub-phase 4i-c entry) | 4i-b |
-| 4i-d | Optional: `Reading.languages` follows the choice | 4i-c |
+| 4i-d | Optional: `Reading.languages` follows the choice. Landed (decisions/vocabulary-model's 2026-11-02 #754, sub-phase 4i-d entry) | 4i-c |
 
 ### The order of work
 
@@ -578,7 +578,7 @@ Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's ar
 | 4f | `[rules]` compiled into a `Locale`'s policy patch; RU and TR_AZ rebuilt from their files. Landed (decisions/vocabulary-model's 2026-10-29 sub-phase 4f entry) | 3b |
 | 4g | the compound given name, a rule kind (#704): built 2026-10-30, `Policy.two_given_names` (rules/O7) | 4f |
 | 4h | the shape inventory per order (#469), starting with the title between a family and a given name under `FAMILY_FIRST`. Landed as an inventory, no reading moved: the arrangements each order is written in, listed with what each reads (decisions/vocabulary-model's 2026-10-30 #754, sub-phase 4h entry); the title between, and bare after, under a family-first order stays a name word, #938 declined on a survey of real names, which write the title in front (decisions/H5's 2026-10-30 #938 entry) | — |
-| 4i | the `Lexicon` surface: configuring by reading, the compiled sets internal, and the choice recorded on the `Lexicon`. Decided 2026-11-02, to be built in four PRs, 4i-a, 4i-b and 4i-c landed (4i-a to 4i-d, "The `Lexicon` surface: configuring by reading" above) | 3b, 3j |
+| 4i | the `Lexicon` surface: configuring by reading, the compiled sets internal, and the choice recorded on the `Lexicon`. Decided 2026-11-02, built in four PRs, 4i-a to 4i-d, all landed (4i-a to 4i-d, "The `Lexicon` surface: configuring by reading" above) | 3b, 3j |
 
 **Interleaved with phase 3** (its sub-phases 3a to 3j, above): 4a can start beside 3a, its `languages` waiting for 3a's index; 4d needs nothing of phase 3, and 4b and 4c need 3a's index only for a listed word's kind, a suffix word's and a particle's; they unblock its packs, `cs`'s generations (3f) and `pt-BR`'s agnomens (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; 4f lands with or after 3b, since both decide what a shipped `Locale` holds; an `is` pack, which no phase 3 sub-phase lists, needs 3b, 4c and 4f. 3j, file-strict, needs 3b alone (3i, which name-strict would have read, is set aside), and 4e and 4h stand apart from both. So the order is 3a and 4a, then 4b to 4d with 3b and 3c (4b after 3a), then 4f, then the packs with 4g, with 4e at any point and 3j at any point after 3b.
 

docs/locales.rst

@@ -389,11 +389,20 @@ at the end of the name, where the default reads each as a name word.
     >>> czech.parse("Jana Nováková roz. Svobodová").maiden
     'Svobodová'
 
-A word read through a pack names no language file in its
-:attr:`~nameparser.Reading.languages`, which lists only the files whose
-reading is on by default, whatever the parser chose: ``ml.`` above names
-none, and ``Mgr.`` names ``en`` and ``fr``, whose Monsignor reading the
-Czech choice dropped for its own.
+A word's :attr:`~nameparser.Reading.languages` names the files whose
+reading is on under the parser's choice: ``ml.`` above names ``cs``,
+whose reading the choice turns on, and ``Mgr.`` names ``cs`` alone, the
+Czech choice having dropped the Monsignor reading of ``en`` and ``fr``
+for its own, which the default names:
+
+.. doctest::
+
+    >>> czech.parse("Jiří Dienstbier ml.").tokens[-1].reading.languages
+    ('cs',)
+    >>> parse("Mgr. Jan Novák").tokens[0].reading.languages
+    ('en', 'fr')
+    >>> czech.parse("Mgr. Jan Novák").tokens[0].reading.languages
+    ('cs',)
 
 A Czech or Slovak woman who keeps her own surname beside her
 husband's writes the two without a hyphen, ``Alena Pešová Řehořová``,

docs/release_log.rst

@@ -16,7 +16,7 @@ Release Log
     - **Add a switch for names with a compound given name.** ``Parser(policy=Policy(two_given_names=True)).parse("Juan Carlos Pérez")`` gives given ``Juan Carlos``, where 1.4.0 through 2.8.0 gave given ``Juan``, middle ``Carlos``. Off by default: under it a middle name beside the given name joins it (``John Quincy Adams`` gives given ``John Quincy``). With ``two_family_names`` on too, the family takes its word first, so ``María José García López`` reads given ``María José``, family ``García López``. A ``Policy`` pickled by 2.8 loads with it off. :issue:`704`
     - **Add Lexicon.given_names, given names two_family_names leaves out of the family.** ``Parser(policy=Policy(two_family_names=True), lexicon=Lexicon.default() | Lexicon(given_names={"luis"})).parse("José Luis Pérez")`` gives middle ``Luis``, family ``Pérez``, where the switch alone, in 2.8.0 as now, gives family ``Luis Pérez``. Empty by default, so the switch alone reads as before; an entry without accents matches the word with them. :issue:`959`
     - **Add Token.reading, the reading the parse took for each word.** ``parse("John Smith PhD Jones").tokens[-1].reading`` is ``Reading('credential' as 'suffix', 'position')``, the reading 2.7's ``Ambiguity.alternative`` gave only for the other side of a fork, and ``ReadingKind`` gains ``patronymic``. The field is not compared, and a ``ParsedName`` pickled by 2.x loads with no readings. :issue:`754`
-    - **Add Reading.languages, the language files that list a word the way it was read.** ``parse("Dr. Juan de la Vega III").tokens[2].reading.languages`` is ``('es', 'fr', 'it', 'nl', 'pt')`` for the particle ``de``, and ``Ambiguity.alternative`` names its files too; a reading by shape or position names none. It is not compared, and a ``Reading`` pickled by 2.7 or 2.8 loads with none. The first parse in a process that meets a listed word loads the reading index once: about 1 ms, or about 16 ms where Python writes no bytecode cache. :issue:`754`, :issue:`753`
+    - **Add Reading.languages, the language files that list a word the way it was read.** ``parse("Dr. Juan de la Vega III").tokens[2].reading.languages`` is ``('es', 'fr', 'it', 'nl', 'pt')`` for the particle ``de``, and ``Ambiguity.alternative`` names its files too; a reading by shape or position names none. Under a choice of languages it names the files the choice reads: ``parser_for(locales.PT)`` names ``('pt',)`` for that ``de``. It is not compared, and a ``Reading`` pickled by 2.7 or 2.8 loads with none. The first parse in a process that meets a listed word loads the reading index once: about 1 ms, or about 16 ms where Python writes no bytecode cache. :issue:`754`, :issue:`753`
     - **Add Czech, Slovak, Slovenian and Croatian packs, locales.CS, SK, SL and HR.** ``parser_for(locales.CS).parse("Jan Novák, MUDr")`` gives title ``MUDr``, given ``Jan``, family ``Novák``, where 1.4.0 through 2.8.0 gave given ``MUDr``, family ``Jan Novák``. Each pack chooses its one language and turns on words the default does not read: ``ml.`` and ``st.`` after the name in all four (``Jiří Dienstbier ml.``), the Czech and Slovak academic titles and ``DrSc.``, and Czech ``Pan`` and ``roz.``. Choosing the language drops ``st``'s Saint readings (``St Clair Smith`` reads given ``St``), which choosing ``fr`` as well keeps. ``CS`` and ``SK`` also read two family names, so ``Alena Pešová Řehořová`` gives family ``Pešová Řehořová`` and two given names read as a given name and a family name (``Petr Pavel Novák`` gives family ``Pavel Novák``); a family comma keeps both. :issue:`753`
     - **Add Hindi and Bengali packs, locales.HI and locales.BN, for the romanized Indic titles.** ``parser_for(locales.HI).parse("Smt Nirmala Sitharaman")`` gives title ``Smt``, given ``Nirmala``, where 1.4.0 through 2.8.0 gave given ``Smt``, middle ``Nirmala``. ``HI`` reads ``Smt``, ``Shrimati`` and ``Shriyut`` as titles and ``BN`` ``Smt``, ``Shrimati`` and ``Adhyapak``; ``Shri`` and ``Sushri`` stay name words under both, since each opens given names (``Shri Kant``, ``Sushri Sangita``). See :doc:`locales`. :issue:`345`, :issue:`753`
     - **Add parser_for(..., strict=True), to read only the chosen languages' words.** ``parser_for(Locale(code="ko", lexicon=Lexicon.empty(), languages=frozenset({"ko"})), strict=True).parse("Kim Do Hyun")`` gives middle ``Do``, family ``Hyun``, where the default, as in 1.4.0 through 2.8.0, and a plain ``ko`` choice give family ``Do Hyun``, ``Do`` read as the Portuguese particle. Only the chosen languages' vocabulary files are read, with the words written in other scripts; every other language's words leave, the shared ones (``Dr``, ``Prof``) and English credentials included, so ``Kim Min Jun, MD`` gives given ``MD`` unless ``en`` is chosen too. A strict call that chooses no language raises ``ValueError``. ``python -m nameparser`` gains ``--strict``, and ``--locale`` takes a language tag no pack has (``--locale ko``). See :doc:`locales`. :issue:`753`

docs/usage.rst

@@ -1256,9 +1256,12 @@ otherwise have been the family name:
     Reading('name' as 'family', 'position')
 
 A reading from the vocabulary also names the language files that list
-the word that way, as BCP 47 tags in ``languages``. A reading by shape
-or position names none, and neither does a word from your own
-``Lexicon`` that no shipped file lists:
+the word that way, as BCP 47 tags in ``languages``: the files whose
+reading is on in the parser's own ``Lexicon``, so a choice of languages
+(:doc:`locales`) names the files it reads. A reading by shape or
+position names none, and neither does a reading you add to your own
+``Lexicon`` of a kind no file holds for that word; adding a kind a
+file does hold turns that file's reading on, and names it:
 
 .. doctest::