Configure a Lexicon by reading; its set fields are internal (#754, 4i-b)

docs/design/decisions/O6-two-family-names.md

@@ -120,7 +120,7 @@ Open: hyphenated surnames block the switch (`Andrés Sánchez-Cid Torres` keeps
 
   The same boxes rendered family box first, read under FAMILY_FIRST on both sides: ES 47.6% → 91.4% → 94.5% (681 fixed, 68 broken), MX 31.4% → 94.6% → 96.3% (465, 132), CO 23.9% → 94.5% → 97.6% (640, 46), PE 12.6% → 95.9% → 95.9% (203, 205), CL 46.3% → 93.4% → 97.0% (729, 38), VE 38.5% → 88.6% → 93.3% (949, 44), AR 82.3% → 59.8% → 90.8% (6,068, 53).
   PSC (`psc-individuals`, the fetch recorded on #849), a second population the list was not built from, by nationality, default → switch → switch and list, fixed and broken: Spanish 6,089 records 31.8% → 80.2% → 82.2% (150, 31), Mexican 402 25.9% → 80.1% → 82.1% (13, 5), Colombian 377 29.2% → 84.4% → 87.0% (15, 5), Venezuelan 237 40.5% → 75.9% → 80.6% (13, 2), Chilean 151 35.1% → 77.5% → 84.8% (12, 1), Peruvian 116 21.6% → 81.0% → 83.6% (5, 2), Argentine 320 81.9% → 45.6% → 81.9% (121, 5).
-  Recompute: `uv run --with openpyxl --with xlrd python tools/es_given_names.py --top N` writes the list for a cutoff (`--write`) and prints its coverage. For ORCID, `local_agreement.read` per record of `local_corpora.load("orcid-researchers")` with `FIELDED["orcid-researchers"]`, records with both boxes, grouped by `country`, under `Parser()`, `Parser(policy=Policy(two_family_names=True))` and that policy with `lexicon=Lexicon.default().add(given_names=...)` holding the tool's first N words, each with `_title_words` of its own lexicon; family-first, the record's `name` replaced by `family + " " + given` and each policy given `name_order=FAMILY_FIRST`. For PSC, `local_agreement.score(rows, FIELDED["psc-individuals"], parser, jobs=1)` over the records whose `group_of` is the nationality, each cell a group's `agreed` over its `scored`. A record is fixed where it agrees only with the list, broken where it agrees only without it. The comparator is the switch alone on the same tree; `local_agreement.py orcid-researchers --base` and `psc-individuals --base` against master match the base, the default reading no list.
+  Recompute: `uv run --with openpyxl --with xlrd python tools/es_given_names.py --top N` writes the list for a cutoff (`--write`) and prints its coverage. For ORCID, `local_agreement.read` per record of `local_corpora.load("orcid-researchers")` with `FIELDED["orcid-researchers"]`, records with both boxes, grouped by `country`, under `Parser()`, `Parser(policy=Policy(two_family_names=True))` and that policy with `lexicon=Lexicon.default() | Lexicon(given_names=...)` (`add(given_names=...)` before 3.0's 4i-b) holding the tool's first N words, each with `_title_words` of its own lexicon; family-first, the record's `name` replaced by `family + " " + given` and each policy given `name_order=FAMILY_FIRST`. For PSC, `local_agreement.score(rows, FIELDED["psc-individuals"], parser, jobs=1)` over the records whose `group_of` is the nationality, each cell a group's `agreed` over its `scored`. A record is fixed where it agrees only with the list, broken where it agrees only without it. The comparator is the switch alone on the same tree; `local_agreement.py orcid-researchers --base` and `psc-individuals --base` against master match the base, the default reading no list.
 - 2026-10-30 #959, decided — THE VETO STAYS A CLAUSE OF O6, THOUGH IT READS A SECOND SIGNAL. docs/design/AGENTS.md ("One rule per piece of evidence") says a clause reading a signal its rule did not already read is a new rule, and the veto reads one O6 did not: the word standing where the family would take one, looked up in the caller's given-name list (the #961 review). Kept in O6 anyway, decided 2026-10-30: the veto is one more member of the class the Except bullet "What is taken must be one name word a family name could be" already holds, beside the initial written alone and the mark with no letter in it. It only declines a take O6 would otherwise make, and reads nothing of its own: with the switch off it is never asked, and where it declines the name reads as with the switch off. A rule of its own would state no reading but O6's absence. The Signal line names the lookup as a second signal joined with ` + `, tagged `position` as P1's vocabulary signal is, by where the word stands, so O6 shows in the split-candidate grep; the next signal O6 gains is the point to split it.
 - 2026-10-30 #753, phase 3g, decided in chat — `locales.ES` TURNS THE SWITCH ON; `locales.PT` DOES NOT. The `es` pack's `[rules]` table writes `two_family_names = true` (sub-phase 4f's schema), compiled into its policy patch, and the pack carries `nameparser/locales/_es_given_names.py`'s list as its lexicon fragment, so a caller choosing it reads `Gabriel García Márquez` with family `García Márquez` and `José Luis Pérez` with family `Pérez`. `two_given_names` stays opt-in in both packs (decisions/O7's phase 3g entry). The table answers two questions, each by its own comparison. Against the DEFAULT: should the pack turn the switch on at all, which is what a caller choosing the pack gets over not choosing it; every Spanish-speaking country gains on ORCID, Argentina least (82.5% → 89.3%), and PSC by nationality gains everywhere but Argentina, which ties. Against O6 ALONE: does the list help once the switch is on, the #959 entry's cutoff question, here re-measured on the pack as shipped, so the column holds the es choice's titles beside the list; against O6 with the list and no choice the pack differs on 9 ORCID records of these countries, all broken by a title the choice turns on (Cuba 7, `Lic.Alina Maren Ferrier`, `Lic Arisleidy Noriega Cisneros`; Venezuela 2, `Doctora Yenny Nereida Montilla Polo`, `PROFESORA CAROLINA GONZALEZ`), the box holding the title the filer typed in; it does in every country but Peru (178 fixed, 189 broken), the one the #959 review found net negative, a loss of 0.1 points that the pack's gain over the default there (16,197 fixed, 146 broken) dwarfs. So the Open line on Latin America stays a question about a better list, not about whether the pack should read two family names. The rows for the United States and Great Britain show the cost to data the pack is not meant for, which a caller choosing it accepts.
 

docs/design/decisions/R8-generation-and-credentials-views.md

@@ -3,21 +3,22 @@
 ## Standing
 
 - `ParsedName.generation` and `ParsedName.credentials` partition `suffix` by the reading each word took, a generation or an agnomen in the first and every other word in the second; they are views, not roles, so `suffix`, `as_dict()`, `replace()`'s fields and the repr are unchanged, item 4 of decisions/vocabulary-model's #754 output-model entry — 2026-10-29 #754, sub-phase 4b.
-- A listed suffix word's kind is read from the reading index (`_reading_index.suffix_kind_of`, through `_vocab.suffix_kind`), since the compiled sets do not record it — 2026-10-29 #754, sub-phase 4b.
+- A listed suffix word's kind is the lexicon's, compiled with its sets (`Lexicon._sets.generation_words`, `credential_words`), which on the default is the shipped vocabulary's — 2026-11-02 #754, sub-phase 4i-b, superseding 4b's read from the reading index.
 
 Elsewhere: the field's composition and the 2026-08-16 decision to keep it one field, which still stands, decisions/suffix-field-composition; whether `credentials` should leave the honorifics out, CLDR having no third field, is a value question under the RFC's Phase 4, Left open (docs/design/vocabulary-model-rfc.md).
 
-Elsewhere: a word the shipped vocabulary lists as a credential reads as one where a caller lists it in `suffix_words_ambiguous`, the gated credential words `kc`, `pc` and `dl` reading so, decisions/S11's 2026-10-30 #965 entry.
+Elsewhere: the gated credential words `kc`, `pc` and `dl`, a credential written as a word beside a name reading, read as credentials, decisions/S11's 2026-10-30 #965 entry.
 
 Open: none in this file.
 
 ## Log
 
 - 2026-10-29 #754, sub-phase 4b (#326) — IMPLEMENTED item 4 of decisions/vocabulary-model's 2026-10-28 #754 entry. SUPERSEDES: decisions/A1's 2026-10-26 #752 ACCEPTED clause that a listed suffix word neither a generation by its tag nor an agnomen reads as `post-nominal`, `jr` and `esq` alike. What the implementation decided, beyond that entry:
-  1. **The kind of a listed suffix word comes from the reading index.** `jr` and `esq` compile into `suffix_words` with the same tags, so 4a read both as `post-nominal` and pinned the gap with strict xfails. Classify writes a kind tag beside `vocab:suffix-word` (`vocab:generation-word`, `vocab:credential-word`), asking `_reading_index.suffix_kind_of` through `_vocab.listed_word_kind_tag`, a memo whose hit costs no frame, and `_vocab.suffix_kind` reads the tag, so `Token.reading` and `Ambiguity.alternative` move together: a lone `Jr` names a `generation` alternative where it named a `post-nominal` one, and a credential written as a word (`博士`) reads `credential`. The vocabulary's courtesy, honorific, professional honorific and status kinds have no reading kind of their own and stay `post-nominal`. `_vocab` imports `_reading_index` for it, which tests/v2/test_layering.py's table records. The first draft looked the kind up in `suffix_kind` from the token's text, which keeps a delimiter glued on (rules/C10): `John Smith, Jr.; MD` read `Jr.;` post-nominal and put it in the credentials, as `John Smith, V; MD` did a numeral, found by the orchestrating review of #947. Classify reads every word by its glued stem already, so the kind is read there, by the key every other tag of the word is read by, and assemble drops the two tags from the published token, `Token.reading` carrying the kind out: `Token` compares its tags, and a published tag would make a name unequal to its own 2.x pickle (tests/v2/test_types.py's 2.8.0 fixture, `…de la Vega III`, caught it); a declared delimiter glued to either end (`Jr./`, `-Jr.`) reads alike, while before the first comma, where C10 reads a glued delimiter as part of the word and classify cuts no stem, `Jr.;` is listed by no set and reaches the suffix only by `period_joined_vocab`, which reads it by its chunks (`Jr.` and `;`, the `Lt.Gov.` path) and tags it `vocab:suffix` with no kind, so `John Smith Jr.; MD` keeps it in the credentials, as `Jr.III` is there after a comma or not, a word made only of generation words; `John Smith Jr; MD`, without the period, reads family `Jr;`. Recorded as rules/R8's cost rather than given a kind of its own (two reviews of the fix commits); a view keeps the delimiter in its word's text, as the suffix does, so `John Smith, Jr.; MD` gives generation 'Jr.;'; and `test_every_suffix_words_reading_alternative_and_index_kind_agree` walks every suffix-set word (826 on 2026-10-29) holding the reading taken, the alternative and the index's kind to one answer.
-  2. **The kind is read whatever the reading's default.** `st`, a Czech generation written `default = false`, reads `generation` when a caller's `Lexicon` lists it, while its `languages` name no file, since none holds it on. A pack turning such a reading on (phase 3's `cs`, 3f) then needs nothing more for the kind; a word only a caller's own `Lexicon` lists has no row and reads `post-nominal`.
+  1. **The kind of a listed suffix word comes from the reading index** (SUPERSEDED by the 2026-11-02 #754, sub-phase 4i-b entry below: it is the lexicon's). `jr` and `esq` compile into `suffix_words` with the same tags, so 4a read both as `post-nominal` and pinned the gap with strict xfails. Classify writes a kind tag beside `vocab:suffix-word` (`vocab:generation-word`, `vocab:credential-word`), asking `_reading_index.suffix_kind_of` through `_vocab.listed_word_kind_tag`, a memo whose hit costs no frame, and `_vocab.suffix_kind` reads the tag, so `Token.reading` and `Ambiguity.alternative` move together: a lone `Jr` names a `generation` alternative where it named a `post-nominal` one, and a credential written as a word (`博士`) reads `credential`. The vocabulary's courtesy, honorific, professional honorific and status kinds have no reading kind of their own and stay `post-nominal`. `_vocab` imports `_reading_index` for it, which tests/v2/test_layering.py's table records. The first draft looked the kind up in `suffix_kind` from the token's text, which keeps a delimiter glued on (rules/C10): `John Smith, Jr.; MD` read `Jr.;` post-nominal and put it in the credentials, as `John Smith, V; MD` did a numeral, found by the orchestrating review of #947. Classify reads every word by its glued stem already, so the kind is read there, by the key every other tag of the word is read by, and assemble drops the two tags from the published token, `Token.reading` carrying the kind out: `Token` compares its tags, and a published tag would make a name unequal to its own 2.x pickle (tests/v2/test_types.py's 2.8.0 fixture, `…de la Vega III`, caught it); a declared delimiter glued to either end (`Jr./`, `-Jr.`) reads alike, while before the first comma, where C10 reads a glued delimiter as part of the word and classify cuts no stem, `Jr.;` is listed by no set and reaches the suffix only by `period_joined_vocab`, which reads it by its chunks (`Jr.` and `;`, the `Lt.Gov.` path) and tags it `vocab:suffix` with no kind, so `John Smith Jr.; MD` keeps it in the credentials, as `Jr.III` is there after a comma or not, a word made only of generation words; `John Smith Jr; MD`, without the period, reads family `Jr;`. Recorded as rules/R8's cost rather than given a kind of its own (two reviews of the fix commits); a view keeps the delimiter in its word's text, as the suffix does, so `John Smith, Jr.; MD` gives generation 'Jr.;'; and `test_every_suffix_words_reading_alternative_and_index_kind_agree` walks every suffix-set word (826 on 2026-10-29) holding the reading taken, the alternative and the index's kind to one answer.
+  2. **The kind is read whatever the reading's default** (SUPERSEDED by the 2026-11-02 #754, sub-phase 4i-b entry below: a caller's `add("st", kind="suffix")` reads `St` as a post-nominal). `st`, a Czech generation written `default = false`, reads `generation` when a caller's `Lexicon` lists it, while its `languages` name no file, since none holds it on. A pack turning such a reading on (phase 3's `cs`, 3f) then needs nothing more for the kind; a word only a caller's own `Lexicon` lists has no row and reads `post-nominal`.
   3. **Each word is placed by its own reading, and separated as the writer separated it.** A `joined` mark is not only the second word of a split credential (`Ph.` `D.`, which takes the first word's reading) but every word of a suffix run written without commas (`Jr. III PhD`), so a view places each word by its reading. Two words of one view are separated by a comma where the suffix holds one anywhere between them, R1's separator (a typed comma, or a semicolon, hyphen, slash or delimiter C10 drops: `MD / Jr. / PhD` gives credentials 'MD, PhD'), and by a space where not, so `MD Jr. PhD` gives credentials 'MD PhD'; the first draft started a new comma-joined entry wherever a word of the other view stood between, writing a comma nobody typed ('MD, PhD'), found by the design-docs review. A connective goes with the word before it (`PhD and MD`, `Jr. and III`), so one joining a generation to a credential dangles at the end of the generation (`Jr. and MD` gives generation 'Jr. and'), accepted rather than given a rule of its own, no corpus name holding the shape.
   4. **A suffix word with no reading is a credential.** `replace()` and `Parser.revise()` write tokens with no `Token.reading` (4a), and a `ParsedName` unpickled from 2.x has none either, so `replace(suffix="Jr.")` reads credentials 'Jr.' and the 2.8 pickle of `Dr. Juan Q. de la Vega III` credentials 'III': nothing records what the word was read as, and the catch-all is the view that claims least. A name word a run took into the suffix by position is a credential too, 4a's reading of it (`Smith, Jr., John` gives credentials 'John').
   5. **The views stay out of `as_dict()`, `--json` and the repr**, as `family_base` and the other derived views do: `as_dict()` is the seven fields `replace()` takes back. They are `render()` keys and assertable fields of a rule's example lines.
   6. **The snapshot records them** (`tools/differential/snapshot.py`, a `RECORDED_LATER` pair): `generation` where the suffix holds one, and `credentials` beside it where the suffix holds both, so a row with neither is a suffix of credentials only, or none. No other key moved on recording.
   - Measured 2026-10-29 on the 4b branch over the committed snapshot (`tools/differential/snapshot.jsonl`, 6,570 readings, rules/R8's own examples included): 1,077 rows hold a suffix, 488 of them a generation, 142 a generation and credentials both. Recompute: count rows with `suffix`, `generation` and `credentials` in the snapshot. Over `cldr_fields.jsonl`, 576 distinct (text, order) renderings carry CLDR's `generation` or `credentials`; 65 parse to exactly CLDR's suffix words, and all 65 split into CLDR's two fields. Recompute: parse each distinct text under its order (`givenFirst` as `GIVEN_FIRST`, `surnameFirst` as `FAMILY_FIRST`, any other order the default), keep those whose `suffix` words, commas removed, equal CLDR's `generation` and `credentials` words together, and compare each view's words with its field's. The RFC's 615 and 75 (2026-10-28) counted renderings, not distinct (text, order) pairs.
+- 2026-11-02 #754, sub-phase 4i-b — A LISTED SUFFIX WORD'S KIND IS THE LEXICON'S. SUPERSEDES items 1 and 2 of the 4b entry above. A caller now states a word's kind (`add("secundus", kind="generation")`), and a view reading the index instead would ignore it in silence, so the compile gives the kinds the sets do not record, `GENERATION_WORDS` and `CREDENTIAL_WORDS`, and classify reads them off the lexicon (rules/R8 amended). On the default they are the index's by the compile invariant (`test_the_default_kind_sets_are_the_shipped_index_s`), so no default parse moves; one test's caller-configured reading did: `Lexicon.default().add("st", kind="suffix")` reads `St` as a post-nominal where 4b read the cs file's generation, written `default = false`. An agnomen needs a name reading, as the build's check 11 requires of a file, so `add(w, kind="agnomen")` alone is refused. decisions/vocabulary-model's 2026-11-02 #754, sub-phase 4i-b entry has the rest.