[derek73]

Sub-phase 4i-b of #754: the public Lexicon break, decided on 2026-11-02 (decisions/vocabulary-model's 2026-11-02 #754, the Lexicon surface entry). A caller configures vocabulary by reading, and the compiled sets become internal. Built on 4i-a (#989).

What changes

• The state. Lexicon holds a base, an ordered edit log, and three fields: surnames, given_names and capitalization_exceptions.

  • The base is the shipped index by reference, none, or a pickled release's rows.
  • The log is compared as written, each edit stored frozen.
  • The compiled sets live in a private _sets field.
  • The choice is not on the Lexicon yet; that is 4i-c.

• Configuring. add(*words, kind, **properties) and remove(*words, kind=None) take kinds and properties as the vocabulary files spell them. remove also takes a category.

• Querying. readings_of(word) returns the new public LexiconReading.

• The repr counts readings by kind, e.g. Lexicon(default + given: -1).

• | concatenates the logs, the left one's first.

• The title warning (2026-11-01, the title warning entry) fires when an add creates or re-adds a title reading that a given reading keeps a title only inside a phrase. The warning names remove(w, kind="given").

• The build's checks 3, 4, 8, 9 and 11's agnomen clause run on each add, and each raises naming the call that fixes it.

• Three refusals, each translated for the words named:

  • add/remove with a set keyword;
  • Lexicon(...) and dataclasses.replace with one;
  • a set attribute read.

  For example, Lexicon.default().add(titles={"dean"}) raises and names .remove('dean', kind='given'). A 2.x pickle is translated the same way (pickle option (b)). The 2.8 fixture's Lexicon.empty().add(titles={"dean"}) loads equal to Lexicon.empty().add("dean", kind="title").

• The views read the lexicon's kinds. These are rules/R8 (the generation and credentials views) and rules/R9 (the patronymic view), both amended.

  • The compile writes three new internal sets to _compiled.py: GENERATION_WORDS, CREDENTIAL_WORDS and PATRONYMIC_PARTICLES.
  • No config module re-exports them.
  • Classify reads them where 4b and 4c looked the kind up in the index.
  • So add("secundus", kind="generation") puts Secundus in the generation view.

• The default stays lazy. Lexicon.default() reads _compiled.py, and parse("John Smith") loads no index; a test holds this.

• _parser._apply_choice keeps the 3b delta until 4i-c retires it. It now writes the delta as readings appended to the log.

  • Over every plain and strict file tag (78 + 83), on three lexicons, the compiled sets equal the old delta's in every field. This is a committed test: test_a_choice_written_as_readings_gives_the_deltas_sets.

Found by review, fixed here

Two reviews ran, one of the code and one of the design docs.

• An agnomen added alone was accepted. add("bisneto", kind="agnomen") with no name reading went through and read as a credential. The build's check 11 refuses that in a file, so add now refuses it too, naming .add(w, kind="family").
• A choice dropped a clashing caller reading in silence. parser_for(locales.CS, base=Parser(lexicon=Lexicon.default().add("st", kind="credential"))) built a parser without the caller's credential. On the base, 3b's delta raised its suffix set guard for this input. _apply_choice now compiles its edits and refuses a result that falls short of the delta's sets, wrapped with the languages chosen (rules/D2). 4i-c replaces this with the decided raise that names the edit.

Choices made inside the decided design

Each is recorded in the new decisions entry, and each is easy to reverse if wrong:

1. add/remove refuse surnames and given_names too. The refusal names lexicon | Lexicon(surnames={...}). The RFC's item 3 says the lists change "through | ... or dataclasses.replace", and it rewrites customize.rst's add(surnames=...) example that way. Its migrate table, though, calls the lists "unchanged".
2. An add of a kind the word holds only off by default turns that reading on. For example, add("st", kind="generation") turns on the generation readings of the cs, hr, sk and sl files.
3. readings_of leaves out default and cased. LexiconReading.properties is a tuple of sorted pairs.

What 2.x could state that readings cannot

These turned up while translating the tests:

• a title that is also a given name written short (chas);
• a possibly-given particle beside an unambiguous credential;
• may_be_initial on a caseless letter;
• a three-word title phrase.

A 2.x pickle holding any of these is refused, naming the fields. An orphan marker, a marker entry whose base set no longer holds the word (for example, remove(titles={"air"}) leaving air a modifier), reads as though the marker went with the word. This was decided on 2026-11-02 and is recorded in the decisions file. 2.x still read such an orphan in two places, and both readings change; the change is accepted:

• John Smith, AIR gives suffix AIR, where 2.8.0 gave given AIR;
• after remove(titles={"sir"}), Sir. Smith gives family Smith, where 2.8.0 gave given Smith.

_LEFT_BEHIND pins both readings.

A loaded 2.8 lexicon keeps the vocabulary it was pickled with. A pickled 2.8 Parser() loads as Lexicon(default + credential: -23; capitalization_exceptions: -13) and reads Jane Doe KC as 2.8.0 did; migrate.rst and the release bullet say so.

Measured

• No default parse moves. tests/v2/test_snapshot.py and test_cldr_agreement.py pass unchanged.
• Frames. The reference row is 311.82 on the base and 311.48 here (tools/perf/call_count.py --against 3804b1a1). _CALL_BASELINE is unchanged and the serial benchmark passes.
• Compile cost. A configured lexicon compiles in about 2.1 ms.
• Rules budget, re-recorded. R8's Accepted bullet grew from 323 to 329 words. R9 grew from 459 to 477 words in Accepted and shrank by one word elsewhere. Both changes are the amendment: the kind now reads "the lexicon's", with its examples. No new signal is read.
• One test expectation moved. Lexicon.default().add("st", kind="suffix") reads St as a post-nominal. 4b's index lookup read it as the cs file's generation, which is written off by default. This is the R8 amendment.

Docs

• docs/customize.rst is rewritten in reading terms:
  • the kinds and properties table;
  • title detection off;
  • |;
  • the ambiguous-word sections.
• docs/migrate.rst gains "Lexicon fields → readings", one row per 2.x field.
• docs/locales.rst, the API reference (LexiconReading), the release-log bullet, nameparser/vocab/README.md, and AGENTS.md's layering, pickling, invariants and Gotchas paragraphs are updated.

Verification

• uv run pytest
• uv run pytest -n0 -m serial
• uv run mypy
• uv run ruff check
• sphinx-build -b html and -b doctest
• python -m doctest README.rst
• the suite with and without namedivider, and on --python 3.15

Left for later

• 4i-c: the choice recorded on the Lexicon, and _apply_choice retired.
• A latent bug that predates this PR: parser_for's wrap of a choice error re-raises type(exc)(msg), which fails on _lexicon._MaskValueError (three-argument __init__). No shipped choice changes a mask, so nothing reaches it today.

Part of #754.

🤖 Generated with Claude Code