Skip to content

Record the choice of languages on the Lexicon (#754, 4i-c) - #992

Merged
derek73 merged 9 commits into
masterfrom
feat/lexicon-choice-4i-c
Nov 2, 2026
Merged

derek73 merged 9 commits into
masterfrom
feat/lexicon-choice-4i-c

Conversation

@derek73

@derek73 derek73 commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

Sub-phase 4i-c of #754: the choice of languages is recorded on the Lexicon, so a choice composes with a caller's readings instead of being applied as a delta.

What changes

  • The choice is recorded. Lexicon.languages and Lexicon.strict are read-only, and Parser.languages and Parser.strict read through to them. Only parser_for sets them publicly. Underneath they are the constructor parameters _languages and _strict, beside _base and _edits, so dataclasses.replace and pickle carry them. They are validated as parser_for validates a choice.
  • Chained choices union. parser_for(EN, base=parser_for(PT)) reads as parser_for(PT, EN): Maria do Carmo gives family do Carmo, and John Smith DO gives suffix DO. Under 3b's delta these gave family Carmo and family DO.
  • A caller's readings apply after the choice. A base that adds do and van as particles keeps both under en. Under strict vi, a base that adds dr keeps it. The delta dropped both.
  • A clash raises at parser_for, naming the edit. For example, locales.CS with a caller's st credential raises: while applying the languages chosen, cs: the lexicon's .add('st', kind='credential') clashes with a reading the choice turns on: ... .remove('st', kind='generation'). The offered remove runs, because it applies after the choice.
  • | refuses two different choices or strictnesses, and the error names parser_for.
  • parser_for's strict has three states (bool | _Unset = UNSET), following the segmenter precedent:
    • Left out: the call keeps the base's strictness. This covers policy-only packs and segmenter swaps on a strict parser.
    • strict=False on a strict base: raises ValueError, and the error offers both spellings.
    • strict=True on a plain base with a choice: makes the combined choice strict.
  • A pack's fragment joins before the caller's lexicon, so the caller's readings and masks win. Of two packs, the later one wins. A shipped mask that the caller's lexicon holds unchanged is not the caller's own statement, so a pack's mask of that word wins, just as a pack's readings beat the shipped ones.
  • A choice over a base built from Lexicon.empty() turns no reading on.
  • Retired: _parser._apply_choice, _choice_delta and _choice_readings, along with _lexicon._MARKER_BASES and _edits_toward's hints. The sweep test_a_choice_written_as_readings_gives_the_deltas_sets is rewritten as test_a_choice_moves_only_the_words_the_callers_log_leaves_alone, over the same 161 tags and three lexicons.
  • Latent bug fixed: parser_for's raise type(exc)(msg) wrap failed on _lexicon._MaskValueError, whose constructor takes three arguments. The wrap now raises a plain TypeError or ValueError, and a test covers both types.

Review round (code review and design-docs review)

Fixed:

  • A strict base could not be chained with a pack that chooses no language. The second review found that the first fix made an explicit strict=False on a strict base return a strict parser without saying so. The UNSET default, decided on that review, fixes both, and the explicit-False row fails on cf3f442b.
  • A pack's mask could not override a shipped mask.
  • Stale 3b/3j present-tense passages in the RFC; the release-log bullet's undefined en/pt; the locales.rst wording on an empty base.
  • The kind-set branch of _edits_toward was dead once the delta retired, so it is removed, with its test.

Recorded as an accepted limit: a mask the caller deleted comes back when a strict choice is chained, because undoing the base's mask moves cannot tell the caller's own deletions from the choice's. The 4i-c decisions entry carries this limit with its example.

The new rows for the two fixes fail on the pre-fix commit.

Choices made inside the decided design are recorded in the new decisions entry (2026-11-02 #754, sub-phase 4i-c). The main one is the case masks. A mask is no reading, and a choice still moves masks as 3b's delta did (cs gives judr the mask JUDr). parser_for moves them in the field: it undoes the base's choice's moves, then applies the union's, so a chained choice moves the masks as the union does.

Docs

  • docs/locales.rst: "Chaining choices" and "Your own words outrank a choice" replace "Choosing in one call".
  • Decisions: a new 2026-11-02 Phase 4: order and shapes, the rules a pack can choose, and the output model #754, sub-phase 4i-c entry. The 3b chained-choices entry and 3j's dropped-listing clause are marked in effect.
  • Also updated: the RFC's status line and work-order row; mechanisms.md#LOCALE-PACKS-PURE-DATA; AGENTS.md (the vocabulary paragraph, the invariants bullet and the pickling bullet); the _locale docstrings; and a release-log bullet under Unreleased.

Measured (over db7e8fc4, Python 3.11.17)

  • Default parses: tests/v2/test_snapshot.py is unchanged.
  • Agreement records: uv run python tools/differential/cldr_agreement.py reports every view matching its record, the per-pack CLDR and Wikidata native views included, with JA measured with the ja extra installed. No pack moves, because no shipped fragment holds a reading or a mask.
  • Frames: tools/perf/call_count.py --against db7e8fc4 gives 311.48 on both, and the serial benchmark passes.
  • Checks run: uv run pytest with and without namedivider, uv run pytest -n0 -m serial, uv run --python 3.15 pytest, uv run mypy, uv run ruff check, sphinx-build -b doctest docs (586 tests, 0 failures) and python -m doctest README.rst.

Part of #754.

🤖 Generated with Claude Code

derek73 and others added 2 commits November 2, 2026 11:22
Lexicon gains _languages/_strict (read as languages/strict, with
Parser.languages/strict reading through); a choice filters the base's
readings and the caller's log applies after it. parser_for unions a
base's choice with its packs', joins the packs' fragments under the
caller's lexicon, and refuses a clash naming the edit. _apply_choice
and its delta retire.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…754, 4i-c)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73 derek73 added this to the 3.0 milestone Nov 2, 2026
@derek73 derek73 added enhancement breaking-change Backwards-incompatible API change vocabulary Per-word vocabulary model and language packs labels Nov 2, 2026
@derek73 derek73 self-assigned this Nov 2, 2026
@derek73 derek73 added design docs Documentation fixes and updates tests and removed breaking-change Backwards-incompatible API change labels Nov 2, 2026
derek73 and others added 7 commits November 2, 2026 11:37
…ck's mask beat a shipped one (#754, 4i-c review)

A chained call choosing no language now keeps base's choice, its
strictness included, where the first draft raised on
parser_for(locales.RU, base=strict_parser). A shipped mask the caller
left unchanged yields to a pack's, as shipped readings do. The kind-set
branch of _edits_toward went with the retired delta. The RFC's 3b and
3j passages are put in the past tense.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Omitted (UNSET) keeps base's recorded strictness; strict=False on a
strict base raises again, as decided; strict=True on a plain base with
a choice makes the union strict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… 4i-c)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…load 4i-b pickles (#754, 4i-c review)

parser_for moves base's masks by the choice and lays the packs' over
them, as before 4i-c; a failed join or choice compile names the first
pack whose fragment fails it (rules/D2), or the languages and the
packs that chose them; _languages and _strict carry _ADDED_IN, so a
4i-b pickle loads choosing nothing (fixture written on db7e8fc).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fallback (#754, 4i-c)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…yields to a pack's (#754, 4i-c)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r the translation's branches (#754, 4i-c)

The translation of a 2.x agnomen offered .add(w, kind='agnomen')
before .add(w, kind='family'), which add() refuses; name readings now
come first. The retired choice delta was the other path through the
translation's property and kind branches, so the refusal table gains
rows reaching them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

design docs Documentation fixes and updates enhancement tests vocabulary Per-word vocabulary model and language packs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant