diff --git a/AGENTS.md b/AGENTS.md index 806f0c13..142da236 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -212,7 +212,7 @@ Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely o ### Vocabulary (`nameparser/vocab/`) and the configuration layer (`nameparser/config/`) -**The default vocabulary is written per word, in TOML, and compiled.** `nameparser/vocab/.toml` holds one file per BCP 47 language tag, each `[[word]]` entry a `text` and its readings keyed by kind. What a file means, how an entry is built, the kinds and properties with the set each compiles into, exclusions, bearer counts and the steps for adding a word are in `nameparser/vocab/README.md`, the reference for anyone editing the TOML; `tests/v2/test_vocab.py` holds its kind, property, derived-marker and reserved-name tables to the names `build.py` accepts and the sets it compiles, so read it rather than restating it here. The parser never reads the TOML: `tools/vocab/build.py` runs the build checks that replaced the modules' import-time asserts (its docstring numbers them), turns every reading into an index row with its file's tag, and writes the rows, and every file's tag (`FILES`, a file holding only exclusions included, for a strict choice to name), as the committed reading index `nameparser/config/_index.py` (#753, phase 3a) and, compiled, as the committed `nameparser/config/_compiled.py`, frozenset literals one entry per line; the build never reads `_index.py` back. A file a shipped pack reads may also hold a `[rules]` table, each key a `Policy` field (`patronymic_rules`, `name_order`, `two_family_names`, `two_given_names`), which the build checks against its `RULE_KEYS` and `PACK_RULES` and writes as `nameparser/config/_rules.py`; `_locale.rules_patch(tag)` turns a file's table into the `PolicyPatch` a pack carries, RU's and TR_AZ's from `ru.toml` and `tr.toml`, and the `two_family_names` of ES, CS and SK from `es.toml`, `cs.toml` and `sk.toml` (#754, 4f; decisions/vocabulary-model's 2026-10-29 4f entry). The compiler is `nameparser/_reading_index.py`'s `compile_readings`, with the kind and property tables it reads (`CATEGORY`, `BEHAVIORAL`, `GATED_WORDS`), so the build and a choice of languages compile alike: `Locale.languages` (3.0, phase 3b) is a choice of BCP 47 tags, which `parser_for` unions across its packs and base's recorded choice and records on the `Lexicon` (`Lexicon.languages`, 3.0's 4i-c), the lexicon compiling its base's readings through `choose` and its own edits after them, `choose` selecting the readings (a chosen file's `default = false` ones on; on a word a chosen file holds a vocabulary reading of, other files' vocabulary readings off, save `mul.toml`'s and `und.toml`'s; a chosen conjunction's `may_be_initial` dropped; under `parser_for(..., strict=True)`, 3j, only the chosen files and native-script vocabulary compiled, through `strict_pool`), `changed_words` deriving a choice's deviation surface which tests/v2/test_locales.py's gate reads, and `strict_ungated` the words a strict choice ungates against the plain one, which tests/v2/test_reading_index.py holds to a reviewed table (decisions/vocabulary-model, 2026-10-29 3b; 2026-10-31 #753, phase 3j); the build keeps the checks and loads that module by path, as it loads `_lexicon.py`, and `tests/v2/test_reading_index.py` holds that the index compiled with no choice is `_compiled.py` exactly. `readings_of(text)` there is the run-time reader of the index, every reading of a word with its file's tag, loaded on first call; `languages_of(text, categories)` is the reader a parse calls, from assemble, for `Reading.languages` (3.0), so the first parse that reads a word from its `Lexicon`'s vocabulary loads the index. Each `config/` module re-exports its constants from `_compiled.py` (`TITLES = _compiled.TITLES`) under its autodoc docstring, so the public names and the docs did not move. `tests/v2/test_vocab.py` fails when `_compiled.py` is stale, and `tests/v2/test_reading_index.py` when the index is, each naming the refresh command, and `test_vocab.py` gives every build check a mutation. The model, the phasing and what was declined: decisions/vocabulary-model. +**The default vocabulary is written per word, in TOML, and compiled.** `nameparser/vocab/.toml` holds one file per BCP 47 language tag, each `[[word]]` entry a `text` and its readings keyed by kind. What a file means, how an entry is built, the kinds and properties with the set each compiles into, exclusions, bearer counts and the steps for adding a word are in `nameparser/vocab/README.md`, the reference for anyone editing the TOML; `tests/v2/test_vocab.py` holds its kind, property, derived-marker and reserved-name tables to the names `build.py` accepts and the sets it compiles, so read it rather than restating it here. The parser never reads the TOML: `tools/vocab/build.py` runs the build checks that replaced the modules' import-time asserts (its docstring numbers them), turns every reading into an index row with its file's tag, and writes the rows, and every file's tag (`FILES`, a file holding only exclusions included, for a strict choice to name), as the committed reading index `nameparser/config/_index.py` (#753, phase 3a) and, compiled, as the committed `nameparser/config/_compiled.py`, frozenset literals one entry per line; the build never reads `_index.py` back. A file a shipped pack reads may also hold a `[rules]` table, each key a `Policy` field (`patronymic_rules`, `name_order`, `two_family_names`, `two_given_names`), which the build checks against its `RULE_KEYS` and `PACK_RULES` and writes as `nameparser/config/_rules.py`; `_locale.rules_patch(tag)` turns a file's table into the `PolicyPatch` a pack carries, RU's and TR_AZ's from `ru.toml` and `tr.toml`, and the `two_family_names` of ES, CS and SK from `es.toml`, `cs.toml` and `sk.toml` (#754, 4f; decisions/vocabulary-model's 2026-10-29 4f entry). The compiler is `nameparser/_reading_index.py`'s `compile_readings`, with the kind and property tables it reads (`CATEGORY`, `BEHAVIORAL`, `GATED_WORDS`), so the build and a choice of languages compile alike: `Locale.languages` (3.0, phase 3b) is a choice of BCP 47 tags, which `parser_for` unions across its packs and base's recorded choice and records on the `Lexicon` (`Lexicon.languages`, 3.0's 4i-c), the lexicon compiling its base's readings through `choose` and its own edits after them, `choose` selecting the readings (a chosen file's `default = false` ones on; on a word a chosen file holds a vocabulary reading of, other files' vocabulary readings off, save `mul.toml`'s and `und.toml`'s; a chosen conjunction's `may_be_initial` dropped; under `parser_for(..., strict=True)`, 3j, only the chosen files and native-script vocabulary compiled, through `strict_pool`), `changed_words` deriving a choice's deviation surface which tests/v2/test_locales.py's gate reads, and `strict_ungated` the words a strict choice ungates against the plain one, which tests/v2/test_reading_index.py holds to a reviewed table (decisions/vocabulary-model, 2026-10-29 3b; 2026-10-31 #753, phase 3j); the build keeps the checks and loads that module by path, as it loads `_lexicon.py`, and `tests/v2/test_reading_index.py` holds that the index compiled with no choice is `_compiled.py` exactly. `readings_of(text)` there is the run-time reader of the index, every reading of a word with its file's tag, loaded on first call; `languages_of(text, categories)` is the reader a default parse calls for `Reading.languages` (3.0), through the lexicon's private `_files` (`_lexicon._ReadingFiles`, one per base, log and choice, hashed by identity so assemble's memo keyed by it costs no frame), which under a choice or a caller's log names the files whose reading is on in that lexicon instead (3.0's 4i-d), so the first parse that reads a word from its `Lexicon`'s vocabulary loads the index. Each `config/` module re-exports its constants from `_compiled.py` (`TITLES = _compiled.TITLES`) under its autodoc docstring, so the public names and the docs did not move. `tests/v2/test_vocab.py` fails when `_compiled.py` is stale, and `tests/v2/test_reading_index.py` when the index is, each naming the refresh command, and `test_vocab.py` gives every build check a mutation. The model, the phasing and what was declined: decisions/vocabulary-model. The `config/` SETS are frozen since 2.2 (#293): there is no `.add()`/`.remove()` on any of them, so a default word list is changed at runtime by configuring a `Lexicon`, never by editing the constant, and the shipped default is changed by editing the TOML and rebuilding. **Neither dict was frozen, and `CAPITALIZATION_EXCEPTIONS` is not covered by anything else either.** `REGEXES` is a compiled-pattern table rather than vocabulary and was never in #293's scope; `CAPITALIZATION_EXCEPTIONS` is vocabulary-shaped and is a decided, in-scope exemption. So the hazard the freeze closes is still live for it: `CAPITALIZATION_EXCEPTIONS['dclinpsy'] = 'DClinPsy'` after the first parse misses the cached `Lexicon.default()`. Same advice — configure the object (`dataclasses.replace(lexicon, capitalization_exceptions=...)`). `tests/v2/test_contracts.py::test_every_vocabulary_constant_is_frozen` names both dicts as explicit exemptions rather than letting its `isinstance` filter drop them. @@ -234,7 +234,7 @@ The `config/` SETS are frozen since 2.2 (#293): there is no `.add()`/`.remove()` The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2.0 and then 3.0 deleted. These conventions apply to all of it. The full design record (rationale, settled-decision logs, dated amendments) lives in untracked `docs/superpowers/specs/`; this section is the enforceable subset. **A commit that establishes or amends one of these conventions must update this section in the same commit** — grep-driven staleness sweeps miss paraphrased prose (see Workflow above), so write-time maintenance is the mechanism, audits are the backstop. - **Module layout**: every module is underscore-private (`_types.py`, `_lexicon.py`, `_policy.py`, `_locale.py`, `_render.py`, `_reading_index.py`, `_pipeline/`, `_parser.py`). The public import surface is exactly `nameparser`, `nameparser.locales` and the vocabulary data modules under `nameparser.config`; `nameparser/__init__.py` holds re-exports and `__all__` only — no logic. -- **Layering is enforced by `tests/v2/test_layering.py`** (exact-module matching; `if TYPE_CHECKING:` imports don't count): `_types` imports nothing internal at module level — its rendering delegates import `_render` at call time; `_lexicon` and `_policy` sit above `_types` independently (`_lexicon` may import `nameparser.config.*` DATA modules, e.g. `config/maiden_markers.py`, and `_reading_index` at call time only, to compile a configured lexicon's readings, since `tools/vocab/build.py` loads it by path, 3.0's 4i-b); `_reading_index` imports nothing internal at module level and the index, `config/_index.py`, at call time, since `tools/vocab/build.py` loads it by path; `_locale` sits on `_lexicon`+`_policy` (plus `_types` for the shared pickle mixin, and `config/_rules.py` at call time, in `rules_patch`); `_render` imports `_types`, `_lexicon` (for `Lexicon.default()` and `_normalize`) and `_policy` (for the scripts that write no initials); `_pipeline/*` imports `_types`+`_lexicon`+`_policy` plus in-package `_pipeline` helpers, `_assemble` also `_reading_index`, for `Reading.languages` (`_vocab` imported it in 3.0's 4b and 4c for a listed suffix word's kind and a patronymic particle's, which since 4i-b the lexicon compiles into `_sets`); `_parser` sits on everything except `_render`, `_reading_index` at call time only, to validate a choice's tags and move the case masks it moves (the choice itself compiles in `_lexicon._built`); `__main__` imports only the public package; locale pack modules (`locales/*.py`) import `_locale`/`_lexicon`/`_policy`/`_types` only (`_types` joined the list in #272 — `locales/ja.py`'s segmenter factory constructs a `Segmentation`), and the `locales/__init__` additionally lazy-imports its packs (PEP 562). Extend the test's `ALLOWED` table when adding a module. +- **Layering is enforced by `tests/v2/test_layering.py`** (exact-module matching; `if TYPE_CHECKING:` imports don't count): `_types` imports nothing internal at module level — its rendering delegates import `_render` at call time; `_lexicon` and `_policy` sit above `_types` independently (`_lexicon` may import `nameparser.config.*` DATA modules, e.g. `config/maiden_markers.py`, and `_reading_index` at call time only, to compile a configured lexicon's readings, since `tools/vocab/build.py` loads it by path, 3.0's 4i-b); `_reading_index` imports nothing internal at module level and the index, `config/_index.py`, at call time, since `tools/vocab/build.py` loads it by path; `_locale` sits on `_lexicon`+`_policy` (plus `_types` for the shared pickle mixin, and `config/_rules.py` at call time, in `rules_patch`); `_render` imports `_types`, `_lexicon` (for `Lexicon.default()` and `_normalize`) and `_policy` (for the scripts that write no initials); `_pipeline/*` imports `_types`+`_lexicon`+`_policy` plus in-package `_pipeline` helpers, `_assemble` also `_reading_index`, for the kinds `Reading.languages` matches (`CATEGORY`; the files themselves it reads through `Lexicon._files`, 4i-d; `_vocab` imported it in 3.0's 4b and 4c for a listed suffix word's kind and a patronymic particle's, which since 4i-b the lexicon compiles into `_sets`); `_parser` sits on everything except `_render`, `_reading_index` at call time only, to validate a choice's tags and move the case masks it moves (the choice itself compiles in `_lexicon._built`); `__main__` imports only the public package; locale pack modules (`locales/*.py`) import `_locale`/`_lexicon`/`_policy`/`_types` only (`_types` joined the list in #272 — `locales/ja.py`'s segmenter factory constructs a `Segmentation`), and the `locales/__init__` additionally lazy-imports its packs (PEP 562). Extend the test's `ALLOWED` table when adding a module. - **Canonical field order** — the seven roles in `Role` enum declaration order, defined once and derived everywhere (properties, `as_dict`, reprs, `comparison_key`). Never restate the order literally. `Role` is a `StrEnum`: members compare as their field-name strings, and `tokens_for()` coerces strings via `_coerce_enum`. - **Method organization**, fixed section order in every class: fields + `__post_init__` validation → alternative constructors → dunders (construction/equality → protocol → operators) → properties → public methods by concern (access → editing → comparison → rendering delegates) → private helpers last, except a helper serving exactly one section may sit at that section's head. - **Validation is eager and fail-loud**: every `raise` states the offending value, the expected form, and the fix. Exception taxonomy: wrong type — including wrong element type inside a collection, bare `str` where an iterable of strings is expected, or a `Mapping` where a plain iterable is expected — raises `TypeError`; well-typed but unacceptable values raise `ValueError`; failed enum lookups stay `ValueError` for any input (stdlib `EnumType` precedent). **When the message hands the reader code to paste, that code has to survive a type checker, and then RUN** — nameparser ships `py.typed`, and a placeholder can type-check and still raise: the 2.4 release sweep's first fix to the `capitalization_exceptions` hint offered `('key', 'Value')`, which passes mypy and fails #459's mask check, so `test_add_capitalization_exceptions_raises_pointing_at_replace` now executes the offered code. #337's segmenterless warning offered `Policy(segment_scripts=())`, an `arg-type` error, because these fields are annotated with what they STORE rather than everything the constructor accepts. Prefer the `frozenset()` / `()` spellings in messages and docstrings, and pin the offered spelling in a test — the warning tests matched on `ja_segmenter` and never checked the actionable half of the message. **A warning emitted in `Parser.__post_init__` needs `parser_for` to re-emit it from its own frame** (the `catch_warnings(record=True)` block at its return): `__post_init__`'s `stacklevel` is sized for direct `Parser(...)` construction, and through `parser_for`'s extra frame the default one-line rendering attributes the warning to the library's own `return Parser(...)` — the exact call the message tells the user to change becomes invisible. No single stacklevel serves both entry points; a new construction warning gets the re-emission for free, but a new CONSTRUCTION SITE for `Parser` inside this package needs its own re-emission or its callers get library-attributed warnings (#337 review). @@ -250,7 +250,7 @@ The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2 - **A claim about WHICH STAGE or WHICH LAYER does something is checkable — check it before writing it.** The pipeline is eight stages with a written ownership map (`ParseState`'s docstring, pinned by `tests/v2/pipeline/test_state.py`), and `parse(s).tokens` prints every token's role and tags, so "extract assigns this", "classify never sees that", "group consumes it" each have a one-command answer. #329's prose claimed delimited maiden content is *"claimed whole before classify has tagged anything inside it"*; measured, `classify` tags the marker fine and only the CONSUMING is missing, because `_group`'s rule walks `pieces` and a token that already carries a role is not in `pieces`. Two different mechanisms, one plausible sentence covering both. That single claim then shipped SIX times across three correction rounds, which is the part worth internalizing: **when a mechanism claim turns out wrong, sweep for where else you wrote it, and sweep again at the END of the change over the words the change itself just added.** Prose density here means one idea lives in a docstring, a case note, a release-log entry and this file at once; the implementer working against a wrong mechanism is the person most likely to restate it; and rewriting a mechanism claim is writing one, so the correction earns the same one-command check as the original — two of the six instances were fresh errors introduced by the sentence fixing the previous one. **Adjacency is the trap.** The claim that feels already-known is the one about the neighbouring stage or the neighbouring layer: one comment block in the 2.x `tests/v2/test_facade_cases.py` got the exception type, the raising layer, the skip mechanism, the count of skipped rows, and which row was blocked all wrong at once — every one a claim about `_config_shim` versus real 1.4.0, written from reasoning, in the file whose whole job is translating between them. What finally held was not better prose but moving the claim into a test (`_CORE_ONLY_IDS`, deleted with the facade in 3.0), which cannot be wrong the way a sentence can. - **Typing/docs**: `from __future__ import annotations`; `frozen=True, slots=True` on every public dataclass; strict-profile mypy flags via per-module overrides in pyproject (`strict = true` itself is not valid per-module). Docstrings state contracts in prose with **no doctest blocks** — `--doctest-modules` makes every example a test; behavior examples go to unit tests per the lean-docs rule. **Document the positive direction of a partial property**: "a non-empty `ambiguities` is a signal to act on" is checkable, while "an empty one means no fork occurred" is a universal negative needing exhaustive verification -- that claim was written twice and falsified twice, at sites the author had not audited. - **The segmenter contract**: the optional `Parser(segmenter=...)` hook is parse-totality's ONE exception (locales spec section 4). Everything inside that exception is a bug in USER CODE, never a fact about the name, so it is surfaced rather than absorbed: the segmenter's own exceptions propagate, and the two protocol violations the stage can detect for itself — an answer of the wrong type, and one cutting at or past the end of the token it was handed — raise `TypeError`/`ValueError` from `_script_segment` for the same reason. The line to hold when adding a check there: a protocol violation by the segmenter's AUTHOR raises, while an adapter's defense against its own third-party library (`locales/ja.py`'s repertoire, length, reconstruction and score guards) declines with `None`, because what those catch is a fact about the content. -- **Pickling**: v2 types must round-trip (`Parser` is picklable by construction, and it holds a `Lexicon`; the one qualifier is that a `Parser` pickles iff its segmenter does — see the segmenter bullet above). Every frozen type assigns `_guarded_getstate`/`_guarded_setstate` (`_types.py`) in its class body (`@dataclass(slots=True)` would override inherited pickle methods) — unpickling fails at the LOAD site on field-layout skew, with one widening: a field added after its type shipped carries `metadata={_types.ADDED_IN: ""}` and a default, and an older pickle missing it loads with the default (`Ambiguity.alternative`, 2.7; `Token.reading`, `Reading.languages`, `Reading.second_surname` and `ParsedName.order`, 3.0.0) -- a missing field without the marker, default or not, an unknown field and a renamed one are still refused, each a row in `tests/v2/test_types.py::_REFUSED_SKEWS`, so a new field on a pickled type takes the marker or breaks every older pickle; the shared guard deliberately does NOT re-validate values (pickle is not a security boundary; canonical state comes from a validated instance). `Lexicon` and `LexiconReading` keep their own copy of the guard (layering); a new unpicklable slot type needs the same treatment plus a round-trip test. Since 3.0's 4i-b a `Lexicon` pickles its lists, its masks, its edit log, its choice of languages (`_languages` and `_strict`, 4i-c, marked `ADDED_IN` so a 4i-b pickle loads choosing nothing, a 2.x state translating to no choice) and its base as a SNAPSHOT of the reading index's rows, so a pickle parses as it did where it was written, a snapshot equal to the running release's index loading as the by-reference default again (decisions/vocabulary-model, 2026-11-02 #754, item 5); its private `_sets`, `_cap_map` and `_chosen` fields are built at construction and never pickled. `Lexicon`'s copy of the guard admits the `ADDED_IN` marker, read under `_lexicon._ADDED_IN`, the same string, since layering keeps `_lexicon` from importing `_types` (`test_lexicon_marks_an_added_field_with_the_shared_key` holds the two equal), so a new `Lexicon` field takes the marker and a default or breaks older pickles. A 2.x state holds the compiled sets instead, and `_lexicon._translated_2x` translates it onto the nearer named base and a per-word log, loading where the compile gives the sets back and refusing otherwise, naming the fields, so the 2.8 `Lexicon`, `Parser` and locale-pack pickles in `tests/v2/data/lexicon_v28.pickle` load as the 3.0 calls that build them. And unlike the shared guard it DOES re-validate: a load goes through the constructor, which checks the base, refuses an edit not in canonical form, and refuses a list or mask entry that is not already normalized, because an old blob would otherwise load a different lexicon in silence (`particles_ambiguous` changed meaning between v1 and v2 without changing its name). +- **Pickling**: v2 types must round-trip (`Parser` is picklable by construction, and it holds a `Lexicon`; the one qualifier is that a `Parser` pickles iff its segmenter does — see the segmenter bullet above). Every frozen type assigns `_guarded_getstate`/`_guarded_setstate` (`_types.py`) in its class body (`@dataclass(slots=True)` would override inherited pickle methods) — unpickling fails at the LOAD site on field-layout skew, with one widening: a field added after its type shipped carries `metadata={_types.ADDED_IN: ""}` and a default, and an older pickle missing it loads with the default (`Ambiguity.alternative`, 2.7; `Token.reading`, `Reading.languages`, `Reading.second_surname` and `ParsedName.order`, 3.0.0) -- a missing field without the marker, default or not, an unknown field and a renamed one are still refused, each a row in `tests/v2/test_types.py::_REFUSED_SKEWS`, so a new field on a pickled type takes the marker or breaks every older pickle; the shared guard deliberately does NOT re-validate values (pickle is not a security boundary; canonical state comes from a validated instance). `Lexicon` and `LexiconReading` keep their own copy of the guard (layering); a new unpicklable slot type needs the same treatment plus a round-trip test. Since 3.0's 4i-b a `Lexicon` pickles its lists, its masks, its edit log, its choice of languages (`_languages` and `_strict`, 4i-c, marked `ADDED_IN` so a 4i-b pickle loads choosing nothing, a 2.x state translating to no choice) and its base as a SNAPSHOT of the reading index's rows, so a pickle parses as it did where it was written, a snapshot equal to the running release's index loading as the by-reference default again (decisions/vocabulary-model, 2026-11-02 #754, item 5); its private `_sets`, `_cap_map`, `_chosen` and `_files` fields are built at construction and never pickled. `Lexicon`'s copy of the guard admits the `ADDED_IN` marker, read under `_lexicon._ADDED_IN`, the same string, since layering keeps `_lexicon` from importing `_types` (`test_lexicon_marks_an_added_field_with_the_shared_key` holds the two equal), so a new `Lexicon` field takes the marker and a default or breaks older pickles. A 2.x state holds the compiled sets instead, and `_lexicon._translated_2x` translates it onto the nearer named base and a per-word log, loading where the compile gives the sets back and refusing otherwise, naming the fields, so the 2.8 `Lexicon`, `Parser` and locale-pack pickles in `tests/v2/data/lexicon_v28.pickle` load as the 3.0 calls that build them. And unlike the shared guard it DOES re-validate: a load goes through the constructor, which checks the base, refuses an edit not in canonical form, and refuses a list or mask entry that is not already normalized, because an old blob would otherwise load a different lexicon in silence (`particles_ambiguous` changed meaning between v1 and v2 without changing its name). - **One sanctioned global**: the (future) cached default `Parser`. Lazily cached FROZEN singletons (`Lexicon.default()`'s `functools.cache`, the future default parser) are constants, not state; any second piece of module-level MUTABLE state requires amending the conventions doc, on purpose, in review. - **Tests**: all v2 tests live in the `tests/v2/` package, one test module per source module plus the cross-cutting ones (`test_reprs.py`, `test_layering.py`, `test_contracts.py`, `test_properties.py`, `test_benchmark.py`, the `cases.py`/`test_cases.py` table, `test_regex_sync.py`, which pins the pipeline's hand-copied patterns against their sources, and `test_snapshot.py`, which holds every corpus name's reading to `tools/differential/snapshot.jsonl` — sharing `_differential_fixtures.py` with `test_differential.py` and `test_corpora.py` for the handles on `tools/differential/`), names stating behavior; the 1.x suite's parsing tests stand ported to `parse()` in `tests/*.py`. Never assert `Lexicon.default()` contents; the narrow sourcing spot-checks in `test_default_sources_v1_vocabulary` that pin the v1→v2 migration contract (e.g. the flipped `particles_ambiguous` model) are the sanctioned exception. diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 11fdd0fa..523b9e59 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -30,7 +30,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - Name readings are hand-declared, and only on a word some language file claims as vocabulary or refuses, so the vocabulary carries no surname list (the segmentation lists, Korean and the ZH pack's, are outside the per-word model) — 2026-10-15/16; 2026-10-19. - Bearer counts are structured provenance (source and count), never used to compute a reading — 2026-10-15/16. - Phase 2 is to put vocabulary, declared name and shape readings in one pool, derive `may_be_given` and `may_be_name` from declared name readings (a count-free one carrying a `note`, `ld` left ungated, `may_be_initial` kept), and decide every clash by one gate of four steps, floor, form, structure and count, asked at the sites that decide today; the reading taken is to surface as `Token.reading` in 3.0 — 2026-10-23 #752. The reading not taken surfaces as `Ambiguity.alternative` since 2.7 (decisions/A1) — 2026-10-26 #752, phase 2m. -- Every reading the parse took or a report names from the vocabulary names the shipped files holding it, `Reading.languages`, looked up at assemble in the reading index by the word and the kind, a name by the field it lands in; the default's files whatever `Lexicon` the parse used, and none for a shape, position, or a word no file holds in that category — 2026-10-29 #754, 4a's follow-up. +- Every reading the parse took or a report names from the vocabulary names the shipped files holding it, `Reading.languages`, looked up at assemble in the readings of the parse's `Lexicon` (the reading index's rows for the default) by the word and the kind, a name by the field it lands in; the files whose reading is on in the `Lexicon` the parse used, so a choice names the files it reads and not those it drops, and none for a shape, position, a caller's own reading, or a word no file holds in that category — 2026-10-29 #754, 4a's follow-up; 2026-11-02 #754, sub-phase 4i-d. - Every numeral rule asks one predicate, `_vocab.reads_as_generation`, the vocabulary's own numerals read inside it; a numeral of two or more letters is the generation wherever the trailing run holds it, while a one-letter one before a credential stays the name word decisions/S10's #721 entry made it — 2026-10-25 #752, phase 2f. - A credential written split across a space is one word, a member of the ambiguous class that the count decides, and the Ph./D. pair starts S8's run from where it stands (decisions/S12) — 2026-10-25 #752, phase 2e. - A listed credential that is also a name, written in capitals in a name of mixed case, is settled as the credential by its writing: it reports nothing and starts the credential's run where no name word stands behind it with no family comma (2026-10-26, Derek), and keeps its other readings, the clash settled at the sites that choose a reading rather than by retagging the word (decisions/S6) — 2026-10-25 #752, phase 2g, #785. @@ -299,7 +299,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 3. **Not filled**, `None`: a token `ParsedName.replace()` builds, a token `Parser.revise()` harvests, whose role the caller forced, and a token loaded from a pickle written before 3.0 (`ADDED_IN = "3.0.0"`; `tests/v2/data/parsedname_v28.pickle`, made by the released 2.8.0 wheel, loads and equals a fresh parse). `capitalized()` carries each reading through. 4. **Cost**, measured 2026-10-29 with `tools/perf/call_count.py`: the reference row moved from 305 to 306 frames on Python 3.11 and from 286 to 287 on 3.15, one frame for its suffix `III`. Every `Reading` a token can take is built once at import (280 of them, ten kinds by seven roles by four sources) and shared, where a construction per token would cost two frames a word; an untagged word, the commonest, is read in C-level tests with no frame. `_CALL_BASELINE` (307 and 288) is not moved. 5. **Left open.** A listed suffix word that the compiled sets do not split by kind reads `post-nominal`, so `Jr`, `Sr` and `Esq.` read alike: the TOML gives `jr` a `generation` reading, but `SUFFIX_WORDS` holds both kinds and classify tags them alike, so only a vocabulary lookup by kind can tell them apart: 3a's reading index, a new `Lexicon` field being set aside (the 2026-10-28 `Lexicon` surface entry makes the compiled sets internal, and `Lexicon`'s own pickle guard refuses an added field). It is the predicate's own answer, which `Ambiguity.alternative` has given since 2.7 (the snapshot's one post-nominal alternative is `Rinpoche`'s), and a kind left unset would have meant a parsed token with no reading or a kind for "not yet known", so the reading taken agrees with it until 4b, whose `generation` view needs the split, reads a listed word's kind from 3a's index; `tests/v2/pipeline/test_assemble.py::test_a_suffix_word_reads_as_its_vocabulary_kind` holds `Jr` and `Sr` as strict expected failures until then. `ReadingKind.PATRONYMIC` exists and no reading takes it until 4c, whose rules write it. This amends the 2026-10-28 #754 entry's item 9: 4b needs 3a for that kind. -- **2026-10-29 #754, 4a's follow-up — `Reading.languages` IS FILLED AT ASSEMBLE FROM 3a's READING INDEX, FOR THE READING A TOKEN TOOK AND FOR A REPORT'S ALTERNATIVE; NO PARSE MOVES.** Completes 4a (the 2026-10-29 #754, sub-phase 4a entry) with the field item 2 of the 2026-10-28 #754 entry names. Until it, no parse read the index (the 2026-10-29 #753, phase 3a entry's load bullet); since it, the first parse that reads a word from the vocabulary of its own `Lexicon` loads it (item 4). +- **2026-10-29 #754, 4a's follow-up (item 1's "names the default's files whatever `Lexicon` the parse used", and that a reading written `default = false` names none, superseded by 2026-11-02 #754, sub-phase 4i-d) — `Reading.languages` IS FILLED AT ASSEMBLE FROM 3a's READING INDEX, FOR THE READING A TOKEN TOOK AND FOR A REPORT'S ALTERNATIVE; NO PARSE MOVES.** Completes 4a (the 2026-10-29 #754, sub-phase 4a entry) with the field item 2 of the 2026-10-28 #754 entry names. Until it, no parse read the index (the 2026-10-29 #753, phase 3a entry's load bullet); since it, the first parse that reads a word from the vocabulary of its own `Lexicon` loads it (item 4). SUPERSEDES: decisions/A1's 2026-10-26 #752 (phase 2m) entry's decline of the reading's `languages`, for 3.0; its reason, that the index describes the default and not the `Lexicon` a parse used, is answered by item 1's semantics rather than removed. 1. **What it holds.** The BCP 47 tags of the shipped files holding the reading taken, on by default, each once, sorted (a word joined from parts and an alternative over several words sorting the union), read through `_reading_index.languages_of(text, kinds)`, which walks the index rows without building an `IndexReading`. A reading matches the rows of every vocabulary kind of its category: a `title`, `particle` or `conjunction` reading its category's rows; a `name` reading the rows listing the word as a name of the field it lands in, `family` for a family or maiden word and `given` for a given or middle one, with the `bound_given` rows in either (a nickname takes any name rows), and a middle word `middle_as_family` folds into the family by the middle it stands as, the policy choosing the reported field and not the files (`John Ma Smith` under it names `en fil`, as without it), so `Ma` read as a family name (`Long Ma`) names `en` and not `fil`, which lists it only as a given name, and a declared name read into a field no file lists it in names none (`van` as a family name, which `nl` lists as given); and every suffix kind the word's suffix rows, a word having one suffix kind (the build's check 4), so `Jr`, which reads `post-nominal` until 4b, names `en`, the file of its `generation` reading. Only a `vocabulary` or `declared` reading is looked up; a shape, position, an initial or a patronymic names no file. A reading written `default = false` names none (`St` in a caller's `suffix_words`, held off by `cs.toml`), and neither does a word a caller's own `Lexicon` adds that no shipped file holds in that category (`Zzq` as a title), while one a file does hold names it (`Dean` added to `titles` names `en`), nor the Korean surnames that divide unspaced hangul, which `config/surnames.py` keeps outside the per-word files: `김` in `김민준` reads `name`, `vocabulary`, with no files. It names the default's files whatever `Lexicon` the parse used, the semantics decisions/A1's 2026-10-26 decline weighed against it: a file holds the reading or not, and which lexicon read it is the caller's own knowledge. Under a choice of languages (3b) a "live" reading becomes one the choice makes, and `languages_of` filtering on `on_by_default` is the place 3b changes. 2. **How a word is looked up.** By the vocabulary's own form of the word (`_normalize`), then with every period removed, as a credential is matched (`M.A` in `John Smith (M.A)`) and as a word listed whole without them is (`Lt.Gen.` as `ltgen`); a credential split across a space whole, its first word joined to the second (`Ph.` `D.` as `phd`), the second taking the first's reading, past a nickname written between them too (`John Smith Ph. "Bob" D.`), where the second word read `credential`, `position` under 4a alone and now takes the first's `credential`, `vocabulary`: the one `Token.reading` this change moves, no role, tag or alternative moving with it; and a word joining listed words with punctuation by its parts, in order, a suffix's parts divided by any mark but the period first (`M.D./Ph.D.`, `MD;`), then any word's by every mark (`Lt.Gov.`), never at a combining mark, which `\W` would count (`प्रोफे.डॉ.` names `hi` and `mr` through `डॉ`), and only where the parse's own `Lexicon` does not list the word whole (in any form, its periods kept or removed): a caller's own `zzq-dr` title or `mac-van` particle was read whole, so it names none of `dr`'s or `van`'s files. An alternative names the files for any word its report names, each read in the field the alternative names. MEASURED 2026-10-29 on `bd02d68f` plus this change over the snapshot's 6,360 distinct names, the empty one among them (20,804 tokens): 3,935 readings are `vocabulary` or `declared` and 3,750 of them name a file; of the 185 that name none, 157 are Korean surnames split off by the list and 28 are declared names read into a field no file lists them in (`van` 17 as a family name, `le` 3, `san`, `la` and `fils` 2 each, `fils` once more and `von` once). Matching name rows by category rather than by field, as this entry's first draft did, 3,778 named a file, all of the 157 aside. By the normalized and period-free forms alone, without the split and the parts, the first draft counted 265 naming none: the 157, 42 words joining listed words, and 66 words of credentials split across a space, 40 by `split:credential` (`Ph.` `D.`) and 26 by `split:member` (`Pharm D`). Recompute: parse every `name` of `tools/differential/snapshot.jsonl` and count the tokens whose `reading.source` is `vocabulary` or `declared` by whether `reading.languages` is empty; for the second figure, look each up with `languages_of` by its normalized and period-free forms only, and sort the misses by the split tags on the word and the word before it. @@ -315,7 +315,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **The no-choice path does not move.** tests/v2/test_snapshot.py passes unchanged; `parser_for` with packs choosing no language returns the base lexicon's value; and `tools/differential/measure.py --base` a checkout of `bd02d68f` reports no move in any committed or local corpus check. - **The deviation surface is derived (item 7).** tests/v2/test_locales.py's `test_non_interference_each_choice` runs every corpus name under `pt`, `en`, `cs`, `de` and `pt` with `en`, requiring the default's reading for every name holding no word of `changed_words` (matched over-wide: each piece whole, periods removed, cut at non-word characters, and a non-ASCII word anywhere), about 1 s a choice on one worker; its negative control removes `e` from `pt`'s surface and fails on the names it moves. tests/v2/test_reading_index.py holds, for every file tag chosen alone, that a choice changes only words its selected files hold. decisions/deviates-registry records the registry half: a registered vocabulary pack's `DEVIATES` is the derived surface from 3d. - (Superseded by 2026-11-02 #754, the `Lexicon` surface, effective with 4i-c on 2026-11-02: a chained choice now unions with the base's.) **A choice chained through `base=` is applied on top of the base's, not unioned with it**, decided in the next entry (option (a) of three weighed there). A `Parser` records no choice, so `parser_for(EN, base=parser_for(PT))` applies `en`'s delta to a lexicon `pt` already changed: `Maria do Carmo` reads family `Carmo` and `John Smith DO` family `DO`, where `parser_for(PT, EN)` keeps both readings (`do Carmo`, `Smith`). Item 1's order-independence holds within one call. - - **`Reading.languages` does not follow a choice: it names the files holding the reading on by default, whatever choice built the parser.** SUPERSEDES: the 4a follow-up entry's expectation, in its item 1, that 3b changes `languages_of`'s filter on `on_by_default`. Measured on this branch merged with that follow-up: under `pt`, the one-case `E` of `ANA SILVA E SOUZA` reads `conjunction` with languages `('it', 'pt')`, though prefer dropped `it`'s reading from the pool; under `cs`, the `st.` of `Jiří Dienstbier st.` reads `generation` with languages `()`, though `cs.toml` holds the reading the choice turned on. Kept, for two reasons: the follow-up decided the field names the default's files whatever `Lexicon` the parse used, a caller's own `St` in `suffix_words` naming none for the same `cs` reading; and a parse cannot know its choice, a `Parser` recording none (the chained-choices entry below), so a field that followed the choice needs the `Parser.languages` left open there. The cost is that a reading a choice turns on names no file and a language a choice dropped is still named; tests/v2/test_parser.py's `test_reading_languages_name_the_default_files_under_a_choice` pins both, so a change with `Parser.languages` moves them deliberately. + - (Superseded by 2026-11-02 #754, sub-phase 4i-d: the field names the files whose reading is on in the parse's own `Lexicon`, which records its choice since 4i-c.) **`Reading.languages` does not follow a choice: it names the files holding the reading on by default, whatever choice built the parser.** SUPERSEDES: the 4a follow-up entry's expectation, in its item 1, that 3b changes `languages_of`'s filter on `on_by_default`. Measured on this branch merged with that follow-up: under `pt`, the one-case `E` of `ANA SILVA E SOUZA` reads `conjunction` with languages `('it', 'pt')`, though prefer dropped `it`'s reading from the pool; under `cs`, the `st.` of `Jiří Dienstbier st.` reads `generation` with languages `()`, though `cs.toml` holds the reading the choice turned on. Kept, for two reasons: the follow-up decided the field names the default's files whatever `Lexicon` the parse used, a caller's own `St` in `suffix_words` naming none for the same `cs` reading; and a parse cannot know its choice, a `Parser` recording none (the chained-choices entry below), so a field that followed the choice needs the `Parser.languages` left open there. The cost is that a reading a choice turns on names no file and a language a choice dropped is still named; tests/v2/test_parser.py's `test_reading_languages_name_the_default_files_under_a_choice` pins both, so a change with `Parser.languages` moves them deliberately. - Not touched: `Lexicon` and `Parser` gain no field; `_locale.py` imports nothing new beyond `_types.ADDED_IN` and `_policy`'s string-and-mapping guard; `_parser.py` imports `_reading_index` at call time, and tests/v2/test_layering.py's table says so. 4f's `[rules]` compiled into a `Locale`'s policy patch needs no change to this shape: a pack's policy patch and its `languages` are separate fields. Strict (item 4) is not built. - **2026-10-29 #753, phase 3b, chained choices (Derek) — SUPERSEDED by 2026-11-02 #754, the `Lexicon` surface (in effect since 4i-c, 2026-11-02 #754, sub-phase 4i-c) — ONLY ONE CALL'S CHOICE COUNTS: A CHOICE CHAINED THROUGH `base=` APPLIES ON TOP OF THE BASE'S, AND A CHOICE DROPS THE CALLER'S OWN BASE ENTRIES THAT IT REMOVES.** Derek decided in chat on 2026-10-29, choosing option (a) of the three the phase 3b entry above weighed. `parser_for` unions the `languages` of the packs passed to it and applies that one choice as a delta against the default's compile; a `Parser` records no choice, so a base built by an earlier `parser_for` carries its choice only as the lexicon it changed. Two consequences, both pinned by tests in tests/v2/test_parser.py so that a change to them is deliberate: 1. **Chained choices compose one after the other.** `parser_for(EN, base=parser_for(PT))` reads `Maria do Carmo` family `Carmo` and `John Smith DO` family `DO`: `en` drops `pt`'s particle and `pt` had dropped `en`'s credential. The way to choose both is one call, `parser_for(PT, EN)`, which keeps both readings (`do Carmo`, suffix `DO`). @@ -512,3 +512,9 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **Accepted: a mask the caller deleted comes back when a strict choice is chained.** Undoing base's choice's mask moves restores the shipped value of each mask that choice dropped, and cannot tell one the caller deleted first: with `bsc` deleted from the default's masks, `parser_for(EN, base=parser_for(KO, base=Parser(lexicon=mine), strict=True), strict=True)` holds `bsc` and `parser_for(KO, EN, base=Parser(lexicon=mine), strict=True)` does not (found by the code review of this PR). Telling them apart would need the masks to be a log as the readings are, which item 3 declined to make them. - **What moved, each the decided supersession.** `parser_for(EN, base=parser_for(PT))` reads `Maria do Carmo` family `do Carmo` and `John Smith DO` suffix `DO`, where 3b's delta read family `Carmo` and family `DO`; `parser_for(EN, base=Parser(lexicon=...add("do", "van", kind="preposition")))` keeps both particles, where the delta kept `van` alone; under strict `vi` a base adding `dr` and `zzq` as titles keeps both, where 3j's delta kept `zzq`; a choice over a base built from `Lexicon.empty()` turns no reading on, where the delta joined the chosen files' off readings; and a pack's fragment yields to the caller's lexicon (no shipped fragment holds a reading or a mask, ZH's and ES's holding lists only, so no shipped pack moves). Each is pinned in tests/v2/test_parser.py. - **Measured.** On this branch over `db7e8fc4`, Python 3.11.17: tests/v2/test_snapshot.py passes unchanged; `uv run python tools/differential/cldr_agreement.py` reports the CLDR locale view and the Wikidata native locale view matching their records (JA not measured, namedivider absent); frames, `uv run python tools/perf/call_count.py --against db7e8fc4`, the reference row 311.48 on both, `_CALL_BASELINE` unchanged and the serial benchmark passing. + +- **2026-11-02 #754, sub-phase 4i-d — `Reading.languages` NAMES THE FILES WHOSE READING IS ON IN THE PARSE'S OWN `Lexicon`; A CHOICE NAMES WHAT IT READS AND NOT WHAT IT DROPS; A CALLER'S READING NAMES NONE; NO PARSE MOVES (decided 2026-11-02 in chat, on the proposal below).** Carries out item 7's optional fourth PR of the 2026-11-02 #754, the `Lexicon` surface entry, and the question its item 4 left to it; supersedes item 1 of the 2026-10-29 #754, 4a's follow-up entry in its default's-files semantics, and the 2026-10-29 #753, phase 3b entry's bullet that `Reading.languages` does not follow a choice, whose second reason, that a parse could not know its choice, 4i-c removed. + - **Decided, three questions.** (1) **The files on in the lexicon, not the chosen files only.** A reading names the files whose reading of that kind is on in the pool the parse's `Lexicon` compiled from: the default's files with no choice; under a plain choice the chosen files and every other file whose reading the choice's preference left on (`de` under `pt` names `pt`, `van` keeps `nl`, which `pt` does not hold), name readings never leaving (decision 15); under strict the chosen files and native-script files, `mul` and `und` gone with their readings. Declined: the chosen files only, which reports `()` for a word only an unchosen file holds, the same answer as "no file holds it", and false. (2) **A caller's log follows the same rule, with or without a choice.** A caller's own reading (file tag `""`) names no file, as before; an `add` of a kind a file holds off by default turns that file's reading on and names it (`Lexicon.default().add("st", kind="generation")` names `cs`, `hr`, `sk` and `sl` for `Jiří Dienstbier st.`, where it named none), and a `remove` stops naming the file whose reading it took. Declined: keeping a lexicon with no choice on the default's files, a second rule for one case. (3) **The `Ph. D.` pair under a strict choice is a separate question.** `split_pair` merges it by its pattern whatever the vocabulary (decisions/phd-merge), so under `parser_for(locales.PT, strict=True)` and `Lexicon.empty()` it still reads `credential`, source `vocabulary`, where the joined `PhD` reads family; it named `en` before this change and names none now. Whether the parse or its source should change goes to a follow-up issue, drafted, not filed. + - **What.** `Lexicon` gains a private slot `_files`, built at construction beside `_sets` and never compared, hashed or pickled: a `_lexicon._ReadingFiles`, one per base, log and choice (`_reading_files`, an `lru_cache`, so equal lexicons share it), hashed by identity. Assemble's two memos, `_languages` and `_part_languages`, take it as a fourth argument, so a hit stays C-level and costs no frame. Its `languages(text, kinds)` answers the default (base `default`, no log, no choice) from the index's rows through `languages_of`, unchanged; a choice with no log from the word's own rows under `choose` (and the strict filter), which is exact since a choice's preference is per word; and a log or a snapshot base from the pool `_built` compiled at construction. Reading the whole pool on every miss was the prototype's first form and cost about 2 ms on the first listed-word parse (`parser_for(locales.PT)`, 0.23 to 2.5 ms), which the per-word path removes. `ParsedName`, `Token` and `Reading` store nothing new, so no `ADDED_IN` marker: a pickled parse keeps the tuple it was parsed with, and an old `Lexicon` or `Parser` pickle builds the slot on load. + - **What moved, measured over the committed corpora's 6,531 names** (on master `854087be` against this branch, Python 3.11.17), only `languages`: under `pt` 309 names, each narrowing (`de` from `es fr it nl pt` to `pt` in 273, `da`, the one-case `E`, `sr`, `dra`, `santa`); under `cs` 11 (`Mgr.` from `en fr` to `cs`; `st.`, `ml.` and `pan`, read only under the choice, from `()` to `cs`); under `es` 299; under strict `en` 109 (`Ma.` from `en fil` to `en`, `Baron` from `de en fr nl` to `en`, `jr.iii` from `en mul` to `en`); under strict `pt` 337, 21 of them the `Ph. D.` pair going to `()`; with no choice and no log none. A check over nine configurations (the default, `pt`, `cs`, `es`, strict `en`, strict `pt`, the default with an `add` and a `remove`, `pt` over a caller's `add`, and `Lexicon.empty()`) found the per-word paths agreeing with the whole pool's answer on every name. tests/v2/test_parser.py's `test_reading_languages_name_the_default_files_under_a_choice`, which pinned the old answer, is rewritten as `test_reading_languages_follow_the_choice`, beside tests for strict, a caller's log, an alternative and a pickled parser. + - **Measured.** tests/v2/test_snapshot.py passes unchanged; `uv run python tools/differential/cldr_agreement.py` reports the CLDR locale view and the Wikidata native locale view matching their records (JA not measured, namedivider absent); frames, `uv run python tools/perf/call_count.py --against 854087be`, the reference row 311.48 to 311.72, all of it the first parse of each word's miss (311.0 on both in a second pass), `_CALL_BASELINE` unchanged and the serial benchmark passing; the first listed-word parse is unchanged in time for the default and for `parser_for(locales.PT)`. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index fbc29a2d..7b634f50 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -463,7 +463,7 @@ How a call reaches a pack's pattern is open, and each answer costs something: `P ### The `Lexicon` surface: configuring by reading -Status: **decided 2026-11-02 as recommended below (decisions/vocabulary-model's 2026-11-02 #754, the `Lexicon` surface entry), item 1's title warning on 2026-11-01 (decisions/vocabulary-model's 2026-11-01 #754, the title warning entry); 4i-a, 4i-b and 4i-c are built (decisions/vocabulary-model's 2026-11-02 #754, sub-phase 4i-a, sub-phase 4i-b and sub-phase 4i-c entries), the optional 4i-d is not.** What is decided is the direction (2026-10-28): a caller configures vocabulary by reading, `Lexicon` derives the sets and gates the build derives, and the compiled sets become internal (on 2026-11-01, 20 of `Lexicon`'s 23 public fields: the 19 of `_lexicon._COMPILED_FIELDS` and the derived `particles_ambiguous`; item 3 keeps the other three), with no 2.x warning (2026-10-29). This section proposes the API that direction leaves open, each question with the options weighed and one recommendation; decisions/vocabulary-model's 2026-11-01 #754, the `Lexicon` surface proposed entry records the proposal and its measurements, and decisions/vocabulary-model's 2026-11-02 #754, the `Lexicon` surface entry the decision, each recommendation accepted and the alternatives below declined. Where this section says "proposed" or "recommended", read "decided"; the wording is kept as the record of what was weighed. Every parse below was run on master `a3b1f3e1` (Python 3.11.17) through a throwaway prototype that compiles a lexicon's readings with the shipped `compile_readings` over `choose(index(), chosen)`, the whole index with its readings off by default, as item 4 has the default hold them, the caller's edits applied after the choice, and folds the result into today's `Lexicon`, so the real parser reads it; the prototype is not committed. +Status: **decided 2026-11-02 as recommended below (decisions/vocabulary-model's 2026-11-02 #754, the `Lexicon` surface entry), item 1's title warning on 2026-11-01 (decisions/vocabulary-model's 2026-11-01 #754, the title warning entry); 4i-a, 4i-b, 4i-c and the optional 4i-d are built (decisions/vocabulary-model's 2026-11-02 #754, sub-phase 4i-a, sub-phase 4i-b, sub-phase 4i-c and sub-phase 4i-d entries).** What is decided is the direction (2026-10-28): a caller configures vocabulary by reading, `Lexicon` derives the sets and gates the build derives, and the compiled sets become internal (on 2026-11-01, 20 of `Lexicon`'s 23 public fields: the 19 of `_lexicon._COMPILED_FIELDS` and the derived `particles_ambiguous`; item 3 keeps the other three), with no 2.x warning (2026-10-29). This section proposes the API that direction leaves open, each question with the options weighed and one recommendation; decisions/vocabulary-model's 2026-11-01 #754, the `Lexicon` surface proposed entry records the proposal and its measurements, and decisions/vocabulary-model's 2026-11-02 #754, the `Lexicon` surface entry the decision, each recommendation accepted and the alternatives below declined. Where this section says "proposed" or "recommended", read "decided"; the wording is kept as the record of what was weighed. Every parse below was run on master `a3b1f3e1` (Python 3.11.17) through a throwaway prototype that compiles a lexicon's readings with the shipped `compile_readings` over `choose(index(), chosen)`, the whole index with its readings off by default, as item 4 has the default hold them, the caller's edits applied after the choice, and folds the result into today's `Lexicon`, so the real parser reads it; the prototype is not committed. **1. Configuring: `add` and `remove` take words and a reading kind.** Proposed: @@ -528,7 +528,7 @@ The `nameparser.config` constants stay public and frozen: they are the default's - A chained call whose `strict` differs from the base's recorded one raises `ValueError`, offering both spellings; raising now keeps a later rule for mixing them an addition. - `_parser._apply_choice` and `_choice_delta` retire: the choice is no longer a delta against the default's compile but a filter over the base's own readings. -Weighed against: (a) recording nothing, as decided on 2026-10-29 for a lexicon of sets, which could not recompile from what it held, so a choice could only be a delta; and (b) a `Parser.languages` field, the 3b session's recommendation, which leaves `parser.lexicon` holding the compiled choice beside a record of it, so the record and what it describes could disagree once a caller replaces the lexicon. The objection to a `Lexicon` field, that its own pickle guard refused an added field, was removed on 2026-10-30 when the guard learned `ADDED_IN` (#960, decisions/A1). Whether `Reading.languages` then follows the choice is left to its own sub-phase (4i-d): the parse can know its choice once the lexicon records it, which answers the 3b entry's reason for not following it, but its memo is keyed by the word alone today and would have to take the lexicon's readings into its key. +Weighed against: (a) recording nothing, as decided on 2026-10-29 for a lexicon of sets, which could not recompile from what it held, so a choice could only be a delta; and (b) a `Parser.languages` field, the 3b session's recommendation, which leaves `parser.lexicon` holding the compiled choice beside a record of it, so the record and what it describes could disagree once a caller replaces the lexicon. The objection to a `Lexicon` field, that its own pickle guard refused an added field, was removed on 2026-10-30 when the guard learned `ADDED_IN` (#960, decisions/A1). Whether `Reading.languages` then follows the choice is left to its own sub-phase (4i-d): the parse can know its choice once the lexicon records it, which answers the 3b entry's reason for not following it, but its memo is keyed by the word alone today and would have to take the lexicon's readings into its key. 4i-d built it on 2026-11-02: the field names the files whose reading is on in the parse's own lexicon, and the memo takes a per-lexicon key (decisions/vocabulary-model's 2026-11-02 #754, sub-phase 4i-d entry). **5. Pickle.** Whether `Lexicon`'s guard learns `ADDED_IN` needs no work here: it has admitted the marker since 2026-10-30 (decisions/A1's #959 prerequisite entry, PR #960). Proposed: a `Lexicon` pickles its five parts, its base as a snapshot of the rows, as today's pickle holds its sets whole, with its ordered log, choice, lists and masks; at load a snapshot equal to the running release's index becomes the by-reference base again (item 3's base equality), and one from another release stays a snapshot, parsing as it did where it was pickled; a pickle of the default holds its base, all 2,261 index rows with their tags, grouped by word as nameparser/config/_index.py writes them (33,122 bytes against today's 27,204 for the default's sets, protocol 4, measured 2026-11-01). Weighed against pickling the base by reference with the caller's edits, which is smaller but makes a pickled `Parser()` load the loading release's vocabulary in silence, the drift decisions/A1's 2026-10-30 entry asks to be avoided or recorded. @@ -562,7 +562,7 @@ All three were checked on a frozen, slotted throwaway dataclass on 2026-11-01. T | 4i-a | The pipeline reads the compiled sets through one internal object, not `Lexicon`'s public fields; the three tools move to it or to the index. No public change; a frozenset test stays a C-level attribute read, so the frame count holds (`_CALL_BASELINE`). Landed 2026-11-02 (decisions/vocabulary-model's 2026-11-02 #754, sub-phase 4i-a entry) | — | | 4i-b | The break: `Lexicon` holds readings, `add`/`remove` by kind, `readings_of` and `LexiconReading`, the narrowed constructor, the three refusals with their translation, the per-kind repr, the warning on a title reading made phrase-only by a `given` reading, the views reading the lexicon's kinds with rules/R8 and rules/R9 amended, the lazy base, the 2.x pickle translation; the tests and docs moved, `docs/migrate.rst`'s table, a release-log bullet. Landed 2026-11-02 (decisions/vocabulary-model's 2026-11-02 #754, sub-phase 4i-b entry) | 4i-a | | 4i-c | The choice recorded on the `Lexicon`: chained choices union, a caller's readings applied after the choice, the strict mismatch raised, `|` refusing two choices, a pack's fragment joined under the caller's lexicon, the compile and its refusals at `parser_for`, the underscore choice parameters validated, `_apply_choice` retired; the chained-choices entry and 3j's dropped-listing clause superseded. Landed 2026-11-02 (decisions/vocabulary-model's 2026-11-02 #754, sub-phase 4i-c entry) | 4i-b | -| 4i-d | Optional: `Reading.languages` follows the choice | 4i-c | +| 4i-d | Optional: `Reading.languages` follows the choice. Landed (decisions/vocabulary-model's 2026-11-02 #754, sub-phase 4i-d entry) | 4i-c | ### The order of work @@ -578,7 +578,7 @@ Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's ar | 4f | `[rules]` compiled into a `Locale`'s policy patch; RU and TR_AZ rebuilt from their files. Landed (decisions/vocabulary-model's 2026-10-29 sub-phase 4f entry) | 3b | | 4g | the compound given name, a rule kind (#704): built 2026-10-30, `Policy.two_given_names` (rules/O7) | 4f | | 4h | the shape inventory per order (#469), starting with the title between a family and a given name under `FAMILY_FIRST`. Landed as an inventory, no reading moved: the arrangements each order is written in, listed with what each reads (decisions/vocabulary-model's 2026-10-30 #754, sub-phase 4h entry); the title between, and bare after, under a family-first order stays a name word, #938 declined on a survey of real names, which write the title in front (decisions/H5's 2026-10-30 #938 entry) | — | -| 4i | the `Lexicon` surface: configuring by reading, the compiled sets internal, and the choice recorded on the `Lexicon`. Decided 2026-11-02, to be built in four PRs, 4i-a, 4i-b and 4i-c landed (4i-a to 4i-d, "The `Lexicon` surface: configuring by reading" above) | 3b, 3j | +| 4i | the `Lexicon` surface: configuring by reading, the compiled sets internal, and the choice recorded on the `Lexicon`. Decided 2026-11-02, built in four PRs, 4i-a to 4i-d, all landed (4i-a to 4i-d, "The `Lexicon` surface: configuring by reading" above) | 3b, 3j | **Interleaved with phase 3** (its sub-phases 3a to 3j, above): 4a can start beside 3a, its `languages` waiting for 3a's index; 4d needs nothing of phase 3, and 4b and 4c need 3a's index only for a listed word's kind, a suffix word's and a particle's; they unblock its packs, `cs`'s generations (3f) and `pt-BR`'s agnomens (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; 4f lands with or after 3b, since both decide what a shipped `Locale` holds; an `is` pack, which no phase 3 sub-phase lists, needs 3b, 4c and 4f. 3j, file-strict, needs 3b alone (3i, which name-strict would have read, is set aside), and 4e and 4h stand apart from both. So the order is 3a and 4a, then 4b to 4d with 3b and 3c (4b after 3a), then 4f, then the packs with 4g, with 4e at any point and 3j at any point after 3b. diff --git a/docs/locales.rst b/docs/locales.rst index 3e965ee9..7947eedf 100644 --- a/docs/locales.rst +++ b/docs/locales.rst @@ -389,11 +389,20 @@ at the end of the name, where the default reads each as a name word. >>> czech.parse("Jana Nováková roz. Svobodová").maiden 'Svobodová' -A word read through a pack names no language file in its -:attr:`~nameparser.Reading.languages`, which lists only the files whose -reading is on by default, whatever the parser chose: ``ml.`` above names -none, and ``Mgr.`` names ``en`` and ``fr``, whose Monsignor reading the -Czech choice dropped for its own. +A word's :attr:`~nameparser.Reading.languages` names the files whose +reading is on under the parser's choice: ``ml.`` above names ``cs``, +whose reading the choice turns on, and ``Mgr.`` names ``cs`` alone, the +Czech choice having dropped the Monsignor reading of ``en`` and ``fr`` +for its own, which the default names: + +.. doctest:: + + >>> czech.parse("Jiří Dienstbier ml.").tokens[-1].reading.languages + ('cs',) + >>> parse("Mgr. Jan Novák").tokens[0].reading.languages + ('en', 'fr') + >>> czech.parse("Mgr. Jan Novák").tokens[0].reading.languages + ('cs',) A Czech or Slovak woman who keeps her own surname beside her husband's writes the two without a hyphen, ``Alena Pešová Řehořová``, diff --git a/docs/release_log.rst b/docs/release_log.rst index da588ccc..6d4749e8 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -16,7 +16,7 @@ Release Log - **Add a switch for names with a compound given name.** ``Parser(policy=Policy(two_given_names=True)).parse("Juan Carlos Pérez")`` gives given ``Juan Carlos``, where 1.4.0 through 2.8.0 gave given ``Juan``, middle ``Carlos``. Off by default: under it a middle name beside the given name joins it (``John Quincy Adams`` gives given ``John Quincy``). With ``two_family_names`` on too, the family takes its word first, so ``María José García López`` reads given ``María José``, family ``García López``. A ``Policy`` pickled by 2.8 loads with it off. :issue:`704` - **Add Lexicon.given_names, given names two_family_names leaves out of the family.** ``Parser(policy=Policy(two_family_names=True), lexicon=Lexicon.default() | Lexicon(given_names={"luis"})).parse("José Luis Pérez")`` gives middle ``Luis``, family ``Pérez``, where the switch alone, in 2.8.0 as now, gives family ``Luis Pérez``. Empty by default, so the switch alone reads as before; an entry without accents matches the word with them. :issue:`959` - **Add Token.reading, the reading the parse took for each word.** ``parse("John Smith PhD Jones").tokens[-1].reading`` is ``Reading('credential' as 'suffix', 'position')``, the reading 2.7's ``Ambiguity.alternative`` gave only for the other side of a fork, and ``ReadingKind`` gains ``patronymic``. The field is not compared, and a ``ParsedName`` pickled by 2.x loads with no readings. :issue:`754` - - **Add Reading.languages, the language files that list a word the way it was read.** ``parse("Dr. Juan de la Vega III").tokens[2].reading.languages`` is ``('es', 'fr', 'it', 'nl', 'pt')`` for the particle ``de``, and ``Ambiguity.alternative`` names its files too; a reading by shape or position names none. It is not compared, and a ``Reading`` pickled by 2.7 or 2.8 loads with none. The first parse in a process that meets a listed word loads the reading index once: about 1 ms, or about 16 ms where Python writes no bytecode cache. :issue:`754`, :issue:`753` + - **Add Reading.languages, the language files that list a word the way it was read.** ``parse("Dr. Juan de la Vega III").tokens[2].reading.languages`` is ``('es', 'fr', 'it', 'nl', 'pt')`` for the particle ``de``, and ``Ambiguity.alternative`` names its files too; a reading by shape or position names none. Under a choice of languages it names the files the choice reads: ``parser_for(locales.PT)`` names ``('pt',)`` for that ``de``. It is not compared, and a ``Reading`` pickled by 2.7 or 2.8 loads with none. The first parse in a process that meets a listed word loads the reading index once: about 1 ms, or about 16 ms where Python writes no bytecode cache. :issue:`754`, :issue:`753` - **Add Czech, Slovak, Slovenian and Croatian packs, locales.CS, SK, SL and HR.** ``parser_for(locales.CS).parse("Jan Novák, MUDr")`` gives title ``MUDr``, given ``Jan``, family ``Novák``, where 1.4.0 through 2.8.0 gave given ``MUDr``, family ``Jan Novák``. Each pack chooses its one language and turns on words the default does not read: ``ml.`` and ``st.`` after the name in all four (``Jiří Dienstbier ml.``), the Czech and Slovak academic titles and ``DrSc.``, and Czech ``Pan`` and ``roz.``. Choosing the language drops ``st``'s Saint readings (``St Clair Smith`` reads given ``St``), which choosing ``fr`` as well keeps. ``CS`` and ``SK`` also read two family names, so ``Alena Pešová Řehořová`` gives family ``Pešová Řehořová`` and two given names read as a given name and a family name (``Petr Pavel Novák`` gives family ``Pavel Novák``); a family comma keeps both. :issue:`753` - **Add Hindi and Bengali packs, locales.HI and locales.BN, for the romanized Indic titles.** ``parser_for(locales.HI).parse("Smt Nirmala Sitharaman")`` gives title ``Smt``, given ``Nirmala``, where 1.4.0 through 2.8.0 gave given ``Smt``, middle ``Nirmala``. ``HI`` reads ``Smt``, ``Shrimati`` and ``Shriyut`` as titles and ``BN`` ``Smt``, ``Shrimati`` and ``Adhyapak``; ``Shri`` and ``Sushri`` stay name words under both, since each opens given names (``Shri Kant``, ``Sushri Sangita``). See :doc:`locales`. :issue:`345`, :issue:`753` - **Add parser_for(..., strict=True), to read only the chosen languages' words.** ``parser_for(Locale(code="ko", lexicon=Lexicon.empty(), languages=frozenset({"ko"})), strict=True).parse("Kim Do Hyun")`` gives middle ``Do``, family ``Hyun``, where the default, as in 1.4.0 through 2.8.0, and a plain ``ko`` choice give family ``Do Hyun``, ``Do`` read as the Portuguese particle. Only the chosen languages' vocabulary files are read, with the words written in other scripts; every other language's words leave, the shared ones (``Dr``, ``Prof``) and English credentials included, so ``Kim Min Jun, MD`` gives given ``MD`` unless ``en`` is chosen too. A strict call that chooses no language raises ``ValueError``. ``python -m nameparser`` gains ``--strict``, and ``--locale`` takes a language tag no pack has (``--locale ko``). See :doc:`locales`. :issue:`753` diff --git a/docs/usage.rst b/docs/usage.rst index 1ab00fa7..75ae6185 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -1256,9 +1256,12 @@ otherwise have been the family name: Reading('name' as 'family', 'position') A reading from the vocabulary also names the language files that list -the word that way, as BCP 47 tags in ``languages``. A reading by shape -or position names none, and neither does a word from your own -``Lexicon`` that no shipped file lists: +the word that way, as BCP 47 tags in ``languages``: the files whose +reading is on in the parser's own ``Lexicon``, so a choice of languages +(:doc:`locales`) names the files it reads. A reading by shape or +position names none, and neither does a reading you add to your own +``Lexicon`` of a kind no file holds for that word; adding a kind a +file does hold turns that file's reading on, and names it: .. doctest:: diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index 1084ee7e..e4360799 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -1011,6 +1011,58 @@ def _choice_compiled(chosen: frozenset[str], return MappingProxyType(_vocab_fields(ri.compile_choice(chosen, strict))) +class _ReadingFiles: + """The files whose readings are on in one lexicon, by word: what + ``Reading.languages`` names (decisions/vocabulary-model, 2026-11-02 + #754, sub-phase 4i-d). One per base, log and choice + (`_reading_files`), held by each `Lexicon` it describes and never + pickled; it hashes by identity, in C, so assemble's memo keyed by + it costs no frame on a hit. + + The default answers from the reading index's rows + (`_reading_index.languages_of`), and a choice with no log from the + word's own rows under `choose`, which is exact because a choice's + preference is per word; neither builds the lexicon's whole pool, + which a log or a snapshot base has built already, at construction.""" + + __slots__ = ("_base", "_edits", "_chosen", "_strict") + + def __init__(self, base: object, edits: tuple[_Edit, ...], + chosen: frozenset[str], strict: bool) -> None: + self._base, self._edits = base, edits + self._chosen, self._strict = chosen, strict + + def languages(self, text: str, kinds: frozenset[str]) -> tuple[str, ...]: + """The tags of the files holding a reading of `text` (as the + vocabulary stores it) of one of `kinds`, on in this lexicon, + sorted; a caller's own reading names none.""" + ri = _index_module() + readings: Iterable[Any] + if self._edits or self._base not in ("default", "empty"): + readings = _built(self._base, self._edits, self._chosen, + self._strict)[0].get(text, ()) + elif self._base == "empty": + return () + elif not self._chosen: + return cast("tuple[str, ...]", ri.languages_of(text, kinds)) + else: + readings = ri.readings_of(text) + if self._strict: + readings = [r for r in readings if r.tag in self._chosen + or ri.is_native_script(r.text)] + readings = ri.choose(readings, self._chosen) + return tuple(sorted({r.tag for r in readings + if r.tag and r.kind in kinds + and r.on_by_default})) + + +@functools.lru_cache(maxsize=64) +def _reading_files(base: object, edits: tuple[_Edit, ...], + chosen: frozenset[str], strict: bool) -> _ReadingFiles: + # one per configuration, so equal lexicons share assemble's memo + return _ReadingFiles(base, edits, chosen, strict) + + def _check_choice(languages: object, strict: object) -> tuple[ frozenset[str], frozenset[str]]: """A recorded choice validated as `parser_for` validates one: its @@ -1777,6 +1829,10 @@ class Lexicon: init=False, repr=False, compare=False, hash=False) _sets: _LexiconSets = field( init=False, repr=False, compare=False, hash=False) + # what assemble keys Reading.languages by: one per base, log and + # choice (_ReadingFiles) + _files: _ReadingFiles = field( + init=False, repr=False, compare=False, hash=False) def __init__( self, *, @@ -1817,6 +1873,7 @@ def __init__( put("_strict", _strict) put("_chosen", chosen) compiled = _compiled_for(self._base, canon, chosen, _strict) + put("_files", _reading_files(self._base, canon, chosen, _strict)) put("_sets", _LexiconSets( **compiled, surnames=self.surnames, given_names=self.given_names)) if removed: diff --git a/nameparser/_pipeline/_assemble.py b/nameparser/_pipeline/_assemble.py index 1575ae7f..9c521757 100644 --- a/nameparser/_pipeline/_assemble.py +++ b/nameparser/_pipeline/_assemble.py @@ -18,9 +18,8 @@ no positional read happened; Policy.patronymic_rules, for the words an ending rule reads as a patronymic (rules/R9); and Lexicon.surnames, suffix_acronyms and suffix_acronyms_ambiguous, for the source of a token's reading where -no tag gives it. Not a Lexicon: the shipped reading index -(_reading_index.languages_of), for Reading.languages, which names the -files and so is the same under every Lexicon. +no tag gives it; and the Lexicon's readings by word (Lexicon._files), +for Reading.languages, the files whose reading is on in it. Structural tokens (dropped maiden markers) are omitted, like delimiter characters. A main-stream token that somehow reaches here with no role @@ -38,7 +37,9 @@ import unicodedata from types import MappingProxyType -from nameparser._lexicon import _VOCAB_FIELDS, Lexicon, _normalize +from nameparser._lexicon import ( + _VOCAB_FIELDS, Lexicon, _normalize, _ReadingFiles, +) from nameparser._pipeline._state import ( AGNOMEN_TAG, AMBIGUOUS_ACRONYM_TAG, AMBIGUOUS_WORD_TAG, ATTACHED_TAG, CREDENTIAL_WORD_TAG, FIELD_READING, GENERATION_WORD_TAG, @@ -52,7 +53,7 @@ name_source, suffix_kind, ) from nameparser._policy import PatronymicRule -from nameparser._reading_index import CATEGORY, languages_of +from nameparser._reading_index import CATEGORY from nameparser._types import ( FOLDED_TAG, GIVEN_FIRST, UNJOINED_CONJUNCTION_TAG, UNJOINED_TAG, Ambiguity, AmbiguityKind, ParsedName, Reading, ReadingKind, @@ -118,27 +119,32 @@ def _kinds(kind: ReadingKind, role: Role) -> frozenset[str]: # Both lookups are memos, C-level on a hit, so a word read before costs -# no frame (decisions/parse-cost), and both are pure functions of the -# shipped files, bounded since what keys them comes from the caller's -# text; so is `_reading`, of the readings that name files. +# no frame (decisions/parse-cost): each is a pure function of the word +# and the lexicon's readings, which `files` stands for, one object per +# base, log and choice hashed by identity (decisions/vocabulary-model, +# 2026-11-02 #754, sub-phase 4i-d), and bounded since what keys them +# comes from the caller's text; so is `_reading`, of the readings that +# name files. @functools.lru_cache(maxsize=4096) -def _languages(text: str, kind: ReadingKind, role: Role) -> tuple[str, ...]: +def _languages(text: str, kind: ReadingKind, role: Role, + files: _ReadingFiles) -> tuple[str, ...]: """The files holding the `kind` reading of `text` as written, in - `role`, Reading.languages: by the vocabulary's own form of the + `role`, on in the parse's lexicon (`files`, its `_files`), + Reading.languages: by the vocabulary's own form of the word, and then with every period removed, the form a credential is matched by ('M.A' in 'John Smith (M.A)') and a word listed whole without them ('Lt.Gen.' as 'ltgen').""" kinds = _kinds(kind, role) n = _normalize(text) - found = languages_of(n, kinds) + found = files.languages(n, kinds) if not found and "." in n: - found = languages_of(n.replace(".", ""), kinds) + found = files.languages(n.replace(".", ""), kinds) return found @functools.lru_cache(maxsize=4096) -def _part_languages(text: str, kind: ReadingKind, - role: Role) -> tuple[str, ...]: +def _part_languages(text: str, kind: ReadingKind, role: Role, + files: _ReadingFiles) -> tuple[str, ...]: """The files of the listed words `text` joins with punctuation ('Lt.Gov.', 'MD/PhD', 'MD;'), sorted: a suffix's parts divided by any mark but the period first, each an acronym ('M.D./Ph.D.'), @@ -146,11 +152,13 @@ def _part_languages(text: str, kind: ReadingKind, own lexicon does not list whole (`_listed_whole`).""" kinds = _kinds(kind, role) n = _normalize(text) - found = _by_parts(n, kinds, ".") if kinds is _SUFFIX_KINDS else () - return found or _by_parts(n, kinds, "") + found = (_by_parts(n, kinds, ".", files) if kinds is _SUFFIX_KINDS + else ()) + return found or _by_parts(n, kinds, "", files) -def _by_parts(n: str, kinds: frozenset[str], keep: str) -> tuple[str, ...]: +def _by_parts(n: str, kinds: frozenset[str], keep: str, + files: _ReadingFiles) -> tuple[str, ...]: """The files of the parts of `n`, sorted: divided by punctuation, symbols and spaces, but for the marks in `keep`, which are removed from each part instead; never by a combining mark, which a `[^\\w]` @@ -160,8 +168,8 @@ def _by_parts(n: str, kinds: frozenset[str], keep: str) -> tuple[str, ...]: for c in n).split() return tuple(sorted({ tag for part in parts - for tag in languages_of(part.replace(keep, "") if keep else part, - kinds)})) + for tag in files.languages( + part.replace(keep, "") if keep else part, kinds)})) @functools.lru_cache(maxsize=4096) @@ -261,9 +269,10 @@ def _alternative(pending: PendingAmbiguity, materialized: tuple[Token, ...], if alt.source in _LISTED: # a loop, not a generator, which would resume a frame per word for t in materialized: - langs = _languages(t.text, alt.kind, role) + langs = _languages(t.text, alt.kind, role, lexicon._files) if (not langs - and (langs := _part_languages(t.text, alt.kind, role)) + and (langs := _part_languages(t.text, alt.kind, role, + lexicon._files)) and _listed_whole(t.text, lexicon)): langs = () found.update(langs) @@ -379,6 +388,7 @@ def assemble(state: ParseState) -> ParsedName: dropped = set(state.dropped) final: dict[int, Token] = {} surnames = state.lexicon._sets.surnames + files = state.lexicon._files folds_middle = state.policy.middle_as_family tokens = state.tokens last = len(tokens) @@ -429,10 +439,10 @@ def assemble(state: ParseState) -> ParsedName: # `_listed_suffix` and its `_normalize`). `_vocab`'s two predicates # also name the sources of the reading NOT taken # (Ambiguity.alternative). - # Reading.languages, the files holding the reading taken, is - # read below from phase 3's reading index for a listed or - # declared reading, through `_languages`, whose hit costs no - # frame (decisions/vocabulary-model, 2026-10-29). + # Reading.languages, the files holding the reading taken, on in + # the parse's lexicon, is read below for a listed or declared + # reading, through `_languages`, whose hit costs no frame + # (decisions/vocabulary-model, 2026-10-29; 2026-11-02, 4i-d). reading: Reading | None = None if role is Role.SUFFIX: # a suffix ends a particle's run: the family can resume @@ -465,9 +475,11 @@ def assemble(state: ParseState) -> ParsedName: # of the two words, which the second then takes written = (t.text + _split_partner(tokens, dropped, i) if prev_split else t.text) - langs = _languages(written, kind, role) if source in _LISTED else () + langs = (_languages(written, kind, role, files) + if source in _LISTED else ()) if (not langs and source in _LISTED - and (langs := _part_languages(written, kind, role)) + and (langs := _part_languages(written, kind, role, + files)) and _listed_whole(written, state.lexicon)): langs = () reading = (_reading(kind, role, source, langs) if langs @@ -573,9 +585,11 @@ def assemble(state: ParseState) -> ParsedName: ReadingSource.POSITION, (), freading.second_surname) final[markers[i]] = Token(text, span, frole, ftags, freading) - langs = _languages(t.text, kind, field) if source in _LISTED else () + langs = (_languages(t.text, kind, field, files) + if source in _LISTED else ()) if (not langs and source in _LISTED - and (langs := _part_languages(t.text, kind, field)) + and (langs := _part_languages(t.text, kind, field, + files)) and _listed_whole(t.text, state.lexicon)): langs = () reading = (_reading(kind, role, source, langs, second) diff --git a/nameparser/_reading_index.py b/nameparser/_reading_index.py index 60f70617..00dff1fa 100644 --- a/nameparser/_reading_index.py +++ b/nameparser/_reading_index.py @@ -430,8 +430,10 @@ def languages_of(text: str, kinds: frozenset[str]) -> tuple[str, ...]: default, of one of `kinds` (keys of :data:`CATEGORY`), each once, in index order; empty for a word no file holds as one of them. `text` is normalized as :func:`readings_of` takes it. What - assemble fills ``Reading.languages`` with (decisions/vocabulary- - model, 2026-10-29); its first call in a process loads the index.""" + ``Reading.languages`` holds under the default lexicon, which reads + it through ``Lexicon._files`` (decisions/vocabulary-model, + 2026-10-29; 2026-11-02, 4i-d); its first call in a process loads + the index.""" out: list[str] = [] for tag, kind, pairs in _by_text().get(text, ()): if (kind in kinds and tag not in out diff --git a/nameparser/_types.py b/nameparser/_types.py index 9151b723..6e928706 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -956,17 +956,23 @@ class Reading: #: shape, or position alone. source: ReadingSource #: The BCP 47 tags of the shipped vocabulary files that hold this - #: reading of the word, on by default (``("es", "fr", "it", "nl", - #: "pt")`` for the particle ``de``, ``("mul",)`` for the title - #: ``Dr.``), sorted; a name reading takes the files listing the - #: word as a name of its field, given or family (``Ma`` read as a + #: reading of the word, on in the parse's own ``Lexicon`` + #: (``("es", "fr", "it", "nl", "pt")`` for the particle ``de`` by + #: default, ``("pt",)`` under a ``pt`` choice, which drops the + #: others' readings of it; ``("mul",)`` for the title ``Dr.``), so + #: a choice of languages names the files it turns on and not those + #: it drops; sorted. A caller's ``add`` of a kind a file holds for + #: the word turns that file's reading on, so the file is named + #: (``add("st", kind="generation")`` names ``cs``, ``hr``, ``sk`` + #: and ``sl``), and a ``remove`` stops naming the files it took a + #: reading from. A name reading takes the files listing the word + #: as a name of its field, given or family (``Ma`` read as a #: family name names ``en``, which lists it so, and not ``fil``, #: which lists it as a given name). Empty for a reading no file - #: holds: a shape, - #: position alone, a name read into a field no file lists it in - #: (``van`` as a family name, ``nl`` listing it as given), a word a - #: caller's own ``Lexicon`` adds that no file holds in that - #: category, or a shipped list kept outside the per-word files (the + #: holds: a shape, position alone, a name read into a field no + #: file lists it in (``van`` as a family name, ``nl`` listing it + #: as given), a caller's ``add`` of a kind no file holds for the + #: word, or a shipped list kept outside the per-word files (the #: Korean surnames that divide ``김민준``). #: A suffix reading is matched by the word's suffix entry whatever #: its finer kind, a word having one. An ``Ambiguity.alternative`` diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index da25a012..898c94be 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -3,6 +3,7 @@ import pickle import re import unicodedata +import warnings import pytest @@ -2830,23 +2831,92 @@ def test_a_packs_fragment_joins_under_the_callers_lexicon() -> None: assert parser_for(second, first).parse("Zzq Smith").title == "Zzq" -def test_reading_languages_name_the_default_files_under_a_choice() -> None: - # decisions/vocabulary-model, 2026-10-29 phase 3b: the field names the - # files holding the reading on by default whatever choice built the - # parser. Under pt, `it`'s dropped reading of `e` is still named; - # under cs, the generation `st` it turned on names no file. Its - # following the choice the lexicon now records is 4i-d's, optional, - # and would change both on purpose - def languages(parser: Parser, name: str, text: str) -> tuple[str, ...]: - token = next(t for t in parser.parse(name).tokens if t.text == text) - assert token.reading is not None - return token.reading.languages +def _languages_of(parser: Parser, name: str, text: str) -> tuple[str, ...]: + token = next(t for t in parser.parse(name).tokens if t.text == text) + assert token.reading is not None + return token.reading.languages + + +def test_reading_languages_follow_the_choice() -> None: + # decisions/vocabulary-model, 2026-11-02 #754, sub-phase 4i-d: the + # field names the files whose reading is on in the parse's own + # lexicon. Under pt, `it`'s reading of `e`, which pt's preference + # drops, is no longer named; under cs, the generation `st` the + # choice turns on names cs; a word only an unchosen file holds keeps + # that file (`van`, nl's), the plain choice keeping its reading on. + # These were ('it', 'pt'), () and ('es', 'fr', 'it', 'nl', 'pt') + # before 4i-d, the files on by default whatever the choice pt, cs = parser_for(_choosing("pt")), parser_for(_choosing("cs")) assert pt.parse("ANA SILVA E SOUZA").family == "SILVA E SOUZA" - assert languages(pt, "ANA SILVA E SOUZA", "E") == ("it", "pt") + assert _languages_of(pt, "ANA SILVA E SOUZA", "E") == ("pt",) assert cs.parse("Jiří Dienstbier st.").suffix == "st." - assert languages(cs, "Jiří Dienstbier st.", "st.") == () - assert languages(pt, "Maria do Carmo", "do") == ("pt",) + assert _languages_of(cs, "Jiří Dienstbier st.", "st.") == ("cs",) + assert _languages_of(pt, "Ana de Souza", "de") == ("pt",) + assert _languages_of(pt, "Vincent van Gogh", "van") == ("nl",) + # with no choice, the files on by default, as before + assert _languages_of(Parser(), "Ana de Souza", "de") == ( + "es", "fr", "it", "nl", "pt") + + +def test_reading_languages_under_a_strict_choice_name_the_chosen_files() -> None: + # a strict choice keeps only the chosen files' readings, so nothing + # else is named, mul.toml's included; a reading the strict lexicon + # does not hold names no file (the Ph./D. pair, merged by its + # pattern whatever the vocabulary: decisions/phd-merge) + en = parser_for(_choosing("en"), strict=True) + assert _languages_of(Parser(), "Baron John Smith", "Baron") == ( + "de", "en", "fr", "nl") + assert _languages_of(en, "Baron John Smith", "Baron") == ("en",) + pt = parser_for(_choosing("pt"), strict=True) + assert pt.parse("John Smith Ph. D.").suffix == "Ph. D." + assert _languages_of(Parser(), "John Smith Ph. D.", "Ph.") == ("en",) + assert _languages_of(pt, "John Smith Ph. D.", "Ph.") == () + + +def test_reading_languages_follow_a_callers_readings() -> None: + # the same rule over a caller's log, with no choice: a caller's own + # reading names no file; an add of a kind a file holds off by + # default turns that file's reading on, and names it; a remove + # stops naming the file whose reading it took out + zzq = Parser(lexicon=Lexicon.default().add("zzq", kind="title")) + assert zzq.parse("Zzq John Smith").title == "Zzq" + assert _languages_of(zzq, "Zzq John Smith", "Zzq") == () + st = Parser(lexicon=Lexicon.default().add("st", kind="generation")) + assert st.parse("Jiří Dienstbier st.").suffix == "st." + assert _languages_of(st, "Jiří Dienstbier st.", "st.") == ( + "cs", "hr", "sk", "sl") + mine = Lexicon.default().remove("baron", kind="title").add( + "baron", kind="title") + assert _languages_of(Parser(lexicon=mine), "Baron John Smith", + "Baron") == () + # a caller's add under a choice turns its kind on as with none + chosen = parser_for(_choosing("pt"), base=st) + assert _languages_of(chosen, "Jiří Dienstbier st.", "st.") == ( + "cs", "hr", "sk", "sl") + # the empty lexicon lists nothing, so names nothing + with warnings.catch_warnings(): + warnings.simplefilter("ignore", UserWarning) + empty = Parser(lexicon=Lexicon.empty()) + assert _languages_of(empty, "John Smith Ph. D.", "Ph.") == () + + +def test_an_alternative_reading_names_the_files_on_under_the_choice() -> None: + name = "da Silva Sr. José Pedro Eduardo Jr., MP" + def alternatives(parser: Parser) -> list[tuple[str, ...]]: + return [a.alternative.languages for a in parser.parse(name).ambiguities + if a.alternative is not None] + assert alternatives(Parser()) == [("it", "pt")] + assert alternatives(parser_for(_choosing("pt"))) == [("pt",)] + + +def test_reading_languages_survive_a_pickled_parser() -> None: + # the lexicon's key is rebuilt on load, never pickled, and equal + # lexicons share it, so a loaded parser reads as the one written + pt = parser_for(_choosing("pt")) + loaded = pickle.loads(pickle.dumps(pt)) + assert loaded.lexicon._files is pt.lexicon._files + assert _languages_of(loaded, "Ana de Souza", "de") == ("pt",) + assert "_files" not in pt.lexicon.__getstate__() def test_a_choice_that_changes_nothing_leaves_the_sets_as_they_are() -> None: