diff --git a/AGENTS.md b/AGENTS.md index 0a0fd4c33..61dc629b6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -218,7 +218,7 @@ The `config/` SETS are frozen since 2.2 (#293): there is no `.add()`/`.remove()` - `titles.py` — `TITLES` (every title reading; prenominals), `GIVEN_NAME_TITLES` (the readings with `addresses = "given"`, e.g. "Sir", which treat the following name as given, not family — a subset of the title readings by construction, though not of `TITLES`: a word that may also be a given name, such as `king` or `sheikh`, compiles to `TITLES_AMBIGUOUS` instead) and `AFTER_GIVEN_TITLES` (the salutation readings with `after_given = true`, read as the title where one ends the given part after a family comma, rules/H8, #835 — a subset by construction); `TITLES_AMBIGUOUS` (a title reading beside a `given` one, derived, compiled INSTEAD of into `TITLES`: a title only inside a title phrase, rules/H9, #752 phase 2i), `TITLE_MODIFIERS` and `TITLE_STYLES` (by kind) and `TITLE_PHRASES` (the two-word phrases `attaches` makes) - `suffixes.py` — `SUFFIX_ACRONYMS` (suffix readings with `acronym = true`, matched with every period removed, e.g. "M.D.") and `SUFFIX_WORDS` (the rest, e.g. "Jr."), disjoint because a word has one suffix kind; `SUFFIX_ACRONYMS_AMBIGUOUS` (an acronym reading on a word with a `given` or `family` reading, derived since 2026-10-25: the build refuses the phase 1 flag `may_be_name`); `SUFFIX_WORDS_AMBIGUOUS` (its word twin, a generation or agnomen reading, or a credential written as a word, on a word with a name reading, `fils`, the agnomens and `kc`, `pc` and `dl`, gated by the count, rules/S11, since 2026-10-25, the credentials since 2026-10-30, #965) with `AGNOMENS` inside it (the `agnomen` kind, whose writing never leans); and `GLUED_HONORIFICS` (#308, `attached`), the subset of `SUFFIX_WORDS` the peel may split off the END of a name token — a separate, harsher bar, since the glued position has no writer-drawn boundary to lean on, and each word refused it carries `excluded = { attached = … }` -- `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (the particles whose word has no `given` reading, each stating `may_be_given = false`; a `given` reading on the word is what makes a particle possibly given, the phase 1 flag `may_be_given = true` refused since 2026-10-25). What the subset licenses is rules/P1's fold in `post_rules` (the "rule 1b" of older issue comments — decisions/legacy-rule-numbers maps the old numbers), and the fold has ONE site: the piece that OPENS the name. A lone never-given particle there, with at least one more name word behind it, starts the family. Titles do not move the opening position (rules/P4), so `Sir de Mesnil` reads family `de Mesnil` under every `name_order`, as `de Mesnil` does. How far the fold reaches depends on the order the name was read under: by default it takes the rest of the name (`de Mesnil Jean` → family `de Mesnil Jean`); under either family-first order it takes the particle run plus ONE name unit and lays the rest out by the order (`de Mesnil Jean` → family `de Mesnil`, given `Jean`); after a family comma it takes the rest of that part whatever the order (`Smith, de Mesnil Jean` → family `Smith de Mesnil Jean`). The leading particle stays a lone piece for the fold to find — `group` does not chain it forward, and the fold walks the particles after it itself — so `de la Vega` reads family `de la Vega` under all three orders. The GIVEN position is not a site. It was a second one until #467, which removed it because under a family-first order that slot holds what the caller DECLARED to be the given name: `Mesnil de` under `FAMILY_FIRST` or `FAMILY_FIRST_GIVEN_LAST` reads given `de`, family `Mesnil`. Nor is a particle inside the name: `Juan de la Vega` under a family-first order reads family `Juan`, given `de la Vega`, with no ambiguity recorded — what the vocabulary forbids is the bare particle as a given name, not a name part that begins with one. A bare `de` has nothing to fold into and is placed by position like any one-word name: given `de` by default, family `de` under both family-first orders. A leading particle OUTSIDE the set is left to position — `van Gogh` is given `van`, family `Gogh` by default and family `van`, given `Gogh` under both family-first orders — and records a `PARTICLE_OR_GIVEN` ambiguity under all three; a never-given particle opening the name records none (`de Mesnil`), and neither does an ambiguous one inside it (`Vincent van Gogh`). `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either +- `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (the particles whose word has no `given` reading, each stating `may_be_given = false`; a `given` reading on the word is what makes a particle possibly given, the phase 1 flag `may_be_given = true` refused since 2026-10-25). What the subset licenses is rules/P1's fold in `post_rules` (the "rule 1b" of older issue comments — decisions/legacy-rule-numbers maps the old numbers), and the fold has ONE site: the piece that OPENS the name. A lone never-given particle there, with at least one more name word behind it, starts the family. Titles do not move the opening position (rules/P4), so `Sir de Mesnil` reads family `de Mesnil` under every `name_order`, as `de Mesnil` does. How far the fold reaches depends on the order the name was read under: by default it takes the rest of the name (`de Mesnil Jean` → family `de Mesnil Jean`); under either family-first order it takes the particle run plus ONE name unit and lays the rest out by the order (`de Mesnil Jean` → family `de Mesnil`, given `Jean`); after a family comma it takes the rest of that part whatever the order (`Smith, de Mesnil Jean` → family `Smith de Mesnil Jean`). The leading particle stays a lone piece for the fold to find — `group` does not chain it forward, and the fold walks the particles after it itself — so `de la Vega` reads family `de la Vega` under all three orders. The GIVEN position is not a site. It was a second one until #467, which removed it because under a family-first order that slot holds what the caller DECLARED to be the given name: `Mesnil de` under `FAMILY_FIRST` or `FAMILY_FIRST_GIVEN_LAST` reads given `de`, family `Mesnil`. Nor is a particle inside the name: `Juan de la Vega` under a family-first order reads family `Juan`, given `de la Vega`, with no ambiguity recorded — what the vocabulary forbids is the bare particle as a given name, not a name part that begins with one. A bare `de` has nothing to fold into and is placed by position like any one-word name: given `de` by default, family `de` under both family-first orders. A leading particle OUTSIDE the set is left to position — `van Gogh` is given `van`, family `Gogh` by default and family `van`, given `Gogh` under both family-first orders — and records a `PARTICLE_OR_GIVEN` ambiguity under all three; a never-given particle opening the name records none (`de Mesnil`), and neither does an ambiguous one inside it (`Vincent van Gogh`). The internal `Lexicon._sets.particles_ambiguous`, the particles holding a `given` reading, is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either - `bound_given_names.py` — `BOUND_GIVEN_NAMES` (`bound_given` readings, e.g. "abdul", "abu"; one that is also a particle needs a `given` reading, a build check); a group-stage rule joins the first non-title piece to its following piece before roles are assigned, reserving a family word unless a family comma has already fixed it or a given-name title stands ahead (#369, rules/P5) (ported from v1's `_join_bound_first_name`) - `given_names.py` — `GIVEN_NAME_ABBREVIATIONS` (#789, a `given` reading carrying `same_as`, the full name: "chas", "wm", Filipino "ma"), the given names written short that rules/H2's shape yields to; nothing else reads it - `conjunctions.py` — `CONJUNCTIONS` (e.g. "and", "of") used to chain multi-word titles and to join name parts (rules/P3), plus `CONJUNCTIONS_AMBIGUOUS` (#383/#479, rules/P8, `may_be_initial`, refused by the build on anything but a cased single letter), the subset of single cased letters that read as an INITIAL in a name written wholly in one case — a marker set @@ -234,7 +234,7 @@ The `config/` SETS are frozen since 2.2 (#293): there is no `.add()`/`.remove()` The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2.0 and then 3.0 deleted. These conventions apply to all of it. The full design record (rationale, settled-decision logs, dated amendments) lives in untracked `docs/superpowers/specs/`; this section is the enforceable subset. **A commit that establishes or amends one of these conventions must update this section in the same commit** — grep-driven staleness sweeps miss paraphrased prose (see Workflow above), so write-time maintenance is the mechanism, audits are the backstop. - **Module layout**: every module is underscore-private (`_types.py`, `_lexicon.py`, `_policy.py`, `_locale.py`, `_render.py`, `_reading_index.py`, `_pipeline/`, `_parser.py`). The public import surface is exactly `nameparser`, `nameparser.locales` and the vocabulary data modules under `nameparser.config`; `nameparser/__init__.py` holds re-exports and `__all__` only — no logic. -- **Layering is enforced by `tests/v2/test_layering.py`** (exact-module matching; `if TYPE_CHECKING:` imports don't count): `_types` imports nothing internal at module level — its rendering delegates import `_render` at call time; `_lexicon` and `_policy` sit above `_types` independently (`_lexicon` may import `nameparser.config.*` DATA modules only, e.g. `config/maiden_markers.py`); `_reading_index` imports nothing internal at module level and the index, `config/_index.py`, at call time, since `tools/vocab/build.py` loads it by path; `_locale` sits on `_lexicon`+`_policy` (plus `_types` for the shared pickle mixin, and `config/_rules.py` at call time, in `rules_patch`); `_render` imports `_types`, `_lexicon` (for `Lexicon.default()` and `_normalize`) and `_policy` (for the scripts that write no initials); `_pipeline/*` imports `_types`+`_lexicon`+`_policy` plus in-package `_pipeline` helpers, `_assemble` also `_reading_index`, for `Reading.languages`, and `_vocab` too, for a listed suffix word's kind and a patronymic particle's (3.0's 4b and 4c); `_parser` sits on everything except `_render`, `_reading_index` at call time only, where a choice of languages is applied; `__main__` imports only the public package; locale pack modules (`locales/*.py`) import `_locale`/`_lexicon`/`_policy`/`_types` only (`_types` joined the list in #272 — `locales/ja.py`'s segmenter factory constructs a `Segmentation`), and the `locales/__init__` additionally lazy-imports its packs (PEP 562). Extend the test's `ALLOWED` table when adding a module. +- **Layering is enforced by `tests/v2/test_layering.py`** (exact-module matching; `if TYPE_CHECKING:` imports don't count): `_types` imports nothing internal at module level — its rendering delegates import `_render` at call time; `_lexicon` and `_policy` sit above `_types` independently (`_lexicon` may import `nameparser.config.*` DATA modules, e.g. `config/maiden_markers.py`, and `_reading_index` at call time only, to compile a configured lexicon's readings, since `tools/vocab/build.py` loads it by path, 3.0's 4i-b); `_reading_index` imports nothing internal at module level and the index, `config/_index.py`, at call time, since `tools/vocab/build.py` loads it by path; `_locale` sits on `_lexicon`+`_policy` (plus `_types` for the shared pickle mixin, and `config/_rules.py` at call time, in `rules_patch`); `_render` imports `_types`, `_lexicon` (for `Lexicon.default()` and `_normalize`) and `_policy` (for the scripts that write no initials); `_pipeline/*` imports `_types`+`_lexicon`+`_policy` plus in-package `_pipeline` helpers, `_assemble` also `_reading_index`, for `Reading.languages` (`_vocab` imported it in 3.0's 4b and 4c for a listed suffix word's kind and a patronymic particle's, which since 4i-b the lexicon compiles into `_sets`); `_parser` sits on everything except `_render`, `_reading_index` at call time only, where a choice of languages is applied; `__main__` imports only the public package; locale pack modules (`locales/*.py`) import `_locale`/`_lexicon`/`_policy`/`_types` only (`_types` joined the list in #272 — `locales/ja.py`'s segmenter factory constructs a `Segmentation`), and the `locales/__init__` additionally lazy-imports its packs (PEP 562). Extend the test's `ALLOWED` table when adding a module. - **Canonical field order** — the seven roles in `Role` enum declaration order, defined once and derived everywhere (properties, `as_dict`, reprs, `comparison_key`). Never restate the order literally. `Role` is a `StrEnum`: members compare as their field-name strings, and `tokens_for()` coerces strings via `_coerce_enum`. - **Method organization**, fixed section order in every class: fields + `__post_init__` validation → alternative constructors → dunders (construction/equality → protocol → operators) → properties → public methods by concern (access → editing → comparison → rendering delegates) → private helpers last, except a helper serving exactly one section may sit at that section's head. - **Validation is eager and fail-loud**: every `raise` states the offending value, the expected form, and the fix. Exception taxonomy: wrong type — including wrong element type inside a collection, bare `str` where an iterable of strings is expected, or a `Mapping` where a plain iterable is expected — raises `TypeError`; well-typed but unacceptable values raise `ValueError`; failed enum lookups stay `ValueError` for any input (stdlib `EnumType` precedent). **When the message hands the reader code to paste, that code has to survive a type checker, and then RUN** — nameparser ships `py.typed`, and a placeholder can type-check and still raise: the 2.4 release sweep's first fix to the `capitalization_exceptions` hint offered `('key', 'Value')`, which passes mypy and fails #459's mask check, so `test_add_capitalization_exceptions_raises_pointing_at_replace` now executes the offered code. #337's segmenterless warning offered `Policy(segment_scripts=())`, an `arg-type` error, because these fields are annotated with what they STORE rather than everything the constructor accepts. Prefer the `frozenset()` / `()` spellings in messages and docstrings, and pin the offered spelling in a test — the warning tests matched on `ja_segmenter` and never checked the actionable half of the message. **A warning emitted in `Parser.__post_init__` needs `parser_for` to re-emit it from its own frame** (the `catch_warnings(record=True)` block at its return): `__post_init__`'s `stacklevel` is sized for direct `Parser(...)` construction, and through `parser_for`'s extra frame the default one-line rendering attributes the warning to the library's own `return Parser(...)` — the exact call the message tells the user to change becomes invisible. No single stacklevel serves both entry points; a new construction warning gets the re-emission for free, but a new CONSTRUCTION SITE for `Parser` inside this package needs its own re-emission or its callers get library-attributed warnings (#337 review). @@ -242,15 +242,15 @@ The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2 - **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary, so emit where the branch is taken, in every stage that takes one, and name the reading not taken. The reading TAKEN (`Token.reading`, 3.0) is the one read later, at assemble, where roles are final, from the role and the tags the deciding stages left, its suffix and name sources by the predicates that name the reading not taken. The contract is mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE; the working rules (branches that change nothing, the field a report names, the stages that decide `PARTICLE_OR_GIVEN`, `alternative`, the reading taken, and how to test an emitter) are in `nameparser/_pipeline/AGENTS.md`, which loads when a session reads a pipeline stage. - **A kind is worth adding only if a reader would hesitate too**: the test is not "does the code take a branch" but whether a person reading that input would genuinely be unsure. "Smith, John V" reads as a middle initial to anyone -- the comma settles it -- so reporting it would be noise that teaches callers to ignore the field, which costs more than the missing report. Reachability of the second branch is necessary, not sufficient. Prefer leaving a fork silent and documenting the omission over emitting on input nobody finds ambiguous. - **Parser owns config-dependent conveniences**: `Parser.matches`/`Parser.capitalized`/`Parser.revise` exist because the `ParsedName` equivalents fall back to DEFAULT config for str/omitted arguments (documented loudly in both docstrings). `revise` harvests tokens from a full sub-parse of each replacement value (tags kept minus `FOLDED_TAG`, roles forced, the R1 entry pass `suffix_entries` re-run over the forced state so a suffix value's entries follow its own commas, ambiguities discarded); the merge tail is shared with `replace()` via `ParsedName._with_field_tokens`. `Parser.capitalized` delegates through `name.capitalized(self.lexicon)` specifically so `_parser` never imports `_render` — keep it that way. -- **Per-word vocabulary fields warn on multi-word entries** (`_normset`/`_normpairs` via `_warn_dead_entry`, UserWarning, never a raise — see the given_name_titles Gotcha for why raising is wrong). `given_name_titles` is the one multi-word-matched field and is exempt; `_edit` passes `warn=False` (add() warns once via the new instance's `__post_init__`; remove() stores nothing). The default vocabulary and every locale pack must stay warning-free (`test_default_lexicon_builds_warning_free`, `test_pack_vocabulary_entries_are_single_words`). -- **Invariants guard harm, not no-ops**: add a constructor check when violating it produces a *wrong parse*, not when it produces *nothing*. A false positive costs a working configuration; a true positive on an inert condition costs the user nothing, so that trade is never worth taking. `suffix_acronyms_ambiguous ∩ suffix_words` is guarded because the overlap loses a family name; `given_name_titles` is not, because an unreachable entry is simply never consulted (see Gotchas). Before adding one, construct the config it forbids and check what actually breaks. **Between raise and silence sits the construction-time `UserWarning`**, for a gap that is statically decidable, harmless to SOME deliberate caller, and indistinguishable-from-working for everyone else: the segmenterless activation (#337 — `parser_for(locales.JA)` without a segmenter behaved exactly like a working parser minus the feature) warns rather than raises because the inert JA registration is itself a pinned property, and a warning is filterable by the caller who wants exactly that. The message must carry every applicable remedy and no inapplicable one (the `ja_segmenter` hint fires only when a Japanese script is among the dead ones). Test fuzzers that legitimately construct such configs suppress the warning by MESSAGE, never by category — a blanket `UserWarning` ignore would mask the next construction diagnostic (`_quiet_parser` in `tests/v2/test_properties.py` is the pattern). +- **A reading is of one word, save a birth marker and a title addressing the given name** (the build's check 8, which `_lexicon._make_edit` runs on every `add` and RAISES on, since a reading is now a caller's statement the compile can refuse as the build does: 3.0's 4i-b). The two lists, `surnames` and `given_names`, and `capitalization_exceptions` keys still WARN on a multi-word entry (`_normset`/`_normpairs` via `_warn_at_caller`, UserWarning, never a raise — an inert entry produces nothing). The default vocabulary and every locale pack must stay warning-free (`test_default_lexicon_builds_warning_free`, `test_pack_vocabulary_entries_are_single_words`). +- **Invariants guard harm, not no-ops**: add a constructor check when violating it produces a *wrong parse*, not when it produces *nothing*. A false positive costs a working configuration; a true positive on an inert condition costs the user nothing, so that trade is never worth taking. Since 3.0's 4i-b a `Lexicon` compiles from readings, which hold the 2.x marker-over-base relations by construction; the checks a caller's edit reaches are the build's checks 3 (a kind and its properties, `may_be_initial` on a single cased letter alone), 4 (one suffix kind per word), 8 (a phrase), 9 (a bound given name that is a particle has a `given` reading) and 11's agnomen clause (an agnomen has a name reading), in `_lexicon._make_edit` and `_check_pool`, and again in `_parser._apply_choice`, which refuses a choice whose readings clash with the caller's, each raising with the call that fixes it, because each is a wrong parse. A given-addressing title run the parse can never build is not checked, because an unreachable entry is simply never consulted (see Gotchas). Before adding one, construct the config it forbids and check what actually breaks. **Between raise and silence sits the construction-time `UserWarning`**, for a gap that is statically decidable, harmless to SOME deliberate caller, and indistinguishable-from-working for everyone else: the segmenterless activation (#337 — `parser_for(locales.JA)` without a segmenter behaved exactly like a working parser minus the feature) warns rather than raises because the inert JA registration is itself a pinned property, and a warning is filterable by the caller who wants exactly that. The message must carry every applicable remedy and no inapplicable one (the `ja_segmenter` hint fires only when a Japanese script is among the dead ones). Test fuzzers that legitimately construct such configs suppress the warning by MESSAGE, never by category — a blanket `UserWarning` ignore would mask the next construction diagnostic (`_quiet_parser` in `tests/v2/test_properties.py` is the pattern). - **Reprs are bounded**: render which fields deviate from a named baseline and by how much, never contents (`Lexicon(default + titles: +2)`). `PolicyPatch`'s repr shows only set (non-UNSET) fields; `_order_repr` must never raise even on an unvalidated patch's garbage `name_order` (PolicyPatch defers validation to apply time); the sweep test in `tests/v2/test_reprs.py` pins that no config repr leaks the UNSET sentinel. -- **Every pipeline stage is one module holding one public function of the same name** (`_tokenize.py`/`tokenize`), with the exceptions named where they stand — `_group.py`'s `marker_run_length`, a shared predicate (mechanisms.md#ONE-PREDICATE-PER-QUESTION), and since #511 `_post_rules.py`'s `suffix_entries`, the R1 entry pass as a function, because `Parser.revise` runs that one pass over a forced-role sub-parse and a stage's tail cannot be called on its own — **and its module docstring declares the contract in three labelled lines**: `Consumes:` what it takes from `ParseState`, `Produces:` what it hands back, `Reads:` which `Policy`/`Lexicon` fields it consults. `Reads:` is the load-bearing one — it makes "which stage do I touch for this feature?" a grep rather than a read-through. A `Reads:` line names a vocabulary set as `Lexicon.`, and every library module but `_lexicon` reads it as `lexicon._sets.` (`_lexicon._LexiconSets`, #754's 4i-a), never off the public field, `surnames` and `given_names` included though they stay public, so that 4i-b can make the other twenty internal without touching a reader; `tests/v2/test_layering.py` refuses a direct read, field by field, in the library and in every tool. The authority for the stage set is `_pipeline/__init__.py`'s `STAGES` (eight, and not public API) with the field-ownership map in `ParseState`'s docstring, pinned by `tests/v2/pipeline/test_state.py`; NOT the rules, which are implementation-free by their own preamble and whose `implemented:` names modules honoring a rule rather than stages. A `_pipeline/` module that is not a stage says so in its first line instead (`_assemble.py`: "Not a stage: …", with a `Reads:` line all the same, since it names a reported field by `Policy.middle_as_family` and a token's reading by three `Lexicon` sets), so absence of the three lines is a claim about the module rather than an oversight. Provenance: §5 of the 2026-07-11 conventions spec, recorded here 2026-08-16. +- **Every pipeline stage is one module holding one public function of the same name** (`_tokenize.py`/`tokenize`), with the exceptions named where they stand — `_group.py`'s `marker_run_length`, a shared predicate (mechanisms.md#ONE-PREDICATE-PER-QUESTION), and since #511 `_post_rules.py`'s `suffix_entries`, the R1 entry pass as a function, because `Parser.revise` runs that one pass over a forced-role sub-parse and a stage's tail cannot be called on its own — **and its module docstring declares the contract in three labelled lines**: `Consumes:` what it takes from `ParseState`, `Produces:` what it hands back, `Reads:` which `Policy`/`Lexicon` fields it consults. `Reads:` is the load-bearing one — it makes "which stage do I touch for this feature?" a grep rather than a read-through. A `Reads:` line names a vocabulary set as `Lexicon.`, and every library module but `_lexicon` reads it as `lexicon._sets.` (`_lexicon._LexiconSets`, #754's 4i-a), never off the public field, `surnames` and `given_names` included though they stay public, which let 4i-b make the other twenty internal without touching a reader; `tests/v2/test_layering.py` refuses a direct read, field by field, in the library and in every tool. The authority for the stage set is `_pipeline/__init__.py`'s `STAGES` (eight, and not public API) with the field-ownership map in `ParseState`'s docstring, pinned by `tests/v2/pipeline/test_state.py`; NOT the rules, which are implementation-free by their own preamble and whose `implemented:` names modules honoring a rule rather than stages. A `_pipeline/` module that is not a stage says so in its first line instead (`_assemble.py`: "Not a stage: …", with a `Reads:` line all the same, since it names a reported field by `Policy.middle_as_family` and a token's reading by three `Lexicon` sets), so absence of the three lines is a claim about the module rather than an oversight. Provenance: §5 of the 2026-07-11 conventions spec, recorded here 2026-08-16. - **Frames are a budget for hot and scaling paths; everywhere else, prefer the shared predicate.** The parser's cost is counted in Python frames (`sys.setprofile` "call" events — deterministic and load-immune, which is why #475 chose them over wall-clock timing), but one frame is small: measured 2026-09-28 on py3.11, a trivial call costs about 13 ns against 40–80 µs for a whole parse, so one frame per parse is roughly 0.03% of it. What the budget exists to catch is how a cost SCALES and how it ACCUMULATES, so weigh an extra frame by four questions, in order. (1) Does it multiply with the input — per token, per member, per pair? Then it matters, and a hand-inlined fast path can earn its place (the inline tag tests ahead of `_pieces.starts_a_credential_run` in its loops, each a necessary half of the predicate so nothing can drift; `_vocab.run_word_fold` was the measured example until #613 deleted the run test it served). Folding a copy into a shared helper is where such a pre-check gets lost: #617 replaced `_comma._title_chains`, whose loop tested the conjunction tag inline, with a call into group's loop, which asks `is_conj_piece` per piece, and every family-comma parse paid two frames per word of the part (`Smith, John` +2) while the reference row fell -- review caught it; the fix gates the call on the tag walk the caller already makes, and `test_a_comma_part_with_no_free_connective_skips_the_join` holds it. Count a comma name, not only the reference row, when a fold touches the comma path. (2) How many names pay it — every parse, where plain names like `Smith, John` must keep their cost, or only a rare shape? (3) Does it move an enforced guard — the ±2% reference band in `test_benchmark.py` (`_CALL_BASELINE`, decisions/parse-cost) or a scaling ratio? (4) What does the inline alternative cost to maintain? A hand copy of a predicate that can drift is the defect class mechanisms.md#ONE-PREDICATE-PER-QUESTION names, and it compounds; a NARROWER question stated exactly is not a copy and may stay inline on that ground alone. A constant frame on a rare shape that no guard sees loses to readability: #544's cleanup declined `is_title_piece` for one frame on comma credential runs (24 ns a parse, no guard moved), a stricter call than this rule asks for; by this rule the inline test is still right, but because it asks the narrower question (a lone word in both vocabularies), not because of the frame. The converse holds too: the frame counter cannot see C-level work (`x in some_list`, slices, string concatenation), so "no new frames" is not "no new cost" — #553's quadratic was a list membership test inside a loop, invisible to every frame guard. - **A claim about WHICH STAGE or WHICH LAYER does something is checkable — check it before writing it.** The pipeline is eight stages with a written ownership map (`ParseState`'s docstring, pinned by `tests/v2/pipeline/test_state.py`), and `parse(s).tokens` prints every token's role and tags, so "extract assigns this", "classify never sees that", "group consumes it" each have a one-command answer. #329's prose claimed delimited maiden content is *"claimed whole before classify has tagged anything inside it"*; measured, `classify` tags the marker fine and only the CONSUMING is missing, because `_group`'s rule walks `pieces` and a token that already carries a role is not in `pieces`. Two different mechanisms, one plausible sentence covering both. That single claim then shipped SIX times across three correction rounds, which is the part worth internalizing: **when a mechanism claim turns out wrong, sweep for where else you wrote it, and sweep again at the END of the change over the words the change itself just added.** Prose density here means one idea lives in a docstring, a case note, a release-log entry and this file at once; the implementer working against a wrong mechanism is the person most likely to restate it; and rewriting a mechanism claim is writing one, so the correction earns the same one-command check as the original — two of the six instances were fresh errors introduced by the sentence fixing the previous one. **Adjacency is the trap.** The claim that feels already-known is the one about the neighbouring stage or the neighbouring layer: one comment block in the 2.x `tests/v2/test_facade_cases.py` got the exception type, the raising layer, the skip mechanism, the count of skipped rows, and which row was blocked all wrong at once — every one a claim about `_config_shim` versus real 1.4.0, written from reasoning, in the file whose whole job is translating between them. What finally held was not better prose but moving the claim into a test (`_CORE_ONLY_IDS`, deleted with the facade in 3.0), which cannot be wrong the way a sentence can. - **Typing/docs**: `from __future__ import annotations`; `frozen=True, slots=True` on every public dataclass; strict-profile mypy flags via per-module overrides in pyproject (`strict = true` itself is not valid per-module). Docstrings state contracts in prose with **no doctest blocks** — `--doctest-modules` makes every example a test; behavior examples go to unit tests per the lean-docs rule. **Document the positive direction of a partial property**: "a non-empty `ambiguities` is a signal to act on" is checkable, while "an empty one means no fork occurred" is a universal negative needing exhaustive verification -- that claim was written twice and falsified twice, at sites the author had not audited. - **The segmenter contract**: the optional `Parser(segmenter=...)` hook is parse-totality's ONE exception (locales spec section 4). Everything inside that exception is a bug in USER CODE, never a fact about the name, so it is surfaced rather than absorbed: the segmenter's own exceptions propagate, and the two protocol violations the stage can detect for itself — an answer of the wrong type, and one cutting at or past the end of the token it was handed — raise `TypeError`/`ValueError` from `_script_segment` for the same reason. The line to hold when adding a check there: a protocol violation by the segmenter's AUTHOR raises, while an adapter's defense against its own third-party library (`locales/ja.py`'s repertoire, length, reconstruction and score guards) declines with `None`, because what those catch is a fact about the content. -- **Pickling**: v2 types must round-trip (`Parser` is picklable by construction, and it holds a `Lexicon`; the one qualifier is that a `Parser` pickles iff its segmenter does — see the segmenter bullet above). Every frozen type assigns `_guarded_getstate`/`_guarded_setstate` (`_types.py`) in its class body (`@dataclass(slots=True)` would override inherited pickle methods) — unpickling fails at the LOAD site on field-layout skew, with one widening: a field added after its type shipped carries `metadata={_types.ADDED_IN: ""}` and a default, and an older pickle missing it loads with the default (`Ambiguity.alternative`, 2.7; `Token.reading`, `Reading.languages`, `Reading.second_surname` and `ParsedName.order`, 3.0.0) -- a missing field without the marker, default or not, an unknown field and a renamed one are still refused, each a row in `tests/v2/test_types.py::_REFUSED_SKEWS`, so a new field on a pickled type takes the marker or breaks every older pickle; the shared guard deliberately does NOT re-validate values (pickle is not a security boundary; canonical state comes from a validated instance). `Lexicon` keeps its own copy of the guard (layering) plus the `mappingproxy` slot rebuild; a new unpicklable slot type needs the same treatment plus a round-trip test. `Lexicon`'s copy of the guard admits the same marker (since 3.0, #959's prerequisite), read under `_lexicon._ADDED_IN`, the same string, since layering keeps `_lexicon` from importing `_types` (`test_lexicon_marks_an_added_field_with_the_shared_key` holds the two equal), so a new `Lexicon` field takes the marker and a default or breaks older `Lexicon`, `Parser` and locale-pack pickles, which `tests/v2/data/lexicon_v28.pickle` holds. An older pickle loads the field's DATACLASS default, not the words a later `Lexicon.default()` compiles into it, so a marked field the default vocabulary fills makes a 2.8 `Parser()` pickle parse differently from the release's `Parser()`, in silence: give such a field a default an older pickle should mean, or record the accepted drift with it (decisions/A1, 2026-10-30; #959's field, filled only by a pack, is the case it does not reach). `Lexicon`'s one slot outside its fields, `_sets` (4i-a), inherited from a plain base so that `dataclasses.fields` and the pickled state stay its fields, is built by `__post_init__` and never pickled, a load rebuilding it. And unlike the shared guard it DOES re-validate, the loaded default with the rest: it re-runs `__post_init__` and refuses a state holding an entry that is not already normalized, because `particles_ambiguous` changed meaning between v1 and v2 without changing its name, and an old blob would otherwise load an inverted lexicon in silence. +- **Pickling**: v2 types must round-trip (`Parser` is picklable by construction, and it holds a `Lexicon`; the one qualifier is that a `Parser` pickles iff its segmenter does — see the segmenter bullet above). Every frozen type assigns `_guarded_getstate`/`_guarded_setstate` (`_types.py`) in its class body (`@dataclass(slots=True)` would override inherited pickle methods) — unpickling fails at the LOAD site on field-layout skew, with one widening: a field added after its type shipped carries `metadata={_types.ADDED_IN: ""}` and a default, and an older pickle missing it loads with the default (`Ambiguity.alternative`, 2.7; `Token.reading`, `Reading.languages`, `Reading.second_surname` and `ParsedName.order`, 3.0.0) -- a missing field without the marker, default or not, an unknown field and a renamed one are still refused, each a row in `tests/v2/test_types.py::_REFUSED_SKEWS`, so a new field on a pickled type takes the marker or breaks every older pickle; the shared guard deliberately does NOT re-validate values (pickle is not a security boundary; canonical state comes from a validated instance). `Lexicon` and `LexiconReading` keep their own copy of the guard (layering); a new unpicklable slot type needs the same treatment plus a round-trip test. Since 3.0's 4i-b a `Lexicon` pickles five things: its lists, its masks, its edit log and its base as a SNAPSHOT of the reading index's rows, so a pickle parses as it did where it was written, a snapshot equal to the running release's index loading as the by-reference default again (decisions/vocabulary-model, 2026-11-02 #754, item 5); its private `_sets` and `_cap_map` fields are built at construction and never pickled. `Lexicon`'s copy of the guard admits the `ADDED_IN` marker, read under `_lexicon._ADDED_IN`, the same string, since layering keeps `_lexicon` from importing `_types` (`test_lexicon_marks_an_added_field_with_the_shared_key` holds the two equal), so a new `Lexicon` field takes the marker and a default or breaks older pickles. A 2.x state holds the compiled sets instead, and `_lexicon._translated_2x` translates it onto the nearer named base and a per-word log, loading where the compile gives the sets back and refusing otherwise, naming the fields, so the 2.8 `Lexicon`, `Parser` and locale-pack pickles in `tests/v2/data/lexicon_v28.pickle` load as the 3.0 calls that build them. And unlike the shared guard it DOES re-validate: a load goes through the constructor, which checks the base, refuses an edit not in canonical form, and refuses a list or mask entry that is not already normalized, because an old blob would otherwise load a different lexicon in silence (`particles_ambiguous` changed meaning between v1 and v2 without changing its name). - **One sanctioned global**: the (future) cached default `Parser`. Lazily cached FROZEN singletons (`Lexicon.default()`'s `functools.cache`, the future default parser) are constants, not state; any second piece of module-level MUTABLE state requires amending the conventions doc, on purpose, in review. - **Tests**: all v2 tests live in the `tests/v2/` package, one test module per source module plus the cross-cutting ones (`test_reprs.py`, `test_layering.py`, `test_contracts.py`, `test_properties.py`, `test_benchmark.py`, the `cases.py`/`test_cases.py` table, `test_regex_sync.py`, which pins the pipeline's hand-copied patterns against their sources, and `test_snapshot.py`, which holds every corpus name's reading to `tools/differential/snapshot.jsonl` — sharing `_differential_fixtures.py` with `test_differential.py` and `test_corpora.py` for the handles on `tools/differential/`), names stating behavior; the 1.x suite's parsing tests stand ported to `parse()` in `tests/*.py`. Never assert `Lexicon.default()` contents; the narrow sourcing spot-checks in `test_default_sources_v1_vocabulary` that pin the v1→v2 migration contract (e.g. the flipped `particles_ambiguous` model) are the sanctioned exception. @@ -260,7 +260,7 @@ The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2 **Before adding a short/common word to `PARTICLES` globally**, test it mid-string against realistic 3-token names, not just check for English-word collisions: Korean/Vietnamese given names put a short syllable in the middle slot (`Park In Hwan`, `Nguyen To Nga`), and Western names put a bare initial there (`John V. Smith`). A word that looks safe ("nobody is named 'to'") can still swallow a real middle name/initial into the last name once it's a global prefix — confirmed regressions for `to`/`in`/`an`/`ten`/`then` and bare `v` this way (PR #191). -**Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `AFTER_GIVEN_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `nameparser/_reading_index.py`'s `BEHAVIORAL` table, a branch in its `compile_readings` adding the words whose reading carries it and a key in its `Compiled` TypedDict, a constant of the same name in `tools/vocab/build.py`'s `CONSTANTS` roster (an import-time assert there holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. A property the compiler reads is kept in the reading index by `indexed()`, which reads `BEHAVIORAL`, so a new one reaches a choice of languages too. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, `SUFFIX_WORDS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). A caller's `Lexicon.add()` to a sub-set does **not** add the word to its base, so a caller adding a brand-new word adds it to both. Two traps a gating set met (#752 phase 2i). Where a caller listing a word in the BASE set has always meant "read it as this everywhere" (the 1.4 recipe `c.titles.add("dean")`), the gated words compile OUT of the base rather than into both, or the gate overrides the caller's listing: `TITLES_AMBIGUOUS` is not a subset of `TITLES` for that reason, and a word in both reads as a title. And `_VOCAB_FIELDS` is read by three rosters: two in `_pipeline/_vocab.py`, the capitals shape's "unlisted" (`in_any_wordlist`, over `_LISTED_FIELDS`, the roster less four fields) and the comma contrast's "claimed as non-name" (`_NON_NAME_FIELDS`, those less `surnames` and `bound_given_names`), and `_assemble._listed_whole`, which reads it whole to tell a word the parse's own lexicon lists for `Reading.languages`. A new field joins all three by default; a field whose words may be names leaves the two in `_vocab`, which the grid found as `John Smith, DEAN` reversed where C6 had read a credential; #789's `given_name_abbreviations` met it again (`John Smith, JAS` and `Jas SMITH, XYZ` reversed), found by review because its grid held only the dotted spellings the set was for -- a new field's grid holds the spellings it must NOT move too. +**Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `AFTER_GIVEN_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `nameparser/_reading_index.py`'s `BEHAVIORAL` table, a branch in its `compile_readings` adding the words whose reading carries it and a key in its `Compiled` TypedDict, a constant of the same name in `tools/vocab/build.py`'s `CONSTANTS` roster (an import-time assert there holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module (or `None` in the roster for a set only `Lexicon` reads, as `GENERATION_WORDS`, `CREDENTIAL_WORDS` and `PATRONYMIC_PARTICLES`, the kinds the views read, are since 4i-b; it then joins `_lexicon._COMPILED_FIELDS` and `_LexiconSets` as well). A property the compiler reads is kept in the reading index by `indexed()`, which reads `BEHAVIORAL`, so a new one reaches a choice of languages too. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, `SUFFIX_WORDS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). A caller states a property on a reading (`add(w, kind="salutation", after_given=True)`), so a caller's word reaches the base and the sub-set together, as a file's does (3.0's 4i-b). Two traps a gating set met (#752 phase 2i). Where a caller listing a word in the BASE set has always meant "read it as this everywhere" (the 1.4 recipe `c.titles.add("dean")`, spelled `remove("dean", kind="given")` since 3.0's 4i-b), the gated words compile OUT of the base rather than into both, or the gate overrides the caller's listing: `TITLES_AMBIGUOUS` is not a subset of `TITLES` for that reason, and a word in both reads as a title. And `_VOCAB_FIELDS` is read by three rosters: two in `_pipeline/_vocab.py`, the capitals shape's "unlisted" (`in_any_wordlist`, over `_LISTED_FIELDS`, the roster less four fields) and the comma contrast's "claimed as non-name" (`_NON_NAME_FIELDS`, those less `surnames` and `bound_given_names`), and `_assemble._listed_whole`, which reads it whole to tell a word the parse's own lexicon lists for `Reading.languages`. A new field joins all three by default; a field whose words may be names leaves the two in `_vocab`, which the grid found as `John Smith, DEAN` reversed where C6 had read a credential; #789's `given_name_abbreviations` met it again (`John Smith, JAS` and `Jas SMITH, XYZ` reversed), found by review because its grid held only the dotted spellings the set was for -- a new field's grid holds the spellings it must NOT move too. **Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base `, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to English titles, honorifics or post-nominals also reads `snapshot.py --local govuk-people` the same way: GOV.UK's display names of ministers, officials and officers, the title run and the post-nominals as an office writes them (`The Rt Hon`, `Sir`, `Baroness`, ranks; `KC`, `CBE`, `KCB`), in 2,093 of whose 6,021 `parse()` read one or both on 2026-10-30. A change to Spanish titles, to the `es` pack, or to how a title marks where a name starts after an office reads `snapshot.py --local dof-signatures` the same way: the names in the signature blocks of Mexico's Diario Oficial for 2025, cut from the clause that names the signatory's office, 65.4% of its 33,565 signatures opening with a title (`Lic.`, `Dr.`, `Mtro.`, `C.`, `Licenciada`, `C.P.`), whose first word `parse()` read as a title in 82.1% and `parser_for(locales.ES)` in 86.8% on 2026-10-31. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Guru` in front, `Shekh`, a title there until #752's phase 2j made it one only inside a title phrase, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base ` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality; and `local_agreement.py orcid-researchers --base `, ORCID holders' own two boxes (given names, family name) from every country, scored by the country they work or live in -- a population each person split for themselves, academic but worldwide, the one to read for a change a language pack makes; and `local_agreement.py patentsview-inventors --base `, US patents' inventors in the two boxes their filings give, split by a professional filer, with how many people filed under each name in each country, the one to read for how common a name's word is among a country's US-patent inventors. A change to titles or post-nominals, or to the `cs` pack, reads `local_agreement.py cz-candidates --base `, and with `--locale cs` too: Czech election candidates' four boxes (the title before the name, given names, surname, the title after it), scored by election, the only population whose titles are stated apart from the name, 36% of its 170,181 records titled (2026-10-30), nearly all academic degrees; its names are composed from the boxes (`Ing. Jan Novák, Ph.D.`), so it says nothing about how a writer spaces or punctuates them. These are the only real fielded populations outside the committed fields files, and no CI run sees them either. @@ -274,11 +274,11 @@ The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2 **Parsing must never write into its config** — `Lexicon` and `Policy` are frozen, so a parse cannot write to them; v1 did, registering dotted abbreviations like `Lt.Gov.` into the shared config through 1.2.x, which made parse results order-dependent and thread-unsafe. Keep any new cache keyed and immutable for the same reason (`tests/test_constants.py`'s `JoinedPieceTests` hold the readings of that bug's inputs). -**The `nameparser/config` vocabulary constants are frozen** (2.2, #293) — `TITLES.add("dean")` raises `AttributeError` at the line that writes it, so a runtime addition goes on a config OBJECT instead: `Parser(lexicon=Lexicon.default().add(titles={"dean"}))`. What the freeze buys: `Lexicon.default()` is `functools.cache`d and reads the constants once, so pre-freeze an edit after the first parse was invisible to `parse()` while one before it was not, which depended on nothing the reader could see (through 2.x the v1 `Constants` copied the constants at every construction too, a third moment). +**The `nameparser/config` vocabulary constants are frozen** (2.2, #293) — `TITLES.add("dean")` raises `AttributeError` at the line that writes it, so a runtime addition goes on a config OBJECT instead: `Parser(lexicon=Lexicon.default().remove("dean", kind="given"))`. What the freeze buys: `Lexicon.default()` is `functools.cache`d and reads the constants once, so pre-freeze an edit after the first parse was invisible to `parse()` while one before it was not, which depended on nothing the reader could see (through 2.x the v1 `Constants` copied the constants at every construction too, a third moment). **Titles permanently shadow first names — be conservative** — any word in `TITLES` is always consumed as a title and can never be parsed as a first name. `"Dean"` is the canonical example: it's a common academic title *and* a common given name, so it is not in `TITLES` but in `TITLES_AMBIGUOUS`, a title only inside a title phrase (`Dean of Students`, `Associate Dean`; rules/H9, #752 phase 2i), and users who need it everywhere add it to a `Lexicon` of their own. A title that is also a given name now has that machinery: a `given` reading on the word moves it there, decided word by word in measured batches (decisions/vocabulary-model's 2026-10-24 order). A rule that reads a title by its neighbours in the title run meets more phrase shapes than the qualifier-then-head it was designed on: the object of a connective (`President of the United States`, `the Queen's Bench`, whose `united`, `states` and `bench` are modifiers with no head after them), a head before `of` (`Member of Parliament`), and a qualifier written after its head (`Chargé d'Affaires`, `Princess Royal`). Phase 2i's first build, "a modifier needs a head title after it", broke `President of the United States`, `Chargé d'Affaires` and the committed `Queen's Bench` row, found by the case table and the committed corpora, and then read a modifier ENDING the name out of H5's trailing run (`Jane Smith Vice.`), found by the design-docs review; put each shape, and the trailing position, in the grid of any such rule. Before adding a word to `TITLES`, ask: "Could this plausibly be someone's given name in any culture?" If yes, don't add it globally; it belongs in a caller-supplied `Lexicon` instead. This same caution applies to international honorifics — `Frau` is a first name in some contexts, and `Prince` and `Sheikh` were titles until 2.7 gave them a `given` reading, which moved them to `TITLES_AMBIGUOUS` (rules/H9). It also applies to any prefix sub-set gated on "never a first name": obscure-looking foreign particles are surprisingly often real given names — `Von` (Von Miller), `Vander` (Brazilian, also the Arcane character). When unsure, exclude — a missing member just means that name isn't auto-handled, whereas a wrong member misparses a real person. The decided exception is a word borne as a name only RARELY in the leading position — mostly a surname written last (`graf`, `greve`, `knight`), a few with small given-name or surname-first counts (`herra`, `hrabia`): the title claim acts only in front, so it ships, and a name that begins with it (`Greve Anna`, `Herra Wijaya` → title) is the accepted cost. That is a judgment by incidence, made per word, with the counts taken recorded on the word's entry in `nameparser/vocab/` (herra's, hrabia's; graf, which predates #606, and greve and knight, which #606 proposed, carry none) and the class in `decisions/salutation-titles` (rules/H1's Accepted clause states the cost), not a rule a sweep can apply: a word that commonly leads a real name (Vietnamese `Ông`, the given names `thiru`, `puan`) is still out of `TITLES`; `marshal` reads as a title only in a phrase (`Field Marshal`). `tools/name_usage.py` produces the counts this judgment starts from (role counts, plus Wikidata citizenship to tell where a family name is written first); its Wikidata column is a lower bound (Shehu Shagari has no given-name statement), so a small count there proves little and a zero proves nothing, and its nam_dict row gives a band per country rather than a count, with a missing name proving nothing either. -**The period-abbreviation title inference runs at the head of the GIVEN-NAME part, not the head of the name** — an unrecognized multi-letter word ending in a single trailing period (`_pieces._PERIOD_ABBREV`, a hand copy of the `period_abbreviation` regex, `{2,}` letters — it was assign's until #424 and group's until #439) is treated as a title in the leading title run, e.g. `"Insp. Jane Morse"` → `title='Insp.'`. "Leading" is per SEGMENT: `_peel_leading_titles` is called for NO_COMMA segment 0, SUFFIX_COMMA segment 0, and FAMILY_COMMA **segment 1**, so `"Morse, Det. Insp. Jane"` → `title='Det. Insp.'` and a lone `"Smith, Xyz."` → `title='Xyz.'` — long-standing, verified against 1.4.0, and the mechanism behind #296 (`"Smith, Jr."` read title in 1.4.0; the comma's credential read takes it as suffix `Jr.` now). The docs said "leading word" until 2026-08-01 and were wrong for every comma path. FAMILY_COMMA **segment 0** peels too since #701, but by VOCABULARY only (rules/H6, in assign's family-comma branch rather than `_peel_leading_titles`): `Dr. Smith, John` → `title='Dr.'`, while `Xyz. Smith, John` keeps family `Xyz. Smith`, and a title-particle (`St`, `Freiherr`) stops the run, as does a title with nothing but post-nominals behind it (`Dr. Jr., John` keeps family `Dr.`); a title/suffix dual with a name word behind it still peels (`Rev. Sr. Smith, Mary`). It adds nothing to the `Lexicon`, so the periodless form (`"Insp"`) is unaffected elsewhere. The `{2,}` length requirement — not a separate initials check — is what excludes single-letter initials like `"J."`; the same word after the given name is left as a middle name. **The inference OUTRANKS vocabulary where it runs**: `"Esq. Smith"` → `title='Esq.'` even though `esq` is suffix-only vocabulary, because the shape rule fires before anything consults the suffix sets. The one vocabulary it yields to, since #789 (phase 2k of #752), is a given name written short (`Lexicon.given_name_abbreviations`, a `given` reading with `same_as` in the TOML): `"Chas. Smith"` → `given='Chas.'`, read off classify's `vocab:given-abbreviation` tag in `is_leading_title` and off the lexicon in `_vocab.is_title_shaped`, the two spellings of one test that `test_is_title_shaped_and_is_leading_title_agree` holds together. The second spelling is what rules/C4's comma count and the title phrase's head run ask, so a change to the shape moves comma names nobody wrote it for: #789's first grid held no comma and missed `Chas. Smith, Ed` (given 'Ed', family 'Chas. Smith' before, suffix 'Ed' after), which only mutating `is_title_shaped`'s half alone showed. Put a word of the ambiguous class after a comma behind the shaped word (`X. Smith, Ed`), and a modifier in front of it (`Vice X. Smith, Ed`), in the grid of any change to the shape, and mutate each spelling separately. **The INFERENCE still runs in one direction only, and that is what the two slots share and where they part** (rewritten 2026-09-08, #316): a period-marked word is claimed by SHAPE at the front and by VOCABULARY at the back, so an unlisted abbreviation opening a name is a title while an unlisted abbreviation ending one is a NAME word (`"John Smith Xyz."` → `family='Xyz.'`). A trailing period-marked word the vocabulary knows as a title now IS one (`"John Smith Prof."` → `title='Prof.'`, `rules/H5`) — the comma path read a lone `Smith, Prof.` that way already, and the given part's `Smith, John Prof.` since 2.3.0 (1.4.0 through 2.2.0 read `Doe, Jane Mrs.` middle 'Mrs.'), so the two agree now — and a trailing period-marked word the UNAMBIGUOUS SUFFIX vocabulary knows is still a suffix, the suffix run being peeled first (`"John Smith Esq."` → `suffix='Esq.'`). A BARE trailing title word stays a name word (`"John Smith Sir"`, `"Mary Jane Pope"`), which is the doctrine line below, save a listed salutation ending the given part after a family comma (rules/H8, `"Doe, Jane Mrs"` → `title='Mrs'`). Meanwhile `period_joined_vocab` resolves INTERIOR-period tokens (`Lt.Gov.`, `Msc.Ed.`) to title-or-suffix by vocabulary, and `_extract._suffix_shaped` treats any period-final delimited content as not-a-nickname. Four sites, and the trailing one now resolves by two vocabularies in order rather than one; unifying the rest is open design work, not settled. (#109; see `docs/usage.rst` "Titles you didn't configure") +**The period-abbreviation title inference runs at the head of the GIVEN-NAME part, not the head of the name** — an unrecognized multi-letter word ending in a single trailing period (`_pieces._PERIOD_ABBREV`, a hand copy of the `period_abbreviation` regex, `{2,}` letters — it was assign's until #424 and group's until #439) is treated as a title in the leading title run, e.g. `"Insp. Jane Morse"` → `title='Insp.'`. "Leading" is per SEGMENT: `_peel_leading_titles` is called for NO_COMMA segment 0, SUFFIX_COMMA segment 0, and FAMILY_COMMA **segment 1**, so `"Morse, Det. Insp. Jane"` → `title='Det. Insp.'` and a lone `"Smith, Xyz."` → `title='Xyz.'` — long-standing, verified against 1.4.0, and the mechanism behind #296 (`"Smith, Jr."` read title in 1.4.0; the comma's credential read takes it as suffix `Jr.` now). The docs said "leading word" until 2026-08-01 and were wrong for every comma path. FAMILY_COMMA **segment 0** peels too since #701, but by VOCABULARY only (rules/H6, in assign's family-comma branch rather than `_peel_leading_titles`): `Dr. Smith, John` → `title='Dr.'`, while `Xyz. Smith, John` keeps family `Xyz. Smith`, and a title-particle (`St`, `Freiherr`) stops the run, as does a title with nothing but post-nominals behind it (`Dr. Jr., John` keeps family `Dr.`); a title/suffix dual with a name word behind it still peels (`Rev. Sr. Smith, Mary`). It adds nothing to the `Lexicon`, so the periodless form (`"Insp"`) is unaffected elsewhere. The `{2,}` length requirement — not a separate initials check — is what excludes single-letter initials like `"J."`; the same word after the given name is left as a middle name. **The inference OUTRANKS vocabulary where it runs**: `"Esq. Smith"` → `title='Esq.'` even though `esq` is suffix-only vocabulary, because the shape rule fires before anything consults the suffix sets. The one vocabulary it yields to, since #789 (phase 2k of #752), is a given name written short (a `given` reading with `same_as`, in the TOML or a caller's `add(w, kind="given", same_as=...)`, compiled into the internal `Lexicon._sets.given_name_abbreviations`): `"Chas. Smith"` → `given='Chas.'`, read off classify's `vocab:given-abbreviation` tag in `is_leading_title` and off the lexicon in `_vocab.is_title_shaped`, the two spellings of one test that `test_is_title_shaped_and_is_leading_title_agree` holds together. The second spelling is what rules/C4's comma count and the title phrase's head run ask, so a change to the shape moves comma names nobody wrote it for: #789's first grid held no comma and missed `Chas. Smith, Ed` (given 'Ed', family 'Chas. Smith' before, suffix 'Ed' after), which only mutating `is_title_shaped`'s half alone showed. Put a word of the ambiguous class after a comma behind the shaped word (`X. Smith, Ed`), and a modifier in front of it (`Vice X. Smith, Ed`), in the grid of any change to the shape, and mutate each spelling separately. **The INFERENCE still runs in one direction only, and that is what the two slots share and where they part** (rewritten 2026-09-08, #316): a period-marked word is claimed by SHAPE at the front and by VOCABULARY at the back, so an unlisted abbreviation opening a name is a title while an unlisted abbreviation ending one is a NAME word (`"John Smith Xyz."` → `family='Xyz.'`). A trailing period-marked word the vocabulary knows as a title now IS one (`"John Smith Prof."` → `title='Prof.'`, `rules/H5`) — the comma path read a lone `Smith, Prof.` that way already, and the given part's `Smith, John Prof.` since 2.3.0 (1.4.0 through 2.2.0 read `Doe, Jane Mrs.` middle 'Mrs.'), so the two agree now — and a trailing period-marked word the UNAMBIGUOUS SUFFIX vocabulary knows is still a suffix, the suffix run being peeled first (`"John Smith Esq."` → `suffix='Esq.'`). A BARE trailing title word stays a name word (`"John Smith Sir"`, `"Mary Jane Pope"`), which is the doctrine line below, save a listed salutation ending the given part after a family comma (rules/H8, `"Doe, Jane Mrs"` → `title='Mrs'`). Meanwhile `period_joined_vocab` resolves INTERIOR-period tokens (`Lt.Gov.`, `Msc.Ed.`) to title-or-suffix by vocabulary, and `_extract._suffix_shaped` treats any period-final delimited content as not-a-nickname. Four sites, and the trailing one now resolves by two vocabularies in order rather than one; unifying the rest is open design work, not settled. (#109; see `docs/usage.rst` "Titles you didn't configure") **Cyrillic suffix regexes need `re.I` even when the pattern is suffix-only** — a Latin title-cased word (`Ivanovich`) keeps its suffix lowercase, so `re.I` seemed skippable; but an irregular Cyrillic suffix can be nearly the whole word (`ильич`), so title-casing capitalizes into the suffix itself (`Ильич`). `east_slavic_patronymic_cyrillic` shipped without `re.I` on the Latin reasoning and silently failed on capitalized irregular forms — don't assume Latin's title-case safety transfers to Cyrillic. (#185) @@ -292,7 +292,7 @@ The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2 **`SUFFIX_ACRONYMS` and `SUFFIX_WORDS` are DISJOINT by construction (since 2026-10-18; an import-time assert from 2026-09-08) — a word has one suffix kind, and its reading is an acronym or a word (`acronym`), so the vocabulary build refuses a word classified twice; the fix is to choose, not to relax the check** — the two branches normalize differently (the word test strips only edge periods, the acronym test strips all of them), so a word in both is matched by two rules and which one fired is unreadable from outside. `esq` was the single overlap and left `SUFFIX_ACRONYMS`, which is what made the assert possible; the whole algebra that had made the overlap safe, and the criterion that decided which set a word belongs to, are in the `suffix-acronym-collisions` entry of `docs/design/decisions/` if anyone wants them. -**`Lexicon.given_name_titles` is deliberately unvalidated against `titles` — do not add a check** — the lookup key is the space-joined run of `Role.TITLE` tokens, built by the parse — matched whole OR by that key's LAST word since #489, a run addressing the way its final title does — and a conjunction inside a run is itself tagged `Role.TITLE`, so `"sir and dame"` is a matchable key whose middle word lives in `conjunctions`. A whole-entry check rejected multi-word entries; a per-word check rejected that one. No static relation over the vocabulary sets decides reachability. An unreachable entry is inert — nothing consults it and nothing misparses — so the condition being "guarded" costs the user nothing while each guard cost a working configuration. If a diagnostic is wanted, it must be non-blocking — the multi-word UserWarning in `_normset` is the shipped example; a raise remains wrong. Note the SHIPPED data cannot carry a multi-word given-name title without a spurious warning: `GIVEN_NAME_TITLES ⊆ TITLES ∪ TITLES_AMBIGUOUS` puts the entry in `titles` or `titles_ambiguous` too, each per-word warned; user-supplied v2 Lexicons are unaffected (`add(given_name_titles=...)` alone is silent). +**A given-name title run is deliberately not checked for whether the parse can build it — do not add a check** — since 3.0's 4i-b it is a title reading with `addresses="given"`, so the word is a title by construction, and a run of several words (`add("lt col", kind="title", addresses="given")`) is a phrase the build's check 8 admits; the lookup key is the space-joined run of `Role.TITLE` tokens, built by the parse — matched whole OR by that key's LAST word since #489, a run addressing the way its final title does — and a conjunction inside a run is itself tagged `Role.TITLE`, so `"sir and dame"` is a matchable key whose middle word lives in `conjunctions`. A whole-entry check rejected multi-word entries; a per-word check rejected that one. No static relation over the vocabulary sets decides reachability. An unreachable entry is inert — nothing consults it and nothing misparses — so the condition being "guarded" costs the user nothing while each guard cost a working configuration. If a diagnostic is wanted, it must be non-blocking — the multi-word UserWarning `_normset` gives a list entry is the shipped example; a raise remains wrong. **`_normalize` must reach a fixed point** — storage and match-time share the one fold, and `Lexicon.__setstate__` re-validates, so a value that changes on re-normalization changes under its owner. `strip().strip(".")` alone is not idempotent (`'. a .'` → `' a '` → `'a'`). The loop is the fix; keep any new stripping inside it. **Anything built on `_normalize` must converge too** — `_fold_words` runs `_normalize` per word and DROPS the words that fold away (`_title_key` is that list space-joined, and `_run_addresses_by_given` reads the list itself, so its last-word arm is the last word of the FOLDED key by construction); keeping the empty slot stored `'lt .'` as `'lt '`, a key match-time can never rebuild (so the entry is silently inert) and `__setstate__` rejects on the next round-trip as "not written by this version". @@ -318,7 +318,7 @@ Don't use the bare `python3 -m doctest .rst` CLI (no `optionflags`) to che **`pr/NNN` local branches** track upstream PRs — don't commit to them by accident. Check `git branch --show-current` before starting work. -**Title vs suffix is positional for BARE words, and the leading period-abbreviation rule overrides even that** — a word matching `TITLES` at the front of a name becomes `title`; the same word matching `SUFFIX_ACRONYMS`/`SUFFIX_WORDS` at the end becomes `suffix` (never both, regardless of the word's real-world meaning). The `TITLES`/suffix overlap was audited in #296 (2026-08-23): the pure postnominals (`jr`, `junior`, `phd`, `do`, `se`) left `TITLES`, the v1-residue `dr`/`sra` left the suffix sets, and the ten words still in both (`ms`, `sr`, `lt`, `ra`, `vc`, and the ranks `cpl`, `cpo`, `cpt`, `csm`, `sgm`; `md` and `sa` were duals too until #853 took `md`'s title reading out, a leading `Md` being the given name Muhammad, and #857 took `sa` out of the vocabulary) are deliberate duals that position decides. External test sources (old issue gists, etc.) sometimes assert `suffix` for a leading professional abbreviation like `RA`/`PD`/`Dipl.-Ing.` — that's the source data being wrong, not a parser bug. Verify position before "fixing" it. Two qualifications the older "purely positional" wording papered over, the first measured 2026-08-01 and the second answered 2026-09-08 (#316): a PERIOD-marked leading word is claimed by the shape rule before any vocabulary is read (`"Esq. Smith"` → `title`, though `esq` is suffix-only), save a listed title and, since #789, a given name written short (`"Chas. Smith"` → `given`), and a PERIOD-marked trailing word is claimed by VOCABULARY — first the suffix sets, which the peel reads before anything else (`"John Smith Esq."` → `suffix`), then the titles (`"John Smith Prof."` → `title='Prof.'`, `rules/H5`) — while an unlisted abbreviation there stays a name part (`"John Smith Xyz."` → `family='Xyz.'`) unless it is written as two or more period-separated chunks, which since 2.4 IS a trailing shape rule (`Policy.unlisted_dotted_suffixes`, default on: `"John Smith X.Y.Z."` → `suffix='X.Y.Z.'`, read by the same words-to-spare count a bare ambiguous acronym takes, `rules/S3`), except paired initials in a script with capitals, which stay a name word (`"Ram Bahadur B.K."` → `family='B.K.'`, #864). The SINGLE trailing period is still not one, deliberately — it is the abbreviation shape any word can wear. The bare word is where "positional" still holds whole -- outside a part past the second comma, which holds no name word, so C2 reads a bare title word there as a title since #603 (`Eric H. Holder, Jr., Attorney General`), and outside a listed salutation ending the given part after a family comma, which rules/H8 reads as the title since #835 (`Doe, Jane Mrs`), the comma having marked the surname's place -- and the reason it must: `TITLES` holds words that are in no suffix set — 787 of them on 2026-10-11 (746 before #606's salutations), measured with `L = Parser().lexicon; len(L.titles - L.suffix_acronyms - L.suffix_words)`, a figure that grows with the vocabulary and never shrinks the argument — and many are ordinary surnames (`pope`, `bishop`, `judge`, `sheriff`, `baron`, `master`, ...), so a vocabulary-first rule over BARE trailing words would read `"Mary Jane Pope"` as `title='Pope'`, `family='Jane'`. The period is what separates the safe case from that one, and it separates it by being a WRITING convention rather than by making the collision go away: `"Mary Jane Pope"` keeps `family='Pope'` while `"Mary Jane Pope."` reads `title='Pope.'`, `family='Jane'`, a cost accepted under the input-is-a-name premise rather than one the rule prevents (the trailing-position bullet in decisions/vocabulary-exclusions, measured 2026-09-09). Since #316 that sentence describes the shipped rule rather than an aspiration. +**Title vs suffix is positional for BARE words, and the leading period-abbreviation rule overrides even that** — a word matching `TITLES` at the front of a name becomes `title`; the same word matching `SUFFIX_ACRONYMS`/`SUFFIX_WORDS` at the end becomes `suffix` (never both, regardless of the word's real-world meaning). The `TITLES`/suffix overlap was audited in #296 (2026-08-23): the pure postnominals (`jr`, `junior`, `phd`, `do`, `se`) left `TITLES`, the v1-residue `dr`/`sra` left the suffix sets, and the ten words still in both (`ms`, `sr`, `lt`, `ra`, `vc`, and the ranks `cpl`, `cpo`, `cpt`, `csm`, `sgm`; `md` and `sa` were duals too until #853 took `md`'s title reading out, a leading `Md` being the given name Muhammad, and #857 took `sa` out of the vocabulary) are deliberate duals that position decides. External test sources (old issue gists, etc.) sometimes assert `suffix` for a leading professional abbreviation like `RA`/`PD`/`Dipl.-Ing.` — that's the source data being wrong, not a parser bug. Verify position before "fixing" it. Two qualifications the older "purely positional" wording papered over, the first measured 2026-08-01 and the second answered 2026-09-08 (#316): a PERIOD-marked leading word is claimed by the shape rule before any vocabulary is read (`"Esq. Smith"` → `title`, though `esq` is suffix-only), save a listed title and, since #789, a given name written short (`"Chas. Smith"` → `given`), and a PERIOD-marked trailing word is claimed by VOCABULARY — first the suffix sets, which the peel reads before anything else (`"John Smith Esq."` → `suffix`), then the titles (`"John Smith Prof."` → `title='Prof.'`, `rules/H5`) — while an unlisted abbreviation there stays a name part (`"John Smith Xyz."` → `family='Xyz.'`) unless it is written as two or more period-separated chunks, which since 2.4 IS a trailing shape rule (`Policy.unlisted_dotted_suffixes`, default on: `"John Smith X.Y.Z."` → `suffix='X.Y.Z.'`, read by the same words-to-spare count a bare ambiguous acronym takes, `rules/S3`), except paired initials in a script with capitals, which stay a name word (`"Ram Bahadur B.K."` → `family='B.K.'`, #864). The SINGLE trailing period is still not one, deliberately — it is the abbreviation shape any word can wear. The bare word is where "positional" still holds whole -- outside a part past the second comma, which holds no name word, so C2 reads a bare title word there as a title since #603 (`Eric H. Holder, Jr., Attorney General`), and outside a listed salutation ending the given part after a family comma, which rules/H8 reads as the title since #835 (`Doe, Jane Mrs`), the comma having marked the surname's place -- and the reason it must: `TITLES` holds words that are in no suffix set — 787 of them on 2026-10-11 (746 before #606's salutations), measured with `L = Parser().lexicon._sets; len(L.titles - L.suffix_acronyms - L.suffix_words)`, a figure that grows with the vocabulary and never shrinks the argument — and many are ordinary surnames (`pope`, `bishop`, `judge`, `sheriff`, `baron`, `master`, ...), so a vocabulary-first rule over BARE trailing words would read `"Mary Jane Pope"` as `title='Pope'`, `family='Jane'`. The period is what separates the safe case from that one, and it separates it by being a WRITING convention rather than by making the collision go away: `"Mary Jane Pope"` keeps `family='Pope'` while `"Mary Jane Pope."` reads `title='Pope.'`, `family='Jane'`, a cost accepted under the input-is-a-name premise rather than one the rule prevents (the trailing-position bullet in decisions/vocabulary-exclusions, measured 2026-09-09). Since #316 that sentence describes the shipped rule rather than an aspiration. ### Tests (`tests/`) diff --git a/docs/customize.rst b/docs/customize.rst index 55bf05941..95a06ba93 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -11,164 +11,182 @@ why the split is drawn there. Vocabulary: Lexicon -------------------- -Adding and removing words -~~~~~~~~~~~~~~~~~~~~~~~~~~ +Configuring by reading +~~~~~~~~~~~~~~~~~~~~~~~ + +A :class:`~nameparser.Lexicon` holds **readings**: what each word may be +read as, in the terms the shipped vocabulary files use. Configure it by +adding and removing readings: .. doctest:: >>> from nameparser import Lexicon, Parser, parse >>> parse("Dean Robert Johns").given 'Dean' - >>> lex = Lexicon.default().add(titles={"dean"}) + >>> [r.kind for r in Lexicon.default().readings_of("dean")] + ['office', 'given', 'family'] + >>> lex = Lexicon.default().remove("dean", kind="given") >>> Parser(lexicon=lex).parse("Dean Robert Johns").title 'Dean' -By default ``Dean`` is a title only inside a title phrase such as -``Dean of Students`` (:ref:`titles-also-given-names`); listing it in -``titles`` makes it a title wherever it stands. - -:meth:`~nameparser.Lexicon.add` and :meth:`~nameparser.Lexicon.remove` -both return a new :class:`~nameparser.Lexicon` — the one you started -from (here, ``Lexicon.default()``) is never mutated. Every field -accepts a plain set of lowercase words, keyword by field name (``titles`` -above; ``particles``, ``suffix_words``, and the rest work the same -way) — see :doc:`modules` for the full field list. - -The default word lists themselves — ``TITLES``, ``PARTICLES`` and the -other frozensets in ``nameparser.config`` — are frozen, so a runtime -addition belongs on a :class:`~nameparser.Lexicon` as above. -``REGEXES`` and ``CAPITALIZATION_EXCEPTIONS`` are the two members the -freeze does not cover — they are still plain dicts. Editing one at -runtime is not a supported override. The parser does not read -``REGEXES`` at all, and ``Lexicon.default()`` is cached, -so an edit to ``CAPITALIZATION_EXCEPTIONS`` after the first parse never -reaches it; extend a :class:`~nameparser.Lexicon`'s pairs instead -(:ref:`case-exceptions`). - -Five of these lists, ``PARTICLES``, ``NON_GIVEN_NAME_PARTICLES``, -``BOUND_GIVEN_NAMES``, ``GIVEN_NAME_TITLES`` and ``SUFFIX_WORDS``, were -renamed in 2.2 to match the field names used here, and two modules -moved with them; the 1.x names are gone in 3.0, and the 2.2.0 entry of -:doc:`release_log` maps them. - -Vocabulary entries are matched one word at a time, with two -exceptions, so a multi-word entry like ``titles={"grand moff"}`` can -never match; the constructor warns when it sees one -(``capitalization_exceptions`` keys included — they are looked up per -word too). The exceptions are ``given_name_titles``, looked up as the -space-joined run of words already read as titles or as that run's last -word — several titles written together are one form of address and the -last one does the addressing, so ``"Her Majesty Queen Elizabeth"`` is -read by ``queen`` — and -``maiden_markers``, matched by lookahead over the words as written: -``maiden_markers={"z domu"}`` matches the pair and neither word alone, -which is how the shipped Polish entry works. The words have to stand -together — a bracketed clause or a comma between them ends the run, and -the first word is then an ordinary name word. Where a phrase entry and a -word entry starting with it are both configured, the phrase wins where -it matches and the word matches everywhere else. No warning is raised -for a multi-word entry in either of these two fields, since there it is -not a mistake. - -The limit is on *storage*, not on the shape a name can have. Adjacent -suffix words are reassembled after they match, so a multi-word -credential is reachable as its component words even though the phrase -itself cannot be stored: - -.. doctest:: +``dean`` holds a title reading (its kind is ``office``) and a ``given`` +reading, and a title that may be a given name is a title only inside a +title phrase such as ``Dean of Students`` +(:ref:`titles-also-given-names`). Removing the ``given`` reading makes +it a title wherever it stands. Adding a title reading instead would +change nothing outside a phrase, so ``add("dean", kind="title")`` warns +and names the ``remove`` above. - >>> from nameparser import parse - >>> parse("John Smith, MD PhD").suffix - 'MD PhD' +:meth:`~nameparser.Lexicon.add` takes the words and a ``kind``, and +:meth:`~nameparser.Lexicon.remove` the words and, optionally, a kind; +both return a new :class:`~nameparser.Lexicon`, the one you started +from (here, ``Lexicon.default()``) never being mutated. Where a word +holds a reading of the kind added, ``add`` updates that reading in every +file holding it; otherwise it adds a reading of the caller's own. +``remove`` with no kind takes out every reading of the word, and with a +category (``"title"``, ``"name"``) every reading of that category. +:meth:`~nameparser.Lexicon.readings_of` answers what a word may be read +as here, each :class:`~nameparser.LexiconReading` naming the shipped +files that hold it; whether a parse *read* it that way is +:attr:`Token.reading `'s question. -That has held since 1.4.0. A credential whose words are not in the -default vocabulary is reached by adding those words, not the phrase. -After a comma the defaults already read an unlisted all-caps run (see -:ref:`all-caps-credentials`), so the difference shows at the end of a -name: +A credential the defaults do not list is one reading per word: .. doctest:: >>> parse("John Smith LEED AP").suffix '' - >>> lex = Lexicon.default().add(suffix_acronyms={"leed", "ap"}) + >>> lex = Lexicon.default().add("leed", "ap", kind="credential") >>> Parser(lexicon=lex).parse("John Smith LEED AP").suffix 'LEED AP' -Removing works the same way, and drops the word from recognition: +Removing works the same way, and a category takes out every reading of +it: .. doctest:: - >>> lean = Lexicon.default().remove(titles={"professor"}) + >>> lean = Lexicon.default().remove("professor", kind="title") >>> Parser(lexicon=lean).parse("Professor Robert Johns").title '' -Marking a subset of another field -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The 2.x set fields (``titles``, ``suffix_acronyms``, +``particles_ambiguous``, ...) are internal since 3.0, compiled from the +readings: ``add(titles=...)``, ``Lexicon(titles=...)`` and +``lexicon.titles`` each raise, the error naming the calls that configure +the same words, and :doc:`migrate` has the table. -Several fields mark a subset of another, qualifying how the base -field's words are read. Their entries belong in the base field too, so -add a word to both, and remove it from the marker first. An entry -outside its base is an *orphan*: four markers refuse one with a -``ValueError`` naming it, because there it would change the parse; the -rest leave it unchecked, because there it is never read. +The default word lists themselves — ``TITLES``, ``PARTICLES`` and the +other frozensets in ``nameparser.config`` — are the default lexicon's +compiled sets, frozen and read-only, so a runtime addition belongs on a +:class:`~nameparser.Lexicon` as above. ``REGEXES`` and +``CAPITALIZATION_EXCEPTIONS`` are the two members the freeze does not +cover — they are still plain dicts. Editing one at runtime is not a +supported override. The parser does not read ``REGEXES`` at all, and +``Lexicon.default()`` is cached, so an edit to +``CAPITALIZATION_EXCEPTIONS`` after the first parse never reaches it; +extend a :class:`~nameparser.Lexicon`'s pairs instead +(:ref:`case-exceptions`). + +Five of these lists, ``PARTICLES``, ``NON_GIVEN_NAME_PARTICLES``, +``BOUND_GIVEN_NAMES``, ``GIVEN_NAME_TITLES`` and ``SUFFIX_WORDS``, were +renamed in 2.2, and two modules moved with them; the 1.x names are gone +in 3.0, and the 2.2.0 entry of :doc:`release_log` maps them. + +Kinds and properties +~~~~~~~~~~~~~~~~~~~~~ + +A kind belongs to a category, which decides what the reading does: .. list-table:: :header-rows: 1 - :widths: 30 34 36 - - * - Marker - - Base - - An orphan - * - ``particles_ambiguous`` - - ``particles`` - - raises ``ValueError`` - * - ``suffix_acronyms_ambiguous`` - - ``suffix_acronyms`` - - raises ``ValueError`` - * - ``honorific_tails`` - - ``suffix_words`` - - raises ``ValueError`` - * - ``suffix_words_ambiguous`` - - ``suffix_words`` - - raises ``ValueError`` - * - ``agnomens`` - - ``suffix_words_ambiguous`` - - is never read - * - ``conjunctions_ambiguous`` - - ``conjunctions`` - - is never read - * - ``after_given_titles`` - - ``titles`` - - is never read - * - ``given_name_titles``, ``title_modifiers``, ``title_styles`` - - ``titles`` or ``titles_ambiguous`` - - is never read - -So ``remove(conjunctions={"e"})`` simply works, and the stale -``conjunctions_ambiguous`` entry is never consulted. -``given_name_titles`` is unchecked for a second reason: it matches a -run of titles as one space-joined string, or by the run's last word, so -a legitimate entry like ``"sir and dame"`` is no single word of -``titles``. ``titles_ambiguous`` is no subset of ``titles`` at all; see -:ref:`titles-also-given-names`. + :widths: 18 52 30 + + * - Category + - Kinds + - Properties + * - title + - ``title``, ``salutation``, ``style``, ``academic``, ``religious``, + ``nobility``, ``military``, ``office``, ``occupation``, + ``kinship``, ``modifier`` + - ``addresses="given"``, ``after_given=True``, ``attaches=[...]`` + * - suffix + - ``suffix``, ``generation``, ``agnomen``, ``credential``, + ``honorific``, ``professional_honorific``, ``status``, + ``courtesy`` + - ``acronym=...``, ``attached="after"`` + * - particle + - ``particle``, ``preposition``, ``article``, ``patronymic``, + ``teknonymic``, ``clan``, ``rank``, ``saint``, ``alias`` + - none + * - conjunction + - ``conjunction`` + - ``may_be_initial=True`` + * - bound_given + - ``bound_given`` + - none + * - relation + - ``birth_marker`` + - none + * - name + - ``given``, ``family`` + - ``same_as="..."`` on ``given`` + +``nameparser/vocab/README.md`` in the repository says what each kind +and property means. A property given its unwritten value (``None``, +``False``) is removed; ``acronym`` is unwritten-true on a +``credential`` alone. Two things are not properties: whether a reading +is on (a caller's always is) and its case mask (:ref:`case-exceptions`). + +A **name** reading beside another reading is what makes a word +ambiguous, as the vocabulary files mark it: a title with a ``given`` +reading is a title only inside a phrase, a particle with one may be a +given name, and a credential or a generation word with a name reading +is read as the suffix only where the writing or the count says so +(:ref:`ambiguous-words`). So ``add("guru", kind="given")`` +keeps ``Guru`` a title only inside a phrase, and +``remove("van", kind="given")`` makes ``van`` a particle that is never +a given name. + +A reading is of one word, with two exceptions, so ``add`` refuses a +phrase of any other kind: a title addressing the given name +(``add("lt col", kind="title", addresses="given")``), matched against +the space-joined run of words already read as titles or by that run's +last word — several titles written together are one form of address +and the last one does the addressing, so ``"Her Majesty Queen +Elizabeth"`` is read by ``queen`` — and a birth marker, matched by +lookahead over the words as written: ``add("z domu", +kind="birth_marker")`` matches the pair and neither word alone, which +is how the shipped Polish entry works. The words have to stand +together — a bracketed clause or a comma between them ends the run, and +the first word is then an ordinary name word. Where a phrase and a word +starting it are both configured, the phrase wins where it matches and +the word matches everywhere else. + +The limit is on what a reading holds, not on the shape a name can have. +Adjacent suffix words are reassembled after they match, so a multi-word +credential is reachable as its component words: + +.. doctest:: + + >>> parse("John Smith, MD PhD").suffix + 'MD PhD' + +That has held since 1.4.0. .. _title-detection-off: Turning title detection off ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The subset rule matters most when clearing a field wholesale. Emptying -``titles`` alone orphans every ``given_name_titles`` entry, so the two -go together, with ``titles_ambiguous``, whose words are titles inside a -phrase ("Dean of Students") without being in ``titles``: +Every title reading of every listed title word goes at once, the +category matching each kind; the word lists in ``nameparser.config`` +name the words: .. doctest:: - >>> d = Lexicon.default() - >>> lean = d.remove(titles=set(d.titles), - ... given_name_titles=set(d.given_name_titles), - ... titles_ambiguous=set(d.titles_ambiguous)) + >>> from nameparser.config.titles import TITLES, TITLES_AMBIGUOUS + >>> lean = Lexicon.default().remove(*TITLES, *TITLES_AMBIGUOUS, + ... kind="title") >>> Parser(lexicon=lean).parse("Dean of Students Jane Smith").title '' >>> Parser(lexicon=lean).parse("Hon Solo").given @@ -177,8 +195,8 @@ phrase ("Dean of Students") without being in ``titles``: Emptying the title vocabulary does not switch titles off entirely, though. A word ending in a period, standing at the front of the part that carries the given name, is read as a title structurally, without -consulting ``titles`` at all — that is what lets unfamiliar ranks and -abbreviations work (see :ref:`abbreviated-titles`): +consulting the vocabulary at all — that is what lets unfamiliar ranks +and abbreviations work (see :ref:`abbreviated-titles`): .. doctest:: @@ -191,16 +209,21 @@ abbreviations work (see :ref:`abbreviated-titles`): Combining two lexicons ~~~~~~~~~~~~~~~~~~~~~~~ -Whole lexicons compose with ``|``, which unions field by field — handy -for keeping a shared house vocabulary separate from a per-source one -and combining them at parser construction: +Whole lexicons compose with ``|`` — handy for keeping a shared house +vocabulary separate from a per-source one and combining them at parser +construction. The right-hand lexicon's edits apply after the left's, +so where the two disagree the later one wins; the ``surnames`` and +``given_names`` lists are unioned, and case masks taken right-biased: .. doctest:: - >>> house = Lexicon.empty().add(titles={"dean"}) - >>> per_source = Lexicon.empty().add(titles={"provost"}) - >>> sorted((house | per_source).titles) - ['dean', 'provost'] + >>> house = Lexicon.empty().add("dean", kind="office") + >>> per_source = Lexicon.empty().add("provost", kind="academic") + >>> both = house | per_source + >>> [r.kind for w in ("dean", "provost") for r in both.readings_of(w)] + ['office', 'academic'] + >>> [r.kind for r in (both | Lexicon.empty().remove("dean")).readings_of("dean")] + [] .. _case-exceptions: @@ -213,8 +236,8 @@ digits, each in the case it should take (``"phd"`` → ``"PhD"``). Case repair lays the mask over the word as it was written and keeps every other character where it stood, so it recases a word and never re-spells it. A value that spells anything else raises ``ValueError`` -when the lexicon is built. Being pair-valued, the field isn't a fit for -``add()``/``remove()``. Change it with ``dataclasses.replace()`` +when the lexicon is built. Being a mask rather than a reading, it isn't +a fit for ``add()``/``remove()``. Change it with ``dataclasses.replace()`` instead, and pass the result to ``capitalized()``: .. doctest:: @@ -295,7 +318,7 @@ inside a longer run: Words that need no entry ^^^^^^^^^^^^^^^^^^^^^^^^ -An acronym already listed in ``suffix_acronyms``, plain or dotted, and +An acronym the lexicon holds as a credential, plain or dotted, and a roman numeral are written in capitals by case repair on its own. Most listed acronyms whose usual spelling is not all capitals already carry a shipped mask (``DSc``, ``PsyD``, ``PharmD``, ``MDiv`` and others): @@ -322,7 +345,7 @@ in plain capitals. The reasoning is the Excluded block for Acronyms the vocabulary doesn't list ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -An acronym ``suffix_acronyms`` doesn't list repairs according to how it +An acronym the lexicon holds no reading of repairs according to how it is written: - **Plain** (``dclinpsy``), it parses as an ordinary name word and repairs @@ -331,7 +354,7 @@ is written: :ref:`unlisted-credentials`) and repairs in all capitals the same as a listed acronym does: ``D.CLINPSY.``. -Either way, give it a ``suffix_acronyms`` entry, or a mask of its own, +Either way, give it a credential reading, or a mask of its own, rather than relying on this fallback. .. _ambiguous-words: @@ -341,21 +364,20 @@ Words that are also ordinary names Some vocabulary words are also ordinary name words: an acronym suffix that is also borne as a surname, a particle that doubles as a given -name, a connective letter that doubles as an initial. Four fields mark -them — ``suffix_acronyms_ambiguous``, ``suffix_words_ambiguous``, -``particles_ambiguous`` and ``conjunctions_ambiguous``, one each for -``suffix_acronyms``, ``suffix_words``, ``particles`` and -``conjunctions``. They add no vocabulary by themselves; they narrow how -an existing entry is read when it appears alone, and the parse reports -the fork as an ambiguity. Titles that are also given names work -differently, with a word list of their own, ``titles_ambiguous`` -(:ref:`titles-also-given-names`). +name, a connective letter that doubles as an initial. What marks them is +a second reading beside the first: a ``given`` or ``family`` reading +beside a credential, a generation word or a particle, and the +``may_be_initial`` property on a one-letter connective. The second +reading adds no vocabulary by itself; it narrows how the word is read +when it appears alone, and the parse reports the fork as an ambiguity. +A title with a ``given`` reading works differently, being a title only +inside a phrase (:ref:`titles-also-given-names`). Credentials that are also surnames ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -``ma`` is a shipped example of ``suffix_acronyms_ambiguous``. It is -both a credential and a common surname, so it is listed there. Written +``ma`` is a shipped example: it holds a credential reading and the +name readings of a common surname. Written with its periods it is a suffix outright; written bare, the capitalization decides: @@ -400,11 +422,11 @@ The full reading of a bare marked acronym, in order: Generation words that are also surnames ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -``suffix_words_ambiguous`` does the same for a suffix word: French -``fils`` and the Brazilian agnomens ``Filho``, ``Neto`` and -``Sobrinho`` mark a generation, and each is a surname too. An agnomen, -written in the case of the name around it and listed in ``agnomens``, -is a suffix only where the name keeps a word for its family. The +A name reading beside a suffix word does the same: French ``fils`` and +the Brazilian agnomens ``Filho``, ``Neto`` and ``Sobrinho`` mark a +generation, and each is a surname too. An agnomen (the ``agnomen`` +kind), written in the case of the name around it, is a suffix only +where the name keeps a word for its family. The writing of ``fils`` speaks as well: in a mixed-case name a Title-case ``Fils`` is the family name even with a word to spare: @@ -417,16 +439,12 @@ writing of ``fils`` speaks as well: in a mixed-case name a Title-case >>> parse("Marie Fils").family 'Fils' -A caller whose data writes an agnomen as part of the surname takes it -out by removing the word from ``suffix_words`` and from the two -markers over it, together: an orphan in ``suffix_words_ambiguous`` -raises ``ValueError``, and one left in ``agnomens`` is simply never read: +A caller whose data writes an agnomen as part of the surname takes out +its suffix reading, leaving its name reading: .. doctest:: - >>> agnomen = {"neto"} - >>> kept = Lexicon.default().remove( - ... agnomens=agnomen, suffix_words_ambiguous=agnomen, suffix_words=agnomen) + >>> kept = Lexicon.default().remove("neto", kind="agnomen") >>> Parser(lexicon=kept).parse("Pedro Alves Neto").family 'Neto' @@ -437,17 +455,16 @@ When you add a word that is both a credential and a name, weigh how often it is one against the other. There are three choices, and none is free: -- **Unambiguous** (``suffix_acronyms`` alone): the credential reading - is taken silently, and a wrong claim can lose a real person's surname. -- **Marked ambiguous** (``suffix_acronyms_ambiguous`` too): the reading - depends on the writing as above, and every bare reading reports the - fork. -- **Left out of** ``suffix_acronyms`` **altogether**: right when the name - reading is far more common, as for an acronym whose credential is - tenuous or specialized beside a common surname. That is what the - default vocabulary did with ``rai`` and ``cha`` in 2.3; a caller who - needs one adds it back with - ``Lexicon.default().add(suffix_acronyms={"cha"})``. +- **A credential alone**: the credential reading is taken silently, + and a wrong claim can lose a real person's surname. +- **A credential and a name reading**: the reading depends on the + writing as above, and every bare reading reports the fork. +- **No credential reading at all**: right when the name reading is far + more common, as for an acronym whose credential is tenuous or + specialized beside a common surname. That is what the default + vocabulary did with ``rai`` and ``cha`` in 2.3; a caller who needs + one adds it back with ``Lexicon.default().add("cha", + kind="credential")``. .. _titles-also-given-names: @@ -459,8 +476,8 @@ Many titles are borne as given names too: ``Dean``, ``King``, name after it, so at the front of a name there is no count of words to read, and what tells the person ``Dean Smith`` from the office in ``Dean of Students`` is the phrase the word stands in. Since 2.7 these -words are listed in ``titles_ambiguous`` rather than ``titles``, and -each is a title only inside a title phrase: behind a style +words hold a ``given`` reading beside their title reading, and each is +a title only inside a title phrase: behind a style (``His Majesty King``) or a modifier (``Associate Dean``), before ``of`` (``Dean of Students``), or in a listed two-word phrase (``Major General``). Anywhere else it is a name word: @@ -490,40 +507,46 @@ standing alone: >>> parse("Grace Smith").ambiguities () -Four fields hold this vocabulary: +Four things in the readings make this vocabulary: -- ``titles_ambiguous`` — the titles read only inside a phrase. A - word listed in ``titles`` as well is a title wherever it stands, so - ``add(titles={"dean"})`` (the first example on this page) is how to - read one as a title everywhere. -- ``title_modifiers`` — the words that qualify a head title +- A ``given`` reading beside a title reading makes the title one read + only inside a phrase; removing it, as the first example on this page + does, reads the word as a title everywhere. +- The ``modifier`` kind is a word that qualifies a head title (``associate``, ``vice``). -- ``title_styles`` — the forms of address a title may stand behind +- The ``style`` kind is a form of address a title may stand behind (``majesty``, ``highness``, ``excellency``). -- ``title_phrases`` — the two-word phrases, each read only where - both words are titles. Removing one reads its words apart: +- ``attaches`` on a title reading names the title kinds, or the title + words, it makes a two-word phrase with, each phrase read only where + both words are titles. Narrowing it reads the words of a phrase it + no longer reaches apart: .. doctest:: >>> parse("Chief Justice John Roberts").title 'Chief Justice' - >>> chief = Lexicon.default().remove(title_phrases={"chief justice"}) + >>> [r.properties for r in Lexicon.default().readings_of("chief")] + [(('attaches', ('military', 'office')),)] + >>> chief = Lexicon.default().add("chief", kind="nobility", + ... attaches=["military"]) >>> Parser(lexicon=chief).parse("Chief Justice John Roberts").title 'Chief' + >>> Parser(lexicon=chief).parse("Chief Sergeant John Roberts").title + 'Chief Sergeant' Particles that are also given names ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -``particles_ambiguous`` is the same idea for surname particles. A -particle listed there may also be a given name, which is what makes a -leading one a decision to take; a particle *not* listed there never -is, so there is nothing to decide. That shows up in what a particle +A ``given`` reading is the same idea for surname particles. A particle +holding one may also be a given name, which is what makes a leading +one a decision to take; a particle *without* one never is, so there is +nothing to decide. That shows up in what a particle standing *alone* at the front of a name does: a listed one is a name -part in its own right, while an unlisted one pulls the rest of the +part in its own right, while one without pulls the rest of the name into the surname and leaves no given name at all. Which field a -*listed* particle lands in is ``name_order``'s question, covered -below; an unlisted one opening the name is the surname under every -order, because a word that can never be a given name leaves the order +particle that may be given lands in is ``name_order``'s question, +covered below; one that may not, opening the name, is the surname +under every order, because a word that can never be a given name leaves the order nothing to decide. .. doctest:: @@ -545,32 +568,32 @@ follows the comma is the given name as usual: >>> parse("de Mesnil, Juan").given # the comma named the surname 'Juan' -If your data never uses ``Van`` as a given name, take it out of the -ambiguous set: a leading ``van`` is then no decision at all, so no +If your data never uses ``Van`` as a given name, take out its +``given`` reading: a leading ``van`` is then no decision at all, so no ambiguity is recorded and it becomes part of the surname — under any ``name_order``, since that is what taking the word out asserted: .. doctest:: - >>> lex = Lexicon.default().remove(particles_ambiguous={"van"}) + >>> lex = Lexicon.default().remove("van", kind="given") >>> Parser(lexicon=lex).parse("van Gogh").family 'van Gogh' -For ``particles_ambiguous`` the default runs the other way from -credentials: a particle that is not borne as a given name belongs -outside the set, which is where ``mc`` and ``ste`` were moved (#360). +For particles the default runs the other way from credentials: a +particle that is not borne as a given name holds no ``given`` reading, +which is where ``mc`` and ``ste`` were moved (#360). .. _connectives-also-initials: One-letter connectives that are also initials ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -``conjunctions_ambiguous`` is the same idea for one-letter connectives. +``may_be_initial`` is the same idea for one-letter connectives. A single letter written against the name's own case is an initial and one written with it is the connective — but a name written wholly in one case, all upper or all lower, says nothing either way, and this is -the set that decides it there. ``e`` and ``i`` are the two entries -shipped: a bare ``E`` or ``I`` initial is common where those letters +the property that decides it there. ``e`` and ``i`` are the two +connectives shipped with it: a bare ``E`` or ``I`` initial is common where those letters between two surnames are rarer, and ``y`` runs the other way, so ``y`` joins even written as a bare capital. @@ -583,7 +606,7 @@ joins even written as a bare capital. >>> parse("Jose e Maria Santos").given # mixed case decides itself 'Jose e Maria' -A member also reports the fork, so a caller can see which reading was +Such a connective also reports the fork, so a caller can see which reading was taken: .. doctest:: @@ -596,7 +619,8 @@ does in Spanish, take it out and the connective reading comes back: .. doctest:: - >>> lex = Lexicon.default().remove(conjunctions_ambiguous={"e"}) + >>> lex = Lexicon.default().add("e", kind="conjunction", + ... may_be_initial=False) >>> Parser(lexicon=lex).parse("jose e maria santos").given 'jose e maria' @@ -606,7 +630,8 @@ joins in a one-case name too: .. doctest:: - >>> lex = Lexicon.default().remove(conjunctions_ambiguous={"i"}) + >>> lex = Lexicon.default().add("i", kind="conjunction", + ... may_be_initial=False) >>> Parser(lexicon=lex).parse("josep carod i rovira").family 'carod i rovira' @@ -615,14 +640,15 @@ never a connective, add the other one instead: .. doctest:: - >>> lex = Lexicon.default().add(conjunctions_ambiguous={"y"}) + >>> lex = Lexicon.default().add("y", kind="conjunction", + ... may_be_initial=True) >>> Parser(lexicon=lex).parse("juan garcia y lopez").middle 'garcia y' Bound given names ~~~~~~~~~~~~~~~~~~ -``bound_given_names`` holds given-name prefixes that attach to the +A ``bound_given`` reading is a given-name prefix that attaches to the following word to form one given name — ``abdul``, ``abu``, ``umm`` and their Arabic-script spellings (``عبد``, ``أبو``, ``أم``) among them: @@ -631,15 +657,16 @@ their Arabic-script spellings (``عبد``, ``أبو``, ``أم``) among them: >>> parse("abdul salam ahmed salem").given 'abdul salam' -Add your own, or empty the set to switch the behavior off entirely: +Add your own, or take out every one to switch the behavior off +entirely: .. doctest:: - >>> lex = Lexicon.default().add(bound_given_names={"mohamad"}) + >>> lex = Lexicon.default().add("mohamad", kind="bound_given") >>> Parser(lexicon=lex).parse("mohamad salam ahmed salem").given 'mohamad salam' - >>> d = Lexicon.default() - >>> off = d.remove(bound_given_names=set(d.bound_given_names)) + >>> from nameparser.config.bound_given_names import BOUND_GIVEN_NAMES + >>> off = Lexicon.default().remove(*BOUND_GIVEN_NAMES, kind="bound_given") >>> Parser(lexicon=off).parse("abdul salam ahmed salem").given 'abdul' @@ -1035,7 +1062,7 @@ matches the word with them: .. doctest:: >>> listed = Parser(policy=Policy(two_family_names=True), - ... lexicon=Lexicon.default().add(given_names={"luis"})) + ... lexicon=Lexicon.default() | Lexicon(given_names={"luis"})) >>> two_names.parse("José Luis Pérez").family 'Luis Pérez' >>> luis = listed.parse("José Luis Pérez") @@ -1161,12 +1188,13 @@ fields. Teaching the splitter a surname ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -To teach the splitter a surname it doesn't ship with, add it to the -``surnames`` vocabulary like any other word: +To teach the splitter a surname it doesn't ship with, join a lexicon +holding it to the ``surnames`` list, a list rather than readings since +it divides a name instead of gating a word: .. doctest:: - >>> lex = Lexicon.default().add(surnames={"김민"}) + >>> lex = Lexicon.default() | Lexicon(surnames={"김민"}) >>> Parser(lexicon=lex).parse("김민준").family '김민' @@ -1218,25 +1246,24 @@ Two behaviors apply however both fields are set: >>> both_off.parse("田中さん").suffix 'さん' -The honorific vocabulary is the peel's off-switch. Removing an entry -from ``honorific_tails`` leaves that honorific glued, while the spaced -form still reads it as a suffix: +The honorific vocabulary is the peel's off-switch. Taking the +``attached`` property off an honorific's reading leaves that honorific +glued, while the spaced form still reads it as a suffix: .. doctest:: - >>> no_san = Parser( - ... lexicon=Lexicon.default().remove(honorific_tails={"さん"})) + >>> no_san = Parser(lexicon=Lexicon.default().add( + ... "さん", kind="honorific", attached=None)) >>> no_san.parse("田中さん").family '田中さん' >>> no_san.parse("田中 さん").suffix 'さん' -``honorific_tails`` marks a subset of ``suffix_words``, so dropping a -word from it alone orphans nothing and needs no matching -``suffix_words`` edit. Emptying it is also how to opt out of what the -peel *costs* a non-ASCII parse: an empty ``honorific_tails`` stops the -peel at its first check, whereas ``segment_scripts`` never gated it and -so cannot turn it off. +The suffix reading stays, so the word is still a suffix written apart. +Taking ``attached`` off every honorific is also how to opt out of what +the peel *costs* a non-ASCII parse: with none attached the peel stops +at its first check, whereas ``segment_scripts`` never gated it and so +cannot turn it off. .. _brackets: @@ -1256,7 +1283,7 @@ no brackets and nothing configured: :ref:`nicknames-and-maiden-names` in the usage guide covers how the marked forms read. The marker words themselves are the -``maiden_markers`` vocabulary, a ``Lexicon`` field, shipped in several +``birth_marker`` readings of a ``Lexicon``, shipped in several languages (see :mod:`nameparser.config.maiden_markers`). Add your own like any other word; a marker you add works bare and in brackets alike: @@ -1266,7 +1293,7 @@ alike: >>> parse("Jane Smith formerly Jones").maiden # not a marker '' >>> marked = Parser( - ... lexicon=Lexicon.default().add(maiden_markers={"formerly"})) + ... lexicon=Lexicon.default().add("formerly", kind="birth_marker")) >>> marked.parse("Jane Smith formerly Jones").maiden 'Jones' >>> marked.parse("Jane Smith (formerly Jones)").maiden @@ -1730,7 +1757,7 @@ you parse. # myapp/names.py from nameparser import Lexicon, Parser, Policy - lex = Lexicon.default().add(titles={"dean"}) + lex = Lexicon.default().remove("dean", kind="given") policy = Policy(strip_emoji=False) parser = Parser(lexicon=lex, policy=policy) diff --git a/docs/design/decisions/O6-two-family-names.md b/docs/design/decisions/O6-two-family-names.md index 2a3351868..8f91f7562 100644 --- a/docs/design/decisions/O6-two-family-names.md +++ b/docs/design/decisions/O6-two-family-names.md @@ -120,7 +120,7 @@ Open: hyphenated surnames block the switch (`Andrés Sánchez-Cid Torres` keeps The same boxes rendered family box first, read under FAMILY_FIRST on both sides: ES 47.6% → 91.4% → 94.5% (681 fixed, 68 broken), MX 31.4% → 94.6% → 96.3% (465, 132), CO 23.9% → 94.5% → 97.6% (640, 46), PE 12.6% → 95.9% → 95.9% (203, 205), CL 46.3% → 93.4% → 97.0% (729, 38), VE 38.5% → 88.6% → 93.3% (949, 44), AR 82.3% → 59.8% → 90.8% (6,068, 53). PSC (`psc-individuals`, the fetch recorded on #849), a second population the list was not built from, by nationality, default → switch → switch and list, fixed and broken: Spanish 6,089 records 31.8% → 80.2% → 82.2% (150, 31), Mexican 402 25.9% → 80.1% → 82.1% (13, 5), Colombian 377 29.2% → 84.4% → 87.0% (15, 5), Venezuelan 237 40.5% → 75.9% → 80.6% (13, 2), Chilean 151 35.1% → 77.5% → 84.8% (12, 1), Peruvian 116 21.6% → 81.0% → 83.6% (5, 2), Argentine 320 81.9% → 45.6% → 81.9% (121, 5). - Recompute: `uv run --with openpyxl --with xlrd python tools/es_given_names.py --top N` writes the list for a cutoff (`--write`) and prints its coverage. For ORCID, `local_agreement.read` per record of `local_corpora.load("orcid-researchers")` with `FIELDED["orcid-researchers"]`, records with both boxes, grouped by `country`, under `Parser()`, `Parser(policy=Policy(two_family_names=True))` and that policy with `lexicon=Lexicon.default().add(given_names=...)` holding the tool's first N words, each with `_title_words` of its own lexicon; family-first, the record's `name` replaced by `family + " " + given` and each policy given `name_order=FAMILY_FIRST`. For PSC, `local_agreement.score(rows, FIELDED["psc-individuals"], parser, jobs=1)` over the records whose `group_of` is the nationality, each cell a group's `agreed` over its `scored`. A record is fixed where it agrees only with the list, broken where it agrees only without it. The comparator is the switch alone on the same tree; `local_agreement.py orcid-researchers --base` and `psc-individuals --base` against master match the base, the default reading no list. + Recompute: `uv run --with openpyxl --with xlrd python tools/es_given_names.py --top N` writes the list for a cutoff (`--write`) and prints its coverage. For ORCID, `local_agreement.read` per record of `local_corpora.load("orcid-researchers")` with `FIELDED["orcid-researchers"]`, records with both boxes, grouped by `country`, under `Parser()`, `Parser(policy=Policy(two_family_names=True))` and that policy with `lexicon=Lexicon.default() | Lexicon(given_names=...)` (`add(given_names=...)` before 3.0's 4i-b) holding the tool's first N words, each with `_title_words` of its own lexicon; family-first, the record's `name` replaced by `family + " " + given` and each policy given `name_order=FAMILY_FIRST`. For PSC, `local_agreement.score(rows, FIELDED["psc-individuals"], parser, jobs=1)` over the records whose `group_of` is the nationality, each cell a group's `agreed` over its `scored`. A record is fixed where it agrees only with the list, broken where it agrees only without it. The comparator is the switch alone on the same tree; `local_agreement.py orcid-researchers --base` and `psc-individuals --base` against master match the base, the default reading no list. - 2026-10-30 #959, decided — THE VETO STAYS A CLAUSE OF O6, THOUGH IT READS A SECOND SIGNAL. docs/design/AGENTS.md ("One rule per piece of evidence") says a clause reading a signal its rule did not already read is a new rule, and the veto reads one O6 did not: the word standing where the family would take one, looked up in the caller's given-name list (the #961 review). Kept in O6 anyway, decided 2026-10-30: the veto is one more member of the class the Except bullet "What is taken must be one name word a family name could be" already holds, beside the initial written alone and the mark with no letter in it. It only declines a take O6 would otherwise make, and reads nothing of its own: with the switch off it is never asked, and where it declines the name reads as with the switch off. A rule of its own would state no reading but O6's absence. The Signal line names the lookup as a second signal joined with ` + `, tagged `position` as P1's vocabulary signal is, by where the word stands, so O6 shows in the split-candidate grep; the next signal O6 gains is the point to split it. - 2026-10-30 #753, phase 3g, decided in chat — `locales.ES` TURNS THE SWITCH ON; `locales.PT` DOES NOT. The `es` pack's `[rules]` table writes `two_family_names = true` (sub-phase 4f's schema), compiled into its policy patch, and the pack carries `nameparser/locales/_es_given_names.py`'s list as its lexicon fragment, so a caller choosing it reads `Gabriel García Márquez` with family `García Márquez` and `José Luis Pérez` with family `Pérez`. `two_given_names` stays opt-in in both packs (decisions/O7's phase 3g entry). The table answers two questions, each by its own comparison. Against the DEFAULT: should the pack turn the switch on at all, which is what a caller choosing the pack gets over not choosing it; every Spanish-speaking country gains on ORCID, Argentina least (82.5% → 89.3%), and PSC by nationality gains everywhere but Argentina, which ties. Against O6 ALONE: does the list help once the switch is on, the #959 entry's cutoff question, here re-measured on the pack as shipped, so the column holds the es choice's titles beside the list; against O6 with the list and no choice the pack differs on 9 ORCID records of these countries, all broken by a title the choice turns on (Cuba 7, `Lic.Alina Maren Ferrier`, `Lic Arisleidy Noriega Cisneros`; Venezuela 2, `Doctora Yenny Nereida Montilla Polo`, `PROFESORA CAROLINA GONZALEZ`), the box holding the title the filer typed in; it does in every country but Peru (178 fixed, 189 broken), the one the #959 review found net negative, a loss of 0.1 points that the pack's gain over the default there (16,197 fixed, 146 broken) dwarfs. So the Open line on Latin America stays a question about a better list, not about whether the pack should read two family names. The rows for the United States and Great Britain show the cost to data the pack is not meant for, which a caller choosing it accepts. diff --git a/docs/design/decisions/R8-generation-and-credentials-views.md b/docs/design/decisions/R8-generation-and-credentials-views.md index 058c85552..64f0e93fb 100644 --- a/docs/design/decisions/R8-generation-and-credentials-views.md +++ b/docs/design/decisions/R8-generation-and-credentials-views.md @@ -3,21 +3,22 @@ ## Standing - `ParsedName.generation` and `ParsedName.credentials` partition `suffix` by the reading each word took, a generation or an agnomen in the first and every other word in the second; they are views, not roles, so `suffix`, `as_dict()`, `replace()`'s fields and the repr are unchanged, item 4 of decisions/vocabulary-model's #754 output-model entry — 2026-10-29 #754, sub-phase 4b. -- A listed suffix word's kind is read from the reading index (`_reading_index.suffix_kind_of`, through `_vocab.suffix_kind`), since the compiled sets do not record it — 2026-10-29 #754, sub-phase 4b. +- A listed suffix word's kind is the lexicon's, compiled with its sets (`Lexicon._sets.generation_words`, `credential_words`), which on the default is the shipped vocabulary's — 2026-11-02 #754, sub-phase 4i-b, superseding 4b's read from the reading index. Elsewhere: the field's composition and the 2026-08-16 decision to keep it one field, which still stands, decisions/suffix-field-composition; whether `credentials` should leave the honorifics out, CLDR having no third field, is a value question under the RFC's Phase 4, Left open (docs/design/vocabulary-model-rfc.md). -Elsewhere: a word the shipped vocabulary lists as a credential reads as one where a caller lists it in `suffix_words_ambiguous`, the gated credential words `kc`, `pc` and `dl` reading so, decisions/S11's 2026-10-30 #965 entry. +Elsewhere: the gated credential words `kc`, `pc` and `dl`, a credential written as a word beside a name reading, read as credentials, decisions/S11's 2026-10-30 #965 entry. Open: none in this file. ## Log - 2026-10-29 #754, sub-phase 4b (#326) — IMPLEMENTED item 4 of decisions/vocabulary-model's 2026-10-28 #754 entry. SUPERSEDES: decisions/A1's 2026-10-26 #752 ACCEPTED clause that a listed suffix word neither a generation by its tag nor an agnomen reads as `post-nominal`, `jr` and `esq` alike. What the implementation decided, beyond that entry: - 1. **The kind of a listed suffix word comes from the reading index.** `jr` and `esq` compile into `suffix_words` with the same tags, so 4a read both as `post-nominal` and pinned the gap with strict xfails. Classify writes a kind tag beside `vocab:suffix-word` (`vocab:generation-word`, `vocab:credential-word`), asking `_reading_index.suffix_kind_of` through `_vocab.listed_word_kind_tag`, a memo whose hit costs no frame, and `_vocab.suffix_kind` reads the tag, so `Token.reading` and `Ambiguity.alternative` move together: a lone `Jr` names a `generation` alternative where it named a `post-nominal` one, and a credential written as a word (`博士`) reads `credential`. The vocabulary's courtesy, honorific, professional honorific and status kinds have no reading kind of their own and stay `post-nominal`. `_vocab` imports `_reading_index` for it, which tests/v2/test_layering.py's table records. The first draft looked the kind up in `suffix_kind` from the token's text, which keeps a delimiter glued on (rules/C10): `John Smith, Jr.; MD` read `Jr.;` post-nominal and put it in the credentials, as `John Smith, V; MD` did a numeral, found by the orchestrating review of #947. Classify reads every word by its glued stem already, so the kind is read there, by the key every other tag of the word is read by, and assemble drops the two tags from the published token, `Token.reading` carrying the kind out: `Token` compares its tags, and a published tag would make a name unequal to its own 2.x pickle (tests/v2/test_types.py's 2.8.0 fixture, `…de la Vega III`, caught it); a declared delimiter glued to either end (`Jr./`, `-Jr.`) reads alike, while before the first comma, where C10 reads a glued delimiter as part of the word and classify cuts no stem, `Jr.;` is listed by no set and reaches the suffix only by `period_joined_vocab`, which reads it by its chunks (`Jr.` and `;`, the `Lt.Gov.` path) and tags it `vocab:suffix` with no kind, so `John Smith Jr.; MD` keeps it in the credentials, as `Jr.III` is there after a comma or not, a word made only of generation words; `John Smith Jr; MD`, without the period, reads family `Jr;`. Recorded as rules/R8's cost rather than given a kind of its own (two reviews of the fix commits); a view keeps the delimiter in its word's text, as the suffix does, so `John Smith, Jr.; MD` gives generation 'Jr.;'; and `test_every_suffix_words_reading_alternative_and_index_kind_agree` walks every suffix-set word (826 on 2026-10-29) holding the reading taken, the alternative and the index's kind to one answer. - 2. **The kind is read whatever the reading's default.** `st`, a Czech generation written `default = false`, reads `generation` when a caller's `Lexicon` lists it, while its `languages` name no file, since none holds it on. A pack turning such a reading on (phase 3's `cs`, 3f) then needs nothing more for the kind; a word only a caller's own `Lexicon` lists has no row and reads `post-nominal`. + 1. **The kind of a listed suffix word comes from the reading index** (SUPERSEDED by the 2026-11-02 #754, sub-phase 4i-b entry below: it is the lexicon's). `jr` and `esq` compile into `suffix_words` with the same tags, so 4a read both as `post-nominal` and pinned the gap with strict xfails. Classify writes a kind tag beside `vocab:suffix-word` (`vocab:generation-word`, `vocab:credential-word`), asking `_reading_index.suffix_kind_of` through `_vocab.listed_word_kind_tag`, a memo whose hit costs no frame, and `_vocab.suffix_kind` reads the tag, so `Token.reading` and `Ambiguity.alternative` move together: a lone `Jr` names a `generation` alternative where it named a `post-nominal` one, and a credential written as a word (`博士`) reads `credential`. The vocabulary's courtesy, honorific, professional honorific and status kinds have no reading kind of their own and stay `post-nominal`. `_vocab` imports `_reading_index` for it, which tests/v2/test_layering.py's table records. The first draft looked the kind up in `suffix_kind` from the token's text, which keeps a delimiter glued on (rules/C10): `John Smith, Jr.; MD` read `Jr.;` post-nominal and put it in the credentials, as `John Smith, V; MD` did a numeral, found by the orchestrating review of #947. Classify reads every word by its glued stem already, so the kind is read there, by the key every other tag of the word is read by, and assemble drops the two tags from the published token, `Token.reading` carrying the kind out: `Token` compares its tags, and a published tag would make a name unequal to its own 2.x pickle (tests/v2/test_types.py's 2.8.0 fixture, `…de la Vega III`, caught it); a declared delimiter glued to either end (`Jr./`, `-Jr.`) reads alike, while before the first comma, where C10 reads a glued delimiter as part of the word and classify cuts no stem, `Jr.;` is listed by no set and reaches the suffix only by `period_joined_vocab`, which reads it by its chunks (`Jr.` and `;`, the `Lt.Gov.` path) and tags it `vocab:suffix` with no kind, so `John Smith Jr.; MD` keeps it in the credentials, as `Jr.III` is there after a comma or not, a word made only of generation words; `John Smith Jr; MD`, without the period, reads family `Jr;`. Recorded as rules/R8's cost rather than given a kind of its own (two reviews of the fix commits); a view keeps the delimiter in its word's text, as the suffix does, so `John Smith, Jr.; MD` gives generation 'Jr.;'; and `test_every_suffix_words_reading_alternative_and_index_kind_agree` walks every suffix-set word (826 on 2026-10-29) holding the reading taken, the alternative and the index's kind to one answer. + 2. **The kind is read whatever the reading's default** (SUPERSEDED by the 2026-11-02 #754, sub-phase 4i-b entry below: a caller's `add("st", kind="suffix")` reads `St` as a post-nominal). `st`, a Czech generation written `default = false`, reads `generation` when a caller's `Lexicon` lists it, while its `languages` name no file, since none holds it on. A pack turning such a reading on (phase 3's `cs`, 3f) then needs nothing more for the kind; a word only a caller's own `Lexicon` lists has no row and reads `post-nominal`. 3. **Each word is placed by its own reading, and separated as the writer separated it.** A `joined` mark is not only the second word of a split credential (`Ph.` `D.`, which takes the first word's reading) but every word of a suffix run written without commas (`Jr. III PhD`), so a view places each word by its reading. Two words of one view are separated by a comma where the suffix holds one anywhere between them, R1's separator (a typed comma, or a semicolon, hyphen, slash or delimiter C10 drops: `MD / Jr. / PhD` gives credentials 'MD, PhD'), and by a space where not, so `MD Jr. PhD` gives credentials 'MD PhD'; the first draft started a new comma-joined entry wherever a word of the other view stood between, writing a comma nobody typed ('MD, PhD'), found by the design-docs review. A connective goes with the word before it (`PhD and MD`, `Jr. and III`), so one joining a generation to a credential dangles at the end of the generation (`Jr. and MD` gives generation 'Jr. and'), accepted rather than given a rule of its own, no corpus name holding the shape. 4. **A suffix word with no reading is a credential.** `replace()` and `Parser.revise()` write tokens with no `Token.reading` (4a), and a `ParsedName` unpickled from 2.x has none either, so `replace(suffix="Jr.")` reads credentials 'Jr.' and the 2.8 pickle of `Dr. Juan Q. de la Vega III` credentials 'III': nothing records what the word was read as, and the catch-all is the view that claims least. A name word a run took into the suffix by position is a credential too, 4a's reading of it (`Smith, Jr., John` gives credentials 'John'). 5. **The views stay out of `as_dict()`, `--json` and the repr**, as `family_base` and the other derived views do: `as_dict()` is the seven fields `replace()` takes back. They are `render()` keys and assertable fields of a rule's example lines. 6. **The snapshot records them** (`tools/differential/snapshot.py`, a `RECORDED_LATER` pair): `generation` where the suffix holds one, and `credentials` beside it where the suffix holds both, so a row with neither is a suffix of credentials only, or none. No other key moved on recording. - Measured 2026-10-29 on the 4b branch over the committed snapshot (`tools/differential/snapshot.jsonl`, 6,570 readings, rules/R8's own examples included): 1,077 rows hold a suffix, 488 of them a generation, 142 a generation and credentials both. Recompute: count rows with `suffix`, `generation` and `credentials` in the snapshot. Over `cldr_fields.jsonl`, 576 distinct (text, order) renderings carry CLDR's `generation` or `credentials`; 65 parse to exactly CLDR's suffix words, and all 65 split into CLDR's two fields. Recompute: parse each distinct text under its order (`givenFirst` as `GIVEN_FIRST`, `surnameFirst` as `FAMILY_FIRST`, any other order the default), keep those whose `suffix` words, commas removed, equal CLDR's `generation` and `credentials` words together, and compare each view's words with its field's. The RFC's 615 and 75 (2026-10-28) counted renderings, not distinct (text, order) pairs. +- 2026-11-02 #754, sub-phase 4i-b — A LISTED SUFFIX WORD'S KIND IS THE LEXICON'S. SUPERSEDES items 1 and 2 of the 4b entry above. A caller now states a word's kind (`add("secundus", kind="generation")`), and a view reading the index instead would ignore it in silence, so the compile gives the kinds the sets do not record, `GENERATION_WORDS` and `CREDENTIAL_WORDS`, and classify reads them off the lexicon (rules/R8 amended). On the default they are the index's by the compile invariant (`test_the_default_kind_sets_are_the_shipped_index_s`), so no default parse moves; one test's caller-configured reading did: `Lexicon.default().add("st", kind="suffix")` reads `St` as a post-nominal where 4b read the cs file's generation, written `default = false`. An agnomen needs a name reading, as the build's check 11 requires of a file, so `add(w, kind="agnomen")` alone is refused. decisions/vocabulary-model's 2026-11-02 #754, sub-phase 4i-b entry has the rest. diff --git a/docs/design/decisions/R9-patronymic-view.md b/docs/design/decisions/R9-patronymic-view.md index b38d9c021..e1d2203ce 100644 --- a/docs/design/decisions/R9-patronymic-view.md +++ b/docs/design/decisions/R9-patronymic-view.md @@ -16,10 +16,11 @@ Open: the Icelandic ending rule, which an `is` pack would turn on (the RFC's wor 1. **Every word of a particle's patronymic reads `patronymic`.** The entry left open whether the particle reads `patronymic`, or only the name after it; the view's text is `bin Mohamad`, so a view over one reading kind needs both, and `ReadingKind.PATRONYMIC`'s docstring already gave "bin Mohamad" as the words it names. The particle's reading is the vocabulary's (`vocabulary`, its files `ar ms` for `bin`), each word after it `position`: what makes it a patronymic is where it stands, after the particle, so it names no file, and its own reading (a name word, a bound given name, a chained particle) is not kept. Weighed against: the particle alone reading `patronymic` and the view adding the words after it, which would put a run-walk in the view and make `Token.reading` say `name` for a word the view counts as the parent's name. 2. **The run is the field, read at assemble.** The words after the particle in its field, a nickname skipped and a suffix ending it, measured over the final roles rather than group's pieces, since P1's fold and P6's attachment move words after group and a leading never-given particle stays a lone piece for the fold to find. So the run is what the family or middle holds after the particle: `ibn Ali Hasan` gives `ibn Ali Hasan` by default and `ibn Ali` under a family-first order, as the family does. Under `middle_as_family` the run follows the field the word stands as, so a folded middle's run stops at the family's own words. 3. **A particle names a parent only with a word after it**, or where P6 attached it after a family comma or a family-first order to the family written before it, whose words then join the run (`Mohamad, Mahathir bin` gives `bin Mohamad` and `Bakar, Ali bin Abu` `bin Abu Bakar`, written in the family's order since P6's words carry the folded mark the family writes first). P6 records the attachment on the words it attaches (`vocab:attached`, beside the folded mark, at both of its sites), which assemble reads and drops as it drops the kind tag, so it is read, not inferred: O3's middle fold writes the same folded mark, and two inferences from shape each lost a case (item 8). The first draft read every patronymic particle read as a particle, and the snapshot showed `Fritz Freiherr und von Bar` reading patronymic `Bar`: the German family name spelled as Hebrew `bar`, ending its field, which group's chain had left a particle. - 4. **The kind is read at classify from the reading index**, as R8's suffix kind is: `particles` compiles every particle kind alike, so classify writes `vocab:patronymic` beside `particle` for a word the shipped files hold as a `patronymic` particle (`_vocab.is_patronymic_particle`, a memo over `_reading_index.holds_kind`), whatever the reading's default, and assemble drops the tag from the published token, a published tag making a name unequal to its 2.x pickle. Every patronymic particle has that kind only (`bar`, `bat`, `bin`, `bint`, `binte`, `binti`, `ibn` and the five native-script forms, 2026-10-29), so the lookup asks for the kind, not the first particle row. A word only a caller's `Lexicon` lists is a plain particle, `ben` included, which `ar.toml` excludes as a patronymic. + 4. **The kind is read at classify from the reading index** (SUPERSEDED by the 2026-11-02 #754, sub-phase 4i-b entry below: from the lexicon), as R8's suffix kind is: `particles` compiles every particle kind alike, so classify writes `vocab:patronymic` beside `particle` for a word the shipped files hold as a `patronymic` particle (`_vocab.is_patronymic_particle`, a memo over `_reading_index.holds_kind`), whatever the reading's default, and assemble drops the tag from the published token, a published tag making a name unequal to its 2.x pickle. Every patronymic particle has that kind only (`bar`, `bat`, `bin`, `bint`, `binte`, `binti`, `ibn` and the five native-script forms, 2026-10-29), so the lookup asks for the kind, not the first particle row. A word only a caller's `Lexicon` lists is a plain particle, `ben` included, which `ar.toml` excludes as a patronymic. 5. **The report's alternative moves with the reading.** A leading patronymic particle read as a given name names `patronymic as family` where it named `particle as family` (`bin Mohamad`), through `_vocab.particle_reading`, which both of assign's particle alternatives now ask, so `Token.reading` and `Ambiguity.alternative` agree, as R8's 4b made them for `Jr`. 6. **The ending rules read a word, not only a rotation.** O1 and O2 fire on exactly three and four name words; the patronymic is written in more shapes than that (`Ivanov, Ivan Ivanovich`, `Ivan Ivanovich Ivanov`, which the rotation leaves alone), so the East Slavic arm reads a middle word carrying O1's ending, and the Turkic arm a marker in a name field with the name word before it in its comma part, wherever the rule is on. The middle is the field a patronymic stands in after a given name, and a family word carrying the ending is a surname as often as not (`Roman Abramovich`). The patterns moved from `_post_rules` to `_vocab` as `has_patronymic_ending` and `is_patronymic_marker`, which the rotations and assemble both ask (mechanisms.md#ONE-PREDICATE-PER-QUESTION), the packs' hand copies pinned against them there. 7. **The view stays out of `as_dict()`, `--json`, `comparison_key()` and the repr**, as R8's do, and is a `render()` key and an assertable field of a rule's example lines. The snapshot records it where it is not empty (`tools/differential/snapshot.py`'s `PATRONYMIC_VIEW`, a `RECORDED_LATER` key), and `tools/differential/cldr_agreement.py` scores it on its own beside the merged `given_names`, as its PENDING_ROLES were written to once `ParsedName` had it. 8. **Review fixes (2026-10-29, the orchestrating review of #948).** (a) P6's attachment and O3's middle fold both write the folded mark, so under `middle_as_family` the first draft read the attached particle's field as the middle and dropped the exception (`Mohamad, Mahathir bin` and `Levi, Omer Bar` read no patronymic): the fix told the attachment by its shape, a folded family particle with nothing after it in its field behind a family word not folded. The second review found that shape wrong both ways: a particle P6 attached with another after it (`Bakar, Ali bin Abu`, a real naming pattern) opened a plain run and lost the family, reading 'bin Abu', under the default policy; and after a family comma, where the family is written first, a trailing particle P6 declines (no given word ahead of it) and O3 folds read as attached (`Rahman, Al bin` under `middle_as_family` gave 'bin Rahman'). P6 now marks what it attaches (item 3), and the field of a word P6 attached is the family under the fold, so a folded middle's run stops short of it (`Ali, Ahmad bin Ahmad Abu`). The third review found that the mark alone had dropped one condition of the shape test, a family word not folded for the attachment to name: behind an empty head (rules/C15) P6 attaches the particle as the whole family, which R2 makes name words without the fold and O3's fold gives a base, so `, Siti Aminah binti` under `middle_as_family` read patronymic 'binti'; an attached particle now names a parent only where the family holds a word P6 and O3 did not fold, and `test_middle_as_family_moves_no_patronymic_r2_does_not_move`, whose comparison skips a shape whose R2 marks differ, says that it cannot see these shapes and leaves them to its sibling's rows. (b) A Turkic marker a caller's `Lexicon` lists as a connective kept that reading while the word before it read `patronymic` (`Ali Ahmad Vali oglu Hasan` gave patronymic 'Vali'): the father is now read only once its marker reads as a name word, built once its marker is read, the token held until then. (c) The Turkic arm read a name field only, while the particle arm reads any: a marker in the maiden name now reads with the maiden word before it (`Leyla née Rasim qizi` gives 'Rasim qizi', `Ali née oglu` none). (d) The user docs said the ending rules fill the view under every order; the East Slavic arm reads only a middle, so it follows the order (`Ivan Ivanovich Ivanov` under FAMILY_FIRST reads given `Ivanovich` and none). (e) Each clause the review found no test for now has one that fails without it: the field test of the run's opener (`Ali bin, Ali`), the folded word left out of P6's family (`Ahmad Ahmad de bin` under FAMILY_FIRST), the suffix ending a run (`bin Ahmad Jr. Hasan, Ali`, whose family resumes behind the suffix) and the name-word test of the marker (the connective marker above); a clause requiring the attached particle to end the family was implied by the others and is gone. The first fix moved the run's field computation into a helper and cost a frame per token, seven on the reference row, caught by re-taking the frames below; it is inline again. 9. **Accepted after the second review.** R2 reads a name part made only of particles as name words, and O3's fold can give such a part a base, so its particle reads as one under the fold alone (`Ali, bin bin Abu` gives 'bin Abu' under `middle_as_family`, none without); `test_middle_as_family_moves_no_patronymic_r2_does_not_move` holds the two policies to the same patronymic words wherever R2's marks agree, 4,494 of 5,080 parses of its grid. Two Turkic patronymics, one in the name and one in the maiden name, are one view (`Leyla Vali qizi née Rasim qizi`), as two nasab particles are. - Measured 2026-10-29 on the 4c branch over `a9884bfb`. The committed snapshot's 6,570 readings of the base: 23 gain a `patronymic` key, every one a patronymic particle's run (the snapshot parses no pack), and one row's alternative moves (`bin Stromae Elizabeth Windsor Maria Teresa MP` under FAMILY_FIRST, `particle as family` to `patronymic as family`); no field moves; rules/R9's examples add 31 names, 6,601 readings in all. Recompute: compare each row of `snapshot.jsonl` with its base row, `patronymic` and `alternatives` set apart. The agreement records gain only `patronymic` entries, each a patronymic stated and not read, none read where the source states none or another: Wikidata 334 (its `patronymic-ru` 126, `patronymic-ri` 41, `patronymic-su` 62, `patronymic-is` 100 and `patronymic-my` 5), the native-script source 17 (`uk-ua` 14, `kk-kz` 3), conventions 6. Under the RU pack, Wikidata's `patronymic-ru` agrees 126 of 126, `patronymic-ri` 41 of 41 and `patronymic-su` 61 of 62, the miss `Оксана Сергіївна Баюл`'s Ukrainian ending; `patronymic-my` agrees 4 of 9 in the default, the five misses written without `bin`. Recompute: parse each `wikidata_fields.jsonl` row stating a patronymic with `cldr_agreement.parse_rendering`, with and without `RU`, and compare `got_roles(...)["patronymic"]` with `expected_roles(...)["patronymic"]` by `same`. Frames per parse (`sys.setprofile` call events, a warm parse), re-taken after the second review's fixes: the reference row `Dr. Juan Q. Xavier de la Vega III` 369 and `John Smith` 153 on both trees under the default policy; a patronymic particle adds two (`Mahathir bin Mohamad` 217 to 219), P6's attached shape nine (`Mohamad, Mahathir bin` 263 to 272, `Bakar, Ali bin Abu` 289 to 298, its family's tokens rebuilt; the look for a family word not folded is a loop, a generator there costing two more and one per word behind an empty head). The opt-in rules pay per word, which is frame question (1) of root AGENTS.md, a cost that scales with the input, recorded here and guarded by the `patronymic_rules` row of `tests/v2/test_benchmark.py`'s `_POLICY_SHAPES`: RU one per middle word read as a name (`Ivan Ivanovich Ivanov` 211 to 213, `John Paul George Ringo Smith` 270 to 273, the reference row 369 to 370), TR_AZ one per parse (`_turkic_patronymics`), one per word in a name field or the maiden name (`is_patronymic_marker`, which O2's rotation also asks once of the last word), and two per marker found (`comma_bucket`), the father's token built once (`John Smith` 153 to 156, the reference row 369 to 376, `Ali Ahmad Vali oglu` 243 to 251: one, four in the arm and one in O2, two). The ending and marker tests are `_vocab`'s shared predicates rather than an inline pre-check, the rules being opt-in and their per-word cost one frame. +- 2026-11-02 #754, sub-phase 4i-b — A PATRONYMIC PARTICLE IS ONE THE LEXICON HOLDS AS ONE. SUPERSEDES item 4 of the 4c entry above. The compile gives `PATRONYMIC_PARTICLES`, which classify reads off the lexicon (rules/R9 amended), so `add("ben", kind="patronymic")` makes `ben` one and a word holding another particle kind is a plain particle; on the default the set is the index's, no patronymic reading being written off by default (2026-11-02), so no default parse moves. diff --git a/docs/design/decisions/suffix-field-composition.md b/docs/design/decisions/suffix-field-composition.md index 78b7b8fa8..f3195bd45 100644 --- a/docs/design/decisions/suffix-field-composition.md +++ b/docs/design/decisions/suffix-field-composition.md @@ -21,7 +21,7 @@ Open: [#326](https://github.com/derek73/python-nameparser/issues/326) — whethe #326 quoted one. A date does not stop rot — #326 carried one too — so before relying on the split, check the set still looks like it: - uv run python -c "from nameparser import Parser; print(sorted(Parser().lexicon.suffix_words))" + uv run python -c "from nameparser import Parser; print(sorted(Parser().lexicon._sets.suffix_words))" What is durable here is the SHAPE of the finding — three kinds of thing, the honorifics the largest — not the three integers. A test asserting the counts is deliberately not the answer: that is the constant-content pattern, and it would fail on every legitimate vocabulary addition. diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index d4f4ab9ea..d828102c6 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -21,6 +21,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - A caller configures vocabulary by reading, `Lexicon` deriving the sets and gates the build derives, and the compiled sets become internal in 3.0 — 2026-10-28 #754, the `Lexicon` surface (Derek). - The configure-by-reading API: `add(*words, kind, **properties)` and `remove(*words, kind=None)`, the views reading a word's kind from the lexicon, `readings_of` returning a public `LexiconReading` and a repr counting readings by kind, a five-part state (base, ordered edit log, choice, lists, masks) compared structurally, a clash a later choice exposes raised at `parser_for` naming the choice, the choice recorded on the `Lexicon` with chained choices unioning, the default's base lazy, a 2.x pickle translated onto the nearer named base and a per-word log, old spellings refused with a per-word translation, built in 4i-a to 4i-d with no default parse moving — 2026-11-02 #754, the `Lexicon` surface (Derek). - Configured by reading, an `add` of a title reading that a `given` reading keeps a title only inside a phrase is to warn, naming `remove(w, kind="given")`, save one that only updates a held title reading's properties — 2026-11-01 #754, the title warning (Derek). +- A 2.x call or pickle that removed a word from its base set but left it in a marker subset reads as though the marker went with the word, for every marker set, the two readings 2.x gave such an orphan accepted as moving — 2026-11-02 #754, a marker left behind. - A `Locale` may carry CLDR-shaped render patterns, display and sorting, a later addition whose shape is agreed; how a call reaches one is open — 2026-10-28 #754, render patterns. - 3.0's output model keeps the seven roles and adds every new field as a view, read from `Token.reading`, the reading each token took, or for `surname2` from rules/O6's tag: `generation` and `credentials` partitioning `suffix`, `patronymic`, and `surname2` where rules/O6 read two family names; `middle` and `surnames` keep their meaning, a compound given name being a rule a caller or pack turns on; the default `str()` renders a name in the order it was read in with its written separators in a normalized form, through `render(spec: str | None = None)`, `None` being that default; a language file's `[rules]` compile into its `Locale`'s policy patch — 2026-10-28 #754 (Derek). - `surname2` is read from O6's mark as `Token.reading` publishes it, `Reading.second_surname`, a field not compared, and not from a published tag, which would make an O6 parse unequal to its 2.x pickle (decisions/R10) — 2026-10-29 #754, sub-phase 4d. @@ -493,3 +494,14 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **The guard.** tests/v2/test_layering.py walks every library module but `_lexicon`, and every file under tools/, and refuses a vocabulary field read off a lexicon, one test per `_VOCAB_FIELDS` entry: in the library only a read through `._sets`, or through a name `sets` that every binding in the module assigns from a `_sets`, passes, `TailRead`'s `read.titles` named as no lexicon, and a `getattr` of a lexicon, or of a field named as a constant off anything but a `_sets`, fails; in the tools, whose counters reuse the field names, a read off `lexicon`, `lex`, `.lexicon`, `Lexicon.default()` or a name any binding gives one of those (an assignment, a walrus, a `Lexicon` annotation) fails, the AGENTS.md recipe's `L = Parser().lexicon` among them. Not followed, and stated in the check's docstring: a lexicon handed through a container, a call's return or an attribute other than `.lexicon`, and aliases per scope rather than per module. Its negative controls: a synthetic read in each spelling, library and tool, per field, must be refused, each binding form of `sets` and of an alias has a row, and `_UNGUARDED_READS` records what the check refuses on today's sources with `._sets` taken out again (114 in the library: 106 attribute loads, 5 roster `getattr`s, 3 of them through the `sets` the undoing makes a lexicon, and 3 `sets =` bindings; 16 in the tools, of 40 files). Mutated on the built tree, one `lex._sets.titles_ambiguous` in `_classify` written back as `lex.titles_ambiguous` failed exactly `test_no_reader_takes_a_vocabulary_field_off_a_lexicon[titles_ambiguous]`. - **How 4i-b plugs in.** 4i-b changes how `_sets` is filled and nothing that reads it: from the lexicon's readings, compiled by `compile_readings`, rather than from its public fields; the default's from nameparser/config/_compiled.py, as `_default_lexicon` reads it today, so `parse("John Smith")` still loads no index (the base stays lazy, the 2026-11-02 entry's item 4). The derived markers the views will read, a listed suffix word's kind and the patronymic particles (the RFC's item 1), join `_LexiconSets` as attributes, and `_vocab.suffix_kind` and classify's particle kind read them there in place of the index. With `dataclasses.fields(Lexicon)` changing in 4i-b anyway, `_sets` may become a private field then, the base class going. `_parser._apply_choice` reads `_sets` but still writes the public fields through `dataclasses.replace`; 4i-b gives it the readings to write until 4i-c retires it. What 4i-b also breaks, outside the readers the guard holds: the tests' 157 public-field reads (the 2026-11-01 count), and AGENTS.md's recipe for the titles in no suffix set, `L.titles - L.suffix_acronyms - L.suffix_words` on `L = Parser().lexicon`, which reads public fields and is rewritten then over `readings_of` or `_sets`. - The default does not move: tests/v2/test_snapshot.py passes unchanged and tests/v2/test_cldr_agreement.py with it; the suite passes on Python 3.11 and 3.15 without the `ja` extra. +- **2026-11-02 #754, sub-phase 4i-b — A `Lexicon` HOLDS READINGS AND IS CONFIGURED BY THEM; ITS SET FIELDS ARE INTERNAL, EVERY 2.x SPELLING REFUSED WITH ITS TRANSLATION; THE VIEWS READ THE LEXICON'S KINDS; A 2.x PICKLE LOADS AS A BASE AND A LOG; NO DEFAULT PARSE MOVES.** Carries out item 7's second PR of the 2026-11-02 #754, the `Lexicon` surface entry, items 1, 2, 3 (but the choice), 5 and 6 of it; the choice recorded on the `Lexicon` is 4i-c's. + - **What.** `Lexicon` is a frozen, slotted dataclass built with `init=False` and a hand-written keyword-only `__init__`, its fields `surnames`, `given_names`, `capitalization_exceptions`, `_base` (`"default"`, `"empty"` or a snapshot of index rows) and `_edits` (a tuple of `_lexicon._Edit`, each an `add` or `remove` stored canonical: words normalized, properties sorted pairs, a property given its unwritten value held as None), with `_cap_map` and `_sets` private fields built at construction. `add(*words, kind, **properties)`, `remove(*words, kind=None)` (a kind or a category), `readings_of(word)` returning the new public `LexiconReading(text, kind, category, properties, languages)`, a repr counting readings by kind (`Lexicon(default + given: -1)`), `|` concatenating the logs, the left's first, and refusing two different non-empty bases. A configured lexicon compiles its pool (the base's rows, the log applied in order) with `compile_readings`, kept per base and log; `Lexicon.default()` reads `_compiled.py` and loads no index, and neither does `parse("John Smith")` (`test_the_default_never_loads_the_reading_index`). The compile gives three internal sets the public ones did not record, `GENERATION_WORDS`, `CREDENTIAL_WORDS` and `PATRONYMIC_PARTICLES`, written to `_compiled.py` and re-exported by no config module (`CONSTANTS` takes `None` for one), which classify reads in place of the index lookups 4b and 4c made: rules/R8 and rules/R9 amended to read "the lexicon's". The title warning, the build's checks 4, 8 and 9 at `add`, and the three refusals are as the 2026-11-01 and 2026-11-02 entries decide; the refusals and the 2.x pickle load share one translation from sets to readings, `_lexicon._edits_toward`. + - **Choices made inside the decided design, each a reading of it rather than a new decision.** (1) `add` and `remove` refuse `surnames` and `given_names` too, naming `lexicon | Lexicon(surnames={...})`: item 3 says the two lists change "through `|` with a fragment ... or `dataclasses.replace`", and the RFC rewrites customize.rst's `add(surnames={"김민"})` as that `|`, though its migrate table calls the lists "unchanged". (2) An `add` of a kind the word holds only off by default turns that reading on, in every file holding it (`add("st", kind="generation")` reads `st` as cs, hr, sk and sl's generation): a caller's reading is on, which is item 1's reason for refusing `default`. (3) `readings_of` leaves out `default` and `cased`, the first being on for every reading it returns and the second a mask the lexicon keeps in `capitalization_exceptions`; `LexiconReading.properties` is a tuple of sorted pairs, so the type hashes. (4) The repr counts a reading whose properties changed as one off and one on (`conjunction: +1-1`), and a log that changes no reading as `edits: N`. (5) `_parser._apply_choice` keeps 3b's delta and writes it as readings: the edits that take each word the choice changes to its sets under the choice, through `_edits_toward` with the choice's own readings as the kinds to take, appended to the log, until 4i-c retires it. For every single file tag a choice may name, the 78 tags holding a reading chosen plainly and the 83 files chosen strictly (2026-11-02), on the default, the empty lexicon and a configured one, the compiled sets equal the delta's in every field, the kind sets included: tests/v2/test_parser.py's `test_a_choice_written_as_readings_gives_the_deltas_sets`, one case per tag, its lexicons `_CHOICE_BASES`. + - **What 2.x could state that readings cannot,** each found translating the tests, and each a configuration, not a default, that no longer has a spelling: a word that is a title and a given name written short at once (`add(titles={"chas"})`, which kept `chas` in `given_name_abbreviations`: the `given` reading that makes the abbreviation keeps a title reading one only inside a phrase, so a title everywhere is the `given` reading removed); a possibly-given particle beside an unambiguous credential (`particles_ambiguous` and `suffix_acronyms` on one word: the `given` reading gates the credential too), whose ordered arms in post_rules' P6 report stay and are held by a test building those sets directly; `may_be_initial` on a caseless letter (`و`), which the build refuses as `add` now does; and a three-word title phrase. A 2.x pickle holding any of them is refused, naming the fields. An orphan marker, by contrast, a marker entry its base set does not hold (`conjunctions_ambiguous` on a word that is no conjunction; `remove(titles={"air"})` leaving `air` a modifier), is dropped (`_lexicon._inert_dropped`), so the spelling translates and the pickle loads as the removal, the marker going with its word (the 2026-11-02 #754, a marker left behind entry below; 2.x did read such an orphan in two places, which that entry accepts moving) (found by the orchestrating review, 2026-11-02: the first draft offered no call for 270 of 786 default titles and 46 of 189 suffix words, and refused those pickles). + - **Measured.** Frames, Python 3.11.17, `uv run python tools/perf/call_count.py --against 3804b1a1`: the reference row 311.82 on the base and 311.48 here, the memo of the index lookups classify no longer calls; `_CALL_BASELINE` is unchanged and the serial benchmark passes. A configured lexicon's compile, `Lexicon.default().add(...)` with the index loaded and the compile cache cleared, 2.09 ms (mean of 20). A pickle of `Lexicon.default()`, protocol 4, 39,658 bytes, its base the index's 2,261 rows. The suite's configuring calls moved to readings, every pinned parse kept but one, a reading's kind: `Lexicon.default().add("st", kind="suffix")` reads `St` as a post-nominal, where 4b's index lookup read the cs file's generation written off (rules/R8's amendment). + - **Found by review, 2026-11-02.** `add("bisneto", kind="agnomen")` alone was accepted and read `Pedro Alves Bisneto` credentials `Bisneto`, an agnomen without the name reading the build's check 11 requires of a file, so `add` now refuses it too, naming `.add(w, kind="family")` (the design-docs review). And `parser_for(locales.CS, base=Parser(lexicon=Lexicon.default().add("st", kind="credential")))` built a parser that dropped the caller's credential in silence, where 3b's delta had raised its suffix set guard: `cs` turns on `st`'s generation, and a word holds one suffix kind. `_apply_choice` now compiles its edits and refuses a result short of the delta's sets, naming the words, wrapped with the languages chosen (rules/D2) (the code review). 4i-c's compile at `parser_for` replaces both with the decided raise naming the edit. + - The default does not move: tests/v2/test_snapshot.py passes unchanged and tests/v2/test_cldr_agreement.py with it. +- **2026-11-02 #754, a marker left behind — A 2.x CALL OR PICKLE THAT REMOVED A WORD FROM ITS BASE SET BUT LEFT IT IN A MARKER SUBSET READS AS THOUGH THE MARKER WENT WITH THE WORD, FOR EVERY MARKER SET; THE TWO READINGS 2.x GAVE SUCH AN ORPHAN ARE ACCEPTED AS MOVING.** Decided in chat on 2026-11-02, on the review of 4i-b's fix commit. In 3.0 a marker is a property of its reading (`addresses="given"` on the title, `may_be_initial` on the conjunction, the `modifier` and `style` kinds, a name reading beside a suffix word) or a kind, so taking the reading out takes the marker: `remove(titles={"sir"})` removes `sir` as a title, and being a given-name title goes with it. 4i-b's `_lexicon._inert_dropped` drops each marker entry its base sets do not hold (`_MARKER_READ_WITH`), on both sides of every compare, so the refusal's translation of such a call and the load of such a pickle agree with it. + - **Accepted: what 2.x read of an orphan, measured on master `3804b1a1` against this tree by the orchestrating review (2026-11-02).** (1) An orphan in `given_name_titles` still made rules/H2's period-shape title address the given name: with `remove(titles={"sir"})`, `Sir. Smith`, `Sir. Elton` and `Smith, Sir.` read given `Smith` and `Elton` on master and family on the tree; `add(given_name_titles={"zork"})` and `Zork. Smith` the same. (2) Any orphan counted as a listed word in the capitals shape's "unlisted" test (`_vocab.in_any_wordlist`): with `remove(titles={"air"})`, `John Smith, AIR` reads given `AIR`, family `John Smith` on master and suffix `AIR` on the tree, unlisted as any word a caller removed. tests/v2/test_lexicon.py's `_LEFT_BEHIND` pins the tree's readings, under the translated call and the loaded pickle alike; recompute by building the 2.x state with the word taken out of its base set alone, as that test's `_state_2x` does, and parsing each text on both trees. + - Weighed against: keeping the orphan's two readings, which would need a reading that marks without its base, the very state the per-word model was built not to express. + diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 2dacc38e1..8c6f2fe98 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -276,7 +276,7 @@ This catalog converts discovery into a one-time cost — it does not remove disc **Contract statement.** The warner walks the stack outward until the first frame outside the library's own modules and warns there, instead of counting frames. -**Lives in.** nameparser/_lexicon.py (_warn_dead_entry), and the related but distinct per-read-location choice is in the #293/#354 bridge (decisions/3-0-reevaluations). First measured instance of the hazard: the delegated bytes-deprecation warnings shipped with fixed stacklevel=2 and needed the follow-up fix (0044073). +**Lives in.** nameparser/_lexicon.py (_warn_at_caller, `_warn_dead_entry` until 3.0's 4i-b), and the related but distinct per-read-location choice is in the #293/#354 bridge (decisions/3-0-reevaluations). First measured instance of the hazard: the delegated bytes-deprecation warnings shipped with fixed stacklevel=2 and needed the follow-up fix (0044073). **Reach for it when.** Adding any warning reachable through more than one public entry point. @@ -386,7 +386,7 @@ print(len(MAIDEN_MARKERS), 'members ->', len(cls), 'structural classes')" **Contract statement.** Where a fix has more than one plausible shape, every candidate is measured before one is chosen — simulated in-process against the real pipeline rather than implemented — and the choice cites the resulting table. -**How it works.** The seams already exist, so simulating is cheap: `Parser(lexicon=dataclasses.replace(Lexicon.default(), ...))` for a vocabulary question, a pytest plugin recompiling one stage for a code question, a monkeypatched predicate for a guard. What the table buys is the discovery that the candidates differ in ways argument does not reveal — #400's obvious fix, adding `abd` to the wordlist, provably changed nothing on the common three-word shape, and the two vocabulary alternatives each silently lost a credential spelling; the winner was a fifth option the wordlist framing had hidden. The counterexample is the same PR's ledger `fields`, widened to include `suffix` on one reviewer's finding without pricing what else the widening claimed: it took `{family, suffix}` from a neighbouring rule and two further shapes from UNEXPLAINED, diffs in which the rule's own subject does not move, and was reverted. One table would have shown that before the commit rather than after. +**How it works.** The seams already exist, so simulating is cheap: `Parser(lexicon=Lexicon.default().add(...))` or `.remove(w, kind=...)` for a vocabulary question (`dataclasses.replace` with a set field before 3.0's 4i-b), a pytest plugin recompiling one stage for a code question, a monkeypatched predicate for a guard. What the table buys is the discovery that the candidates differ in ways argument does not reveal — #400's obvious fix, adding `abd` to the wordlist, provably changed nothing on the common three-word shape, and the two vocabulary alternatives each silently lost a credential spelling; the winner was a fifth option the wordlist framing had hidden. The counterexample is the same PR's ledger `fields`, widened to include `suffix` on one reviewer's finding without pricing what else the widening claimed: it took `{family, suffix}` from a neighbouring rule and two further shapes from UNEXPLAINED, diffs in which the rule's own subject does not move, and was reverted. One table would have shown that before the commit rather than after. **Reach for it when.** A review names a defect and the fix looks obvious — that is when the option space is least examined, and the obvious fix is the candidate whose cost nobody has priced. Not a mechanism with a home module; the technique IS the seams above, which is why it sits here rather than in the catalog. @@ -445,7 +445,7 @@ A second axis cuts across the jobs: whether the names are real or invented, labe - Release notes describe intent; TAGS are ground truth. Before trusting a changelog cross-reference about what shipped where, run `git diff -- ` — the "1.2.1 pickle shim" claim recurred twice from a changelog line describing a thing that never shipped. - Never pipe a check's output when its exit code matters. Under zsh, `snapshot.py | tail` makes `$?` tail's status. Redirect to a file and read the file. - Mutation-test a new guard before believing it, and mutate the thing the guard watches — a survivor usually means the fixture satisfies the invariant for free. A guard over a HAND COPY of a pipeline set has two such holes, and #744's review found both in one test of tools/vocab/build.py's check 2, which copies the tokenizer's sets because the tool cannot import the package it writes. Pinning a copy to its source sees only the classes someone copied: the first test pinned `DIVIDERS` alone, and emoji, which the tokenizer strips under `Policy.strip_emoji`, had no copy and no check, so `dr★` compiled and the test passed. Its replacement swept every codepoint `_tokenize._ignorable` drops but accepted any refusal short of a TOML error, so the space in `dr x` passed because the phrase-title check refused it, and a class check 2 never learned would pass too wherever a later check happened to refuse its test text. Drive the sweep from the pipeline's own predicate, match the refusal by the checked rule's own messages, and record the control (hiding the emoji check behind its pinned constant fails the sweep: `test_every_character_the_tokenizer_ignores_is_refused` in tests/v2/test_vocab.py, which now leaves the space out, a phrase's word separator). The predicate reaches only what it decides, though: the interpunct between East Asian characters (rules/T3) is divided by context in `_tokenize_region`, never by `_ignorable`, so hiding that check passes the sweep (measured in #746's review, 2026-10-19). What holds it is its `MUTATIONS` row (`text-with-an-east-asian-interpunct`, input `先·生`); the `EAST_ASIAN_RANGES` and `INTERPUNCT` pins hold only the copies, and stay green with the check gone. Name such a class where the guard stands, and the test that holds it, rather than let the sweep imply it. -- A claim about what a vocabulary entry DOES is a claim about the parse WITHOUT it, so take that parse (`Parser(lexicon=Lexicon.default().remove(titles={...}))`) and compare: the parse with the entry present shows only that the output is consistent with it. #773's first note on `wm` said that removing it would move `Wm. Smith`, read off the parse with the word listed; with the word removed, `Wm. Smith` read exactly as before, rules/H2's leading-abbreviation title then claiming `Wm.` either way. The corrected note then said only the bare `Wm Smith` moves, and that was wrong too (#789, 2026-10-24): it took the off-switch parse at the head of the name only, and a trailing `Wm.` (rules/H5) and a `Wm.` before a family comma (rules/H6), which read vocabulary rather than shape, moved as well. A shape rule shadows an entry only in the spelling and the slot it reads, so take the off-switch parse for each spelling, bare and period-marked, in each position a rule reads the word in. +- A claim about what a vocabulary entry DOES is a claim about the parse WITHOUT it, so take that parse (`Parser(lexicon=Lexicon.default().remove(w, kind=...))`) and compare: the parse with the entry present shows only that the output is consistent with it. #773's first note on `wm` said that removing it would move `Wm. Smith`, read off the parse with the word listed; with the word removed, `Wm. Smith` read exactly as before, rules/H2's leading-abbreviation title then claiming `Wm.` either way. The corrected note then said only the bare `Wm Smith` moves, and that was wrong too (#789, 2026-10-24): it took the off-switch parse at the head of the name only, and a trailing `Wm.` (rules/H5) and a `Wm.` before a family comma (rules/H6), which read vocabulary rather than shape, moved as well. A shape rule shadows an entry only in the spelling and the slot it reads, so take the off-switch parse for each spelling, bare and period-marked, in each position a rule reads the word in. - Verify the restore, not just the mutation: diff against the pre-mutation copy; do not trust the harness's own restore report. - Run all the gates, not the ones you remember: ruff runs before mypy and pytest in CI, and each has caught what the others passed. - Purge __pycache__ between same-length source mutations; stale bytecode makes a changed file measure as unchanged. diff --git a/docs/design/rules/R8-generation-and-credentials-views.md b/docs/design/rules/R8-generation-and-credentials-views.md index 1a2899bdd..10fb3fe8f 100644 --- a/docs/design/rules/R8-generation-and-credentials-views.md +++ b/docs/design/rules/R8-generation-and-credentials-views.md @@ -56,11 +56,11 @@ ``` "Smith, Jr., John" → credentials="John" ``` -- The kind of a word listed as a suffix word is the shipped vocabulary's, read whatever the reading's default, so a word a caller's own `Lexicon` lists in `suffix_words` alone is in the credentials, a generation word included, and so is a generation word a caller lists as an acronym; one the caller also lists in `suffix_words_ambiguous` or `agnomens` reads as a generation or an agnomen by that marker, save a word the shipped vocabulary lists as a credential, which listed in `suffix_words_ambiguous` reads as a credential, as the gated `KC` does (S11), and listed in `agnomens` reads as an agnomen. +- The kind of a word listed as a suffix word is the lexicon's, the kind of the suffix reading its `Lexicon` holds, so a word a caller adds as a generation (`add("secundus", kind="generation")`) reads as one, and a word added as a credential, as a plain `suffix`, or as an honorific, status or courtesy word is in the credentials, a credential written as a word (`acronym=False`) as the gated `KC` is (S11); an agnomen reads as an agnomen. On the default lexicon the kind is the shipped vocabulary's, which compiles it, and a lexicon whose readings a caller removed reads the kind it holds, not the files'. ``` "John Smith KC" → credentials="KC" ``` --- **Signal:** none in the name: the reading each suffix word took -**history:** decisions/R8 · **implemented:** nameparser/_types.py, nameparser/_pipeline/_vocab.py, nameparser/_pipeline/_classify.py +**history:** decisions/R8 · **implemented:** nameparser/_types.py, nameparser/_pipeline/_classify.py diff --git a/docs/design/rules/R9-patronymic-view.md b/docs/design/rules/R9-patronymic-view.md index b87711559..e8bbc50b5 100644 --- a/docs/design/rules/R9-patronymic-view.md +++ b/docs/design/rules/R9-patronymic-view.md @@ -4,7 +4,7 @@ **Then:** the words read as a patronymic, in whichever field the parse put them, make the patronymic view, every field keeping them, and the view joins them with spaces in written order, the family's words in the order the family writes them. -- **A patronymic particle**: a particle the shipped vocabulary holds as a patronymic, read as a particle, with the words after it in its field, the parent's name. +- **A patronymic particle**: a particle the lexicon holds as a patronymic, read as a particle, with the words after it in its field, the parent's name. - **The East Slavic rule** (`PatronymicRule.EAST_SLAVIC`, which RU turns on): a middle word carrying O1's patronymic ending. - **The Turkic rule** (`PatronymicRule.TURKIC`, which TR_AZ turns on): a standalone patronymic marker, O2's, with the name word before it in its comma part, the father's name, a marker in the maiden name taking the maiden name's word before it. @@ -94,9 +94,9 @@ "Siti binti Ahmad binti Hasan" → patronymic="binti Ahmad binti Hasan" "Leyla Vali qizi née Rasim qizi" [tr_az] → patronymic="Vali qizi Rasim qizi" ``` -- A patronymic particle is the shipped vocabulary's, read whatever the reading's default, as R8 reads a suffix word's kind, so a word only a caller's `Lexicon` lists as a particle (`ben`, which the vocabulary excludes as a patronymic) is a plain particle. +- A patronymic particle is one whose `Lexicon` holds a `patronymic` reading of it, as R8 reads a suffix word's kind: on the default the shipped vocabulary's, so `ben`, which the vocabulary excludes as a patronymic, is no particle there, a caller's `add("ben", kind="patronymic")` makes it one, and a word a caller adds as another particle kind is a plain particle. - A word the particle's run or the Turkic marker takes reads `patronymic` by position, its own reading (a name word, a bound given name, a particle) not kept in `Token.reading`. --- **Signal:** neighbor + shape: a listed patronymic particle before the parent's name or an opt-in marker after it, and an opt-in patronymic ending on a middle word -**history:** decisions/R9 · **implemented:** nameparser/_types.py, nameparser/_pipeline/_assemble.py, nameparser/_pipeline/_classify.py, nameparser/_pipeline/_vocab.py +**history:** decisions/R9 · **implemented:** nameparser/_types.py, nameparser/_pipeline/_assemble.py, nameparser/_pipeline/_classify.py diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index cde5af8ac..6a56d711d 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -463,7 +463,7 @@ How a call reaches a pack's pattern is open, and each answer costs something: `P ### The `Lexicon` surface: configuring by reading -Status: **decided 2026-11-02 as recommended below (decisions/vocabulary-model's 2026-11-02 #754, the `Lexicon` surface entry), item 1's title warning on 2026-11-01 (decisions/vocabulary-model's 2026-11-01 #754, the title warning entry); 4i-a is built (decisions/vocabulary-model's 2026-11-02 #754, sub-phase 4i-a entry), the rest is not.** What is decided is the direction (2026-10-28): a caller configures vocabulary by reading, `Lexicon` derives the sets and gates the build derives, and the compiled sets become internal (on 2026-11-01, 20 of `Lexicon`'s 23 public fields: the 19 of `_lexicon._COMPILED_FIELDS` and the derived `particles_ambiguous`; item 3 keeps the other three), with no 2.x warning (2026-10-29). This section proposes the API that direction leaves open, each question with the options weighed and one recommendation; decisions/vocabulary-model's 2026-11-01 #754, the `Lexicon` surface proposed entry records the proposal and its measurements, and decisions/vocabulary-model's 2026-11-02 #754, the `Lexicon` surface entry the decision, each recommendation accepted and the alternatives below declined. Where this section says "proposed" or "recommended", read "decided"; the wording is kept as the record of what was weighed. Every parse below was run on master `a3b1f3e1` (Python 3.11.17) through a throwaway prototype that compiles a lexicon's readings with the shipped `compile_readings` over `choose(index(), chosen)`, the whole index with its readings off by default, as item 4 has the default hold them, the caller's edits applied after the choice, and folds the result into today's `Lexicon`, so the real parser reads it; the prototype is not committed. +Status: **decided 2026-11-02 as recommended below (decisions/vocabulary-model's 2026-11-02 #754, the `Lexicon` surface entry), item 1's title warning on 2026-11-01 (decisions/vocabulary-model's 2026-11-01 #754, the title warning entry); 4i-a and 4i-b are built (decisions/vocabulary-model's 2026-11-02 #754, sub-phase 4i-a and sub-phase 4i-b entries), 4i-c and 4i-d are not.** What is decided is the direction (2026-10-28): a caller configures vocabulary by reading, `Lexicon` derives the sets and gates the build derives, and the compiled sets become internal (on 2026-11-01, 20 of `Lexicon`'s 23 public fields: the 19 of `_lexicon._COMPILED_FIELDS` and the derived `particles_ambiguous`; item 3 keeps the other three), with no 2.x warning (2026-10-29). This section proposes the API that direction leaves open, each question with the options weighed and one recommendation; decisions/vocabulary-model's 2026-11-01 #754, the `Lexicon` surface proposed entry records the proposal and its measurements, and decisions/vocabulary-model's 2026-11-02 #754, the `Lexicon` surface entry the decision, each recommendation accepted and the alternatives below declined. Where this section says "proposed" or "recommended", read "decided"; the wording is kept as the record of what was weighed. Every parse below was run on master `a3b1f3e1` (Python 3.11.17) through a throwaway prototype that compiles a lexicon's readings with the shipped `compile_readings` over `choose(index(), chosen)`, the whole index with its readings off by default, as item 4 has the default hold them, the caller's edits applied after the choice, and folds the result into today's `Lexicon`, so the real parser reads it; the prototype is not committed. **1. Configuring: `add` and `remove` take words and a reading kind.** Proposed: @@ -560,7 +560,7 @@ All three were checked on a frozen, slotted throwaway dataclass on 2026-11-01. T | Sub-phase | What | Needs | |---|---|---| | 4i-a | The pipeline reads the compiled sets through one internal object, not `Lexicon`'s public fields; the three tools move to it or to the index. No public change; a frozenset test stays a C-level attribute read, so the frame count holds (`_CALL_BASELINE`). Landed 2026-11-02 (decisions/vocabulary-model's 2026-11-02 #754, sub-phase 4i-a entry) | — | -| 4i-b | The break: `Lexicon` holds readings, `add`/`remove` by kind, `readings_of` and `LexiconReading`, the narrowed constructor, the three refusals with their translation, the per-kind repr, the warning on a title reading made phrase-only by a `given` reading, the views reading the lexicon's kinds with rules/R8 and rules/R9 amended, the lazy base, the 2.x pickle translation; the tests and docs moved, `docs/migrate.rst`'s table, a release-log bullet | 4i-a | +| 4i-b | The break: `Lexicon` holds readings, `add`/`remove` by kind, `readings_of` and `LexiconReading`, the narrowed constructor, the three refusals with their translation, the per-kind repr, the warning on a title reading made phrase-only by a `given` reading, the views reading the lexicon's kinds with rules/R8 and rules/R9 amended, the lazy base, the 2.x pickle translation; the tests and docs moved, `docs/migrate.rst`'s table, a release-log bullet. Landed 2026-11-02 (decisions/vocabulary-model's 2026-11-02 #754, sub-phase 4i-b entry) | 4i-a | | 4i-c | The choice recorded on the `Lexicon`: chained choices union, a caller's readings applied after the choice, the strict mismatch raised, `|` refusing two choices, a pack's fragment joined under the caller's lexicon, the compile and its refusals at `parser_for`, the underscore choice parameters validated, `_apply_choice` retired; the chained-choices entry and 3j's dropped-listing clause superseded | 4i-b | | 4i-d | Optional: `Reading.languages` follows the choice | 4i-c | @@ -578,7 +578,7 @@ Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's ar | 4f | `[rules]` compiled into a `Locale`'s policy patch; RU and TR_AZ rebuilt from their files. Landed (decisions/vocabulary-model's 2026-10-29 sub-phase 4f entry) | 3b | | 4g | the compound given name, a rule kind (#704): built 2026-10-30, `Policy.two_given_names` (rules/O7) | 4f | | 4h | the shape inventory per order (#469), starting with the title between a family and a given name under `FAMILY_FIRST`. Landed as an inventory, no reading moved: the arrangements each order is written in, listed with what each reads (decisions/vocabulary-model's 2026-10-30 #754, sub-phase 4h entry); the title between, and bare after, under a family-first order stays a name word, #938 declined on a survey of real names, which write the title in front (decisions/H5's 2026-10-30 #938 entry) | — | -| 4i | the `Lexicon` surface: configuring by reading, the compiled sets internal, and the choice recorded on the `Lexicon`. Decided 2026-11-02, to be built in four PRs, 4i-a landed (4i-a to 4i-d, "The `Lexicon` surface: configuring by reading" above) | 3b, 3j | +| 4i | the `Lexicon` surface: configuring by reading, the compiled sets internal, and the choice recorded on the `Lexicon`. Decided 2026-11-02, to be built in four PRs, 4i-a and 4i-b landed (4i-a to 4i-d, "The `Lexicon` surface: configuring by reading" above) | 3b, 3j | **Interleaved with phase 3** (its sub-phases 3a to 3j, above): 4a can start beside 3a, its `languages` waiting for 3a's index; 4d needs nothing of phase 3, and 4b and 4c need 3a's index only for a listed word's kind, a suffix word's and a particle's; they unblock its packs, `cs`'s generations (3f) and `pt-BR`'s agnomens (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; 4f lands with or after 3b, since both decide what a shipped `Locale` holds; an `is` pack, which no phase 3 sub-phase lists, needs 3b, 4c and 4f. 3j, file-strict, needs 3b alone (3i, which name-strict would have read, is set aside), and 4e and 4h stand apart from both. So the order is 3a and 4a, then 4b to 4d with 3b and 3c (4b after 3a), then 4f, then the packs with 4g, with 4e at any point and 3j at any point after 3b. diff --git a/docs/locales.rst b/docs/locales.rst index 8d12167f2..7c7342c5e 100644 --- a/docs/locales.rst +++ b/docs/locales.rst @@ -51,18 +51,14 @@ reasoning behind the default doesn't apply to you — add the entry: >>> from nameparser import Lexicon, Parser >>> sayyid_lex = Lexicon.default().add( - ... titles={"سيد"}, given_name_titles={"سيد"}) + ... "سيد", kind="title", addresses="given") >>> sayyid = Parser(lexicon=sayyid_lex).parse("سيد محمد") >>> sayyid.title, sayyid.given ('سيد', 'محمد') -Both fields, because ``given_name_titles`` is a marker over ``titles`` -rather than a separate vocabulary: ``titles`` makes the word a title at -all, and listing it in ``given_name_titles`` says the honorific -precedes the *given* name — as Arabic ones do — so the word after it -isn't read as a family name. Listing it in ``given_name_titles`` alone -does nothing, and says nothing: the word is never read as a title, so -the entry is never consulted. +One reading, a title, with the property ``addresses="given"``, which +says the honorific precedes the *given* name — as Arabic ones do — so +the word after it isn't read as a family name. East Asian names need no pack ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -234,9 +230,11 @@ credential: .. doctest:: >>> from nameparser import Parser - >>> mine = Lexicon(particles=frozenset({"do", "van"}), - ... surnames=Lexicon.default().surnames) - >>> sorted(parser_for(en, base=Parser(lexicon=mine)).lexicon.particles) + >>> mine = Lexicon(surnames=Lexicon.default().surnames).add( + ... "do", "van", kind="preposition") + >>> chosen = parser_for(en, base=Parser(lexicon=mine)).lexicon + >>> [w for w in ("do", "van") + ... if any(r.category == "particle" for r in chosen.readings_of(w))] ['van'] .. _strict-choice: @@ -836,7 +834,7 @@ see :meth:`Policy.patched() `. .. doctest:: >>> from nameparser import Lexicon, Locale, PolicyPatch, parser_for - >>> kapitan_lex = Lexicon.empty().add(titles={"kapitan"}) + >>> kapitan_lex = Lexicon.empty().add("kapitan", kind="military") >>> corp_pack = Locale(code="mycorp", lexicon=kapitan_lex, ... policy=PolicyPatch(middle_as_family=True)) >>> kapitan = parser_for(corp_pack).parse("Kapitan Anna Maria Schmidt") @@ -856,17 +854,17 @@ The lexicon fragment is validated on its own The ``policy`` half is a patch, but the ``lexicon`` half is not. A pack's :class:`~nameparser.Lexicon` is a complete value in its own -right and is validated on its own, before it is unioned onto the base -— so a fragment listing a word in ``particles_ambiguous``, -``suffix_acronyms_ambiguous``, ``honorific_tails`` or -``suffix_words_ambiguous`` must also list it in the set that field marks -(``particles``, ``suffix_acronyms``, ``suffix_words``). A fragment holding ``particles_ambiguous={"van"}`` -without ``particles={"van"}`` raises ``ValueError`` before any union -happens. ``given_name_titles``, ``conjunctions_ambiguous``, -``after_given_titles`` and ``agnomens`` are not checked this way, and can mark a word only the base holds. ``zh`` is the -shipped worked example: its ``Lexicon(surnames=...)`` has to satisfy -every ``Lexicon`` rule standing alone, before anything unions it onto -the base. +right: its readings are checked as they are added (a phrase, a +``may_be_initial`` on anything but a single cased letter, an agnomen +with no name reading), and again when ``|`` joins them onto the base, +where a reading that clashes with one the base holds raises with the +pack's code. A fragment holding ``Lexicon.empty().add("dc", +kind="generation")`` fails there, ``dc`` being a credential in the +default and a word holding one suffix kind. A reading that marks +another, as a ``given`` reading makes a particle possibly given, needs +nothing else in the fragment: ``Lexicon.empty().add("mc", kind="given")`` +marks the base's particle ``mc``. ``zh`` is the shipped worked example: +its ``Lexicon(surnames=...)`` holds a list and no reading. Contributing a pack to nameparser ---------------------------------- diff --git a/docs/migrate.rst b/docs/migrate.rst index 573842063..02f5b1545 100644 --- a/docs/migrate.rst +++ b/docs/migrate.rst @@ -12,8 +12,10 @@ per old name. The 3.0 API is the one 2.0 introduced — ``parse``, ``Parser``, ``Lexicon``, ``Policy``, ``Locale`` and ``ParsedName``, with :mod:`nameparser.locales` — and its names and signatures carry into 3.0 -gaining additions only. The vocabulary data modules in -``nameparser.config`` (``titles``, ``particles`` and the rest) stay. +gaining additions only, save one: a ``Lexicon`` is configured by +*reading* in 3.0, its 2.x set fields internal (`Lexicon fields → +readings`_). The vocabulary data modules in ``nameparser.config`` +(``titles``, ``particles`` and the rest) stay. Separately, the output model changes in 3.0: ``str()`` now renders a name in the order it was read in, with the separators it was written with (`Changed in 3.0`_). Further output changes come with the 3.0 @@ -42,12 +44,12 @@ exceptions `Finding the sites`_ names: the `Attribute map`_ * - ``Constants(...)``, ``constants.copy()`` and the shared ``CONSTANTS`` - - ``Parser(lexicon=Lexicon.default().add(titles={"dean"}))``, and - the `Config map`_ for each setting + - ``Parser(lexicon=Lexicon.default().remove("dean", + kind="given"))``, and the `Config map`_ for each setting * - ``SetManager``, ``TupleManager`` and ``RegexTupleManager`` in ``nameparser.config`` - - ``Lexicon`` fields take plain sets: - ``Lexicon.default().add(titles={"dean"})`` + - ``Lexicon.add`` and ``Lexicon.remove`` take words and a reading + kind: ``Lexicon.default().add("leed", kind="credential")`` * - ``nameparser.util`` and its ``lc`` - ``value.lower().strip(".")`` * - Loading a pickled ``HumanName`` @@ -291,14 +293,16 @@ Vocabulary sets → Lexicon ~~~~~~~~~~~~~~~~~~~~~~~~~ The vocabulary sets of a 1.x or 2.x ``Constants`` (and so of the shared -``CONSTANTS``) map onto :class:`~nameparser.Lexicon` fields: +``CONSTANTS``) map onto the 2.x :class:`~nameparser.Lexicon` field of +the second column, and each of those onto the readings of +`Lexicon fields → readings`_ below: .. list-table:: :header-rows: 1 - :widths: 30 40 30 + :widths: 34 33 33 * - 1.x/2.x ``Constants`` - - 3.0 ``Lexicon`` + - 2.x ``Lexicon`` field - Note * - ``titles`` - ``titles`` @@ -329,32 +333,122 @@ The vocabulary sets of a 1.x or 2.x ``Constants`` (and so of the shared - * - ``capitalization_exceptions`` - ``capitalization_exceptions`` - - Pair-valued, so not ``add()``/``remove()``; extend the defaults - with ``dataclasses.replace()`` as :ref:`case-exceptions` shows. - Since 2.4 a value is a case mask, so ``{"md": "M.D."}`` repairs - ``md`` to ``MD`` - -Some ``Lexicon`` fields never had a ``Constants`` attribute — -``conjunctions_ambiguous``, ``maiden_markers``, ``surnames``, -``honorific_tails``, ``after_given_titles``, ``suffix_words_ambiguous``, -``agnomens`` and ``given_name_abbreviations`` — so code coming from -``Constants`` has nothing to translate for them. + - A field still. Pair-valued, so not ``add()``/``remove()``; extend + the defaults with ``dataclasses.replace()`` as + :ref:`case-exceptions` shows. Since 2.4 a value is a case mask, + so ``{"md": "M.D."}`` repairs ``md`` to ``MD`` .. warning:: - ``non_first_name_prefixes`` and ``particles_ambiguous`` mark - **complementary** sets, not the same set under a new name. ``non_first_name_prefixes`` lists particles that are *never* read as - a given name; ``particles_ambiguous`` lists the particles that - *may* be read as one. The same holds for the config constant behind - it: ``particles.NON_GIVEN_NAME_PARTICLES`` marks the never-given - set, so it is the complement of ``particles_ambiguous`` too, however - much the two names suggest otherwise. Translating a customization - means flipping the set: ``particles_ambiguous = lexicon.particles - - constants.non_first_name_prefixes``. Copying - ``non_first_name_prefixes`` straight into ``particles_ambiguous`` - silently inverts which particles are allowed to double as a given - name. + a given name, which 3.0 spells as a particle with no ``given`` + reading; a particle that *may* be read as one holds a ``given`` + reading too. The config constant ``particles.NON_GIVEN_NAME_PARTICLES`` + marks the never-given set, however much its name suggests + otherwise. So a 1.x ``non_first_name_prefixes`` entry is + ``remove(word, kind="given")``, and a word missing from it + ``add(word, kind="given")``. + +Lexicon fields → readings +~~~~~~~~~~~~~~~~~~~~~~~~~ + +A 3.0 :class:`~nameparser.Lexicon` holds readings, what a word may be +read as in the vocabulary files' terms, and compiles the sets a parse +reads from them; the 2.x set fields are internal. A caller states what +a word is, and the lexicon derives which sets that puts it in, so +making ``Guru`` a possible given name is ``add("guru", kind="given")`` +where 2.x needed ``remove(titles={"guru"}).add(titles_ambiguous={"guru"})``. +Each 2.x spelling raises where it is written, translated for the words +it names: + +- ``add(...)`` or ``remove(...)`` with a set field as a keyword raises + ``TypeError`` naming the calls for those words, so + ``Lexicon.default().add(titles={"dean"})`` names + ``.remove('dean', kind='given')``; +- ``Lexicon(titles=...)`` and ``dataclasses.replace(lexicon, + titles=...)`` raise the same way; +- reading ``lexicon.titles`` raises ``AttributeError``, naming + :meth:`~nameparser.Lexicon.readings_of`, and for what a parse read, + :attr:`Token.reading `. + +.. list-table:: + :header-rows: 1 + :widths: 32 68 + + * - 2.x ``Lexicon`` field + - 3.0 reading + * - ``titles`` + - ``add(w, kind="title")``, or a finer title kind (``office``, + ``academic``, ...); where the word holds a ``given`` reading, + ``remove(w, kind="given")`` makes it a title everywhere + * - ``titles_ambiguous`` + - ``add(w, kind="title")`` and ``add(w, kind="given")`` + * - ``given_name_titles`` + - ``add(w, kind="title", addresses="given")`` + * - ``after_given_titles`` + - ``add(w, kind="salutation", after_given=True)`` + * - ``title_modifiers`` + - ``add(w, kind="modifier")`` + * - ``title_styles`` + - ``add(w, kind="style")`` + * - ``title_phrases`` + - ``attaches=[...]`` on the first word's title reading, naming the + title words or kinds it pairs with + * - ``suffix_acronyms`` + - ``add(w, kind="credential")`` + * - ``suffix_acronyms_ambiguous`` + - a credential and a ``family`` reading + * - ``suffix_words`` + - ``add(w, kind="suffix")``, or ``generation`` for a word that + numbers its bearer (read by the ``generation`` view) + * - ``suffix_words_ambiguous`` + - a ``generation`` and a ``family`` reading + * - ``agnomens`` + - a ``family`` reading, then ``add(w, kind="agnomen")``, an agnomen + needing the name reading that gates it + * - ``honorific_tails`` + - ``attached="after"`` on the suffix reading + * - ``particles`` + - ``add(w, kind="particle")``, or a finer particle kind + * - ``particles_ambiguous`` + - a particle and a ``given`` reading, so + ``remove(particles_ambiguous=...)`` is ``remove(w, kind="given")`` + * - ``conjunctions`` + - ``add(w, kind="conjunction")`` + * - ``conjunctions_ambiguous`` + - ``may_be_initial=True`` on the conjunction reading + * - ``bound_given_names`` + - ``add(w, kind="bound_given")`` + * - ``given_name_abbreviations`` + - ``add(w, kind="given", same_as="")`` + * - ``maiden_markers`` + - ``add(w, kind="birth_marker")``, a phrase whole (``"z domu"``) + * - ``surnames``, ``given_names``, ``capitalization_exceptions`` + - unchanged, fields still; ``add``/``remove`` no longer take the + two lists, which join with ``lexicon | Lexicon(surnames={...})`` + +A ``Lexicon`` pickled by 2.x holds the compiled sets, and loads where +readings compile them back: as the default or empty lexicon with each +word that differs replaced by the readings it compiles to, so a 2.8 +``Lexicon.empty().add(titles={"dean"})`` loads equal to 3.0's +``Lexicon.empty().add("dean", kind="title")``. One no readings can +express (a word in ``particles_ambiguous`` and ``suffix_acronyms`` but +not ``suffix_acronyms_ambiguous``, say) is refused, naming the fields; +so are a ``Parser`` and a ``Locale`` holding it. A loaded lexicon keeps +the vocabulary it was pickled with, not 3.0's: a 2.8.0 pickle of +``Lexicon.default()``, or of a ``Parser()``, loads as the 3.0 default +with the 23 credentials and 13 case masks 3.0 added taken out again, so it reads +``"Jane Doe KC"`` as 2.8.0 did. Build a new ``Parser()`` for 3.0's +vocabulary. + +A 2.x call or pickle that took a word out of a set but left it in a +marker over that set (``remove(titles={"air"})``, which left ``air`` in +``title_modifiers``) reads as though the marker went with the word: in +3.0 the marker is a property of the reading, and goes with it. 2.x still +read such a leftover in two places, so two readings move: +``"John Smith, AIR"`` gives suffix ``AIR`` where 2.8.0 gave given ``AIR``, +family ``John Smith``, and after ``remove(titles={"sir"})``, +``"Sir. Smith"`` gives family ``Smith`` where 2.8.0 gave given ``Smith``. Default word lists are frozen ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -384,7 +478,7 @@ to a parser: >>> from nameparser import Lexicon, Parser >>> parse("Dean Smith").given # a title only in a phrase 'Dean' - >>> dean_parser = Parser(lexicon=Lexicon.default().add(titles={"dean"})) + >>> dean_parser = Parser(lexicon=Lexicon.default().remove("dean", kind="given")) >>> dean_parser.parse("Dean Smith").title 'Dean' @@ -715,7 +809,7 @@ grep your fixtures for, with what 1.4 read: holds for ``Princess``, ``Sultana``, ``Shaikh``, ``Sheik``, ``Shekh``, ``Shaik``, ``Cheikh``, ``Mahdi``, ``Bard``, ``Buddha``, ``Siddha``, ``Gyani`` and ``Wing``, and for a lone qualifier such as ``Grand``. - ``Lexicon.default().add(titles={"king"})`` makes one a title + ``Lexicon.default().remove("king", kind="given")`` makes one a title everywhere. - ``"Chief Justice John Roberts"`` and ``"Major General John Smith"`` (2.7) — a title phrase is one title: 1.4 read title ``Chief``, first diff --git a/docs/modules.rst b/docs/modules.rst index eb9d3c57c..075cb0493 100644 --- a/docs/modules.rst +++ b/docs/modules.rst @@ -92,6 +92,9 @@ Configuration .. autoclass:: nameparser.Lexicon :members: +.. autoclass:: nameparser.LexiconReading + :members: + .. autoclass:: nameparser.Policy :members: @@ -201,11 +204,12 @@ Locales Default vocabulary ------------------ -The data modules in ``nameparser.config``: the word lists -``Lexicon.default()`` is built from, -and the compiled patterns in ``regexes``. To change a default for your -parses, configure a :class:`~nameparser.Lexicon` rather than the -module (:doc:`customize`). Coming from 1.x? See :doc:`migrate`. +The data modules in ``nameparser.config``: the sets +``Lexicon.default()`` compiles its readings to, read-only, and the +compiled patterns in ``regexes``. To ask what a word may be read as, +call :meth:`Lexicon.readings_of() `; to +change a default for your parses, configure a +:class:`~nameparser.Lexicon` rather than the module (:doc:`customize`). Coming from 1.x? See :doc:`migrate`. .. automodule:: nameparser.config.titles :members: diff --git a/docs/release_log.rst b/docs/release_log.rst index 969d4c4ca..8beb2c6ad 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -14,7 +14,7 @@ Release Log - **Add ParsedName.patronymic, the words naming the bearer's parent.** ``parse("Mahathir bin Mohamad")`` gives patronymic ``bin Mohamad`` beside family ``bin Mohamad``, which 1.4.0 through 2.8.0 gave alone and which is unchanged; under ``locales.RU``, ``Ivan Ivanovich Ivanov`` gives patronymic ``Ivanovich`` and keeps it as the middle name, and under ``locales.TR_AZ`` a marker such as ``oglu`` reads with the name before it. A view like ``family_base``, not a field of ``as_dict()``, read only from a patronymic particle (``bin``, ``binti``, ``binte``, ``ibn``, ``bint``, ``bar``, ``bat`` and their Arabic and Hebrew forms) or a rule a locale turns on, so ``Anwar Ibrahim`` has none, and a name written by ``replace()``, or unpickled from 2.x, has none either. ``Ambiguity.alternative`` of a leading ``bin`` read as a given name names a patronymic where 2.7.0 and 2.8.0 named a particle. :issue:`754` - **Add ParsedName.surname2, the second of two family names.** ``Parser(policy=Policy(two_family_names=True)).parse("Gabriel García Márquez")`` gives surname2 ``Márquez`` beside family ``García Márquez``, which 2.8.0 gave alone and which is unchanged. A view like ``family_base``, not a field of ``as_dict()``, filled only where ``two_family_names`` reads the family as two, so the default, a family comma (``Lloyd Webber, Andrew``) and a hyphenated family have none, and so does a name unpickled from 2.x or one whose family ``replace()`` rewrote. ``Reading.second_surname`` marks its words; it is not compared. :issue:`754` - **Add a switch for names with a compound given name.** ``Parser(policy=Policy(two_given_names=True)).parse("Juan Carlos Pérez")`` gives given ``Juan Carlos``, where 1.4.0 through 2.8.0 gave given ``Juan``, middle ``Carlos``. Off by default: under it a middle name beside the given name joins it (``John Quincy Adams`` gives given ``John Quincy``). With ``two_family_names`` on too, the family takes its word first, so ``María José García López`` reads given ``María José``, family ``García López``. A ``Policy`` pickled by 2.8 loads with it off. :issue:`704` - - **Add Lexicon.given_names, given names two_family_names leaves out of the family.** ``Parser(policy=Policy(two_family_names=True), lexicon=Lexicon.default().add(given_names={"luis"})).parse("José Luis Pérez")`` gives middle ``Luis``, family ``Pérez``, where the switch alone, in 2.8.0 as now, gives family ``Luis Pérez``. Empty by default, so the switch alone reads as before; an entry without accents matches the word with them. :issue:`959` + - **Add Lexicon.given_names, given names two_family_names leaves out of the family.** ``Parser(policy=Policy(two_family_names=True), lexicon=Lexicon.default() | Lexicon(given_names={"luis"})).parse("José Luis Pérez")`` gives middle ``Luis``, family ``Pérez``, where the switch alone, in 2.8.0 as now, gives family ``Luis Pérez``. Empty by default, so the switch alone reads as before; an entry without accents matches the word with them. :issue:`959` - **Add Token.reading, the reading the parse took for each word.** ``parse("John Smith PhD Jones").tokens[-1].reading`` is ``Reading('credential' as 'suffix', 'position')``, the reading 2.7's ``Ambiguity.alternative`` gave only for the other side of a fork, and ``ReadingKind`` gains ``patronymic``. The field is not compared, and a ``ParsedName`` pickled by 2.x loads with no readings. :issue:`754` - **Add Reading.languages, the language files that list a word the way it was read.** ``parse("Dr. Juan de la Vega III").tokens[2].reading.languages`` is ``('es', 'fr', 'it', 'nl', 'pt')`` for the particle ``de``, and ``Ambiguity.alternative`` names its files too; a reading by shape or position names none. It is not compared, and a ``Reading`` pickled by 2.7 or 2.8 loads with none. The first parse in a process that meets a listed word loads the reading index once: about 1 ms, or about 16 ms where Python writes no bytecode cache. :issue:`754`, :issue:`753` - **Add Czech, Slovak, Slovenian and Croatian packs, locales.CS, SK, SL and HR.** ``parser_for(locales.CS).parse("Jan Novák, MUDr")`` gives title ``MUDr``, given ``Jan``, family ``Novák``, where 1.4.0 through 2.8.0 gave given ``MUDr``, family ``Jan Novák``. Each pack chooses its one language and turns on words the default does not read: ``ml.`` and ``st.`` after the name in all four (``Jiří Dienstbier ml.``), the Czech and Slovak academic titles and ``DrSc.``, and Czech ``Pan`` and ``roz.``. Choosing the language drops ``st``'s Saint readings (``St Clair Smith`` reads given ``St``), which choosing ``fr`` as well keeps. ``CS`` and ``SK`` also read two family names, so ``Alena Pešová Řehořová`` gives family ``Pešová Řehořová`` and two given names read as a given name and a family name (``Petr Pavel Novák`` gives family ``Pavel Novák``); a family comma keeps both. :issue:`753` @@ -26,6 +26,7 @@ Release Log **Breaking Changes** + - **Configure a Lexicon by reading; its set fields are internal.** ``Lexicon.default().remove("dean", kind="given")`` reads ``Dean Smith`` title ``Dean``, the spelling of 2.8.0's ``Lexicon.default().add(titles={"dean"})``, which now raises ``TypeError`` naming that call, as do ``Lexicon(titles=...)`` and ``dataclasses.replace`` with a set field, and reading ``lexicon.titles`` raises ``AttributeError``; ``add(*words, kind=..., **properties)``, ``remove(*words, kind=None)`` and ``readings_of(word)``, returning ``LexiconReading``, take the vocabulary files' kinds, and :doc:`migrate` has a row per 2.8.0 field. A 2.8.0 pickle of a ``Lexicon``, ``Parser`` or ``Locale`` loads where readings compile its sets back, keeping the vocabulary it was pickled with rather than taking 3.0's, and is refused otherwise; a 2.x call or pickle that left a word in a marker set over a set it took the word out of reads as though the marker went with it (after ``remove(titles={"air"})``, ``John Smith, AIR`` gives suffix ``AIR``, given ``AIR`` in 2.8.0). :issue:`754` - **Remove HumanName, Constants and the rest of the 1.x compatibility layer.** ``from nameparser import HumanName`` raises ``ImportError``, where 1.4.0 through 2.8.0 imported it (2.8.0 warning where it was constructed); ``parse("Dr. Juan de la Vega")`` reads the same name, with given ``Juan`` and family ``de la Vega``. ``nameparser.parser``, ``nameparser.util``, ``nameparser.config``'s ``Constants``, ``CONSTANTS`` and managers, and the five word-list names and two module paths 2.2 renamed go too, and a pickled ``HumanName`` or ``Constants`` no longer loads; the word lists in ``nameparser.config`` stay. See :doc:`migrate`. :issue:`357` **Behavior Changes** diff --git a/nameparser/__init__.py b/nameparser/__init__.py index 66c9cc684..6edfa9bfb 100644 --- a/nameparser/__init__.py +++ b/nameparser/__init__.py @@ -5,7 +5,7 @@ __license__ = "LGPL" __url__ = "https://github.com/derek73/python-nameparser" -from nameparser._lexicon import Lexicon +from nameparser._lexicon import Lexicon, LexiconReading from nameparser._locale import Locale from nameparser._parser import Parser, parse, parser_for from nameparser._policy import ( @@ -40,7 +40,7 @@ "Span", "Role", "Token", "Ambiguity", "AmbiguityKind", "Reading", "ReadingKind", "ReadingSource", "ParsedName", "STABLE_TAGS", "Segmentation", "Segmenter", - "Lexicon", "Policy", "PolicyPatch", "PatronymicRule", "Script", "UNSET", + "Lexicon", "LexiconReading", "Policy", "PolicyPatch", "PatronymicRule", "Script", "UNSET", "CapsSuffixes", "GIVEN_FIRST", "FAMILY_FIRST", "FAMILY_FIRST_GIVEN_LAST", "DEFAULT_NICKNAME_DELIMITERS", "DEFAULT_SCRIPT_ORDERS", "Locale", diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index ee80297f5..1a2ed53cd 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -1,9 +1,12 @@ -"""Immutable vocabulary configuration for the 2.0 API. - -Layering: may import nameparser.config DATA modules (the imports in -_default_lexicon() are the authoritative list) as the single source of -vocabulary -- never the nameparser.config package itself. Enforced by -tests/v2/test_layering.py. +"""Immutable vocabulary configuration: a Lexicon holds readings, what +each word may be read as, and compiles the sets a parse reads from them +(3.0's 4i-b; decisions/vocabulary-model, 2026-11-02 #754). + +Layering: may import nameparser.config DATA modules as the single +source of vocabulary -- never the nameparser.config package itself -- +and nameparser._reading_index, the shared compiler and the index's +reader, at call time only: tools/vocab/build.py loads this module by +path. Enforced by tests/v2/test_layering.py. """ from __future__ import annotations @@ -14,8 +17,8 @@ import warnings from collections.abc import Iterable, Mapping, Set from dataclasses import dataclass, field -from types import FrameType, MappingProxyType -from typing import Any, cast +from types import FrameType, MappingProxyType, ModuleType +from typing import TYPE_CHECKING, Any, NamedTuple, cast #: Vocabulary set fields, in declaration order. add()/remove() operate #: on exactly these and reject capitalization_exceptions (its entries @@ -39,66 +42,6 @@ #: importing `_types`, so tests/v2/test_lexicon.py holds the two equal. _ADDED_IN = "nameparser.added_in" -#: (marker, base, why) triples. Each marker QUALIFIES how entries of -#: its base vocabulary are read and carries no vocabulary of its own, -#: so an entry outside the base is a configuration mistake -- but the -#: mistake differs per pair, and the reason is recorded here rather -#: than generalized, because an orphan is NOT simply inert. Nor is the -#: qualification one-directional: the first two NARROW their base (an -#: entry is read as vocabulary in fewer places), while honorific_tails -#: WIDENS it, granting a suffix word the glued position on top of the -#: whole-token match every suffix word already gets. -#: -#: * particles_ambiguous: _assign keys on the tag alone, so an orphan -#: makes the parse emit a spurious particle-or-given ambiguity. -#: * suffix_acronyms_ambiguous: _vocab returns True on the ambiguous -#: set before testing suffix_acronyms, so an orphan silently turns a -#: word into a period-gated suffix. -#: * honorific_tails: script_segment peels the tail into its own token -#: before classify ever runs, so an orphan splits the name and leaves -#: the fragment stranded inside it -- worse than not peeling at all. -#: Its base is deliberately NARROWER than what actually claims the -#: peeled piece: suffix_as_written ORs suffix_words with the -#: non-ambiguous suffix_acronyms, so a tail listed only as an acronym -#: would classify fine yet is rejected here. Accepted, and a decision -#: rather than an oversight -- the three-term predicate is easy to -#: get wrong in the dangerous direction (an ambiguous acronym admitted -#: as a tail would peel a period-gated word off a real name), and -#: nothing needs the acronym half: the shipped tails are CJK -#: honorifics, which are words. -#: The shipped vocabulary holds the relation by construction: its -#: build puts a word in GLUED_HONORIFICS only through an `attached` -#: property on a word reading (tools/vocab/build.py). The check here -#: is the only thing that holds a caller's own. -#: -#: Three fields are deliberately NOT here and have no check of their own. -#: given_name_titles -- see the note in __post_init__ for why every -#: attempt at one rejected working configurations. conjunctions_ambiguous -#: -- an orphan is never consulted because both the classify fork and -#: its ambiguity emitter require the base entry as well, so an orphan -#: decides nothing, and AGENTS.md's invariants rule guards harm, not -#: no-ops. after_given_titles -- the same: classify tags a word from it -#: only where the word is in titles too, so an orphan is never read. -#: agnomens -- the same: classify reads it only for a word already in -#: suffix_words_ambiguous, so an orphan decides nothing. -#: title_modifiers, title_styles and title_phrases -- the same: the -#: title phrase test (rules/H9) reads them only for words that are -#: titles, so an orphan decides nothing. titles_ambiguous is no subset -#: at all: its words are titles only inside a phrase, so a caller lists -#: one there without listing it in titles, and one in both is a title -#: everywhere. -_SUBSET_FIELDS = ( - ("particles_ambiguous", "particles", - "an orphan emits a spurious particle-or-given ambiguity"), - ("suffix_acronyms_ambiguous", "suffix_acronyms", - "an orphan silently becomes a period-gated suffix"), - ("honorific_tails", "suffix_words", - "an orphan splits the name and leaves the tail inside it"), - ("suffix_words_ambiguous", "suffix_words", - "an orphan silently becomes a suffix gated by the count"), -) - - #: The fields whose entries may be PHRASES: stored space-joined and #: folded per word (_title_key), and exempt from the dead-entry warning #: below. Every other field is matched one word at a time, where a @@ -327,7 +270,7 @@ def _reject_buffer(value: object, label: str, plural: str) -> None: ) -def _warn_dead_entry(message: str) -> None: +def _warn_at_caller(message: str) -> None: # A fixed stacklevel always lands on library internals: the call # depth differs per entry point (Lexicon(), add(), unpickle, # dataclasses.replace). Walk out of this module (and dataclasses' @@ -407,7 +350,7 @@ def _normset( # interior whitespace test; split() covers all Unicode # whitespace and n != "".join(n.split())): - _warn_dead_entry( + _warn_at_caller( f"Lexicon.{field_name} entries are matched one word at " f"a time; multi-word entry {w!r} can never match. " f"Split it into separate entries") @@ -544,7 +487,7 @@ def _normpairs( # config that was always going to be a no-op. # interior whitespace test; split() covers all Unicode whitespace if normalized_key != "".join(normalized_key.split()): - _warn_dead_entry( + _warn_at_caller( f"capitalization_exceptions keys are matched one word " f"at a time; multi-word key {k!r} can never match. " f"Split it into per-word entries") @@ -567,26 +510,51 @@ def _normpairs( return tuple(sorted(deduped.items())) +#: The two lists outside the per-word model (decisions/vocabulary-model's +#: Standing line on name readings): `Lexicon` fields, changed through +#: ``|`` with a fragment or ``dataclasses.replace``, and copied into +#: `_LexiconSets` beside the compiled sets. +_LIST_FIELDS = ("surnames", "given_names") + +#: The 20 sets a compile of readings gives, under the names the 2.x +#: `Lexicon` fields had: internal since 3.0 (#754, 4i-b), read through +#: `Lexicon._sets`, and the names an old spelling is refused by. +_SET_FIELDS = tuple(f for f in _VOCAB_FIELDS if f not in _LIST_FIELDS) + +#: The kinds those sets do not record, which the views read (rules/R8, +#: rules/R9): `_LexiconSets` attributes after the `_VOCAB_FIELDS` ones, +#: each a compiled constant of its own. +_KIND_FIELDS = ("generation_words", "credential_words", + "patronymic_particles") + +#: Each marker set over the base set it qualifies. A compile from +#: readings holds every pair by construction, the marker existing only +#: beside its base reading; `_parser._apply_choice` reads the pairs to +#: drop a word's marks where a choice drops it from the base. +_MARKER_BASES = ( + ("particles_ambiguous", "particles"), + ("suffix_acronyms_ambiguous", "suffix_acronyms"), + ("honorific_tails", "suffix_words"), + ("suffix_words_ambiguous", "suffix_words"), +) + + @dataclass(frozen=True, slots=True, eq=False, repr=False) class _LexiconSets: - """The vocabulary sets a parse reads, held apart from the public - `Lexicon` fields: the pipeline, `_render`, `_parser` and the tools - read `lexicon._sets.titles`, never `lexicon.titles` (#754, 4i-a; - tests/v2/test_layering.py holds it). One attribute per - `_VOCAB_FIELDS` entry, in that order, each a frozenset, so a - membership test stays a C-level attribute read and costs no frame. - - Built once per `Lexicon`, in its `__post_init__`, from the - lexicon's own fields, and never compared, hashed or pickled: a - `Lexicon`'s equality and pickle are its fields', and a load or a - `dataclasses.replace` builds this again. 4i-b, which makes the - compiled fields internal, changes only how it is filled: from the - lexicon's readings rather than from its public fields, the default - from nameparser/config/_compiled.py as today, so the reading index - stays unloaded on `parse("John Smith")`; the derived markers the - views will read (a listed suffix word's kind, the patronymic - particles) join it as attributes of their own. Nothing that reads - it moves then. + """The vocabulary sets a parse reads: the pipeline, `_render`, + `_parser` and the tools read `lexicon._sets.titles` (#754, 4i-a; + tests/v2/test_layering.py holds it), the 2.x public fields having + become internal in 4i-b. One attribute per `_VOCAB_FIELDS` entry, in + that order, then one per `_KIND_FIELDS` entry, each a frozenset, so + a membership test stays a C-level attribute read and costs no frame. + + Built once per `Lexicon`, at construction: the compiled sets from + its readings by `compile_readings`, or for `Lexicon.default()` from + nameparser/config/_compiled.py, so the reading index stays unloaded + on `parse("John Smith")`; the two lists from its fields. Never + compared, hashed or pickled: a `Lexicon`'s equality and pickle are + its fields', and a load or a `dataclasses.replace` builds this + again. Not a mapping and no repr of its own: it holds whole word lists, and a repr of them would be unbounded.""" @@ -613,141 +581,1008 @@ class _LexiconSets: title_styles: frozenset[str] title_phrases: frozenset[str] given_names: frozenset[str] + #: the suffix words whose suffix reading is a generation (`jr`, + #: `iii`), a credential written as a word (`kc`), and the particles + #: read as patronymics (`bin`): the kinds rules/R8's and rules/R9's + #: views read, which the sets above do not record + generation_words: frozenset[str] + credential_words: frozenset[str] + patronymic_particles: frozenset[str] -class _HoldsSets: - """`Lexicon`'s one slot outside its dataclass fields, holding its - `_LexiconSets`. A slot inherited from a plain base, because a field, - even one built `init=False`, would change `dataclasses.fields`, the - pickled state and the 2.8 pickles' layout check, and 4i-a changes - nothing public; `@dataclass(slots=True)` takes no slot of its own - beside its fields. Set once, by `Lexicon.__post_init__`.""" +#: Lexicon field -> the compiled constant it is read from, for every +#: set the vocabulary compiles (nameparser/_reading_index.py's +#: Compiled); `particles_ambiguous` is derived (_vocab_fields below). +#: tests/v2/test_parser.py holds that these, read from the compile of +#: no choice, are Lexicon.default()'s. +_COMPILED_FIELDS = { + "titles": "TITLES", + "given_name_titles": "GIVEN_NAME_TITLES", + "suffix_acronyms": "SUFFIX_ACRONYMS", + "suffix_words": "SUFFIX_WORDS", + "suffix_acronyms_ambiguous": "SUFFIX_ACRONYMS_AMBIGUOUS", + "particles": "PARTICLES", + "conjunctions": "CONJUNCTIONS", + "conjunctions_ambiguous": "CONJUNCTIONS_AMBIGUOUS", + "bound_given_names": "BOUND_GIVEN_NAMES", + "given_name_abbreviations": "GIVEN_NAME_ABBREVIATIONS", + "maiden_markers": "MAIDEN_MARKERS", + "honorific_tails": "GLUED_HONORIFICS", + "after_given_titles": "AFTER_GIVEN_TITLES", + "suffix_words_ambiguous": "SUFFIX_WORDS_AMBIGUOUS", + "agnomens": "AGNOMENS", + "titles_ambiguous": "TITLES_AMBIGUOUS", + "title_modifiers": "TITLE_MODIFIERS", + "title_styles": "TITLE_STYLES", + "title_phrases": "TITLE_PHRASES", + "generation_words": "GENERATION_WORDS", + "credential_words": "CREDENTIAL_WORDS", + "patronymic_particles": "PATRONYMIC_PARTICLES", +} - __slots__ = ("_sets",) - _sets: _LexiconSets + +def _vocab_fields(compiled: Mapping[str, Any]) -> dict[str, frozenset[str]]: + """The sets a compile of the vocabulary gives, by `_LexiconSets` + name (the constants of nameparser/config/_compiled.py, or a compile + of readings), the two lists aside.""" + out = {name: compiled[key] for name, key in _COMPILED_FIELDS.items()} + # FLIPPED from v1: v1 marked the never-given subset; the sets mark + # the may-be-given one + out["particles_ambiguous"] = (compiled["PARTICLES"] + - compiled["NON_GIVEN_NAME_PARTICLES"]) + return out + + +@functools.cache +def _default_compiled() -> Mapping[str, frozenset[str]]: + # the compiled data module is the default's one source, so the + # default never loads the reading index (decisions/vocabulary- + # model, 2026-10-29 3a; its 2026-11-02 #754 entry, item 4) + import nameparser.config._compiled as compiled + return MappingProxyType(_vocab_fields(vars(compiled))) + + +_NO_SETS: Mapping[str, frozenset[str]] = MappingProxyType( + {name: frozenset() for name in _COMPILED_FIELDS} | { + "particles_ambiguous": frozenset()}) + + +# -- readings ------------------------------------------------------------ + +#: A reading's properties as the log and the repr compare them: sorted +#: (key, value) pairs, a list value a sorted tuple. +_Props = tuple[tuple[str, Any], ...] + + +class _Edit(NamedTuple): + """One `add` or `remove`, stored frozen as written (decisions/ + vocabulary-model, 2026-11-02 #754, item 3): its words normalized, + its kind (None on a `remove` of every reading), its properties + sorted pairs, a property given its unwritten value held as None, + which removes it. Compared and hashed as a tuple.""" + + op: str + words: tuple[str, ...] + kind: str | None + props: _Props + + +def _index_module() -> ModuleType: + # the reading index's reader, imported at call time: tools/vocab/ + # build.py loads this module by path, before the index it rewrites + # can be trusted to import (root AGENTS.md's layering bullet) + import nameparser._reading_index as ri + return ri + + +def _clean(props: Mapping[str, Any]) -> _Props: + """A reading's properties as a caller states them: `default` (a + caller's reading is on) and `cased` (a case mask, configured through + `capitalization_exceptions`) left out.""" + return tuple(sorted((k, v) for k, v in props.items() + if k not in ("default", "cased"))) + + +def _unwritten(kind: str, prop: str) -> object: + """The value `prop` takes on a `kind` reading that does not write + it, read through _reading_index.ReadingProperties: `acronym` is + true on a credential, `addresses` the family name.""" + if prop == "acronym": + return kind == "credential" + if prop == "addresses": + return "family" + return False + + +def _prop_value(kind: str, prop: str, value: object) -> object: + """`value` canonical for the log, None where it is the unwritten + one (which removes the property).""" + if value is None or (isinstance(value, (bool, str)) + and value == _unwritten(kind, prop)): + return None + if prop in ("acronym", "after_given", "may_be_initial"): + if not isinstance(value, bool): + raise TypeError(f"{prop} must be True or False, got {value!r}") + return value + if prop == "addresses": + if value != "given": + raise ValueError(f"addresses must be 'given' or 'family', " + f"got {value!r}") + return value + if prop == "attached": + if value != "after": + raise ValueError(f"attached must be 'after' (a tail peeled " + f"off the end of a name token) or None, " + f"got {value!r}") + return value + if prop == "attaches": + if isinstance(value, (str, bytes)) or isinstance(value, Mapping): + raise TypeError(f"attaches must be an iterable of title kinds " + f"or title words, e.g. attaches=['military'], " + f"got {value!r}") + try: + items: tuple[object, ...] = tuple(cast("Iterable[object]", value)) + except TypeError: + raise TypeError(f"attaches must be an iterable of title kinds " + f"or title words, got {value!r}") from None + words = [x for x in items if isinstance(x, str)] + if len(words) != len(items): + raise TypeError(f"attaches entries must be strings, got " + f"{value!r}") + folded = tuple(sorted({_normalize(x) for x in words} - {""})) + return folded or None + # same_as, a given name written short's full name (#789) + if not isinstance(value, str) or not value.strip(): + raise TypeError(f"same_as must be the full given name, a " + f"non-empty string, got {value!r}") + return value + + +def _check_kind(kind: object, op: str) -> str | None: + ri = _index_module() + if kind is None and op == "remove": + return None + if not isinstance(kind, str): + raise TypeError(f"{op}() takes kind as a string, a vocabulary " + f"kind such as 'title' or 'credential', got " + f"{kind!r}") + if kind in ri.CATEGORY: + return kind + categories = set(ri.CATEGORY.values()) + if kind in categories: + if op == "remove": + return kind + raise ValueError( + f"{kind!r} is a category with no kind of its name; add() " + f"takes one of its kinds: " + ", ".join( + k for k, c in ri.CATEGORY.items() if c == kind)) + raise ValueError( + f"unknown reading kind {kind!r}; the kinds are " + + ", ".join(ri.CATEGORY) + + (" (or a category, on remove: " + + ", ".join(sorted(categories)) + ")" if op == "remove" else "")) + + +def _edit_words(words: tuple[object, ...], op: str) -> tuple[str, ...]: + if not words: + raise TypeError(f"{op}() takes at least one word, e.g. " + f"{op}('dean', kind='title')") + out = [] + for w in words: + _reject_buffer(w, f"{op}() words", "strings") + if not isinstance(w, str): + raise TypeError( + f"{op}() takes each word as its own argument, a string, " + f"got {w!r}; unpack a collection: {op}(*words, kind=...)") + n = _title_key(w.split()) + if not n: + raise ValueError(f"{op}() word {w!r} normalizes to empty " + f"(lowercase + strip full stops/whitespace " + f"leaves nothing)") + out.append(n) + return tuple(out) + + +def _edit_props(kind: str, props: Mapping[str, object]) -> _Props: + ri = _index_module() + allowed = ri.indexed(kind) - {"default", "cased"} + out: dict[str, object] = {} + for prop, value in props.items(): + if prop == "default": + raise ValueError( + "add() takes no default: a caller's reading is on, which " + "is the caller's choice; take one out with remove(word, " + "kind=...)") + if prop == "cased": + raise ValueError( + "add() takes no cased: a case mask is configured through " + "capitalization_exceptions, e.g. dataclasses.replace(" + "lexicon, capitalization_exceptions=(*lexicon." + "capitalization_exceptions, ('phd', 'PhD')))") + if prop not in allowed: + raise ValueError( + f"a {kind} reading takes " + + (", ".join(sorted(allowed)) or "no properties") + + f"; got {prop!r}") + out[prop] = _prop_value(kind, prop, value) + return tuple(sorted(out.items())) + + +def _make_edit(op: object, words: object, kind: object, + props: object) -> _Edit: + """An edit checked and canonical: what `add` and `remove` store and + what a constructor's `_edits` must already be.""" + if op not in ("add", "remove"): + raise ValueError(f"an edit is 'add' or 'remove', got {op!r}") + if not isinstance(words, tuple): + raise TypeError(f"an edit's words are a tuple, got {words!r}") + ws = _edit_words(words, op) + k = _check_kind(kind, op) + if op == "remove": + if props: + raise ValueError(f"remove() takes no properties, got {props!r}") + return _Edit(op, ws, k, ()) + assert k is not None + if not isinstance(props, (tuple, Mapping)): + raise TypeError(f"an edit's properties are pairs, got {props!r}") + p = _edit_props(k, dict(props)) + ri = _index_module() + category = ri.CATEGORY[k] + for w in ws: + # the build's check 8: a phrase is matched whole only as a birth + # marker or a run of titles addressing the given name + if " " in w and not (k == "birth_marker" or ( + category == "title" and ("addresses", "given") in p)): + raise ValueError( + f"{w!r} is a phrase, which can never match as a {k}: only " + f"a birth_marker and a title with addresses='given' are " + f"matched as a whole run; add its words one by one") + # the build refuses may_be_initial on anything but a cased single + # letter: no other word reads as an initial (rules/P8) + if ("may_be_initial", True) in p and not ( + len(w) == 1 and w.upper() != w.lower()): + raise ValueError( + f"may_be_initial marks a single cased letter that may read " + f"as an initial; {w!r} is not one") + return _Edit(op, ws, k, p) + + +def _apply_edit(pool: dict[str, tuple[Any, ...]], edit: _Edit) -> None: + """Apply `edit` to `pool` (word -> its readings) in place. An add of + a kind the word holds updates that reading's properties in every + file holding it, and turns a reading written `default = false` on; + otherwise it adds the caller's own reading, which names no file.""" + ri = _index_module() + for w in edit.words: + rs = pool.get(w, ()) + if edit.op == "remove": + kept = tuple(r for r in rs if edit.kind is not None + and r.kind != edit.kind and r.category != edit.kind) + elif any(r.kind == edit.kind for r in rs): + kept = tuple( + ri.IndexReading(w, r.tag, r.kind, MappingProxyType( + _merged(r.props, edit.props))) + if r.kind == edit.kind else r for r in rs) + else: + kept = rs + (ri.IndexReading(w, "", edit.kind, MappingProxyType( + {k: v for k, v in edit.props if v is not None})),) + if kept: + pool[w] = kept + else: + pool.pop(w, None) + + +def _merged(props: Mapping[str, Any], update: _Props) -> dict[str, Any]: + out = {k: v for k, v in props.items() if k != "default"} + for k, v in update: + if v is None: + out.pop(k, None) + else: + out[k] = v + return out + + +Pool = Mapping[str, tuple[Any, ...]] + + +@functools.lru_cache(maxsize=8) +def _base_pool(base: object) -> Pool: + """A base's readings by word, on by default or not: the shipped + index for `"default"`, none for `"empty"`, a snapshot's own rows.""" + if base == "empty": + return MappingProxyType({}) + ri = _index_module() + rows = ri._by_text().items() if base == "default" else base + return MappingProxyType({ + text: tuple(ri.from_row(text, row) for row in rs) + for text, rs in rows}) # type: ignore[union-attr] + + +def _check_pool(pool: Pool, words: Iterable[str]) -> None: + """The build's checks a caller's edits can reach, over the words + they touch (checks 3 and 8 run per edit, in _make_edit): one suffix + kind per word (check 4), an agnomen has a name reading (check 11's + agnomen clause), and a bound given name that is also a particle has + a given reading (check 9).""" + for w in dict.fromkeys(words): + on = [r for r in pool.get(w, ()) if r.on_by_default] + suffix = {r.kind: r for r in on if r.category == "suffix"} + if len(suffix) > 1: + held = sorted(k for k, r in suffix.items() if r.tag) + drop = held[0] if held else sorted(suffix)[0] + raise ValueError( + f"{w!r} would hold two suffix kinds, " + f"{' and '.join(sorted(suffix))}, and a word holds one; " + f"remove the one it holds first: .remove({w!r}, " + f"kind={drop!r})") + kinds = {r.category for r in on} | {r.kind for r in on} + # check 11's agnomen clause: an agnomen is written in the case of + # the name around it, so only the count gates it, and the gate is + # a name reading beside it (rules/S11) + if "agnomen" in kinds and "name" not in kinds: + raise ValueError( + f"{w!r} is an agnomen with no name reading, and an agnomen " + f"is read as the suffix only where a name reading gates " + f"it by the count: write .add({w!r}, kind='family') " + f"first, then the agnomen") + if {"bound_given", "particle"} <= kinds and "given" not in kinds: + raise ValueError( + f"{w!r} is a bound given name and a particle with no given " + f"reading, and a particle that never starts a given name " + f"cannot bind one: .add({w!r}, kind='given'), or " + f".remove({w!r}, kind='bound_given')") + + +@functools.lru_cache(maxsize=128) +def _built(base: object, edits: tuple[_Edit, ...]) -> tuple[ + Pool, Mapping[str, frozenset[str]]]: + """A lexicon's readings by word and the sets they compile to, kept + per base and log: a compile costs milliseconds, paid once per + configuration (decisions/vocabulary-model, 2026-11-01 #754).""" + pool = dict(_base_pool(base)) + for edit in edits: + _apply_edit(pool, edit) + _check_pool(pool, (w for e in edits for w in e.words)) + ri = _index_module() + try: + compiled = ri.compile_readings( + r for rs in pool.values() for r in rs) + except ri.CompileError as exc: + raise ValueError(str(exc)) from None + return MappingProxyType(pool), MappingProxyType(_vocab_fields(compiled)) + + +def _compiled_for(base: object, + edits: tuple[_Edit, ...]) -> Mapping[str, frozenset[str]]: + if not edits: + if base == "default": + return _default_compiled() + if base == "empty": + return _NO_SETS + return _built(base, edits)[1] + + +def _check_base(base: object) -> object: + """`"default"`, `"empty"`, or a snapshot of index rows from a pickle + (decisions/vocabulary-model, 2026-11-02 #754, items 3 and 5), which + becomes `"default"` where it is the running release's index.""" + if base in ("default", "empty"): + return base + if not isinstance(base, tuple): + raise TypeError(f"a Lexicon's base is 'default', 'empty' or a " + f"snapshot of index rows, got {base!r}") + ri = _index_module() + from nameparser.config._index import READINGS + if base == READINGS: + return "default" + shape = ("a Lexicon snapshot is (word, rows) pairs, each row (tag, " + "kind, properties), the properties (name, value) pairs") + for entry in base: + if not (isinstance(entry, tuple) and len(entry) == 2 + and isinstance(entry[0], str) + and isinstance(entry[1], tuple)): + raise TypeError(f"{shape}; got {entry!r}") + word, rows = entry + for row in rows: + if not (isinstance(row, tuple) and len(row) == 3): + raise TypeError(f"{shape}; {word!r} has the row {row!r}") + tag, kind, pairs = row + if not isinstance(tag, str) or not isinstance(kind, str): + raise TypeError(f"{shape}, the tag and the kind strings; " + f"{word!r} has the row {row!r}") + if kind not in ri.CATEGORY: + raise ValueError( + f"a Lexicon snapshot holds the reading kind {kind!r}, " + f"which this release does not know") + if not isinstance(pairs, tuple) or not all( + isinstance(p, tuple) and len(p) == 2 + and isinstance(p[0], str) for p in pairs): + raise TypeError(f"{shape}; {word!r} has the row {row!r}") + for name, value in pairs: + if name not in ri.indexed(kind): + raise ValueError( + f"a Lexicon snapshot gives {word!r}'s {kind} reading " + f"the property {name!r}, which a {kind} reading " + f"does not take in this release") + try: + hash(value) + except TypeError: + raise TypeError( + f"a Lexicon snapshot gives {word!r}'s {kind} reading " + f"{name}={value!r}; a value is a string, a bool or " + f"a tuple") from None + return base + + +# -- the 2.x set fields, translated -------------------------------------- + +#: The fields whose membership puts a word in a category. A marker set +#: implies its base here; a 2.x orphan, a marker entry its base does not +#: hold, never reaches this, `_inert_dropped` having taken it out. +_TITLE_FIELDS = frozenset({ + "titles", "titles_ambiguous", "given_name_titles", "after_given_titles", + "title_modifiers", "title_styles"}) +_ACRONYM_FIELDS = frozenset({"suffix_acronyms", "suffix_acronyms_ambiguous"}) +_WORD_FIELDS = frozenset({ + "suffix_words", "suffix_words_ambiguous", "agnomens", "honorific_tails", + "generation_words", "credential_words"}) +_PARTICLE_FIELDS = frozenset({ + "particles", "particles_ambiguous", "patronymic_particles"}) + + +def _props_by_kind(readings: Iterable[Any]) -> dict[str, list[dict[str, Any]]]: + out: dict[str, list[dict[str, Any]]] = {} + for r in readings: + out.setdefault(r.kind, []).append(dict(_clean(r.props))) + return out + + +def _by_kind(readings: Iterable[Any]) -> dict[str, dict[str, Any]]: + out: dict[str, dict[str, Any]] = {} + for r in readings: + out.setdefault(r.kind, dict(_clean(r.props))) + return out + + +def _target_readings(word: str, member: frozenset[str], + heads: frozenset[str] | None, + on: dict[str, dict[str, Any]], + hints: dict[str, dict[str, Any]], + kinds_known: bool) -> dict[str, dict[str, Any]]: + """The readings, kind -> properties, that compile `word` into the + sets `member` names, keeping each reading `word` holds (`on`) where + it fits and taking a kind it lacks from `hints`, then from the + category's own kind. `heads` are the title words a phrase starting + with `word` must reach, None where they do not change. A best + effort: the caller compiles the result and compares.""" + ri = _index_module() + + def kinds(category: str) -> list[str]: + return ([k for k in on if ri.CATEGORY[k] == category] + or [k for k in hints if ri.CATEGORY[k] == category]) + + def props(kind: str) -> dict[str, Any]: + return dict(on[kind] if kind in on else hints.get(kind, {})) + + out: dict[str, dict[str, Any]] = {} + if member & _TITLE_FIELDS or heads: + ks = kinds("title") or ["title"] + if "title_modifiers" in member: + ks = ["modifier"] + else: + ks = [k for k in ks if k != "modifier"] or ["title"] + if "title_styles" in member and "style" not in ks: + ks = [k for k in ks if k != "title"] + ["style"] + elif "title_styles" not in member: + ks = [k for k in ks if k != "style"] or ["title"] + for k in ks: + p = props(k) + p.pop("addresses", None) + p.pop("after_given", None) + if "given_name_titles" in member: + p["addresses"] = "given" + if "after_given_titles" in member: + p["after_given"] = True + if heads is not None or k not in on: + p.pop("attaches", None) + if heads: + p["attaches"] = tuple(sorted(heads)) + out[k] = p + particled = bool(member & _PARTICLE_FIELDS) + if particled: + ks = kinds("particle") or ["particle"] + if kinds_known: + if "patronymic_particles" in member: + ks = ["patronymic"] + else: + ks = [k for k in ks if k != "patronymic"] or ["particle"] + for k in ks: + out[k] = props(k) + if member & {"conjunctions", "conjunctions_ambiguous"}: + p = props("conjunction") + p.pop("may_be_initial", None) + if "conjunctions_ambiguous" in member: + p["may_be_initial"] = True + out["conjunction"] = p + if "bound_given_names" in member: + out["bound_given"] = props("bound_given") + if "maiden_markers" in member: + out["birth_marker"] = props("birth_marker") + acronym, worded = member & _ACRONYM_FIELDS, member & _WORD_FIELDS + held: str | None = next(iter(kinds("suffix")), None) + titled = "titles" in member + given_needed = (("titles_ambiguous" in member and not titled) + or "particles_ambiguous" in member + or "given_name_abbreviations" in member) + suffix: str | None = None + if acronym and not worded: + suffix = held or "credential" + elif worded: + if "agnomens" in member: + suffix = "agnomen" + elif kinds_known and "generation_words" in member: + suffix = "generation" + elif kinds_known and "credential_words" in member: + suffix = "credential" + elif "suffix_words_ambiguous" in member: + # an agnomen is its own kind, so a word that stops being one + # is the generation the gated class otherwise holds + suffix = (held if held in ri.GATED_WORDS and held != "agnomen" + else "generation") + else: + suffix = held or "suffix" + if suffix == "agnomen" or (kinds_known and suffix in ( + "generation", "credential")) or ( + suffix in ri.GATED_WORDS and given_needed): + suffix = "suffix" + # the name readings, which gate the vocabulary readings beside them: + # a word's own kept where they gate nothing they must not + names = {k: dict(p) for k, p in on.items() if ri.CATEGORY[k] == "name"} + given_ok = (not (titled and not particled) + and not (particled and "particles_ambiguous" not in member)) + named_ok = not ((acronym and "suffix_acronyms_ambiguous" not in member) + or (worded and suffix in ri.GATED_WORDS + and "suffix_words_ambiguous" not in member)) + if not given_ok or not named_ok: + names.pop("given", None) + if not named_ok: + names.pop("family", None) + if given_needed and "given" not in names: + names["given"] = props("given") + if "given" in names: + if "given_name_abbreviations" in member: + names["given"].setdefault("same_as", word) + else: + names["given"].pop("same_as", None) + if (member & {"suffix_acronyms_ambiguous", "suffix_words_ambiguous"} + and not names): + names["family"] = {} + if suffix is not None: + p = props(suffix) + p.pop("acronym", None) + p.pop("attached", None) + if acronym and not worded: + if suffix != "credential": + p["acronym"] = True + else: + if suffix == "credential": + p["acronym"] = False + if "honorific_tails" in member: + p["attached"] = "after" + out[suffix] = p + out.update(names) + return out + + +#: (marker, the sets one of which holds a word its marker qualifies): no +#: reading compiles a marker without its base, a property needing its +#: reading, so `_inert_dropped` takes a 2.x entry outside them out of +#: both sides of every compare, and a 2.x spelling or pickle that left +#: one behind -- `remove(titles={"air"})` keeping `air` a modifier -- +#: reads as though the marker went with its word. 2.x did read such an +#: orphan in two places, which move, an accepted difference +#: (decisions/vocabulary-model, 2026-11-02 #754, a marker left behind): +#: an orphan given-name title still made rules/H2's period-shape title +#: address the given name (`Sir. Smith` read given `Smith`), and any +#: orphan counted as a listed word in the capitals shape (`John Smith, +#: AIR` read given `AIR`). +_MARKER_READ_WITH = ( + ("given_name_titles", ("titles", "titles_ambiguous")), + ("after_given_titles", ("titles",)), + ("title_modifiers", ("titles", "titles_ambiguous")), + ("title_styles", ("titles", "titles_ambiguous")), + ("conjunctions_ambiguous", ("conjunctions",)), + ("particles_ambiguous", ("particles",)), + ("patronymic_particles", ("particles",)), + ("suffix_acronyms_ambiguous", ("suffix_acronyms",)), + ("suffix_words_ambiguous", ("suffix_words",)), + ("honorific_tails", ("suffix_words",)), + ("generation_words", ("suffix_words",)), + ("credential_words", ("suffix_words",)), + ("agnomens", ("suffix_words_ambiguous",)), +) + + +def _inert_dropped(sets: Mapping[str, frozenset[str]], + ) -> dict[str, frozenset[str]]: + """`sets` without the entries no reading compiles: a marker's word + its base sets do not hold, and a title phrase or a given-name title + run holding a word that is no title. In the order of + `_MARKER_READ_WITH`, so an agnomen whose ambiguous suffix word went + goes too.""" + out = dict(sets) + for marker, bases in _MARKER_READ_WITH: + if marker in out: + read = frozenset().union(*(out[b] for b in bases)) + out[marker] = frozenset( + w for w in out[marker] + if w in read or (marker == "given_name_titles" and " " in w + and all(x in read for x in w.split()))) + titles = out["titles"] | out["titles_ambiguous"] + out["title_phrases"] = frozenset( + p for p in out["title_phrases"] + if " " in p and all(w in titles for w in p.split(" "))) + return out + + +def _edits_toward(pool: Pool, current: Mapping[str, frozenset[str]], + target: Mapping[str, frozenset[str]], + hints: Pool | None = None) -> tuple[_Edit, ...]: + """The edits that take the readings of `pool`, which compile to the + sets `current`, toward the sets `target` (by `_LexiconSets` name; + a set it leaves out stays as `current` has it), word by word: for + each word whose memberships differ, a `remove` of each kind it no + longer holds, then an `add` of each kind it lacks or whose + properties change. The one translation from sets to readings, which + the refusals of a 2.x spelling, the load of a 2.x pickle and + `_parser._apply_choice` share; a best effort, so each caller + compiles the result where it must be exact. `hints` are readings a + word's new kinds are taken from before its held ones (a choice's).""" + ri = _index_module() + kinds_known = all(k in target for k in _KIND_FIELDS) + fields = _SET_FIELDS + (_KIND_FIELDS if kinds_known else ()) + full = _inert_dropped({f: target.get(f, current[f]) for f in fields}) + current = _inert_dropped({f: current[f] for f in fields}) + changed: set[str] = set() + for f in fields: + if f == "title_phrases": + changed |= {p.split(" ", 1)[0] + for p in full[f] ^ current[f]} + else: + changed |= full[f] ^ current[f] + + def heads(phrases: frozenset[str], word: str) -> frozenset[str]: + # a 2.x entry of one word is no phrase, and no head + return frozenset(p.partition(" ")[2] for p in phrases + if p.partition(" ")[0] == word and " " in p) + + out: list[_Edit] = [] + for w in sorted(changed): + held = pool.get(w, ()) + on = _by_kind(r for r in held if r.on_by_default) + everything = _by_kind(held) + # a new kind is taken from the hints first, then from the + # readings the word holds turned off + hinted = _by_kind((hints or {}).get(w, ())) + for k, p in everything.items(): + hinted.setdefault(k, p) + member = frozenset(f for f in fields + if f != "title_phrases" and w in full[f]) + new_heads = heads(full["title_phrases"], w) + moved = new_heads != heads(current["title_phrases"], w) + want = _target_readings(w, member, new_heads if moved else None, + on, hinted, kinds_known) + dropped = [k for k in on if k not in want] + gone_categories = {k for k in dropped if ri.CATEGORY[k] == k} + for k in dropped: + out.append(_Edit("remove", (w,), k, ())) + # every reading of a kind the add reaches, files disagreeing (a + # given name written short in one file, `ma`, and plain in + # another), so the delta takes each of them to `p` + after = {k: ps for k, ps in _props_by_kind(held).items() + if k not in dropped and ri.CATEGORY[k] not in gone_categories} + for k, p in want.items(): + if k in after: + if k in on and all(q == p for q in after[k]): + continue + delta = {x: v for x, v in p.items() + if any(q.get(x) != v for q in after[k])} + delta |= {x: None for q in after[k] for x in q if x not in p} + else: + delta = p + out.append(_make_edit("add", (w,), k, delta)) + return tuple(out) + + +def _fields_of(sets: _LexiconSets | Mapping[str, frozenset[str]], + ) -> dict[str, frozenset[str]]: + get = (sets.__getitem__ if isinstance(sets, Mapping) + else functools.partial(getattr, sets)) + return {f: get(f) for f in _SET_FIELDS + _KIND_FIELDS} + + +def _rendered(edits: Iterable[_Edit]) -> list[str]: + """Edits as the calls that make them, consecutive ones of one kind + and properties joined into one call.""" + calls: list[tuple[str, list[str], str | None, _Props]] = [] + for e in edits: + if calls and calls[-1][0] == e.op and calls[-1][2:] == (e.kind, + e.props): + calls[-1][1].extend(e.words) + else: + calls.append((e.op, list(e.words), e.kind, e.props)) + out = [] + for op, words, kind, props in calls: + args = [repr(w) for w in words] + if kind is not None: + args.append(f"kind={kind!r}") + args += [f"{k}={_shown_value(kind, k, v)}" for k, v in props] + out.append(f".{op}({', '.join(args)})") + return out +def _shown_value(kind: str | None, prop: str, value: object) -> str: + """A logged property value as a caller writes it: a removed one as + its unwritten value where it has one (`may_be_initial=False`).""" + if value is None and kind is not None and prop in ( + "acronym", "addresses", "after_given", "may_be_initial"): + value = _unwritten(kind, prop) + if isinstance(value, tuple) and len(value) > _SHOWN_WORDS: + # bounded: a phrase taken out of a kind's reach lists every + # other title word the kind reached (`chief justice`) + return (repr(list(value[:_SHOWN_WORDS]))[:-1] + + f", ... {len(value) - _SHOWN_WORDS} more]") + return repr(list(value) if isinstance(value, tuple) else value) + + +#: The most words a refusal shows of one property's list. +_SHOWN_WORDS = 6 + + +#: The most calls a refusal spells out before it counts the rest. +_SHOWN_CALLS = 8 + + +def _translation(lexicon: Lexicon, changes: Mapping[str, frozenset[str]], + prefix: str = "lexicon") -> str: + """What a 2.x set spelling asks of `lexicon`, as 3.0 calls: the + sets it would leave, translated word by word (_edits_toward, the + translation the 2.x pickle load uses).""" + target = {f: v for f, v in changes.items() if f in _SET_FIELDS} + current = _fields_of(lexicon._sets) + # every set compared, so a reading that also moves a set the call did + # not name (a given reading gating a credential) is no exact match; + # a word in both titles and titles_ambiguous compared as in titles + # alone, as a 2.x pickle is (the two spellings parse alike) + wanted = {f: target.get(f, current[f]) for f in _SET_FIELDS} + wanted["titles_ambiguous"] = wanted["titles_ambiguous"] - wanted["titles"] + wanted = _inert_dropped(wanted) + try: + edits = _edits_toward(lexicon._pool(), current, target) + result = _inert_dropped( + _built(lexicon._base, lexicon._edits + edits)[1]) + exact = all(result[f] == wanted[f] for f in _SET_FIELDS) + except (TypeError, ValueError): + edits, exact = (), False + if not edits and exact: + return ("the lexicon already reads those words so, an entry it " + "would add or remove being a marker without its word's " + "reading, which goes with that reading in 3.0; no call is " + "needed") + if not edits: + return ("no readings give those words exactly those sets (a word " + "holds one suffix kind, an acronym or a word, and a " + "property is read only beside its reading); " + "docs/migrate.rst's Lexicon table has every field") + calls = _rendered(edits) + shown = "".join(calls[:_SHOWN_CALLS]) + ( + f" ... ({len(calls) - _SHOWN_CALLS} calls more)" + if len(calls) > _SHOWN_CALLS else "") + if not exact: + return ("no readings compile to exactly that set; the nearest is " + f"{prefix}{shown}, and docs/migrate.rst's Lexicon table " + "has every field") + return f"write {prefix}{shown}" + + +def _refusal(where: str, lexicon: Lexicon, + removed: Mapping[str, object]) -> TypeError: + """The TypeError for a 2.x spelling, translated for the words it + names (decisions/vocabulary-model, 2026-11-02 #754, item 6).""" + unknown = [k for k in removed + if k not in _SET_FIELDS + _LIST_FIELDS + and k != "capitalization_exceptions"] + if unknown: + return TypeError(f"{where} got an unexpected keyword argument " + f"{unknown[0]!r}") + if "capitalization_exceptions" in removed: + return TypeError( + "capitalization_exceptions holds key->value pairs; use " + "dataclasses.replace(lexicon, capitalization_exceptions=(*lexicon." + "capitalization_exceptions, ('key', 'KEY'))) instead of " + f"{where}") + lists = [k for k in removed if k in _LIST_FIELDS] + if lists: + name = lists[0] + given = removed[name] + words = (sorted(given) if isinstance(given, Iterable) + and not isinstance(given, (str, bytes, Mapping)) + and all(isinstance(w, str) for w in given) and given + else ["word"]) + shown = "{" + ", ".join(map(repr, words)) + "}" + code = (f"dataclasses.replace(lexicon, {name}=lexicon.{name} - " + f"{shown})" if where.startswith("remove") + else f"lexicon | Lexicon({name}={shown})") + return TypeError( + f"{name} is a list, changed with a fragment since 3.0, not " + f"{where}: write {code}") + return TypeError( + f"{where} takes no {', '.join(removed)} since 3.0: a Lexicon is " + f"configured by reading, add(*words, kind=..., **properties) and " + f"remove(*words, kind=...); for these words, " + + _translation(lexicon, { + f: cast("frozenset[str]", v) for f, v in removed.items()}, + # a bare constructor call names the base it starts from + "Lexicon.empty()" if where.startswith("Lexicon()") + and lexicon == Lexicon.empty() else "lexicon")) + + +def _translated_2x(state: Mapping[str, object]) -> dict[str, object]: + """A 2.x `Lexicon` pickle's state as 3.0 constructor arguments: the + sets translated onto the nearer of the two named bases, as the + repr counts it (the fewest entries differing), and a log replacing + each word whose memberships differ from that base's compile + (decisions/vocabulary-model, 2026-11-02 #754, item 5). Raises + ValueError where no readings compile the pickled sets back, naming + the fields; a word in both `titles` and `titles_ambiguous` is + compared as in `titles` alone, the two spellings parsing alike.""" + expected = set(_VOCAB_FIELDS) | {"capitalization_exceptions"} + missing_set = expected - set(state) - {"given_names"} + if missing_set or not set(state) <= expected: + missing = ", ".join(sorted(missing_set)) or "none" + unexpected = ", ".join(sorted(set(state) - expected)) or "none" + raise ValueError( + f"incompatible Lexicon pickle: missing fields: {missing}; " + f"unexpected fields: {unexpected}") + sets: dict[str, frozenset[str]] = {} + drifted = [] + for name in _VOCAB_FIELDS: + given = state.get(name, frozenset()) + try: + folded = _normset(cast("Iterable[str]", given), name, warn=False) + except (TypeError, ValueError) as exc: + raise ValueError(f"incompatible Lexicon pickle: {exc}") from None + lost = frozenset(cast("Iterable[str]", given)) - folded + if lost: + drifted.append(f"{name}: {', '.join(sorted(lost))}") + sets[name] = folded + if drifted: + raise ValueError( + "incompatible Lexicon pickle: entries are not normalized " + f"({'; '.join(drifted)}); this state was not written by " + "this version of nameparser") + sets["titles_ambiguous"] -= sets["titles"] + # an entry 2.x never read (an orphan marker) is no part of what the + # pickle parsed as, and no reading compiles one + wanted = _inert_dropped({f: sets[f] for f in _SET_FIELDS}) + + def distance(compiled: Mapping[str, frozenset[str]]) -> int: + compiled = _inert_dropped(compiled) + return sum(len(wanted[f] ^ compiled[f]) for f in _SET_FIELDS) + + base = min(("default", "empty"), + key=lambda b: distance(_compiled_for(b, ()))) + try: + edits = _edits_toward(_base_pool(base), _compiled_for(base, ()), + wanted) + got = _inert_dropped(_compiled_for(base, edits)) + except ValueError as exc: + raise ValueError(f"a 2.x Lexicon pickle that 3.0 readings cannot " + f"express: {exc}") from None + differ = [f for f in _SET_FIELDS if got[f] != wanted[f]] + if differ: + raise ValueError( + f"a 2.x Lexicon pickle that 3.0 readings cannot express: no " + f"readings compile its {', '.join(differ)} back; rebuild the " + f"lexicon in 3.0 (docs/migrate.rst's Lexicon table)") + return {"surnames": sets["surnames"], + "given_names": sets["given_names"], + "capitalization_exceptions": state["capitalization_exceptions"], + "_base": base, "_edits": edits} + + +# -- the public types ---------------------------------------------------- + @dataclass(frozen=True, slots=True) -class Lexicon(_HoldsSets): - """The vocabulary a parser matches against: which words are - titles, particles, suffixes, and so on. Immutable and hashable. +class LexiconReading: + """One reading a :class:`Lexicon` holds for a word, as + :meth:`Lexicon.readings_of` returns it: what the word may be read + as, in the vocabulary files' terms (decisions/vocabulary-model, + 2026-11-02 #754, item 2). + + ``kind`` is a vocabulary kind as the language files write it + (``"office"``, ``"credential"``, ``"preposition"``, ``"given"``), + finer than :class:`ReadingKind`, which is the parse's; ``category`` + its category (``"title"``, ``"suffix"``, ``"particle"``, + ``"conjunction"``, ``"bound_given"``, ``"relation"``, ``"name"``); + ``properties`` its behavioral properties as sorted ``(name, value)`` + pairs, spelled as the files spell them (``("addresses", "given")``), + a list value a sorted tuple; ``languages`` the BCP 47 tags of the + shipped files holding it, sorted, empty for a reading a caller + added. Whether the parse READ a word this way is the parse's + question, :attr:`Token.reading`: a title reading beside a ``given`` + one is a title only inside a title phrase (rules/H9).""" + + text: str + kind: str + category: str + properties: tuple[tuple[str, Any], ...] = () + languages: tuple[str, ...] = () + + def __post_init__(self) -> None: + ri = _index_module() + for name in ("text", "kind", "category"): + if not isinstance(getattr(self, name), str): + raise TypeError(f"LexiconReading.{name} must be a string, " + f"got {getattr(self, name)!r}") + if self.kind not in ri.CATEGORY: + raise ValueError(f"unknown reading kind {self.kind!r}; the " + f"kinds are {', '.join(ri.CATEGORY)}") + if self.category != ri.CATEGORY[self.kind]: + raise ValueError( + f"a {self.kind} reading's category is " + f"{ri.CATEGORY[self.kind]!r}, got {self.category!r}") + for name in ("properties", "languages"): + if not isinstance(getattr(self, name), tuple): + raise TypeError(f"LexiconReading.{name} must be a tuple, " + f"got {getattr(self, name)!r}") + + def __getstate__(self) -> dict[str, object]: + return {f.name: getattr(self, f.name) + for f in dataclasses.fields(self)} + + def __setstate__(self, state: dict[str, object]) -> None: + # the layout check the other frozen types share + # (_types._guarded_setstate), held here by layering + expected = {f.name for f in dataclasses.fields(LexiconReading)} + if set(state) != expected: + missing = ", ".join(sorted(expected - set(state))) or "none" + unexpected = ", ".join(sorted(set(state) - expected)) or "none" + raise ValueError( + f"incompatible LexiconReading pickle: missing fields: " + f"{missing}; unexpected fields: {unexpected}") + for name, value in state.items(): + object.__setattr__(self, name, value) + + +@dataclass(frozen=True, slots=True, init=False, repr=False) +class Lexicon: + """The vocabulary a parser matches against: which words may be read + as titles, particles, suffixes, and so on. Immutable and hashable. Start from :meth:`default` (the shipped vocabulary) or - :meth:`empty`, derive variants with :meth:`add` / :meth:`remove` / - ``|`` (union), and pass the result to ``Parser(lexicon=...)``. - Entries are normalized at construction -- lowercased, NFC-composed, + :meth:`empty`, configure it by READING -- :meth:`add` and + :meth:`remove` take words and a vocabulary kind as the language + files write it, ``add("leed", "ap", kind="credential")`` -- join + fragments with ``|``, and pass the result to ``Parser(lexicon=...)``. + Ask what a word may be read as with :meth:`readings_of`. + + Words are normalized as they are stored -- lowercased, NFC-composed, edge full stops stripped -- so matching is case-insensitive. - Vocabulary entries are single words -- a multi-word entry warns at - construction and can never match. Three fields are exempt, and they - differ in HOW they match: ``given_name_titles`` is looked up - against the run of words the parse has ALREADY read as titles -- - the whole run space-joined, or that run's last word -- - ``maiden_markers`` is matched by lookahead, longest first, over - words that need not be markers on their own (``"z domu"``), and - ``title_phrases`` holds two-word phrases, stored space-joined and - matched against two adjacent title words (``"chief justice"``). - Field docs below show examples, not full - contents; inspect any field's shipped vocabulary directly, e.g. - ``Lexicon.default().conjunctions``.""" - - #: Pre-nominal titles ("dr", "sir", "capt", ...). Full default - #: list: :data:`~nameparser.config.titles.TITLES`. - titles: frozenset[str] = frozenset() - #: Titles whose single name word, before or after them, reads as a - #: GIVEN name ("sir", "sister", ...) rather than a family name. Full - #: default list: :data:`~nameparser.config.titles.GIVEN_NAME_TITLES`. - given_name_titles: frozenset[str] = frozenset() - #: Post-nominal acronym suffixes, matched with or without periods - #: ("phd" matches "PhD" and "Ph.D."). Full default list: - #: :data:`~nameparser.config.suffixes.SUFFIX_ACRONYMS`. - suffix_acronyms: frozenset[str] = frozenset() - #: Post-nominal word suffixes ("jr", "esquire", "iii", ...). Full - #: default list: - #: :data:`~nameparser.config.suffixes.SUFFIX_WORDS`. - suffix_words: frozenset[str] = frozenset() - #: Subset of suffix_acronyms whose bare forms are also common - #: surnames ("ma", "do"). Written with periods one is a suffix - #: outright; written bare, capitals decide in a mixed-case name - #: ("Jack MA" is the credential, "Jack Ma" keeps his family name) - #: and otherwise the word counts as a suffix only where a name - #: word is left beside it ("John Smith ma"), rules/S6 and rules/S4. Full - #: default list: - #: :data:`~nameparser.config.suffixes.SUFFIX_ACRONYMS_AMBIGUOUS`. - suffix_acronyms_ambiguous: frozenset[str] = frozenset() - #: Family-name particles that chain onto the following piece - #: ("van", "de", "bin", ...). Full default list: - #: :data:`~nameparser.config.particles.PARTICLES`. - particles: frozenset[str] = frozenset() - #: Subset of particles that can also BE a given name ("Van - #: Johnson", but also "Van Buren"). Membership decides nothing - #: about chaining: the prefix chain skips the name's first piece - #: and never consults this set, so it leaves a leading particle a - #: piece of its own whether listed or not -- "de Mesnil" groups - #: into two pieces exactly as "van Gogh" does, and since #367 the - #: NAME in "Dr. de Mesnil" and "Dr. Van Johnson" groups into those - #: same two pieces behind the title piece, a title not being - #: part of the name it precedes. What membership - #: decides is what becomes of that piece afterwards. Under ANY - #: ``name_order`` a member records a particle-or-given ambiguity - #: and a non-member records none, and a non-member is additionally - #: folded back into the family name once roles exist WHERE IT OPENS - #: THE NAME, so the whole name is the surname ("de Mesnil" -- a bare "de", with nothing to - #: fold into, is left alone). That fold is order-independent too - #: (#359) -- not because the word could not be a given name, which - #: it can where position forces it (``parse("de")`` reports given - #: "de"), but because a particle OPENING the name has the rest of - #: the name to join forward to, and that evidence is positional - #: rather than vocabulary, so no declared order contradicts it. Which field a MEMBER's piece - #: lands in is ``name_order``'s question, not this set's. - #: No constant of its own -- the default derives - #: as particles minus - #: :data:`~nameparser.config.particles.NON_GIVEN_NAME_PARTICLES` - #: (which marks the opposite, never-given subset). - particles_ambiguous: frozenset[str] = frozenset() - #: Words or characters that join surrounding pieces into one - #: ("and", "&", "y", "и", ...). Full default list: - #: :data:`~nameparser.config.conjunctions.CONJUNCTIONS`. - conjunctions: frozenset[str] = frozenset() - #: Subset of conjunctions that read as an INITIAL rather than a - #: connective in a name written wholly in one case ("e": "jose e - #: maria santos" reads middle "e maria", where "juan garcia y - #: lopez" joins because "y" is not a member). Mixed-case input is - #: decided by the writing instead and never consults this set, and - #: neither does a caseless letter (Arabic و), which has no case to - #: read. A member additionally reports - #: :attr:`~nameparser.AmbiguityKind.CONJUNCTION_OR_INITIAL` where a - #: name word of its part follows it (with none, it connects - #: nothing); a non-member reports nothing, its reading not being in - #: doubt. - #: Full default list: - #: :data:`~nameparser.config.conjunctions.CONJUNCTIONS_AMBIGUOUS`. - #: Entries need not also be in ``conjunctions``; one that is not is - #: simply never consulted, so it is inert rather than an error. - conjunctions_ambiguous: frozenset[str] = frozenset() - #: Given-name prefixes that bind to the following word to form one - #: given name ("abdul" -> "Abdul Salam"); never standalone names. - #: Full default list: - #: :data:`~nameparser.config.bound_given_names.BOUND_GIVEN_NAMES`. - bound_given_names: frozenset[str] = frozenset() - #: Given names written short ("chas" for Charles, "wm" for William, - #: Filipino "ma" for María). Period-marked and opening the part of - #: the name that carries the given name, an entry reads as that - #: given name, where an unlisted abbreviation there reads as a - #: title by its shape (rules/H2): "Chas. Smith" reads given - #: "Chas.", "Xyz. Smith" title "Xyz.". An entry also in ``titles`` - #: is a title there, which is what listing it as one asks; anywhere - #: else an entry is read as it would be unlisted. Full default - #: list: - #: :data:`~nameparser.config.given_names.GIVEN_NAME_ABBREVIATIONS`. - given_name_abbreviations: frozenset[str] = frozenset() - #: Marker words introducing a birth surname, routed to the maiden - #: field ("née", "geb.", "rozená", ...). An entry may be a PHRASE - #: ("z domu"): entries are matched by lookahead, longest first, so - #: a phrase wins where it matches and a word entry that starts one - #: still matches on its own everywhere else. A phrase matches only - #: where its words stand together -- a bracketed clause or a comma - #: between them ends the run. Full default list: - #: :data:`~nameparser.config.maiden_markers.MAIDEN_MARKERS`. - maiden_markers: frozenset[str] = frozenset() + + Since 3.0 the sets a parse reads are compiled from the readings and + are internal; the 2.x set fields (``titles``, ``suffix_acronyms``, + ...) are refused wherever they were written, each refusal naming + the calls that configure the same words (docs/migrate.rst has the + table). Three fields stay, being outside the per-word model: + ``surnames``, ``given_names`` and ``capitalization_exceptions``. + + State is a base (the shipped vocabulary, none, or a pickled + release's), the ordered log of :meth:`add` and :meth:`remove` calls + applied after it, and the three fields; two lexicons are equal when + all of them are, the log compared call by call as written, so two + logs that read alike can still differ (decisions/vocabulary-model, + 2026-11-02 #754, item 3).""" + #: Family names for the unspaced-name segmentation stage (#271), #: matched longest-first against the start of the FIRST token #: written wholly in a script :attr:`Policy.segment_scripts @@ -755,91 +1590,9 @@ class Lexicon(_HoldsSets): #: carries the Korean census list #: (:data:`~nameparser.config.surnames.KOREAN_SURNAMES`); Chinese #: surnames ship in locales.ZH because Han segmentation is opt-in. + #: A list, outside the per-word model: add to it with a fragment, + #: ``lexicon | Lexicon(surnames={...})``. surnames: frozenset[str] = frozenset() - #: Honorifics, and credential, office and generational words, that - #: may be peeled off the END of a name token (#308, #707), matched longest-first: 田中さん splits into 田中 and さん - #: before the tokens are classified. Every entry must also be a - #: :attr:`suffix_words` entry -- the peeled tail is claimed by - #: suffix classification like any other post-nominal. Deliberately - #: NOT gated on :attr:`Policy.segment_scripts - #: ` (unlike :attr:`surnames` - #: above): 田中さん peels under the default policy, where HAN is in - #: no activation set, because a tail entry carries its own license - #: to fire. Entries are matched against the RAW token text, and - #: only within a name containing a non-ASCII character, so an ASCII - #: or mixed-case entry is at best conditionally active -- a ``"Jr"`` - #: entry is stored ``"jr"`` and matches only lowercase text. The - #: field is effectively CJK-scoped in 2.1, which is what the shipped - #: vocabulary is. Full default list: - #: :data:`~nameparser.config.suffixes.GLUED_HONORIFICS`. - honorific_tails: frozenset[str] = frozenset() - #: Salutations that read as a title where one ENDS the given part - #: after a family comma, written bare: "Doe, Jane Mrs" reads title - #: "Mrs" where an office or rank word there stays a middle name - #: ("Doe, John Pope"), rules/H8. A period-marked title word reads as - #: a title there whether listed here or not (rules/H5). Entries need - #: not also be in ``titles``; one that is not is never consulted. - #: Full default list: - #: :data:`~nameparser.config.titles.AFTER_GIVEN_TITLES`. - after_given_titles: frozenset[str] = frozenset() - #: Subset of suffix_words that are also borne as names ("fils", - #: "neto", and the credentials "kc", "pc" and "dl", #965). Read as a suffix only where the name has words to - #: spare, as an ambiguous acronym is: "Pedro Alves Neto" ends in a - #: suffix, "Ana Neto" keeps her family name, and either reading is - #: reported (rules/S11). Written in Title case in a name of mixed - #: case it leans to the name even with words to spare, and in lower - #: case it is the suffix after a comma that names the family, - #: unless it is also in ``agnomens``. Full default list: - #: :data:`~nameparser.config.suffixes.SUFFIX_WORDS_AMBIGUOUS`. - suffix_words_ambiguous: frozenset[str] = frozenset() - #: Subset of suffix_words_ambiguous whose writing says nothing: - #: an agnomen ("filho", "neto") is written in the case of the name - #: around it, so only the words to spare decide (rules/S11). An - #: entry outside suffix_words_ambiguous is never consulted. Full - #: default list: :data:`~nameparser.config.suffixes.AGNOMENS`. - agnomens: frozenset[str] = frozenset() - #: Titles that may also be a given name ("dean", "justice", - #: "major"), each read as a title only inside a title phrase: after a - #: modifier, a style, or a title it stands in ``title_phrases`` - #: behind; or before "of" or a title it stands in ``title_phrases`` - #: in front of. Elsewhere it is a name word, and where it opens the - #: name the parse reports ``title-or-name`` if a phrase of - #: ``title_phrases`` the parse can read holds it (rules/H9), so "Grace Smith", which no - #: phrase holds, reports nothing, and emptying ``title_phrases`` - #: silences the report: "Dean Smith" - #: and "Mr Major Smith" read given "Dean" and "Major", "Dean of - #: Students Smith" and "Major General Smith" a title. An entry also - #: in ``titles`` is a title wherever it stands, which is what listing - #: it there asks, so the default lists these words here alone. Full - #: default list: - #: :data:`~nameparser.config.titles.TITLES_AMBIGUOUS`. - titles_ambiguous: frozenset[str] = frozenset() - #: Titles that only qualify a head title ("vice", "wing", "grand"). - #: At the head of the name one is a title only with a head title - #: behind it in its run of titles, so "Wing Commander Lau" keeps the - #: title where "Wing Lau" reads "Wing" as the given name; after a - #: comma it is read as any title is (rules/H9), save one also in - #: ``titles_ambiguous``, which is read everywhere as at the head. One - #: standing before a word of ``titles_ambiguous`` puts that word in - #: a title phrase. Entries in neither ``titles`` nor - #: ``titles_ambiguous`` are never consulted. Full default - #: list: :data:`~nameparser.config.titles.TITLE_MODIFIERS`. - title_modifiers: frozenset[str] = frozenset() - #: Honorific styles ("hon", "excellency", "highness"): one standing - #: before a word of ``titles_ambiguous`` puts that word in a title - #: phrase, as in "The Hon. Justice Smith" (rules/H9). Entries in - #: neither ``titles`` nor ``titles_ambiguous`` are never consulted. - #: Full default list: - #: :data:`~nameparser.config.titles.TITLE_STYLES`. - title_styles: frozenset[str] = frozenset() - #: Two-word title phrases, a title and the title it qualifies - #: ("chief justice", "major general"), stored space-joined: a word - #: of ``titles_ambiguous`` on either side of one is in a title - #: phrase (rules/H9). A phrase is consulted only if it is two - #: words, each in ``titles`` or ``titles_ambiguous``. Full default - #: list: - #: :data:`~nameparser.config.titles.TITLE_PHRASES`. - title_phrases: frozenset[str] = frozenset() #: Given names read by two-family-name handling alone #: (:attr:`Policy.two_family_names #: `, rules/O6): the word that @@ -851,10 +1604,7 @@ class Lexicon(_HoldsSets): #: by default, so the switch reads as without it; a caller passes #: a list of its own, and ``locales.ES`` carries a Spanish list, with #: the switch on. - #: Added in 3.0, so a Lexicon pickled by 2.x loads with it empty, - #: which is what an older pickle means: no pack filled it then. - given_names: frozenset[str] = field( - default=frozenset(), metadata={_ADDED_IN: "3.0.0"}) + given_names: frozenset[str] = frozenset() #: Lowercase word -> case mask used by capitalized(): the word's #: own letters and digits, each in the case it takes ("phd" -> #: "PhD"), laid over the word as written, so the one entry @@ -863,90 +1613,63 @@ class Lexicon(_HoldsSets): #: and is never written into the word -- repair keeps the writer's #: own punctuation. A value that does not spell the key's letters #: and digits raises ValueError. Pair-valued: change it with - #: dataclasses.replace(), not add()/remove(); read it as a mapping - #: via capitalization_exceptions_map. Full default mapping: + #: dataclasses.replace(); read it as a mapping via + #: capitalization_exceptions_map. Full default mapping: #: :data:`~nameparser.config.capitalization.CAPITALIZATION_EXCEPTIONS`. # Canonical storage: sorted tuple of (key, value) pairs. The # constructor tolerates any Mapping (or pair iterable) at runtime and # canonicalizes here; this closes the caller-aliasing hole and keeps # Lexicon hashable. Read via capitalization_exceptions_map. capitalization_exceptions: tuple[tuple[str, str], ...] = () + # The base and the log: constructor parameters under underscore + # names, so that dataclasses.replace and copy.replace carry them, + # and no public API (decisions/vocabulary-model, 2026-11-02 #754, + # item 3). 4i-c adds the choice beside them. + _base: Any = "empty" + _edits: tuple[_Edit, ...] = () _cap_map: Mapping[str, str] = field( - init=False, repr=False, compare=False, hash=False, - default_factory=lambda: MappingProxyType({})) - - def __post_init__(self) -> None: - for name in _VOCAB_FIELDS: - object.__setattr__(self, name, _normset(getattr(self, name), name)) - canonical = _normpairs(self.capitalization_exceptions) - object.__setattr__(self, "capitalization_exceptions", canonical) - object.__setattr__(self, "_cap_map", MappingProxyType(dict(canonical))) - for marker, base, why in _SUBSET_FIELDS: - orphans = getattr(self, marker) - getattr(self, base) - if orphans: - raise ValueError( - f"{marker} marks a subset of {base}; " - f"not in {base}: {', '.join(sorted(orphans))} " - f"({why}). Add them to {base}, or drop them from " - f"{marker}" - ) - # NOT validated: given_name_titles against titles. The lookup - # key is the space-joined run of Role.TITLE tokens, built by the - # parse rather than drawn from this vocabulary, and a - # conjunction inside a run is itself tagged Role.TITLE -- so - # "sir and dame" is a matchable key whose middle word is in - # conjunctions, not titles. A whole-entry check rejected - # multi-word entries; a per-word check rejected that one. An - # unreachable entry here is inert: nothing consults it, nothing - # misparses. That is the cheap failure, and guarding it proved - # the expensive one -- three working configurations broken - # across two attempts. Do not add a third. - # - # The v2 form of the vocabulary build's check that a bound given - # name which is also a particle may be given (tools/vocab/build.py, - # check 9). The build guards the shipped data; this guards - # vocabulary a caller supplies. - contradictory = ( - self.bound_given_names & self.particles) - self.particles_ambiguous - if contradictory: - raise ValueError( - f"bound_given_names entries that are also particles must be " - f"in particles_ambiguous; not in particles_ambiguous: " - f"{', '.join(sorted(contradictory))}. A particle that never " - f"starts a given name cannot also bind one — add them to " - f"particles_ambiguous, or drop them from bound_given_names" - ) - # suffix_as_written ORs the acronym and word branches, so a word - # membership bypasses the period gate the ambiguous set exists - # to impose: listing 'ma' in suffix_words would make "Jack Ma" - # read as a suffix and lose the family name. - gate_bypassed = self.suffix_acronyms_ambiguous & self.suffix_words - if gate_bypassed: + init=False, repr=False, compare=False, hash=False) + _sets: _LexiconSets = field( + init=False, repr=False, compare=False, hash=False) + + def __init__( + self, *, + surnames: Iterable[str] = frozenset(), + given_names: Iterable[str] = frozenset(), + capitalization_exceptions: (Mapping[str, str] + | Iterable[tuple[str, str]]) = (), + _base: object = "empty", + _edits: Iterable[_Edit] = (), + **removed: object, + ) -> None: + put = functools.partial(object.__setattr__, self) + put("surnames", _normset(surnames, "surnames")) + put("given_names", _normset(given_names, "given_names")) + canonical = _normpairs(capitalization_exceptions) + put("capitalization_exceptions", canonical) + put("_cap_map", MappingProxyType(dict(canonical))) + put("_base", _check_base(_base)) + if isinstance(_edits, (str, bytes)) or not isinstance( + _edits, Iterable): + raise TypeError(f"_edits must be a tuple of edits, got " + f"{_edits!r}") + edits = tuple(_edits) + for e in edits: + if not isinstance(e, tuple) or len(e) != 4: + raise TypeError(f"an edit is (op, words, kind, props), " + f"got {e!r}") + canon = tuple(_make_edit(*e) for e in edits) + if canon != edits: raise ValueError( - f"an ambiguous suffix acronym must not also be a suffix " - f"word; in both: {', '.join(sorted(gate_bypassed))}. The " - f"word branch matches without periods, which bypasses the " - f"period gate suffix_acronyms_ambiguous exists to impose" - ) - - # the mirror of the check above: suffix_as_written ORs the - # unambiguous acronym branch too, so a gated suffix word listed - # there is claimed as a suffix with no count ('Ana Neto' would - # lose her family name, rules/S11) - word_gate_bypassed = self.suffix_words_ambiguous & ( - self.suffix_acronyms - self.suffix_acronyms_ambiguous) - if word_gate_bypassed: - raise ValueError( - f"an ambiguous suffix word must not also be an unambiguous " - f"suffix acronym; in both: " - f"{', '.join(sorted(word_gate_bypassed))}. The acronym " - f"branch matches it outright, which bypasses the count " - f"suffix_words_ambiguous exists to impose; drop it from " - f"suffix_acronyms or from suffix_words_ambiguous" - ) - # last, from the normalized fields: what the parse reads - object.__setattr__(self, "_sets", _LexiconSets( - **{name: getattr(self, name) for name in _VOCAB_FIELDS})) + "_edits holds an edit not in canonical form; build a " + "Lexicon's log with add() and remove()") + put("_edits", canon) + compiled = _compiled_for(self._base, canon) + put("_sets", _LexiconSets( + **compiled, surnames=self.surnames, given_names=self.given_names)) + if removed: + raise _refusal("Lexicon() (and dataclasses.replace)", self, + self._changed("set", removed)) # -- constructors ---------------------------------------------------- @@ -960,56 +1683,65 @@ def default(cls) -> Lexicon: # -- dunders ---------------------------------------------------------- - def _deltas_from(self, baseline: Lexicon) -> list[tuple[str, int, int]]: - deltas = [] - for name in _VOCAB_FIELDS + ("capitalization_exceptions",): - mine = set(getattr(self, name)) - theirs = set(getattr(baseline, name)) - added, removed = len(mine - theirs), len(theirs - mine) - if added or removed: - deltas.append((name, added, removed)) - return deltas + # hidden from a type checker, which would otherwise read any + # attribute as defined and stop flagging `lexicon.titles` itself + if not TYPE_CHECKING: + def __getattr__(self, name: str) -> object: + if name in _SET_FIELDS: + raise AttributeError( + f"Lexicon.{name} is internal since 3.0, compiled from " + f"the lexicon's readings: ask lexicon.readings_of(word) " + f"what a word may be read as, and a parse's " + f"Token.reading what it was read as") + raise AttributeError( + f"{type(self).__name__!r} object has no attribute {name!r}") def __repr__(self) -> str: - # Bounded: renders only which fields deviate from the nearer of - # the two named constructors and by how many entries -- never the - # entries themselves (design rule, see nameparser._types module - # docstring). Diffing empty()-built lexicons against default() - # would tell the wrong story ("default minus the entire - # default vocabulary"). + # Bounded: counts readings by kind against the nearer of the two + # named lexicons, never naming a word (the design rule in + # nameparser._types' module docstring) if self == Lexicon.default(): return "Lexicon(default)" if self == Lexicon.empty(): return "Lexicon(empty)" - candidates = [(label, self._deltas_from(baseline)) - for label, baseline in (("default", Lexicon.default()), - ("empty", Lexicon.empty()))] - label, deltas = min( - candidates, key=lambda c: sum(a + r for _, a, r in c[1])) - rendered = ", ".join( - name + ": " + "".join( - part for part, n in ((f"+{a}", a), (f"-{r}", r)) if n) - for name, a, r in deltas) - return f"Lexicon({label} + {rendered})" + candidates = [(label, self._deltas_from(other)) + for label, other in (("default", Lexicon.default()), + ("empty", Lexicon.empty()))] + label, (kinds, lists) = min( + candidates, + key=lambda c: sum(a + r for _, a, r in c[1][0] + c[1][1])) + + def counts(deltas: list[tuple[str, int, int]]) -> str: + return ", ".join( + name + ": " + "".join( + part for part, n in ((f"+{a}", a), (f"-{r}", r)) if n) + for name, a, r in deltas) + + rendered = "; ".join(filter(None, (counts(kinds), counts(lists)))) + return (f"Lexicon({label} + {rendered or f'edits: {len(self._edits)}'})") def __getstate__(self) -> dict[str, object]: - # _cap_map is a MappingProxyType, which pickle rejects; ship every - # other slot and rebuild the proxy from the canonical tuple on load. - return {f.name: getattr(self, f.name) - for f in dataclasses.fields(self) if f.name != "_cap_map"} + # the base as a snapshot of its rows, so a pickle parses as it + # did where it was written (decisions/vocabulary-model, + # 2026-11-02 #754, item 5) + base = self._base + if base == "default": + from nameparser.config._index import READINGS + base = READINGS + return {"surnames": self.surnames, "given_names": self.given_names, + "capitalization_exceptions": self.capitalization_exceptions, + "_base": base, "_edits": tuple(tuple(e) for e in self._edits)} def __setstate__(self, state: dict[str, object]) -> None: - # Fail at the unpickle site if the state comes from a different - # Lexicon field layout (version skew) -- silently loading it - # would defer the failure to some distant attribute read. - # Layout check and message kept in sync with - # _types._guarded_setstate by design (layering keeps this module - # import-free of _types): the one skew admitted is a field - # marked `_ADDED_IN` with a default, missing from a pickle - # written before it existed. Its default joins the state before - # anything is set, so the validation below reads it like any - # value the pickle held. - fields = [f for f in dataclasses.fields(Lexicon) if f.name != "_cap_map"] + # A 2.x state holds the compiled sets, translated onto a base + # and a log; a 3.0 one is checked for its layout as + # _types._guarded_setstate checks the other frozen types', the + # one skew admitted a field marked `_ADDED_IN` with a default + # (none yet), and then built through the constructor, which + # validates every value as for a caller's. + if "titles" in state: + state = _translated_2x(state) + fields = [f for f in dataclasses.fields(Lexicon) if f.init] expected = {f.name for f in fields} added = {f.name: f.default for f in fields if _ADDED_IN in f.metadata @@ -1020,43 +1752,26 @@ def __setstate__(self, state: dict[str, object]) -> None: unexpected = ", ".join(sorted(set(state) - expected)) or "none" raise ValueError( f"incompatible Lexicon pickle: missing fields: {missing}; " - f"unexpected fields: {unexpected}" - ) + f"unexpected fields: {unexpected}") defaulted = {name for name in added if name not in state} state = {**{name: added[name] for name in defaulted}, **state} - for name, value in state.items(): - object.__setattr__(self, name, value) - # Re-run construction validation rather than trusting the blob. - # The layout check above catches SHAPE skew; this catches - # CONTENT skew, which is likelier -- particles_ambiguous flipped - # meaning between v1's never-given set and v2's may-be-given set - # without changing its name, so an old pickle would otherwise - # load a semantically inverted lexicon in silence. Rebuilds - # _cap_map for free, which this used to do by hand. - self.__post_init__() - # __post_init__ also normalizes, and quietly accepting a - # rewritten value would make this a place where caller data - # changes without a word. A pickle this library wrote is always + Lexicon.__init__(self, **state) # type: ignore[misc, arg-type] + # The constructor normalizes, and quietly accepting a rewritten + # value would make this a place where caller data changes + # without a word. A pickle this library wrote is always # normalized already (_normalize converges), so a difference # here means the state came from somewhere else -- say so while # the offending entries can still be named. - # One pass per field, in _VOCAB_FIELDS declaration order. cast is - # safe: __post_init__ just ran _normset over each of these and - # raised for anything that was not an iterable of str. drifted = [] - for name in _VOCAB_FIELDS: + for name in _LIST_FIELDS: lost = frozenset( cast("Iterable[str]", state[name])) - getattr(self, name) if lost: drifted.append(f"{name}: {', '.join(sorted(lost))}") - # the pair field too, or ten fields raise and the eleventh is - # quietly re-canonicalized (keys normalized, sorted, deduped) - given_pairs = cast("Iterable[tuple[str, str]]", - state["capitalization_exceptions"]) - if tuple(given_pairs) != self.capitalization_exceptions: + given_pairs = state["capitalization_exceptions"] + if (isinstance(given_pairs, tuple) + and given_pairs != self.capitalization_exceptions): drifted.append("capitalization_exceptions") - # A drifted field the pickle lacked drifted in its CLASS default, - # which no pickle can fix: name the default, not the pickle. faulty = [d for d in drifted if d.split(":")[0] in defaulted] if faulty: raise ValueError( @@ -1064,26 +1779,34 @@ def __setstate__(self, state: dict[str, object]) -> None: f"({'; '.join(faulty)}): a field marked _ADDED_IN loads " "its default into every pickle written before it, so " "the default must be normalized entries; this is a " - "defect in nameparser, not in the pickle" - ) + "defect in nameparser, not in the pickle") if drifted: raise ValueError( "incompatible Lexicon pickle: entries are not normalized " f"({'; '.join(drifted)}); this state was not written by " - "this version of nameparser" - ) + "this version of nameparser") def __or__(self, other: Lexicon) -> Lexicon: + """The union of two lexicons: the lists unioned, the masks taken + right-biased, and the logs concatenated, this one's first, so a + later lexicon's edits apply after an earlier one's (decisions/ + vocabulary-model, 2026-11-02 #754, item 3); the base is the one + that is not empty, and two different ones raise.""" if not isinstance(other, Lexicon): return NotImplemented - updates: dict[str, object] = { - name: getattr(self, name) | getattr(other, name) - for name in _VOCAB_FIELDS - } - # right-biased on key conflicts, mirroring later-wins for scalars + if "empty" in (self._base, other._base) or self._base == other._base: + base = other._base if self._base == "empty" else self._base + else: + raise ValueError( + "cannot join two lexicons built on different bases (the " + "shipped vocabulary of two releases); rebuild one from " + "Lexicon.default() or Lexicon.empty()") merged = dict(self._cap_map) | dict(other._cap_map) - updates["capitalization_exceptions"] = tuple(sorted(merged.items())) - return dataclasses.replace(self, **updates) # type: ignore[arg-type] + return Lexicon( + surnames=self.surnames | other.surnames, + given_names=self.given_names | other.given_names, + capitalization_exceptions=tuple(sorted(merged.items())), + _base=base, _edits=self._edits + other._edits) # -- properties ------------------------------------------------------- @@ -1091,147 +1814,183 @@ def __or__(self, other: Lexicon) -> Lexicon: def capitalization_exceptions_map(self) -> Mapping[str, str]: return self._cap_map + # -- access ----------------------------------------------------------- + + def readings_of(self, word: str) -> tuple[LexiconReading, ...]: + """The readings this lexicon holds for `word`: one per kind and + properties, each with the shipped files that hold it, a reading + a caller added naming none; empty for a word it holds no reading + of. `word` is normalized as the lexicon stores it, a phrase + (``"z domu"``) whole. "Is X a title here?" is ``any(r.category + == "title" for r in lexicon.readings_of(x))``; whether a parse + read it as one is :attr:`Token.reading`'s.""" + _reject_buffer(word, "readings_of()", "characters") + if not isinstance(word, str): + raise TypeError(f"readings_of() takes a word, a string, got " + f"{word!r}") + text = _title_key(word.split()) + ri = _index_module() + grouped: dict[tuple[str, _Props], list[str]] = {} + for r in self._pool().get(text, ()): + if r.on_by_default: + grouped.setdefault((r.kind, _clean(r.props)), []).append( + r.tag) + order = list(ri.CATEGORY) + return tuple( + LexiconReading(text, kind, ri.CATEGORY[kind], props, + tuple(sorted(t for t in set(tags) if t))) + for (kind, props), tags in sorted( + grouped.items(), + key=lambda item: (order.index(item[0][0]), repr(item[0][1])))) + # -- editing ---------------------------------------------------------- - def _edit(self, op: str, entries: Mapping[str, Iterable[str]]) -> Lexicon: - updates: dict[str, frozenset[str]] = {} + # `kind` is required, and a type checker reads it so: the runtime + # default exists only to catch a 2.x call, which names no kind, and + # refuse it with its translation rather than with Python's own + # missing-argument error + if TYPE_CHECKING: + def add(self, *words: str, kind: str, + **properties: object) -> Lexicon: ... + else: + def add(self, *words: str, kind: str | None = None, + **properties: object) -> Lexicon: + """This lexicon with a reading of `kind` added to each word. + `kind` is required; the signature's default exists only so + that a 2.x call, which names no kind, is refused with its + translation. It is a vocabulary kind as the language files + write it, a key + of ``nameparser._reading_index.CATEGORY`` (``"title"``, + ``"office"``, ``"credential"``, ``"generation"``, + ``"preposition"``, ``"conjunction"``, ``"bound_given"``, + ``"birth_marker"``, ``"given"``, ``"family"``, ...), and + `properties` that kind's behavioral properties as the files + spell them (``addresses="given"``, ``after_given=True``, + ``attaches=[...]``, ``acronym=...``, ``attached="after"``, + ``may_be_initial=True``, ``same_as=...`` on a ``given`` + reading); a property given its unwritten value (``None``, + ``False``) is removed. Where the word holds a reading of `kind` + already, the add updates its properties in every file holding + it; otherwise it adds the caller's own reading, which names no + file. Warns where it makes a title reading that a ``given`` + reading keeps a title only inside a title phrase (rules/H9), + naming the ``remove`` that makes it a title everywhere.""" + old = {k: properties.pop(k) for k in list(properties) + if k in _SET_FIELDS + _LIST_FIELDS + or k == "capitalization_exceptions"} + if old: + raise _refusal("add()", self, self._changed("add", old)) + if kind is None: + raise TypeError("add() missing required keyword argument " + "'kind', e.g. add('leed', kind='credential')") + edit = _make_edit("add", words, kind, properties) + added = dataclasses.replace(self, _edits=self._edits + (edit,)) + ri = _index_module() + if ri.CATEGORY[edit.kind] == "title": # type: ignore[index] + before = self._pool() + for w in edit.words: + if (w in added._sets.titles_ambiguous + and not _updates(before.get(w, ()), edit)): + _warn_at_caller( + f"{w!r} holds a given reading, so it reads as a " + f"title only inside a title phrase (rules/H9) and " + f"this add reads no name differently elsewhere; to " + f"read it as a title everywhere, write " + f".remove({w!r}, kind='given')") + return added + + def remove(self, *words: str, kind: str | None = None, + **removed: object) -> Lexicon: + """This lexicon without each word's readings of `kind`: a kind, + or a category, matching every kind of it (``kind="title"`` takes + out ``dean``'s ``office`` reading, ``kind="name"`` its ``given`` + and ``family`` ones), or None, taking out every reading, so the + word leaves the vocabulary. Removing a reading the word does not + hold changes nothing now, and is kept: a later choice of + languages may turn that reading on.""" + if removed: + raise _refusal("remove()", self, self._changed("remove", removed)) + edit = _make_edit("remove", words, kind, ()) + return dataclasses.replace(self, _edits=self._edits + (edit,)) + + # -- private helpers -------------------------------------------------- + + def _pool(self) -> Pool: + if not self._edits: + return _base_pool(self._base) + return _built(self._base, self._edits)[0] + + def _changed(self, op: str, + entries: Mapping[str, object]) -> dict[str, object]: + """A 2.x call's set fields as the sets it would leave, `op` + being "add", "remove" or "set" (a constructor's or + `dataclasses.replace`'s), for its refusal to translate; a list + or the masks pass through, refused by name.""" + out: dict[str, object] = {} for name, words in entries.items(): - if name == "capitalization_exceptions": - raise TypeError( - "capitalization_exceptions holds key->value pairs; " - "use dataclasses.replace(lexicon, " - "capitalization_exceptions=(*lexicon." - "capitalization_exceptions, ('key', 'KEY'))) " - "instead of " - f"{op}()" - ) - if name not in _VOCAB_FIELDS: - raise TypeError( - f"unknown Lexicon field {name!r}; valid fields: " - f"{', '.join(_VOCAB_FIELDS)}" - ) - current: frozenset[str] = getattr(self, name) - # warn=False: this pass only computes the new set membership, - # never stores it directly. add() still warns exactly once, - # from the replaced instance's own __post_init__ -> _normset; - # remove() never reaches __post_init__ with the dead entry - # (it is subtracted out here), so it stays silent -- correct, - # since a removal stores nothing a warning could be about. - # That silence covers the entry BEING REMOVED only: a - # different multi-word entry still stored re-warns from the - # derived instance's __post_init__, since the warning is - # per-construction by design. - normalized = _normset(words, name, warn=False) - updates[name] = (current | normalized if op == "add" - else current - normalized) - # mypy's dataclasses.replace() typing checks a **dict's single - # value type against every field's type (it can't see which keys - # are actually present behind the unpack), so a homogeneous - # frozenset[str] dict is flagged against the tuple/Mapping-typed - # capitalization_exceptions/_cap_map fields even though this dict - # never contains those keys (guarded above). - return dataclasses.replace(self, **updates) # type: ignore[arg-type] - - def add(self, **entries: Iterable[str]) -> Lexicon: - return self._edit("add", entries) - - def remove(self, **entries: Iterable[str]) -> Lexicon: - return self._edit("remove", entries) - - -#: Lexicon field -> the compiled constant it is read from, for every -#: field the vocabulary compiles (nameparser/_reading_index.py's -#: Compiled); `particles_ambiguous` is derived (_vocab_fields below), -#: and `surnames` is KOREAN_SURNAMES, a segmentation list outside the -#: per-word model. tests/v2/test_parser.py holds that these fields, -#: read from the compile of no choice, are Lexicon.default()'s. -_COMPILED_FIELDS = { - "titles": "TITLES", - "given_name_titles": "GIVEN_NAME_TITLES", - "suffix_acronyms": "SUFFIX_ACRONYMS", - "suffix_words": "SUFFIX_WORDS", - "suffix_acronyms_ambiguous": "SUFFIX_ACRONYMS_AMBIGUOUS", - "particles": "PARTICLES", - "conjunctions": "CONJUNCTIONS", - "conjunctions_ambiguous": "CONJUNCTIONS_AMBIGUOUS", - "bound_given_names": "BOUND_GIVEN_NAMES", - "given_name_abbreviations": "GIVEN_NAME_ABBREVIATIONS", - "maiden_markers": "MAIDEN_MARKERS", - "honorific_tails": "GLUED_HONORIFICS", - "after_given_titles": "AFTER_GIVEN_TITLES", - "suffix_words_ambiguous": "SUFFIX_WORDS_AMBIGUOUS", - "agnomens": "AGNOMENS", - "titles_ambiguous": "TITLES_AMBIGUOUS", - "title_modifiers": "TITLE_MODIFIERS", - "title_styles": "TITLE_STYLES", - "title_phrases": "TITLE_PHRASES", -} - - -def _vocab_fields(compiled: Mapping[str, Any]) -> dict[str, frozenset[str]]: - """The set-valued Lexicon fields a compile of the vocabulary gives - (the constants of nameparser/config/_compiled.py, or a choice's - compile of the reading index), `surnames` aside.""" - out = {name: compiled[key] for name, key in _COMPILED_FIELDS.items()} - # FLIPPED from v1, as in _default_lexicon below - out["particles_ambiguous"] = (compiled["PARTICLES"] - - compiled["NON_GIVEN_NAME_PARTICLES"]) - return out + if name not in _SET_FIELDS: + out[name] = words + continue + try: + given = _normset(cast("Iterable[str]", words), name, + warn=False) + except (TypeError, ValueError): + given = frozenset() + current = getattr(self._sets, name) + out[name] = (given if op == "set" else current | given + if op == "add" else current - given) + return out + + def _deltas_from(self, other: Lexicon) -> tuple[ + list[tuple[str, int, int]], list[tuple[str, int, int]]]: + ri = _index_module() + mine, theirs = _keyed(self._pool()), _keyed(other._pool()) + kinds = [] + for kind in ri.CATEGORY: + a = sum(1 for key in mine - theirs if key[1] == kind) + r = sum(1 for key in theirs - mine if key[1] == kind) + if a or r: + kinds.append((kind, a, r)) + lists = [] + for name in _LIST_FIELDS + ("capitalization_exceptions",): + m, t = set(getattr(self, name)), set(getattr(other, name)) + if m != t: + lists.append((name, len(m - t), len(t - m))) + return kinds, lists + + +def _keyed(pool: Pool) -> set[tuple[str, str, _Props]]: + """The readings on in `pool`, each once, as the repr counts them: + by word, kind and properties, whatever file holds them.""" + return {(r.text, r.kind, _clean(r.props)) + for rs in pool.values() for r in rs if r.on_by_default} + + +def _updates(held: Iterable[Any], edit: _Edit) -> bool: + """Whether `edit` only updates the properties of a reading of its + kind the word holds: the title warning's exemption (decisions/ + vocabulary-model, 2026-11-01 #754, the title warning).""" + return any(r.kind == edit.kind + and _clean(_merged(r.props, edit.props)) != _clean(r.props) + for r in held) @functools.cache def _default_lexicon() -> Lexicon: # the config data modules, compiled from nameparser/vocab/, are the - # single source of the default vocabulary. - from nameparser.config.bound_given_names import BOUND_GIVEN_NAMES - from nameparser.config.given_names import GIVEN_NAME_ABBREVIATIONS + # single source of the default vocabulary: its sets are read from + # nameparser/config/_compiled.py, never compiled from the index here + # (_compiled_for), so Lexicon.default() loads no index. from nameparser.config.capitalization import CAPITALIZATION_EXCEPTIONS - from nameparser.config.conjunctions import ( - CONJUNCTIONS, CONJUNCTIONS_AMBIGUOUS) - from nameparser.config.maiden_markers import MAIDEN_MARKERS - from nameparser.config.particles import NON_GIVEN_NAME_PARTICLES, PARTICLES - from nameparser.config.suffixes import ( - AGNOMENS, GLUED_HONORIFICS, SUFFIX_ACRONYMS, SUFFIX_ACRONYMS_AMBIGUOUS, - SUFFIX_WORDS, SUFFIX_WORDS_AMBIGUOUS, - ) from nameparser.config.surnames import KOREAN_SURNAMES - from nameparser.config.titles import ( - AFTER_GIVEN_TITLES, GIVEN_NAME_TITLES, TITLE_MODIFIERS, TITLE_PHRASES, - TITLE_STYLES, TITLES, TITLES_AMBIGUOUS) - - # every vocabulary constant is a frozenset since #293, so each one - # feeds its strictly-typed frozenset[str] field as it stands -- and - # this cache reading them ONCE is the reason they are frozen. A - # mutated module set reached this Lexicon only when the edit landed - # before the first call; after it, the cache was already built and - # the same edit was invisible here, which the code doing the - # mutating could not see. + + # every vocabulary constant is a frozenset since #293, and this + # cache reading them ONCE is the reason they are frozen. return Lexicon( - titles=TITLES, - given_name_titles=GIVEN_NAME_TITLES, - suffix_acronyms=SUFFIX_ACRONYMS, - suffix_words=SUFFIX_WORDS, - suffix_acronyms_ambiguous=SUFFIX_ACRONYMS_AMBIGUOUS, - particles=PARTICLES, - # FLIPPED from v1: v1 marks the never-given subset; v2 marks the - # may-be-given subset (migration: complement translation). - particles_ambiguous=PARTICLES - NON_GIVEN_NAME_PARTICLES, - conjunctions=CONJUNCTIONS, - conjunctions_ambiguous=CONJUNCTIONS_AMBIGUOUS, - bound_given_names=BOUND_GIVEN_NAMES, - given_name_abbreviations=GIVEN_NAME_ABBREVIATIONS, - maiden_markers=MAIDEN_MARKERS, surnames=KOREAN_SURNAMES, - honorific_tails=GLUED_HONORIFICS, - after_given_titles=AFTER_GIVEN_TITLES, - suffix_words_ambiguous=SUFFIX_WORDS_AMBIGUOUS, - agnomens=AGNOMENS, - titles_ambiguous=TITLES_AMBIGUOUS, - title_modifiers=TITLE_MODIFIERS, - title_styles=TITLE_STYLES, - title_phrases=TITLE_PHRASES, # pass canonical pair-tuples so this strictly-typed call site never - # feeds a Mapping to the tuple-annotated field; __post_init__ - # still tolerates a Mapping at runtime for interactive use + # feeds a Mapping to the tuple-annotated field; __init__ still + # tolerates a Mapping at runtime for interactive use capitalization_exceptions=tuple(sorted(CAPITALIZATION_EXCEPTIONS.items())), + _base="default", ) diff --git a/nameparser/_parser.py b/nameparser/_parser.py index ea8bfd655..9d10af85a 100644 --- a/nameparser/_parser.py +++ b/nameparser/_parser.py @@ -11,7 +11,10 @@ import dataclasses import functools import warnings +from collections.abc import Mapping from dataclasses import dataclass, field +from types import MappingProxyType +from typing import Any from nameparser._lexicon import Lexicon from nameparser._locale import Locale @@ -319,25 +322,62 @@ def _choice_delta(chosen: frozenset[str], strict: bool = False) -> tuple[ return fields, masks +@functools.cache +def _choice_readings(chosen: frozenset[str], + strict: bool = False) -> Mapping[str, tuple[Any, ...]]: + """The readings the choice of the file tags `chosen` compiles, by + word: the kinds `_apply_choice` gives a word the choice changes.""" + from nameparser._reading_index import choose, index, strict_pool + by_text: dict[str, list[Any]] = {} + for r in choose(strict_pool(chosen) if strict else index(), chosen): + by_text.setdefault(r.text, []).append(r) + return MappingProxyType({t: tuple(rs) for t, rs in by_text.items()}) + + def _apply_choice(lexicon: Lexicon, languages: frozenset[str], strict: bool = False) -> Lexicon: """`lexicon` under the choice of `languages`, a delta against the default's compile: what the choice drops leaves `lexicon` whoever put it there, a caller included, what it turns on joins, and every - other entry stays as the caller had it.""" - from nameparser._lexicon import _SUBSET_FIELDS - fields, (unmasked, masks) = _choice_delta( - _reading_index_selected(languages, strict), strict) - updates: dict[str, object] = { - name: (getattr(lexicon._sets, name) - gone) | joined - for name, (gone, joined) in fields.items()} + other entry stays as the caller had it. Written as readings, the + edits that take each changed word's readings to its sets under the + choice, appended to the lexicon's log (#754, 4i-b), until 4i-c + records the choice on the `Lexicon` and retires this delta.""" + from nameparser._lexicon import ( + _MARKER_BASES, _compiled_for, _edits_toward, _fields_of) + chosen = _reading_index_selected(languages, strict) + fields, (unmasked, masks) = _choice_delta(chosen, strict) + current = _fields_of(lexicon._sets) + target = {name: (current[name] - gone) | joined + for name, (gone, joined) in fields.items()} # a word the choice drops from a base set leaves that set's markers - # too, a caller's own mark of it included: marking a word the - # vocabulary no longer holds is the orphan Lexicon refuses - for marker, base, _ in _SUBSET_FIELDS: + # too, a caller's own mark of it included: no reading compiles a + # marker without its base + for marker, base in _MARKER_BASES: if base in fields and fields[base][0]: - kept = updates.get(marker, getattr(lexicon._sets, marker)) - updates[marker] = kept - fields[base][0] # type: ignore[operator] + target[marker] = target.get(marker, current[marker]) - fields[base][0] + updates: dict[str, object] = {} + if target: + wanted = current | target + edits = _edits_toward(lexicon._pool(), current, wanted, + _choice_readings(chosen, strict)) + if edits: + updates["_edits"] = lexicon._edits + edits + # a reading the choice turns on can clash with one the + # caller wrote (`st`'s cs generation beside a caller's + # credential: one suffix kind per word), and no readings + # give the delta's sets back; refused, as construction + # refuses the same clash (decisions/vocabulary-model, + # 2026-11-02 #754, item 3), where 3b's delta raised the + # set guard it broke + got = _fields_of(_compiled_for(lexicon._base, + lexicon._edits + edits)) + clash = sorted({w for f in wanted for w in got[f] ^ wanted[f]}) + if clash: + raise ValueError( + f"the readings the choice turns on clash with the " + f"lexicon's own for {', '.join(map(repr, clash[:8]))}; " + f"remove the lexicon's reading of each before choosing") if unmasked or masks: cap = {k: v for k, v in lexicon.capitalization_exceptions if k not in unmasked} diff --git a/nameparser/_pipeline/_assemble.py b/nameparser/_pipeline/_assemble.py index b340107b2..1575ae7f8 100644 --- a/nameparser/_pipeline/_assemble.py +++ b/nameparser/_pipeline/_assemble.py @@ -540,8 +540,8 @@ def assemble(state: ParseState) -> ParsedName: field = (Role.MIDDLE if role is Role.FAMILY and folds_middle and FOLDED_TAG in tags and ATTACHED_TAG not in tags else role) - # rules/R9: "a particle the shipped vocabulary holds as a - # patronymic", read as a particle, opens a patronymic that + # rules/R9: "a particle the lexicon holds as a patronymic", + # read as a particle, opens a patronymic that # takes the words after it in its field; an ending rule a # policy turns on reads its own words (decisions/R9, #754) opens = attaches = False diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index 9a80b6bb2..c18a65f38 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -12,7 +12,9 @@ recorded for the later stages that read it (#289/#516) and left alone where an earlier stage already asked. Reads: every Lexicon vocabulary field except surnames and -honorific_tails, which script_segment consumes upstream; and, since +honorific_tails, which script_segment consumes upstream, and the kinds +of its suffix words and particles (Lexicon's generation_words, +credential_words and patronymic_particles, internal sets, 3.0); and, since 2.4, Policy.unlisted_dotted_suffixes and Policy.unlisted_caps_suffixes, which decide whether an UNLISTED dotted or all-caps token joins the ambiguous credential class by SHAPE (#516) -- the all-caps half here @@ -72,7 +74,7 @@ from nameparser._policy import CapsSuffixes from nameparser._pipeline._state import ( AGNOMEN_TAG, AMBIGUOUS_ACRONYM_TAG, AMBIGUOUS_WORD_TAG, CREDENTIAL_WORD_TAG, - GIVEN_ABBREVIATION_TAG, PATRONYMIC_TAG, SHAPE_ACRONYM_TAG, + GENERATION_WORD_TAG, GIVEN_ABBREVIATION_TAG, PATRONYMIC_TAG, SHAPE_ACRONYM_TAG, SPLIT_CREDENTIAL_TAG, SPLIT_MEMBER_TAG, TITLE_AMBIGUOUS_TAG, ParseState, Deferred, PendingAmbiguity, PendingReading, WorkToken, copy_with, @@ -81,7 +83,7 @@ from nameparser._pipeline._vocab import ( BUILTIN_GLUED_DELIMITERS, SPLIT_CHUNK, SPLIT_PAIR_OPENS, _INITIAL, caps_shape_candidate, composed_len, glued_cores, glued_stem, is_cased_paired_initials, - is_initial, is_one_case, is_patronymic_particle, listed_word_kind_tag, + is_initial, is_one_case, period_joined_vocab, split_pair, suffix_as_written, tag_marker_runs, title_readings, @@ -160,18 +162,19 @@ def _tags_for(token: WorkToken, n: str, state: ParseState, if n in lex._sets.agnomens: tags.add(AGNOMEN_TAG) # rules/R8: "The kind of a word listed as a suffix word is the - # shipped vocabulary's" -- a gated credential (`kc`, #965) reads - # as a credential, not the generation the class otherwise holds - elif listed_word_kind_tag(n) == CREDENTIAL_WORD_TAG: + # lexicon's" -- a gated credential (`kc`, #965) reads as a + # credential, not the generation the class otherwise holds + elif n in lex._sets.credential_words: tags.add(CREDENTIAL_WORD_TAG) elif n in lex._sets.suffix_words: tags.add("vocab:suffix-word") # rules/R8: "The kind of a word listed as a suffix word is the - # shipped vocabulary's" -- read here, from the word as this - # stage reads it (decisions/R8, #326) - kind_tag = listed_word_kind_tag(n) - if kind_tag is not None: - tags.add(kind_tag) + # lexicon's" -- read here, from the word as this stage reads it + # (decisions/R8, #326; #754, 4i-b) + if n in lex._sets.generation_words: + tags.add(GENERATION_WORD_TAG) + elif n in lex._sets.credential_words: + tags.add(CREDENTIAL_WORD_TAG) if n in lex._sets.suffix_acronyms_ambiguous: tags.add(AMBIGUOUS_ACRONYM_TAG) # rules/P7: "A one-letter word written with its period is an @@ -181,10 +184,9 @@ def _tags_for(token: WorkToken, n: str, state: ParseState, tags.add("particle") if n in lex._sets.particles_ambiguous: tags.add("vocab:particle-ambiguous") - # rules/R9: "a particle the shipped vocabulary holds as a - # patronymic" -- its kind, read here as a suffix word's is - # (decisions/R9, #754) - if is_patronymic_particle(n): + # rules/R9: "a particle the lexicon holds as a patronymic" -- its + # kind, read here as a suffix word's is (decisions/R9, #754) + if n in lex._sets.patronymic_particles: tags.add(PATRONYMIC_TAG) # rules/P8: "a single-letter connective reads as an initial # where the writing says so: written as a bare Latin capital in a diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index 5d6b70299..d7ca54446 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -43,9 +43,7 @@ builds one, and the tags they read are this module's vocabulary. Layering: imports _lexicon, _policy, and _pipeline._state (WorkToken, -comma_bucket, PendingReading -- read-only, never a whole ParseState), -and _reading_index for the one fact no compiled set records, the kind -of a listed suffix word (listed_word_kind_tag, 3.0's 4b). +comma_bucket, PendingReading -- read-only, never a whole ParseState). """ from __future__ import annotations @@ -60,7 +58,6 @@ ) from nameparser._policy import (CapsSuffixes, Policy, Script, _JA_SCRIPTS, _NO_INITIALS, _SCRIPT_RANGES, _script_matcher) -from nameparser._reading_index import holds_kind, suffix_kind_of from nameparser._pipeline._state import ( AGNOMEN_TAG, AMBIGUOUS_ACRONYM_TAG, AMBIGUOUS_WORD_TAG, COMMA_CHARS, CREDENTIAL_WORD_TAG, Deferred, GENERATION_WORD_TAG, @@ -186,7 +183,7 @@ def in_initialless_script(text: str) -> bool: # 'Xyz. Smith, Ed' (UNLISTED, H2-shaped) both read family 'Dr. # Smith'/'Xyz. Smith', given 'Ed' at the peel (since #701 the listed # 'Dr.' then peels into the title, rules/H6), because is_leading_title -# reads H2's shape and a bare `folded in lexicon.titles` lookup does +# reads H2's shape and a bare `folded in lexicon._sets.titles` lookup does # not -- but name_word_count's OWN count disagreed: 'Xyz.' counted as # a name word where 'Dr.' did not, so 'Xyz. Smith, Ed' alone flipped # the comma structure to a credential run @@ -221,7 +218,7 @@ def in_initialless_script(text: str) -> bool: def is_title_shaped(text: str, lexicon: Lexicon) -> bool: """Whether TEXT reads as a title by H2's shape -- the LISTED half being a separate lookup at each caller's own site - (is_title_piece/lexicon.titles), and the one vocabulary read here + (is_title_piece/lexicon._sets.titles), and the one vocabulary read here the given name written short, which outranks the shape (rules/H2, #789: 'Chas.' is no title). `name_word_count` and `title_readings` below are its callers; `_pieces.is_leading_title` asks the same question @@ -1090,7 +1087,7 @@ def caps_shape_candidate(text: str, lexicon: Lexicon, policy: Policy, asks of words classify left untagged). A hand-written roster is a second place to remember, and it had already gone wrong: it named eleven of the thirteen fields, leaving `surnames` and `honorific_tails` out, so - `Lexicon.default().add(surnames={"dupont"})` still read + a `Lexicon` whose `surnames` listed "dupont" still read `Jean Pierre DUPONT` as suffix `DUPONT` -- a caller listing a word as a SURNAME and getting it read as a credential is this switch's own worst failure, arriving through the one wordlist that says @@ -1507,7 +1504,7 @@ def surname_unit_tags(text: str, lexicon: Lexicon, # of itself, asked of the phrases `attaches` compiled rather than of a # property of its own def in_a_title_phrase(text: str, lexicon: Lexicon) -> bool: - """Whether a phrase of `lexicon.title_phrases` that classify can + """Whether a phrase of `lexicon._sets.title_phrases` that classify can read holds `text`.""" return _normalize(text) in _phrase_words( lexicon._sets.title_phrases, lexicon._sets.titles, lexicon._sets.titles_ambiguous) @@ -1655,7 +1652,7 @@ def name_word_count(texts: Sequence[str], lexicon: Lexicon, predicate, the same one is_wholly_suffix asks, so the two agree about what a suffix word is; the title half asks BOTH the listed lookup and H2's shape test (`is_title_shaped`) -- a bare - `lexicon.titles` lookup here read 'Xyz.' as a name word where the + `lexicon._sets.titles` lookup here read 'Xyz.' as a name word where the leading peel reads it as a title, and that divergence is recorded once, at `is_title_shaped` itself (mechanisms.md#ONE-PREDICATE-PER-QUESTION). @@ -2265,43 +2262,6 @@ def suffix_kind(text: str, else ReadingSource.VOCABULARY) -#: The tag naming a listed suffix word's kind in the shipped reading -#: index, for the kinds with a reading kind of their own; a courtesy, an -#: honorific or a status reads as a post-nominal word, and an agnomen -#: has its own set and tag. -_LISTED_WORD_KIND_TAGS = {"generation": GENERATION_WORD_TAG, - "credential": CREDENTIAL_WORD_TAG} - - -# rules/R8: "The kind of a word listed as a suffix word is the shipped -# vocabulary's, read whatever the reading's default" -- the kind the -# views partition the suffix by (decisions/R8, #326). A memo, C-level -# on a hit, so a suffix word read before costs no frame; bounded since -# its key is the caller's text. -@functools.lru_cache(maxsize=4096) -def listed_word_kind_tag(n: str) -> str | None: - """The kind tag classify writes beside "vocab:suffix-word" on a word - whose normalized form `n` is the one it read (a delimiter glued on - removed, rules/C10): the kind the compiled sets do not record (`jr` - and `esq` both compile into `suffix_words`), read from the shipped - files' reading index (3.0's 4b), so a word only a caller's - `Lexicon` lists has none.""" - return _LISTED_WORD_KIND_TAGS.get(suffix_kind_of(n) or "") - - -# rules/R9: "a particle the shipped vocabulary holds as a patronymic" -# -- read at classify, as a listed suffix word's kind is (decisions/R9, -# #754). A memo for the reason `listed_word_kind_tag` is one. -@functools.lru_cache(maxsize=4096) -def is_patronymic_particle(n: str) -> bool: - """Whether a file holds the normalized word `n` as a patronymic - particle (`bin`, `ibn`, `bat`), on by default or not: the kind the - compiled `particles` set does not record, read from the shipped - reading index (3.0's 4c), so a word only a caller's `Lexicon` - lists is a plain particle.""" - return holds_kind(n, "patronymic") - - def name_reading(token: WorkToken, role: Role | Deferred) -> PendingReading: """The name reading of a word a decision read as something else, landing in `role`: an initial where the word is one letter or diff --git a/nameparser/_reading_index.py b/nameparser/_reading_index.py index 7e71c5530..60f70617c 100644 --- a/nameparser/_reading_index.py +++ b/nameparser/_reading_index.py @@ -445,8 +445,10 @@ def suffix_kind_of(text: str) -> str | None: whose category is ``suffix``), on by default or not; None for a word no file holds as a suffix. A word has one suffix kind in every file (the build's check 4), so the first row says it. `text` is - normalized as :func:`readings_of` takes it. What the parse reads a - listed suffix word's kind by (`_vocab.suffix_kind`, 3.0's 4b).""" + normalized as :func:`readings_of` takes it. The shipped files' + answer only: since 3.0's 4i-b the parse reads a suffix word's kind + from its own `Lexicon`, which compiles it (GENERATION_WORDS, + CREDENTIAL_WORDS), and this agrees with the default's.""" for _tag, kind, _pairs in _by_text().get(text, ()): if CATEGORY[kind] == "suffix": return kind @@ -456,9 +458,9 @@ def suffix_kind_of(text: str) -> str | None: def holds_kind(text: str, kind: str) -> bool: """Whether a file holds `text` in a reading of `kind` (a key of :data:`CATEGORY`), on by default or not. `text` is normalized as - :func:`readings_of` takes it. What the parse reads a listed - particle's patronymic kind by (`_vocab.is_patronymic_particle`, - 3.0's 4c).""" + :func:`readings_of` takes it. The shipped files' answer only: since + 3.0's 4i-b the parse reads a particle's patronymic kind from its + own `Lexicon` (PATRONYMIC_PARTICLES).""" return any(k == kind for _tag, k, _pairs in _by_text().get(text, ())) @@ -493,6 +495,9 @@ class Compiled(TypedDict): SUFFIX_ACRONYMS_AMBIGUOUS: frozenset[str] SUFFIX_ACRONYMS: frozenset[str] CAPITALIZATION_EXCEPTIONS: dict[str, str] + GENERATION_WORDS: frozenset[str] + CREDENTIAL_WORDS: frozenset[str] + PATRONYMIC_PARTICLES: frozenset[str] def compile_readings(readings: Iterable[ReadingProperties]) -> Compiled: @@ -529,6 +534,9 @@ def compile_readings(readings: Iterable[ReadingProperties]) -> Compiled: agnomens: set[str] = set() ambiguous: set[str] = set() acronyms: set[str] = set() + generations: set[str] = set() + credential_words: set[str] = set() + patronymics: set[str] = set() cased: dict[str, str] = {} for r, t in texts: p = r.props @@ -555,8 +563,14 @@ def compile_readings(readings: Iterable[ReadingProperties]) -> Compiled: ambiguous_words.add(t) if r.kind == "agnomen": agnomens.add(t) + elif r.kind == "generation": + generations.add(t) + elif r.kind == "credential": + credential_words.add(t) elif r.category == "particle": particles.add(t) + if r.kind == "patronymic": + patronymics.add(t) if t not in given: never_given.add(t) elif r.category == "conjunction": @@ -613,6 +627,14 @@ def compile_readings(readings: Iterable[ReadingProperties]) -> Compiled: SUFFIX_ACRONYMS_AMBIGUOUS=frozenset(ambiguous), SUFFIX_ACRONYMS=frozenset(acronyms), CAPITALIZATION_EXCEPTIONS=dict(sorted(cased.items())), + # the kinds the compiled sets above do not record, which the + # generation, credentials and patronymic views read (rules/R8, + # rules/R9): a suffix WORD's kind, `jr` and `esq` both being in + # SUFFIX_WORDS, and a particle's. Internal, re-exported by no + # config module (#754, 4i-b) + GENERATION_WORDS=frozenset(generations), + CREDENTIAL_WORDS=frozenset(credential_words), + PATRONYMIC_PARTICLES=frozenset(patronymics), ) # One form per post-nominal: the word test strips only edge periods # and the acronym test strips all of them, so a word in both is diff --git a/nameparser/_render.py b/nameparser/_render.py index 7693b3bf0..4383cdbf3 100644 --- a/nameparser/_render.py +++ b/nameparser/_render.py @@ -536,10 +536,10 @@ def _cap_word(word: str, role: Role, tags: frozenset[str], # `conjunctions` are disjoint in the default vocabulary and in # every locale pack, so no shipped conjunction can sit in an # all-particle part and carry the mark. That is a property of the - # shipped DATA, not an invariant -- both sets are public, - # configurable API, and a caller's Lexicon may put one word in - # both, the way _pipeline/_post_rules.py's arms allow for. Measured: - # under `Lexicon.default().add(particles={'y'})`, `anh y van` has + # shipped DATA, not an invariant -- a caller's Lexicon may give one + # word both readings, the way _pipeline/_post_rules.py's arms allow + # for. Measured: under `Lexicon.default().add('y', kind='particle')`, + # `anh y van` has # an all-particle family whose `y` carries both tags and the mark, # and gives 'Anh y Van'; gating this conjunct too would give # 'Anh Y Van'. That is pinned by test_repair_keeps_a_conjunction_ diff --git a/nameparser/config/_compiled.py b/nameparser/config/_compiled.py index 6318f634a..ffe64e786 100644 --- a/nameparser/config/_compiled.py +++ b/nameparser/config/_compiled.py @@ -4,7 +4,8 @@ uv run python tools/vocab/build.py The public names are re-exported, with their documentation, by the -nameparser.config modules (titles, particles, suffixes, ...). +nameparser.config modules (titles, particles, suffixes, ...); the last +three, the kinds those sets do not record, are read by Lexicon alone. """ @@ -2701,3 +2702,152 @@ 'ua': 'Ua', 'ó': 'Ó', } + +GENERATION_WORDS: frozenset[str] = frozenset({ + '2', + 'eldri', + 'fils', + 'i', + 'ii', + 'iii', + 'iv', + 'jaunesnioji', + 'jaunesnysis', + 'jaunākais', + 'jaunākā', + 'jnr', + 'jr', + 'junior', + 'júnior', + 'mladšia', + 'mladší', + 'mlajša', + 'mlajši', + 'mlađa', + 'mlađi', + 'nuorempi', + 'snr', + 'sr', + 'starejša', + 'starejši', + 'starija', + 'stariji', + 'staršia', + 'starší', + 'v', + 'vanhempi', + 'vecākais', + 'vecākā', + 'vyresnioji', + 'vyresnysis', + 'yngri', + 'ƙarami', + 'σίνιορ', + 'τζούνιορ', + 'малодшая', + 'малодшы', + 'мл', + 'млади', + 'младша', + 'младшая', + 'младши', + 'младший', + 'млађа', + 'млађи', + 'молодша', + 'молодший', + 'помладиот', + 'постариот', + 'стари', + 'старија', + 'старији', + 'старша', + 'старшая', + 'старши', + 'старший', + 'старэйшая', + 'старэйшы', + 'կրտսեր', + "ג'וניור", + 'ג׳וניור', + 'ג’וניור', + 'האב', + 'הבן', + 'الأب', + 'الإبن', + 'الابن', + 'جونئیر', + 'جونيئر', + 'جونيور', + 'جونیئر', + 'جونیور', + 'سينيور', + 'سینئر', + 'سینیئر', + 'سینیور', + 'जुनियर', + 'जूनियर', + 'ज्युनियर', + 'ज्यूनियर', + 'सीनियर', + 'কনিষ্ঠ', + 'জুনিয়র', + 'জুনিয়ৰ', + 'জ্যেষ্ঠ', + 'সিনিয়র', + 'ਜੂਨੀਅਰ', + 'ਸੀਨੀਅਰ', + 'જુનિયર', + 'જૂનિયર', + 'સિનિયર', + 'ଜୁନିଅର', + 'ଜୁନିୟର୍', + 'இளையவர்', + 'ஜூனியர்', + 'மூத்தவர்', + 'జూనియర్', + 'సీనియర్', + 'ಜೂನಿಯರ್', + 'ജൂനിയർ', + 'സീനിയർ', + 'ජූනියර්', + 'จูเนียร์', + 'ซีเนียร์', + 'ຈູເນຍ', + 'ဂျူနီယာ', + 'ၵျူႇၼီႇယႃႇ', + 'უმცროსი', + 'უფროსი', + 'ጁኒየር', + 'ጁንየር', + 'ជូនៀរ', + 'ジュニア', + '二世', + '시니어', + '주니어', +}) + +CREDENTIAL_WORDS: frozenset[str] = frozenset({ + 'dl', + 'kc', + 'pc', + '医学博士', + '博士', + '醫學博士', + '박사', +}) + +PATRONYMIC_PARTICLES: frozenset[str] = frozenset({ + 'bar', + 'bat', + 'bin', + 'bint', + 'binte', + 'binti', + 'ibn', + 'בן', + 'בת', + 'ابن', + 'بن', + 'بنت', +}) diff --git a/nameparser/config/capitalization.py b/nameparser/config/capitalization.py index 1d70c0a11..0fd4e7bc8 100644 --- a/nameparser/config/capitalization.py +++ b/nameparser/config/capitalization.py @@ -36,8 +36,8 @@ (``d.clinpsy.``) it is a suffix by shape alone, with no vocabulary entry needed to read it as one, and repairs in all capitals the same as a listed acronym does (``D.CLINPSY.``). Either way, a caller who -wants a specific spelling needs a ``suffix_acronyms`` entry -(``Lexicon.add(suffix_acronyms={...})``) or a mask of +wants a specific spelling needs a credential reading +(``Lexicon.add(..., kind="credential")``) or a mask of its own. The map also carries the Irish patronymic particles ``ó``, ``ní`` and ``ua`` (#604): case repair writes a particle in lowercase (``de la Vega``), Irish writes these capitalized (``Liam Ó Murchú``), diff --git a/nameparser/config/surnames.py b/nameparser/config/surnames.py index 579a301fc..440fcf639 100644 --- a/nameparser/config/surnames.py +++ b/nameparser/config/surnames.py @@ -38,7 +38,7 @@ single-syllable surnames (Kim/Lee/Park alone cover ~45% of the population) plus the two-syllable surnames in current use. A coverage floor, not the complete census roster: extend with -``Lexicon.default().add(surnames={...})``. +``Lexicon.default() | Lexicon(surnames={...})``. """ diff --git a/nameparser/vocab/README.md b/nameparser/vocab/README.md index a984145d0..15fac7533 100644 --- a/nameparser/vocab/README.md +++ b/nameparser/vocab/README.md @@ -111,7 +111,7 @@ A reading's key is its **kind**. The table lists the kinds in their canonical or | `kinship` | `title` | `TITLES` | | `modifier` | `title` | `TITLES`, `TITLE_MODIFIERS` | | `suffix` | `suffix` | `SUFFIX_WORDS` | -| `generation` | `suffix` | `SUFFIX_WORDS` | +| `generation` | `suffix` | `SUFFIX_WORDS`, `GENERATION_WORDS` | | `agnomen` | `suffix` | `SUFFIX_WORDS`, `SUFFIX_WORDS_AMBIGUOUS`, `AGNOMENS` | | `credential` | `suffix` | `SUFFIX_ACRONYMS` | | `honorific` | `suffix` | `SUFFIX_WORDS` | @@ -121,7 +121,7 @@ A reading's key is its **kind**. The table lists the kinds in their canonical or | `particle` | `particle` | `PARTICLES` | | `preposition` | `particle` | `PARTICLES` | | `article` | `particle` | `PARTICLES` | -| `patronymic` | `particle` | `PARTICLES` | +| `patronymic` | `particle` | `PARTICLES`, `PATRONYMIC_PARTICLES` | | `teknonymic` | `particle` | `PARTICLES` | | `clan` | `particle` | `PARTICLES` | | `rank` | `particle` | `PARTICLES` | @@ -133,7 +133,7 @@ A reading's key is its **kind**. The table lists the kinds in their canonical or | `given` | `name` | nothing | | `family` | `name` | nothing | -A suffix reading goes to `SUFFIX_ACRONYMS` when its `acronym` property is true and to `SUFFIX_WORDS` otherwise; the column shows where each kind goes with `acronym` unwritten. A word has at most one suffix kind (check 4). A `given` reading carrying `same_as` goes to `GIVEN_NAME_ABBREVIATIONS`, a given name written short (see [Name readings](#name-readings)). +A suffix reading goes to `SUFFIX_ACRONYMS` when its `acronym` property is true and to `SUFFIX_WORDS` otherwise; the column shows where each kind goes with `acronym` unwritten. `GENERATION_WORDS`, `CREDENTIAL_WORDS` (a credential written as a word) and `PATRONYMIC_PARTICLES` record the kinds the other sets do not, which the `generation`, `credentials` and `patronymic` views read ([rules/R8](../../docs/design/rules/R8-generation-and-credentials-views.md), [rules/R9](../../docs/design/rules/R9-patronymic-view.md)); `Lexicon` reads them, and no `nameparser.config` module re-exports them. A word has at most one suffix kind (check 4). A `given` reading carrying `same_as` goes to `GIVEN_NAME_ABBREVIATIONS`, a given name written short (see [Name readings](#name-readings)). `generation` is a word that numbers or ranks its bearer among relatives of the same name, written after the name: junior and senior and their translations, `fils`, the roman numerals the vocabulary lists. `agnomen` is the Brazilian and Portuguese kind of it (`filho`, `neto`, `sobrinho` and their feminines): it is written in the case of the name around it, so its writing settles nothing, and an agnomen reading needs a name reading on its word (check 11), which gates it by the count alone ([rules/S11](../../docs/design/rules/S11-suffix-word-that-is-also-a-name.md), a suffix word that is also a name). The column shows an agnomen with that name reading. @@ -178,11 +178,11 @@ A property is **behavioral** when it compiles into a set and so changes how name | `addresses` | behavioral | `title` | `"family"` | `"given"` adds the word to `GIVEN_NAME_TITLES`: `Sir John` reads `John` as the given name ([rules/H1](../../docs/design/rules/H1-title-and-one-name-word.md), a title and one name word) | | `attaches` | behavioral | `title` | none | `TITLE_PHRASES`: the title kinds, or the title words, this word stands before in a title phrase, a list (`chief = { attaches = ["office", "military"] }`, `sergeant = { attaches = ["major"] }`, a word where a kind would reach too far), compiled into the two-word phrases it makes with each title word of those kinds and each word named (`chief justice`, `sergeant major`); a title that may be a given name on either side of one is a title there ([rules/H9](../../docs/design/rules/H9-title-phrase.md), the title phrase) | | `after_given` | behavioral | `title` | false | `AFTER_GIVEN_TITLES`, on a `salutation` reading only: written bare at the end of the given part after a family comma, the word is the title, as in `Doe, Jane Mrs` ([rules/H8](../../docs/design/rules/H8-salutation-ending-the-given-part.md), a salutation ending the given part). A salutation borne as a given name or a surname is left out with an exclusion of the property, its counts beside it (`goodman`, `herr`), since a middle name may be either | -| `acronym` | behavioral | `suffix` | true for `credential`, false for every other kind | true: `SUFFIX_ACRONYMS`, matched with every period removed (`M.D.` is `md`); false: `SUFFIX_WORDS` | +| `acronym` | behavioral | `suffix` | true for `credential`, false for every other kind | true: `SUFFIX_ACRONYMS`, matched with every period removed (`M.D.` is `md`); false: `SUFFIX_WORDS`, and a credential's `CREDENTIAL_WORDS`, a credential written as a word | | `attached` | behavioral | `suffix` | absent | `GLUED_HONORIFICS`, on a word reading only: `{ side = "after" }` splits the word off the end of a name token, as in `田中さん` ([rules/W2](../../docs/design/rules/W2-glued-honorific-peel.md), the glued-honorific peel) | | `may_be_given` | stated | `particle` | none: required where the word has no `given` reading | `false` is required where the word has no `given` reading in any file, and that absence, not the property, is what puts the word in `NON_GIVEN_NAME_PARTICLES`: a particle opening the name starts the family name ([rules/P1](../../docs/design/rules/P1-lone-particle-fold.md), the lone particle fold). `true` is refused, naming the `given` reading to write instead (check 11) | | `may_be_initial` | behavioral | `conjunction` | false | `CONJUNCTIONS_AMBIGUOUS`, on a cased single letter only: read as an initial in a name written in one case, as in `JOHN E SMITH` ([rules/P8](../../docs/design/rules/P8-single-letter-connective-against-the-initial-it-spells.md), a single-letter connective against the initial it spells) | -| `default` | behavioral | `title`, `suffix`, `particle`, `conjunction`, `bound_given`, `relation`, `name` | true | `false` keeps the reading out of every compiled set, `SUFFIX_WORDS` included for a generation: recorded and checked as a reading, for a pack to turn on in phase 3, where its writing offers the default no cue (`i`, the German regnal ordinal, written `I.`, exactly as an initial is). An exclusion does not refuse it, and it cannot read two ways beside the word's live readings (checks 4 to 6); a name reading beside it marks nothing. On a `given` or `family` reading (3.0's phase 3a) it derives no marker in the default, so the word's other readings compile as they would without it, until a pack turns the name on. It does not count as the name reading an agnomen on by default needs (check 11), and a particle whose word's only `given` readings are off is refused, which side of check 11 it takes being left to the first pack that needs one; a bound given name that is also a particle is refused with it, before check 9 would ask for its `given` reading. `true` is refused: it is what every reading is unwritten | +| `default` | behavioral | `title`, `suffix`, `particle`, `conjunction`, `bound_given`, `relation`, `name` | true | `false` keeps the reading out of every compiled set, `SUFFIX_WORDS` and `GENERATION_WORDS` included for a generation: recorded and checked as a reading, for a pack to turn on in phase 3, where its writing offers the default no cue (`i`, the German regnal ordinal, written `I.`, exactly as an initial is). An exclusion does not refuse it, and it cannot read two ways beside the word's live readings (checks 4 to 6); a name reading beside it marks nothing. On a `given` or `family` reading (3.0's phase 3a) it derives no marker in the default, so the word's other readings compile as they would without it, until a pack turns the name on. It does not count as the name reading an agnomen on by default needs (check 11), and a particle whose word's only `given` readings are off is refused, which side of check 11 it takes being left to the first pack that needs one; a bound given name that is also a particle is refused with it, before check 9 would ask for its `given` reading. `true` is refused: it is what every reading is unwritten | | `cased` | behavioral | `title`, `suffix`, `particle`, `conjunction`, `bound_given`, `relation` | the word as written | `CAPITALIZATION_EXCEPTIONS`: the case `capitalized()` gives the word (`PhD`), a mask over its letters and digits, never a replacement (#459) | | `note` | descriptive | every category, a `name` reading included | | free text: why the reading is as it is | | `issue` | descriptive | every category but `name` | | an issue number, or a list of them | diff --git a/tests/test_bound_given_names.py b/tests/test_bound_given_names.py index 31db7ac64..8e96f8e56 100644 --- a/tests/test_bound_given_names.py +++ b/tests/test_bound_given_names.py @@ -1,5 +1,3 @@ -import dataclasses - from nameparser import Lexicon, Parser, parse from tests.base import ParsedNameTestBase @@ -141,8 +139,9 @@ def test_suffix_comma_abu_bakr_al_baghdadi(self) -> None: # --- opt-out --- def test_opt_out_via_clear(self) -> None: - """Clearing bound_given_names restores prior behavior.""" - lexicon = dataclasses.replace(Lexicon.default(), bound_given_names=frozenset()) + """Removing every bound_given reading restores prior behavior.""" + default = Lexicon.default() + lexicon = default.remove(*default._sets.bound_given_names, kind='bound_given') hn = Parser(lexicon=lexicon).parse("abdul salam ahmed salem") self.m(hn.given, "abdul", hn) self.m(hn.middle, "salam ahmed", hn) diff --git a/tests/test_constants.py b/tests/test_constants.py index dfeb1129a..07ae9236d 100644 --- a/tests/test_constants.py +++ b/tests/test_constants.py @@ -1,5 +1,3 @@ -import dataclasses - from nameparser import Lexicon, Parser, parse from tests.base import ParsedNameTestBase @@ -8,36 +6,38 @@ class LexiconCustomizationTests(ParsedNameTestBase): def test_add_title(self) -> None: - lexicon = Lexicon.default().add(titles={'te'}) - self.assertEqual(len(Lexicon.default().titles) + 1, len(lexicon.titles)) + lexicon = Lexicon.default().add('te', kind='title') + self.assertEqual(len(Lexicon.default()._sets.titles) + 1, len(lexicon._sets.titles)) hn = Parser(lexicon=lexicon).parse("Te Awanui-a-Rangi Black") self.m(hn.title, "Te", hn) self.m(hn.given, "Awanui-a-Rangi", hn) self.m(hn.family, "Black", hn) def test_remove_title(self) -> None: - lexicon = Lexicon.default().remove(titles={'hon'}) - self.assertEqual(len(Lexicon.default().titles) - 1, len(lexicon.titles)) + lexicon = Lexicon.default().remove('hon', kind='title') + self.assertEqual(len(Lexicon.default()._sets.titles) - 1, len(lexicon._sets.titles)) hn = Parser(lexicon=lexicon).parse("Hon Solo") self.m(hn.given, "Hon", hn) self.m(hn.family, "Solo", hn) def test_add_multiple_arguments(self) -> None: - lexicon = Lexicon.default().add(titles={'dean', 'Chemistry'}) + lexicon = Lexicon.default().add('chemistry', kind='title').remove('dean', kind='given') hn = Parser(lexicon=lexicon).parse("Assoc Dean of Chemistry Robert Johns") self.m(hn.title, "Assoc Dean of Chemistry", hn) self.m(hn.given, "Robert", hn) self.m(hn.family, "Johns", hn) def test_remove_multiple_arguments(self) -> None: - lexicon = Lexicon.default().remove(titles={'hon', 'ms'}) + lexicon = Lexicon.default().remove('hon', 'ms', kind='title') hn = Parser(lexicon=lexicon).parse("Ms Hon Solo") self.m(hn.given, "Ms", hn) self.m(hn.middle, "Hon", hn) self.m(hn.family, "Solo", hn) def test_chain_multiple_arguments(self) -> None: - lexicon = Lexicon.default().remove(titles={'hon', 'ms'}).add(titles={'dean'}) + lexicon = ( + Lexicon.default().remove('hon', 'ms', kind='title').remove('dean', kind='given') + ) hn = Parser(lexicon=lexicon).parse("Dean Ms Hon Solo") self.m(hn.title, "Dean", hn) self.m(hn.given, "Ms", hn) @@ -45,7 +45,8 @@ def test_chain_multiple_arguments(self) -> None: self.m(hn.family, "Solo", hn) def test_clear_removes_all_entries(self) -> None: - lexicon = dataclasses.replace(Lexicon.default(), titles=frozenset()) + default = Lexicon.default() + lexicon = default.remove(*default._sets.titles, kind='title') hn = Parser(lexicon=lexicon).parse("Ms Hon Solo") self.m(hn.given, "Ms", hn) self.m(hn.middle, "Hon", hn) diff --git a/tests/test_particles.py b/tests/test_particles.py index 13a2cd34f..749775fd8 100644 --- a/tests/test_particles.py +++ b/tests/test_particles.py @@ -270,7 +270,7 @@ def test_non_first_name_prefix_with_custom_title(self) -> None: # path. Title is consumed first, so the fold does # not fire and the surname is already correct; asserts we don't corrupt # the title case. - parser = Parser(lexicon=Lexicon.default().add(titles={'gunny'})) + parser = Parser(lexicon=Lexicon.default().add('gunny', kind='title')) hn = parser.parse("Gunny de Mesnil") self.m(hn.title, "Gunny", hn) self.m(hn.given, "", hn) diff --git a/tests/test_python_api.py b/tests/test_python_api.py index ddb7f5cdc..b705a715d 100644 --- a/tests/test_python_api.py +++ b/tests/test_python_api.py @@ -74,8 +74,10 @@ def test_matches_parsed_name_operand(self) -> None: def test_matches_parses_str_with_the_parsers_lexicon(self) -> None: # the custom title must not be a default one ('chancellor' is!), or # this passes without the custom parse path ever mattering - self.assertNotIn('zephyrmark', Lexicon.default().titles) - parser = Parser(lexicon=Lexicon.default().add(titles={'zephyrmark'})) + self.assertNotIn( + 'title', + {r.category for r in Lexicon.default().readings_of('zephyrmark')}) + parser = Parser(lexicon=Lexicon.default().add('zephyrmark', kind='title')) hn = parser.parse("Zephyrmark Jane Smith") # the str operand is parsed with the parser's lexicon, so the custom # title is recognized in the comma form; parsed with the default, diff --git a/tests/test_suffixes.py b/tests/test_suffixes.py index 206f0db9d..dd7b9ad89 100644 --- a/tests/test_suffixes.py +++ b/tests/test_suffixes.py @@ -1,5 +1,3 @@ -import dataclasses - import pytest from nameparser import Lexicon, ParsedName, Parser, Policy, parse @@ -451,21 +449,22 @@ def test_suffix_in_double_quotes(self) -> None: self.m(hn.nickname, "", hn) def test_suffix_acronyms_ambiguous_custom_entry_stays_nickname(self) -> None: - # A custom suffix_acronyms_ambiguous entry keeps a suffix_acronyms - # member classified as a nickname instead of a suffix, confirming - # the exception list -- not a hardcoded check -- drives the behavior. + # A custom credential with a family reading beside it (compiled + # into suffix_acronyms_ambiguous) stays a nickname instead of a + # suffix, confirming the readings -- not a hardcoded check -- drive + # the behavior. # Title case: capitals in a mixed-case name settle the member as # the credential (rules/S1, #810). - lexicon = Lexicon.default().add( - suffix_acronyms={'xyz'}, - suffix_acronyms_ambiguous={'xyz'}, + lexicon = ( + Lexicon.default().add('xyz', kind='credential').add('xyz', kind='family') ) hn = Parser(lexicon=lexicon).parse("Andrew Perkins (Xyz)") self.m(hn.nickname, "Xyz", hn) self.m(hn.suffix, "", hn) def test_suffix_acronyms_ambiguous_removal_routes_to_suffix(self) -> None: - # Removing 'jd' from a custom suffix_acronyms_ambiguous flips Jd + # Removing the name readings that make every listed credential + # ambiguous (suffix_acronyms_ambiguous, compiled) flips Jd # from nickname to suffix. Uses a trailing-position name (unlike the # JEFFREY (JD) BRICKEN regression guard in test_nicknames.py) so # parse_full_name's trailing-run suffix detection actually picks it @@ -473,8 +472,9 @@ def test_suffix_acronyms_ambiguous_removal_routes_to_suffix(self) -> None: # Title case, which leaves the member a nickname while it is listed # (rules/S1, #810). assert parse("Andrew Perkins (Jd)").nickname == "Jd" - lexicon = dataclasses.replace( - Lexicon.default(), suffix_acronyms_ambiguous=frozenset()) + default = Lexicon.default() + lexicon = default.remove( + *default._sets.suffix_acronyms_ambiguous, kind='name') hn = Parser(lexicon=lexicon).parse("Andrew Perkins (Jd)") self.m(hn.nickname, "", hn) self.m(hn.suffix, "Jd", hn) diff --git a/tests/v2/pipeline/test_assemble.py b/tests/v2/pipeline/test_assemble.py index fee4bd9b4..a4c945cca 100644 --- a/tests/v2/pipeline/test_assemble.py +++ b/tests/v2/pipeline/test_assemble.py @@ -12,13 +12,12 @@ from ..cases import CASES, Case -_LEX = Lexicon( - titles=frozenset({"dr"}), - particles=frozenset({"de", "la", "van"}), - particles_ambiguous=frozenset({"van"}), - suffix_words=frozenset({"iii"}), - maiden_markers=frozenset({"née"}), -) +_LEX = (Lexicon() + .add("dr", kind="title") + .add("de", "la", "van", kind="particle") + .add("van", kind="given") # an ambiguous particle + .add("iii", kind="suffix") + .add("née", kind="birth_marker")) def _parse(text: str) -> ParsedName: @@ -390,7 +389,7 @@ def test_a_caller_surname_reads_from_the_vocabulary_only_where_split_off( ) -> None: # spaced, a caller's own Latin surname is placed by position; the # list decides only a division of one written word - lex = Lexicon.default().add(surnames={"smith"}) + lex = Lexicon.default() | Lexicon(surnames={"smith"}) pn = Parser(lexicon=lex).parse("John Smith") assert pn.tokens[-1].reading == Reading("name", "family", "position") # type: ignore[arg-type] @@ -447,9 +446,10 @@ def test_every_suffix_words_reading_alternative_and_index_kind_agree( lex = Lexicon.default() kinds = {"generation": "generation", "agnomen": "agnomen", "credential": "credential"} - words = sorted(lex.suffix_words | lex.suffix_acronyms - | lex.suffix_words_ambiguous - | lex.suffix_acronyms_ambiguous) + sets = lex._sets + words = sorted(sets.suffix_words | sets.suffix_acronyms + | sets.suffix_words_ambiguous + | sets.suffix_acronyms_ambiguous) assert len(words) > 800 mismatched = [] reported = 0 @@ -545,19 +545,19 @@ def test_a_reading_names_the_files_holding_it( assert tok.reading.languages == languages -@pytest.mark.parametrize("field,word,text,written", [ - ("titles", "zzq-dr", "Zzq-Dr John Smith", "Zzq-Dr"), - ("particles", "mac-van", "John mac-van Smith", "mac-van"), +@pytest.mark.parametrize("kind,word,text,written", [ + ("title", "zzq-dr", "Zzq-Dr John Smith", "Zzq-Dr"), + ("particle", "mac-van", "John mac-van Smith", "mac-van"), # listed without the periods it is written with - ("titles", "drma", "Dr.Ma Smith", "Dr.Ma"), + ("title", "drma", "Dr.Ma Smith", "Dr.Ma"), ]) def test_a_callers_whole_word_names_no_file_of_its_parts( - field: str, word: str, text: str, written: str) -> None: + kind: str, word: str, text: str, written: str) -> None: # the parse read it whole, so the files of `dr` or `van` are not # its reading's; the default's own joined words still name theirs # (`Lt.Gov.` in _EXPECTED_LANGUAGES, and `Dr.Ma` below) assert parse(text).tokens[0].reading is not None - lex = Lexicon.default().add(**{field: {word}}) + lex = Lexicon.default().add(word, kind=kind) (tok,) = [t for t in Parser(lexicon=lex).parse(text).tokens if t.text == written] assert tok.reading is not None @@ -568,13 +568,15 @@ def test_a_callers_whole_word_names_no_file_of_its_parts( def test_a_reading_from_a_callers_own_vocabulary_names_no_file() -> None: # a word no file holds, and one a file holds only off by default # (`st`, a Czech generation written `default = false`): its kind is - # the word's, read whatever the reading's default (4b), so a pack - # turning it on reads it alike, but no file holds it on - lex = Lexicon.default().add(titles={"zzq"}, suffix_words={"st"}) + # the lexicon's since 4i-b (rules/R8's amendment), the caller's + # plain suffix reading a post-nominal where 4b read the file's off + # generation, and no file holds either on + lex = (Lexicon.default().add("zzq", kind="title") + .add("st", kind="suffix")) pn = Parser(lexicon=lex).parse("Zzq John Smith St") readings = {t.text: t.reading for t in pn.tokens} assert readings["Zzq"] == Reading("title", "title", "vocabulary") # type: ignore[arg-type] - assert readings["St"] == Reading("generation", "suffix", "vocabulary") # type: ignore[arg-type] + assert readings["St"] == Reading("post-nominal", "suffix", "vocabulary") # type: ignore[arg-type] assert readings["Zzq"] is not None and readings["Zzq"].languages == () assert readings["St"] is not None and readings["St"].languages == () @@ -584,8 +586,8 @@ def test_a_callers_whole_suffix_and_its_alternative_name_no_file_of_its_parts( # 'md' and 'ma' are in en, and the default reads 'MD-ZZ' by its # parts; listed whole, the token and the report's alternative name # none of their files - lex = Lexicon.default().add(suffix_acronyms={"md-zz", "ma-zz"}, - suffix_acronyms_ambiguous={"ma-zz"}) + lex = (Lexicon.default().add("md-zz", "ma-zz", kind="credential") + .add("ma-zz", kind="family")) parser = Parser(lexicon=lex) (md,) = [t for t in parser.parse("John Smith MD-ZZ").tokens if t.text == "MD-ZZ"] @@ -657,7 +659,7 @@ def test_every_particle_reads_patronymic_exactly_where_the_index_says( # file holds it as a patronymic particle, whatever its default from nameparser._reading_index import holds_kind lex = Lexicon.default() - words = sorted(lex.particles) + words = sorted(lex._sets.particles) assert len(words) > 100 found = set() mismatched = [] diff --git a/tests/v2/pipeline/test_assign.py b/tests/v2/pipeline/test_assign.py index f86c047a7..cd43aa95b 100644 --- a/tests/v2/pipeline/test_assign.py +++ b/tests/v2/pipeline/test_assign.py @@ -27,15 +27,16 @@ (Policy(name_order=FAMILY_FIRST_GIVEN_LAST), "family"), ] -_LEX = Lexicon( - titles=frozenset({"dr", "mr", "mrs", "sir", "sr"}), - given_name_titles=frozenset({"sir"}), - suffix_acronyms=frozenset({"phd", "md"}), - suffix_words=frozenset({"jr", "iii", "v", "sr"}), - particles=frozenset({"de", "la", "van"}), - particles_ambiguous=frozenset({"van"}), - conjunctions=frozenset({"and"}), - maiden_markers=frozenset({"née"}), +_LEX = ( + Lexicon() + .add("dr", "mr", "mrs", "sr", kind="title") + .add("sir", kind="title", addresses="given") + .add("phd", "md", kind="credential") + .add("jr", "iii", "v", "sr", kind="suffix") + .add("de", "la", "van", kind="particle") + .add("van", kind="given") # an ambiguous particle + .add("and", kind="conjunction") + .add("née", kind="birth_marker") ) @@ -291,8 +292,8 @@ def test_convention_details_name_the_field_the_word_lands_in( # only other details at this site that name a field -- H4's PEEL # shape ("Lord Chancellor") deliberately names none, because H1 # retags that word after assign under the default order. - lex = _LEX.add(suffix_words={"rinpoche"}, conjunctions={"of"}, - titles={"bishop"}) + lex = (_LEX.add("rinpoche", kind="suffix").add("of", kind="conjunction") + .add("bishop", kind="title")) (amb,) = _reported(text, policy, lexicon=lex) assert amb.kind is kind assert amb.detail == detail.format(role=role) @@ -305,7 +306,7 @@ def test_leading_particle_detail_follows_the_effective_order() -> None: # resolves family-first through _effective_order while name_order # is untouched and still reads given-first. The head piece is the # FAMILY name here, and the detail has to say so. - han = _LEX.add(particles={"毛"}, particles_ambiguous={"毛"}) + han = _LEX.add("毛", kind="particle").add("毛", kind="given") out = _assigned("毛 泽东", lexicon=han) assert out.policy.name_order[0] is Role.GIVEN assert out.policy.script_orders[0][0] is Script.HAN @@ -412,8 +413,7 @@ def test_the_trailing_title_is_transparent_to_the_suffix_peel() -> None: reserve keeps a bare ambiguous acronym the family of a two-word name, and takes it as a credential when a full name remains. """ - lex = _LEX.add(suffix_acronyms={"ma"}, - suffix_acronyms_ambiguous={"ma"}) + lex = _LEX.add("ma", kind="credential").add("ma", kind="family") # MOVED by #289, not deleted: spliced, 'John MA' is the bare # 'Jack MA' shape -- an ALL-CAPS ambiguous acronym in a mixed-case # name leans credential and is taken with no words to spare, so @@ -531,7 +531,7 @@ def test_family_first_order() -> None: def test_family_first_given_last_order() -> None: # Pinned against the actual built pipeline (2026-07-12), not guessed: - # 'Van' is particles_ambiguous and non-leading, so group's prefix + # 'Van' is an ambiguous particle and non-leading, so group's prefix # chain (which is name_order-agnostic -- it lands upstream of assign # and doesn't consult Policy) absorbs 'Van Anh Thu' into ONE piece # before assign ever sees it. With only two name pieces left, diff --git a/tests/v2/pipeline/test_classify.py b/tests/v2/pipeline/test_classify.py index fe9af186a..339eb209c 100644 --- a/tests/v2/pipeline/test_classify.py +++ b/tests/v2/pipeline/test_classify.py @@ -23,32 +23,33 @@ from nameparser._policy import Policy from nameparser._types import AmbiguityKind, Role -_LEX = Lexicon( - titles=frozenset({"dr", "sir"}), - given_name_titles=frozenset({"sir"}), - suffix_acronyms=frozenset({"phd", "ma"}), - suffix_words=frozenset({"jr", "v"}), - suffix_acronyms_ambiguous=frozenset({"ma"}), - particles=frozenset({"de", "la", "van"}), - particles_ambiguous=frozenset({"van"}), +_LEX = ( + Lexicon() + .add("dr", kind="title") + .add("sir", kind="title", addresses="given") + .add("phd", "ma", kind="credential") + .add("ma", kind="family") # an ambiguous credential + .add("jr", "v", kind="suffix") + .add("de", "la", "van", kind="particle") + .add("van", kind="given") # an ambiguous particle # 'e' and 'y' are both shipped single-letter conjunctions and the # fork treats them differently, which is the whole point of the - # subset; й is COPIED from the shipped conjunctions (Ukrainian, - # #267) so the collision + # `may_be_initial` mark; й is COPIED from the shipped conjunctions + # (Ukrainian, #267) so the collision # test_cyrillic_initial_outranks_the_conjunction pins is one that # really ships and the reader can check against the defaults. The # copies are local: this file never reads the shipped sets. # 'i' (#397) is the second marked entry and the only one that is - # ALSO in suffix_words here, which is what makes the collision cell - # below reachable. - conjunctions=frozenset({"and", "e", "i", "y", "й"}), - conjunctions_ambiguous=frozenset({"e", "i"}), - bound_given_names=frozenset({"abdul"}), + # ALSO a suffix word in some lexicon below, which is what makes the + # collision cell reachable. + .add("and", "y", "й", kind="conjunction") + .add("e", "i", kind="conjunction", may_be_initial=True) + .add("abdul", kind="bound_given") # "née" and the unaccented "nee" are both shipped (English writes # the French marker either way, AGENTS.md); a clause-truncation # test below needs the unaccented spelling since _normalize does # not fold accents away. - maiden_markers=frozenset({"née", "nee"}), + .add("née", "nee", kind="birth_marker") ) @@ -199,7 +200,7 @@ def test_a_trailing_uppercase_suffix_makes_the_name_mixed_case() -> None: # shipped vocabulary: unlisted, 'III' is an all-caps word after a # comma behind two name words, which the default reads as a # credential run since #564 and reports -- a different fork. - lex = dataclasses.replace(_LEX, suffix_words=_LEX.suffix_words | {"iii"}) + lex = _LEX.add("iii", kind="suffix") out = _classified_with("john e jones, III", lex) assert "conjunction" in _tags(out, "e") assert out.ambiguities == () @@ -208,7 +209,7 @@ def test_a_trailing_uppercase_suffix_makes_the_name_mixed_case() -> None: def test_emptying_the_subset_restores_joining_for_e() -> None: # the knob, and the reason there is no switch: remove 'e' and a # one-case name joins it again, silently. - lex = dataclasses.replace(_LEX, conjunctions_ambiguous=frozenset()) + lex = _LEX.add("e", "i", kind="conjunction", may_be_initial=False) out = _classified_with("jose e maria santos", lex) assert "conjunction" in _tags(out, "e") assert "initial" not in _tags(out, "e") @@ -248,29 +249,13 @@ def test_removing_the_link_from_the_subset_restores_joining() -> None: # the Catalan/Polish removal recipe docs/customize.rst teaches, # exercised: drop 'i' from the marked subset and a one-case name # joins it again, silently, which is what a ca/pl corpus wants. - lex = dataclasses.replace( - _LEX, conjunctions_ambiguous=_LEX.conjunctions_ambiguous - {"i"}) + lex = _LEX.add("i", kind="conjunction", may_be_initial=False) out = _classified_with("josep carod i rovira", lex) assert "conjunction" in _tags(out, "i") assert "initial" not in _tags(out, "i") assert out.ambiguities == () -def test_an_orphan_marker_is_inert_even_in_the_emitter() -> None: - # 'e' left in conjunctions_ambiguous but removed from conjunctions - # -- legal since the pair is not in _SUBSET_FIELDS (decisions/P8) - # -- never takes the fork, so 'E' in an all-upper name falls to the - # else branch and is tagged "initial" by is_initial() alone, same as - # any bare capital. Without the emitter's base-vocabulary gate this - # would still report a connective that the lexicon no longer has. - orphan = dataclasses.replace( - _LEX, conjunctions=_LEX.conjunctions - {"e"}) - out = _classified_with("JOSE E MARIA SANTOS", orphan) - assert "initial" in _tags(out, "E") - assert "conjunction" not in _tags(out, "E") - assert out.ambiguities == () - - def test_a_caseless_connective_never_enters_the_fork() -> None: # Arabic و has no case, so token.upper() == token.lower() and # today's rule stands whatever the name's case class is. Marked @@ -278,11 +263,14 @@ def test_a_caseless_connective_never_enters_the_fork() -> None: # decides anything: without this, و is absent from # conjunctions_ambiguous the same way every other shipped # conjunction but 'e' is, so the test would pass even if the - # casedness check were deleted outright. - lex = dataclasses.replace( - _LEX, - conjunctions=_LEX.conjunctions | frozenset({"و"}), - conjunctions_ambiguous=_LEX.conjunctions_ambiguous | frozenset({"و"})) + # casedness check were deleted outright. Since 3.0's 4i-b no + # configuration marks it, `add` refusing `may_be_initial` on a + # caseless letter as the build does, so the sets are built by hand: + # the gate is the stage's own defense, and this still holds it. + lex = _LEX.add("و", kind="conjunction") + object.__setattr__(lex, "_sets", dataclasses.replace( + lex._sets, conjunctions_ambiguous=( + lex._sets.conjunctions_ambiguous | frozenset({"و"})))) out = _classified_with("محمد و علي", lex) assert "conjunction" in _tags(out, "و") assert out.ambiguities == () @@ -551,8 +539,8 @@ def test_no_listed_member_ever_carries_the_shape_tag() -> None: # whole while `suffix_as_written`'s period-free acronym lookup # ('ab') misses it, so the chunk view reached the shape branch and # 'Jack A.B.' read family where 'Jack MA' reads suffix. - dotted = Lexicon.default().add(suffix_acronyms={"a.b"}, - suffix_acronyms_ambiguous={"a.b"}) + dotted = (Lexicon.default().add("a.b", kind="credential") + .add("a.b", kind="family")) for lex, texts in ((Lexicon.default(), ("John Smith MA", "John Smith Ma", "Smith, MA", "John Smith BA", "John Smith X.Y.Z.")), @@ -565,7 +553,7 @@ def test_no_listed_member_ever_carries_the_shape_tag() -> None: policy=policy).items(): if SHAPE_ACRONYM_TAG in tags: assert _normalize(word) not in \ - lex.suffix_acronyms_ambiguous, (text, word) + lex._sets.suffix_acronyms_ambiguous, (text, word) def test_delimited_content_never_joins_the_shape_class() -> None: @@ -775,20 +763,19 @@ def test_the_caps_shape_is_a_testable_predicate() -> None: == _tags_by_text("John Smith X.Y")["X.Y"]) - def _surname_unit_sweep() -> list[str]: lex = Lexicon.default() words: set[str] = {"Smith", "Ma", "M.D.", "Ph.D.", "JD.CPA", "Msc.Ed.", "Lt.Gov.", "X.Y.Z.", "V.", "I", "v", "Jr.", "de."} for field in ("particles", "suffix_acronyms", "suffix_words", "conjunctions", "bound_given_names", "titles"): - for w in getattr(lex, field): + for w in getattr(lex._sets, field): if " " not in w: words |= {w, w.title(), w.upper()} # rules/P7's spelling: a one-letter particle written with its # period is an initial, and both copies must say so (#604) words |= {w + "." for w in set(words) - if len(w) == 1 and w.lower() in lex.particles} + if len(w) == 1 and w.lower() in lex._sets.particles} return sorted(words) diff --git a/tests/v2/pipeline/test_comma.py b/tests/v2/pipeline/test_comma.py index fcab289ec..5af58fd87 100644 --- a/tests/v2/pipeline/test_comma.py +++ b/tests/v2/pipeline/test_comma.py @@ -16,11 +16,11 @@ from nameparser._types import AmbiguityKind, Role # synthetic vocabulary: behavior given a lexicon, never default() contents -_LEX = Lexicon( - suffix_acronyms=frozenset({"phd", "md", "do", "dds", "ma", "ed"}), - suffix_acronyms_ambiguous=frozenset({"ma", "ed", "do"}), - suffix_words=frozenset({"jr", "v"}), -) +_LEX = (Lexicon() + .add("phd", "md", "do", "dds", "ma", "ed", kind="credential") + # a family reading makes the credential ambiguous + .add("ma", "ed", "do", kind="family") + .add("jr", "v", kind="suffix")) def _decided(text: str, lexicon: Lexicon = _LEX, @@ -250,7 +250,7 @@ def test_the_run_test_reads_lenient_comma_suffixes() -> None: # is_suffix_lenient bypasses the initial veto), strict under # Policy(lenient_comma_suffixes=False) (v1's is_suffix, which the # veto reads as a middle initial instead). - lex = _LEX.add(suffix_words={"b"}) + lex = _LEX.add("b", kind="suffix") text = "John Smith, Ma B." state = ParseState(original=text, lexicon=lex, policy=Policy()) out = _comma.decide(classify(segment(tokenize(extract_delimited(state))))) @@ -314,7 +314,7 @@ def test_a_listed_surname_still_carries_the_name_contrast() -> None: # is a different question. A draft shared one predicate for both # and this read given 'XYZ'. The all-caps spelling is the control: # no word before the comma is written as a name, so no contrast. - lex = Lexicon.default().add(surnames={"smith", "jones"}) + lex = Lexicon.default() | Lexicon(surnames={"smith", "jones"}) out = _decided("Smith Jones, XYZ", lexicon=lex) assert out.structure is Structure.SUFFIX_COMMA assert out.tokens[2].role is Role.SUFFIX @@ -391,9 +391,9 @@ def test_a_generational_connective_joins_no_title_after_the_comma() -> None: # it in a plain join, which only caller vocabulary can spell. The # control is the same word as a plain connective, which joins the # titles. - generational = Lexicon.default().add(conjunctions={"og"}, - suffix_words={"og"}) - plain = Lexicon.default().add(conjunctions={"og"}) + generational = (Lexicon.default().add("og", kind="conjunction") + .add("og", kind="suffix")) + plain = Lexicon.default().add("og", kind="conjunction") name = Parser(lexicon=generational).parse("Smith, Mr. og Mrs.") assert (name.title, name.suffix) == ("Mr. Mrs.", "og") name = Parser(lexicon=plain).parse("Smith, Mr. og Mrs.") diff --git a/tests/v2/pipeline/test_group.py b/tests/v2/pipeline/test_group.py index ac982354e..e1cf8ac90 100644 --- a/tests/v2/pipeline/test_group.py +++ b/tests/v2/pipeline/test_group.py @@ -1,5 +1,4 @@ import bisect -import dataclasses from collections.abc import Sequence from typing import cast @@ -23,18 +22,19 @@ from nameparser._policy import Policy, Script from nameparser._types import AmbiguityKind, Role -_LEX = Lexicon( - titles=frozenset({"mr", "mrs", "secretary", "the", "state"}), - suffix_acronyms=frozenset({"phd"}), - suffix_words=frozenset({"jr"}), - particles=frozenset({"de", "la", "van", "von", "und", "zu"}), - particles_ambiguous=frozenset({"van"}), - conjunctions=frozenset({"and", "of", "the", "y"}), - bound_given_names=frozenset({"abdul"}), - maiden_markers=frozenset({"née", "geb"}), +_LEX = ( + Lexicon() + .add("mr", "mrs", "secretary", "the", "state", kind="title") + .add("phd", kind="credential") + .add("jr", kind="suffix") + .add("de", "la", "van", "von", "und", "zu", kind="particle") + .add("van", kind="given") # an ambiguous particle + .add("and", "of", "the", "y", kind="conjunction") + .add("abdul", kind="bound_given") + .add("née", "geb", kind="birth_marker") ) -# NOTE: "the" sits in both titles and conjunctions here, mirroring v1's -# real vocabulary overlap; the subset rule only constrains particles. +# NOTE: "the" holds both a title and a conjunction reading here, +# mirroring v1's real vocabulary overlap. def _grouped(text: str, policy: Policy | None = None, @@ -115,7 +115,7 @@ def test_a_title_that_is_also_a_particle_stops_the_scan() -> None: # family Smith). Spelled here as the overlap a CONFIG can create, # which is the same shape: tests/test_constants.py::test_add_title # adds `te` to the titles while `te` ships as a particle. - overlap = dataclasses.replace(_LEX, titles=_LEX.titles | {"van"}) + overlap = _LEX.add("van", kind="title") assert _piece_texts(_grouped("Van von Richthofen", lexicon=overlap)) == \ [["Van", "von Richthofen"]] assert _piece_texts(_grouped("Van Johnson Smith", lexicon=overlap)) == \ @@ -174,10 +174,11 @@ def test_the_phd_merge_declines_at_the_head_of_a_name() -> None: def test_von_und_zu_bridges() -> None: # conjunction "und" joins two prefixes; the joined piece is a derived # prefix and still chains onto the following name (v1 PR #191). - # `und` is a CONNECTIVE here and no particle: _LEX lists it among - # the particles, which walks the particle chain instead and passed + # `und` is a CONNECTIVE here and no particle: _LEX reads it as a + # particle, which walks the particle chain instead and passed # with P3's prefix derivation deleted (#617's test review) - lexicon = _LEX.remove(particles={"und"}).add(conjunctions={"und"}) + lexicon = _LEX.remove("und", kind="particle").add( + "und", kind="conjunction") out = _grouped("Otto von und zu Habsburg", lexicon=lexicon) assert _piece_texts(out) == [["Otto", "von und zu Habsburg"]] @@ -230,7 +231,7 @@ def test_leading_marker_is_not_consumed() -> None: #: `nee` a marker AND `Nee` a surname in one vocabulary -- the collision #: the clause-size guard exists for. _LEX alone leaves `Nee` untagged, #: which would let the guard tests pass under every mutant. -_NEE_LEX = _LEX.add(maiden_markers=frozenset({"nee"})) +_NEE_LEX = _LEX.add("nee", kind="birth_marker") def test_delimited_marker_is_dropped() -> None: @@ -302,8 +303,8 @@ def test_clause_containment_survives_script_segmentation() -> None: one place the two meet: 王小明 becomes 王 + 小明 before group runs, which shifts every token index after it while the spans stay exact, and the marker is still the token the clause drops.""" - lex = Lexicon(surnames=frozenset({"王"}), - maiden_markers=frozenset({"旧姓"})) + lex = Lexicon(surnames=frozenset({"王"})).add( + "旧姓", kind="birth_marker") policy = Policy(segment_scripts=frozenset({Script.HAN}), maiden_delimiters=frozenset({("（", "）")})) state = ParseState(original="王小明（旧姓 李四）", lexicon=lex, @@ -329,8 +330,8 @@ def test_extra_suffix_delimiter_splits_tail_entries() -> None: # suffix ENTRIES (each rendered independently) and is itself # dropped, rather than becoming a suffix token or fusing the # entries on either side into one "joined" run. - lex = Lexicon(suffix_acronyms=frozenset({"phd"}), - suffix_words=frozenset({"jr", "v", "md"})) + lex = (Lexicon().add("phd", kind="credential") + .add("jr", "v", "md", kind="suffix")) out = _grouped("Smith, John, V MD / PhD", Policy(extra_suffix_delimiters=frozenset({"/"})), lexicon=lex) @@ -357,9 +358,8 @@ def test_suffix_comma_name_segment_gets_no_additional_count() -> None: assert _piece_texts(out) == [["John", "y", "Smith"], []] -_DUAL_LEX = dataclasses.replace( - _LEX, bound_given_names=frozenset({"abdul", "dual"}), - suffix_acronyms=frozenset({"phd", "dual"})) +_DUAL_LEX = _LEX.add("dual", kind="bound_given").add( + "dual", kind="credential") def test_a_bound_word_in_two_vocabularies_still_joins() -> None: @@ -429,7 +429,7 @@ def test_a_chain_never_leaves_a_marker_standing_alone() -> None: out = _grouped(text) for seg, texts in zip(out.pieces, _piece_texts(out)): for k, (piece, shown) in enumerate(zip(seg, texts)): - if shown.lower() not in _LEX.maiden_markers or k == 0: + if shown.lower() not in _LEX._sets.maiden_markers or k == 0: continue assert not _piece_has_tag(out, seg[k - 1], "particle"), ( f"{text!r}: marker {shown!r} left standing after " @@ -597,9 +597,7 @@ def test_the_bound_given_join_never_absorbs_a_marker() -> None: f"the piece {shown!r}") -_GIVEN_NAME_TITLE_LEX = dataclasses.replace( - _LEX, titles=_LEX.titles | {"sir"}, - given_name_titles=frozenset({"sir"})) +_GIVEN_NAME_TITLE_LEX = _LEX.add("sir", kind="title", addresses="given") def test_a_given_name_title_licenses_the_bound_join_with_one_word_to_spare() -> None: @@ -714,7 +712,7 @@ def test_the_numeral_is_read_after_a_suffix_word_that_is_also_a_title() -> None: # family again: 'abdul Smith jr V' read given 'abdul Smith', # suffix 'jr, V'. Found by the design-docs review. The shipped # vocabulary has jr in TITLES; the test lexicon gets the same. - out = _grouped("abdul Smith jr V", lexicon=_LEX.add(titles={"jr"})) + out = _grouped("abdul Smith jr V", lexicon=_LEX.add("jr", kind="title")) assert _piece_texts(out) == [["abdul", "Smith", "jr", "V"]] @@ -796,8 +794,7 @@ def test_the_join_declines_a_suffix_after_a_family_comma_too() -> None: # -- #425: the reserve runs assign's peel over the post-join view -_AMBIGUOUS_LEX = _LEX.add(suffix_acronyms={"ma"}, - suffix_acronyms_ambiguous={"ma"}) +_AMBIGUOUS_LEX = _LEX.add("ma", kind="credential").add("ma", kind="family") def test_the_reserve_mirrors_the_bare_acronym_fork() -> None: @@ -919,7 +916,7 @@ def test_the_chain_keeps_an_acronym_assign_will_not_peel() -> None: # Berg', family 'Ma' -- 1.4.0's reading, and the reviews' find). # The chain asks the peel again over what it leaves, and takes the # acronym assign will not peel. - lex = _AMBIGUOUS_LEX.add(titles={"st"}, particles={"st"}) + lex = _AMBIGUOUS_LEX.add("st", kind="title").add("st", kind="particle") out = _grouped("St van Berg Ma", lexicon=lex) assert _piece_texts(out) == [["St", "van Berg Ma"]] # MOVED by #289, not deleted: with a given word of its own the @@ -1044,12 +1041,12 @@ def test_the_view_check_asks_about_the_member_and_not_about_the_run( then declines 'MA' for want of words to spare. A check asking whether the reader takes SOMETHING answers yes there, and the member becomes the FAMILY name.""" - # NOTE the accented marker: this module's `_LEX` ships - # maiden_markers={"née", "geb"} and NOT the unaccented "nee", so + # NOTE the accented marker: this module's `_LEX` reads "née" and + # "geb" as birth markers and NOT the unaccented "nee", so # the plain spelling takes nothing at all here and the test would # pass vacuously. cases.py's row of the same shape uses the # DEFAULT vocabulary, where both spellings are markers. - lex = _AMBIGUOUS_LEX.add(suffix_acronyms={"phd"}) + lex = _AMBIGUOUS_LEX.add("phd", kind="credential") out = _grouped("JOHN NÉE JONES SMITH MA PHD", lexicon=lex) assert _maiden_texts(out) == ["JONES", "SMITH", "MA"] assert len(_suffix_forks(out)) == 1 @@ -1069,7 +1066,7 @@ def test_a_marker_in_a_third_comma_part_is_a_word() -> None: credentials whole, and a marker there takes nothing (#601). Until #601 the clause was taken whole there, maiden 'Jones MA'.""" out = _grouped("Smith, John, Jr née Jones MA", - lexicon=_AMBIGUOUS_LEX.add(suffix_words={"jr"})) + lexicon=_AMBIGUOUS_LEX.add("jr", kind="suffix")) assert _maiden_texts(out) == [] assert _suffix_forks(out) == [] @@ -1249,8 +1246,8 @@ def test_an_unlisted_abbreviation_is_as_transparent_as_a_title() -> None: # phrase whose FIRST WORD is itself an entry -- the longest-first case, # where a site that stopped at the first word would still find a # marker and simply find a shorter one. -_MARKER_LEX = Lexicon( - maiden_markers=frozenset({"née", "z domu", "geb", "geb von"})) +_MARKER_LEX = Lexicon().add("née", "z domu", "geb", "geb von", + kind="birth_marker") def _marker_sites(spelling: str) -> dict[str, int]: @@ -1258,7 +1255,7 @@ def _marker_sites(spelling: str) -> dict[str, int]: a name whose marker is written `spelling`.""" clause = spelling.split() + ["Jones"] answers = {"predicate": maiden_marker_run(clause, - _MARKER_LEX.maiden_markers)} + _MARKER_LEX._sets.maiden_markers)} def staged(text: str) -> ParseState: return classify(segment(tokenize(extract_delimited(ParseState( @@ -1400,9 +1397,9 @@ def test_the_marker_placements_reach_both_answers() -> None: # letter -- and a test using the shipped sets would walk the right # branch while proving nothing about it (_pipeline/AGENTS.md: "Pin the # decision, not the vocabulary"). -_LINK_LEX = _LEX.add(conjunctions={"i"}, suffix_words={"i"}) +_LINK_LEX = _LEX.add("i", kind="conjunction").add("i", kind="suffix") #: the same letter as a connective that is NOT generational vocabulary -_PLAIN_LEX = _LEX.add(conjunctions={"i"}) +_PLAIN_LEX = _LEX.add("i", kind="conjunction") def test_a_connective_piece_counts_toward_the_carve_outs_total() -> None: @@ -1453,11 +1450,11 @@ def test_the_right_hand_test_reads_the_peel_not_the_suffix_piece( # suffix-piece test refuses it (the initial veto, # `_vocab.is_initial`) while assign's trailing peel takes it. Spelled with the piece test # alone, the link would swallow the generation here. - lex = _LINK_LEX.add(suffix_words={"v"}) + lex = _LINK_LEX.add("v", kind="suffix") out = _grouped("Josep Lluis Carod i V", lexicon=lex) assert _piece_texts(out) == [["Josep", "Lluis", "Carod", "i", "V"]] plain = _grouped("Josep Lluis Carod i V", - lexicon=_PLAIN_LEX.add(suffix_words={"v"})) + lexicon=_PLAIN_LEX.add("v", kind="suffix")) assert _piece_texts(plain) == [["Josep", "Lluis", "Carod i V"]] @@ -1619,7 +1616,7 @@ def test_a_one_case_letter_is_an_initial_and_never_counts() -> None: # unmoved, and the name reads as it always did. The EXCLUSIVITY is # the load-bearing part, not the order of the two tests -- measured # (swapping them moves nothing). - lex = _LINK_LEX.add(conjunctions_ambiguous={"i"}) + lex = _LINK_LEX.add("i", kind="conjunction", may_be_initial=True) out = _grouped("josep carod i rovira", lexicon=lex) assert _piece_texts(out) == [["josep", "carod", "i", "rovira"]] @@ -1628,7 +1625,7 @@ def test_the_class_reaches_a_callers_own_connective() -> None: # rules/P3 is keyed on the class throughout: a caller who adds # 'v' to their connectives gets the Catalan link's behavior for # it, because 'v' is generational vocabulary the way 'i' is. - p = Parser(lexicon=Lexicon.default().add(conjunctions={"v"})) + p = Parser(lexicon=Lexicon.default().add("v", kind="conjunction")) joined = p.parse("Josep Carod v Rovira") assert (joined.given, joined.family) == ("Josep", "Carod v Rovira") trailing = p.parse("John Quincy Smith v") @@ -1641,7 +1638,7 @@ def test_a_callers_non_generational_letter_is_outside_the_condition( # roman numeral letter that is NOT suffix vocabulary, so the # both-sides condition declines to ask and BOTH positions join, # which is what they did before this change too. - p = Parser(lexicon=Lexicon.default().add(conjunctions={"x"})) + p = Parser(lexicon=Lexicon.default().add("x", kind="conjunction")) assert p.parse("John Quincy Smith x").family == "Smith x" assert p.parse("Josep Carod x Rovira").family == "Carod x Rovira" @@ -1655,7 +1652,7 @@ def test_the_count_reaches_a_connective_that_is_particle_vocabulary( # Now it agrees with it -- the direction the rule wants. Measured # before this change: given 'Juan', middle 'Velasquez', family # 'y Garcia', family_base 'Garcia'. - p = Parser(lexicon=Lexicon.default().add(particles={"y"})) + p = Parser(lexicon=Lexicon.default().add("y", kind="particle")) out = p.parse("Juan Velasquez y Garcia") assert (out.given, out.middle, out.family) == ( "Juan", "", "Velasquez y Garcia") @@ -1675,14 +1672,15 @@ def test_the_count_reaches_a_connective_that_is_particle_vocabulary( # outside the generational vocabulary the walk never stopped at it in # the first place, so the pair says which arm does the work. # -# `ma` is added to BOTH acronym sets where the right-hand neighbour -# has to be an AMBIGUOUS credential: such a word carries no +# `ma` is added as a credential with a family reading where the +# right-hand neighbour has to be an AMBIGUOUS credential: such a word +# carries no # `vocab:suffix` tag, so the piece test cannot refuse it and only the # peel bound keeps it out of the clause. -_LINK_MA_LEX = _LINK_LEX.add(suffix_acronyms={"ma"}, - suffix_acronyms_ambiguous={"ma"}) -_PLAIN_MA_LEX = _PLAIN_LEX.add(suffix_acronyms={"ma"}, - suffix_acronyms_ambiguous={"ma"}) +_LINK_MA_LEX = _LINK_LEX.add("ma", kind="credential").add( + "ma", kind="family") +_PLAIN_MA_LEX = _PLAIN_LEX.add("ma", kind="credential").add( + "ma", kind="family") def test_a_link_inside_a_maiden_clause_does_not_end_it() -> None: @@ -1882,8 +1880,8 @@ def test_a_multi_letter_link_of_the_suffix_vocabulary_joins_by_the_same_rule( # caller-reachable past it. Nothing SHIPPED reaches it -- 'i' is # the only member of the class in the default vocabulary and in # every locale pack -- which is why the lexicon is built here. - lex = Lexicon.default().add(conjunctions={"og"}, - suffix_words={"og"}) + lex = Lexicon.default().add("og", kind="conjunction").add( + "og", kind="suffix") parser = Parser(lexicon=lex) # nothing on its right: it is the generation it also spells lone = parser.parse("John Quincy Smith og") diff --git a/tests/v2/pipeline/test_pieces.py b/tests/v2/pipeline/test_pieces.py index d762adf03..136b18bd3 100644 --- a/tests/v2/pipeline/test_pieces.py +++ b/tests/v2/pipeline/test_pieces.py @@ -692,14 +692,14 @@ def test_own_words_is_the_name_s_own_span_and_its_clause_cut() -> None: # one function (#289/#516). state = _state_through("segment", "Jane née Jones Smith") texts, cut = own_words(state.tokens, state.comma_offsets, - state.lexicon.maiden_markers) + state.lexicon._sets.maiden_markers) assert texts == ["Jane"] assert cut == 1 # a delimited clause's tokens arrive with a role already set, so # they are not the name's own words either state = _state_through("segment", "Andrew (Andy) Perkins") texts, cut = own_words(state.tokens, state.comma_offsets, - state.lexicon.maiden_markers) + state.lexicon._sets.maiden_markers) assert texts == ["Andrew", "Perkins"] assert cut == 3 # `comma_offsets` is load-bearing only for a marker PHRASE whose @@ -712,10 +712,10 @@ def test_own_words_is_the_name_s_own_span_and_its_clause_cut() -> None: # cut lands at 'z' instead. Measured 2026-09-17. state = _state_through("segment", "Anna z, domu Nowak") texts, cut = own_words(state.tokens, state.comma_offsets, - state.lexicon.maiden_markers) + state.lexicon._sets.maiden_markers) assert texts == ["Anna", "z", "domu", "Nowak"] assert cut == 4 - texts, cut = own_words(state.tokens, (), state.lexicon.maiden_markers) + texts, cut = own_words(state.tokens, (), state.lexicon._sets.maiden_markers) assert texts == ["Anna"] assert cut == 1 @@ -726,17 +726,17 @@ def test_own_words_takes_a_marker_map_when_the_caller_has_one() -> None: state = _state_through("segment", "Jane née Jones Smith") heads = {1: "vocab:maiden-marker", 2: "vocab:maiden-marker-cont"} assert own_words(state.tokens, state.comma_offsets, - state.lexicon.maiden_markers, + state.lexicon._sets.maiden_markers, heads) == (["Jane"], 1) # an incomplete phrase entry: no head tag, so the map's answer is # "no clause" -- and the self-built path agrees, since both now # run the SAME completion test (#289/#516) state = _state_through("segment", "Anna z Nowak") assert own_words(state.tokens, state.comma_offsets, - state.lexicon.maiden_markers, + state.lexicon._sets.maiden_markers, {}) == (["Anna", "z", "Nowak"], 3) assert own_words(state.tokens, state.comma_offsets, - state.lexicon.maiden_markers)[1] == 3 + state.lexicon._sets.maiden_markers)[1] == 3 def test_own_words_two_spellings_agree_on_the_one_case_verdict() -> None: @@ -759,11 +759,11 @@ def test_own_words_two_spellings_agree_on_the_one_case_verdict() -> None: state = _state_through("segment", text) folded = [_normalize(t.text) for t in state.tokens] built_map = tag_marker_runs(state.tokens, state.comma_offsets, - state.lexicon.maiden_markers, folded) + state.lexicon._sets.maiden_markers, folded) tagged = own_words(state.tokens, state.comma_offsets, - state.lexicon.maiden_markers, built_map) + state.lexicon._sets.maiden_markers, built_map) walked = own_words(state.tokens, state.comma_offsets, - state.lexicon.maiden_markers) + state.lexicon._sets.maiden_markers) assert tagged == walked, text assert is_one_case(tagged[0]) == is_one_case(walked[0]), text @@ -785,7 +785,7 @@ def test_tag_marker_runs_answers_in_ascending_index_order() -> None: state = _state_through("segment", text) folded = [_normalize(t.text) for t in state.tokens] keys = list(tag_marker_runs(state.tokens, state.comma_offsets, - state.lexicon.maiden_markers, + state.lexicon._sets.maiden_markers, folded)) assert keys == sorted(keys), text @@ -860,7 +860,10 @@ def test_a_title_the_fixed_point_splices_out_keeps_no_peel_pick() -> None: # the only way here. Recorded negative control (2026-10-01): with # the drop removed, this reads the same fields and reports # 'suffix-or-name' on 'Ma.', a word the parse took as a title. - parser = Parser(lexicon=Lexicon.default().add(titles={"ma"})) + # `ma`'s given readings come off first, or rules/H9 keeps it a + # title only inside a title phrase. + parser = Parser(lexicon=Lexicon.default().remove("ma", kind="given") + .add("ma", kind="title")) name = parser.parse("John Smith Ma. PhD Jr.") assert (name.title, name.given, name.family, name.suffix) == ( "Ma.", "John", "Smith", "PhD Jr.") diff --git a/tests/v2/pipeline/test_post_rules.py b/tests/v2/pipeline/test_post_rules.py index 08d35cc68..a97f53d5a 100644 --- a/tests/v2/pipeline/test_post_rules.py +++ b/tests/v2/pipeline/test_post_rules.py @@ -27,28 +27,30 @@ # found three such words when it was written (#395): `dr` and `md` were # absent while shipping as titles, and `la` sat in the never-given half # while shipping as ambiguous. -_LEX = Lexicon( - titles=frozenset({"mr", "sir", "dr"}), - given_name_titles=frozenset({"sir"}), - after_given_titles=frozenset({"mr"}), - particles=frozenset({"de", "der", "ibn", "la", "van", "vd"}), - particles_ambiguous=frozenset({"la", "van"}), +_LEX = ( + Lexicon() + .add("dr", kind="title") + .add("mr", kind="title", after_given=True) + .add("sir", kind="title", addresses="given") + .add("de", "der", "la", "van", "vd", kind="particle") + .add("ibn", kind="patronymic") + .add("la", "van", kind="given") # ambiguous particles # `씨` is a ONE-LETTER suffix word, which starts no credential run # (rules/S8's two-letter minimum -- any longer suffix word would, # an honorific included since #679), the filler a test needs where # a credential would now take the rest of the part - suffix_words=frozenset({"jr", "씨"}), + .add("jr", "씨", kind="suffix") # `vd` mirrors its shipped dual membership -- particle AND - # UNAMBIGUOUS suffix acronym -- which is what puts a name on P6's + # UNAMBIGUOUS credential -- which is what puts a name on P6's # suffix arm below. `ma` mirrors the ambiguous class, which starts # no run either. - suffix_acronyms=frozenset({"md", "vd", "ma"}), - suffix_acronyms_ambiguous=frozenset({"ma"}), - # and `ma` is a given name written short too (Filipino María, #789), - # read so only period-marked at the head of the given part - given_name_abbreviations=frozenset({"ma"}), - conjunctions=frozenset({"y"}), - bound_given_names=frozenset({"abdul"}), + .add("md", "vd", "ma", kind="credential") + # and `ma` is a given name written short too (Filipino María, + # #789), read so only period-marked at the head of the given part; + # the given reading is also what makes its credential ambiguous + .add("ma", kind="given", same_as="maría") + .add("y", kind="conjunction") + .add("abdul", kind="bound_given") ) @@ -290,7 +292,7 @@ def test_leading_never_given_particle_folds_into_family() -> None: def test_leading_ambiguous_particle_stays_given() -> None: - # 'van' is particles_ambiguous: the given reading stands (v1 parity) + # 'van' is an ambiguous particle: the given reading stands (v1 parity) out = _parsed("van Gogh") assert _by_role(out, Role.GIVEN) == "van" assert _by_role(out, Role.FAMILY) == "Gogh" @@ -377,7 +379,7 @@ def test_leading_piece_scan_skips_pieces_that_hold_no_name( ("de", "de", ""), # the leading piece is not a particle ("Juan de Mesnil", "Juan", "de Mesnil"), - # 'van' is particles_ambiguous -- out of the rule's scope in EVERY + # 'van' is an ambiguous particle -- out of the rule's scope in EVERY # order, so a family-first order still splits at the particle (#360) ("van Gogh", "van", "Gogh"), ]) @@ -607,7 +609,7 @@ def test_no_never_given_particle_is_left_as_the_given_name( nothing about which words are in the set. """ lex = Lexicon.default() - never_given = sorted(lex.particles - lex.particles_ambiguous) + never_given = sorted(lex._sets.particles - lex._sets.particles_ambiguous) assert never_given, "no never-given particles to exercise" failures = [] for particle in never_given: @@ -766,9 +768,8 @@ def test_the_fixture_guard_fires( # -- the same inertness it exists to prevent. assert not _fixture_mirrors_shipped("de la Cruz Juan", Policy()) monkeypatch.setattr(sys.modules[__name__], "_LEX", - dataclasses.replace( - _LEX, particles=frozenset(), - particles_ambiguous=frozenset())) + _LEX.remove("de", "der", "ibn", "la", "van", "vd", + kind="particle")) assert "'de'" in _fixture_mirrors_shipped("de la Cruz Juan", Policy()) @@ -782,7 +783,7 @@ def test_the_stop_reads_the_effective_order_not_the_policy() -> None: # never read family-first. Needs a custom lexicon (every shipped # particle is Latin, and Latin has no script_orders entry), so it # builds the state directly rather than going through _parsed. - lex = Lexicon(particles=frozenset({"ノ"})) + lex = Lexicon().add("ノ", kind="particle") policy = Policy(name_order=FAMILY_FIRST, script_orders=((Script.KATAKANA, GIVEN_FIRST),)) out = run(ParseState(original="ノ クルス フアン カルロス", @@ -879,11 +880,17 @@ def test_a_dual_membership_word_decides_rather_than_raising() -> None: # the table for the attachment to decline # (mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE -- report the # branch actually taken). - lex = dataclasses.replace( - _LEX, - particles=frozenset(_LEX.particles | {"zz"}), - particles_ambiguous=frozenset(_LEX.particles_ambiguous | {"zz"}), - suffix_acronyms=frozenset(_LEX.suffix_acronyms | {"zz"})) + # + # Since 3.0's 4i-b no configuration reaches it either: the `given` + # reading that makes a particle possibly given also gates a + # credential beside it, so the word is an ambiguous acronym too. + # The arms stay ordered all the same, and the sets are built here + # by hand, so the stage's defense is still what this holds. + lex = (_LEX.add("zz", kind="particle").add("zz", kind="given") + .add("zz", kind="credential")) + object.__setattr__(lex, "_sets", dataclasses.replace( + lex._sets, suffix_acronyms_ambiguous=( + lex._sets.suffix_acronyms_ambiguous - {"zz"}))) out = run(ParseState(original="Berg, Jan zz", lexicon=lex, policy=Policy())) # token order, so the attachment shows as the run joining the @@ -1228,7 +1235,7 @@ def test_an_all_particle_part_keeps_r2s_mark_and_gains_no_other( # the part is all particles R2's mark already readmits every word, # a word that is BOTH particle and connective included. Swapping # the branches moves the caller's own rows. - p = Parser(lexicon=Lexicon.default().add(particles={"y"})) + p = Parser(lexicon=Lexicon.default().add("y", kind="particle")) r = p.parse("Juan de y") y = [t for t in r.tokens if t.text == "y"][0] assert UNJOINED_TAG in y.tags @@ -1246,7 +1253,7 @@ def test_a_word_that_is_both_particle_and_connective_is_marked_too( # Without the arm the both-vocabulary word takes neither mark, # drops as an ordinary family particle, and the part's own base # disagrees with the view over it. - p = Parser(lexicon=Lexicon.default().add(particles={"y"})) + p = Parser(lexicon=Lexicon.default().add("y", kind="particle")) r = p.parse("Juan de y and") assert r.family == "de y and" assert r.initials() == "J. y. a." @@ -1303,7 +1310,7 @@ def test_two_family_names_counts_every_listed_hyphen_as_two_words( #: rules/O6's given-name veto (#959) reads Lexicon.given_names, which #: the default leaves empty; these rows carry the list a pack ships. -_ES_GIVEN = Lexicon.default().add(given_names=GIVEN_NAMES) +_ES_GIVEN = Lexicon.default() | Lexicon(given_names=GIVEN_NAMES) def _roles(text: str, policy: Policy, lexicon: Lexicon) -> tuple[str, ...]: @@ -1399,10 +1406,10 @@ def test_a_caller_entry_matches_as_written_and_by_its_key() -> None: # An entry with its accent matches the word written with it; one # without matches every spelling, the key dropping the accent. o6 = Policy(two_family_names=True) - accented = Lexicon.default().add(given_names={"Lúis"}) + accented = Lexicon.default() | Lexicon(given_names={"Lúis"}) assert _roles("José Lúis Pérez", o6, accented)[2] == "Pérez" assert _roles("José Luis Pérez", o6, accented)[2] == "Luis Pérez" - bare = Lexicon.default().add(given_names={"luis"}) + bare = Lexicon.default() | Lexicon(given_names={"luis"}) for written in ("Luis", "Lúis", "LUIS", "Luís"): assert _roles(f"José {written} Pérez", o6, bare)[2] == "Pérez", written diff --git a/tests/v2/pipeline/test_script_segment.py b/tests/v2/pipeline/test_script_segment.py index 2563273ed..55ec7d1c0 100644 --- a/tests/v2/pipeline/test_script_segment.py +++ b/tests/v2/pipeline/test_script_segment.py @@ -25,14 +25,15 @@ # 欧 AND 欧阳 both present on purpose: compound-before-single must # pick 欧阳 (the issue's own acceptance example); same for 남/남궁. # "jr" so the comma cases can reach SUFFIX_COMMA. -_LEX = Lexicon(surnames=frozenset({"毛", "欧", "欧阳", "김", "남", "남궁"}), - suffix_words=frozenset({"jr"})) -# The peel's own vocabulary (#308). Tails are also suffix words here, -# as the shipped config asserts they are: the neighbour rule in the +_LEX = Lexicon(surnames=frozenset({"毛", "欧", "欧阳", "김", "남", "남궁"}) + ).add("jr", kind="suffix") +# The peel's own vocabulary (#308): suffix readings carrying +# `attached="after"`, so tails are also suffix words here, as the +# shipped config asserts they are: the neighbour rule in the # segmenter tests below reads that membership, so a stage lexicon that # omitted it would pin a shape no real configuration can have. _TAILS = frozenset({"씨", "님", "선생님", "박사", "さん", "先生", "様"}) -_LEX_TAILS = _LEX.add(suffix_words=_TAILS, honorific_tails=_TAILS) +_LEX_TAILS = _LEX.add(*sorted(_TAILS), kind="suffix", attached="after") def _run(text: str, policy: Policy = _HAN, lexicon: Lexicon = _LEX, @@ -163,7 +164,7 @@ def test_a_family_comma_stands_the_surname_split_down() -> None: # boundary the writer did not draw ("남궁 민수" with a space). # The SPLIT only -- the stage as a whole is not inert under a # family comma since #312, and would not be on a lexicon carrying - # honorific_tails (_LEX does not; see the peel's own cases below). + # attached tails (_LEX does not; see the peel's own cases below). out = _run("남궁민수, 지훈", policy=_HANGUL) assert out.structure is Structure.FAMILY_COMMA assert _texts(out) == ["남궁민수", "지훈"] @@ -740,7 +741,7 @@ def test_an_interpunct_gates_the_split_but_not_the_peel() -> None: # still an honorific, so the peel crosses it (#312). 威 is a # surname in this lexicon precisely so the first clause bites: # without the gate 威廉 would divide 威 + 廉. - lex = _LEX_TAILS.add(surnames={"威"}) + lex = _LEX_TAILS | Lexicon(surnames={"威"}) state = segment(tokenize(ParseState( original="威廉·莎士比亚先生", lexicon=lex, policy=_HAN))) assert state.interpunct_offsets @@ -753,9 +754,8 @@ def test_a_peeled_tail_is_not_a_surname_site() -> None: # then treat it as a name: 남궁 opens with the listed surname 남, # so without the post-nominal guard the stage would split the # honorific it had just created. - lex = Lexicon(surnames=frozenset({"남"}), - suffix_words=frozenset({"남궁"}), - honorific_tails=frozenset({"남궁"})) + lex = Lexicon(surnames=frozenset({"남"})).add( + "남궁", kind="suffix", attached="after") assert _texts(_run("Anderson남궁", policy=_HANGUL, lexicon=lex)) == ["Anderson", "남궁"] @@ -806,7 +806,7 @@ def test_the_peel_scans_the_name_runs_and_no_further() -> None: # step over -- "V." ends in no tail, and 씨 would stay glued. And # a later junk segment would take the peel off the person's own # name onto 박씨, peeling the wrong 씨 of two. - lex = _LEX_TAILS.add(suffix_words={"v"}) + lex = _LEX_TAILS.add("v", kind="suffix") assert _texts(_run("Dr 김민준씨, V.", policy=_HANGUL, lexicon=lex)) == ["Dr", "김", "민준", "씨", "V."] assert _texts(_run("Dr 김민준씨, Jr., 박씨", policy=_HANGUL, @@ -862,7 +862,7 @@ def test_a_wholly_suffix_run_after_a_family_comma_is_declined() -> None: # ends in no listed tail, so the peel is abandoned and さん stays # glued to 田中 (#319). The peel asks is_wholly_suffix, C1's run # predicate, and declines the run. - lex = _LEX_TAILS.add(suffix_words={"v"}) + lex = _LEX_TAILS.add("v", kind="suffix") assert _texts(_run("田中さん, V.", policy=_HANGUL, lexicon=lex)) == ["田中", "さん", "V."] @@ -876,8 +876,8 @@ def test_a_long_wholly_suffix_post_comma_run_is_declined_too() -> None: # purpose -- a scan that reached this run would step over PhD and # MD (both post-nominals by the strict test) and stop on "V.", # which ends in no tail, abandoning the peel exactly as above. - lex = _LEX_TAILS.add(suffix_words={"v"}, - suffix_acronyms={"md", "phd"}) + lex = _LEX_TAILS.add("v", kind="suffix").add( + "md", "phd", kind="credential") assert _texts(_run("田中さん, V. MD PhD", policy=_HANGUL, lexicon=lex)) == ["田中", "さん", "V.", "MD", "PhD"] @@ -895,7 +895,7 @@ def test_a_third_segment_does_not_reopen_the_declined_run() -> None: # scan it -- test_the_peel_crosses_a_family_comma_and_stops_there # above); the question here is only whether its presence changes # the answer about the SECOND. - lex = _LEX_TAILS.add(suffix_words={"v"}) + lex = _LEX_TAILS.add("v", kind="suffix") assert _texts(_run("田中さん, V., 太郎", policy=_HANGUL, lexicon=lex)) == ["田中", "さん", "V.", "太郎"] @@ -913,8 +913,8 @@ def test_a_configured_suffix_delimiter_reaches_the_peel() -> None: # admitted by plain core membership. Each peels only with its # delimiter configured; the bare-policy line under each is the # baseline that says so. - lex = _LEX_TAILS.add(suffix_words={"v"}, - suffix_acronyms={"md", "phd"}) + lex = _LEX_TAILS.add("v", kind="suffix").add( + "md", "phd", kind="credential") # not '/': a word joined by a slash or a hyphen whose every chunk # is listed reads as a suffix with no delimiter configured (#752) plus = Policy(extra_suffix_delimiters=frozenset({"+"})) @@ -948,7 +948,7 @@ def test_the_peel_crosses_a_comma_and_a_dot_at_once() -> None: # crosses both and reaches 太郎さん on the far side of the comma. # Neither gate alone pins this: keying the cross-comma reach on # interpunct-free input passes every other case in this file. - lex = _LEX_TAILS.add(surnames={"威"}) + lex = _LEX_TAILS | Lexicon(surnames={"威"}) state = segment(tokenize(ParseState( original="威廉·莎士比亚, 太郎さん", lexicon=lex, policy=_HAN))) assert state.interpunct_offsets diff --git a/tests/v2/pipeline/test_segment.py b/tests/v2/pipeline/test_segment.py index 8df9c5ef4..45e080611 100644 --- a/tests/v2/pipeline/test_segment.py +++ b/tests/v2/pipeline/test_segment.py @@ -8,11 +8,11 @@ from nameparser._types import AmbiguityKind # synthetic vocabulary: behavior given a lexicon, never default() contents -_LEX = Lexicon( - suffix_acronyms=frozenset({"phd", "md", "do", "dds", "ma", "ed"}), - suffix_acronyms_ambiguous=frozenset({"ma", "ed", "do"}), - suffix_words=frozenset({"jr", "v"}), -) +_LEX = (Lexicon() + .add("phd", "md", "do", "dds", "ma", "ed", kind="credential") + # a family reading makes the credential ambiguous + .add("ma", "ed", "do", kind="family") + .add("jr", "v", kind="suffix")) def _segmented(text: str) -> ParseState: diff --git a/tests/v2/pipeline/test_vocab.py b/tests/v2/pipeline/test_vocab.py index 3a7a70e74..883dc30bb 100644 --- a/tests/v2/pipeline/test_vocab.py +++ b/tests/v2/pipeline/test_vocab.py @@ -1,5 +1,5 @@ -import dataclasses import unicodedata +from collections.abc import Callable import pytest @@ -23,11 +23,10 @@ from nameparser._policy import (Policy, Script, _NO_INITIALS, _SCRIPT_RANGES) -_LEX = Lexicon( - suffix_acronyms=frozenset({"phd", "ma"}), - suffix_words=frozenset({"jr", "v"}), - suffix_acronyms_ambiguous=frozenset({"ma"}), -) +_LEX = (Lexicon() + .add("phd", "ma", kind="credential") + .add("ma", kind="family") # an ambiguous credential + .add("jr", "v", kind="suffix")) def test_is_initial() -> None: @@ -70,7 +69,7 @@ def test_suffix_lookup_reads_every_full_stop(word: str) -> None: # is_initial stays False on every spelling: the initial veto is a # repertoire test on the raw text (#320) and the stop set does not # touch it. - lex = Lexicon(suffix_words=frozenset({"씨"})) + lex = Lexicon().add("씨", kind="suffix") assert is_suffix_strict(word, lex) assert not is_initial(word) @@ -94,7 +93,7 @@ def test_strict_suffix_veto_skips_cjk() -> None: honorific being recognized. _normalize strips the trailing period, so '씨.' reaches the vocabulary as '씨' -- the veto was the only thing rejecting it.""" - lex = Lexicon(suffix_words=frozenset({"씨", "様"})) + lex = Lexicon().add("씨", "様", kind="suffix") assert is_suffix_strict("씨.", lex) assert is_suffix_strict("様.", lex) @@ -205,10 +204,8 @@ def test_strict_excludes_bare_ambiguous_even_when_in_acronyms() -> None: assert is_suffix_strict("M.A.", _LEX) -_SUFFIX_LEX = Lexicon( - suffix_acronyms=frozenset({"phd", "md"}), - suffix_words=frozenset({"jr", "v"}), -) +_SUFFIX_LEX = (Lexicon().add("phd", "md", kind="credential") + .add("jr", "v", kind="suffix")) def test_the_built_in_cores_are_cores_every_policy_has() -> None: @@ -258,7 +255,7 @@ def test_is_wholly_suffix() -> None: # load-bearing: it is not suffix vocabulary in ANY lexicon, so the # pair fails without the merge. ('Ph.' happens to fail alone here # too, but only because _SUFFIX_LEX is synthetic -- under - # Lexicon.default(), 'ph' IS in suffix_acronyms.) + # Lexicon.default(), 'ph' IS a listed credential.) assert is_wholly_suffix(["Ph.", "D."], lex, pol) # either period may be left off ('Ph D', 'PH D'), v1's fix_phd assert is_wholly_suffix(["Ph", "D"], lex, pol) @@ -375,7 +372,7 @@ def test_period_joined_vocab_retires_the_single_character_chunk() -> None: # itself is asserted below, not just described, so a future # vocabulary change cannot silently drift this test's premise. lex = Lexicon.default() - assert {c for c in lex.suffix_acronyms | lex.suffix_words + assert {c for c in lex._sets.suffix_acronyms | lex._sets.suffix_words if len(c) == 1 and c.isascii()} == {"2", "i", "v"} assert period_joined_vocab("R.A.I.", lex) == "shape" assert period_joined_vocab("X.Y.I.", lex) == "shape" @@ -471,7 +468,7 @@ def test_caps_shape_candidate_reads_joined_capitals( def test_caps_shape_candidate_refuses_a_listed_joined_word() -> None: # unlisted means the whole word in no wordlist, as for the unjoined # word - lex = Lexicon.default().add(surnames={"dupont-moretti"}) + lex = Lexicon.default() | Lexicon(surnames={"dupont-moretti"}) assert not caps_shape_candidate("DUPONT-MORETTI", lex, Policy(), one_case=False) assert caps_shape_candidate("DUPONT-MORETTI", Lexicon.default(), @@ -494,40 +491,67 @@ def test_ambiguous_class_candidate_admits_a_by_shape_member() -> None: def test_a_listed_dotted_entry_is_not_read_by_shape() -> None: - # Review round, 2026-09-18. A caller may list a DOTTED entry in - # `suffix_acronyms_ambiguous`, and then the whole token matches + # Review round, 2026-09-18. A caller may list a DOTTED ambiguous + # credential (one with a name reading), and then the whole token + # matches # that set while `suffix_as_written`'s period-free acronym lookup # ('ab') misses it -- so the chunk view reached the shape verdict # and called a LISTED member by-shape, which silences the case # lean everywhere downstream (`_pieces.listed_lean` declines # wherever SHAPE_ACRONYM_TAG rides). Both halves of the class # test the same membership now. - lex = Lexicon.default().add(suffix_acronyms={"a.b"}, - suffix_acronyms_ambiguous={"a.b"}) + lex = (Lexicon.default().add("a.b", kind="credential") + .add("a.b", kind="family")) assert not ambiguous_class_candidate("A.B.", lex, Policy()) # the shipped vocabulary carries no dotted ambiguous entry, so # nothing default moves assert ambiguous_class_candidate("A.B.", Lexicon.default(), Policy()) -def test_a_callers_own_conjunction_marker_keeps_its_word_a_name() -> None: - # The exclusion end to end through a CALLER's vocabulary rather - # than the shipped one: `conjunctions_ambiguous` has no subset - # check of its own (an orphan decides nothing, _lexicon's own - # note), so a caller can list a word there alone -- and the caps - # shape test must still decline it. 'John Smith ZZQ' reads suffix - # 'ZZQ' with the switch on and the default vocabulary; one - # wordlist entry is the whole difference (2026-09-18 review - # round). - on = Policy(unlisted_caps_suffixes=CapsSuffixes.EVERYWHERE) - plain = Parser(policy=on).parse("John Smith ZZQ") - assert (plain.family, plain.suffix) == ("Smith", "ZZQ") - listed = Parser( - lexicon=Lexicon.default().add(conjunctions_ambiguous={"zzq"}), - policy=on).parse("John Smith ZZQ") - assert (listed.middle, listed.family, listed.suffix) == ( - "Smith", "ZZQ", "") - assert listed.ambiguities == () + + +#: For each compiled vocabulary set, the readings that put a word into +#: it, as the calls that add them (3.0 configures by reading, #754). +#: `conjunctions_ambiguous` has none for a word of more than one +#: letter: `may_be_initial` marks a single cased letter only, which the +#: caps shape never reads, so the field's exclusion has no word to +#: reach. +_INTO_FIELD: dict[str, Callable[[Lexicon, str], Lexicon]] = { + "titles": lambda lx, w: lx.add(w, kind="title"), + "given_name_titles": lambda lx, w: lx.add( + w, kind="title", addresses="given"), + "suffix_acronyms": lambda lx, w: lx.add(w, kind="credential"), + "suffix_words": lambda lx, w: lx.add(w, kind="suffix"), + "suffix_acronyms_ambiguous": lambda lx, w: lx.add( + w, kind="credential").add(w, kind="family"), + "particles": lambda lx, w: lx.add(w, kind="particle"), + "particles_ambiguous": lambda lx, w: lx.add( + w, kind="particle").add(w, kind="given"), + "conjunctions": lambda lx, w: lx.add(w, kind="conjunction"), + "bound_given_names": lambda lx, w: lx.add(w, kind="bound_given"), + "given_name_abbreviations": lambda lx, w: lx.add( + w, kind="given", same_as="zzqxander"), + "maiden_markers": lambda lx, w: lx.add(w, kind="birth_marker"), + "surnames": lambda lx, w: lx | Lexicon(surnames={w}), + "honorific_tails": lambda lx, w: lx.add( + w, kind="suffix", attached="after"), + "after_given_titles": lambda lx, w: lx.add( + w, kind="salutation", after_given=True), + "suffix_words_ambiguous": lambda lx, w: lx.add( + w, kind="generation").add(w, kind="family"), + # the family reading first: an agnomen needs its gate (check 11) + "agnomens": lambda lx, w: lx.add(w, kind="family").add( + w, kind="agnomen"), + "titles_ambiguous": lambda lx, w: lx.add(w, kind="title").add( + w, kind="given"), + "title_modifiers": lambda lx, w: lx.add(w, kind="modifier"), + "title_styles": lambda lx, w: lx.add(w, kind="style"), + # a phrase the word heads; the given reading keeps the word itself + # out of the titles (rules/H9) + "title_phrases": lambda lx, w: lx.add( + w, kind="title", attaches=["dr"]).add(w, kind="given"), + "given_names": lambda lx, w: lx | Lexicon(given_names={w}), +} def test_the_caps_exclusion_covers_every_vocabulary_field() -> None: @@ -540,41 +564,28 @@ def test_the_caps_exclusion_covers_every_vocabulary_field() -> None: # that says "this is a family name". # # Measured per field rather than asserted about the source: each - # loop adds the same unlisted word to ONE field and checks the - # predicate declines it. A field whose exclusion is dropped fails - # here by name. + # loop gives the same unlisted word the readings that put it into + # ONE field, checks it landed there, and checks the predicate + # declines it. A field whose exclusion is dropped fails here by + # name. on = Policy(unlisted_caps_suffixes=CapsSuffixes.EVERYWHERE) base = Lexicon.default() assert caps_shape_candidate("ZZQX", base, on, False) - # H9: two fields are left out on purpose -- a title that may - # be a given name reads, outside a phrase, as the unlisted word it - # is there, and a phrase is two words -- and #789's third: a given - # name written short is read only by H2's shape; and #959's fourth, - # the given names only O6 reads - for field in ("titles_ambiguous", "title_phrases", - "given_name_abbreviations", "given_names"): - assert caps_shape_candidate( - "ZZQX", base.add(**{field: {"zzqx"}}), on, False), field - for field in _VOCAB_FIELDS: + assert set(_INTO_FIELD) == set(_VOCAB_FIELDS) - {"conjunctions_ambiguous"} + for field, into in _INTO_FIELD.items(): + lex = into(base, "zzqx") + member = "zzqx dr" if field == "title_phrases" else "zzqx" + assert member in getattr(lex._sets, field), field + # H9: two fields are left out on purpose -- a title that may + # be a given name reads, outside a phrase, as the unlisted + # word it is there, and a phrase is two words -- and #789's + # third: a given name written short is read only by H2's + # shape; and #959's fourth, the given names only O6 reads if field in ("titles_ambiguous", "title_phrases", "given_name_abbreviations", "given_names"): - continue - extra: dict[str, set[str]] = {field: {"zzqx"}} - if field in ("honorific_tails", "suffix_words_ambiguous"): - # honorific_tails ⊆ suffix_words is a Lexicon invariant, - # so this entry cannot be added alone -- which is also why - # the field was already excluded transitively, and why it - # joins the roster for completeness rather than for a - # behavior change (suffix_words_ambiguous the same) - extra["suffix_words"] = {"zzqx"} - elif field == "given_name_titles": - extra["titles"] = {"zzqx"} # ⊆ titles in practice - elif field == "particles_ambiguous": - extra["particles"] = {"zzqx"} # enforced subset - elif field == "suffix_acronyms_ambiguous": - extra["suffix_acronyms"] = {"zzqx"} # enforced subset - lex = base.add(**extra) # type: ignore[arg-type] - assert not caps_shape_candidate("ZZQX", lex, on, False), field + assert caps_shape_candidate("ZZQX", lex, on, False), field + else: + assert not caps_shape_candidate("ZZQX", lex, on, False), field def test_is_title_shaped_is_h2_s_shape_alone() -> None: @@ -598,8 +609,8 @@ def test_is_title_shaped_is_h2_s_shape_alone() -> None: # that: the same word unlisted wears the shape again assert not is_title_shaped("Chas.", lex) assert not is_title_shaped("MA.", lex) - assert is_title_shaped("Chas.", lex.remove( - given_name_abbreviations={"chas"})) + assert is_title_shaped("Chas.", lex.add( + "chas", kind="given", same_as=None)) def test_name_word_count_counts_names_not_tokens() -> None: @@ -967,10 +978,8 @@ def test_a_listed_member_written_in_period_closed_chunks_is_dotted( chunk is the single trailing period any word can wear and stays outside the gate, and the chunks must still spell the member: the gate reads the spelling, the membership test the letters.""" - lex = Lexicon( - suffix_acronyms=frozenset({"meng", "lac", "ma", "phd"}), - suffix_acronyms_ambiguous=frozenset({"meng", "lac", "ma"}), - ) + lex = (Lexicon().add("meng", "lac", "ma", "phd", kind="credential") + .add("meng", "lac", "ma", kind="family")) for text in ("M.Eng.", "m.eng.", "L.Ac.", "M.A."): assert is_suffix_strict(text, lex), text # one chunk, a missing closing period, the bare word, and chunks @@ -1003,7 +1012,10 @@ def test_two_dotted_letters_merge_only_outside_a_cased_script() -> None: # script has capitals (#872's predicate), so they merge only in a # caseless one, and only where the fold is listed, as no default # entry is - lex = Lexicon.default().add(suffix_acronyms={"jr", "พบ"}) + # (`jr` a credential here, so that its fold is listed too: a word + # holds one suffix kind, so its generation reading comes off first) + lex = Lexicon.default().remove("jr", kind="generation").add( + "jr", "พบ", kind="credential") assert split_pair("J.", "R.", lex) is None assert split_pair("พ.", "บ.", lex) == "split" assert split_pair("พ.", "บ.", Lexicon.default()) is None @@ -1038,8 +1050,8 @@ def test_the_c_level_pre_tests_are_necessary_conditions_of_split_pair( # merged whatever the vocabulary, as v1's fix_phd did assert SPLIT_PAIR_OPENS.fullmatch(first), (first, second) else: - assert (fold in lex.suffix_acronyms - or fold in lex.suffix_words), (first, second) + assert (fold in lex._sets.suffix_acronyms + or fold in lex._sets.suffix_words), (first, second) def test_cased_paired_initials_is_the_bare_shape_in_a_cased_script() -> None: @@ -1106,7 +1118,7 @@ def test_the_ph_d_pair_merges_when_both_its_words_are_listed() -> None: # #752's second review: classify's two-suffix-words refusal stood in # front of the Ph./D. answer, which merges whatever the vocabulary # lists - parser = Parser(lexicon=Lexicon.default().add(suffix_words={"ph", "d"})) + parser = Parser(lexicon=Lexicon.default().add("ph", "d", kind="suffix")) assert parser.parse("John Ph. D. Smith").suffix == "Ph. D." assert parser.parse("John Smith Ph. D. Jones").suffix == "Ph. D. Jones" @@ -1175,8 +1187,9 @@ def test_title_readings(words: list[str], head: bool, def test_a_word_a_caller_lists_as_a_title_is_one_everywhere() -> None: # the 1.4-era recipe for a title the default leaves out: listing it - # makes it a title wherever it stands, phrase or none - lex = Lexicon.default().add(titles={"dean"}) + # makes it a title wherever it stands, phrase or none; since 3.0 + # that is taking its given reading away (rules/H9) + lex = Lexicon.default().remove("dean", kind="given") name = Parser(lexicon=lex).parse("Dean Smith") assert (name.title, name.family, name.ambiguities) == ("Dean", "Smith", ()) @@ -1186,8 +1199,8 @@ def test_a_contested_title_in_a_phrase_addresses_as_its_vocabulary_says() -> Non # (no shipped word is both yet) takes the given-name title tag once a # partner makes it a title, so H1 reads the one name word after it as # the given name, as 'Sir John' reads - lex = Lexicon.default().add(titles_ambiguous={"zed"}, - given_name_titles={"zed"}) + lex = (Lexicon.default().add("zed", kind="title", addresses="given") + .add("zed", kind="given")) phrase = Parser(lexicon=lex).parse("Hon Zed John") assert (phrase.title, phrase.given, phrase.family) == ("Hon Zed", "John", "") # the tag classify documents for a given-name title, which a word @@ -1197,20 +1210,29 @@ def test_a_contested_title_in_a_phrase_addresses_as_its_vocabulary_says() -> Non assert (alone.title, alone.given, alone.family) == ("", "Zed", "John") -@pytest.mark.parametrize("phrases, reports", [ - (frozenset(), False), +def _without_title_phrases(lex: Lexicon) -> Lexicon: + """`lex` with every title reading's `attaches` taken off, so it + reads no two-word title phrase.""" + for head in sorted({p.split()[0] for p in lex._sets.title_phrases}): + for r in lex.readings_of(head): + if dict(r.properties).get("attaches"): + lex = lex.add(head, kind=r.kind, attaches=None) + return lex + + +@pytest.mark.parametrize("attaches, reports", [ + (None, False), # a title before the word: classify reads the pair, so the word may # head a title of itself and opening a name it reports - (frozenset({"chief grace"}), True), - # 'your' is no title, so classify never reads the pair, and a phrase - # it never reads makes no word report - (frozenset({"your grace"}), False), - # classify reads two-word phrases only - (frozenset({"chief of grace"}), False), + (("chief", "nobility", ["grace"]), True), ]) def test_only_a_phrase_classify_can_read_makes_a_contested_word_report( - phrases: frozenset[str], reports: bool) -> None: - lex = dataclasses.replace(Lexicon.default(), title_phrases=phrases) + attaches: tuple[str, str, list[str]] | None, reports: bool) -> None: + lex = _without_title_phrases(Lexicon.default()) + assert not lex._sets.title_phrases + if attaches is not None: + head, kind, partners = attaches + lex = lex.add(head, kind=kind, attaches=partners) kinds = [a.kind.value for a in Parser(lexicon=lex).parse( "Grace Smith").ambiguities] assert kinds == (["title-or-name"] if reports else []) diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index e7729145b..3f57493c7 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -57,8 +57,8 @@ "R5": {"words": 250, "then_words": 154, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "R6": {"words": 400, "then_words": 265, "because_sentences": 2, "except": 1, "precedence": 0, "accepted": 5, "accepted_words": 289}, "R7": {"words": 142, "then_words": 30, "because_sentences": 2, "except": 0, "precedence": 1, "accepted": 2, "accepted_words": 74}, - "R8": {"words": 163, "then_words": 61, "because_sentences": 3, "except": 1, "precedence": 0, "accepted": 7, "accepted_words": 323}, - "R9": {"words": 398, "then_words": 123, "because_sentences": 3, "except": 5, "precedence": 0, "accepted": 13, "accepted_words": 459}, + "R8": {"words": 163, "then_words": 61, "because_sentences": 3, "except": 1, "precedence": 0, "accepted": 7, "accepted_words": 329}, + "R9": {"words": 397, "then_words": 122, "because_sentences": 3, "except": 5, "precedence": 0, "accepted": 13, "accepted_words": 477}, "R10": {"words": 168, "then_words": 26, "because_sentences": 2, "except": 2, "precedence": 0, "accepted": 6, "accepted_words": 160}, "S1": {"words": 306, "then_words": 26, "because_sentences": 1, "except": 2, "precedence": 3, "accepted": 0, "accepted_words": 0}, "S2": {"words": 658, "then_words": 120, "because_sentences": 1, "except": 0, "precedence": 7, "accepted": 3, "accepted_words": 222}, diff --git a/tests/v2/rules_doc.py b/tests/v2/rules_doc.py index a0b6c66b1..b28d18b50 100644 --- a/tests/v2/rules_doc.py +++ b/tests/v2/rules_doc.py @@ -182,7 +182,7 @@ def has_boundary_or_waiver(self) -> bool: #: is the policy's name above plus the fragment's, ``es_given_names`` #: naming the list ``nameparser/locales/_es_given_names.py`` ships, so #: a rule names the data as it names the switch. -_ES_GIVEN = Lexicon.default().add(given_names=GIVEN_NAMES) +_ES_GIVEN = Lexicon.default() | Lexicon(given_names=GIVEN_NAMES) CONFIGS: dict[str, tuple[Policy, Lexicon]] = { "two_family_names+es_given_names": ( POLICIES["two_family_names"], _ES_GIVEN), diff --git a/tests/v2/test_contracts.py b/tests/v2/test_contracts.py index db36f47cf..54d1a5a26 100644 --- a/tests/v2/test_contracts.py +++ b/tests/v2/test_contracts.py @@ -220,8 +220,8 @@ def test_the_documented_replacements_for_an_in_place_edit_work() -> None: do INSTEAD of ``TITLES.add("dean")`` was the one claim about the freeze with nothing checking it. "dean" is the canonical example for this: a common academic title and a common given name, so it is - deliberately absent from the shipped ``TITLES`` and a caller who - wants it has to add it themselves. + deliberately not a title outside a title phrase in the shipped + vocabulary, and a caller who wants it everywhere has to say so. """ from nameparser import Lexicon, Parser, parse from nameparser.config.titles import TITLES @@ -231,8 +231,11 @@ def test_the_documented_replacements_for_an_in_place_edit_work() -> None: TITLES.add("dean") # type: ignore[attr-defined] assert parse("Dean Smith").title == "" - # the recipe: an extended Lexicon - parser = Parser(lexicon=Lexicon.default().add(titles={"dean"})) + # the recipe: a configured Lexicon. "dean" holds a title reading + # and a given one, so it reads as a title only inside a title + # phrase (rules/H9); removing the given reading makes it a title + # everywhere + parser = Parser(lexicon=Lexicon.default().remove("dean", kind="given")) assert parser.parse("Dean Smith").title == "Dean" diff --git a/tests/v2/test_es_given_names.py b/tests/v2/test_es_given_names.py index a9d89c357..9bbc1bb94 100644 --- a/tests/v2/test_es_given_names.py +++ b/tests/v2/test_es_given_names.py @@ -53,10 +53,10 @@ def _non_name_readings(lexicon: Lexicon) -> dict[str, frozenset[str]]: too, `del` and `da` among them, which the tool also leaves out), a title (an ambiguous one too), a suffix and a conjunction.""" fields = { - "particle": lexicon.particles, - "title": lexicon.titles | lexicon.titles_ambiguous, - "suffix": lexicon.suffix_acronyms | lexicon.suffix_words, - "conjunction": lexicon.conjunctions, + "particle": lexicon._sets.particles, + "title": lexicon._sets.titles | lexicon._sets.titles_ambiguous, + "suffix": lexicon._sets.suffix_acronyms | lexicon._sets.suffix_words, + "conjunction": lexicon._sets.conjunctions, } return {kind: frozenset(given_name_key(w) for w in words) for kind, words in fields.items()} diff --git a/tests/v2/test_layering.py b/tests/v2/test_layering.py index 09dcaeb3e..b0d919768 100644 --- a/tests/v2/test_layering.py +++ b/tests/v2/test_layering.py @@ -56,8 +56,11 @@ # are skipped by _nameparser_imports and need no entry. "_types.py": ("nameparser._render", "nameparser._parser"), # intent: DATA modules only -- mechanically this admits any config - # submodule, and since 3.0 the package holds nothing else - "_lexicon.py": ("nameparser.config.",), + # submodule, and since 3.0 the package holds nothing else; and the + # reading index's reader and compiler, at call time only, since + # tools/vocab/build.py loads this file by path (3.0's 4i-b: a + # Lexicon compiles its readings) + "_lexicon.py": ("nameparser.config.", "nameparser._reading_index"), "_policy.py": ("nameparser._types",), # the shared compiler and the reading index's reader (#753, 3a): the # index, a config DATA module, and nothing else. That the import is @@ -89,12 +92,10 @@ # ParseState. Widening this past _state is the tell that a text- # level predicate has grown a dependency on a stage. # _reading_index joined in 3.0's 4b for a listed suffix word's - # kind, a fact about the word's text no compiled set records - # (`_vocab.suffix_kind`), and it imports nothing internal at - # module level. + # kind and left in 4i-b, the lexicon compiling the kind into sets + # of its own (Lexicon._sets.generation_words and its siblings). "_pipeline/_vocab.py": ("nameparser._lexicon", "nameparser._policy", - "nameparser._pipeline._state", - "nameparser._reading_index"), + "nameparser._pipeline._state"), "_pipeline/_extract.py": _PIPELINE_STAGE_ALLOWED, "_pipeline/_tokenize.py": _PIPELINE_STAGE_ALLOWED, "_pipeline/_script_segment.py": _PIPELINE_STAGE_ALLOWED, @@ -353,7 +354,9 @@ def test_every_frozen_dataclass_carries_the_pickle_guards() -> None: # never pickled: a Lexicon pickles its fields and # rebuilds this from them at load (#754, 4i-a) continue - if cls.__name__ == "Lexicon": + if cls.__name__ in ("Lexicon", "LexiconReading"): + # each keeps a copy of the guard, layering keeping + # _lexicon import-free of _types (LexiconReading, 4i-b) assert "__getstate__" in cls.__dict__ assert "__setstate__" in cls.__dict__ continue @@ -606,13 +609,15 @@ def test_the_check_finds_a_tool_read_through_an_alias( #: The reads 4i-a moved, as the check counts them on today's sources #: with `._sets` taken out again: the recorded negative control over #: the real tree, so the check above cannot pass because its walk -#: reaches nothing. The library's 114 are 106 attribute loads, 5 roster -#: `getattr`s (3 of them through `sets`, which the undone `sets = -#: lexicon` makes a lexicon) and the 3 `sets =` bindings; the tools' -#: 16 are attribute loads, every tool scanned. +#: reaches nothing. The library's 112 are 106 attribute loads, 3 roster +#: `getattr`s, all through `sets`, which the undone `sets = lexicon` +#: makes a lexicon, and the 3 `sets =` bindings; the tools' 16 are +#: attribute loads, every tool scanned. 4i-a's count was 114, with two +#: more `getattr`s in `_parser._apply_choice`, which reads the sets +#: through `_lexicon._fields_of` since 4i-b. #: A reader added through `_sets` moves these, and they are re-taken #: then: they count what the check would refuse with the move undone. -_UNGUARDED_READS = {"library": 114, "tools": 16} +_UNGUARDED_READS = {"library": 112, "tools": 16} def test_the_check_refuses_every_read_4i_a_moved() -> None: diff --git a/tests/v2/test_lexicon.py b/tests/v2/test_lexicon.py index 5b0719946..b46f855b9 100644 --- a/tests/v2/test_lexicon.py +++ b/tests/v2/test_lexicon.py @@ -2,6 +2,8 @@ import dataclasses import pickle import re +import subprocess +import sys import unicodedata import warnings from collections.abc import Callable @@ -10,56 +12,63 @@ import pytest -from nameparser import Parser +from nameparser import LexiconReading, Parser from nameparser._lexicon import ( - Lexicon, _ADDED_IN, _MaskValueError, _PHRASE_FIELDS, _VOCAB_FIELDS, - _default_lexicon, _normalize, _title_key, + Lexicon, _ADDED_IN, _KIND_FIELDS, _LIST_FIELDS, _MaskValueError, + _SET_FIELDS, _VOCAB_FIELDS, _default_lexicon, _fields_of, _normalize, + _title_key, ) from nameparser._policy import Script, _SCRIPT_RANGES from nameparser.config.suffixes import SUFFIX_ACRONYMS, SUFFIX_WORDS -def test_entries_are_normalized_at_construction() -> None: - lex = Lexicon(titles=frozenset({"Dr.", "MR"})) - assert lex.titles == frozenset({"dr", "mr"}) +def _quiet(lexicon: Callable[[], Lexicon]) -> Lexicon: + """A lexicon built where the title warning would fire, the warning + being no part of what the test asks.""" + with warnings.catch_warnings(): + warnings.simplefilter("ignore", UserWarning) + return lexicon() + + +# -- construction and normalization ---------------------------------------- + +def test_words_are_normalized_as_they_are_stored() -> None: + lex = Lexicon.empty().add("Dr.", "MR", kind="title") + assert lex._sets.titles == frozenset({"dr", "mr"}) + assert [r.text for r in lex.readings_of("DR.")] == ["dr"] def test_default_sources_v1_vocabulary() -> None: - lex = Lexicon.default() - assert "dr" in lex.titles - assert "van" in lex.particles - assert "phd" in lex.suffix_acronyms + sets = Lexicon.default()._sets + assert "dr" in sets.titles + assert "van" in sets.particles + assert "phd" in sets.suffix_acronyms # flipped model: 'dos' is never-given in v1, so NOT ambiguous here - assert "dos" in lex.particles and "dos" not in lex.particles_ambiguous - assert "van" in lex.particles_ambiguous + assert "dos" in sets.particles and "dos" not in sets.particles_ambiguous + assert "van" in sets.particles_ambiguous # the policy is 'e' and not 'y' (decisions/P8): a bare E initial is # common, where 'y' between two surnames is the commonest Hispanic compound - assert "e" in lex.conjunctions_ambiguous and "y" not in lex.conjunctions_ambiguous + assert "e" in sets.conjunctions_ambiguous + assert "y" not in sets.conjunctions_ambiguous # CAPITALIZATION_EXCEPTIONS maps 'phd' -> 'PhD', a case mask since # #459 (verbatim, not normalized -- only keys are lowercased and # period-stripped at construction; values pass through unchanged). - assert lex.capitalization_exceptions_map["phd"] == "PhD" - # maiden markers source from the same data-module pattern (#274); - # non-colliding Cyrillic entries live in the default per the locales - # design's sorting rule, and both ё/е spellings are listed because - # case normalization does not fold them. - assert "geborene" in lex.maiden_markers - assert "урожденная" in lex.maiden_markers and "урождённая" in lex.maiden_markers + assert Lexicon.default().capitalization_exceptions_map["phd"] == "PhD" + # maiden markers source from the same data-module pattern (#274), + # both ё/е spellings listed because case normalization does not + # fold them + assert "geborene" in sets.maiden_markers + assert {"урожденная", "урождённая"} <= sets.maiden_markers # #269 fidelity: entries whose spelling casefold() would mutate must - # arrive in the lexicon exactly as authored in the data modules - # (final sigma intact, ß intact) -- the review that caught 'κοσ'. - assert "κος" in lex.titles and "κοσ" not in lex.titles - assert "großfürst" in lex.titles and "grossfürst" not in lex.titles + # arrive exactly as authored (final sigma intact, ß intact) + assert "κος" in sets.titles and "κοσ" not in sets.titles + assert "großfürst" in sets.titles and "grossfürst" not in sets.titles def test_default_is_cached_single_instance() -> None: assert Lexicon.default() is Lexicon.default() -def test_particles_ambiguous_must_be_subset_of_particles() -> None: - with pytest.raises(ValueError, match="subset"): - Lexicon(particles_ambiguous=frozenset({"van"})) - def test_capitalization_exceptions_canonical_and_no_aliasing() -> None: exceptions = {"phd": "PhD", "ii": "II"} @@ -72,28 +81,6 @@ def test_capitalization_exceptions_canonical_and_no_aliasing() -> None: assert lex2 == lex3 and hash(lex2) == hash(lex3) -def test_lexicon_is_hashable() -> None: - assert isinstance(hash(Lexicon.default()), int) - - -def test_lexicon_rejects_bare_string_vocab() -> None: - with pytest.raises(TypeError, match="bare string"): - Lexicon(titles="dr") # type: ignore[arg-type] - - -def test_lexicon_rejects_non_str_vocab_entries() -> None: - with pytest.raises(TypeError, match="entries must be strings"): - Lexicon(titles={"Dr.", 42}) # type: ignore[arg-type] - - -def test_lexicon_rejects_non_iterable_vocab_field() -> None: - # A non-iterable (an int) used to fall through to tuple(entries) - # and surface as a bare "'int' object is not iterable" with no - # hint which field was wrong. - with pytest.raises(TypeError, match="titles.*iterable"): - Lexicon(titles=5) # type: ignore[arg-type] - with pytest.raises(TypeError, match="titles.*iterable"): - Lexicon.empty().add(titles=5) # type: ignore[arg-type] def test_capitalization_exceptions_rejects_non_iterable() -> None: @@ -101,17 +88,6 @@ def test_capitalization_exceptions_rejects_non_iterable() -> None: Lexicon(capitalization_exceptions=5) # type: ignore[arg-type] -def test_entries_normalizing_to_empty_raise() -> None: - # "." or "" is a data bug (stray split artifact, empty CSV cell); - # dropping it silently would also let a future data-module typo - # vanish instead of failing CI. One rule for vocab AND exception keys. - with pytest.raises(ValueError, match="normalizes to empty"): - Lexicon(titles=frozenset({"Dr.", "."})) - with pytest.raises(ValueError, match="normalizes to empty"): - Lexicon.empty().add(titles={" "}) - with pytest.raises(ValueError, match="normalizes to empty"): - Lexicon(capitalization_exceptions=(("...", "X"), ("phd", "PhD"))) - def test_colliding_exception_keys_dedupe_last_wins() -> None: # 'Phd.' and 'phd' collide under edge-period normalization @@ -282,83 +258,6 @@ def test_mask_value_error_survives_pickle_and_copy() -> None: assert restored.__notes__ == ["seen in worker 3"] -def test_add_and_remove_return_new_lexicons() -> None: - # "zqtitle" is a synthetic word absent from v1's TITLES data (unlike - # e.g. "dra", the feminine "dr." abbreviation, which is already there). - base = Lexicon.default() - lex = base.add(titles={"zqtitle"}).remove(suffix_words={"esquire"}) - assert "zqtitle" in lex.titles and "zqtitle" not in base.titles - # precondition, or the removal assertion passes vacuously (this is a - # guard for the operation under test, not a vocabulary content pin) - assert "esquire" in base.suffix_words - assert "esquire" not in lex.suffix_words - - -def test_add_unknown_field_raises_with_valid_names() -> None: - with pytest.raises(TypeError, match="prefixes"): - Lexicon.default().add(prefixes={"van"}) # v1 name: helpful error - - -def test_add_capitalization_exceptions_raises_pointing_at_replace() -> None: - # The offered spelling is a tuple, not a dict: the field is - # annotated as one, so a pasted dict fails mypy (AGENTS.md). And - # it must RUN as pasted: the value is a case mask of its key, so - # ('key', 'Value') type-checks and then raises (2.4 release sweep). - offered = re.escape( - "dataclasses.replace(lexicon, capitalization_exceptions=" - "(*lexicon.capitalization_exceptions, ('key', 'KEY')))") - with pytest.raises(TypeError, match=offered) as raised: - Lexicon.default().add(capitalization_exceptions={"x": "X"}) - code = re.search(r"dataclasses\.replace\(.*?\)\)\)", - str(raised.value)) - assert code is not None - pasted = eval(code[0], {"dataclasses": dataclasses, - "lexicon": Lexicon.default()}) - assert pasted.capitalization_exceptions_map["key"] == "KEY" - - -def test_union_is_fieldwise_and_right_biased_for_exceptions() -> None: - a = dataclasses.replace(Lexicon.empty(), - capitalization_exceptions=(("phd", "PhD"),)) - a = a.add(titles={"dr"}) - b = dataclasses.replace(Lexicon.empty(), - capitalization_exceptions=(("phd", "Ph.D."),)) - b = b.add(titles={"mr"}) - u = a | b - assert u.titles == frozenset({"dr", "mr"}) - assert u.capitalization_exceptions_map["phd"] == "Ph.D." # right wins - - -def test_remove_breaking_subset_invariant_raises() -> None: - lex = Lexicon(particles=frozenset({"van"}), particles_ambiguous=frozenset({"van"})) - with pytest.raises(ValueError, match="subset"): - lex.remove(particles={"van"}) # would orphan particles_ambiguous - - -def test_pickle_round_trip_preserves_equality_and_cap_map() -> None: - # _cap_map holds a MappingProxyType, which pickle rejects; Lexicon - # must round-trip anyway because Parser is picklable by construction - # (spec: 2026-07-11-v2-core-api-design.md) and a Parser holds a Lexicon. - import pickle - - for lex in (Lexicon.default(), - Lexicon.empty().add(titles={"Dr."})): - loaded = pickle.loads(pickle.dumps(lex)) - assert loaded == lex - assert hash(loaded) == hash(lex) - assert (loaded.capitalization_exceptions_map - == lex.capitalization_exceptions_map) - - -def test_lexicon_rejects_mapping_for_plain_vocab_field() -> None: - # A dict here almost always means the caller confused the field with - # capitalization_exceptions; silently keeping just the keys would be - # the lone quiet coercion on an otherwise fail-loud surface. - with pytest.raises(TypeError, match="mapping"): - Lexicon(titles={"Dr.": "Doctor"}) # type: ignore[arg-type] - with pytest.raises(TypeError, match="mapping"): - Lexicon.empty().add(titles={"Dr.": "Doctor"}) - def test_capitalization_exceptions_rejects_malformed_shapes() -> None: with pytest.raises(TypeError, match="bare string"): @@ -371,295 +270,53 @@ def test_capitalization_exceptions_rejects_malformed_shapes() -> None: Lexicon(capitalization_exceptions=("ab",)) # type: ignore[arg-type] -def test_setstate_rejects_mismatched_field_layout() -> None: - # A pickle from a different Lexicon version (a field added without - # the ADDED_IN marker, or renamed) - # must fail at load time with a message naming the mismatch, not at - # some later attribute read far from the unpickle site. - lex = Lexicon.empty() - good_state = lex.__getstate__() - missing = dict(good_state) - del missing["titles"] - with pytest.raises(ValueError, match="titles"): - Lexicon.__new__(Lexicon).__setstate__(missing) - extra = dict(good_state) - extra["zq_future_field"] = frozenset() - with pytest.raises(ValueError, match="zq_future_field"): - Lexicon.__new__(Lexicon).__setstate__(extra) - - -def test_lexicon_marks_an_added_field_with_the_shared_key() -> None: - # one key for every guard: a Lexicon field marked with the key the - # other types read, and the other way round - from nameparser import _types - assert _ADDED_IN == _types.ADDED_IN - - -_UNSET = object() - - -def _mark(monkeypatch: pytest.MonkeyPatch, name: str, - default: object = _UNSET) -> None: - """Give a shipped Lexicon field the `_ADDED_IN` marker, as a field - added in a later release would carry, and optionally another - default. Patches the Field object `dataclasses.fields` returns, so - only the pickle guard sees it.""" - f = Lexicon.__dataclass_fields__[name] - monkeypatch.setattr(f, "metadata", - MappingProxyType({**f.metadata, _ADDED_IN: "9.9"})) - if default is not _UNSET: - monkeypatch.setattr(f, "default", default) - - -def _lexicon_state(*without: str, **renamed: str) -> dict[str, object]: - state = Lexicon.default().__getstate__() - for name in without: - del state[name] - for old, new in renamed.items(): - state[new] = state.pop(old) - return state - - -#: (field marked, what loads in its place, the refusal the same state -#: meets with the field unmarked). The third column is the negative -#: control: the guard before it learned the marker refused exactly -#: this, so a widening that stopped working fails the admitted half. -_ADMITTED = [ - ("title_phrases", frozenset(), - "missing fields: title_phrases; unexpected fields: none"), - ("capitalization_exceptions", (), - "missing fields: capitalization_exceptions; unexpected fields: none"), -] - - -@pytest.mark.parametrize("name,loaded,unmarked", _ADMITTED, - ids=[row[0] for row in _ADMITTED]) -def test_setstate_admits_a_marked_field_missing_from_an_older_pickle( - monkeypatch: pytest.MonkeyPatch, name: str, loaded: object, - unmarked: str) -> None: - state = _lexicon_state(name) - with pytest.raises(ValueError, match=re.escape(unmarked)): - Lexicon.__new__(Lexicon).__setstate__(dict(state)) - _mark(monkeypatch, name) - restored = Lexicon.__new__(Lexicon) - restored.__setstate__(dict(state)) - assert getattr(restored, name) == loaded - assert restored == dataclasses.replace(Lexicon.default(), **{name: loaded}) # type: ignore[arg-type] - assert restored.capitalization_exceptions_map == dict( - restored.capitalization_exceptions) - - -#: (marks, state, refusal): every skew the guard refused before it -#: learned the marker, still refused beside a marked field, the rows of -#: tests/v2/test_types.py's _REFUSED_SKEWS for the shared guard. -_LEXICON_REFUSED = [ - pytest.param({"title_phrases": _UNSET}, lambda: _lexicon_state("titles"), - "missing fields: titles; unexpected fields: none", - id="missing-unmarked-field"), - pytest.param({"title_phrases": _UNSET}, - lambda: _lexicon_state("titles", "title_phrases"), - "missing fields: titles; unexpected fields: none", - id="missing-unmarked-field-beside-a-missing-marked-one"), - pytest.param({"title_phrases": _UNSET}, - lambda: {**_lexicon_state(), "zq_future": frozenset()}, - "missing fields: none; unexpected fields: zq_future", - id="unexpected-field"), - pytest.param({"title_phrases": _UNSET}, - lambda: {**_lexicon_state("title_phrases"), - "zq_future": frozenset()}, - "missing fields: none; unexpected fields: zq_future", - id="unexpected-field-beside-a-missing-marked-one"), - pytest.param({"title_phrases": _UNSET}, - lambda: _lexicon_state(titles="title"), - "missing fields: titles; unexpected fields: title", - id="renamed-field"), - pytest.param({"title_phrases": _UNSET}, - lambda: _lexicon_state(title_phrases="title_phrase"), - "missing fields: none; unexpected fields: title_phrase", - id="renamed-marked-field"), - # a marker with no default to load is no license: the field is - # still required, as in the shared guard - pytest.param({"title_phrases": dataclasses.MISSING}, - lambda: _lexicon_state("title_phrases"), - "missing fields: title_phrases; unexpected fields: none", - id="marked-field-without-a-default"), - # the default is validated like a value the pickle held: one that - # does not survive normalization is refused, not silently rewritten - pytest.param({"titles": frozenset({"DR."})}, - lambda: _lexicon_state("titles"), - "default is not normalized (titles: DR.)", - id="marked-field-whose-default-is-not-normalized"), -] - - -@pytest.mark.parametrize("marks,state,match", _LEXICON_REFUSED) -def test_setstate_still_refuses_every_skew_beside_a_marked_field( - monkeypatch: pytest.MonkeyPatch, marks: dict[str, object], - state: Callable[[], dict[str, object]], match: str) -> None: - for name, default in marks.items(): - _mark(monkeypatch, name, default) - with pytest.raises(ValueError, match=re.escape(match)): - Lexicon.__new__(Lexicon).__setstate__(state()) - - -def test_a_lexicon_parser_and_locale_pickled_by_2_8_0_load() -> None: - """tests/v2/data/lexicon_v28.pickle is a dict of - `Lexicon.empty().add(titles={"dean"})`, a `Parser` holding it and - `locales.RU`, pickled (protocol 4) by the released 2.8.0 wheel. - Made 2026-10-30 with `PYTHONSAFEPATH=1 uv run --isolated - --no-project --with 'nameparser==2.8.0'` from a directory outside - the checkout, version and module path asserted (AGENTS.md's v1 - comparison trap). A field a later release adds to Lexicon takes - `_ADDED_IN` and a default, or this fails. pickle.load is safe - here: a repo-controlled fixture. - - What it holds is that each object LOADS: only the Lexicon, built - from words named here, is compared whole. The pack and the policy - are checked by type and code, since whole equality with - `locales.RU` or `Parser().policy` would fail on any 3.0 change to - their content, which is no layout skew and no reason to - re-record.""" - from nameparser import Locale, Policy - with open(Path(__file__).parent / "data" / "lexicon_v28.pickle", - "rb") as f: - loaded = pickle.load(f) - lex = Lexicon.empty().add(titles={"dean"}) - assert loaded["lexicon"] == lex - assert loaded["parser"].lexicon == lex - assert isinstance(loaded["parser"].policy, Policy) - assert isinstance(loaded["locale"], Locale) - assert loaded["locale"].code == "ru" - assert isinstance(loaded["locale"].lexicon, Lexicon) - # #959's given_names, added in 3.0 and marked, loads empty: the - # default an older pickle means, no pack having filled it in 2.8 - for loaded_lexicon in (loaded["lexicon"], loaded["parser"].lexicon, - loaded["locale"].lexicon): - assert loaded_lexicon.given_names == frozenset() - - -def test_given_names_round_trips() -> None: - lex = Lexicon.default().add(given_names={"luis", "José"}) - assert pickle.loads(pickle.dumps(lex)) == lex - - -def test_given_names_loads_from_2_8_0_only_by_its_marker( - monkeypatch: pytest.MonkeyPatch) -> None: - # The negative control for the test above: the same 2.8.0 pickle - # with given_names' _ADDED_IN marker taken off is refused, so the - # load above is the marker's doing (#959). - from dataclasses import fields - f = next(f for f in fields(Lexicon) if f.name == "given_names") - assert _ADDED_IN in f.metadata - monkeypatch.setattr(f, "metadata", MappingProxyType( - {k: v for k, v in f.metadata.items() if k != _ADDED_IN})) - with open(Path(__file__).parent / "data" / "lexicon_v28.pickle", - "rb") as fh, pytest.raises(ValueError, match="given_names"): - pickle.load(fh) - - -def test_normalization_lowercases_and_keeps_interior_periods() -> None: - # v1's lc() semantics: lowercase, EDGE periods trimmed, interior - # periods KEPT ('J.R.' must not collapse to 'jr' and hit the - # periodless vocabulary -- v1 parity, pinned live 2026-07-17). - # Suffix-ACRONYM membership alone strips periods (see - # _vocab.suffix_as_written). - lex = Lexicon(titles=frozenset({"STRAßE", "Ph.D", "Dr."})) - assert lex.titles == frozenset({"straße", "ph.d", "dr"}) - - -def test_normalization_preserves_spellings_casefold_would_mutate() -> None: - # lower(), not casefold(): casefold's caseless-matching folds would - # rewrite stored vocabulary -- Greek final sigma flattens ('κος' -> - # the misspelling 'κοσ') and German ß expands ('großfürst' -> - # 'grossfürst'). lower() follows Unicode SpecialCasing contextually - # ('ΚΟΣ' -> 'κος') and keeps entries as authored, matching v1's lc(). - lex = Lexicon(titles=frozenset({"ΚΟΣ", "Großfürst"})) - assert lex.titles == frozenset({"κος", "großfürst"}) - - -def test_suffix_ambiguous_must_be_subset_of_acronyms() -> None: - # same invariant as particles_ambiguous <= particles: the ambiguous - # set marks a subset of suffix_acronyms, and classify's period gate - # depends on the membership tests agreeing about it - with pytest.raises(ValueError, match="subset"): - Lexicon(suffix_acronyms_ambiguous=frozenset({"ma"})) - - -def test_an_ambiguous_suffix_word_may_not_be_an_unambiguous_acronym() -> None: - # the mirror of the gate-bypass raise: the unambiguous acronym - # branch claims a word outright, so a gated suffix word listed there - # loses its count ('Ana Neto' would read suffix 'Neto', rules/S11) - with pytest.raises(ValueError, match="bypasses the count"): - Lexicon.default().add(suffix_acronyms={"neto"}) +def test_multiword_capitalization_key_warns() -> None: + with pytest.warns(UserWarning, match="matched one word at a time"): + dataclasses.replace( + Lexicon.empty(), + capitalization_exceptions=(("zqx zqy", "ZqXZqY"),)) -def test_the_shipped_suffix_sets_are_disjoint() -> None: - # the vocabulary build refuses the overlap (tools/vocab/build.py, - # run over the shipped files by tests/v2/test_vocab.py), but over the - # TOML, not over the constants a running process holds, so the - # invariant over the shipped constants gets a named test as well. The two - # sets normalize differently -- the word test strips only edge - # periods, the acronym test strips all of them -- so a word in both - # is matched by two rules and which one fired is unreadable from - # outside. The last overlap, 'esq', left the acronyms 2026-09-08 - # (decisions/suffix-acronym-collisions). - # - # The SHIPPED sets only: Lexicon has no such invariant, and a - # caller who wants the overlap in their own vocabulary keeps it. - assert SUFFIX_ACRONYMS & SUFFIX_WORDS == frozenset() - default = Lexicon.default() - assert default.suffix_acronyms & default.suffix_words == frozenset() +def test_a_multiword_key_skips_the_mask_check_entirely() -> None: + """#459 review: the mask check ran before the multi-word-key + warning, so a key that can never match -- 'ph d' has no word in + capitalized()'s per-word lookup -- still raised over a value that + does not spell it (AGENTS.md: an invariant guards harm, and an + unreachable entry does none). Reordered so a multi-word key only + warns, storing its value verbatim regardless of the mask + question.""" + with pytest.warns(UserWarning, match="matched one word at a time"): + lex = dataclasses.replace( + Lexicon.empty(), + capitalization_exceptions=(("ph d", "Doctor"),)) + assert lex.capitalization_exceptions_map == {"ph d": "Doctor"} -def test_given_name_titles_may_hold_a_multi_word_phrase() -> None: - # given_name_titles is looked up against the SPACE-JOINED title run - # (_pipeline/_post_rules.py), not per token, so "grand duke" is a - # legitimate entry. Its words must be titles -- that is what makes - # the run a title in the first place -- but the phrase itself is - # not, and must not be required to be. - lex = Lexicon(titles=frozenset({"grand", "duke"}), - given_name_titles=frozenset({"grand duke"})) - assert "grand duke" in lex.given_name_titles -@pytest.mark.parametrize("spelling", ["lt. col", "lt col", "Lt. Col."]) -def test_given_name_titles_folds_per_word_so_abbreviations_match( - spelling: str) -> None: - # The key is built per word and rejoined, so an abbreviated entry - # has to fold the same way the parse folds the title run. Storing - # 'lt. col' verbatim kept the interior period and matched nothing -- - # a silent no-op on the config surface, the failure this field is - # most prone to (it has no validation by design). - # - # It is the whole-run arm that matches here, and only that arm: - # `col` alone is not a given-name title, so #489's last-word arm - # must not have replaced the phrase lookup. - base = Lexicon.default() - lex = dataclasses.replace( - base, - titles=base.titles | {"lt", "col"}, - given_name_titles=base.given_name_titles | {spelling}) - assert "lt col" in lex.given_name_titles - assert "col" not in lex.given_name_titles - assert Parser(lexicon=lex).parse("Lt. Col. Smith").given == "Smith" +def test_default_lexicon_builds_warning_free() -> None: + # Would have caught the eight dead entries the previous commit + # repaired. + _default_lexicon.cache_clear() + try: + with warnings.catch_warnings(): + warnings.simplefilter("error") + Lexicon.default() + finally: + _default_lexicon.cache_clear() -def test_the_shipped_given_name_titles_are_every_one_a_single_word() -> None: - # The invariant that makes the whole-run arm dead for the shipped - # vocabulary: it can only ever match a one-word run, which the - # last-word arm reads the same way (#489). A phrase entry here - # would give that arm reach the docstring says it has not. - assert not any(" " in t for t in Lexicon.default().given_name_titles) - - -def test_a_run_matches_by_its_last_word_when_the_whole_run_does_not() -> None: - # The other arm's premise, on the shipped vocabulary: 'dr sir' is - # no entry and never could be, and `sir` is the run's last word - # (#489). What the parse then reads is the `Dr. Sir John` case - # row's, and is not asserted twice. +def test_default_lexicon_ships_korean_surnames_only() -> None: lex = Lexicon.default() - assert "dr sir" not in lex.given_name_titles - assert "sir" in lex.given_name_titles + assert "김" in lex.surnames and "남궁" in lex.surnames + # Han surnames are locales.ZH's cargo (segmentation opt-in), so + # the DEFAULT set is wholly hangul -- structural check, not + # content-pinning. The spans come from the shared table rather than + # a second hand-copy of 0xAC00/0xD7A3 (test_locales.py's precedent). + hangul = _SCRIPT_RANGES[Script.HANGUL] + assert all(all(any(lo <= ord(c) <= hi for lo, hi in hangul) for c in s) + and 1 <= len(s) <= 2 for s in lex.surnames) + @pytest.mark.parametrize("entry", ["lt .", "lt . col", ". col", ". ."]) @@ -672,124 +329,6 @@ def test_given_name_titles_fold_is_a_fixed_point(entry: str) -> None: assert once == once.strip() and " " not in once -def test_a_title_word_that_folds_away_leaves_no_gap() -> None: - # 'lt .' stored 'lt' before this field folded per word. A per-word - # join that keeps the empty slot stores 'lt ', which match time can - # never build -- silently inert config, the exact failure the - # per-word fold was introduced to remove. - base = Lexicon.default() - lex = dataclasses.replace( - base, titles=base.titles | {"lt", "col"}, - given_name_titles=base.given_name_titles | {"lt ."}) - assert "lt" in lex.given_name_titles - assert Parser(lexicon=lex).parse("Lt. John").given == "John" - - -def test_the_constructor_and_add_store_an_entry_identically() -> None: - # add() goes through dataclasses.replace, which re-runs - # __post_init__ -- a non-idempotent fold converges there and not in - # the constructor, so the two paths silently disagree. - base = dataclasses.replace(Lexicon.default(), - titles=Lexicon.default().titles | {"lt", "col"}) - built = dataclasses.replace( - base, given_name_titles=base.given_name_titles | {"lt . col"}) - assert built.given_name_titles == base.add( - given_name_titles={"lt . col"}).given_name_titles - - -def test_an_all_punctuation_given_name_title_is_rejected() -> None: - # every other vocab field raises on this; this one must not be - # special just because it folds per word - with pytest.raises(ValueError, match="normalizes to empty"): - Lexicon(given_name_titles=frozenset({". ."})) - - -def test_given_name_titles_is_not_constrained_against_titles() -> None: - # Deliberately unvalidated. The lookup key is the joined run of - # Role.TITLE tokens, which the PARSE builds, and a conjunction - # inside a run is title-tagged too -- so "sir and dame" is a - # matchable key whose middle word is a conjunction, not a title. - # A per-word check rejected it; a whole-entry check rejected - # multi-word entries. An unreachable entry is inert rather than - # harmful, so nothing is rejected here. - assert Lexicon(titles=frozenset({"sir", "dame"}), - given_name_titles=frozenset({"sir and dame"})) - # and an entry nothing will ever consult is accepted, not rejected - assert Lexicon(given_name_titles=frozenset({"zzqnotatitle"})) - - -@pytest.mark.parametrize(("make", "marker", "base"), [ - (lambda w: Lexicon(particles_ambiguous=w), - "particles_ambiguous", "particles"), - (lambda w: Lexicon(suffix_acronyms_ambiguous=w), - "suffix_acronyms_ambiguous", "suffix_acronyms"), - (lambda w: Lexicon(honorific_tails=w), - "honorific_tails", "suffix_words"), - (lambda w: Lexicon(suffix_words_ambiguous=w), - "suffix_words_ambiguous", "suffix_words"), -], ids=["particles_ambiguous", "suffix_acronyms_ambiguous", "honorific_tails", - "suffix_words_ambiguous"]) -def test_subset_error_names_the_fix( - make: Callable[[frozenset[str]], Lexicon], marker: str, base: str -) -> None: - # the constraint alone leaves the reader to infer the remedy, and - # the remedy is direction-dependent: a caller who was ADDING wants - # to add to the base, one who was REMOVING wants to drop from the - # marker. Naming both keeps the message right either way. - with pytest.raises(ValueError, match=f"Add them to {base}, or drop " - f"them from {marker}"): - make(frozenset({"zzqnovel"})) - - -def test_bound_given_name_that_is_a_particle_must_be_ambiguous() -> None: - # the vocabulary build checks this on the shipped data (check 9): - # NON_GIVEN_NAME_PARTICLES stays disjoint from BOUND_GIVEN_NAMES. In - # 2.0's complement model that reads as bound_given_names & particles - # <= particles_ambiguous. A particle declared never-to-start-a-given- - # name cannot simultaneously bind one; without the check, one of the - # two contradictory declarations is silently discarded. - with pytest.raises(ValueError, match="particles_ambiguous"): - Lexicon(particles=frozenset({"dos"}), - particles_ambiguous=frozenset(), - bound_given_names=frozenset({"dos"})) - - -def test_bound_given_name_may_overlap_an_ambiguous_particle() -> None: - # the shipped case: "abu" is both, and legitimately so - lex = Lexicon(particles=frozenset({"abu"}), - particles_ambiguous=frozenset({"abu"}), - bound_given_names=frozenset({"abu"})) - assert "abu" in lex.bound_given_names - # and a bound name that is not a particle at all is unconstrained - assert "abdul" in Lexicon(bound_given_names=frozenset({"abdul"}) - ).bound_given_names - - -def test_default_lexicon_satisfies_the_bound_particle_invariant() -> None: - d = Lexicon.default() - assert not (d.bound_given_names & d.particles) - d.particles_ambiguous - - -def test_removing_a_title_leaves_its_given_name_marker_alone() -> None: - # the mirror of the above: removing from titles does not raise, and - # the now-inert marker entry simply stops being reachable - lex = Lexicon(titles=frozenset({"sheikh"}), - given_name_titles=frozenset({"sheikh"})) - lean = lex.remove(titles={"sheikh"}) - assert lean.titles == frozenset() - assert lean.given_name_titles == frozenset({"sheikh"}) - - -def test_removing_a_conjunction_leaves_its_ambiguous_marker_alone() -> None: - # conjunctions_ambiguous is the same shape as given_name_titles: not - # in _SUBSET_FIELDS, so an orphan is inert rather than rejected -- - # the marker is never consulted once its base entry is gone - # (decisions/P8). - lean = Lexicon.default().remove(conjunctions={"e"}) - assert "e" not in lean.conjunctions - assert "e" in lean.conjunctions_ambiguous - - @pytest.mark.parametrize("word", [ "dr.", " Dr. ", ". a .", ". .", "..x..", " .b. ", # the three non-ASCII full stops (#322), alone and mixed with @@ -843,21 +382,84 @@ def test_normalize_folds_every_full_stop_and_composes_nfd() -> None: assert _normalize("Ｍ．Ｂ．Ａ．") == "ｍ．ｂ．ａ" -def test_an_entry_that_is_only_full_stops_is_rejected() -> None: - # the "normalizes to empty" guard reads the same stop set as the - # fold, so an ideographic stop on its own is the data bug an ASCII - # one already is - with pytest.raises(ValueError, match="normalizes to empty"): - Lexicon(titles=frozenset({"。"})) + +def test_normalization_preserves_spellings_casefold_would_mutate() -> None: + # lower(), not casefold(): casefold's caseless-matching folds would + # rewrite stored vocabulary -- Greek final sigma flattens ('κος' -> + # the misspelling 'κοσ') and German ß expands ('großfürst' -> + # 'grossfürst'). lower() follows Unicode SpecialCasing contextually + # ('ΚΟΣ' -> 'κος') and keeps entries as authored, matching v1's lc(). + lex = Lexicon.empty().add("ΚΟΣ", "Großfürst", "Ph.D", kind="title") + # and an interior period survives: 'J.R.' must not collapse to 'jr' + assert lex._sets.titles == frozenset({"κος", "großfürst", "ph.d"}) + + +def test_a_word_that_normalizes_to_empty_raises() -> None: + # "." or "" is a data bug (a stray split, an empty CSV cell), for a + # word as for a mask key + for word in (" ", ".", "。", ". ."): + with pytest.raises(ValueError, match="normalizes to empty"): + Lexicon.empty().add(word, kind="title") + with pytest.raises(ValueError, match="normalizes to empty"): + Lexicon(capitalization_exceptions=(("...", "X"), ("phd", "PhD"))) + + +@pytest.mark.parametrize("word", [42, {"dr"}, ["dr"], None]) +def test_each_word_is_its_own_string_argument(word: object) -> None: + # a collection passed as one word is the 2.x habit (`titles={"dr"}` + # moved to the front); the message says to unpack it + with pytest.raises(TypeError, match=r"unpack a collection: add\(\*words"): + Lexicon.empty().add(word, kind="title") # type: ignore[arg-type] + + +@pytest.mark.parametrize("value", [b"dr", bytearray(b"dr"), memoryview(b"dr")]) +@pytest.mark.parametrize("build", [ + lambda v: Lexicon.empty().add(v, kind="title"), + lambda v: Lexicon.empty().remove(v), + lambda v: Lexicon.default().readings_of(v), + lambda v: Lexicon(surnames=v), + lambda v: Lexicon(capitalization_exceptions=v), +]) +def test_every_lexicon_entry_point_rejects_a_buffer_with_a_decode_hint( + build: Callable[[object], object], value: object) -> None: + # Binary sequences iterate to INTS, so the generic entry check + # reported "entries must be strings, got 100" -- the byte value of + # 'd', naming neither the cause nor the fix. v1 shipped a decode + # hint for exactly this (#238). Every entry point, a guard on one + # member of a family belonging on all of it (AGENTS.md). + with pytest.raises(TypeError, match="decode"): + build(value) + + +@pytest.mark.parametrize("field", _LIST_FIELDS) +def test_a_list_rejects_a_bare_string_a_mapping_and_a_non_string( + field: str) -> None: + with pytest.raises(TypeError, match="bare string"): + Lexicon(**{field: "kim"}) # type: ignore[arg-type] + with pytest.raises(TypeError, match="mapping"): + Lexicon(**{field: {"kim": "x"}}) # type: ignore[arg-type] + with pytest.raises(TypeError, match="entries must be strings"): + Lexicon(**{field: {"kim", 42}}) # type: ignore[arg-type] + with pytest.raises(TypeError, match=f"{field}.*iterable"): + Lexicon(**{field: 5}) # type: ignore[arg-type] + + +@pytest.mark.parametrize("field", _LIST_FIELDS) +def test_a_multi_word_list_entry_warns_at_the_callers_line(field: str) -> None: + # a list is matched one word at a time, so a phrase can never match + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter("always") + Lexicon(**{field: {"zqx zqy"}}) # type: ignore[arg-type] + assert [x.filename for x in w] == [__file__] + assert "matched one word at a time" in str(w[0].message) def test_an_nfd_authored_entry_is_stored_composed() -> None: # the storage fold composes NFC (#322), so a caller who authors a # surname in NFD gets the NFC entry -- and raw NFD input, which # segmentation matches without folding, then finds no entry and - # goes unsplit (decisions/W1's 2026-07-29 NFD bullet, ja amendment 1a: NFD - # degrades to no-split, never to a wrong split). Pinned - # because the surname site's comment relies on it. + # goes unsplit (decisions/W1's 2026-07-29 NFD bullet: NFD degrades + # to no-split, never to a wrong split) nfd = unicodedata.normalize("NFD", "김") lex = Lexicon(surnames=frozenset({nfd})) assert lex.surnames == frozenset({"김"}) @@ -865,391 +467,1053 @@ def test_an_nfd_authored_entry_is_stored_composed() -> None: def test_every_shipped_entry_is_already_nfc() -> None: - # the measurement behind _normalize's docstring, as a standing - # assertion: NFC at storage changes no shipped entry, so the - # vocabulary a caller reads back is the vocabulary that was - # authored. A data module written in NFD would fail here. - # - # The roster is _VOCAB_FIELDS, which is the roster __post_init__ - # and __setstate__ normalize -- so a fourteenth vocabulary field - # has to join it for add/remove to work at all, and joining it - # gets the field checked here for free (#322/#323 review). + # NFC at storage changes no shipped entry, so the vocabulary a caller + # reads back is the vocabulary that was authored lex = Lexicon.default() - for field in _VOCAB_FIELDS: - for word in getattr(lex, field): + for field, words in _fields_of(lex._sets).items(): + for word in words: assert unicodedata.normalize("NFC", word) == word, (field, word) for pair in lex.capitalization_exceptions: for word in pair: assert unicodedata.normalize("NFC", word) == word, pair -def test_unpickling_revalidates_invariants() -> None: - # the field-layout guard catches SHAPE skew, but the likeliest skew - # is same-name/flipped-meaning: particles_ambiguous inverted sense - # between v1's never-given set and v2's may-be-given set. A state - # dict that violates an invariant must not load in silence. - lex = Lexicon(particles=frozenset({"van"}), - particles_ambiguous=frozenset({"van"})) - state = lex.__getstate__() - state["particles"] = frozenset() # orphans particles_ambiguous - forged = Lexicon.__new__(Lexicon) - with pytest.raises(ValueError, match="particles_ambiguous"): - forged.__setstate__(state) +def test_the_lists_stay_fields() -> None: + lex = Lexicon(surnames=frozenset({"毛", "欧阳"})) + assert lex.surnames == frozenset({"毛", "欧阳"}) + merged = Lexicon(surnames=frozenset({"김"})) | Lexicon(surnames=frozenset({"박"})) + assert merged.surnames == frozenset({"김", "박"}) + assert dataclasses.replace(merged, surnames={"김"}).surnames == {"김"} # type: ignore[arg-type] + assert (Lexicon.default() | Lexicon(given_names={"Luis", "José"}) + ).given_names == frozenset({"luis", "josé"}) -def test_unpickling_rejects_unnormalized_entries() -> None: - # a genuine pickle always holds normalized entries, so state that - # does not is foreign. Correcting it silently would make the load - # path a fourth place where caller data is rewritten without a - # word; say so instead, at the site that can still name the entry. - lex = Lexicon(titles=frozenset({"dr"})) - state = lex.__getstate__() - state["titles"] = {"Sir ", "DR."} - restored = Lexicon.__new__(Lexicon) - with pytest.raises(ValueError, match="not normalized"): - restored.__setstate__(state) +# -- configuring by reading: add and remove ---------------------------------- +def test_add_and_remove_return_new_lexicons() -> None: + base = Lexicon.default() + lex = base.add("zqtitle", kind="title").remove("esquire") + assert "zqtitle" in lex._sets.titles and "zqtitle" not in base._sets.titles + # precondition, or the removal passes vacuously + assert "esquire" in base._sets.suffix_words + assert "esquire" not in lex._sets.suffix_words + assert lex.readings_of("esquire") == () -def test_unpickling_rejects_uncanonical_capitalization_exceptions() -> None: - # the pair field gets the same treatment as the vocab sets: it is - # re-canonicalized by __post_init__ (normalized keys, sorted, - # deduped), so accepting a rewritten value would leave one field - # silently corrected while ten raise - lex = Lexicon(capitalization_exceptions=(("phd", "PhD"),)) - state = lex.__getstate__() - state["capitalization_exceptions"] = ((" PhD. ", "Ph.D."), ("aa", "AA")) - restored = Lexicon.__new__(Lexicon) - with pytest.raises(ValueError, match="not normalized"): - restored.__setstate__(state) +def test_remove_takes_a_kind_or_a_category() -> None: + # dean holds an office reading and the given and family readings + lex = Lexicon.default() + assert {r.kind for r in lex.readings_of("dean")} == { + "office", "given", "family"} + assert {r.kind for r in lex.remove("dean", kind="office") + .readings_of("dean")} == {"given", "family"} + # a category matches every kind of it: `title` is both + assert {r.kind for r in lex.remove("dean", kind="title") + .readings_of("dean")} == {"given", "family"} + assert {r.kind for r in lex.remove("dean", kind="name") + .readings_of("dean")} == {"office"} + + +def test_add_of_a_held_kind_updates_it_in_every_file() -> None: + # `e`'s conjunction is Italian and Portuguese, one-case fork and all + lex = Lexicon.default() + assert lex.readings_of("e")[0].languages == ("it", "pt") + settled = lex.add("e", kind="conjunction", may_be_initial=False) + (reading,) = settled.readings_of("e") + assert (reading.properties, reading.languages) == ((), ("it", "pt")) + assert "e" not in settled._sets.conjunctions_ambiguous + assert Parser(lexicon=settled).parse("ANA SILVA E SOUZA").family == ( + "SILVA E SOUZA") + + +def test_add_of_a_held_kind_turns_a_reading_written_off_on() -> None: + # st's generation reading is cs, hr, sk and sl's, off by default; + # an add of that kind turns it on in each file, beside its name + # reading, so it is a suffix word gated by the count + lex = Lexicon.default() + assert "st" not in lex._sets.suffix_words + on = lex.add("st", kind="generation") + (generation,) = [r for r in on.readings_of("st") if r.kind == "generation"] + assert generation.languages == ("cs", "hr", "sk", "sl") + assert "st" in on._sets.suffix_words_ambiguous + assert "st" in on._sets.generation_words + + +def test_a_callers_own_reading_names_no_file() -> None: + lex = Lexicon.default().add("leed", "ap", kind="credential") + assert lex.readings_of("leed") == ( + LexiconReading("leed", "credential", "suffix"),) + assert Parser(lexicon=lex).parse("John Smith LEED AP").suffix == "LEED AP" + + +def test_a_property_given_its_unwritten_value_is_removed() -> None: + lex = Lexicon.empty().add("sir", kind="nobility", addresses="given") + assert lex.readings_of("sir")[0].properties == (("addresses", "given"),) + for unwritten in ("family", None): + cleared = lex.add("sir", kind="nobility", addresses=unwritten) + assert cleared.readings_of("sir")[0].properties == () + # acronym is unwritten-true on a credential alone + worded = Lexicon.empty().add("kc", kind="credential", acronym=False) + assert "kc" in worded._sets.suffix_words + assert "kc" in worded._sets.credential_words + assert worded.readings_of("kc")[0].properties == (("acronym", False),) + + +def test_a_derived_marker_follows_from_the_readings() -> None: + # a given reading beside a credential gates it, as the build derives + # it from a file: `dc` becomes an ambiguous acronym + lex = Lexicon.default().add("dc", kind="given") + assert "dc" in lex._sets.suffix_acronyms_ambiguous + assert "dc" not in Lexicon.default()._sets.suffix_acronyms_ambiguous + + +@pytest.mark.parametrize(("call", "error", "match"), [ + (lambda: Lexicon.empty().add("x"), TypeError, "missing required keyword argument 'kind'"), # type: ignore[call-arg] + (lambda: Lexicon.empty().add(kind="title"), TypeError, "at least one word"), + (lambda: Lexicon.empty().remove(), TypeError, "at least one word"), + (lambda: Lexicon.empty().add("x", kind="titel"), ValueError, "unknown reading kind 'titel'"), + (lambda: Lexicon.empty().add("x", kind=3), TypeError, "kind as a string"), # type: ignore[arg-type] + # a category with no kind of its name names its kinds + (lambda: Lexicon.empty().add("x", kind="name"), ValueError, "kinds: given, family"), + (lambda: Lexicon.empty().add("x", kind="relation"), ValueError, "kinds: birth_marker"), + (lambda: Lexicon.empty().remove("x", kind="titel"), ValueError, "or a category, on remove"), + (lambda: Lexicon.empty().add("x", kind="title", default=False), ValueError, "a caller's reading is on"), + (lambda: Lexicon.empty().add("x", kind="title", cased="X"), ValueError, "capitalization_exceptions"), + (lambda: Lexicon.empty().add("x", kind="title", acronym=True), ValueError, "a title reading takes addresses, after_given, attaches"), + (lambda: Lexicon.empty().add("x", kind="family", same_as="y"), ValueError, "a family reading takes no properties"), + (lambda: Lexicon.empty().add("x", kind="title", after_given="yes"), TypeError, "after_given must be True or False"), + (lambda: Lexicon.empty().add("x", kind="title", addresses="first"), ValueError, "addresses must be 'given' or 'family'"), + (lambda: Lexicon.empty().add("x", kind="title", attaches="military"), TypeError, "attaches must be an iterable"), + (lambda: Lexicon.empty().add("x", kind="honorific", attached="before"), ValueError, "attached must be 'after'"), + (lambda: Lexicon.empty().add("x", kind="given", same_as=""), TypeError, "same_as must be the full given name"), + # the build's check 8: a phrase matches only as a marker or a run of + # titles addressing the given name + (lambda: Lexicon.empty().add("chief justice", kind="title"), ValueError, "a phrase, which can never match as a title"), + # rules/P8's marker reads a single cased letter alone + (lambda: Lexicon.empty().add("and", kind="conjunction", may_be_initial=True), ValueError, "not one"), + # the build's check 4: one suffix kind per word, naming the fix + (lambda: Lexicon.default().add("dc", kind="generation"), ValueError, re.escape(".remove('dc', kind='credential')")), + # check 11: an agnomen is gated by a name reading beside it + (lambda: Lexicon.default().add("bisneto", kind="agnomen"), ValueError, re.escape(".add('bisneto', kind='family')")), + (lambda: Lexicon.default().remove("neto", kind="name"), ValueError, "agnomen with no name reading"), + # check 9: a bound given name that is a particle may be given + (lambda: Lexicon.empty().add("dos", kind="particle").add("dos", kind="bound_given"), ValueError, re.escape(".add('dos', kind='given')")), +]) +def test_add_and_remove_refuse_with_the_fix( + call: Callable[[], object], error: type[Exception], match: str) -> None: + with pytest.raises(error, match=match): + call() -def test_unpickling_a_real_pickle_is_the_identity() -> None: - # the corollary: anything this library produced round-trips - # unchanged, including entries whose normalized form needed more - # than one strip pass to converge - for lex in (Lexicon.default(), Lexicon.empty(), - Lexicon(titles=frozenset({". a ."}))): - assert pickle.loads(pickle.dumps(lex)) == lex +def test_a_phrase_reading_is_stored_folded_per_word() -> None: + lex = Lexicon.empty().add("Z. Domu", kind="birth_marker").add( + "Lt. Col.", kind="title", addresses="given") + assert lex._sets.maiden_markers == frozenset({"z domu"}) + assert lex._sets.given_name_titles == frozenset({"lt col"}) + assert lex.readings_of("z. domu")[0].kind == "birth_marker" -def test_remove_error_offers_the_removal_remedy_too() -> None: - # a caller removing from the base does not want to be told to add - # it back; the message names both directions - lex = Lexicon(particles=frozenset({"van"}), - particles_ambiguous=frozenset({"van"})) - with pytest.raises(ValueError, - match="drop them from particles_ambiguous"): - lex.remove(particles={"van"}) +@pytest.mark.parametrize("spelling", ["lt. col", "lt col", "Lt. Col."]) +def test_a_given_title_run_folds_per_word_so_abbreviations_match( + spelling: str) -> None: + # The run's key is built per word and rejoined, so an abbreviated + # phrase folds the way the parse folds the title run; it is the + # whole-run arm that matches, `col` alone addressing the family + lex = Lexicon.default().add(spelling, kind="title", addresses="given") + assert "lt col" in lex._sets.given_name_titles + assert "col" not in lex._sets.given_name_titles + assert Parser(lexicon=lex).parse("Lt. Col. Smith").given == "Smith" -@pytest.mark.parametrize("value", [b"dr", bytearray(b"dr"), memoryview(b"dr")]) -@pytest.mark.parametrize("build", [ - lambda v: Lexicon(titles=v), - lambda v: Lexicon.empty().add(titles=v), - lambda v: Lexicon(capitalization_exceptions=v), -]) -def test_every_lexicon_entry_point_rejects_a_buffer_with_a_decode_hint( - build: Callable[[object], object], value: object) -> None: - # Binary sequences iterate to INTS, so the generic entry check - # reported "entries must be strings, got 100" -- the byte value of - # 'd', naming neither the cause nor the fix. v1 shipped a decode - # hint for exactly this (#238). - # - # Every entry point, because the first version of this fix covered - # the vocabulary fields and left capitalization_exceptions behind: - # guarding one member of a family and not its siblings is the - # single most repeated defect on this branch. - with pytest.raises(TypeError, match="decode"): - build(value) +def test_the_shipped_given_name_titles_are_every_one_a_single_word() -> None: + # what makes the whole-run arm dead for the shipped vocabulary: it + # can only match a one-word run, which the last-word arm reads the + # same way (#489) + sets = Lexicon.default()._sets + assert not any(" " in t for t in sets.given_name_titles) + # and the last-word arm's premise: 'dr sir' is no entry, 'sir' is + assert "dr sir" not in sets.given_name_titles + assert "sir" in sets.given_name_titles -# Derived from the exemption, not written out beside it: an exemption -# that widens must take fields OUT of this list, where the parametrized -# test below stops covering them and the count assertion notices. -_PER_WORD_FIELDS = [f for f in _VOCAB_FIELDS if f not in _PHRASE_FIELDS] +def test_the_shipped_suffix_sets_are_disjoint() -> None: + # The two sets normalize differently -- the word test strips only + # edge periods, the acronym test all of them -- so a word in both + # is matched by two rules (decisions/suffix-acronym-collisions); a + # compile from readings refuses two suffix kinds on a word, and one + # kind is an acronym or a word + assert SUFFIX_ACRONYMS & SUFFIX_WORDS == frozenset() + sets = Lexicon.default()._sets + assert sets.suffix_acronyms & sets.suffix_words == frozenset() -def test_only_the_phrase_fields_are_exempt_from_the_warning() -> None: - # The exemption is exactly the kind of edit that silently widens: - # one more field in _PHRASE_FIELDS costs nothing at construction and - # turns a dead entry back into a silent no-op. Three fields, named. - assert _PHRASE_FIELDS == ("given_name_titles", "maiden_markers", - "title_phrases") - assert len(_PER_WORD_FIELDS) == len(_VOCAB_FIELDS) - 3 +def test_default_lexicon_satisfies_the_bound_particle_invariant() -> None: + d = Lexicon.default()._sets + assert not (d.bound_given_names & d.particles) - d.particles_ambiguous -@pytest.mark.parametrize("field", _PER_WORD_FIELDS) -def test_multiword_entry_warns_per_field(field: str) -> None: - # Guard the whole family: every per-word field warns, not one. - with pytest.warns(UserWarning, match="matched one word at a time"): - if field in ("particles_ambiguous", "suffix_acronyms_ambiguous", - "honorific_tails", "suffix_words_ambiguous"): - # subset fields need their base populated to construct - base = {"particles_ambiguous": "particles", - "suffix_acronyms_ambiguous": "suffix_acronyms", - "honorific_tails": "suffix_words", - "suffix_words_ambiguous": "suffix_words"}[field] - Lexicon.empty().add(**{base: ["zqx zqy"], field: ["zqx zqy"]}) - else: - Lexicon.empty().add(**{field: ["zqx zqy"]}) - - -def test_multiword_given_name_title_does_not_warn() -> None: - # given_name_titles matches space-joined runs -- multi-word entries - # are the point (see the __post_init__ note in _lexicon.py). - with warnings.catch_warnings(): - warnings.simplefilter("error") - Lexicon.empty().add(given_name_titles=["lt col"]) +def test_removing_a_honorific_tails_attachment_is_the_peels_off_switch() -> None: + # dropping `attached` leaves the glued name unsplit while the SPACED + # spelling still reads the honorific, a suffix word either way + lex = Lexicon.default().add("さん", kind="honorific", attached=None) + assert "さん" not in lex._sets.honorific_tails + p = Parser(lexicon=lex) + glued = p.parse("田中さん") + assert (glued.family, glued.suffix) == ("田中さん", "") + spaced = p.parse("田中 さん") + assert (spaced.family, spaced.suffix) == ("田中", "さん") + on = Parser().parse("田中さん") + assert (on.family, on.suffix) == ("田中", "さん") -def test_multiword_maiden_marker_does_not_warn() -> None: - # The second exemption, and the one whose warning was wrong rather - # than merely unhelpful: markers are matched by lookahead, and - # "split it into separate entries" reads 'Maria Kowalska z domu - # Nowak' as maiden 'domu Nowak'. - with warnings.catch_warnings(): - warnings.simplefilter("error") - Lexicon.empty().add(maiden_markers=["z domu"]) +def test_removing_after_given_is_rule_h8s_off_switch() -> None: + # the bare salutation is a middle name after a family comma, while + # the period-marked spelling reads as the title by H5's shape + lex = Lexicon.default().add("mrs", kind="salutation", after_given=False) + p = Parser(lexicon=lex) + assert (p.parse("Doe, Jane Mrs").title, + p.parse("Doe, Jane Mrs").middle) == ("", "Mrs") + assert p.parse("Doe, Jane Mrs.").title == "Mrs." + assert Parser().parse("Doe, Jane Mrs").title == "Mrs" + # and with the title reading gone there is nothing to read after + # the given name + gone = Parser(lexicon=Lexicon.default().remove("mrs", kind="title")) + assert (gone.parse("Doe, Jane Mrs").title, + gone.parse("Doe, Jane Mrs").middle) == ("", "Mrs") -def test_phrase_fields_store_the_phrase_folded_per_word() -> None: - # The shared storage rule: space-joined, each word normalized on - # its own. Whole-phrase _normalize would keep 'z.' interior period - # and the entry would never match. - lex = Lexicon.empty().add(maiden_markers=["Z. Domu"], - given_name_titles=["Lt. Col"]) - assert lex.maiden_markers == frozenset({"z domu"}) - assert lex.given_name_titles == frozenset({"lt col"}) +def test_every_shipped_after_given_title_is_reachable() -> None: + # each entry must reach rules/H8 at the slot it is listed for, so a + # member another reading claims first shows up as silently inert + p = Parser() + unreached = [w for w in sorted(p.lexicon._sets.after_given_titles) + if p.parse(f"Doe, Jane {w.title()}").title != w.title()] + assert not unreached, unreached -def test_multiword_capitalization_key_warns() -> None: - with pytest.warns(UserWarning, match="matched one word at a time"): - dataclasses.replace( - Lexicon.empty(), - capitalization_exceptions=(("zqx zqy", "ZqXZqY"),)) +# -- the views read the lexicon's kinds (R8 and R9, #754 4i-b) ----------------- + +def test_the_suffix_views_read_the_kind_the_lexicon_states() -> None: + # R8, the generation and credentials views: the kind of a listed + # suffix word is the lexicon's, so a + # word a caller lists as a generation reads as one (the 2.x spelling, + # a suffix word with no kind, read credentials `Secundus`) + name = "Marcus Tullius Secundus" + generation = Parser(lexicon=Lexicon.default().add( + "secundus", kind="generation")).parse(name) + assert (generation.generation, generation.credentials) == ("Secundus", "") + plain = Parser(lexicon=Lexicon.default().add( + "secundus", kind="suffix")).parse(name) + assert (plain.generation, plain.credentials) == ("", "Secundus") + + +def test_the_patronymic_view_reads_the_kind_the_lexicon_states() -> None: + # R9, the patronymic view: `bin` read as a patronymic particle while + # the lexicon + # holds that kind, a plain particle once it holds another + name = "Mahathir bin Mohamad" + assert Parser().parse(name).patronymic == "bin Mohamad" + plain = Lexicon.default().add("bin", kind="preposition").remove( + "bin", kind="patronymic") + parsed = Parser(lexicon=plain).parse(name) + assert (parsed.family, parsed.patronymic) == ("bin Mohamad", "") + + +def test_the_default_kind_sets_are_the_shipped_index_s() -> None: + # by the compile invariant: what the views read on the default is + # what the shipped files say (the index lookup 4b and 4c read) + from nameparser._reading_index import holds_kind, suffix_kind_of + sets = Lexicon.default()._sets + for word in sets.suffix_words: + kind = suffix_kind_of(word) + assert (word in sets.generation_words) == (kind == "generation"), word + assert (word in sets.credential_words) == (kind == "credential"), word + for word in sets.particles: + assert (word in sets.patronymic_particles) == holds_kind( + word, "patronymic"), word + + +# -- the 2.x spellings, refused with their translation ---------------------- + +#: (name, 2.x spelling, 3.0 spelling): the recipes the RFC's table +#: carries (docs/design/vocabulary-model-rfc.md, item 1), each parsing +#: every field alike under both. The 2.x side is built here from the +#: sets the 2.x call would leave, through the translation a 2.x pickle +#: loads by, which is itself under test below. +_RECIPES = [ + ("Dean Smith", lambda L: L.remove("dean", kind="given")), + ("John Smith LEED AP", lambda L: L.add("leed", "ap", kind="credential")), + ("Professor Plum", lambda L: L.remove("professor", kind="title")), + ("Guru Prasad Sharma", lambda L: L.add("guru", kind="given")), + ("Ram Bahadur DC", lambda L: L.remove("dc", kind="credential")), + ("Van Johnson", lambda L: L.remove("van", kind="given")), + ("ANA SILVA E SOUZA", + lambda L: L.add("e", kind="conjunction", may_be_initial=False)), + ("Mohamad Ali Khan", lambda L: L.add("mohamad", kind="bound_given")), + ("Jane Smith formerly Jones", + lambda L: L.add("formerly", kind="birth_marker")), + ("山田さん", lambda L: L.add("さん", kind="honorific", attached=None)), +] -def test_a_multiword_key_skips_the_mask_check_entirely() -> None: - """#459 review: the mask check ran before the multi-word-key - warning, so a key that can never match -- 'ph d' has no word in - capitalized()'s per-word lookup -- still raised over a value that - does not spell it (AGENTS.md: an invariant guards harm, and an - unreachable entry does none). Reordered so a multi-word key only - warns, storing its value verbatim regardless of the mask - question.""" - with pytest.warns(UserWarning, match="matched one word at a time"): - lex = dataclasses.replace( - Lexicon.empty(), - capitalization_exceptions=(("ph d", "Doctor"),)) - assert lex.capitalization_exceptions_map == {"ph d": "Doctor"} +@pytest.mark.parametrize(("text", "spell"), _RECIPES, + ids=[r[0] for r in _RECIPES]) +def test_each_recipe_moves_the_parse_it_names( + text: str, spell: Callable[[Lexicon], Lexicon]) -> None: + # the recipe's own negative control: the default reads it otherwise + configured = Parser(lexicon=spell(Lexicon.default())).parse(text) + assert configured.as_dict() != Parser().parse(text).as_dict() + + +#: A 2.x call per site: (site, call, the words it names). The refusal +#: translates those words, and the translation runs. +_REFUSED_CALLS = [ + ("add", lambda: Lexicon.default().add(titles={"dean"}), # type: ignore[call-arg] + ".remove('dean', kind='given')"), + ("add", lambda: Lexicon.default().add(suffix_acronyms={"leed", "ap"}), # type: ignore[call-arg] + ".add('ap', 'leed', kind='credential')"), + ("remove", lambda: Lexicon.default().remove(particles_ambiguous={"van"}), + ".remove('van', kind='given')"), + ("remove", lambda: Lexicon.default().remove(conjunctions_ambiguous={"e"}), + ".add('e', kind='conjunction', may_be_initial=False)"), + ("remove", lambda: Lexicon.default().remove(honorific_tails={"さん"}), + ".add('さん', kind='honorific', attached=None)"), + ("Lexicon()", lambda: Lexicon(titles={"dean"}), + "Lexicon.empty().add('dean', kind='title')"), + ("replace", lambda: dataclasses.replace( # type: ignore[call-arg] + Lexicon.default(), maiden_markers={"geb", "formerly"}), + None), +] -def test_default_lexicon_builds_warning_free() -> None: - # Would have caught the eight dead entries the previous commit - # repaired. - _default_lexicon.cache_clear() - try: - with warnings.catch_warnings(): - warnings.simplefilter("error") - Lexicon.default() - finally: - _default_lexicon.cache_clear() +@pytest.mark.parametrize(("site", "call", "translation"), _REFUSED_CALLS) +def test_a_2x_spelling_is_refused_with_calls_that_run( + site: str, call: Callable[[], object], translation: str | None) -> None: + with pytest.raises(TypeError, match="since 3.0") as raised: + call() + message = str(raised.value) + if translation is not None: + assert message.endswith("write lexicon" + translation) or \ + message.endswith("write " + translation), message + # the offered calls run, on the lexicon they name + code = message.rsplit("write ", 1)[1] + built = eval(code, {"Lexicon": Lexicon, # noqa: S307 + "lexicon": Lexicon.default()}) + assert isinstance(built, Lexicon) + + +@pytest.mark.parametrize("field", _SET_FIELDS) +def test_every_2x_set_field_is_refused_at_all_four_sites(field: str) -> None: + for call in (lambda: Lexicon.empty().add(**{field: {"zzq"}}), # type: ignore[arg-type] + lambda: Lexicon.empty().remove(**{field: {"zzq"}}), # type: ignore[arg-type] + lambda: Lexicon(**{field: {"zzq"}}), # type: ignore[arg-type] + lambda: dataclasses.replace(Lexicon.empty(), + **{field: {"zzq"}})): # type: ignore[arg-type] + with pytest.raises(TypeError, match=f"takes no {field} since 3.0"): + call() + with pytest.raises(AttributeError, match=re.escape( + f"Lexicon.{field} is internal since 3.0")) as raised: + getattr(Lexicon.default(), field) + assert "readings_of" in str(raised.value) + assert "Token.reading" in str(raised.value) + + +def test_a_list_or_the_masks_through_add_is_refused_with_its_spelling() -> None: + # the offered code names the caller's own words and RUNS as pasted + # (root AGENTS.md: a message's code must type-check and run) + for call, expected in ( + (lambda: Lexicon.default().add(surnames={"김민"}), # type: ignore[call-arg] + "lexicon | Lexicon(surnames={'김민'})"), + (lambda: Lexicon.default().remove(given_names={"luis"}), + "dataclasses.replace(lexicon, given_names=lexicon.given_names" + " - {'luis'})")): + with pytest.raises(TypeError, match=re.escape(expected)) as raised: + call() + code = str(raised.value).rsplit("write ", 1)[1] + pasted = eval(code, {"dataclasses": dataclasses, # noqa: S307 + "Lexicon": Lexicon, "lexicon": Lexicon.default()}) + assert isinstance(pasted, Lexicon) + with pytest.raises(TypeError, match="unexpected keyword argument 'prefixes'"): + Lexicon.default().remove(prefixes={"van"}) # a v1 name + with pytest.raises(TypeError, match="unexpected keyword argument 'zq'"): + Lexicon(zq=1) -def test_multiword_entry_warns_from_constructor_and_unpickle() -> None: - with pytest.warns(UserWarning, match="matched one word at a time"): - lex = Lexicon(titles=frozenset({"zqx zqy"})) - with pytest.warns(UserWarning, match="matched one word at a time"): - pickle.loads(pickle.dumps(lex)) +def test_add_capitalization_exceptions_raises_pointing_at_replace() -> None: + # The offered spelling is a tuple, not a dict: the field is + # annotated as one, so a pasted dict fails mypy (AGENTS.md). And + # it must RUN as pasted: the value is a case mask of its key, so + # ('key', 'Value') type-checks and then raises (2.4 release sweep). + offered = re.escape( + "dataclasses.replace(lexicon, capitalization_exceptions=" + "(*lexicon.capitalization_exceptions, ('key', 'KEY')))") + with pytest.raises(TypeError, match=offered) as raised: + Lexicon.default().add(capitalization_exceptions={"x": "X"}) # type: ignore[call-arg] + code = re.search(r"dataclasses\.replace\(.*?\)\)\)", + str(raised.value)) + assert code is not None + pasted = eval(code[0], {"dataclasses": dataclasses, # noqa: S307 + "lexicon": Lexicon.default()}) + assert pasted.capitalization_exceptions_map["key"] == "KEY" -def test_remove_does_not_warn() -> None: - # remove() stores nothing; a dead-entry warning there would be a - # false positive with wrong advice. - with warnings.catch_warnings(): - warnings.simplefilter("error") - Lexicon.default().remove(titles=["zqx zqy"]) +def test_a_spelling_that_already_reads_so_says_no_call_is_needed() -> None: + with pytest.raises(TypeError, match="no call is needed"): + Lexicon.default().add(titles={"dr"}) # type: ignore[call-arg] -def test_multiword_warning_points_at_caller_line() -> None: - # the frame walk is the feature: both entry depths must attribute - # the warning to THIS file, not library internals - with warnings.catch_warnings(record=True) as w: - warnings.simplefilter("always") - Lexicon(titles=frozenset({"zqx zqy"})) - Lexicon.empty().add(titles=["zqx zqy"]) - assert len(w) >= 2 - assert all(x.filename == __file__ for x in w) +# -- the title warning (decisions/vocabulary-model, 2026-11-01) ------------- -def test_add_warns_exactly_once() -> None: - # the warn=False plumbing exists to prevent the double warning - # (_edit's pass + the new instance's __post_init__); count it +def _warned(build: Callable[[], object]) -> list[warnings.WarningMessage]: with warnings.catch_warnings(record=True) as w: warnings.simplefilter("always") - Lexicon.empty().add(titles=["zqx zqy"]) - assert len([x for x in w if "matched one word" in str(x.message)]) == 1 - - -def test_remove_of_a_stored_dead_entry_is_silent() -> None: - # the scenario the design argues for: user got the warning, now - # removes the dead entry -- silently. (An UNRELATED edit on a - # lexicon still holding one re-warns; that is per-construction - # warning, by design.) - with pytest.warns(UserWarning): - dirty = Lexicon.empty().add(titles=["zqx zqy"]) + build() + return [x for x in w if "title only inside a title phrase" in str(x.message)] + + +@pytest.mark.parametrize(("build", "warns"), [ + # a title reading created on a word whose given reading keeps it a + # title only inside a phrase: the 1.4 recipe's trap + (lambda: Lexicon.default().add("dean", kind="title"), True), + # the word's own kind re-added unchanged: the same trap + (lambda: Lexicon.default().add("dean", kind="office"), True), + # a property update of a held title reading is the caller's + # intent, and the warning's fix would widen it + (lambda: Lexicon.default().add("prince", kind="nobility", + attaches=["military"]), False), + # no given reading, so a title everywhere + (lambda: Lexicon.default().add("zzq", kind="title"), False), + (lambda: Lexicon.default().remove("dean", kind="given").add( + "dean", kind="title"), False), + # not a title reading + (lambda: Lexicon.default().add("dean", kind="family"), False), +], ids=["created", "re-added", "property-update", "no-given-reading", + "given-removed-first", "not-a-title"]) +def test_the_title_warning_fires_where_the_add_changes_nothing_outside_a_phrase( + build: Callable[[], object], warns: bool) -> None: + assert bool(_warned(build)) is warns + + +def test_the_title_warning_names_the_fix_at_the_callers_line() -> None: + (w,) = _warned(lambda: Lexicon.default().add("dean", kind="title")) + assert w.filename == __file__ + assert ".remove('dean', kind='given')" in str(w.message) + # and the add it warns about does read no name differently + lex = _quiet(lambda: Lexicon.default().add("dean", kind="title")) + assert Parser(lexicon=lex).parse("Dean Smith").as_dict() == \ + Parser().parse("Dean Smith").as_dict() + + +def test_nothing_but_add_warns() -> None: + lex = _quiet(lambda: Lexicon.default().add("dean", kind="title")) with warnings.catch_warnings(): warnings.simplefilter("error") - dirty.remove(titles=["zqx zqy"]) + dataclasses.replace(lex, surnames={"김"}) # type: ignore[arg-type] + pickle.loads(pickle.dumps(lex)) + lex | Lexicon.empty() + Lexicon.default().remove("dean", kind="given") -def test_surnames_is_a_vocab_field() -> None: - lex = Lexicon.empty().add(surnames={"毛", "欧阳"}) - assert lex.surnames == frozenset({"毛", "欧阳"}) - assert lex.remove(surnames={"毛"}).surnames == frozenset({"欧阳"}) - merged = Lexicon(surnames=frozenset({"김"})) | Lexicon(surnames=frozenset({"박"})) - assert merged.surnames == frozenset({"김", "박"}) - +# -- readings_of and LexiconReading ---------------------------------------- -def test_default_lexicon_ships_korean_surnames_only() -> None: +def test_readings_of_answers_with_kind_category_and_files() -> None: lex = Lexicon.default() - assert "김" in lex.surnames and "남궁" in lex.surnames - # Han surnames are locales.ZH's cargo (segmentation opt-in), so - # the DEFAULT set is wholly hangul -- structural check, not - # content-pinning. The spans come from the shared table rather than - # a second hand-copy of 0xAC00/0xD7A3 (test_locales.py's precedent). - hangul = _SCRIPT_RANGES[Script.HANGUL] - assert all(all(any(lo <= ord(c) <= hi for lo, hi in hangul) for c in s) - and 1 <= len(s) <= 2 for s in lex.surnames) + assert lex.readings_of("Dean") == ( + LexiconReading("dean", "office", "title", (), ("en",)), + LexiconReading("dean", "given", "name", (), ("en",)), + LexiconReading("dean", "family", "name", (), ("en",))) + assert lex.readings_of("zzqnotaword") == () + assert any(r.category == "title" for r in lex.readings_of("Dr.")) + + +def test_readings_of_answers_for_the_lexicon_not_the_shipped_files() -> None: + lex = Lexicon.default().remove("dean", kind="given") + assert {r.kind for r in lex.readings_of("dean")} == {"office", "family"} + assert Lexicon.empty().readings_of("dean") == () + + +def test_a_lexicon_reading_is_a_frozen_hashable_picklable_value() -> None: + reading = Lexicon.default().readings_of("sir")[0] + assert hash(reading) == hash(copy.copy(reading)) + assert pickle.loads(pickle.dumps(reading)) == reading + with pytest.raises(dataclasses.FrozenInstanceError): + reading.kind = "title" # type: ignore[misc] + with pytest.raises(ValueError, match="category is 'title'"): + LexiconReading("sir", "nobility", "suffix") + with pytest.raises(ValueError, match="unknown reading kind"): + LexiconReading("sir", "knight", "title") + with pytest.raises(TypeError, match="properties must be a tuple"): + LexiconReading("sir", "nobility", "title", {}) # type: ignore[arg-type] + state = reading.__getstate__() + with pytest.raises(ValueError, match="unexpected fields: zq"): + LexiconReading.__new__(LexiconReading).__setstate__({**state, "zq": 1}) + + +# -- state: equality, the log, |, replace ----------------------------------- +def test_lexicon_is_hashable() -> None: + assert isinstance(hash(Lexicon.default()), int) + assert isinstance(hash(Lexicon.default().add("x", kind="title")), int) + + +def test_two_lexicons_are_equal_when_their_logs_are_as_written() -> None: + a = Lexicon.default().add("leed", kind="credential") + assert a == Lexicon.default().add("leed", kind="credential") + assert hash(a) == hash(Lexicon.default().add("leed", kind="credential")) + # structural, not "the same compiled readings": a log that undoes + # itself compiles alike and is not equal (the decided cost) + undone = a.remove("leed", kind="credential") + assert _fields_of(undone._sets) == _fields_of(Lexicon.default()._sets) + assert undone != Lexicon.default() + # and the order is part of it + one = Lexicon.empty().add("x", kind="title").add("y", kind="title") + two = Lexicon.empty().add("y", kind="title").add("x", kind="title") + assert one != two and one._sets.titles == two._sets.titles + # the base is: an edit does not make a default-based lexicon empty + assert Lexicon.default().add("x", kind="title") != \ + Lexicon.empty().add("x", kind="title") + + +def test_a_later_edit_reads_after_an_earlier_one() -> None: + lex = Lexicon.empty() + assert lex.add("x", kind="title").remove("x")._sets.titles == frozenset() + assert lex.remove("x").add("x", kind="title")._sets.titles == {"x"} -def test_honorific_tails_is_a_vocab_field() -> None: - # every tail is also a suffix word -- the peeled piece is claimed - # by suffix classification -- so each construction carries both - lex = Lexicon.empty().add(honorific_tails={"씨", "さん"}, - suffix_words={"씨", "さん"}) - assert lex.honorific_tails == frozenset({"씨", "さん"}) - assert lex.remove(honorific_tails={"씨"}).honorific_tails == \ - frozenset({"さん"}) - merged = (Lexicon(honorific_tails=frozenset({"씨"}), - suffix_words=frozenset({"씨"})) - | Lexicon(honorific_tails=frozenset({"様"}), - suffix_words=frozenset({"様"}))) - assert merged.honorific_tails == frozenset({"씨", "様"}) - - -def test_removing_a_honorific_tail_is_the_peels_off_switch() -> None: - # The off-switch docs/customize.rst documents, at parse level: - # dropping a tail leaves the glued name unsplit while the SPACED - # spelling still routes the honorific, since suffix_words is a - # separate field and only the peel consults honorific_tails. - # honorific_tails is the one marker field a caller can empty - # without a matching base edit -- an orphan is what raises, and a - # removal cannot make one. - lex = Lexicon.default().remove(honorific_tails={"さん"}) - p = Parser(lexicon=lex) - glued = p.parse("田中さん") - assert (glued.family, glued.suffix) == ("田中さん", "") - spaced = p.parse("田中 さん") - assert (spaced.family, spaced.suffix) == ("田中", "さん") - # the baseline the removal is against, so the test cannot pass by - # the peel being broken outright - on = Parser().parse("田中さん") - assert (on.family, on.suffix) == ("田中", "さん") +def test_replace_carries_the_base_and_the_log() -> None: + lex = Lexicon.default().add("leed", kind="credential") + kept = dataclasses.replace(lex, surnames={"김"}) # type: ignore[arg-type] + assert kept._edits == lex._edits and kept._base == "default" + assert "leed" in kept._sets.suffix_acronyms + if sys.version_info >= (3, 13): + assert copy.replace(lex, given_names={"luis"})._edits == lex._edits # type: ignore[attr-defined] -def test_after_given_titles_is_a_vocab_field() -> None: - # #835: add, remove and | reach it, and it is normalized like every - # other field, so the period-marked entry matches the bare word - lex = Lexicon.empty().add(after_given_titles={"Mrs.", "mr"}) - assert lex.after_given_titles == frozenset({"mrs", "mr"}) - respelled = (Lexicon.default().remove(after_given_titles={"mrs"}) - .add(after_given_titles={"Mrs."})) - assert Parser(lexicon=respelled).parse("Doe, Jane Mrs").title == "Mrs" - assert lex.remove(after_given_titles={"mr"}).after_given_titles == \ - frozenset({"mrs"}) - merged = (Lexicon(after_given_titles=frozenset({"mrs"})) - | Lexicon(after_given_titles=frozenset({"miss"}))) - assert merged.after_given_titles == frozenset({"mrs", "miss"}) - with pytest.raises(TypeError): - Lexicon(after_given_titles="mrs") # type: ignore[arg-type] - edited = Lexicon.default().add(after_given_titles={"zzq"}) - assert pickle.loads(pickle.dumps(edited)) == edited - assert "after_given_titles: +1" in repr(edited) - - -def test_removing_an_after_given_title_is_rule_h8s_off_switch() -> None: - # Dropping the marker leaves the bare salutation a middle name after - # a family comma, while the period-marked spelling still reads as - # the title by H5's shape, which never consults the marker - lex = Lexicon.default().remove(after_given_titles={"mrs"}) - p = Parser(lexicon=lex) - assert (p.parse("Doe, Jane Mrs").title, - p.parse("Doe, Jane Mrs").middle) == ("", "Mrs") - assert p.parse("Doe, Jane Mrs.").title == "Mrs." - # the baseline the removal is against - assert Parser().parse("Doe, Jane Mrs").title == "Mrs" +def test_an_edit_log_not_written_by_add_or_remove_is_refused() -> None: + lex = Lexicon.empty().add("dr", kind="title") + for bad in ((("add", ("DR",), "title", ()),), + (("add", ("dr",), "titel", ()),), + (("rename", ("dr",), "title", ()),), + ("add",)): + with pytest.raises((TypeError, ValueError)): + dataclasses.replace(lex, _edits=bad) # type: ignore[arg-type] -def test_every_shipped_after_given_title_is_reachable() -> None: - # a property of the set, not its contents: each entry must reach - # rules/H8 at the slot it is listed for, so a member that another - # reading claims first (a suffix dual, a particle, a bound given - # name) shows up here as silently inert - p = Parser() - unreached = [w for w in sorted(p.lexicon.after_given_titles) - if p.parse(f"Doe, Jane {w.title()}").title != w.title()] - assert not unreached, unreached +def test_union_joins_the_logs_in_order_and_the_lists() -> None: + a = dataclasses.replace(Lexicon.empty().add("dr", kind="title"), + capitalization_exceptions=(("phd", "PhD"),)) + b = dataclasses.replace(Lexicon.empty().add("mr", kind="title"), + capitalization_exceptions=(("phd", "Ph.D."),), + surnames={"김"}) # type: ignore[arg-type] + u = a | b + assert u._sets.titles == frozenset({"dr", "mr"}) + assert u._edits == a._edits + b._edits + assert u.surnames == frozenset({"김"}) + assert u.capitalization_exceptions_map["phd"] == "Ph.D." # right wins + # later wins: b's removal applies after a's add + assert (a | Lexicon.empty().remove("dr"))._sets.titles == frozenset() + # the base that is not empty + assert (Lexicon.empty() | Lexicon.default())._base == "default" + assert (Lexicon.default() | Lexicon(surnames={"김"}))._base == "default" -@pytest.mark.parametrize(("lexicon", "text", "word"), [ - # in the marker, removed from the titles - (Lexicon.default().remove(titles={"mrs"}), "Doe, Jane Mrs", "Mrs"), - # added to the marker only - (Lexicon.default().add(after_given_titles={"zzq"}), "Doe, Jane Zzq", - "Zzq"), -], ids=["removed-from-titles", "marker-only"]) -def test_an_after_given_title_that_is_no_title_is_never_read( - lexicon: Lexicon, text: str, word: str) -> None: - # the orphan contract docs/customize.rst states: an entry is read - # only for a word that is a title, so an orphan is inert rather than - # an error (classify tags the marker only where the word is a title) - name = Parser(lexicon=lexicon).parse(text) - assert (name.title, name.middle) == ("", word) - - -@pytest.mark.parametrize("field", _PHRASE_FIELDS) -def test_a_shipped_phrase_entry_is_stored_as_written(field: str) -> None: - """The half of a phrase entry's storage rule config's own - assert_normalized cannot see. It checks the single-spaced lowercase - form; the per-WORD period strip is _lexicon._title_key's. So the - question is asked here, and it is stronger than the fold: does - Lexicon store the shipped constant UNCHANGED? An entry written - 'z. domu' passes import-time hygiene and is silently rewritten to - 'z domu' -- inert as a lookup key nobody wrote. - - Over _PHRASE_FIELDS rather than over maiden_markers, so a third - phrase field arrives already pinned.""" - from nameparser.config.maiden_markers import MAIDEN_MARKERS - from nameparser.config.titles import GIVEN_NAME_TITLES, TITLE_PHRASES - shipped = {"maiden_markers": MAIDEN_MARKERS, - "given_name_titles": GIVEN_NAME_TITLES, - "title_phrases": TITLE_PHRASES}[field] - assert getattr(Lexicon.default(), field) == frozenset(shipped), ( - f"{field}: Lexicon rewrote the shipped constant, so the entries " - f"it stores are not the ones the data module wrote -- " - f"{sorted(frozenset(shipped) - getattr(Lexicon.default(), field))} " - f"were folded away or changed") + +def test_union_refuses_two_different_bases() -> None: + from nameparser.config._index import READINGS + older = Lexicon(_base=READINGS[1:]) + assert older._base != "default" + with pytest.raises(ValueError, match="different bases"): + Lexicon.default() | older def test_the_internal_sets_hold_each_field_under_its_own_name() -> None: - # 4i-a: the parse reads Lexicon._sets, built from the fields by - # keyword. The roster pinned here so that a field added to one and - # not the other fails at once (4i-b adds the derived markers); the - # identity check, so that a field filled from the wrong set does. + # the roster pinned so that a field added to one and not the other + # fails at once; the kinds the views read follow the 2.x sets (4i-b) from nameparser._lexicon import _LexiconSets assert [f.name for f in dataclasses.fields(_LexiconSets)] == list( - _VOCAB_FIELDS) - for lexicon in (Lexicon.default(), Lexicon.empty().add(titles={"x"})): - for name in _VOCAB_FIELDS: + _VOCAB_FIELDS + _KIND_FIELDS) + for lexicon in (Lexicon.default(), Lexicon(surnames={"김"})): + for name in _LIST_FIELDS: assert getattr(lexicon._sets, name) is getattr(lexicon, name), name + + +def test_the_default_never_loads_the_reading_index() -> None: + # decisions/vocabulary-model, 2026-11-02 #754, item 4: the default's + # base is held by reference and its sets read from _compiled.py, so + # a parse that reads no word from the vocabulary loads no index; the + # first readings_of does + code = ("import sys, nameparser\n" + "nameparser.Lexicon.default()\n" + "nameparser.parse('John Smith')\n" + "before = 'nameparser.config._index' in sys.modules\n" + "nameparser.Lexicon.default().readings_of('dean')\n" + "print(before, 'nameparser.config._index' in sys.modules)\n") + out = subprocess.run([sys.executable, "-c", code], capture_output=True, + text=True, check=True).stdout + assert out.split() == ["False", "True"] + + +def test_lexicon_marks_an_added_field_with_the_shared_key() -> None: + # one key for every guard: a Lexicon field marked with the key the + # other types read, and the other way round + from nameparser import _types + assert _ADDED_IN == _types.ADDED_IN + + + +# -- pickle (decisions/vocabulary-model, 2026-11-02 #754, item 5) ---------- + +def test_pickle_round_trip_preserves_equality_and_cap_map() -> None: + # _cap_map holds a MappingProxyType, which pickle rejects; Lexicon + # must round-trip anyway because a Parser holds one + for lex in (Lexicon.default(), Lexicon.empty(), + Lexicon.empty().add("Dr.", kind="title"), + _quiet(lambda: Lexicon.default().add("dean", kind="title") + .remove("van", kind="given")), + Lexicon(surnames={"김"}, given_names={"luis"}, + capitalization_exceptions={"phd": "PhD"})): + loaded = pickle.loads(pickle.dumps(lex)) + assert loaded == lex + assert hash(loaded) == hash(lex) + assert (loaded.capitalization_exceptions_map + == lex.capitalization_exceptions_map) + + +def test_the_default_pickles_its_rows_and_loads_by_reference() -> None: + # a pickle carries the base's rows, so it parses as it did where it + # was written; rows equal to the running release's become the + # by-reference base again, so a round trip within a release is equal + from nameparser.config._index import READINGS + state = Lexicon.default().__getstate__() + assert state["_base"] == READINGS + assert pickle.loads(pickle.dumps(Lexicon.default()))._base == "default" + + +def test_another_releases_rows_stay_a_snapshot_and_parse_by_it() -> None: + from nameparser.config._index import READINGS + state = Lexicon.default().__getstate__() + # a release whose index lacked `dr` + state["_base"] = tuple(e for e in READINGS if e[0] != "dr") + older = Lexicon.__new__(Lexicon) + older.__setstate__(state) + assert older._base != "default" and older != Lexicon.default() + assert "dr" not in older._sets.titles + # bare: written `Dr.`, H2's shape reads a title whatever is listed + assert Parser(lexicon=older).parse("Dr John Smith").title == "" + assert pickle.loads(pickle.dumps(older)) == older + + +def test_a_snapshot_of_a_kind_this_release_does_not_know_is_refused() -> None: + state = Lexicon.default().__getstate__() + state["_base"] = (("zzq", (("en", "regnal", ()),)),) + with pytest.raises(ValueError, match="kind 'regnal'"): + Lexicon.__new__(Lexicon).__setstate__(state) + + +_UNSET = object() + + +def _mark(monkeypatch: pytest.MonkeyPatch, name: str, + default: object = _UNSET) -> None: + """Give a shipped Lexicon field the `_ADDED_IN` marker, as a field + added in a later release would carry, and optionally another + default. Patches the Field object `dataclasses.fields` returns, so + only the pickle guard sees it.""" + f = Lexicon.__dataclass_fields__[name] + monkeypatch.setattr(f, "metadata", + MappingProxyType({**f.metadata, _ADDED_IN: "9.9"})) + if default is not _UNSET: + monkeypatch.setattr(f, "default", default) + + +def _lexicon_state(*without: str, **renamed: str) -> dict[str, object]: + state = Lexicon.default().__getstate__() + for name in without: + del state[name] + for old, new in renamed.items(): + state[new] = state.pop(old) + return state + + +#: (field marked, what loads in its place, the refusal the same state +#: meets with the field unmarked). The third column is the negative +#: control: the guard before it learned the marker refused exactly +#: this, so a widening that stopped working fails the admitted half. +_ADMITTED = [ + ("given_names", frozenset(), + "missing fields: given_names; unexpected fields: none"), + ("capitalization_exceptions", (), + "missing fields: capitalization_exceptions; unexpected fields: none"), +] + + +@pytest.mark.parametrize("name,loaded,unmarked", _ADMITTED, + ids=[row[0] for row in _ADMITTED]) +def test_setstate_admits_a_marked_field_missing_from_an_older_pickle( + monkeypatch: pytest.MonkeyPatch, name: str, loaded: object, + unmarked: str) -> None: + state = _lexicon_state(name) + with pytest.raises(ValueError, match=re.escape(unmarked)): + Lexicon.__new__(Lexicon).__setstate__(dict(state)) + _mark(monkeypatch, name) + restored = Lexicon.__new__(Lexicon) + restored.__setstate__(dict(state)) + assert getattr(restored, name) == loaded + assert restored == dataclasses.replace(Lexicon.default(), **{name: loaded}) # type: ignore[arg-type] + assert restored.capitalization_exceptions_map == dict( + restored.capitalization_exceptions) + + +#: (marks, state, refusal): every skew the guard refused before it +#: learned the marker, still refused beside a marked field, the rows of +#: tests/v2/test_types.py's _REFUSED_SKEWS for the shared guard. +_LEXICON_REFUSED = [ + pytest.param({"given_names": _UNSET}, lambda: _lexicon_state("surnames"), + "missing fields: surnames; unexpected fields: none", + id="missing-unmarked-field"), + pytest.param({"given_names": _UNSET}, + lambda: _lexicon_state("surnames", "given_names"), + "missing fields: surnames; unexpected fields: none", + id="missing-unmarked-field-beside-a-missing-marked-one"), + pytest.param({"given_names": _UNSET}, + lambda: {**_lexicon_state(), "zq_future": frozenset()}, + "missing fields: none; unexpected fields: zq_future", + id="unexpected-field"), + pytest.param({"given_names": _UNSET}, + lambda: {**_lexicon_state("given_names"), + "zq_future": frozenset()}, + "missing fields: none; unexpected fields: zq_future", + id="unexpected-field-beside-a-missing-marked-one"), + pytest.param({"given_names": _UNSET}, + lambda: _lexicon_state(_edits="edits"), + "missing fields: _edits; unexpected fields: edits", + id="renamed-field"), + pytest.param({"given_names": _UNSET}, + lambda: _lexicon_state(given_names="given_name"), + "missing fields: none; unexpected fields: given_name", + id="renamed-marked-field"), + # a marker with no default to load is no license: the field is + # still required, as in the shared guard + pytest.param({"given_names": dataclasses.MISSING}, + lambda: _lexicon_state("given_names"), + "missing fields: given_names; unexpected fields: none", + id="marked-field-without-a-default"), + # the default is validated like a value the pickle held: one that + # does not survive normalization is refused, not silently rewritten + pytest.param({"surnames": frozenset({"DR."})}, + lambda: _lexicon_state("surnames"), + "default is not normalized (surnames: DR.)", + id="marked-field-whose-default-is-not-normalized"), +] + + +@pytest.mark.parametrize("marks,state,match", _LEXICON_REFUSED) +def test_setstate_still_refuses_every_skew_beside_a_marked_field( + monkeypatch: pytest.MonkeyPatch, marks: dict[str, object], + state: Callable[[], dict[str, object]], match: str) -> None: + for name, default in marks.items(): + _mark(monkeypatch, name, default) + with pytest.raises(ValueError, match=re.escape(match)): + Lexicon.__new__(Lexicon).__setstate__(state()) + + +def test_unpickling_rejects_unnormalized_entries() -> None: + # a genuine pickle always holds normalized entries, so state that + # does not is foreign: say so, at the site that can name the entry + state = Lexicon(surnames={"kim"}).__getstate__() + state["surnames"] = {"Kim ", "LEE."} + with pytest.raises(ValueError, match="not normalized"): + Lexicon.__new__(Lexicon).__setstate__(state) + state = Lexicon(capitalization_exceptions=(("phd", "PhD"),)).__getstate__() + state["capitalization_exceptions"] = ((" PhD. ", "Ph.D."), ("aa", "AA")) + with pytest.raises(ValueError, match="not normalized"): + Lexicon.__new__(Lexicon).__setstate__(state) + state = Lexicon.empty().add("dr", kind="title").__getstate__() + state["_edits"] = (("add", ("DR.",), "title", ()),) + with pytest.raises(ValueError, match="canonical"): + Lexicon.__new__(Lexicon).__setstate__(state) + + +def _state_2x(sets: dict[str, frozenset[str]], + masks: tuple[tuple[str, str], ...] = (), + **changed: frozenset[str]) -> dict[str, object]: + """A 2.8 Lexicon's pickled state: its 21 set fields, no + given_names (a 3.0 field), and its masks.""" + state: dict[str, object] = {f: sets[f] for f in _VOCAB_FIELDS + if f != "given_names"} + state.update(changed) + state["capitalization_exceptions"] = masks + return state + + +def _loaded(state: dict[str, object]) -> Lexicon: + lex = Lexicon.__new__(Lexicon) + lex.__setstate__(state) + return lex + + +def test_a_lexicon_parser_and_locale_pickled_by_2_8_0_load() -> None: + """tests/v2/data/lexicon_v28.pickle is a dict of + `Lexicon.empty().add(titles={"dean"})`, a `Parser` holding it and + `locales.RU`, pickled (protocol 4) by the released 2.8.0 wheel. + Made 2026-10-30 with `PYTHONSAFEPATH=1 uv run --isolated + --no-project --with 'nameparser==2.8.0'` from a directory outside + the checkout, version and module path asserted (AGENTS.md's v1 + comparison trap). pickle.load is safe here: a repo-controlled + fixture. + + A 2.x state holds the compiled sets, translated at load onto the + nearer named base and a per-word log (pickle option (b)): the + lexicon built from Lexicon.empty() loads as the one-entry log a + fresh 3.0 call writes. The pack and the policy are checked by + type and code, since whole equality with `locales.RU` or + `Parser().policy` would fail on any 3.0 change to their content.""" + from nameparser import Locale, Policy + with open(Path(__file__).parent / "data" / "lexicon_v28.pickle", + "rb") as f: + loaded = pickle.load(f) + lex = Lexicon.empty().add("dean", kind="title") + assert loaded["lexicon"] == lex + assert loaded["parser"].lexicon == lex + assert isinstance(loaded["parser"].policy, Policy) + assert isinstance(loaded["locale"], Locale) + assert loaded["locale"].code == "ru" + assert loaded["locale"].lexicon == Lexicon.empty() + # #959's given_names, added in 3.0, loads empty: no pack filled it + for loaded_lexicon in (loaded["lexicon"], loaded["parser"].lexicon, + loaded["locale"].lexicon): + assert loaded_lexicon.given_names == frozenset() + + +def test_a_2x_default_loads_as_the_default() -> None: + default = Lexicon.default() + sets = _fields_of(default._sets) | {"surnames": default.surnames} + loaded = _loaded(_state_2x(sets, default.capitalization_exceptions)) + assert loaded == default + + +def test_a_2x_title_listed_beside_its_gate_loads_as_a_title() -> None: + # the commonest 2.x customization, `add(titles={"dean", + # "chancellor"})`: a word in both titles and titles_ambiguous is + # compared as in titles alone, the two spellings parsing alike, and + # loads as the removal of its given reading + default = Lexicon.default() + sets = _fields_of(default._sets) | {"surnames": default.surnames} + loaded = _loaded(_state_2x( + sets, default.capitalization_exceptions, + titles=sets["titles"] | {"dean", "chancellor"})) + assert loaded == default.remove("chancellor", kind="given").remove( + "dean", kind="given") + for text in ("Dean Smith", "Chancellor Jones, Mary", "Smith, Dean"): + assert Parser(lexicon=loaded).parse(text).title.lower() in ( + "dean", "chancellor") + + +def test_a_2x_state_no_readings_express_is_refused_naming_the_fields() -> None: + # a given reading translated for particles_ambiguous also marks the + # acronym ambiguous, so the sets do not come back + empty: dict[str, frozenset[str]] = {f: frozenset() for f in _VOCAB_FIELDS} + with pytest.raises(ValueError, match="suffix_acronyms_ambiguous"): + _loaded(_state_2x(empty, particles={"xo"}, particles_ambiguous={"xo"}, # type: ignore[arg-type] + suffix_acronyms={"xo"})) # type: ignore[arg-type] + + +def test_a_2x_state_missing_a_field_or_not_normalized_is_refused() -> None: + empty: dict[str, frozenset[str]] = {f: frozenset() for f in _VOCAB_FIELDS} + state = _state_2x(empty) + del state["titles_ambiguous"] # a 2.6 pickle: added in 2.7 + with pytest.raises(ValueError, match="missing fields: titles_ambiguous"): + _loaded(state) + with pytest.raises(ValueError, match=re.escape("not normalized (titles: DR.)")): + _loaded(_state_2x(empty, titles={"DR."})) # type: ignore[arg-type] + + +# -- the refusals and translation arms no other test reaches ---------------- + +@pytest.mark.parametrize(("call", "error", "match"), [ + (lambda: Lexicon(surnames={"."}), ValueError, "normalizes to empty"), + (lambda: Lexicon.empty().add("x", kind="office", attaches=5), TypeError, "attaches must be an iterable"), + (lambda: Lexicon.empty().add("x", kind="office", attaches=["a", 5]), TypeError, "attaches entries must be strings"), + (lambda: Lexicon(_edits=(("add", "x", "title", ()),)), TypeError, "words are a tuple"), # type: ignore[arg-type] + (lambda: Lexicon(_edits=(("remove", ("x",), "title", (("a", 1),)),)), ValueError, "remove\\(\\) takes no properties"), # type: ignore[arg-type] + (lambda: Lexicon(_edits=(("add", ("x",), "title", 5),)), TypeError, "properties are pairs"), # type: ignore[arg-type] + (lambda: Lexicon(_edits=5), TypeError, "_edits must be a tuple"), # type: ignore[arg-type] + (lambda: Lexicon(_base=5), TypeError, "base is 'default', 'empty'"), + (lambda: Lexicon(_base=(("x", ("bad",)),)), TypeError, "snapshot is \\(word, rows\\)"), + (lambda: Lexicon(_base=(("x", (("en", ["title"], ()),)),)), TypeError, "the tag and the kind strings"), + (lambda: Lexicon(_base=(("x", ((3, "title", ()),)),)), TypeError, "the tag and the kind strings"), + (lambda: Lexicon(_base=(("x", (("en", "title", (("attaches", ["a"]),)),)),)), TypeError, "a value is a string"), + (lambda: Lexicon(_base=(("x", (("en", "title", (("nonsense", 1),)),)),)), ValueError, "the property 'nonsense'"), + (lambda: Lexicon(_base=(("x", (("en", "given", (("addresses", "given"),)),)),)), ValueError, "the property 'addresses'"), + (lambda: Lexicon(_base=(("x", (("en", "title", ("bad",)),)),)), TypeError, "has the row"), + (lambda: LexiconReading(5, "title", "title"), TypeError, "text must be a string"), # type: ignore[arg-type] + (lambda: Lexicon.default().readings_of(5), TypeError, "takes a word"), # type: ignore[arg-type] +]) +def test_a_malformed_argument_is_refused_by_name( + call: Callable[[], object], error: type[Exception], match: str) -> None: + with pytest.raises(error, match=match): + call() + + +def test_union_with_a_non_lexicon_is_not_implemented() -> None: + assert Lexicon.empty().__or__(5) is NotImplemented # type: ignore[operator] + + +def test_a_2x_state_with_a_malformed_field_is_refused() -> None: + empty: dict[str, frozenset[str]] = {f: frozenset() for f in _VOCAB_FIELDS} + with pytest.raises(ValueError, match="incompatible Lexicon pickle"): + _loaded(_state_2x(empty, titles=5)) # type: ignore[arg-type] + + +def test_a_2x_spelling_with_a_malformed_value_still_says_since_3_0() -> None: + with pytest.raises(TypeError, match="since 3.0"): + Lexicon.default().add(titles=5) # type: ignore[call-arg] + + +@pytest.mark.parametrize(("sets", "kinds"), [ + # an acronym reading on a word whose held suffix kind is a word kind + ({"suffix_acronyms": {"jr"}}, {"generation": (("acronym", True),)}), + # a credential the 2.x sets list as a word: acronym false + ({"suffix_words": {"zzc"}}, {"suffix": ()}), +]) +def test_the_translation_keeps_a_held_suffix_kind( + sets: dict[str, set[str]], kinds: dict[str, tuple[object, ...]]) -> None: + from nameparser._lexicon import _edits_toward + lex = Lexicon.default() + current = _fields_of(lex._sets) + target = {f: frozenset(current[f] | w) for f, w in sets.items()} + (word,) = next(iter(sets.values())) + target |= {f: current[f] - {word} for f in _SET_FIELDS + if f not in sets and word in current[f] + and f not in ("suffix_acronyms_ambiguous",)} + edits = _edits_toward(lex._pool(), current, target) + built = dataclasses.replace(lex, _edits=edits) + got = {r.kind: r.properties for r in built.readings_of(word) + if r.category == "suffix"} + assert got == kinds + + +def test_a_choice_translation_reads_the_kinds_it_is_given() -> None: + # the kind sets name a word's kind where a choice's delta moves it + # (`_apply_choice`): a patronymic, a credential written as a word, + # and a gated generation that a possibly-given particle reading + # beside it turns into a plain suffix word + from nameparser._lexicon import _edits_toward + lex = Lexicon.empty().add("zza", "zzb", kind="particle").add( + "zzc", kind="generation") + current = _fields_of(lex._sets) + target = dict(current) + target["patronymic_particles"] = frozenset({"zza"}) + target["suffix_words"] = current["suffix_words"] | {"zzb"} + target["credential_words"] = frozenset({"zzb"}) + target["particles"] = current["particles"] - {"zzb"} + target["generation_words"] = frozenset() + target["particles"] = target["particles"] | {"zzc"} + target["particles_ambiguous"] = frozenset({"zzc"}) + built = dataclasses.replace(lex, _edits=_edits_toward( + lex._pool(), current, target)) + assert {r.kind for r in built.readings_of("zza")} == {"patronymic"} + assert [(r.kind, r.properties) for r in built.readings_of("zzb")] == [ + ("credential", (("acronym", False),))] + assert {r.kind for r in built.readings_of("zzc")} == { + "suffix", "particle", "given"} + + +def test_a_2x_multi_word_title_is_refused_as_a_pickle_3_0_cannot_express() -> None: + # 2.x warned on a multi-word entry in a per-word set and kept it; a + # reading of a phrase is a birth marker or a given-addressing title + # alone (the build's check 8), so the load names the pickle + empty: dict[str, frozenset[str]] = {f: frozenset() for f in _VOCAB_FIELDS} + with pytest.raises(ValueError, match="3.0 readings cannot express"): + _loaded(_state_2x(empty, titles=frozenset({"grand moff"}))) + + +@pytest.mark.parametrize(("field", "word", "spelled"), [ + # 2.8 left the word in a marker its base no longer held, where no + # parse read it (the orphan contract): it loads as the removal + ("titles", "air", lambda L: L.remove("air", kind="modifier")), + ("conjunctions", "e", lambda L: L.remove("e", kind="conjunction")), +]) +def test_a_2x_removal_leaving_an_inert_marker_loads_as_the_removal( + field: str, word: str, spelled: Callable[[Lexicon], Lexicon]) -> None: + default = Lexicon.default() + sets = _fields_of(default._sets) | {"surnames": default.surnames} + loaded = _loaded(_state_2x(sets, default.capitalization_exceptions, + **{field: sets[field] - {word}})) + assert loaded == spelled(default) + for text in ("Air Marshal John Smith", "ANA SILVA E SOUZA"): + assert Parser(lexicon=loaded).parse(text).as_dict() == \ + Parser(lexicon=spelled(default)).parse(text).as_dict() + + +@pytest.mark.parametrize(("call", "said"), [ + (lambda: Lexicon.empty().add(titles={"zzq"}, title_modifiers={"zzq"}), # type: ignore[call-arg] + "write lexicon.add('zzq', kind='modifier')"), + (lambda: Lexicon.empty().add(titles={"zzq"}, title_styles={"zzq"}), # type: ignore[call-arg] + "write lexicon.add('zzq', kind='style')"), + (lambda: Lexicon.empty().add(conjunctions={"q"}, # type: ignore[call-arg] + conjunctions_ambiguous={"q"}), + "write lexicon.add('q', kind='conjunction', may_be_initial=True)"), + (lambda: Lexicon.empty().add(suffix_words={"zzq"}, # type: ignore[call-arg] + suffix_words_ambiguous={"zzq"}), + "write lexicon.add('zzq', kind='generation').add('zzq', kind='family')"), + (lambda: Lexicon.empty().add(suffix_words={"zzq"}, # type: ignore[call-arg] + honorific_tails={"zzq"}), + "write lexicon.add('zzq', kind='suffix', attached='after')"), + # an agnomen taken out of its set is the gated class's generation + (lambda: Lexicon.default().remove(agnomens={"neto"}), # type: ignore[call-arg] + "write lexicon.remove('neto', kind='agnomen').add('neto', kind='generation')"), + # a phrase taken out of a kind's reach lists the words left, bounded + (lambda: Lexicon.default().remove(title_phrases={"chief justice"}), + "more]"), + # one suffix kind per word: no readings give an acronym and a word + (lambda: Lexicon.default().add(suffix_acronyms={"jr"}), # type: ignore[call-arg] + "no readings give those words exactly those sets"), + # the given reading a possibly-given particle needs gates the + # credential too, so the nearest is offered + (lambda: Lexicon.empty().add(particles={"xo"}, # type: ignore[call-arg] + particles_ambiguous={"xo"}, + suffix_acronyms={"xo"}), + "the nearest is lexicon.add('xo', kind='particle')"), + # a phrase no reading takes: the translation itself is refused + (lambda: Lexicon.empty().add(titles={"grand moff"}), # type: ignore[call-arg] + "no readings give those words exactly those sets"), +]) +def test_each_2x_spelling_is_translated_or_said_to_have_none( + call: Callable[[], object], said: str) -> None: + with pytest.raises(TypeError, match=re.escape(said)): + call() + + +#: (field, word, text, the reading under the 2.x state with the word +#: removed from its base set and left in a marker): the marker goes with +#: the word, decided 2026-11-02 (decisions/vocabulary-model's #754, a +#: marker left behind entry). 2.x read the orphan in two places, and +#: each row pins the reading that changed, so a translation keeping the +#: orphan's effect fails here, where the set compare above cannot see it. +_LEFT_BEHIND = [ + # given_name_titles: 2.x's orphan still made H2's period-shape title + # address the given name, reading given 'Smith' + ("titles", "sir", "Sir. Smith", {"title": "Sir.", "family": "Smith"}), + ("titles", "sir", "Smith, Sir.", {"title": "Sir.", "family": "Smith"}), + # title_modifiers: 2.x's orphan still counted as listed in the + # capitals shape, reading given 'AIR', family 'John Smith' + ("titles", "air", "John Smith, AIR", + {"given": "John", "family": "Smith", "suffix": "AIR"}), +] + + +@pytest.mark.parametrize(("field", "word", "text", "reads"), _LEFT_BEHIND) +def test_a_marker_left_behind_goes_with_its_word( + field: str, word: str, text: str, reads: dict[str, str]) -> None: + default = Lexicon.default() + sets = _fields_of(default._sets) | {"surnames": default.surnames} + loaded = _loaded(_state_2x(sets, default.capitalization_exceptions, + **{field: sets[field] - {word}})) + with pytest.raises(TypeError, match="since 3.0") as raised: + default.remove(**{field: {word}}) # type: ignore[arg-type] + spelled = eval(str(raised.value).rsplit("write ", 1)[1], # noqa: S307 + {"lexicon": default}) + for lexicon in (loaded, spelled): + parsed = Parser(lexicon=lexicon).parse(text).as_dict() + assert {f: parsed[f] for f in reads} == reads + assert not {f for f, v in parsed.items() if v and f not in reads} diff --git a/tests/v2/test_local_agreement.py b/tests/v2/test_local_agreement.py index 4b8d96bdb..9b6b7601c 100644 --- a/tests/v2/test_local_agreement.py +++ b/tests/v2/test_local_agreement.py @@ -477,7 +477,7 @@ def test_a_score_read_in_workers_is_the_score_read_in_process( def test_a_score_in_workers_refuses_a_parser_it_cannot_send_them() -> None: from nameparser import Lexicon - configured = Parser(lexicon=Lexicon.default().add(titles={"partner"})) + configured = Parser(lexicon=Lexicon.default().add("partner", kind="title")) with pytest.raises(ValueError, match=r"parser_of\(''\)"): agreement.score(list(_ALL), _PSC_FIELDS, configured, jobs=2) agreement.score(list(_ALL), _PSC_FIELDS, configured) # the control: in process diff --git a/tests/v2/test_locale.py b/tests/v2/test_locale.py index 5474adf5d..d1a118653 100644 --- a/tests/v2/test_locale.py +++ b/tests/v2/test_locale.py @@ -53,7 +53,7 @@ def test_locale_validates_component_types() -> None: def test_locale_with_lexicon_pickles_round_trip() -> None: import pickle - loc = Locale(code="ru", lexicon=Lexicon.empty().add(titles={"Dr."})) + loc = Locale(code="ru", lexicon=Lexicon.empty().add("Dr.", kind="title")) assert pickle.loads(pickle.dumps(loc)) == loc diff --git a/tests/v2/test_locales.py b/tests/v2/test_locales.py index 8f5498418..93d9b2bd7 100644 --- a/tests/v2/test_locales.py +++ b/tests/v2/test_locales.py @@ -790,10 +790,10 @@ def test_ja_stacked_on_zh_pays_the_zh_packs_documented_cost() -> None: def test_pack_vocabulary_entries_are_single_words() -> None: # The shipped-vocabulary guard (test_lexicon.py's # test_default_lexicon_builds_warning_free), extended to the OTHER - # vocabulary source: a locale pack's lexicon fields are matched one - # word at a time same as Lexicon.default()'s, so a multi-word entry - # in any field but given_name_titles would be dead. Checked - # directly against field contents (not by capturing the + # vocabulary source: a locale pack's compiled sets (`_sets`) are + # matched one word at a time same as Lexicon.default()'s, so a + # multi-word entry in any set but given_name_titles would be dead. + # Checked directly against the sets' contents (not by capturing the # construction-time warning) because a pack module is imported -- # and its Locale built -- at most once per process; a later test in # this file may already have forced the import, which would make a @@ -809,7 +809,7 @@ def test_pack_vocabulary_entries_are_single_words() -> None: for field in _VOCAB_FIELDS: if field == "given_name_titles": continue - for entry in getattr(lexicon, field): + for entry in getattr(lexicon._sets, field): assert entry == "".join(entry.split()), ( f"{code}: {field} entry {entry!r} contains " f"whitespace and can never match") @@ -903,7 +903,7 @@ def test_ru_plus_tr_az_unions_patronymic_rules() -> None: def test_pack_over_custom_base() -> None: - base = Parser(lexicon=Lexicon.default().add(titles={"zqxcustom"})) + base = Parser(lexicon=Lexicon.default().add("zqxcustom", kind="title")) p = parser_for(locales.RU, base=base) n = p.parse("Zqxcustom Иван Петрович") assert n.title == "Zqxcustom" # base lexicon survives the fold diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 7b139071e..b6712f667 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -97,7 +97,7 @@ def test_module_parse_reuses_the_default_parser() -> None: def test_parser_for_stacks_locales() -> None: ru = Locale(code="ru", - lexicon=Lexicon.empty().add(titles={"г-н"}), + lexicon=Lexicon.empty().add("г-н", kind="title"), policy=PolicyPatch(patronymic_rules=frozenset( {PatronymicRule.EAST_SLAVIC}))) p = parser_for(ru) @@ -458,16 +458,17 @@ def _overlap_parser(policy: Policy | None = None) -> Parser: needs, whatever the shipped data says today: the leading words become titles and particles, the chained words ambiguous particles. - `particles` covers both because `_SUBSET_FIELDS` requires - `particles_ambiguous <= particles`; the leading words are deliberately - NOT made ambiguous, since that membership does nothing for them and - asserting it is how the wrong set got written down in the first place. + Both get a particle reading; the chained words a given reading too, + which is what makes a particle ambiguous. The leading words are given + none here, since that membership does nothing for them and asserting + it is how the wrong set got written down in the first place (the + shipped `freiherr`, `do` and `st` hold one of their own, which this + leaves as it is). """ - lex = Lexicon.default().add( - titles=_TITLE_PARTICLES, - particles=_TITLE_PARTICLES | _CHAINED_PARTICLES, - particles_ambiguous=_CHAINED_PARTICLES, - ) + lex = (Lexicon.default() + .add(*sorted(_TITLE_PARTICLES), kind="title") + .add(*sorted(_TITLE_PARTICLES | _CHAINED_PARTICLES), kind="particle") + .add(*sorted(_CHAINED_PARTICLES), kind="given")) return Parser(lexicon=lex, policy=policy or Policy()) @@ -477,21 +478,22 @@ def test_the_chained_emitter_is_reachable_by_construction() -> None: Asserted against a lexicon built here, so what it pins is the emitter rather than today's word lists. Three controls carry the weight, one - per membership the claim rests on: drop the leading word's `particles` - and the chain is gone; drop the chained word's `particles_ambiguous` - and the report is gone; drop the leading word's `particles_ambiguous` - and nothing moves at all -- which is the whole correction, since + per reading the claim rests on: drop the leading word's particle + reading and the chain is gone; drop the chained word's given reading + (its `particles_ambiguous` membership) and the report is gone; give + the leading word a given reading too and nothing moves at all -- which is the whole correction, since asserting THAT membership is what this file used to do. If this test fails, the emitter is gone or broken, or one of the two - words this test builds has lost a membership it supplies itself. An + words this test builds has lost a reading it supplies itself. An empty `titles & particles_ambiguous` in the SHIPPED vocabulary does not fail it and does not mean the emitter is unreachable. """ word = "zzoverlap" - base = Lexicon.default().add( - particles=_CHAINED_PARTICLES, particles_ambiguous=_CHAINED_PARTICLES) - overlap = base.add(titles={word}, particles={word}) + base = (Lexicon.default() + .add(*sorted(_CHAINED_PARTICLES), kind="particle") + .add(*sorted(_CHAINED_PARTICLES), kind="given")) + overlap = base.add(word, kind="title").add(word, kind="particle") text = f"{word} van Johnson" # the emitter, reached by construction @@ -505,7 +507,7 @@ def test_the_chained_emitter_is_reachable_by_construction() -> None: # control 1 -- leading word a plain title, NOT a particle: no chain # at all, and _assign reports the other side of the same fork. - title_only = Parser(lexicon=base.add(titles={word})).parse(text) + title_only = Parser(lexicon=base.add(word, kind="title")).parse(text) assert (title_only.given, title_only.family) == ("van", "Johnson") assert [a.detail for a in title_only.ambiguities] == [ "leading 'van' may be a family-name particle; read as a given name"] @@ -515,8 +517,7 @@ def test_the_chained_emitter_is_reachable_by_construction() -> None: # that there is no decision left to report. Without this control the # assertion above could not tell "the emitter ran" from "the chain # happened to produce this grouping". - unambiguous = dataclasses.replace( - overlap, particles_ambiguous=overlap.particles_ambiguous - {"van"}) + unambiguous = overlap.remove("van", kind="given") quiet = Parser(lexicon=unambiguous).parse(text) assert (quiet.given, quiet.family) == (chained.given, chained.family) assert quiet.ambiguities == () @@ -527,7 +528,7 @@ def test_the_chained_emitter_is_reachable_by_construction() -> None: # executable half of the correction; without it the claim that # `titles & particles` is the right set lives only in a comment, and # a fixture supplying all three sets could never contradict it. - also_ambiguous = overlap.add(particles_ambiguous={word}) + also_ambiguous = overlap.add(word, kind="given") same = Parser(lexicon=also_ambiguous).parse(text) assert (same.given, same.family) == (chained.given, chained.family) assert [a.detail for a in same.ambiguities] == [amb.detail] @@ -1028,7 +1029,7 @@ def test_each_suffix_or_name_branch_describes_itself() -> None: def test_parser_matches_uses_its_own_config() -> None: - p = Parser(lexicon=Lexicon.default().add(titles=["moff"])) + p = Parser(lexicon=Lexicon.default().add("moff", kind="title")) a = p.parse("Moff Tarkin") # ParsedName.matches falls back to the DEFAULT parser for the str # argument, which reads "Moff" as a given name -- mismatch. @@ -1209,7 +1210,7 @@ def test_revise_writes_a_connective_mark_the_sub_parse_did_not( # rather than by any stage. Without it the 'y' carries no mark at # all, drops as an ordinary family particle, and the field and # the view disagree again. - p = Parser(lexicon=Lexicon.default().add(particles={"y"})) + p = Parser(lexicon=Lexicon.default().add("y", kind="particle")) revised = p.revise(p.parse("John Smith"), family="and y") assert revised.family == "and y" assert revised.initials() == "J. a. y." @@ -1229,7 +1230,7 @@ def test_revise_keeps_r2s_precedence_over_the_connective_mark( # rendered answer is the same either way, which is why this # asserts the TAGS: the contract is that one mark is written, and # no rendered view tells the two apart. - p = Parser(lexicon=Lexicon.default().add(particles={"y"})) + p = Parser(lexicon=Lexicon.default().add("y", kind="particle")) revised = p.revise(p.parse("John Smith"), family="de y") assert revised.initials() == "J. d. y." y = [t for t in revised.tokens if t.text == "y"][0] @@ -1370,7 +1371,7 @@ def test_the_known_round_trip_exceptions_are_one_limit() -> None: # suffix-bearing corpus (two, measured 2026-09-10; three since # 2026-10-06). p = Parser() - tails = Lexicon.default().honorific_tails + tails = Lexicon.default()._sets.honorific_tails def ends_in_a_tail_without_being_one(name: str) -> bool: return any(word not in tails @@ -1994,7 +1995,7 @@ def test_a_phrase_marker_outranks_the_word_it_starts_with() -> None: # vocabulary reads, so the control below is what makes the first # assertion mean anything. configured = Parser(lexicon=Lexicon.default().add( - maiden_markers=["geb von"])) + "geb von", kind="birth_marker")) assert configured.parse("Jane Smith geb von Braun").maiden == "Braun" assert configured.parse("Jane Smith geb Braun").maiden == "Braun" assert parse("Jane Smith geb von Braun").maiden == "von Braun" @@ -2007,12 +2008,14 @@ def test_the_catalan_recipe_unmarks_the_link_for_one_case_names() -> None: `i` ships in the MARKED subset, so a name written wholly in one case reads it as an initial and reports the fork. A caller whose data is Catalan knows better, and the knob the rule names is - `remove(conjunctions_ambiguous={"i"})` -- which leaves the letter + `add("i", kind="conjunction", may_be_initial=False)` -- which + leaves the letter a connective and takes the one-case fork out of its way. The prose said so; nothing ran it. """ catalan = Parser( - lexicon=Lexicon.default().remove(conjunctions_ambiguous={"i"})) + lexicon=Lexicon.default().add( + "i", kind="conjunction", may_be_initial=False)) for text in ("JOSEP CAROD I ROVIRA", "josep carod i rovira"): name = catalan.parse(text) assert name.family == text.split(" ", 1)[1] @@ -2167,7 +2170,7 @@ def test_a_decomposed_conjunction_reports_as_its_composed_twin() -> None: by `й` only once a caller marks it ambiguous -- no default entry is decomposable.""" parser = Parser(lexicon=Lexicon.default().add( - conjunctions_ambiguous={"й"})) + "й", kind="conjunction", may_be_initial=True)) text = "ivan й petrov" composed = parser.parse(text) assert [a.kind for a in composed.ambiguities] == [ @@ -2189,8 +2192,8 @@ def test_a_decomposed_chunked_credential_reads_as_its_composed_twin() -> None: default policy; with that off and the acronym listed, the chunked gate is the only reader, and it read NFD `Ž.Šč.` as no credential.""" parser = Parser( - lexicon=Lexicon.default().add(suffix_acronyms={"žšč"}, - suffix_acronyms_ambiguous={"žšč"}), + lexicon=Lexicon.default().add("žšč", kind="credential") + .add("žšč", kind="family"), policy=Policy(unlisted_dotted_suffixes=False)) text = "John Smith Ž.Šč." decomposed = unicodedata.normalize("NFD", text) @@ -2246,7 +2249,7 @@ def test_a_vetted_cjk_credential_word_peels_off_a_glued_name( def test_a_declined_cjk_credential_word_does_not_peel_off_a_glued_name( word: str) -> None: # 長 alone is a borne given name, so 阿部長 can be 阿部 + 長 (#707). - assert word not in Parser().lexicon.honorific_tails + assert word not in Parser().lexicon._sets.honorific_tails assert Parser().parse(f"田中{word}").suffix == "" @@ -2371,19 +2374,23 @@ def test_the_ph_d_pair_merges_whatever_the_vocabulary_says() -> None: # without 'phd' still read the pair as one suffix; the split # credential's vocabulary pre-test must not stand in front of that # answer (#752's review) - p = Parser(lexicon=Lexicon.default().remove(suffix_acronyms={"phd"})) + p = Parser(lexicon=Lexicon.default().remove("phd", kind="credential")) assert p.parse("John Smith Ph. D.").suffix == "Ph. D." h = p.parse("John Smith, Ph. D.") assert (h.given, h.family, h.suffix) == ("John", "Smith", "Ph. D.") def test_a_given_name_written_short_listed_as_a_title_is_a_title() -> None: - # #789: the caller who lists the word as a title gets 1.4's reading + # #789: the caller who reads the word as a title gets 1.4's reading # back, the leading peel asking the titles before the shape the - # given_name_abbreviations set outranks + # given_name_abbreviations set outranks. Since 3.0's 4i-b the word + # leaves that set on the way: its given reading, which made it an + # abbreviation, would also keep a title reading one only inside a + # phrase (rules/H9), so a title everywhere is the given reading + # removed, and 2.x's title beside an abbreviation has no spelling assert parse("Chas. Smith").given == "Chas." # baseline - p = Parser(lexicon=Lexicon.default().add(titles={"chas"})) - assert "chas" in p.lexicon.given_name_abbreviations + p = Parser(lexicon=Lexicon.default().remove("chas", kind="given") + .add("chas", kind="title")) assert p.parse("Chas. Smith").title == "Chas." @@ -2397,20 +2404,21 @@ def _choosing(*tags: str, code: str = "choice", def test_the_compiled_fields_are_the_default_lexicons() -> None: - from nameparser._lexicon import _VOCAB_FIELDS, _vocab_fields + from nameparser._lexicon import _KIND_FIELDS, _VOCAB_FIELDS, _vocab_fields from nameparser._reading_index import compile_choice default = Lexicon.default() fields = _vocab_fields(compile_choice(frozenset())) # surnames is KOREAN_SURNAMES and given_names a pack's list (#959), - # neither compiled from the vocabulary + # neither compiled from the vocabulary; the kind sets the views read + # are compiled beside the 2.x fields assert {f for f in fields} | {"surnames", "given_names"} == set( - _VOCAB_FIELDS) + _VOCAB_FIELDS) | set(_KIND_FIELDS) for name, words in fields.items(): - assert getattr(default, name) == words, name + assert getattr(default._sets, name) == words, name def test_a_pack_choosing_no_language_leaves_the_vocabulary_as_it_is() -> None: - base = Parser(lexicon=Lexicon.default().add(titles={"zzq"})) + base = Parser(lexicon=Lexicon.default().add("zzq", kind="title")) assert parser_for(Locale(code="xx", lexicon=Lexicon.empty()), base=base).lexicon == base.lexicon @@ -2465,16 +2473,16 @@ def test_the_packs_choices_are_one_choice_in_any_order() -> None: def test_a_choice_is_a_delta_that_keeps_a_callers_vocabulary() -> None: - base = Parser(lexicon=Lexicon.default().add(titles={"zzq"})) + base = Parser(lexicon=Lexicon.default().add("zzq", kind="title")) got = parser_for(_choosing("pt"), base=base) - assert "zzq" in got.lexicon.titles + assert "zzq" in got.lexicon._sets.titles assert got.parse("ANA SILVA E SOUZA").family == "SILVA E SOUZA" def test_a_packs_own_fragment_is_unioned_on_after_the_choice() -> None: # pt drops en's credential reading of `do`; the pack's fragment puts # it back, as a caller listing a word has always asked for - pack = _choosing("pt", lexicon=Lexicon.empty().add(suffix_acronyms={"do"})) + pack = _choosing("pt", lexicon=Lexicon.empty().add("do", kind="credential")) assert parser_for(pack).parse("John Smith DO").suffix == "DO" @@ -2494,11 +2502,12 @@ def test_a_lowercase_region_selects_the_regions_file() -> None: def test_a_word_a_choice_drops_leaves_the_callers_markers_of_it_too() -> None: - # a caller marked `vd` possibly given; en's credential drops nl's - # preposition, so the word leaves `particles`, and the caller's mark - # would be an orphan the Lexicon refuses (found by review, 3b) - base = Parser(lexicon=Lexicon.default().add(particles_ambiguous={"vd"})) - got = parser_for(_choosing("en"), base=base).lexicon + # a caller marked `vd` possibly given (a given reading beside its + # particle one); en's credential drops nl's preposition, so the word + # leaves `particles`, and the caller's mark with it, a marker + # compiling only beside its base (found by review, 3b) + base = Parser(lexicon=Lexicon.default().add("vd", kind="given")) + got = parser_for(_choosing("en"), base=base).lexicon._sets assert "vd" not in got.particles and "vd" not in got.particles_ambiguous @@ -2506,16 +2515,16 @@ def test_an_error_applying_a_choice_names_the_choice( monkeypatch: pytest.MonkeyPatch) -> None: # an error the Lexicon raises while the choice is applied names the # languages chosen, as a patch error names its pack. No shipped - # choice raises, so a delta joining an orphan marker stands in for - # one + # choice raises, so a delta joining a mask that is not a string + # stands in for one (an orphan marker did until 4i-b, which left a + # choice no way to write one) import nameparser._parser as parser_mod - orphan: object = ({"particles_ambiguous": (frozenset(), frozenset({"zzq"}))}, - (frozenset(), ())) + bad_mask: object = ({}, (frozenset(), (("zzq", 5),))) monkeypatch.setattr(parser_mod, "_choice_delta", - lambda chosen, strict=False: orphan) - with pytest.raises(ValueError, + lambda chosen, strict=False: bad_mask) + with pytest.raises(TypeError, match="while applying the languages chosen, en, pt: " - "particles_ambiguous marks"): + "capitalization_exceptions entries must be"): parser_for(_choosing("pt", code="aa"), _choosing("en", code="bb")) @@ -2564,13 +2573,26 @@ def test_a_choice_chained_through_base_applies_after_the_bases() -> None: assert chained.lexicon != together.lexicon +def test_a_choice_whose_reading_clashes_with_the_callers_is_refused() -> None: + # cs turns on `st`'s generation beside a caller's credential, and a + # word holds one suffix kind: 3b's delta broke a set guard here, and + # 4i-b's readings refuse it too, rather than drop the caller's + # reading in silence (found by review of 4i-b) + from nameparser import locales + mine = Parser(lexicon=Lexicon.default().add("st", kind="credential")) + with pytest.raises(ValueError, match=( + r"while applying the languages chosen, cs: the readings the " + r"choice turns on clash with the lexicon's own for 'st'")): + parser_for(locales.CS, base=mine) + + def test_a_choice_drops_a_callers_own_entry_it_removes() -> None: # decided with the chained choices: the delta removes what the # choice drops whoever listed it, and keeps every untouched entry - mine = Lexicon(particles=frozenset({"do", "van"}), - surnames=Lexicon.default().surnames) + mine = Lexicon(surnames=Lexicon.default().surnames).add( + "do", "van", kind="particle") got = parser_for(_choosing("en"), base=Parser(lexicon=mine)).lexicon - assert got.particles == {"van"} + assert got._sets.particles == {"van"} def test_reading_languages_name_the_default_files_under_a_choice() -> None: @@ -2597,7 +2619,7 @@ def test_a_choice_that_changes_nothing_leaves_the_lexicon_itself() -> None: import nameparser._parser as parser_mod from nameparser._reading_index import changed_words, selected assert changed_words(selected(["sv"])) == frozenset() - mine = Lexicon.default().add(titles={"zzq"}) + mine = Lexicon.default().add("zzq", kind="title") assert parser_mod._apply_choice(mine, frozenset({"sv"})) is mine @@ -2670,7 +2692,7 @@ def test_a_file_with_no_reading_is_a_strict_choice_only() -> None: with pytest.raises(ValueError, match=r"vi\.toml holds no reading yet.*" r"strict=True"): parser_for(_choosing("vi")) - assert "do" not in parser_for(_choosing("vi"), strict=True).lexicon.particles + assert "do" not in parser_for(_choosing("vi"), strict=True).lexicon._sets.particles def test_a_strict_choice_still_refuses_mul_and_an_unknown_tag() -> None: @@ -2688,8 +2710,8 @@ def test_strict_is_a_bool() -> None: def test_a_strict_choice_is_a_delta_over_the_union_in_any_order() -> None: # a caller's listing of a dropped word leaves, a word no file holds # stays (measured in the decisions entry); the union, then strict - base = Parser(lexicon=Lexicon.default().add(titles={"dr", "zzq"})) - got = parser_for(_choosing("vi"), base=base, strict=True).lexicon + base = Parser(lexicon=Lexicon.default().add("dr", "zzq", kind="title")) + got = parser_for(_choosing("vi"), base=base, strict=True).lexicon._sets assert "zzq" in got.titles and "dr" not in got.titles ko, en = _choosing("ko", code="aa"), _choosing("en", code="bb") assert (parser_for(ko, en, strict=True).lexicon @@ -2698,5 +2720,49 @@ def test_a_strict_choice_is_a_delta_over_the_union_in_any_order() -> None: def test_a_packs_fragment_is_unioned_on_after_the_strict_compile() -> None: - pack = _choosing("ko", lexicon=Lexicon.empty().add(suffix_acronyms={"ra"})) + pack = _choosing("ko", lexicon=Lexicon.empty().add("ra", kind="credential")) assert parser_for(pack, strict=True).parse("Kim Min Ra").suffix == "Ra" + + +def _choice_tags() -> list[tuple[str, bool]]: + from nameparser._reading_index import UNCHOOSABLE, files, tags + return ([(t, False) for t in sorted(tags() - UNCHOOSABLE)] + + [(t, True) for t in sorted(files() - UNCHOOSABLE)]) + + +#: The lexicons a choice is written into: the default, the empty one, +#: and one a caller configured (a removed gate, an added title, two +#: added particles, one of them a word a choice turns a suffix on for) +_CHOICE_BASES = { + "default": lambda: Lexicon.default(), + "empty": lambda: Lexicon.empty(), + "configured": lambda: Lexicon.default().remove("van", "dean", kind="given") + .add("dean", kind="title").add("zzz", "st", kind="preposition"), +} + + +@pytest.mark.parametrize("tag,strict", _choice_tags(), + ids=lambda v: str(v)) +def test_a_choice_written_as_readings_gives_the_deltas_sets( + tag: str, strict: bool) -> None: + # 4i-b writes 3b's delta as readings (`_apply_choice` through + # `_lexicon._edits_toward`): for every single file tag a choice may + # name, plain and strict, on each of _CHOICE_BASES, the compiled sets + # are the delta applied to the base's sets, field by field, the kind + # sets the views read included (decisions/vocabulary-model's + # 2026-11-02 #754, sub-phase 4i-b entry counts this population) + from nameparser._lexicon import _MARKER_BASES, _fields_of + from nameparser._parser import ( + _apply_choice, _choice_delta, _reading_index_selected) + chosen = _reading_index_selected(frozenset({tag}), strict) + fields, _ = _choice_delta(chosen, strict) + for name, build in _CHOICE_BASES.items(): + lexicon = build() + current = _fields_of(lexicon._sets) + want = {f: (current[f] - gone) | joined + for f, (gone, joined) in fields.items()} + for marker, base in _MARKER_BASES: + if base in fields and fields[base][0]: + want[marker] = want.get(marker, current[marker]) - fields[base][0] + got = _fields_of(_apply_choice(lexicon, frozenset({tag}), strict)._sets) + assert got == current | want, name diff --git a/tests/v2/test_properties.py b/tests/v2/test_properties.py index 9d3e2152f..54e336d63 100644 --- a/tests/v2/test_properties.py +++ b/tests/v2/test_properties.py @@ -24,6 +24,7 @@ Script, parse, ) from nameparser._lexicon import _VOCAB_FIELDS +from nameparser._reading_index import CATEGORY as _READING_KINDS from nameparser._pipeline import run from nameparser._pipeline._state import (AMBIGUOUS_ACRONYM_TAG, ParseState) @@ -157,12 +158,14 @@ def test_a_leading_ambiguous_particle_is_reported_once_and_only_once( # vocabulary, for the reason test_parser.py's _TITLE_PARTICLES block # gives; a no-op against today's data, and #360-proof against # tomorrow's. - lex = Lexicon.default().add(titles={"freiherr"}, particles={"freiherr"}) + lex = (Lexicon.default().add("freiherr", kind="title") + .add("freiherr", kind="particle")) # bound-given prefixes are excluded, not overlooked: 'abu' is both # an ambiguous particle and a bound given prefix, so whether it # forks depends on whether the bound join fired -- a second rule, # covered by the case corpus rather than by this sweep - particles = sorted(lex.particles_ambiguous - lex.bound_given_names) + particles = sorted(lex._sets.particles_ambiguous + - lex._sets.bound_given_names) assert particles, "no ambiguous particles to exercise" failures = [] for particle in particles: @@ -1304,43 +1307,91 @@ def test_particle_fork_is_never_double_reported(text: str) -> None: "김", "남궁", "남", "毛", "欧阳", ]) -# given_name_titles is the one field matched as a space-joined run, so +# given_name_titles is the one set matched as a space-joined run, so # a multi-word phrase there is meaningful (not dead) and must not warn. # Derived from _VOCAB rather than duplicated, so the two pools cannot # drift apart. _TITLE_VOCAB = st.one_of(_VOCAB, st.just("grand duke")) +#: Each compiled vocabulary set, by its `_LexiconSets` name, and the +#: readings (kind, properties) a word drawn for it is given -- the +#: readings that compile a word into that set, since 3.0 configures a +#: Lexicon by reading (#754, 4i-b). One draw per set, as the 2.x +#: strategy drew one per field, so every set is fuzzed in every example. +_READINGS_FOR: dict[str, tuple[tuple[str, dict[str, object]], ...]] = { + "titles": (("title", {}),), + "given_name_titles": (("title", {"addresses": "given"}),), + "suffix_acronyms": (("credential", {}),), + "suffix_words": (("suffix", {}),), + "suffix_acronyms_ambiguous": (("credential", {}), ("family", {})), + "particles": (("particle", {}),), + "particles_ambiguous": (("particle", {}), ("given", {})), + "conjunctions": (("conjunction", {}),), + "conjunctions_ambiguous": (("conjunction", {"may_be_initial": True}),), + "bound_given_names": (("bound_given", {}),), + "given_name_abbreviations": (("given", {"same_as": "jack"}),), + "maiden_markers": (("birth_marker", {}),), + "honorific_tails": (("honorific", {"attached": "after"}),), + "after_given_titles": (("salutation", {"after_given": True}),), + "suffix_words_ambiguous": (("generation", {}), ("family", {})), + "agnomens": (("agnomen", {}), ("family", {})), + "titles_ambiguous": (("title", {}), ("given", {})), + "title_modifiers": (("modifier", {}),), + "title_styles": (("style", {}),), + # a phrase of the word and any title of kind `title` drawn + "title_phrases": (("title", {"attaches": ["title"]}),), +} -def _fix_invariants(**fields: frozenset[str]) -> dict[str, frozenset[str]]: - """Repair a random draw into a legal Lexicon instead of generating - one legally. - - Drawing dependent subsets directly (particles_ambiguous from - whatever particles happened to be drawn) makes the strategy tree - deep and mostly rejects; intersecting after the fact keeps every - draw usable and still reaches every shape. The seven rules are - Lexicon's own, restated here on purpose -- if one changes, this - fails loudly rather than silently fuzzing a narrower space. - """ - fields["particles_ambiguous"] &= fields["particles"] - fields["suffix_acronyms_ambiguous"] &= fields["suffix_acronyms"] - fields["suffix_words"] -= fields["suffix_acronyms_ambiguous"] - # order matters: must run AFTER the suffix_words subtraction above, - # or that later subtraction could re-orphan a tail this repair just - # fixed - fields["honorific_tails"] &= fields["suffix_words"] - fields["suffix_words_ambiguous"] &= fields["suffix_words"] - fields["suffix_words_ambiguous"] -= ( - fields["suffix_acronyms"] - fields["suffix_acronyms_ambiguous"]) - fields["bound_given_names"] -= ( - fields["particles"] - fields["particles_ambiguous"]) - return fields - - -# Derived, never listed: a new vocabulary field must be fuzzed the day -# it is added, and a hand-copied list would leave it silently unfuzzed -# -- the invisible gap this whole layer exists to prevent. +# Derived, never listed: a new vocabulary set must be fuzzed the day it +# is added, and a hand-copied list would leave it silently unfuzzed -- +# the invisible gap this whole layer exists to prevent. The lists stay +# constructor fields; every other set is reached through its readings, +# and a set with no entry above fails here rather than going unfuzzed. _SET_FIELDS = _VOCAB_FIELDS +assert set(_READINGS_FOR) == set(_SET_FIELDS) - {"surnames", "given_names"} + +#: The order the drawn readings are added in: titles first, so a title +#: reading never lands on a word already holding a given one (the add +#: that would warn, rules/H9), then the names, so a bound given name +#: that is also a particle holds its given reading before it is bound +#: (an add the Lexicon would refuse) -- the compiled sets are the same +#: in any order. +_CATEGORY_ORDER = ("title", "name", "suffix", "particle", "conjunction", + "bound_given", "relation") + + +def _fix_invariants(drawn: dict[str, frozenset[str]], + ) -> dict[str, dict[str, dict[str, object]]]: + """Repair a random draw into legal readings instead of generating + them legally, word -> kind -> properties. + + Drawing dependent memberships directly makes the strategy tree deep + and mostly rejects; repairing after the fact keeps every draw + usable and still reaches every shape. The three rules are the + Lexicon's own refusals, restated here on purpose -- if one changes, + this fails loudly rather than silently fuzzing a narrower space: a + word holds one suffix kind (the first drawn, in `_READINGS_FOR` + order, is kept); `may_be_initial` marks only a single cased letter; + and a bound given name that is a particle with no given reading + stops being a bound given name. + """ + out: dict[str, dict[str, dict[str, object]]] = {} + for name, readings in _READINGS_FOR.items(): + for word in sorted(drawn[name]): + held = out.setdefault(word, {}) + for kind, props in readings: + if _READING_KINDS[kind] == "suffix" and any( + _READING_KINDS[k] == "suffix" and k != kind for k in held): + continue + if props.get("may_be_initial") and not ( + len(word) == 1 and word.upper() != word.lower()): + props = {} + held.setdefault(kind, {}).update(props) + for word, held in out.items(): + if ("bound_given" in held and "given" not in held + and any(_READING_KINDS[k] == "particle" for k in held)): + del held["bound_given"] + return out @st.composite @@ -1357,14 +1408,24 @@ def _recased(draw: st.DrawFn, word: str) -> str: @st.composite def _lexicons(draw: st.DrawFn) -> Lexicon: - fields = {name: draw(st.frozensets( + drawn = {name: draw(st.frozensets( _TITLE_VOCAB if name == "given_name_titles" else _VOCAB, max_size=5)) for name in _SET_FIELDS} keys = draw(st.lists(_VOCAB, max_size=3)) caps = [(key, draw(_recased(key))) for key in keys] - return Lexicon(capitalization_exceptions=tuple(caps), - **_fix_invariants(**fields)) + lexicon = Lexicon(capitalization_exceptions=tuple(caps), + surnames=drawn["surnames"], + given_names=drawn["given_names"]) + readings = _fix_invariants(drawn) + adds = sorted( + ((_READING_KINDS[kind], kind, sorted(props.items()), word) + for word, held in readings.items() + for kind, props in held.items()), + key=lambda a: (_CATEGORY_ORDER.index(a[0]), a[1], repr(a[2]), a[3])) + for _category, kind, props, word in adds: + lexicon = lexicon.add(word, kind=kind, **dict(props)) + return lexicon # script_orders' legal values are as restricted as name_order's (only @@ -1435,7 +1496,7 @@ def _names_using(draw: st.DrawFn, lexicon: Lexicon, note. """ vocab = sorted({w for name in _SET_FIELDS - for w in getattr(lexicon, name)}) + for w in getattr(lexicon._sets, name)}) # script_segment (#271, #308) holds the two halves a space-joined # name can never reach, and BOTH need their token derived from the # draw: waiting for a drawn entry and a matching literal to @@ -1458,8 +1519,12 @@ def _names_using(draw: st.DrawFn, lexicon: Lexicon, # derandomize=True seed -- peel and surname split are told apart by # its tail_tag argument -- and origin/master's version of this # strategy counts ZERO surname splits against two peels, while this - # one counts 23 against one. That figure is reproducible by anyone - # in one command and moves only when these strategies do. + # one counted 23 against one when this was written; re-taken on + # 2026-11-02, the 2.8-era strategy at 3.0's 4i-b base counted 9 + # against 1, and the reading-built strategy that replaced it 7 + # against 5, its `attaches` making real title phrases in the pool. + # Reproducible in one command, and moving only when these + # strategies do. # Being in the POOL is not the same as being in the NAME, and for # the surname half that gap was the whole story. Two things must # coincide before the shape is legal at all -- a HANGUL surname @@ -1532,7 +1597,7 @@ def _names_using(draw: st.DrawFn, lexicon: Lexicon, # every index sampled_from draws -- which would defeat # derandomize=True on the whole strategy, not just this slice. unspaced = sorted(w + "민준" for w in lexicon.surnames) - glued = sorted("민준" + w for w in lexicon.honorific_tails) + glued = sorted("민준" + w for w in lexicon._sets.honorific_tails) # plain names and structure characters are always available, so the # pool is never empty even for an empty lexicon pieces = st.sampled_from( @@ -1631,8 +1696,14 @@ def test_config_values_are_hashable_and_reusable( _POLICY_FIELDS = tuple(f.name for f in dataclasses.fields(Policy)) -@given(st.sampled_from(_SET_FIELDS + ("capitalization_exceptions",)), - _HOSTILE) +#: What a caller configures a Lexicon through: its three fields, and +#: since 3.0 a reading of each kind, the word given to add() (#754, +#: 4i-b). Derived from the kind table, so a new kind is fuzzed too. +_LEXICON_INPUTS = ("surnames", "given_names", "capitalization_exceptions", + *(f"kind:{k}" for k in _READING_KINDS)) + + +@given(st.sampled_from(_LEXICON_INPUTS), _HOSTILE) @settings(max_examples=200, deadline=None, derandomize=True) def test_bad_lexicon_field_fails_cleanly(field: str, value: object) -> None: """A rejected configuration must be a DOCUMENTED rejection. @@ -1642,7 +1713,11 @@ def test_bad_lexicon_field_fails_cleanly(field: str, value: object) -> None: where the message no longer names the field the caller typed. """ try: - lexicon = Lexicon(**{field: value}) # type: ignore[arg-type] + if field.startswith("kind:"): + lexicon = Lexicon.empty().add( + value, kind=field[len("kind:"):]) # type: ignore[arg-type] + else: + lexicon = Lexicon(**{field: value}) # type: ignore[arg-type] except (ValueError, TypeError): return # Accepted, so it has to survive an actual parse -- construction @@ -1822,8 +1897,8 @@ def _class_letters(lexicon: Lexicon) -> frozenset[str]: grids ask this of every row and there are five lexicons between them. """ - return frozenset(w for w in lexicon.conjunctions - if w in lexicon.suffix_words) + return frozenset(w for w in lexicon._sets.conjunctions + if w in lexicon._sets.suffix_words) #: The two grids' lexicon variants, module-level so @@ -1831,13 +1906,13 @@ def _class_letters(lexicon: Lexicon) -> frozenset[str]: #: `_rows` for what the third element of each declares. _CONNECTIVE_LEXICONS: _Variants = ( ("default", Lexicon.default(), frozenset()), - ("conj+v", Lexicon.default().add(conjunctions={"v"}), + ("conj+v", Lexicon.default().add("v", kind="conjunction"), frozenset({"v"})), - ("part+y", Lexicon.default().add(particles={"y"}), + ("part+y", Lexicon.default().add("y", kind="particle"), frozenset({"y"}))) _OFF_SWITCH_LEXICONS: _Variants = ( ("default", Lexicon.default(), frozenset()), - ("conj+v", Lexicon.default().add(conjunctions={"v"}), + ("conj+v", Lexicon.default().add("v", kind="conjunction"), frozenset({"v"}))) @@ -2164,7 +2239,7 @@ def test_the_mark_is_exactly_the_parts_with_nothing_to_join() -> None: Stated over the PAIR of marks deliberately: keyed on the new mark alone it fails 272 times, every one of them under - `add(particles={"y"})`, where the part is all-particle and R2's + `add("y", kind="particle")`, where the part is all-particle and R2's OLD mark does the readmitting. Mutation-checked, re-measured 2026-09-20 on the trimmed grid: neutering the walk's lone-connective arm, so the new mark is never written, fails this @@ -2189,7 +2264,7 @@ def test_initials_take_only_base_words_and_drop_only_joiners() -> None: Mutation-checked, re-measured 2026-09-20 on the trimmed grid: swapping the two branches of the mark walk fails this on 94 parses (188 over the full cross product) -- `Carod y` under - `add(particles={"y"})`, whose base is empty while its `y` + `add("y", kind="particle")`, whose base is empty while its `y` initials. """ failures = _connective_findings()["INV3/4"] @@ -2319,7 +2394,8 @@ def _off_switch(parser: Parser, letters: frozenset[str]) -> Parser: key = (id(parser), letters) if key not in _OFF_PARSERS: _OFF_PARSERS[key] = Parser( - lexicon=parser.lexicon.remove(conjunctions=set(letters)), + lexicon=parser.lexicon.remove(*sorted(letters), + kind="conjunction"), policy=parser.policy) return _OFF_PARSERS[key] @@ -2482,7 +2558,7 @@ def _off_switch_findings() -> dict[str, list[str]]: # marked subset, so 'carod y rovira, v' and its ALL-CAPS twin # disagree, identically at the parent 46651750 (20 rows, # measured 2026-09-20, every one of them strict-comma). - if letters <= parser.lexicon.conjunctions_ambiguous: + if letters <= parser.lexicon._sets.conjunctions_ambiguous: roles_by[(label, text)] = tuple( (tok.text.lower(), tok.role.value, AMBIGUOUS_ACRONYM_TAG in tok.tags) for tok in on.tokens) @@ -2944,7 +3020,7 @@ def _link_texts() -> dict[str, str]: tokens and then read the first parse at those offsets. """ out: dict[str, str] = {} - markers = sorted(Lexicon.default().maiden_markers) + markers = sorted(Lexicon.default()._sets.maiden_markers) for head, marker, body, tail, comma in itertools.product( _LINK_HEADS, markers, _LINK_BODIES, _LINK_TAILS, (False, True)): @@ -3517,7 +3593,7 @@ def _without_particle_masks() -> Lexicon: default = Lexicon.default() return dataclasses.replace(default, capitalization_exceptions=tuple( (key, mask) for key, mask in default.capitalization_exceptions - if key not in default.particles)) + if key not in default._sets.particles)) @pytest.mark.parametrize("module", sorted(_P7_SITE_EFFECT)) @@ -3735,8 +3811,10 @@ def test_a_given_name_written_short_reads_as_a_given_name() -> None: shape fire somewhere, so a pass is not a grid the shape never reaches.""" lexicon = Lexicon.default() - off = lexicon.remove( - given_name_abbreviations=set(lexicon.given_name_abbreviations)) + abbreviations = sorted(lexicon._sets.given_name_abbreviations) + # the set emptied: each word keeps its given reading, without the + # full name that makes it one written short + off = lexicon.add(*abbreviations, kind="given", same_as=None) def fields(name: ParsedName, plain: str = "", head: str = "") -> dict[str, str]: return {k: v.replace(plain, head) if plain else v @@ -3754,7 +3832,7 @@ def fields(name: ParsedName, plain: str = "", head: str = "") -> dict[str, str]: # is compared only with the set emptied) stand_in = Parser(policy=dataclasses.replace( policy, unlisted_caps_suffixes=CapsSuffixes.OFF)) - for word in sorted(lexicon.given_name_abbreviations): + for word in abbreviations: # the bare spellings too, which the set must never move for head, plain in ((word.title() + ".", "Jack"), (word.upper() + ".", "JACK"), diff --git a/tests/v2/test_reading_index.py b/tests/v2/test_reading_index.py index add5e098d..ff9ee2af2 100644 --- a/tests/v2/test_reading_index.py +++ b/tests/v2/test_reading_index.py @@ -372,3 +372,8 @@ def test_strict_ungated_compares_with_the_plain_choice_not_the_default() -> None cs = frozenset({"cs"}) assert "st" not in ri.compile_choice(frozenset())["SUFFIX_WORDS"] assert "st" in ri.strict_ungated(cs) + + +def test_a_word_no_file_holds_as_a_suffix_has_no_suffix_kind() -> None: + assert ri.suffix_kind_of("dean") is None + assert ri.suffix_kind_of("zzqnotaword") is None diff --git a/tests/v2/test_render.py b/tests/v2/test_render.py index 490183c8e..ec9c35ede 100644 --- a/tests/v2/test_render.py +++ b/tests/v2/test_render.py @@ -279,11 +279,12 @@ def test_repair_keeps_a_conjunction_lowercase_in_a_particle_part() -> None: # mechanisms.md#VOCABULARY-OVERLAP-AS-PRECONDITION: "assert the # intersection as a precondition" -- built here rather than found, # so what has to hold is the half not being added. - assert "y" in Lexicon.default().conjunctions, ( + assert any(r.category == "conjunction" + for r in Lexicon.default().readings_of("y")), ( "'y' left the default conjunctions; this test builds the " "particle/conjunction overlap it needs from the other side and " "no longer has one. Pick another shipped conjunction.") - lex = Lexicon.default().add(particles={"y"}) + lex = Lexicon.default().add("y", kind="particle") p = Parser(lexicon=lex) van = p.parse("Anh y Van") @@ -313,11 +314,12 @@ def test_initials_readmits_a_conjunction_in_a_particle_part() -> None: values quoted in prose by decisions/, mechanisms.md and `_render.py` from going stale unnoticed. """ - assert "y" in Lexicon.default().conjunctions, ( + assert any(r.category == "conjunction" + for r in Lexicon.default().readings_of("y")), ( "'y' left the default conjunctions; this test builds the " "particle/conjunction overlap it needs from the other side and " "no longer has one. Pick another shipped conjunction.") - p = Parser(lexicon=Lexicon.default().add(particles={"y"})) + p = Parser(lexicon=Lexicon.default().add("y", kind="particle")) # the part the mark has turned into name words: every word of it # initials, the conjunction included, and the base agrees @@ -360,7 +362,7 @@ def test_initials_readmits_a_shared_word_between_particles() -> None: reaches -- INV2's connective grid never puts the shared word between particles. """ - p = Parser(lexicon=Lexicon.default().add(particles={"y"})) + p = Parser(lexicon=Lexicon.default().add("y", kind="particle")) de_y_la = p.parse("Juan de y la") assert de_y_la.family_base == "de y la" assert de_y_la.initials() == "J. d. y. l." @@ -865,7 +867,7 @@ def test_a_mask_recases_the_word_as_the_writer_punctuated_it() -> None: == "John Smith Ph. D." # the mask is asked BEFORE the acronym clause, which would give # 'BSC' -- bsc is a listed acronym too - assert "bsc" in Lexicon.default().suffix_acronyms + assert "bsc" in Lexicon.default()._sets.suffix_acronyms # forced: a mixed-case corpus name R5 would otherwise hold back assert str(parse("Dr. med. univ. Margit Popp, MSc").capitalized( force=True)) == "Dr. Med. Univ. Margit Popp, MSc" @@ -1484,7 +1486,8 @@ def test_a_link_inside_a_hyphenated_word_keeps_its_lowercase() -> None: assert mixed.capitalized() == mixed assert str(mixed.capitalized(force=True)) == "Jose Ortega-y-Gasset" # not connective vocabulary, so not reached (decisions/R4) - assert "a" not in Lexicon.default().conjunctions + assert not any(r.category == "conjunction" + for r in Lexicon.default().readings_of("a")) assert str(parse("Te Awanui-a-Rangi Black").capitalized( force=True)) == "Te Awanui-A-Rangi Black" # a single letter marked with a period is an initial there, @@ -1514,7 +1517,7 @@ def test_the_hyphen_is_the_writers_join_even_in_a_one_case_name() -> None: name is in. The cost is a one-case name whose hyphenated bare initials spell a connective -- 'J-E-P DUPONT' repairs to 'J-e-P Dupont', where 1.4.0 and the parent gave 'J-E-P'. And - conjunctions_ambiguous, P8's knob, does not reach a hyphenated + a `may_be_initial` conjunction reading, P8's knob, does not reach a hyphenated word: marking 'y' moves the spaced spelling only.""" for text, repaired in (("J-E-P DUPONT", "J-e-P Dupont"), ("JOHN A-Y-B SMITH", "John A-y-B Smith"), @@ -1522,7 +1525,7 @@ def test_the_hyphen_is_the_writers_join_even_in_a_one_case_name() -> None: ("maria silva e sousa", "Maria Silva E Sousa")): assert str(parse(text).capitalized()) == repaired, text marked = Parser(lexicon=Lexicon.default().add( - conjunctions_ambiguous={"y"})) + "y", kind="conjunction", may_be_initial=True)) for text, default, under_mark in ( ("JOSE ORTEGA Y GASSET", "Jose Ortega y Gasset", "Jose Ortega Y Gasset"), @@ -1544,8 +1547,8 @@ def test_a_shipped_mask_spells_its_suffix_in_either_single_case() -> None: lexicon = Lexicon.default() pairs = tuple((key, mask) for key, mask in lexicon.capitalization_exceptions - if key in lexicon.suffix_acronyms - or key in lexicon.suffix_words) + if key in lexicon._sets.suffix_acronyms + or key in lexicon._sets.suffix_words) assert pairs # an empty map would make the loop vacuous for key, mask in pairs: for text in (f"john smith {key}", f"JOHN SMITH {key.upper()}"): @@ -1565,7 +1568,7 @@ def test_a_shipped_particle_mask_outranks_the_particle_lowercase() -> None: lexicon = Lexicon.default() pairs = tuple((key, mask) for key, mask in lexicon.capitalization_exceptions - if key in lexicon.particles) + if key in lexicon._sets.particles) assert pairs # an empty selection would make the loop vacuous for key, mask in pairs + (("de", "de"),): for text in (f"john {key} smith", f"JOHN {key.upper()} SMITH"): @@ -1728,7 +1731,7 @@ def test_initials_has_no_lexicon_so_a_spliced_field_is_all_name_words()\ for giving `Parser` an `initials` crossing. A fallback to `Lexicon.default()` was written and dropped: it guesses a vocabulary, and under a caller's own the guess erases a whole - field (`Lexicon.default().add(particles={'y'})`, family 'de y'). + field (`Lexicon.default().add('y', kind='particle')`, family 'de y'). """ p = Parser() base = p.parse("john smith") @@ -1738,7 +1741,7 @@ def test_initials_has_no_lexicon_so_a_spliced_field_is_all_name_words()\ # the vocabulary a fallback would have had to guess, and the field # it erased when it guessed wrong - y_lex = Lexicon.default().add(particles={"y"}) + y_lex = Lexicon.default().add("y", kind="particle") py = Parser(lexicon=y_lex) assert py.parse("Juan de y").initials() == "J. d. y." assert py.parse("Juan Perez").replace(family="de y").initials() == "J. d. y." @@ -1906,7 +1909,7 @@ def test_a_marked_particle_lowercases_only_where_the_particle_arm_would( name = given_last.parse("IVOI PAUL D'") assert name.given == "D'" # the declared given slot, not P6's assert name.capitalized().given == "D'" - hyphened = Parser(lexicon=Lexicon.default().add(particles={"van-der"})) + hyphened = Parser(lexicon=Lexicon.default().add("van-der", kind="particle")) assert str(hyphened.capitalized(hyphened.parse("jan van-der"))) == \ "Jan Van-Der" diff --git a/tests/v2/test_reprs.py b/tests/v2/test_reprs.py index 41f83ac99..3813a948b 100644 --- a/tests/v2/test_reprs.py +++ b/tests/v2/test_reprs.py @@ -82,8 +82,8 @@ def test_policy_repr_shows_only_nondefault_fields() -> None: def test_lexicon_repr_is_bounded() -> None: assert repr(Lexicon.default()) == "Lexicon(default)" - lex = Lexicon.default().add(titles={"zqx", "zqy"}) - assert repr(lex) == "Lexicon(default + titles: +2)" + lex = Lexicon.default().add("zqx", "zqy", kind="title") + assert repr(lex) == "Lexicon(default + title: +2)" assert "zqx" not in repr(lex) # never dump contents @@ -107,8 +107,8 @@ def test_lexicon_repr_diffs_against_nearer_baseline() -> None: # An empty()-built Lexicon must not render as default() minus ~700 # entries -- diff against whichever named constructor is nearer. assert repr(Lexicon.empty()) == "Lexicon(empty)" - lex = Lexicon.empty().add(titles={"zqx"}) - assert repr(lex) == "Lexicon(empty + titles: +1)" + lex = Lexicon.empty().add("zqx", kind="title") + assert repr(lex) == "Lexicon(empty + title: +1)" def test_policy_patch_repr_shows_only_set_fields() -> None: diff --git a/tests/v2/test_shape_inventory.py b/tests/v2/test_shape_inventory.py index fc756326b..22300b805 100644 --- a/tests/v2/test_shape_inventory.py +++ b/tests/v2/test_shape_inventory.py @@ -103,7 +103,7 @@ def test_every_order_covers_every_position() -> None: _LEX = Lexicon.default() -_SUFFIXES = _LEX.suffix_words | _LEX.suffix_acronyms +_SUFFIXES = _LEX._sets.suffix_words | _LEX._sets.suffix_acronyms _DOTTED = re.compile(r"[^\W\d_]\.[^\W\d_]\.") _ABBREVIATION = re.compile(r"[^\W\d_]{2,}\.") _ROMAN = re.compile(r"(?=[IVXLCDM]+$)M*(C[MD]|D?C{0,3})(X[CL]|L?X{0,3})(I[XV]|V?I{0,3})") @@ -114,7 +114,7 @@ def _fold(word: str) -> str: def _is_title(word: str) -> bool: - return _fold(word) in _LEX.titles or bool(_ABBREVIATION.fullmatch(word)) + return _fold(word) in _LEX._sets.titles or bool(_ABBREVIATION.fullmatch(word)) def _head(t: str) -> list[str]: @@ -127,7 +127,7 @@ def _name_words(words: list[str]) -> int: word after it) count as none, a connective as its word.""" return sum(1 for w in words if not _is_title(w) and _fold(w) not in _SUFFIXES - and _fold(w) not in _LEX.particles) + and _fold(w) not in _LEX._sets.particles) def _after_comma(t: str) -> list[str]: @@ -143,19 +143,19 @@ def _after_comma(t: str) -> list[str]: bool(_after_comma(t)) and _is_title(_after_comma(t)[0])), "title-between": lambda t: len(t.split()) > 2 and not _is_title(t.split()[0]) and _is_title(t.split()[1]), - "title-end": lambda t: _fold(t.split()[-1]) in _LEX.titles, + "title-end": lambda t: _fold(t.split()[-1]) in _LEX._sets.titles, "title-after-comma": lambda t: bool(_head(t)) and _is_title(t.split(",")[-1].split()[0]), "title-after-empty-head": lambda t: t.startswith(",") - and _fold(t.split()[-1]) in _LEX.titles, + and _fold(t.split()[-1]) in _LEX._sets.titles, "title-particle-head": lambda t: all( - _fold(w) in _LEX.titles and _fold(w) in _LEX.particles + _fold(w) in _LEX._sets.titles and _fold(w) in _LEX._sets.particles for w in t.split()[:2]), "dual-in-title-run": lambda t: _is_title(t.split()[0]) - and _fold(t.split()[1]) in _LEX.titles + and _fold(t.split()[1]) in _LEX._sets.titles and _fold(t.split()[1]) in _SUFFIXES, "shape-title": lambda t: any( - _ABBREVIATION.fullmatch(w) and _fold(w) not in _LEX.titles + _ABBREVIATION.fullmatch(w) and _fold(w) not in _LEX._sets.titles for w in t.split()[:2]), "suffix-no-comma": lambda t: _fold(t.split()[-1]) in _SUFFIXES and "," not in t.rsplit(" ", 1)[0][-1:], @@ -166,7 +166,7 @@ def _after_comma(t: str) -> list[str]: "numeral": lambda t: any(_ROMAN.fullmatch(w) for w in t.split()), "dotted-suffix": lambda t: bool(_DOTTED.search(t.split()[-1])), "undotted-suffix": lambda t: "." not in t.split()[-1] - and _fold(t.split()[-1]) in _LEX.suffix_words, + and _fold(t.split()[-1]) in _LEX._sets.suffix_words, "comma-full-head": lambda t: _name_words(_head(t)) >= 2 and "," in t, "comma-one-word-head": lambda t: _name_words(_head(t)) == 1 and "," in t, @@ -174,11 +174,11 @@ def _after_comma(t: str) -> list[str]: "nickname-only-head": lambda t: "," in t and _head(t)[0][:1] in "(\"" and len(_head(t)) == 1, "connective-before-comma": lambda t: "," in t - and any(_fold(w) in _LEX.conjunctions for w in _head(t)), + and any(_fold(w) in _LEX._sets.conjunctions for w in _head(t)), "particle-before-comma": lambda t: "," in t - and any(_fold(w) in _LEX.particles for w in _head(t)), + and any(_fold(w) in _LEX._sets.particles for w in _head(t)), "middle": lambda t: _name_words(t.split(",")[0].split()) >= 3, - "particle": lambda t: any(_fold(w) in _LEX.particles + "particle": lambda t: any(_fold(w) in _LEX._sets.particles for w in t.split()), "nickname": lambda t: '"' in t or "(" in t, "nfd": lambda t: unicodedata.normalize("NFC", t) != t, diff --git a/tests/v2/test_types.py b/tests/v2/test_types.py index bd652c2d6..997342fed 100644 --- a/tests/v2/test_types.py +++ b/tests/v2/test_types.py @@ -355,19 +355,20 @@ def test_the_suffix_views_partition_it_by_reading( assert (pn.generation, pn.credentials) == (generation, credentials) -@pytest.mark.parametrize("field,kind", [ - # a caller's word in `suffix_words` alone has no row to read a kind - # from; its gate's marker or the agnomen set gives it one by its tag - ({"suffix_words": {"zzz"}}, "post-nominal"), - ({"suffix_words": {"zzz"}, "suffix_words_ambiguous": {"zzz"}}, - "generation"), - ({"suffix_words": {"zzz"}, "suffix_words_ambiguous": {"zzz"}, - "agnomens": {"zzz"}}, "agnomen"), +@pytest.mark.parametrize("kinds,kind", [ + # a caller's own suffix reading names no file; its kind, and the + # family reading that gates it by the count, give it a reading kind + (("suffix",), "post-nominal"), + (("generation", "family"), "generation"), + # the family reading first: an agnomen needs its gate (check 11) + (("family", "agnomen"), "agnomen"), ]) def test_a_callers_suffix_word_reads_by_its_marker_else_post_nominal( - field: dict[str, set[str]], kind: str) -> None: - pn = Parser(lexicon=Lexicon.default().add(**field)).parse( - "Pedro Alves Silva zzz") + kinds: tuple[str, ...], kind: str) -> None: + lexicon = Lexicon.default() + for k in kinds: + lexicon = lexicon.add("zzz", kind=k) + pn = Parser(lexicon=lexicon).parse("Pedro Alves Silva zzz") (tok,) = pn.tokens_for("suffix") assert tok.reading is not None and tok.reading.kind.value == kind assert pn.generation == ("" if kind == "post-nominal" else "zzz") @@ -1214,7 +1215,7 @@ def test_a_turkic_marker_a_caller_lists_as_a_connective_names_no_father( ) -> None: # the marker keeps the connective reading the caller's Lexicon gives # it, so neither it nor the word before it is a patronymic - parser = Parser(lexicon=Lexicon.default().add(conjunctions={"oglu"}), + parser = Parser(lexicon=Lexicon.default().add("oglu", kind="conjunction"), policy=Policy(patronymic_rules={"turkic"})) # type: ignore[arg-type] pn = parser.parse("Ali Ahmad Vali oglu Hasan") assert pn.patronymic == "" diff --git a/tests/v2/test_vocab.py b/tests/v2/test_vocab.py index 75fcc9230..deeec7d03 100644 --- a/tests/v2/test_vocab.py +++ b/tests/v2/test_vocab.py @@ -85,29 +85,12 @@ def test_the_shipped_files_pass_every_check_with_warnings_as_errors( def test_the_compiled_sets_are_the_default_lexicon(shipped: dict[str, str]) -> None: + from nameparser._lexicon import _fields_of, _vocab_fields c = _compile(shipped) lex = Lexicon.default() - rebuilt = Lexicon( - titles=c["TITLES"], given_name_titles=c["GIVEN_NAME_TITLES"], - suffix_acronyms=c["SUFFIX_ACRONYMS"], suffix_words=c["SUFFIX_WORDS"], - suffix_acronyms_ambiguous=c["SUFFIX_ACRONYMS_AMBIGUOUS"], - particles=c["PARTICLES"], - particles_ambiguous=c["PARTICLES"] - c["NON_GIVEN_NAME_PARTICLES"], - conjunctions=c["CONJUNCTIONS"], - conjunctions_ambiguous=c["CONJUNCTIONS_AMBIGUOUS"], - bound_given_names=c["BOUND_GIVEN_NAMES"], - given_name_abbreviations=c["GIVEN_NAME_ABBREVIATIONS"], - maiden_markers=c["MAIDEN_MARKERS"], surnames=lex.surnames, - honorific_tails=c["GLUED_HONORIFICS"], - after_given_titles=c["AFTER_GIVEN_TITLES"], - suffix_words_ambiguous=c["SUFFIX_WORDS_AMBIGUOUS"], - agnomens=c["AGNOMENS"], - titles_ambiguous=c["TITLES_AMBIGUOUS"], - title_modifiers=c["TITLE_MODIFIERS"], title_styles=c["TITLE_STYLES"], - title_phrases=c["TITLE_PHRASES"], - capitalization_exceptions=tuple(c["CAPITALIZATION_EXCEPTIONS"].items()), - ) - assert rebuilt == lex + assert _vocab_fields(c) == _fields_of(lex._sets) + assert lex.capitalization_exceptions == tuple( + sorted(c["CAPITALIZATION_EXCEPTIONS"].items())) def test_the_toml_ships_as_package_data() -> None: @@ -1073,7 +1056,9 @@ def _sets_fed(kind: str, props: str = "{}", text: str = "zzq", #: sets, against the same kind written bare, is the property's effect. _PROPERTY_PROBES = { "addresses": ("title", '{ addresses = "given" }', "zzq"), - "acronym": ("generation", "{ acronym = true }", "zzq"), + # a credential written as a word, the one acronym change that moves + # the kind set the credentials view reads (CREDENTIAL_WORDS, 4i-b) + "acronym": ("credential", "{ acronym = false }", "zzq"), "attached": ("honorific", '{ attached = { side = "after" } }', "zzq"), "may_be_initial": ("conjunction", "{ may_be_initial = true }", "q"), "cased": ("title", '{ cased = "ZzQ" }', "zzq"), @@ -1248,11 +1233,11 @@ def test_attaches_compiles_to_a_phrase_with_each_title_of_its_kinds() -> None: def test_a_shipped_phrase_names_two_titles() -> None: # every phrase the build writes is two words, each a title or a # title that may be a given name, so the phrase test can reach it - lex = Lexicon.default() - for phrase in lex.title_phrases: + sets = Lexicon.default()._sets + for phrase in sets.title_phrases: words = phrase.split(" ") assert len(words) == 2, phrase - assert all(w in lex.titles or w in lex.titles_ambiguous + assert all(w in sets.titles or w in sets.titles_ambiguous for w in words), phrase diff --git a/tools/vocab/build.py b/tools/vocab/build.py index 2e0bbfd70..dbd5a656e 100644 --- a/tools/vocab/build.py +++ b/tools/vocab/build.py @@ -1261,7 +1261,8 @@ def _behavior(r: Reading, prop: str) -> object: #: The constants _compiled.py defines, in its order, each with the -#: nameparser.config module that re-exports it. +#: nameparser.config module that re-exports it, or None for one only +#: `Lexicon` reads, a kind the public sets do not record (#754, 4i-b). CONSTANTS = ( ("GIVEN_NAME_TITLES", "titles"), ("TITLES", "titles"), @@ -1284,6 +1285,9 @@ def _behavior(r: Reading, prop: str) -> object: ("SUFFIX_ACRONYMS_AMBIGUOUS", "suffixes"), ("SUFFIX_ACRONYMS", "suffixes"), ("CAPITALIZATION_EXCEPTIONS", "capitalization"), + ("GENERATION_WORDS", None), + ("CREDENTIAL_WORDS", None), + ("PATRONYMIC_PARTICLES", None), ) @@ -1365,7 +1369,8 @@ def compile_files(files: Mapping[str, str]) -> Compiled: {REFRESH} The public names are re-exported, with their documentation, by the -nameparser.config modules (titles, particles, suffixes, ...). +nameparser.config modules (titles, particles, suffixes, ...); the last +three, the kinds those sets do not record, are read by Lexicon alone. """ '''