From 872974c45e9f5e6a51f58a7ae31e169d2950f03e Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 3 Nov 2026 18:19:14 -0800 Subject: [PATCH 1/2] Take a hyphenated surname under two_family_names; example both switches (#959) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit rules/O6 now counts a hyphenated word as one name word when it is the word taken, so `Andrés Sánchez-Cid Torres` under `two_family_names` gives family 'Sánchez-Cid Torres' where 2.8.0 kept 'Sánchez-Cid' a middle name. A hyphenated word every part of which is a listed given name stays (`Ana María-José López` under the Spanish list), and so do hyphenated initials. A hyphen in the family still counts its words. Decided in chat over ORCID's self-split names: under locales.ES the change fixes 186 of 10,794 hyphenated records and breaks 3, where counting a hyphen as one everywhere broke 520. rules/O7's precedence over O6 gains the both-switches example `María del Carmen de la Vega`, which decisions/O6 listed as Open. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/O6-two-family-names.md | 6 ++- docs/design/rules/O6-two-family-names.md | 10 ++-- docs/design/rules/O7-two-given-names.md | 4 +- docs/release_log.rst | 1 + nameparser/_pipeline/_post_rules.py | 54 ++++++++++++++------ tests/v2/cases.py | 24 +++++++-- tests/v2/pipeline/test_post_rules.py | 8 ++- tests/v2/rules_budget.json | 4 +- tests/v2/test_types.py | 2 +- tools/differential/corpus_rules.jsonl | 3 ++ tools/differential/snapshot.jsonl | 3 ++ 11 files changed, 88 insertions(+), 31 deletions(-) diff --git a/docs/design/decisions/O6-two-family-names.md b/docs/design/decisions/O6-two-family-names.md index 8f91f7562..ac13efdbb 100644 --- a/docs/design/decisions/O6-two-family-names.md +++ b/docs/design/decisions/O6-two-family-names.md @@ -6,7 +6,7 @@ - `locales.ES` turns the switch on, carrying the Spanish given-name list beside it, one pack for Spain and Latin America alike, Argentina included; `locales.PT` leaves it off until a Portuguese given-name list exists, decided in chat — 2026-10-30 #753, phase 3g. - `locales.CS` and `locales.SK` turn the switch on with no given-name list, a net gain in every real population measured (CLDR's fictional renderings lose), two given names the recorded cost; `locales.SL` and `locales.HR` leave it off, decided in chat — 2026-10-30 #753, phase 3f amended. - Both family names stay in `family` through 2.x; a second-surname field is the RFC's Phase 4 output model (docs/design/vocabulary-model-rfc.md, "The rules a pack can choose"), and a pack may later switch this rule on — 2026-10-24 #754; landed as the `surname2` view, `family` keeping both, 2026-10-29 #754, sub-phase 4d. -- The family's words are counted as P1's fold counts them, a connective join and a hyphen counting as their words, so a family already holding two keeps what it has; the word taken must count as one by the same count and be no initial written alone — 2026-10-24 #754. +- The family's words are counted as P1's fold counts them, a connective join and a hyphen counting as their words, so a family already holding two keeps what it has; the word taken must count as one, a hyphenated word counting as one there, and be no initial written alone nor hyphenated initials — 2026-10-24 #754; 2026-11-03 #959 (Derek). - A name its script (W4) reads in an order other than the declared `name_order` is left alone; where the two agree it is read like any other — 2026-10-24 #754. - Under a family-first order the first two name words are the family, the remainder laid out as the positions after a family — 2026-10-24 #754. - The switch runs before `middle_as_family` and is not refused beside it: given-first, the two together read as `middle_as_family` alone — 2026-10-24 #754. @@ -16,7 +16,7 @@ - The veto stays a clause of this rule though it reads a second signal, the taken word looked up in a list, and the rule is a split candidate — 2026-10-30 #959, decided. - The Spanish list, which `locales.ES` carries as its lexicon fragment since sub-phase 3g (a caller may pass a list of its own), is generated, not per-word vocabulary: INE's 2025 census tables, pooled over every name borne in Spain, cut at a measured rank (1,000 words, 90% given use) by `tools/es_given_names.py`; a change to it is a cutoff change — 2026-10-30 #959. -Open: hyphenated surnames block the switch (`Andrés Sánchez-Cid Torres` keeps `Sánchez-Cid` a middle name), of the 466 Spanish records the issue's held-out per-country list still read wrong, 271 are this shape (measured on the issue, 2026-10-30) — #959. No rule example shows both switches on over a given name a particle completes (`María del Carmen de la Vega`, given `María del Carmen` under O7 alone and family `del Carmen de la Vega` with O6 on too) — #959. Why the Spanish list does less in Latin America, and whether a list per region or another selection rule would serve it better than the one list `locales.ES` carries (the 2026-10-30 #959 entry's Latin America bullet; the 2026-10-30 #753 phase 3g entry measures the pack against the default there) — #959. A name written with ñ dropped (`Begona`, `Inigo`) matches no entry, the key keeping the tilde; whether to fold ñ for data that strips it — #959. +Open: why the Spanish list does less in Latin America, and whether a list per region or another selection rule would serve it better than the one list `locales.ES` carries (the 2026-10-30 #959 entry's Latin America bullet; the 2026-10-30 #753 phase 3g entry measures the pack against the default there) — #959. A name written with ñ dropped (`Begona`, `Inigo`) matches no entry, the key keeping the tilde; whether to fold ñ for data that strips it — #959. ## Log @@ -163,3 +163,5 @@ Open: hyphenated surnames block the switch (`Andrés Sánchez-Cid Torres` keeps | ORCID, country SK | 12,962 | 94.84% → 96.40% | 347 | 144 | Every broken record, in all three, has two or more words in the given box, read as a given name and a family name: no population-wide Czech or Slovak given-name list is published to veto the word, the register's frequencies having been withdrawn after 2016's; ČSÚ's top 100 newborn given names a year (CC BY 4.0) is a list, and the open lead. The circular ceiling such a list could reach, and the sources looked for, are in the vocabulary-model entry. +- 2026-11-03 #959 (Derek) — A HYPHENATED WORD COUNTS AS ONE WHEN IT IS THE WORD TAKEN, UNLESS EVERY PART OF IT IS A LISTED GIVEN NAME OR AN INITIAL. SUPERSEDES, in part, the 2026-10-24 #754 entry's count of the word taken, in which a hyphen joined two words without making them one, so `Andrés Sánchez-Cid Torres` kept `Sánchez-Cid` a middle name (the #754 review's reading of `Ana María-José López`, middle 'María-José', which the first draft had read as family 'María-José López'). The issue found this shape behind 271 of the 466 Spanish records the given-name veto still read wrong. Measured 2026-11-03 on the cached `orcid-researchers` corpus, ES, MX, CO, PE, CL, VE and AR at most 20,000 records a country with both boxes (135,735 records), over the 10,794 distinct records holding a hyphen in either box, rendered given first and read right where both the family and the given names match the boxes (`local_agreement.fold`), under `parser_for(locales.ES)`, on 39839b91 against prototypes of three readings: a hyphen counting as one everywhere O6 counts (family and word taken) reads 10,040 right against 10,281, fixing 277 and breaking 520, most of them a family box of two surnames written with a hyphen (`Garcia-Gomez`), which then counts as one and takes a given name; a hyphen counting as one in the word taken alone reads 10,464, fixing 186 (ES 125, MX 24, CL 20, PE 6, CO 5, VE 5, AR 1) and breaking 3 (`Aristide Tanyi-Jong Akem`, `Hilda Ines-Romero Menacho`, `Benjamin Moraga-Arias .`); and the same with a veto on a hyphenated word every part of which is a listed given name, which moves none of these records. Decided: the word-taken count with the veto, the veto keeping the rule's own `Ana María-José López` where the Spanish list is loaded (middle 'María-José'), while without a list the switch takes it (family 'María-José López'), its usual cost (rules/O6's Accepted). Hyphenated initials (`Ana M.-J. López`) stay as an initial written alone stays, found building it. A hyphen in the family still counts its words, so `Javier Olmedo Pérez-Montaut` keeps `Olmedo` a middle name: reading `Pérez-Montaut` as one surname is the 520 above. The built change reads the same 10,464 and moves the same 186 and 3, and with the switch alone (`Policy(two_family_names=True)`, no list) 10,292 to 10,475, the same records. Recompute: for each record of `local_agreement.records(local_corpora.load("orcid-researchers"))` in those countries, both boxes non-empty, at most 20,000 a country, keep those with a hyphen (`_post_rules._HYPHENS`) in either box, parse `f"{given} {family}"` with each tree's parser, `nameparser.__file__` asserted, and compare. +- 2026-11-03 #959 — BOTH SWITCHES OVER A GIVEN NAME A PARTICLE COMPLETES GET AN EXAMPLE. The 2026-10-30 entry's Open named the shape no rule example showed: `María del Carmen de la Vega` reads family 'del Carmen de la Vega' with both switches on, O6 taking the chain before O7 can complete the given name with it, and given 'María del Carmen' under O7 alone. rules/O7's precedence bullet against two-family-name handling now says so and carries both rows; no behavior moves. diff --git a/docs/design/rules/O6-two-family-names.md b/docs/design/rules/O6-two-family-names.md index 9b3ab6d9d..fbd6eb6fa 100644 --- a/docs/design/rules/O6-two-family-names.md +++ b/docs/design/rules/O6-two-family-names.md @@ -16,9 +16,11 @@ "García Márquez Gabriel" family-first+two_family_names → given="Gabriel" "García Márquez Ana María" family-first-given-last+two_family_names → given="María" ``` -- **A name word is counted as P1's fold counts one**, so a particle run counts with the word it attaches to: "de la Vega" is one word and takes the word before it. +- **A name word is counted as P1's fold counts one**, so a particle run counts with the word it attaches to: "de la Vega" is one word and takes the word before it. A hyphenated word counts as one when it is the word taken, where a hyphen in the family counts its words (Except). ``` "Juan Pérez de la Vega" two_family_names → family="Pérez de la Vega" + "Andrés Sánchez-Cid Torres" two_family_names → family="Sánchez-Cid Torres" + "Torres Sánchez-Cid Andrés" family-first+two_family_names → family="Torres Sánchez-Cid" ``` - **The family keeps the two as units**, the one written second being the second-surname view (R10), whichever of the two the rule took. ``` @@ -59,13 +61,14 @@ ``` ", Ana García López" two_family_names → family="" ``` -- **What is taken must be one name word a family name could be**: a part counting as two words (a hyphenated given name, a bound given-name pair, P5) stays where it is, and so do an initial written alone, which writes a given name, and a mark or number with no letter in it. A particle chain (P2) is never split, so where taking it would leave no given name the rule declines. A listed given name stays where it is too, under either order, and a list written without accents matches the word written with them: the caller's vocabulary names it a word only a given name is, so the name holds one family name. A listed word behind a particle is a surname unit, taken as before. +- **What is taken must be one name word a family name could be**: a bound given-name pair (P5), counting as two words, stays where it is, and so do an initial written alone or hyphenated initials, which write given names, and a mark or number with no letter in it. A particle chain (P2) is never split, so where taking it would leave no given name the rule declines. A listed given name stays where it is too, under either order, and so does a hyphenated word every part of which is one, and a list written without accents matches the word written with them: the caller's vocabulary names it a word only a given name is, so the name holds one family name. A listed word behind a particle is a surname unit, taken as before. ``` "José Luis Pérez" two_family_names+es_given_names → family="Pérez" "Jose Luis Perez" two_family_names+es_given_names → middle="Luis" "Pérez José Luis" family-first+two_family_names+es_given_names → family="Pérez" "Pérez Luis José" family-first-given-last+two_family_names+es_given_names → family="Pérez" - "Ana María-José López" two_family_names → family="López" + "Ana María-José López" two_family_names+es_given_names → family="López" + "Ana M.-J. López" two_family_names → family="López" "Ana abdul Rahman López" two_family_names → family="López" "Ana M. García" two_family_names → family="García" "Ana María García - López" two_family_names → family="López" @@ -99,6 +102,7 @@ - A name with one family name behind two given names loses its second given name to the family, which is the shape the switch cannot tell from the one it is for: most Argentine and English names of three words or more read wrong under it, and so do Portuguese and Brazilian names whose writer gave the last family name alone as the family. That is why it is off by default and a caller turns it on only for data written this way. ``` "Juan Carlos Pérez" two_family_names → family="Carlos Pérez" + "Ana María-José López" two_family_names → family="María-José López" ``` - A patronymic is a name word like any other here, a patronymic rule on or off, so where no rotation fired it can join the family. ``` diff --git a/docs/design/rules/O7-two-given-names.md b/docs/design/rules/O7-two-given-names.md index 4e624333b..5eb2aa3cc 100644 --- a/docs/design/rules/O7-two-given-names.md +++ b/docs/design/rules/O7-two-given-names.md @@ -68,7 +68,7 @@ **Precedence:** -- **Against two-family-name handling** ("Gabriel García Márquez" with both switches on): that rule reads first, since the caller who turns both on says the family holds two names, and this rule takes from the middles it leaves. A name of three words then keeps one given name, and a name of four whose third word that rule takes reads two and two. A listed given name that rule leaves where it is (O6) is a middle, which this rule takes where it stands beside the given name, so a name of three words whose middle is listed reads two given names and one family name. +- **Against two-family-name handling** ("Gabriel García Márquez" with both switches on): that rule reads first, since the caller who turns both on says the family holds two names, and this rule takes from the middles it leaves. A name of three words then keeps one given name, and a name of four whose third word that rule takes reads two and two. A listed given name that rule leaves where it is (O6) is a middle, which this rule takes where it stands beside the given name, so a name of three words whose middle is listed reads two given names and one family name. A given name a particle completes is a particle chain that rule takes into the family first, so it is the family's with both on, where this rule alone reads it as the given name. ``` "Gabriel García Márquez" two_family_names+two_given_names → given="Gabriel" "José Luis Pérez" two_family_names+two_given_names+es_given_names → given="José Luis" @@ -76,6 +76,8 @@ "Ana María Luis Pérez" two_family_names+two_given_names+es_given_names → middle="Luis" "María José García López" two_family_names+two_given_names → given="María José" "María José García López" two_family_names+two_given_names → family="García López" + "María del Carmen de la Vega" two_family_names+two_given_names → family="del Carmen de la Vega" + "María del Carmen de la Vega" two_given_names → given="María del Carmen" ``` - **Against compound-family handling** ("Hassan Mohamad Ali" with both switches on): this rule reads first, as a bound given-name pair (P5) joins before the fold, and O3 then folds whatever middles are left. ``` diff --git a/docs/release_log.rst b/docs/release_log.rst index 6acda49f7..221a85cf9 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -36,6 +36,7 @@ Release Log **Behavior Changes** + - **Fix two_family_names leaving a hyphenated surname out of the family.** ``Parser(policy=Policy(two_family_names=True)).parse("Andrés Sánchez-Cid Torres")`` gives family ``Sánchez-Cid Torres``, where 2.8.0 gave middle ``Sánchez-Cid``, family ``Torres``; a hyphenated given name is taken too unless every part of it is a listed given name, as under ``locales.ES`` (``Ana María-José López`` keeps middle ``María-José``), and hyphenated initials stay. :issue:`959` - **Fix particles joined by a connective reading the connective as the family's base.** ``parse("Fritz Freiherr und von Bar")`` gives family_base ``Freiherr und von Bar`` and initials ``F. F. v. B.``, where 2.0.0 through 2.8.0 gave family_base ``und``; a connective ending the family is still its base (``Juan de y`` gives ``y``), and one joining particles before a name word stays in it (``Alois von und zu Liechtenstein`` gives ``und Liechtenstein``). :issue:`939` - **Render a name in the order it was read in, keeping its written separators.** ``str(parse("김민준"))`` gives ``'김민준'``, where 2.1.0 through 2.8.0 gave ``'민준 김'``; a name read family-first renders family-first, and a comma before a suffix stays (``John Smith, Jr.``, ``Smith, Jr.``), while a family comma between name words never does (``Smith, John`` gives ``'John Smith'``). A spec you pass to ``render()`` renders as written, so 2.x's default spec keeps given-first output, and a name pickled by 2.x renders as it did. ``initials()`` gives nothing for a word in Han, Hangul or kana. The new ``ParsedName.order`` records the order; ``render()``'s spec now defaults to ``None``. See :doc:`migrate`. :issue:`754` - **Fix a credential written in Title case before a name word taking the name with it.** ``parse("Badrul Akmal Hisham Md Yusoff")`` gives middle ``Akmal Hisham Md``, family ``Yusoff``, as 1.4.0 through 2.3.0 did, where 2.4.0 through 2.8.0 gave family ``Hisham``, suffix ``Md Yusoff``; ``MD``, ``M.D.`` and the part after a family comma (``Smith, John Md Jones``) still start the credential's run. :issue:`1019` diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 92d68ef79..677738835 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -40,7 +40,7 @@ import bisect import re -from nameparser._lexicon import _normalize, _run_addresses_by_given +from nameparser._lexicon import Lexicon, _normalize, _run_addresses_by_given from nameparser._pipeline._assign import ( _name_alt, _name_pieces, _name_positions, ) @@ -54,7 +54,7 @@ WorkToken, _NEVER_FLIPPED, comma_bucket, copy_with, ) from nameparser._pipeline._vocab import ( - delimiter_cores, has_patronymic_ending, is_listed_given_name, + delimiter_cores, has_patronymic_ending, is_initial, is_listed_given_name, is_patronymic_marker, suffix_reading, unit_ends, ) from nameparser._policy import PatronymicRule @@ -250,21 +250,43 @@ def _fold_reach(tokens: list[WorkToken], name_idx: list[int]) -> int: _HYPHENS = re.compile("[-\u2010\u2011\u2012\u2013\u2212\ufe63\uff0d]") -def _family_word_count(tokens: list[WorkToken], family: list[int]) -> int: +def _family_word_count(tokens: list[WorkToken], family: list[int], + hyphen_joins: bool = False) -> int: """The family's name words as rules/O6 counts them: a particle run counts with the word it attaches to, a connective counts as nothing and a hyphen joins words without making them one, so "de la Vega" is one word and "Ortega y Gasset" and "García-López" two. A family of nothing but particles counts each of them, R2 reading - them as ordinary name words.""" - words = sum(len([part for part in _HYPHENS.split(tokens[i].text) - if part]) + them as ordinary name words. `hyphen_joins` counts a hyphenated word + as one, as the word O6 takes is counted (#959).""" + words = sum(1 if hyphen_joins + else len([part for part in _HYPHENS.split(tokens[i].text) + if part]) for i in family if "particle" not in tokens[i].tags and "conjunction" not in tokens[i].tags) return words or len(family) +def _hyphenated_initials(text: str) -> bool: + """A hyphenated word every part of which is an initial ("M.-J."), + which writes given names as an initial written alone does (#959).""" + parts = [part for part in _HYPHENS.split(text) if part] + return len(parts) > 1 and all(is_initial(part) for part in parts) + + +def _listed_given(text: str, lexicon: Lexicon) -> bool: + """rules/O6's veto: `text` is a listed given name, or a hyphenated + word every part of which is one ("María-José" under the Spanish + list), which the taken word's count would otherwise let through + (#959).""" + if is_listed_given_name(text, lexicon): + return True + parts = [part for part in _HYPHENS.split(text) if part] + return len(parts) > 1 and all(is_listed_given_name(part, lexicon) + for part in parts) + + def _take_second_family_name(state: ParseState, tokens: list[WorkToken]) -> None: # rules/O6: "the family name takes the name word beside it, so @@ -288,14 +310,16 @@ def _take_second_family_name(state: ParseState, if len(units) < 2: return take = units[-1] if state.order[0] is Role.GIVEN else units[0] - # What is taken must be ONE name word, counted as the family is, so - # a hyphenated given name or a bound given-name pair (P5) stays put; - # and a word a family name could be: not an initial, which writes a - # given name (one written alone; "J.R." is taken), nor a lone dash - # or other mark with no letter in it. - if _family_word_count(tokens, take) != 1: + # What is taken must be ONE name word, so a bound given-name pair + # (P5) stays put; rules/O6: "A hyphenated word counts as one when it + # is the word taken" (decisions/O6, #959), where the family's own + # hyphen counts two; and a word a family name could be: not an + # initial, which writes a given name (one written alone; "J.R." is + # taken), nor a lone dash or other mark with no letter in it. + if _family_word_count(tokens, take, hyphen_joins=True) != 1: return - if len(take) == 1 and "initial" in tokens[take[0]].tags: + if len(take) == 1 and ("initial" in tokens[take[0]].tags + or _hyphenated_initials(tokens[take[0]].text)): return if not any(ch.isalpha() for i in take for ch in tokens[i].text): return @@ -305,8 +329,8 @@ def _take_second_family_name(state: ParseState, # either order. Only a unit of one token is asked: a word # behind a particle ("de Miguel") is a unit of two, a surname taken # as before. History: decisions/O6 (#959). - if len(take) == 1 and is_listed_given_name(tokens[take[0]].text, - state.lexicon): + if len(take) == 1 and _listed_given(tokens[take[0]].text, + state.lexicon): return if state.order[0] is Role.GIVEN: # given, middles..., family: the middle unit standing right diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 1d78389b3..1fec228ca 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -563,13 +563,27 @@ def _check_cjk_shape_purity(self) -> None: {"given": "Ana", "middle": "M.", "family": "García"}, policy=_TWO, classification="parity", notes="#754 review: an initial writes a given name"), - Case("two_family_names_leaves_a_hyphenated_given_name_before_the_family", + Case("two_family_names_takes_a_hyphenated_word_as_one", + "Andrés Sánchez-Cid Torres", + {"given": "Andrés", "family": "Sánchez-Cid Torres"}, + policy=_TWO, classification="feat(#959)", + notes="a hyphenated word counts as one when it is the word " + "taken; 2.8.0 kept 'Sánchez-Cid' a middle name, its two " + "words counted as the family's are"), + Case("two_family_names_takes_a_hyphenated_given_name_without_a_list", "Ana María-José López", - {"given": "Ana", "middle": "María-José", "family": "López"}, + {"given": "Ana", "family": "María-José López"}, + policy=_TWO, classification="feat(#959)", + notes="the switch's cost (O6's Accepted): with no given-name " + "list nothing says María-José is a given name; the " + "#754 review's reading, middle 'María-José', is what " + "the Spanish list's veto gives"), + Case("two_family_names_leaves_hyphenated_initials_before_the_family", + "Ana M.-J. López", + {"given": "Ana", "middle": "M.-J.", "family": "López"}, policy=_TWO, classification="parity", - notes="#754 review: the hyphen's two words count against the " - "taken word as they count in the family; the first draft " - "read family 'María-José López'"), + notes="hyphenated initials write given names, as an initial " + "written alone does (#959)"), Case("two_family_names_leaves_a_bound_given_pair_before_the_family", "Ana abdul Rahman López", {"given": "Ana", "middle": "abdul Rahman", "family": "López"}, diff --git a/tests/v2/pipeline/test_post_rules.py b/tests/v2/pipeline/test_post_rules.py index cedfa4120..77e27ada0 100644 --- a/tests/v2/pipeline/test_post_rules.py +++ b/tests/v2/pipeline/test_post_rules.py @@ -1360,8 +1360,12 @@ def _roles(text: str, policy: Policy, lexicon: Lexicon) -> tuple[str, ...]: ("José", "Luis Pérez", ""), ("José", "Luis Pérez", "")), ("Ana Martín López", GIVEN_FIRST, ("Ana", "", "Martín López"), ("Ana", "", "Martín López")), + # a hyphenated word taken counts as one, and the veto reads each + # of its parts (#959): every part listed, it stays ("Ana María-Luis López", GIVEN_FIRST, - ("Ana", "María-Luis", "López"), ("Ana", "María-Luis", "López")), + ("Ana", "", "María-Luis López"), ("Ana", "María-Luis", "López")), + ("Ana María-Torres López", GIVEN_FIRST, + ("Ana", "", "María-Torres López"), ("Ana", "", "María-Torres López")), ("José Luis Pérez-García", GIVEN_FIRST, ("José", "Luis", "Pérez-García"), ("José", "Luis", "Pérez-García")), # a listed word behind a particle is a surname unit, taken as @@ -1387,7 +1391,7 @@ def test_the_veto_counts_its_rows() -> None: moved = [r for r in _VETO_EFFECT if r[2] != r[3]] assert {r[1] for r in moved} == {GIVEN_FIRST, FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST} - assert len(moved) == 11 and len(_VETO_EFFECT) - len(moved) == 7 + assert len(moved) == 12 and len(_VETO_EFFECT) - len(moved) == 7 def test_the_veto_leaves_the_word_to_two_given_names() -> None: diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index 6526414cb..2201e7f9a 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -39,8 +39,8 @@ "O3": {"words": 47, "then_words": 18, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "O4": {"words": 91, "then_words": 57, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "O5": {"words": 478, "then_words": 160, "because_sentences": 2, "except": 6, "precedence": 1, "accepted": 0, "accepted_words": 0}, - "O6": {"words": 652, "then_words": 161, "because_sentences": 2, "except": 5, "precedence": 4, "accepted": 8, "accepted_words": 297}, - "O7": {"words": 598, "then_words": 175, "because_sentences": 2, "except": 4, "precedence": 3, "accepted": 6, "accepted_words": 220}, + "O6": {"words": 682, "then_words": 183, "because_sentences": 2, "except": 5, "precedence": 4, "accepted": 8, "accepted_words": 297}, + "O7": {"words": 633, "then_words": 175, "because_sentences": 2, "except": 4, "precedence": 3, "accepted": 6, "accepted_words": 220}, "P1": {"words": 415, "then_words": 287, "because_sentences": 2, "except": 0, "precedence": 1, "accepted": 4, "accepted_words": 319}, "P2": {"words": 315, "then_words": 279, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 3, "accepted_words": 167}, "P3": {"words": 639, "then_words": 598, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, diff --git a/tests/v2/test_types.py b/tests/v2/test_types.py index 1a8b67063..0fad3665a 100644 --- a/tests/v2/test_types.py +++ b/tests/v2/test_types.py @@ -1347,7 +1347,7 @@ def test_a_patronymic_parsedname_pickled_by_2_8_0_loads_equal() -> None: ("Ana María de la Vega Rodríguez", _GF, False, ""), # O6's other declines ("Ana García", _GF, False, ""), - ("Ana María-José López", _GF, False, ""), + ("Ana M.-J. López", _GF, False, ""), ("Ana abdul Rahman López", _GF, False, ""), ("Ana M. García", _GF, False, ""), ("Juan de la Vega Rodríguez", _FF, False, ""), diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 8a41c2f08..2378165a9 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -62,6 +62,7 @@ "Ana J.R. García" "Ana Jorge Flores" "Ana M. García" +"Ana M.-J. López" "Ana Martín López" "Ana María 3 López" "Ana María García - López" @@ -84,6 +85,7 @@ "Andrew Perkins (MBA)" "Andrew Perkins (PhD)" "Andrew Perkins «MBA»" +"Andrés Sánchez-Cid Torres" "Anh Do" "Anh Van Do" "Anh van Do" @@ -1078,6 +1080,7 @@ "The Mahdi Muhammad Ahmad" "The Rev. Deacon John Smith" "The Right Hon. the President of the Queen's Bench Division" +"Torres Sánchez-Cid Andrés" "VI, AD" "Van Buren, Ed" "Van Johnson" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index c5b250c5b..da7aba433 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -211,6 +211,7 @@ {"name": "Ana J.R. García", "given": "Ana", "middle": "J.R.", "family": "García", "initials": "A. J. G."} {"name": "Ana Jorge Flores", "given": "Ana", "middle": "Jorge", "family": "Flores", "initials": "A. J. F."} {"name": "Ana M. García", "given": "Ana", "middle": "M.", "family": "García", "initials": "A. M. G."} +{"name": "Ana M.-J. López", "given": "Ana", "middle": "M.-J.", "family": "López", "initials": "A. M. L."} {"name": "Ana Marija Vera Kovač", "given": "Ana", "middle": "Marija Vera", "family": "Kovač", "initials": "A. M. V. K."} {"name": "Ana Martín López", "given": "Ana", "middle": "Martín", "family": "López", "initials": "A. M. L."} {"name": "Ana María 3 López", "given": "Ana", "middle": "María 3", "family": "López", "initials": "A. M. 3. L."} @@ -255,6 +256,7 @@ {"name": "Andrew Perkins, Jr., Col. (Ret)", "title": "Col.", "given": "Andrew", "family": "Perkins", "suffix": "Jr., Ret", "initials": "A. P.", "generation": "Jr.", "credentials": "Ret"} {"name": "Andrews, M.D.", "family": "Andrews", "suffix": "M.D.", "initials": "A."} {"name": "Andrés David Padilla Mejía", "given": "Andrés", "middle": "David Padilla", "family": "Mejía", "initials": "A. D. P. M."} +{"name": "Andrés Sánchez-Cid Torres", "given": "Andrés", "middle": "Sánchez-Cid", "family": "Torres", "initials": "A. S. T."} {"name": "Ane Miren Urrestaratzu", "given": "Ane", "middle": "Miren", "family": "Urrestaratzu", "initials": "A. M. U."} {"name": "Angiviller, Charles Claude de La Billarderie d'", "given": "Charles", "middle": "Claude de La Billarderie", "family": "d' Angiviller", "initials": "C. C. B. A.", "capitalized_force": "Charles Claude de la Billarderie d' Angiviller"} {"name": "Angoulême, Louis Antoine d'Artois d'", "given": "Louis", "middle": "Antoine d'Artois", "family": "d' Angoulême", "initials": "L. A. d. A.", "capitalized_force": "Louis Antoine D'Artois d' Angoulême"} @@ -3697,6 +3699,7 @@ {"name": "Tomás Iglesias Pérez", "given": "Tomás", "middle": "Iglesias", "family": "Pérez", "initials": "T. I. P."} {"name": "Tony Miranda", "given": "Tony", "family": "Miranda", "initials": "T. M."} {"name": "Torres Ana", "order": "FAMILY_FIRST", "given": "Ana", "family": "Torres", "initials": "A. T."} +{"name": "Torres Sánchez-Cid Andrés", "given": "Torres", "middle": "Sánchez-Cid", "family": "Andrés", "initials": "T. S. A."} {"name": "Torres, Ana", "given": "Ana", "family": "Torres", "initials": "A. T."} {"name": "Tounens, Antoine de", "given": "Antoine", "family": "de Tounens", "initials": "A. T."} {"name": "Traberth, Julie", "given": "Julie", "family": "Traberth", "initials": "J. T."} From c80f23ff72aecabf8ddb2a5e608e14d979a86f64 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 3 Nov 2026 18:49:07 -0800 Subject: [PATCH 2/2] Review fixes for the hyphenated surname taken (#959) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - The veto reads each part of a hyphenated word: every part a listed given name or an initial keeps it (`M.-José` under the Spanish list, `M.-J.`); measured neutral on ORCID. - A hyphenated bound pair (`Ana Abdul-Rahman López`) is taken like any hyphenated word, recorded under rules/O6's Accepted: a veto for it cost 2 ORCID surnames (`Abdul-Hadi Pru`) and kept none. - Policy.two_family_names' docstring, the release range (2.6.0 through 2.8.0), decisions/O6's 10,038 figure and its recipe's dedup step. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/O6-two-family-names.md | 4 +-- docs/design/rules/O6-two-family-names.md | 4 ++- docs/release_log.rst | 2 +- nameparser/_pipeline/_post_rules.py | 32 ++++++++------------ nameparser/_policy.py | 9 ++++-- tests/v2/rules_budget.json | 2 +- tools/differential/corpus_rules.jsonl | 2 ++ tools/differential/snapshot.jsonl | 2 ++ 8 files changed, 29 insertions(+), 28 deletions(-) diff --git a/docs/design/decisions/O6-two-family-names.md b/docs/design/decisions/O6-two-family-names.md index ac13efdbb..c2f821714 100644 --- a/docs/design/decisions/O6-two-family-names.md +++ b/docs/design/decisions/O6-two-family-names.md @@ -6,7 +6,7 @@ - `locales.ES` turns the switch on, carrying the Spanish given-name list beside it, one pack for Spain and Latin America alike, Argentina included; `locales.PT` leaves it off until a Portuguese given-name list exists, decided in chat — 2026-10-30 #753, phase 3g. - `locales.CS` and `locales.SK` turn the switch on with no given-name list, a net gain in every real population measured (CLDR's fictional renderings lose), two given names the recorded cost; `locales.SL` and `locales.HR` leave it off, decided in chat — 2026-10-30 #753, phase 3f amended. - Both family names stay in `family` through 2.x; a second-surname field is the RFC's Phase 4 output model (docs/design/vocabulary-model-rfc.md, "The rules a pack can choose"), and a pack may later switch this rule on — 2026-10-24 #754; landed as the `surname2` view, `family` keeping both, 2026-10-29 #754, sub-phase 4d. -- The family's words are counted as P1's fold counts them, a connective join and a hyphen counting as their words, so a family already holding two keeps what it has; the word taken must count as one, a hyphenated word counting as one there, and be no initial written alone nor hyphenated initials — 2026-10-24 #754; 2026-11-03 #959 (Derek). +- The family's words are counted as P1's fold counts them, a connective join and a hyphen counting as their words, so a family already holding two keeps what it has; the word taken must count as one, a hyphenated word counting as one there unless it writes given names (each part a listed given name or an initial), and be no initial written alone — 2026-10-24 #754; 2026-11-03 #959 (Derek). - A name its script (W4) reads in an order other than the declared `name_order` is left alone; where the two agree it is read like any other — 2026-10-24 #754. - Under a family-first order the first two name words are the family, the remainder laid out as the positions after a family — 2026-10-24 #754. - The switch runs before `middle_as_family` and is not refused beside it: given-first, the two together read as `middle_as_family` alone — 2026-10-24 #754. @@ -163,5 +163,5 @@ Open: why the Spanish list does less in Latin America, and whether a list per re | ORCID, country SK | 12,962 | 94.84% → 96.40% | 347 | 144 | Every broken record, in all three, has two or more words in the given box, read as a given name and a family name: no population-wide Czech or Slovak given-name list is published to veto the word, the register's frequencies having been withdrawn after 2016's; ČSÚ's top 100 newborn given names a year (CC BY 4.0) is a list, and the open lead. The circular ceiling such a list could reach, and the sources looked for, are in the vocabulary-model entry. -- 2026-11-03 #959 (Derek) — A HYPHENATED WORD COUNTS AS ONE WHEN IT IS THE WORD TAKEN, UNLESS EVERY PART OF IT IS A LISTED GIVEN NAME OR AN INITIAL. SUPERSEDES, in part, the 2026-10-24 #754 entry's count of the word taken, in which a hyphen joined two words without making them one, so `Andrés Sánchez-Cid Torres` kept `Sánchez-Cid` a middle name (the #754 review's reading of `Ana María-José López`, middle 'María-José', which the first draft had read as family 'María-José López'). The issue found this shape behind 271 of the 466 Spanish records the given-name veto still read wrong. Measured 2026-11-03 on the cached `orcid-researchers` corpus, ES, MX, CO, PE, CL, VE and AR at most 20,000 records a country with both boxes (135,735 records), over the 10,794 distinct records holding a hyphen in either box, rendered given first and read right where both the family and the given names match the boxes (`local_agreement.fold`), under `parser_for(locales.ES)`, on 39839b91 against prototypes of three readings: a hyphen counting as one everywhere O6 counts (family and word taken) reads 10,040 right against 10,281, fixing 277 and breaking 520, most of them a family box of two surnames written with a hyphen (`Garcia-Gomez`), which then counts as one and takes a given name; a hyphen counting as one in the word taken alone reads 10,464, fixing 186 (ES 125, MX 24, CL 20, PE 6, CO 5, VE 5, AR 1) and breaking 3 (`Aristide Tanyi-Jong Akem`, `Hilda Ines-Romero Menacho`, `Benjamin Moraga-Arias .`); and the same with a veto on a hyphenated word every part of which is a listed given name, which moves none of these records. Decided: the word-taken count with the veto, the veto keeping the rule's own `Ana María-José López` where the Spanish list is loaded (middle 'María-José'), while without a list the switch takes it (family 'María-José López'), its usual cost (rules/O6's Accepted). Hyphenated initials (`Ana M.-J. López`) stay as an initial written alone stays, found building it. A hyphen in the family still counts its words, so `Javier Olmedo Pérez-Montaut` keeps `Olmedo` a middle name: reading `Pérez-Montaut` as one surname is the 520 above. The built change reads the same 10,464 and moves the same 186 and 3, and with the switch alone (`Policy(two_family_names=True)`, no list) 10,292 to 10,475, the same records. Recompute: for each record of `local_agreement.records(local_corpora.load("orcid-researchers"))` in those countries, both boxes non-empty, at most 20,000 a country, keep those with a hyphen (`_post_rules._HYPHENS`) in either box, parse `f"{given} {family}"` with each tree's parser, `nameparser.__file__` asserted, and compare. +- 2026-11-03 #959 (Derek) — A HYPHENATED WORD COUNTS AS ONE WHEN IT IS THE WORD TAKEN, UNLESS EVERY PART OF IT IS A LISTED GIVEN NAME OR AN INITIAL. SUPERSEDES, in part, the 2026-10-24 #754 entry's count of the word taken, in which a hyphen joined two words without making them one, so `Andrés Sánchez-Cid Torres` kept `Sánchez-Cid` a middle name (the #754 review's reading of `Ana María-José López`, middle 'María-José', which the first draft had read as family 'María-José López'). The issue found this shape behind 271 of the 466 Spanish records the given-name veto still read wrong. Measured 2026-11-03 on the cached `orcid-researchers` corpus, ES, MX, CO, PE, CL, VE and AR at most 20,000 records a country with both boxes (135,735 records), over the 10,794 distinct records holding a hyphen in either box, rendered given first and read right where both the family and the given names match the boxes (`local_agreement.fold`), under `parser_for(locales.ES)`, on 39839b91 against prototypes of three readings: a hyphen counting as one everywhere O6 counts (family and word taken) reads 10,038 right against 10,281, fixing 277 and breaking 520, most of them a family box of two surnames written with a hyphen (`Garcia-Gomez`), which then counts as one and takes a given name; a hyphen counting as one in the word taken alone reads 10,464, fixing 186 (ES 125, MX 24, CL 20, PE 6, CO 5, VE 5, AR 1) and breaking 3 (`Aristide Tanyi-Jong Akem`, `Hilda Ines-Romero Menacho`, `Benjamin Moraga-Arias .`); and the same with a veto on a hyphenated word every part of which is a listed given name, which moves none of these records. Decided: the word-taken count with the veto, the veto keeping the rule's own `Ana María-José López` where the Spanish list is loaded (middle 'María-José'), while without a list the switch takes it (family 'María-José López'), its usual cost (rules/O6's Accepted). Found building it and in its review, and folded into the veto, which reads each part: a part that is an initial counts as a listed given name does, so hyphenated initials (`Ana M.-J. López`) stay as an initial written alone stays, and `M.-José` stays under the list, which moves no ORCID record. A hyphenated bound given-name pair (`Ana Abdul-Rahman López`, middle 'Abdul-Rahman' in 2.8.0) is taken like any hyphenated word, family 'Abdul-Rahman López', where the pair written as two words (P5) stays: a veto on a hyphenated word whose first part is a bound given name was built and backed out the same day, since in this population it cost 2 records whose hyphenated word is a surname (`Paola Valentina Abdul-Hadi Pru`, `Giorgia Anette Abd-El-Kader Espinoza`) and kept none, the hyphen being how those surnames are written; rules/O6's Accepted records the cost. A hyphen in the family still counts its words, so `Javier Olmedo Pérez-Montaut` keeps `Olmedo` a middle name: reading `Pérez-Montaut` as one surname is the 520 above. The built change reads the same 10,464 and moves the same 186 and 3, and with the switch alone (`Policy(two_family_names=True)`, no list) 10,292 to 10,475, the same records. Recompute: for each record of `local_agreement.records(local_corpora.load("orcid-researchers"))` in those countries, both boxes non-empty, at most 20,000 a country, keep those with a hyphen (`_post_rules._HYPHENS`) in either box, distinct by given box, family box and country (two records are duplicates), parse `f"{given} {family}"` with each tree's parser, `nameparser.__file__` asserted, and compare. - 2026-11-03 #959 — BOTH SWITCHES OVER A GIVEN NAME A PARTICLE COMPLETES GET AN EXAMPLE. The 2026-10-30 entry's Open named the shape no rule example showed: `María del Carmen de la Vega` reads family 'del Carmen de la Vega' with both switches on, O6 taking the chain before O7 can complete the given name with it, and given 'María del Carmen' under O7 alone. rules/O7's precedence bullet against two-family-name handling now says so and carries both rows; no behavior moves. diff --git a/docs/design/rules/O6-two-family-names.md b/docs/design/rules/O6-two-family-names.md index fbd6eb6fa..3d16bd26e 100644 --- a/docs/design/rules/O6-two-family-names.md +++ b/docs/design/rules/O6-two-family-names.md @@ -61,7 +61,7 @@ ``` ", Ana García López" two_family_names → family="" ``` -- **What is taken must be one name word a family name could be**: a bound given-name pair (P5), counting as two words, stays where it is, and so do an initial written alone or hyphenated initials, which write given names, and a mark or number with no letter in it. A particle chain (P2) is never split, so where taking it would leave no given name the rule declines. A listed given name stays where it is too, under either order, and so does a hyphenated word every part of which is one, and a list written without accents matches the word written with them: the caller's vocabulary names it a word only a given name is, so the name holds one family name. A listed word behind a particle is a surname unit, taken as before. +- **What is taken must be one name word a family name could be**: a bound given-name pair (P5) written as two words stays where it is, and so do an initial written alone, which writes a given name, and a mark or number with no letter in it. A particle chain (P2) is never split, so where taking it would leave no given name the rule declines. A listed given name stays where it is too, under either order, and so does a hyphenated word every part of which is one or an initial, and a list written without accents matches the word written with them: the caller's vocabulary names it a word only a given name is, so the name holds one family name. A listed word behind a particle is a surname unit, taken as before. ``` "José Luis Pérez" two_family_names+es_given_names → family="Pérez" "Jose Luis Perez" two_family_names+es_given_names → middle="Luis" @@ -69,6 +69,7 @@ "Pérez Luis José" family-first-given-last+two_family_names+es_given_names → family="Pérez" "Ana María-José López" two_family_names+es_given_names → family="López" "Ana M.-J. López" two_family_names → family="López" + "Ana M.-José López" two_family_names+es_given_names → family="López" "Ana abdul Rahman López" two_family_names → family="López" "Ana M. García" two_family_names → family="García" "Ana María García - López" two_family_names → family="López" @@ -103,6 +104,7 @@ ``` "Juan Carlos Pérez" two_family_names → family="Carlos Pérez" "Ana María-José López" two_family_names → family="María-José López" + "Ana Abdul-Rahman López" two_family_names → family="Abdul-Rahman López" ``` - A patronymic is a name word like any other here, a patronymic rule on or off, so where no rotation fired it can join the family. ``` diff --git a/docs/release_log.rst b/docs/release_log.rst index 221a85cf9..c7a28184d 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -36,7 +36,7 @@ Release Log **Behavior Changes** - - **Fix two_family_names leaving a hyphenated surname out of the family.** ``Parser(policy=Policy(two_family_names=True)).parse("Andrés Sánchez-Cid Torres")`` gives family ``Sánchez-Cid Torres``, where 2.8.0 gave middle ``Sánchez-Cid``, family ``Torres``; a hyphenated given name is taken too unless every part of it is a listed given name, as under ``locales.ES`` (``Ana María-José López`` keeps middle ``María-José``), and hyphenated initials stay. :issue:`959` + - **Fix two_family_names leaving a hyphenated surname out of the family.** ``Parser(policy=Policy(two_family_names=True)).parse("Andrés Sánchez-Cid Torres")`` gives family ``Sánchez-Cid Torres``, where 2.6.0 through 2.8.0 gave middle ``Sánchez-Cid``, family ``Torres``; a hyphenated given name is taken too unless every part of it is a listed given name, as under ``locales.ES`` (``Ana María-José López`` keeps middle ``María-José``), and hyphenated initials stay. :issue:`959` - **Fix particles joined by a connective reading the connective as the family's base.** ``parse("Fritz Freiherr und von Bar")`` gives family_base ``Freiherr und von Bar`` and initials ``F. F. v. B.``, where 2.0.0 through 2.8.0 gave family_base ``und``; a connective ending the family is still its base (``Juan de y`` gives ``y``), and one joining particles before a name word stays in it (``Alois von und zu Liechtenstein`` gives ``und Liechtenstein``). :issue:`939` - **Render a name in the order it was read in, keeping its written separators.** ``str(parse("김민준"))`` gives ``'김민준'``, where 2.1.0 through 2.8.0 gave ``'민준 김'``; a name read family-first renders family-first, and a comma before a suffix stays (``John Smith, Jr.``, ``Smith, Jr.``), while a family comma between name words never does (``Smith, John`` gives ``'John Smith'``). A spec you pass to ``render()`` renders as written, so 2.x's default spec keeps given-first output, and a name pickled by 2.x renders as it did. ``initials()`` gives nothing for a word in Han, Hangul or kana. The new ``ParsedName.order`` records the order; ``render()``'s spec now defaults to ``None``. See :doc:`migrate`. :issue:`754` - **Fix a credential written in Title case before a name word taking the name with it.** ``parse("Badrul Akmal Hisham Md Yusoff")`` gives middle ``Akmal Hisham Md``, family ``Yusoff``, as 1.4.0 through 2.3.0 did, where 2.4.0 through 2.8.0 gave family ``Hisham``, suffix ``Md Yusoff``; ``MD``, ``M.D.`` and the part after a family comma (``Smith, John Md Jones``) still start the credential's run. :issue:`1019` diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 677738835..43d668a16 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -268,23 +268,14 @@ def _family_word_count(tokens: list[WorkToken], family: list[int], return words or len(family) -def _hyphenated_initials(text: str) -> bool: - """A hyphenated word every part of which is an initial ("M.-J."), - which writes given names as an initial written alone does (#959).""" +def _hyphenated_given(text: str, lexicon: Lexicon) -> bool: + """A hyphenated word that writes given names, which the taken word's + count (a hyphenated word counting as one) would otherwise let through + (#959): every part of it a listed given name or an initial + ("María-José" under the Spanish list, "M.-J.", "M.-José").""" parts = [part for part in _HYPHENS.split(text) if part] - return len(parts) > 1 and all(is_initial(part) for part in parts) - - -def _listed_given(text: str, lexicon: Lexicon) -> bool: - """rules/O6's veto: `text` is a listed given name, or a hyphenated - word every part of which is one ("María-José" under the Spanish - list), which the taken word's count would otherwise let through - (#959).""" - if is_listed_given_name(text, lexicon): - return True - parts = [part for part in _HYPHENS.split(text) if part] - return len(parts) > 1 and all(is_listed_given_name(part, lexicon) - for part in parts) + return len(parts) > 1 and all(is_initial(part) or is_listed_given_name(part, lexicon) + for part in parts) def _take_second_family_name(state: ParseState, @@ -318,8 +309,7 @@ def _take_second_family_name(state: ParseState, # taken), nor a lone dash or other mark with no letter in it. if _family_word_count(tokens, take, hyphen_joins=True) != 1: return - if len(take) == 1 and ("initial" in tokens[take[0]].tags - or _hyphenated_initials(tokens[take[0]].text)): + if len(take) == 1 and "initial" in tokens[take[0]].tags: return if not any(ch.isalpha() for i in take for ch in tokens[i].text): return @@ -329,8 +319,10 @@ def _take_second_family_name(state: ParseState, # either order. Only a unit of one token is asked: a word # behind a particle ("de Miguel") is a unit of two, a surname taken # as before. History: decisions/O6 (#959). - if len(take) == 1 and _listed_given(tokens[take[0]].text, - state.lexicon): + # A hyphenated word writing given names stays too (#959). + if len(take) == 1 and ( + is_listed_given_name(tokens[take[0]].text, state.lexicon) + or _hyphenated_given(tokens[take[0]].text, state.lexicon)): return if state.order[0] is Role.GIVEN: # given, middles..., family: the middle unit standing right diff --git a/nameparser/_policy.py b/nameparser/_policy.py index c65b24af5..98b54d2b0 100644 --- a/nameparser/_policy.py +++ b/nameparser/_policy.py @@ -699,9 +699,12 @@ class Policy: #: García-López", "Ana María de la Vega Rodríguez"), titles and #: suffixes count as no name word, and a name of two name words is #: unchanged. The word taken must be one name word a family name - #: could be, so an initial written alone ("Ana M. García"), a - #: hyphenated given name ("Ana María-José López") and a bound - #: given-name pair stay where they are. Under a family-first + #: could be, so an initial written alone ("Ana M. García"), hyphenated + #: initials ("Ana M.-J. López") and a bound given-name pair stay where + #: they are; a hyphenated word counts as one when + #: taken ("Andrés Sánchez-Cid Torres" gives family ``Sánchez-Cid + #: Torres``), so a hyphenated given name is taken unless every part + #: of it is a listed given name or an initial (``Lexicon.given_names``). Under a family-first #: name_order the FIRST two name words are the family; a name its #: script reads in an order other than name_order (a spaced Han or #: Hangul name under the default) is left alone, though under a diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index 2201e7f9a..9836a6ffc 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -39,7 +39,7 @@ "O3": {"words": 47, "then_words": 18, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "O4": {"words": 91, "then_words": 57, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "O5": {"words": 478, "then_words": 160, "because_sentences": 2, "except": 6, "precedence": 1, "accepted": 0, "accepted_words": 0}, - "O6": {"words": 682, "then_words": 183, "because_sentences": 2, "except": 5, "precedence": 4, "accepted": 8, "accepted_words": 297}, + "O6": {"words": 683, "then_words": 183, "because_sentences": 2, "except": 5, "precedence": 4, "accepted": 8, "accepted_words": 297}, "O7": {"words": 633, "then_words": 175, "because_sentences": 2, "except": 4, "precedence": 3, "accepted": 6, "accepted_words": 220}, "P1": {"words": 415, "then_words": 287, "because_sentences": 2, "except": 0, "precedence": 1, "accepted": 4, "accepted_words": 319}, "P2": {"words": 315, "then_words": 279, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 3, "accepted_words": 167}, diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 2378165a9..5ce19194f 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -56,6 +56,7 @@ "Aliyev, oglu" "Alves, Pedro Fils" "Alves, Pedro Neto" +"Ana Abdul-Rahman López" "Ana Fo" "Ana García" "Ana García (Anita) López" @@ -63,6 +64,7 @@ "Ana Jorge Flores" "Ana M. García" "Ana M.-J. López" +"Ana M.-José López" "Ana Martín López" "Ana María 3 López" "Ana María García - López" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index da7aba433..3ade00e78 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -205,6 +205,7 @@ {"name": "Amélie Beaury-Saurel", "given": "Amélie", "family": "Beaury-Saurel", "initials": "A. B."} {"name": "Amīn as-Sulṭān ʿAlī Aṣġar Ḫān", "given": "Amīn", "middle": "as-Sulṭān ʿAlī Aṣġar", "family": "Ḫān", "initials": "A. a. ʿ. A. Ḫ.", "capitalized_force": "Amīn As-Sulṭān ʿAlī Aṣġar Ḫān"} {"name": "An tOll. An Dr. Brigid Aoibhe Sophia de Bhulbh", "given": "An", "middle": "tOll. An Dr. Brigid Aoibhe Sophia", "family": "de Bhulbh", "initials": "A. t. A. D. B. A. S. B.", "capitalized_force": "An Toll. An Dr. Brigid Aoibhe Sophia de Bhulbh"} +{"name": "Ana Abdul-Rahman López", "given": "Ana", "middle": "Abdul-Rahman", "family": "López", "initials": "A. A. L."} {"name": "Ana Fo", "given": "Ana", "family": "Fo", "ambiguities": ["SUFFIX_OR_NAME"], "alternatives": ["SUFFIX_OR_NAME: agnomen as suffix (vocabulary)"], "initials": "A. F."} {"name": "Ana García", "given": "Ana", "family": "García", "initials": "A. G."} {"name": "Ana García (Anita) López", "given": "Ana", "middle": "García", "family": "López", "nickname": "Anita", "initials": "A. G. L."} @@ -212,6 +213,7 @@ {"name": "Ana Jorge Flores", "given": "Ana", "middle": "Jorge", "family": "Flores", "initials": "A. J. F."} {"name": "Ana M. García", "given": "Ana", "middle": "M.", "family": "García", "initials": "A. M. G."} {"name": "Ana M.-J. López", "given": "Ana", "middle": "M.-J.", "family": "López", "initials": "A. M. L."} +{"name": "Ana M.-José López", "given": "Ana", "middle": "M.-José", "family": "López", "initials": "A. M. L."} {"name": "Ana Marija Vera Kovač", "given": "Ana", "middle": "Marija Vera", "family": "Kovač", "initials": "A. M. V. K."} {"name": "Ana Martín López", "given": "Ana", "middle": "Martín", "family": "López", "initials": "A. M. L."} {"name": "Ana María 3 López", "given": "Ana", "middle": "María 3", "family": "López", "initials": "A. M. 3. L."}