Skip to content

Refuse five unattested readings: bon, dí, heer as particles; herren, heren as titles (#766) - #919

Merged
derek73 merged 2 commits into
masterfrom
fix/issue-766-unattested-vocabulary
Oct 27, 2026
Merged

derek73 merged 2 commits into
masterfrom
fix/issue-766-unattested-vocabulary

Conversation

@derek73

@derek73 derek73 commented Oct 27, 2026

Copy link
Copy Markdown
Owner

Closes #766.

Carries out #766's proposal: five vocabulary readings with no attested use in a person's name are replaced by exclusions quoting the evidence, and each word keeps or gains dated name readings.

word file reading refused name readings
bon und.toml particle given (SSA 274, Wikidata 9), family (Census 1,123, Wikidata 102) — unchanged
dí und.toml particle given (Wikidata 7) — unchanged
heer nl.toml particle (rank) given (SSA 272); new family (Census 2010 2,007, Wikidata 113, 2026-10-27)
herren de.toml salutation family (Census 4,466, Wikidata 37), re-counted 2026-10-27, unchanged
heren nl.toml salutation (with its after_given mark) family (Census 123, Wikidata 1), re-counted 2026-10-27, unchanged

Compiled sets: PARTICLES −3, TITLES −2, AFTER_GIVEN_TITLES −1 (heren). und.toml still holds both its words (now name readings and exclusions), so it is not empty and the build needed nothing new. The v1 shim needs nothing either: a word removed from the vocabulary simply leaves v1's default titles/prefixes, as sa did.

What moves

Grid over all three name orders and the comma shapes, against master (6ae585f):

  • Marie Bon Durand → middle Bon, family Durand (was family Bon Durand, 1.4.0 through 2.6.0); Li Dí Wang and Jan Heer Smit the same
  • Herren Hans → given Herren (was title Herren); Heren Jan the same
  • Doe, Jan Heren → middle Heren: back to 1.4.0 through 2.5.0's reading (2.6.0 read title Heren via rules/H8)
  • a leading Bon/Dí/Heer no longer reports particle-or-given
  • the Additional unit tests mostly for German & Dutch names + more degrees #18 test string Johan de heer Van Kampen reads middle de heer, family Van Kampen; no test or doc example held it

Each claim above was checked against the 1.4.0, 2.5.0 and 2.6.0 wheels.

Committed corpora: no moves (test_snapshot.py, CLDR, GND, Wikidata native and locale views all match their records), so nothing was re-recorded.

Local corpora (measure.py --base): 79 entries moved: ctgov-officials 14, fec-individuals 9, jrc-names 30, psc-individuals 20, tse-donors 6; nepal-candidates, tw-donors and ukec-individuals none. By shape:

  1. A real particle plus the word as a surname (de Heer, le Bon, van Bon, di/dal Bon): this used to be an all-particle family. Now the word is the base, so initials() stops counting the particle (K. d. H. → K. H.), and case repair writes the particle as it does for any particle surname. Turned up by the measurement, not predicted: it's an improvement.
  2. The word between given name and surname → a middle name. psc-individuals: 4 records now agree with the boxes their filers typed, and none newly disagree (the 1 "changed" record is a set-aside title box).
  3. A leading Bon/Heer given name (Korean and Filipino Bon) loses its particle-or-given report.
  4. Herren/Heren opening a name → given name instead of title (3 entries, one of them a Brazilian given name Heren).
  5. Smaller ones: a trailing or hyphenated Heer/Bon is no longer lowercased by forced case repair (Dhillon-Heer, Bon-Kwon).

Accepted cost (please confirm): a compound surname written with Bon as its own first word, e.g. the Italian X Bon Valsassina and the stage name Bon Jovi, now reads middle Bon. That is how any double surname reads by position; Policy(two_family_names=True) reads it right. jrc-names reads 5 more persons' spellings alike.

Docs, same diff

  • decisions/vocabulary-model: a Standing bullet, the und.toml Standing bullet revised, Should bon, dí, heer, herren and heren stay in the vocabulary? None is attested in a person's name #766 removed from Open:, and a dated Log entry with the measurement and its recompute recipe
  • decisions/H8: a Log line saying heren left the salutations
  • vocab/README.md: the und description, and the rank example (heer → freiherrin)
  • release_log 2.7.0 bullet; migrate.rst "Changed in" entry (the heading widened to "2.3 through 2.7", since it already listed a 2.7 shape)

I left decided off the exclusions: that's for you to add once you accept them.

Checks

Full suite (32,083 passed), -n0 -m serial (43 passed; an earlier run failed its wall-clock ratios under load 40, and the base checkout failed the same way under that load), sphinx-build -b doctest docs, python -m doctest README.rst, ruff, mypy, build.py --check.

🤖 Generated with Claude Code

derek73 and others added 2 commits October 27, 2026 14:36
…n and heren as titles (#766)

None had a recorded source or an attested use in a person's name, and each
is borne as a name. Each reading becomes an exclusion quoting the evidence;
the words keep or gain dated name readings (heer: Census 2010 surname 2,007,
Wikidata family 113). PARTICLES loses bon, dí and heer; TITLES loses herren
and heren, which also leaves AFTER_GIVEN_TITLES.

Marie Bon Durand reads middle 'Bon', family 'Durand' (1.4.0-2.6.0: family
'Bon Durand'); Herren Hans reads given 'Herren' (was title); Doe, Jan Heren
reads middle 'Heren' again, as 1.4.0-2.5.0 did. No committed corpus name
moves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73 derek73 added enhancement vocabulary Per-word vocabulary model and language packs labels Oct 27, 2026
@derek73 derek73 self-assigned this Oct 27, 2026
@derek73
derek73 merged commit 30843d0 into master Oct 27, 2026
12 checks passed
@derek73 derek73 added this to the 2.7 milestone Oct 27, 2026
@derek73
derek73 deleted the fix/issue-766-unattested-vocabulary branch October 28, 2026 02:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement vocabulary Per-word vocabulary model and language packs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Should bon, dí, heer, herren and heren stay in the vocabulary? None is attested in a person's name

1 participant