Skip to content

Add the Hindi and Bengali packs, locales.HI and locales.BN (3.0, 3e) - #958

Merged
derek73 merged 5 commits into
masterfrom
feat/v3-pack-3e-hi-bn
Oct 30, 2026
Merged

derek73 merged 5 commits into
masterfrom
feat/v3-pack-3e-hi-bn

Conversation

@derek73

@derek73 derek73 commented Oct 30, 2026 •

Copy link
Copy Markdown
Owner

Phase 3e of #753: two vocabulary packs for #345's romanized Indic titles. Each is nothing but its choice of language, with no DEVIATES (the 3d pattern).

What ships

  • locales.HI turns on smt, shrimati and shriyut (hi.toml, salutation readings default = false, after_given like their Devanagari spellings).
  • locales.BN turns on smt, shrimati (bn.toml, the same readings) and adhyapak (academic, default = false).
  • Example: parser_for(locales.HI).parse("Smt Nirmala Sitharaman") gives title Smt, given Nirmala. 1.4.0 through 2.8.0, and the default, give given Smt, middle Nirmala.
  • Where a turned-on word reads as a title: wherever the title vocabulary is read: the head, behind a listed title, after a family comma, ahead of one (rules/H6), ending the given part (rules/H8), dotted at the end (rules/H5), and inside a period-joined token.
  • No default parse moves. _compiled.py is current and the committed snapshot is unchanged.

Left out, measured

  • shri. Its exclusion stays, with the counts.
    • ORCID's whole extract has 198 given boxes opening with Shri, nearly all given names (Shri Kant, Shri Lakshmi; 52 hold Shri alone).
    • Turned on, it broke 20 filed names and fixed none: PSC 2, ORCID 5, PatentsView 4, FEC 9 (BANGALEY, SHRI lost its given). The only fix anywhere was jrc-names' Shri Dharmendra Pradhan.
    • The clash is in the pack's own language, so neither prefer nor strict settles it. That is the RFC's same-language route (a given reading plus the title phrase, as default work), left Open.
  • sushri. 14 of ORCID's 16 openers are Odia given names. It is left for strict, with no reading written: a plain hi choice would turn a default = false reading on for exactly the data that holds those names.
  • The 3d question (a given reading of श्री in hi.toml/bn.toml): answered no. A Hindi or Bengali choice wants the salutation.

Moves under the choices

Measured 2026-10-30 on b57468d plus this change, with --base a detached master checkout.

  • Committed corpora: none.
  • CLDR locale view: hi+hi_Latn 31/48 and bn 14/24 both ways. The native-script hi-in (60/60) and bn-in/bn-bd (40/40) are unchanged too. Records gain the rows.
  • nepal-candidates: 17 names under each choice (16 Shrimati, plus Smt.Nagiya, accepted).
  • Pairs: 125,833 → 125,839 alike; 11 now alike, 5 now apart. All 5 pair the word with श्रीमति (short i), which hi.toml does not list (Open).
  • Other local corpora: jrc-names, ctgov, psc, orcid, patentsview, fec and ukec show none under the final word set. That null is over populations holding no row with smt, shrimati, shriyut or adhyapak as a word; nepal-candidates' 17 rows are the only ones any source holds.
  • Accepted costs:
    • Shrimati Kushwaha and Shrimati Shetty read with no given name.
    • NPPES's 4 SMT credentials read as the title in any comma part holding them (Jane Doe, SMT, Jane Doe, PhD SMT).
  • shrimati ships, weighed: it leads as the title in all 16 nepal-candidates romanizations (by their Devanagari), against 1 ORCID person bearing it as a given name.
  • shriyut and adhyapak ship with no measured leading use: no bearer of either opens a name (adhyapak's 17 bearers are all family names, written last). Both are admitted on their native spellings' title readings.

The full record and recipe are in the 2026-10-30 phase 3e entry of docs/design/decisions/vocabulary-model.md.

Also

  • cldr_agreement.py maps hi, hi_Latn → HI and bn → BN, plus the native strata hi-in, bn-in and bn-bd.
  • Rotators cover every changed word in each shape.
  • Docs: docs/locales.rst (table, a "Romanized Indic titles" subsection) and the README/locales available() doctests.
  • An Unreleased release-log bullet.
  • A note in nameparser/vocab/AGENTS.md: count the boxes a title candidate opens, not only the boxes holding it.
  • The second commit is the fix round from a design-docs review: the slots the titles reach (H5/H6), adhyapak's cost, the Recompute recipe, the RFC status line, Shreemati in the Open.
  • The third commit restates the evidence after the two PR reviews:
    • the CLDR basis of the श्री answer (bn unmeasured);
    • the dotted forms' moves named by site;
    • the two no-given moves, the shrimati weighing and the population behind each null;
    • the NPPES SMT cost;
    • hi/bn added to the contributor checklist.
  • The fourth commit is the second-round fixes:
    • the Standing criterion restated for words with no measured leading use;
    • the dotted forms' sites taken from a comma-shape sweep (rules/C2 included);
    • SMT's cost covering every comma part that holds it;
    • both sides of Srimati's count.

Checks

  • uv run pytest, -n0 -m serial, mypy, ruff, sphinx-build -b doctest and python -m doctest README.rst: all pass.
  • The suite on 3.15 (a separate venv) and without namedivider: pass.
  • test_locales and test_cldr_agreement with namedivider: pass.

🤖 Generated with Claude Code

derek73 and others added 2 commits October 30, 2026 13:10
Two vocabulary packs, each nothing but its choice of language, for #345's
romanized Indic titles: hi.toml turns on smt, shrimati and shriyut, and
bn.toml smt, shrimati and adhyapak, each a `default = false` reading, so
`Smt Nirmala Sitharaman` reads title `Smt` under either pack. No default
parse moves.

shri and sushri stay out, measured: shri opens given names in every
population that writes it (198 of ORCID's given boxes; turned on it broke
20 filed names across PSC, ORCID, PatentsView and FEC and fixed none), a
clash in the pack's own language that neither prefer nor strict settles;
sushri is an Odia given name, left for strict. Its exclusion keeps shri's
counts. The CLDR and native-script locale views measure both packs
(unchanged agreement, recorded).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The packs' titles are read wherever the title vocabulary is, so a dotted
spelling moves ending the name (rules/H5) and a word before a family
comma moves (rules/H6), adhyapak's family-name cost included; rotators
cover both slots. The Recompute bullet names FEC's command and the tree
that was measured (the exclusion removed first, bn.toml too); shri's 198
openers include 52 lone ones; the RFC records 3e as landed; 3d's श्री
question is answered; Shreemati joins the Open; the pack docstrings no
longer name shri.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73 derek73 added enhancement vocabulary Per-word vocabulary model and language packs labels Oct 30, 2026
@derek73 derek73 self-assigned this Oct 30, 2026
derek73 and others added 3 commits October 30, 2026 13:37
The 3d question's answer names the CLDR renderings it rests on (hi, kok,
mr) and says bn's is unmeasured; the dotted spellings' moves are named by
site (H5, H6, H8, a period-joined chunk), H2 reading the rest already;
the two nepal-candidates moves that leave no given name or take a glued
token are stated; shrimati is weighed against its two name bearers; each
"none" says the population held no row with the words; NPPES's SMT
credential, read as the title after a comma under the pack, is an
accepted cost on smt's note; the contributor checklist lists hi and bn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…itles

From the second-round review: the Standing criterion names the case of a
word with no measured leading use (shriyut) or only family-name bearers
(adhyapak), each with its own bullet; the dotted spellings' sites come
from a sweep of comma shapes, the part past a second comma (rules/C2)
among them; SMT's cost covers every comma part holding it; shrimati's
name side counts one person; Srimati's Open gives both sides of its
count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the third-round review: the Standing criterion counts persons
pooled across the populations measured, and decides only the words a
pack's phase lists, so Shreemati and Srimati stay open; shrimati's
bullet cites it as stated, with ORCID's split alone; the dotted sites
name the empty head (rules/C15) and a bracketed trailing word, and a
rotator holds the empty head.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement vocabulary Per-word vocabulary model and language packs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant