Skip to content

Add locales.NE, the first language pack: a choice of ne (3.0, 3d) - #955

Merged
derek73 merged 3 commits into
masterfrom
feat/v3-pack-3d-ne
Oct 30, 2026
Merged

derek73 merged 3 commits into
masterfrom
feat/v3-pack-3d-ne

Conversation

@derek73

@derek73 derek73 commented Oct 30, 2026 •

Copy link
Copy Markdown
Owner

Sub-phase 3d of #753's work order: the first language pack, and the pattern 3e–3h copy where a pack is a choice alone.

What changes

  • nameparser/vocab/ne.toml gains given = { default = false, note = … } readings on श्री, guru and गुरु. They go in ne.toml because a chosen tag selects its own file and its ancestors' only (3b), and ne has none. The title readings stay where they were. A name reading never prefers, so under ne each word holds both readings and derives TITLES_AMBIGUOUS, a title only inside a title phrase (rules/H9, the title phrase).
  • locales.NE = Locale(code="ne", lexicon=Lexicon.empty(), languages=frozenset({"ne"})), registered in locales/__init__.py. It declares no DEVIATES.
    • tests/v2/test_locales.py's _deviates(code) ors the surface derived from _reading_index.changed_words into the predicate of every pack with languages. A hand DEVIATES therefore needs to declare only what a fragment or patch beside a choice changes. Nothing refuses one that restates the choice's words.
    • The registry contract refuses any DEVIATES attribute (even None) on a pack that is nothing but a choice, and requires one of every other pack. The rotators cover each word in each script, at the head, behind a listed title and after a family comma.
  • local_corpora.py alike --languages: the pair measure under a choice, against the default's flags (the tree's own or --base's), as snapshot.py --languages does. It re-keys every kept alike base once (_parse joins _ALIKE_CODE).
    • Both flags read the bare choice, not a registered pack's fragment or patch. That is the whole of NE; both tools' docs state the limit.
  • cldr_agreement.py maps CLDR's ne file and Wikidata's ne-np stratum to NE. cldr_agreement_locales.json gains NE's rows, and neither score moves.
  • hi.toml: not touched. Its श्री note (a given reading there would make the title one only inside a title phrase) stays true of the default. 3e can edit it freely.

The default does not move

build.py --check passes, _compiled.py is unchanged, and _index.py gains three rows. tests/v2/test_snapshot.py passes with no re-record.

Under a family-first order

The words are name words placed by position. Under a family-first order they land in the family name: श्री राम बहादुर थापा gives family श्री, with given राम under FAMILY_FIRST and given थापा (middle राम बहादुर) under FAMILY_FIRST_GIVEN_LAST. The docs and the release bullet say so.

What ne moves (2026-10-30, on 40675146 plus this change)

  • Committed corpora (snapshot.py --languages ne): 6 entries.
    • 4 are CLDR's hi, kok and mr renderings opening with the salutation श्री (CLDR title श्री). They lose the title, the cost the user docs state: use the pack for Nepali data.
    • Guru Gobind Singh and Guru Nanak read given Guru.
  • CLDR per pack: ne stays 15/24 and Wikidata ne-np 10/10, both ways. Neither file holds one of the words.
  • nepal-candidates (snapshot.py --local nepal-candidates --base <checkout> --languages ne): 296 names move. In every one the word leaves the title for the given name: 209 श्री, 42 Guru, 41 गुरु, 1 GURU, 1 श्री., 2 सु.श्री.
  • Pairs (local_corpora.py alike nepal-candidates --languages ne): 125,833 → 126,040 of 126,981 read alike, with 211 now alike and 4 now apart.
    • All 211 hold श्री, which the romanization writes as a name word (Shree 191, Shri 6, Sri 2).
    • Of the 4 now apart, 3 romanize श्री as Mr.. The fourth is the open defect below.

Open

  • A glued period-joined title reads as a name word under ne, in both directions. period_joined_vocab makes a token a title when ANY chunk is in titles, and श्री has moved to titles_ambiguous.
    • It costs सु.श्री. (Sushri, written with its periods): 2 nepal-candidates names read given.
    • It fixes श्री.राम थापा, which reads given श्री.राम instead of title.
    • A fix would move the default for every TITLES_AMBIGUOUS word, and its grid should hold both shapes.

Recorded in decisions/vocabulary-model's 2026-10-30 phase 3d entry and decisions/deviates-registry. The RFC status line is updated.

Checks

  • uv run pytest: 33,000 passed, on 3.11 and on 3.15 (-n 4; the default -n auto hit an xdist INTERNALERROR at load ~100).
  • mypy, ruff, build.py --check, sphinx-build -b doctest and python -m doctest README.rst are clean.
  • pytest -n0 -m serial: 42 passed at load ~30. Earlier runs at load 50 to 100 failed a different set of wall-clock ratios each time, and so did master at 40675146.
  • The ja extra was absent for every run.

🤖 Generated with Claude Code

A choice of `ne` turns on `given` readings of श्री, guru and गुरु, written
`default = false` in ne.toml, so each is a title only inside a title
phrase (rules/H9): `Guru Prasad Sharma` reads given `Guru` under the pack,
title `Guru` by default. The default's compile does not move; only the
reading index gains the three rows.

locales.NE is nothing but its choice and declares no DEVIATES: the
registry contract in tests/v2/test_locales.py derives a vocabulary
pack's surface from changed_words and refuses a hand-written one there.
local_corpora.py alike takes --languages for the pair measure, and
cldr_agreement.py scores NE on CLDR's ne file and Wikidata's ne-np
stratum (unchanged, 15/24 and 10/10).

Measured on nepal-candidates: 211 pairs now read alike, 4 now apart; the
decisions entry records the moves, the CLDR hi/kok/mr cost and the open
period-joined सु.श्री. case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73 derek73 added enhancement tooling vocabulary Per-word vocabulary model and language packs labels Oct 30, 2026
@derek73 derek73 self-assigned this Oct 30, 2026
derek73 and others added 2 commits October 30, 2026 03:23
…d review)

The gate ors the surface derived from a pack's languages into every
pack's predicate, so a hand DEVIATES declares only what a fragment or a
patch beside a choice changes. snapshot.py and alike --languages say they
read the choice alone. The docs and the release bullet say ne's words are
name words placed by position (family under a family-first order), the
warning excepts a choice pack, and the decisions entry's open item records
that period_joined_vocab moves both ways (सु.श्री. costs, श्री.राम gains).
The guru note counts 43 names moved.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…m (3d review 2)

FAMILY_FIRST reads given राम, FAMILY_FIRST_GIVEN_LAST given थापा. A hand
copy of a choice's words is no longer needed beside a patch, but nothing
refuses one; a pure choice pack must not define DEVIATES at all
(hasattr, so DEVIATES = None is refused too). The warning's aside no
longer swallows "cannot tell whose name it is looking at".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73
derek73 merged commit 71d2a81 into master Oct 30, 2026
13 checks passed
derek73 added a commit that referenced this pull request Oct 30, 2026
Squashed for the rebase onto #955: the pack commit and its two review rounds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
derek73 added a commit that referenced this pull request Oct 30, 2026
Squashed for the rebase onto #955: the pack commit and its two review rounds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
derek73 added a commit that referenced this pull request Oct 30, 2026
Squashed for the rebase onto #955: the pack commit and its two review rounds.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement tooling vocabulary Per-word vocabulary model and language packs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant