Repository navigation
Design 3.0's language packs and output model (#753, #754) - #940
Merged
Merged
Conversation
…e pool (#753) The RFC's Phase 3 section rewritten and a dated decisions/vocabulary-model entry, every item marked proposed until Derek decides it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… findings (#753) Strict cannot be built on today's files (name readings sit where their count came from), so it waits; the add-only pack contract and the 3-0 Standing line are named as amended; counts, pointers and scope fixed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…estions (#753) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… are views (#754) A spec draft for review, every item marked proposed. The RFC's Phase 4 section gains a status line, the [rules]-into-policy-patch proposal, the output model (Token.reading; generation/credentials, patronymic and surname2 as views; middle unchanged; render defaults following the read order), an API-change table against the 3.0 freeze, phase 3's Q1 confirmed, and a 4a-4h work order interleaved with 3a-3h. decisions/vocabulary-model gains the dated, proposed 2026-10-28 #754 entry carrying the measurements from the fielded corpora. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scope the render exclusion to word order, so a given-first interpunct name keeps its dot; re-count the corpus figures over distinct entries under their declared orders and name the ending tests; point at phase 3's unmerged draft instead of an entry master lacks; say surname2 is read from O6's tag; leave order out of comparison; list the four reserved names as without a sub-phase; drop the Đỗ promise Token.reading cannot keep and the release plan no entry records. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rules/R2 says the base is the family without its leading particles; the code removes every working particle. Each interior one leads its own name unit, so the per-surname reading and the code agree. Proposed: amend R2 to the code. Measured over the committed corpus and tse-donors, in the default policy and with middle_as_family. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…754) Comma names drop the comma as a separator; the str() change reaches given-first names with an interpunct; order's exclusion from == is left open; suffix counts re-measured per word (33/77/64) with the recipe stated; CLDR samples counted as distinct names; R2's proposal points at #937's unmerged branch, covers particle runs and trailing particles (39/505), and names migrate.rst among the sites landing it touches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
family_base removes every working particle wherever it stands, and R2 now says so, its Because calling the base a view the same under every naming convention rather than a sort key. decisions/R2 records the Portuguese last-surname and Spanish first-surname filing conventions with sources (RAE's rules a and e read from the page), and the RFC leaves open whether a pack supplies a convention's sort key. R2's size grows by three example lines and reads no new signal; budget re-recorded. Leaving ParsedName.order out of == is recorded as an accepted cost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two words are joined by their written separator when they stand one after the other in the same order in the text and the rendering. A family comma never qualifies, since the rendering reverses its pair; a comma before a suffix does, so 'John Smith, Jr.' keeps it (456 of the 1,008 committed entries with a suffix after a family write one). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A kept separator is whitespace, nothing, an interpunct, a nakaguro or one comma; delimiters and consumed marker words take the spec's joiner. Only a family comma keeps a name given-first. The suffix-comma count is re-taken (214 of 1,003 keep the comma) and the differing render of 'Smith, John, Jr.' is recorded as accepted. R2's entry says where the base departs from the Spanish key (rule b's La and San), speaks of R2's old Then in the past, and the RFC keeps R2's all-particle exception. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A shape for later: a pack may carry given-first, surname-first and sorting patterns. The RFC maps Phase 4's views to CLDR's person-name fields and names the gap (CLDR's prefix and core are the first surname's), proposes a casefolded key method, and leaves open whether a pattern is reached through the name or the parser. Two Standing pointers gain their entry's head words, now that two 2026-10-28 #754 entries exist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A spec joiner that decorates a field (the ') ' after a maiden name) stands over a kept separator; a comma may be ASCII or full-width, with the counts' ASCII scope stated. The CLDR table splits a one-unit surname (family, today) from a two-unit one, and option (a) names its cost of a new way to ask for a pattern, render() taking a format spec. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Title tokens show behaviors, not kinds, and ReadingKind reads every title as TITLE. The RFC's Phase 4 Left open asks whether Reading gains an optional vocabulary-kind field, and so whether phase 3's reading index carries kinds; no title word holds two kinds today (816 checked). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A shape question for before the freeze: keep a public field per behavior set (22 today, 11 added since 2.0.0), or answer word, kind and property queries from phase 3's reading index and keep the sets internal. The RFC states the costs on each side: frozen fields, pickles, reprs, how a caller adds a word, and per-token frames. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A spec joiner that decorates a field now precedes the kept-separator branch, and a kept comma carries the whitespace written beside it, so an unspaced full-width comma is kept too. Five of the 728 hold a full-width family comma; the decisions copy scopes its counts to the ASCII comma. Left open becomes lists, with the Lexicon fields question set apart as a shape question; its pickle cost is corrected (Lexicon's own guard admits no ADDED_IN field) and the index's load cost added. The title-kind item notes that phase 3's index already holds a kind per row, and that titles are split by name collision, not kind. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rule now covers every separator: kept ones (nothing, whitespace, interpunct, nakaguro, one comma of any kind the parse reads, Arabic included), the spec's decoration between two fields standing instead, and the spec's joiner for everything else, in both copies. 36 of the 728 hold a family comma the parse reads (5 full-width, 31 Arabic). The index load cost carries its no-cache qualifier, and the title-kind item narrows to whether Reading exposes the vocabulary's kind. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A kept separator keeps its kind, not its written whitespace: any whitespace becomes one space, an interpunct or nakaguro stands alone, an ASCII or Arabic comma becomes comma-space, and a full-width comma stands alone. The written whitespace stays in ParsedName.original and the token spans. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The change list no longer counts whitespace-separated names, which the normalized rule leaves as they are, and the Standing line says the separators are normalized. Only an interpunct the parse reads as a separator stands alone; a spaced one is a name word (rules/T3). The interpunct comma count reads every comma the parse reads, the decisions copy names directional marks and the Arabic counts, the whitespace note sits after the rule, and the index-cost comparison names the cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…754) U+3000 is whitespace like any other, so it renders as one ASCII space; the full-width comma's own-width reason is scoped to the comma. The Japanese comma is noted as a parser question outside the design. The per-word index's import cost now matches decisions/A1's measurement (0.35 ms of 18.2 ms), where two copies had 0.4 of 19.5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The U+3000 example no longer reads as today's output, which is turned round; the decisions copy states the full-width comma's reason, says ASCII space, and carries the Japanese-comma note; the change list covers only separators between words the rendering keeps side by side. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
render(spec=None): None is the order-aware default str() passes, and a caller's spec renders exactly as written, so 2.x specs keep their meaning. The family-first layout keeps title first and suffix last with the name roles in order's sequence, checked against CLDR's surnameFirst samples. Both marked proposed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The proposed render(spec=None) is named a signature change, not an addition, in both copies and the API table. A family-comma name keeps a given-first order, so it renders by today's spec. The CLDR comparison now says most samples put the title after the family (50 of 70), so the title's place in the family-first default stays open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing in 3a-3h said whose names a name reading's are, which strict waits on. Record it as open beside the strict paragraph and in the decisions entry's item 4: an attribute naming each name reading's bearers' languages, filled word by word, since a box count says where a word was typed, not whose name it is (do's ORCID family box is led by Brazil's particle, none of its 7,227 boxes do alone). Proposed as 3i, after 3a, neutral; nothing decided. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…753) - The reading index loads for every caller, since Reading.languages (4a) is filled from it on every parse; lazy loading would need a Reading to carry its word as a hidden key. Cached import measured at about 0.45 ms against about 20 ms for import nameparser; the earlier 11 ms was taken under PYTHONDONTWRITEBYTECODE. - The Locale/Policy dependency is answered by the 2026-10-28 freeze. - Work order: 3f and 3h need 4b, 3g needs 4d and 4g, 4f lands with or after 3b; RU and TR_AZ wait for 4f. - The shape-questions-first paragraph is history: the output model was agreed 2026-10-28 (#754). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The phase table's rows 3 and 4 say both states: phase 3's design proposed 2026-10-27 and not agreed, phase 4's output model agreed 2026-10-28. The log keeps both entries in date order. Phase 4's text now cites phase 3 in this file rather than its branch, mentions 3i in the interleaving, and drops the stale claim that the reading index loads only for a caller who chooses a language. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
render(spec: str | None = None) is taken: None is the order-aware default str() passes. The family-first layout's title place stays open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The family-first default keeps the title first, the one place the parse reads every title back from; reading titles where CLDR writes them is a separate parser issue. Vocabulary is configured by reading in 3.0 and the compiled sets become internal, the first break decided under the core API freeze; its 2.x warning is open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Items 1, 2, 4, 5, 6 and 8 are accepted as written: the choice of languages on Locale.languages, one mechanism, strict waiting for 3i, the reading index loaded for every caller, tag selection by file and ancestors, and third-party files deferred. Items 3, 7 and 9-12 stay proposed. The add-only pack contract is amended in 3b's PR, as the items tie it there. A pack's lexicon fragment and Locale(lexicon=...)'s successor now point at the Lexicon-surface entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Items 3, 7 and 9-12 are accepted as written, so all twelve are decided and the entry's head and the RFC's status say agreed. 3g's connective stays an open design inside the work order. The add-only pack contract is still amended in 3b's PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
R2's example 'da Silva do Amaral de Souza, Miguel' joins corpus_rules.jsonl and the snapshot as one new entry; no recorded reading moves, and every agreement record still matches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- rules/R2 gains an Except: a connective between two words of a part is set aside when the part is asked whether every word is particle vocabulary, so `Fritz Freiherr und von Bar` reads base 'Freiherr und von Bar'; today's base 'und' is a deviates: marker for #939. A connective ending the part stays a name word (`Juan de y` boundary). rules/R3's all-particles Except follows it, with its own marker. decisions/R2 records the case, dated 2026-10-28. - decisions/3-0-reevaluations records the render() signature break as waived, taking no 2.x warning, and updates the render-defaults and harness Standing lines. - The 3i counts are recounted at c8357e7 and dated; the Ministro reproduction names both titles added; Lim Mei Ling Jessica is scoped to FAMILY_FIRST. - rules_budget re-recorded (R2 and R3 grew by the new clause); corpus_rules gains the new example. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- rules/R4's all-particles Except sets a connective between the part's words aside as R2 does, with `FRITZ FREIHERR UND VON BAR` marked deviates: #939 (today 'Fritz freiherr und von bar'). - rules/R2 and decisions/R2 say a joining connective anchors no base on its own but stays in the base view, instead of "is never the base", which its own example and R3's `Jon Dough and` contradict. - decisions/R2 names R4 beside R3, and its unjoined-mark Standing line is scoped to a connective ending the part. - rules_budget re-recorded; corpus_rules and the snapshot gain the new example, no recorded reading moved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- rules/R2, R3 and R4's connective set-aside reaches only a connective that is not also particle vocabulary; a word a caller lists as both keeps R2's readmission wherever it stands (the readings the #461 tests and INV2 pin). The default lists no word as both, so this is stated in prose with no example line. - decisions/R2's unjoined-mark Standing line again covers every connective and says how it relates to the set-aside; the case-repair value moves onto the all-capitals spelling; the Open line names R4. - decisions/R3 and decisions/R4 each gain a dated #939 pointer. - R4's new sentence is trimmed to the test; rules_budget re-recorded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- tests/v2/test_render.py gains a test pinning a word listed as both a particle and a conjunction where it stands between particles (`Juan de y la` base 'de y la', initials 'J. d. y. l.'; `Fritz Freiherr y von Bar` 'F. F. y. v. B.'), guarding R2's scoping of its connective set-aside. A mutation setting aside every interior connective of an all-particle part fails it, while the neighboring #461 test still passes. - decisions/R2 names the test pinning each reading; the interior ones were attributed to tests that never reach them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts resolved: decisions/3-0-reevaluations keeps the removal's entries beside the render-signature waiver and the Lexicon surface entry, its render Standing line pointing at the decided defaults and its harness line under Not standing; decisions/R3 and decisions/R4 keep both same-day entries; rules_budget re-recorded rather than merged. #937's test comment quoting R2's old Then is re-quoted to the amended one, and the design branch's prose no longer treats #937 as pending or points at the retired "clean slate". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
decided 2026-10-29: like the render() signature, the Lexicon surface break takes no 2.x warning. Unlike it, the break does change calls that work today, but 2.8 was the last 2.x feature release and no deprecation path ships for the compiled-set fields, so 3.0's release notes carry the change. decisions/3-0-reevaluations gains THE LEXICON WARNING entry and its Standing line; decisions/vocabulary-model and the RFC no longer list the warning as open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lands the agreed design for vocabulary phase 3 (language packs) and phase 4 (order, shape and the output model), decided 2026-10-28. It holds no parser change. The record lives in the docs; this description only indexes it.
Phase 3: language packs (#753)
Overview:
docs/design/vocabulary-model-rfc.md, Phase 3. Record:decisions/vocabulary-model.md, 2026-10-27 #753 entry. All twelve items were agreed 2026-10-28:Locale.languages, unioned byparser_for. Strict is an argument of the whole choice.default = falseis allowed on name readings. When turned on, they derive markers and never settle a clash (decision 15 unchanged).snapshot.pytaking a choice.default = falsereadings.Phase 4: the output model (#754)
Overview: the RFC's Phase 4 section. Record:
decisions/vocabulary-model.md, 2026-10-28 #754 entry, plus its render patterns andLexiconsurface entries.generation/credentials,patronymicandsurname2are views read from a newToken.reading(or from rules/O6's tag).family.middlekeeps its meaning (Should a two-word given name likeMaría JoséorMin Hofillgiven, instead of given + middle? #704).[rules]compile into aLocale's policy patch.render(spec: str | None = None)is a signature break that was decided and waived: it takes no 2.x warning, since no call that works today changes meaning (decisions/3-0-reevaluations, THE RENDER SIGNATURE).Lexiconis configured by reading, and its compiled sets become internal in 3.0. This is a break decided under the freeze, and its 2.x warning was waived on 2026-10-29: 2.8 was the last 2.x feature release, so 3.0's release notes carry the change (decisions/3-0-reevaluations, THE LEXICON SURFACE and THE LEXICON WARNING).Wang Xiaoming SenatorandWang Dr Xiaomingread the title, where CLDR writes surname-first names? #938.Review
A design-docs review over the combined diff led to three fix rounds, each reviewed in turn:
c8357e74.A test now pins a word listed as both a particle and a conjunction where it stands between particles. A mutation that sets aside every interior connective fails it.
R2 amendment and #939
decisions/R2, 2026-10-28 Phase 4: order and shapes, the rules a pack can choose, and the output model #754).Fritz Freiherr und von Barshould read base 'Freiherr und von Bar' where today it gives 'und'.deviates: #939markers sit in rules/R2 (family_base), R3 (initials) and R4 (capitalized).rules_budget.jsonwas re-recorded for R2, R3 and R4.corpus_rules.jsonland the snapshot gain the new examples, and no recorded reading moved.Merged in
#937 (the facade removal) is merged into this branch. Its test comment quoting R2's old Then is re-quoted to the amended one.
Still open
Lexicon's own pickle guard learning theADDED_INmarker.Closes nothing: #753 and #754 stay open for implementation.
Refs #753, #754, #939
🤖 Generated with Claude Code