[derek73]

What this adds

A new scored local corpus, cz-candidates: the candidate registers the Czech Statistical Office (ČSÚ) publishes on volby.gov.cz, starting with the 2022 municipal elections (kv2022) and the 2025 Chamber election (ps2025). It is the first local corpus where titles are stated separately from the name. Each candidate filed four boxes: the title before the name (TITULPRED), the given names (JMENO), the surname (PRIJMENI) and the title after it (TITULZA). 36% of the records carry a title, nearly all academic degrees.

• tools/differential/local_volby.py fetches each election's index page and reads the register's CSV zip name off it. The zip name carries a date stamp, and a guessed stamp gets an HTML 404 page, so the name is never hard-coded. An answer that is not a zip is retried and then refused. The fetch reads the UTF-8 csv_od/ member by column name, because the elections' registers differ in every other column. It keeps one record per distinct set of boxes per election, so the corpus counts records, not candidates: 31,770 of kv2022's rows fold into another record, most of them different people sharing all four boxes. The fetch refuses:
  • a register missing a box's column;
  • a register whose last row is cut short (the CSV row-count gotcha in AGENTS.md);
  • a register holding a different number of candidates than its pin in VOLBY_ELECTIONS. A register is not final once its election is held: kv2022's carries by-elections through 2024 and was re-issued on 2026-03-28, so the pin is what protects;
  • a register with too many malformed or nameless rows;
  • a register where fewer than a fifth of candidates are titled.
    Adding an election is one row in VOLBY_ELECTIONS.
• The name is composed, not written by anyone. It is built as TITULPRED JMENO PRIJMENI, TITULZA (Ing. Jan Novák, Ph.D.), so it has no case, spacing or punctuation of its own. The module docstring, the README and the credit line all say so. meta.json records VOLBY_READ_VERSION, and a corpus built by a different version of compose is refused on load. A comma typed inside a box before the last (Dopitová, roz. Lišková in the surname box, Ing., Bc. in the title box: 21 records) makes a comma shape nobody wrote. Those records are kept as filed and counted (records_with_a_comma_inside_a_box), an accepted cost stated in the docstring and README: the score reads the boxes, so dropping the records would only hide the miss.
• Licence. CC BY 4.0, per https://csu.gov.cz/podminky_pro_vyuzivani_a_dalsi_zverejnovani_statistickych_udaju_csu (re-read 2026-10-30): "Statistické informace Českého statistického úřadu … jsou licencovány v souladu s CC BY 4.0." The page also asks that derived data be marked as derived, and the credit line does that. The names are personal data, so they are cached and never committed, like the other local corpora.
• local_agreement.py changes:
  • This is the first source with a box after the name. When a source has a suffix box, the suffix is compared with that box alone and is no longer appended to the family.
  • A new Fielded.sets_aside flag switches off the title set-aside. That rule was built for PSC's free-text title box. Here it would have excused 1,059 records, each a degree or rank the parser missed and none a job typed into the box. Of them, 756 hold dis in the box after the name (re.search(r"\bdis\b", box.replace(".", "").casefold())), and 179 of the rest hold et in the title box (re.search(r"\bet\b", box); 182 by that test alone, since 3 hold both).
  • --locale cs scores parser_for(locales.CS) against its own record (agreement.cs.jsonl, with kept --base records keyed by locale too). A pack's score and the default's are never read as each other. An unknown locale is refused before any record is read or base scored.
  • Merging moves the scorer fingerprint for every source. parser_of joins _SCORER_CODE, and read, score and _read_chunk change. Every existing shared agreement record (psc-individuals, orcid-researchers, patentsview-inventors) will be refused as written by another scorer until re-recorded with --record --tree <base>. --base copes on its own: its kept records are keyed by the fingerprint, so it scores the base again.
• Wired into SOURCES, so snapshot.py --local, tally, name_usage.py and measure.py all pick it up. There is a README table row and paragraph, and a sentence in AGENTS.md's "Validating a new parsing rule". There is no release-log bullet: Add Companies House's PSC register as a local corpus, with a local agreement score #849 and Add the nepal-candidates local corpus: Nepal's 2079 BS local election candidates #866 added none for a tools-only change.

Measured on the fetch of 2026-10-30

There are 170,181 records: 165,736 of kv2022's 197,506 candidates and 4,445 of ps2025's 4,475. The fetch took 25 s and the score 3.4 s. Each cell is default / locales.CS:

election	records	every field	every field, titled records	box before the name (filled)	box after the name (filled)	family
kv2022	165,736	96.98% / 96.99%	92.93% / 92.97%	97.77% / 97.83% of 56,721	54.35% / 54.50% of 7,233	99.03% / 99.03%
ps2025	4,445	96.18% / 96.22%	92.79% / 92.89%	97.02% / 97.12% of 1,946	81.50% / 81.94% of 454	98.47% / 98.47%
all	170,181	96.96% / 96.97%	92.93% / 92.97%	97.74% / 97.81% of 58,667	55.95% / 56.12% of 7,687	99.01% / 99.01%

The survey's sample gave 95.0% for the post-nominal box. That figure was taken over titled candidates, most of whom left the box after the name empty, so it agreed trivially. Over the boxes that are actually filled, agreement is 56%, because DiS. fills 42% of them.

What the misses are, on the whole corpus (default / cs):

• 3,240 / 3,240: DiS. after the name, the box after the name missed. Predicate: the box after the name holds dis (re.search(r"\bdis\b", box.replace(".", "").casefold())), 3,248 records, and that box disagrees in 3,240. Of those, 3,232 read DiS. as a title (the parsed title holds dis by the same test): Jan Novák, DiS. gives title DiS.. The other 8 read the whole name before the comma as the family (DiS. et DiS. 6, a bare DiS 1, BBus, DiS. 1). The mechanism: the part after the comma is not suffix vocabulary, so it reads as a family comma (rules/C1). rules/H2 then reads the unlisted abbreviation at the head of the given part as a title, and rules/C8 reads the part before the comma as a whole name. Any unlisted abbreviation behaves the same: Jan Novák, Xyz. gives title Xyz., and so do dipl. um. (27), Dr. (10) and ArtD..
• 1,483 / 1,315: a two-word surname read as a middle name plus the family. For example, a married double surname Pešová Řehořová is read as middle Pešová. With Policy(two_family_names=True) the family box's misses fall from 1,683 to 825 (default) and 986 (cs), and the given box's from 1,828 / 1,644 to 743 / 732.
• 164 / 153: a connective between two titles that only their shape makes titles, read as given names. Ing. et Ing. (102 of 102 missed), Bc. et Bc. (38 of 38). Mgr. et Mgr. reads correctly in 218 of 223, because Mgr. is in the vocabulary. Under cs, JUDr. et Mgr. reads correctly too, because the pack adds JUDr..
• 11 / 245: a generation word typed into the surname box (Novák ml. 177, st. 57 under cs). It reads as the suffix it is, and the score counts it against the empty box after the name. Here the box is out of place, not the parse.
• The rest are small leads: English degrees filed before the name (BA, BSc, B.A., 36) read as given names; BA (Hons) read with (Hons) as a nickname; roz. typed into the surname box; and the surnames Ceh and Mls read as credentials.

This PR does not change parser behaviour. Issue drafts for the first three shapes are with the orchestrating session for approval and have not been filed.

Checks

• uv run pytest: 33,546 passed.
• uv run mypy, uv run ruff check: clean.
• The new and changed tests also pass on 3.15.
• New tests: tests/v2/test_local_volby.py, a fixture-served index page and zip covering each refusal. The last-row check has a negative control: with its short-row arm removed, the cut-inside-a-quoted-field case fails.
• New test_local_agreement.py tests cover the suffix box, sets_aside, and --locale with its record, its kept --base record and its workers.
• Fetched for real once, and the counts above come from that fetch.

🤖 Generated with Claude Code