Skip to content

locale.nl_langinfo(locale.ERA) does not work for past eras #126727

Closed
Closed
locale.nl_langinfo(locale.ERA) does not work for past eras#126727
Labels
3.12only security fixes3.13only security fixes3.14bugs and security fixestype-bugAn unexpected behavior, bug, or error

Description

@serhiy-storchaka
serhiy-storchaka
opened on Nov 12, 2024

Bug report

According to the Posix specification (https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap07.html#tag_07_03_05_02), nl_langinfo(ERA) should return a string containing semicolon separated era description segments. But in Glibc it uses NUL instead of a semicolon as a separator. As result, locale.nl_langinfo(locale.ERA) in Python only returns the first segment, corresponding to the last (current) era. For example, in Japanese locale the result cannot be used for data before year 2020:

>>> import locale
>>> locale.setlocale(locale.LC_ALL, 'ja_JP')
'ja_JP'
>>> locale.nl_langinfo(locale.ERA)
'+:2:2020/01/01:+*:令和:%EC%Ey年'

This issue is similar to #124969, but at least the result can be used for the current date.

cc @methane, @kulikjak.

Linked PRs

Activity

  1. serhiy-storchaka
    added
    type-bugAn unexpected behavior, bug, or error
    3.12only security fixes
    3.13only security fixes
    3.14bugs and security fixes
    on Nov 12, 2024
  2. serhiy-storchaka
    added this to Locale issues 🗺on Nov 12, 2024
  3. github-project-automation
    moved this to Todo in Locale issues 🗺on Nov 12, 2024
  4. serhiy-storchaka
    moved this from Todo to In Progress in Locale issues 🗺on Nov 12, 2024
  5. serhiy-storchaka
    added a commit that references this issue on Nov 12, 2024

    pythongh-126727: Fix locale.nl_langinfo(locale.ERA)

    80200a2

  6. serhiy-storchaka commented on Nov 12, 2024

    @serhiy-storchaka
    serhiy-storchaka
    on Nov 12, 2024
    MemberAuthor

    On my computer (Linux) the following script

    import locale, subprocess
alllocales = subprocess.check_output(['locale', '-a']).decode().split()
for loc in alllocales:
    if '.' in loc or '@' in loc:
        continue
    try:
        _ = locale.setlocale(locale.LC_ALL, loc)
    except locale.Error:
        continue
    era = locale.nl_langinfo(locale.ERA)
    if era:
        print(loc, era.count(';'), era)

    now produces the following output:

    cmn_TW 2 +:2:1913/01/01:+*:民國:%EC%Ey年;+:1:1912/01/01:1912/12/31:民國:%EC元年;+:1:1911/12/31:-*:民前:%EC%Ey年
hak_TW 2 +:2:1913/01/01:+*:民國:%EC%Ey年;+:1:1912/01/01:1912/12/31:民國:%EC元年;+:1:1911/12/31:-*:民前:%EC%Ey年
ja_JP 10 +:2:2020/01/01:+*:令和:%EC%Ey年;+:1:2019/05/01:2019/12/31:令和:%EC元年;+:2:1990/01/01:2019/04/30:平成:%EC%Ey年;+:1:1989/01/08:1989/12/31:平成:%EC元年;+:2:1927/01/01:1989/01/07:昭和:%EC%Ey年;+:1:1926/12/25:1926/12/31:昭和:%EC元年;+:2:1913/01/01:1926/12/24:大正:%EC%Ey年;+:1:1912/07/30:1912/12/31:大正:%EC元年;+:6:1873/01/01:1912/07/29:明治:%EC%Ey年;+:1:0001/01/01:1872/12/31:西暦:%EC%Ey年;+:1:-0001/12/31:-*:紀元前:%EC%Ey年
japanese 10 +:2:2020/01/01:+*:令和:%EC%Ey年;+:1:2019/05/01:2019/12/31:令和:%EC元年;+:2:1990/01/01:2019/04/30:平成:%EC%Ey年;+:1:1989/01/08:1989/12/31:平成:%EC元年;+:2:1927/01/01:1989/01/07:昭和:%EC%Ey年;+:1:1926/12/25:1926/12/31:昭和:%EC元年;+:2:1913/01/01:1926/12/24:大正:%EC%Ey年;+:1:1912/07/30:1912/12/31:大正:%EC元年;+:6:1873/01/01:1912/07/29:明治:%EC%Ey年;+:1:0001/01/01:1872/12/31:西暦:%EC%Ey年;+:1:-0001/12/31:-*:紀元前:%EC%Ey年
lo_LA 0 +:1:-543/01/01:+*:ພ.ສ.:%EC %Ey
lzh_TW 2 +:2:1913/01/01:+*:民國:%EC%Ey年;+:1:1912/01/01:1912/12/31:民國:%EC元年;+:1:1911/12/31:-*:民前:%EC%Ey年
nan_TW 2 +:2:1913/01/01:+*:民國:%EC%Ey年;+:1:1912/01/01:1912/12/31:民國:%EC元年;+:1:1911/12/31:-*:民前:%EC%Ey年
thai 0 +:1:-543/01/01:+*:พ.ศ.:%EC %Ey
th_TH 0 +:1:-543/01/01:+*:พ.ศ.:%EC %Ey
zh_TW 2 +:2:1913/01/01:+*:民國:%EC%Ey年;+:1:1912/01/01:1912/12/31:民國:%EC元年;+:1:1911/12/31:-*:民前:%EC%Ey年

    The ERA values are not set on FreeBSD and Illumos, and I suppose on macOS and Solaris too. It seems that currently they are only set on Linux.

  7. serhiy-storchaka
    added 3 commits that reference this issue on Nov 21, 2024

    gh-126727: Fix locale.nl_langinfo(locale.ERA) (GH-126730)

    4803cd0

    [3.13] pythongh-126727: Fix locale.nl_langinfo(locale.ERA) (pythonGH-…

    a72886c

    [3.12] pythongh-126727: Fix locale.nl_langinfo(locale.ERA) (pythonGH-…

    5c1157e
  8. serhiy-storchaka
    closed this as completedon Nov 21, 2024
  9. github-project-automation
    moved this from In Progress to Done in Locale issues 🗺on Nov 21, 2024
  10. serhiy-storchaka
    added 2 commits that reference this issue on Nov 21, 2024

    [3.12] gh-126727: Fix locale.nl_langinfo(locale.ERA) (GH-126730) (GH-…

    4b4e0db

    [3.13] gh-126727: Fix locale.nl_langinfo(locale.ERA) (GH-126730) (GH-…

    6e5e7bc

  11. kulikjak commented on Nov 27, 2024

    @kulikjak
    kulikjak
    on Nov 27, 2024
    Contributor

    Hi @serhiy-storchaka, I am sorry I couldn't test this earlier.

    While they are indeed not set on Illumos, we have them on Oracle Solaris for some locales:

    ja_JP.PCK 4 +:2:2020/01/01:+*:令和:%EC%Ey年;+:1:2019/05/01:2019/12/31:令和:%EC元年;+:2:1990/01/01:2019/04/30:平成:%EC%Ey年;+:1:1989/01/08:1989/12/31:平成:%EC元年;+:2:1927/01/01:1989/01/07:昭和:%EC%Ey年
ja_JP.UTF-8 4 +:2:2020/01/01:+*:令和:%EC%Ey年;+:1:2019/05/01:2019/12/31:令和:%EC元年;+:2:1990/01/01:2019/04/30:平成:%EC%Ey年;+:1:1989/01/08:1989/12/31:平成:%EC元年;+:2:1927/01/01:1989/01/07:昭和:%EC%Ey年
ja_JP.eucJP 4 +:2:2020/01/01:+*:令和:%EC%Ey年;+:1:2019/05/01:2019/12/31:令和:%EC元年;+:2:1990/01/01:2019/04/30:平成:%EC%Ey年;+:1:1989/01/08:1989/12/31:平成:%EC元年;+:2:1927/01/01:1989/01/07:昭和:%EC%Ey年
th_TH.TIS620 0 +:0:-543/01/01:+*::พ.ศ. %Ey
zh_CN.GB18030 1 +:0:0000/01/01:+*:公元:%EC%Ey年;+:1:-0001/12/31:-*:公元前:%EC%Ey年
zh_CN.GB18030@pinyin 1 +:0:0000/01/01:+*:公元:%EC%Ey年;+:1:-0001/12/31:-*:公元前:%EC%Ey年
zh_CN.GB18030@radical 1 +:0:0000/01/01:+*:公元:%EC%Ey年;+:1:-0001/12/31:-*:公元前:%EC%Ey年
zh_CN.GB18030@stroke 1 +:0:0000/01/01:+*:公元:%EC%Ey年;+:1:-0001/12/31:-*:公元前:%EC%Ey年
zh_TW.BIG5@zhuyin 4 +:2:1913/01/01:+*:中華民國:%EC%Ey年;+:1:1912/01/01:1912/12/31:中華民國:%EC元年;+:1:1911/12/31:-*:民國:%EC%Ey年;+:1:1/1/1:1911/12/31:西元:%EC%Ey年;+:1:-1/12/31:-*:西元前:%EC%Ey年
zh_TW.EUC@zhuyin 4 +:2:1913/01/01:+*:中華民國:%EC%Ey年;+:1:1912/01/01:1912/12/31:中華民國:%EC元年;+:1:1911/12/31:-*:民國:%EC%Ey年;+:1:1/1/1:1911/12/31:西元:%EC%Ey年;+:1:-1/12/31:-*:西元前:%EC%Ey年
zh_TW.UTF-8@zhuyin 4 +:2:1913/01/01:+*:中華民國:%EC%Ey年;+:1:1912/01/01:1912/12/31:中華民國:%EC元年;+:1:1911/12/31:-*:民國:%EC%Ey年;+:1:1/1/1:1911/12/31:西元:%EC%Ey年;+:1:-1/12/31:-*:西元前:%EC%Ey年

    Unfortunately, we have fewer Japanese ERAs than Linux (which causes the newly added test to fail), but I created #127327 to handle this.

  12. serhiy-storchaka
    added a commit that references this issue on Dec 5, 2024

    GH-126727: Fix test_era_nl_langinfo with Japanese ERAs on Solaris (GH…

    d958d9f
  13. miss-islington
    added 2 commits that reference this issue on Dec 5, 2024

    pythonGH-126727: Fix test_era_nl_langinfo with Japanese ERAs on Solar…

    088196e

    pythonGH-126727: Fix test_era_nl_langinfo with Japanese ERAs on Solar…

    139494e

  14. serhiy-storchaka commented on Dec 5, 2024

    @serhiy-storchaka
    serhiy-storchaka
    on Dec 5, 2024
    MemberAuthor

    Thank you @kulikjak.

  15. serhiy-storchaka
    added 2 commits that reference this issue on Dec 5, 2024

    [3.12] GH-126727: Fix test_era_nl_langinfo with Japanese ERAs on Sola…

    675324d

    [3.13] GH-126727: Fix test_era_nl_langinfo with Japanese ERAs on Sola…

    2da829b
  16. srinivasreddy
    added a commit that references this issue on Jan 8, 2025

    pythonGH-126727: Fix test_era_nl_langinfo with Japanese ERAs on Solar…

    9914861
  17. ebonnal
    added a commit that references this issue on Jan 12, 2025

    pythongh-126727: Fix locale.nl_langinfo(locale.ERA) (pythonGH-126730)

    9df4953
  18. serhiy-storchaka
    marked locale.nl_langinfo(locale.ERA) is broken #125411 as a duplicate of this issue on Jul 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.12only security fixes3.13only security fixes3.14bugs and security fixestype-bugAn unexpected behavior, bug, or error

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions