Repository navigation
fractions: Should we support unicode in width/precision formatting fields? #135025
Description
Bug report
Bug description:
Currently, specification allows only [0-9] digits. Though, actual implementation permits unicode symbols for float/Decimal's, but not Fraction's:
>>> f"{decimal.Decimal('123'):.١١f}" # arabic 11 in precision
'123.00000000000'
>>> f"{fractions.Fraction('123'):.١١f}"
Traceback (most recent call last):
File "<python-input-9>", line 1, in <module>
f"{fractions.Fraction('123'):.١١f}"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sk/src/cpython/Lib/fractions.py", line 600, in __format__
raise ValueError(
...<2 lines>...
)
ValueError: Invalid format specifier '.١١f' for object of type 'Fraction'
>>> f"{float(fractions.Fraction('123')):.١١f}"
'123.00000000000'Quick tests shows no measurable performance penalty with unicode support:
$ python -m timeit -s 'from fractions import Fraction as F' 'format(F(123), ".11f")'
10000 loops, best of 5: 39.2 usec per loop
$ python -m timeit -s 'from fractions import Fraction as F' 'format(F(123), ".١١f")' # with patch
5000 loops, best of 5: 40.2 usec per loop
a patch
diff --git a/Lib/fractions.py b/Lib/fractions.py
index 063f28478c..b4120b2beb 100644
--- a/Lib/fractions.py
+++ b/Lib/fractions.py
@@ -170,7 +170,7 @@ def _round_to_figures(n, d, figures):
(?P<zeropad>0(?=[0-9]))?
(?P<minimumwidth>0|[1-9][0-9]*)?
(?P<thousands_sep>[,_])?
- (?:\.(?P<precision>0|[1-9][0-9]*))?
+ (?:\.(?P<precision>0|\d*))?
(?P<presentation_type>[eEfFgG%])
""", re.DOTALL | re.VERBOSE).fullmatch
CPython versions tested on:
CPython main branch
Operating systems tested on:
No response
Activity
serhiy-storchaka commentedon Jun 2, 2025
It was perhaps overlooked in Python 2 to Python 3 transition.
I think that non-ASCII digits should be deprecated here.
skirpichev commentedon Jun 3, 2025
Maybe. My archeological research traced this story down to the PEP 3101 implementation. Note that PEP text is vague about width/precision fields. It says:
Though, I see no tests for this "feature".
I think so.
On another hand, support doesn't look too costly and it should be easy to adjust the documentation and the fractions module code.
CC @ericvsmith
ericvsmith commentedon Jun 3, 2025
If we were doing it all over again, I'd argue that we should accept only ASCII numbers in format strings (for precision and width). At this point, as much as I'd like to deprecate it, I think we should just document it and move on. And I guess if we do that, we should allow it for Fractions, too.
skirpichev commentedon Jun 4, 2025
Does make sense. But,
maybe we can leave things as is here? This will complicate things for alternative implementations for no good reasons. (I also guess that this "feature" was introduced unintentionally.) It seems, PyPy3.11 doesn't support it:
Python 3.11.11 (0253c85bf5f8, Feb 26 2025, 10:42:42) [PyPy 7.3.19 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)] on linux Type "help", "copyright", "credits" or "license" for more information. >>>> f"{float('123'):.١١f}" Traceback (most recent call last): File "<python-input-0>", line 1, in <module> f"{float('123'):.١١f}" ValueError: no precision givenserhiy-storchaka commentedon Jun 4, 2025
Accepting non-ASCII digits creates security risks, because some non-ASCII digits look like ASCII digits with different value.
Note that while non-ASCII digits are accepted in string to number conversion, they are not accepted in Python numerical literals.
Support of non-ASCII digits in regular expressions was deprecated in 3.11 and removed in 3.12 (see #91760).
We are currently in process of making
strptime()behavior more strict and consistent -- non-ASCII digits will only be allowed in locales that use them and only for fields for which they are used.See also https://peps.python.org/pep-0672/#confusable-digits . cc @encukou
encukou commentedon Jun 4, 2025
I'd be +1 for deprecating these everywhere, including the
intconstructor, with a long deprecation period. But I can't commit the time to push the PEP through.We can't support all numeral systems anyway. From that point of view, supporting ones that use decimal digits is a rather arbitrary choice.
Things like
int('੪٦', 20)are just nonsense.ASCII only is consistent, predictable, and ultimately more secure.
But I'm -1 for only deprecating this in less-important places, like formatting fields. That just feels like a way to avoid the discussion. If we, as the CPython project, have an opinion on this, it should be clear and consistent.
For documentation, I think this should be pointed out as a CPython implementation detail. Other implementations should be free to not support it, if they don't mind the incompatibility.
skirpichev commentedon Jun 10, 2025
Does this require a PEP if we deprecate just very odd cases?
Like that:
Agreed. Lets reject non-ascii strings for base!=10.
Why? Like the deprecated behavior in regular expressions, this was rather introduced as an accident. And it not supported by some Python implementations. Lets just document that as a CPython implementation detail and then deprecate.
BTW, there is an old issue for documentation of string parsing by numerical types: #54790
serhiy-storchaka commentedon Jun 10, 2025
Please do not confuse two different things. Accepting non-ASCII digits in user data is one thing, and accepting them in format specifiers is other thing. With f-strings, the latter is close to code than to user data. Non-ASCII digits are not supported in Python numerical literals, even if they are supported in Python identifiers.
There is a use for ASCII-only string-to-number conversions, because some protocols only support ASCII digits, and simply using int() or float() can be inadequately lenient. For now, the solution is to use the
str.isascii()check (this was the reason to introducestr.isascii()at first place). In future we may addascii=Trueorscript='ascii'options. But this is very different issue.encukou commentedon Jun 10, 2025
It would be great to get to two different things, but here we have 3+ different things: what
intaccepts, what the parser accepts as literals,[0-9]+. There are more variations inmatchpatterns,re,str.formatindices...Do we have a good idea what all these should be?
skirpichev commentedon Jun 19, 2025
Ok, what's a plan? Should someone open a d.p.o thread to discuss first the more wide picture?
ericvsmith commentedon Jun 22, 2025
While what f-strings accept is in fact code and not user input, I always consider what
str.formataccepts as possibly user-provided, such as in translations or what-not. But they're processed the same: they're handled by the object's__format__method. So the line isn't quite so clear.vstinner commentedon Jun 24, 2025
I'm in favor of deprecating non-ASCII digits in format strings, but I would prefer to leave
int(value)constructor unchanged even if there are weird cases with bases other than 10.skirpichev commentedon Oct 27, 2025
@encukou, are you still against deprecating unicode digits just for formatting?
If so, I'll provide patch to allow them for Fraction's, but leaving this stuff undocumented. In this way at least we will have less incompatibilities across different stdlib number types.
Maybe someone else will someday bring this discussion on wider audience.
pythongh-135025: allow unicode in floating-point format strings for F…
encukou commentedon Oct 27, 2025
I still think we should have a clear idea of the whole picture before making changes.
Distinguishing “user data” and “code” seems reasonable to me, even though the line is not always clear. Deprecating non-ASCII digits for “code” sounds good, but IMO we should first agree that this is the intended direction. Otherwise this feels like changing one arbitrary behaviour to better match another -- which is IMO worse than keeping the status quo.
[-]Should we support unicode in width/precision formatting fields?[/-][+]fractions: Should we support unicode in width/precision formatting fields?[/+]ericvsmith commentedon Oct 27, 2025
I'd like to deprecate non-ascii chars. But as @encukou points out, it's good to think about user data vs. code. Unfortunately for
__format__, the format spec can be specified as user data (forstr.format, at least). This would typically be through mechanisms such as config files or translation utilities. Is it worth saying that such translations are "code"? I'm not sure.serhiy-storchaka commentedon Oct 27, 2025
Format strings is a mix of user data and code. They can be created by non-programmers, but the repleceable parts is the code. It makes sense to use non-ascii digits in the literal parts, but the argument indices and format strings are not shown to the end user, they are interpreted as code, and there are no more reasons to support non-ascii digits there than in the Python code.