Repository navigation
Decimal to int conversion with engineering notation costly despite failing #139957
Description
Activity
cc @serhiy-storchaka @tim-one I'm not sure we have a specialization in the "parser" but do you think it could make sense?
Note that the conversion from
Decimalsucceeds. It's the attempt to print the resultingintat the prompt that fails.Python 3.13.7 (main, Aug 15 2025, 23:01:14) [Clang 17.0.0 (clang-1700.0.13.5)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from decimal import Decimal >>> x = Decimal('1e10000') >>> i = int(x) # succeeds >>> i # fails Traceback (most recent call last): File "<python-input-3>", line 1, in <module> i ValueError: Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit
So this doesn't really have anything to do with
Decimal. It's no different from trying to print10**10000directly. OrFraction("1e10000"), for that matter.>>> from fractions import Fraction >>> f = Fraction('1e10000') >>> i = int(f) # succeeds >>> i # fails Traceback (most recent call last): File "<python-input-7>", line 1, in <module> i # fails ^ ValueError: Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit
Reacted by Sergey B KirpichevReacted by Tim PetersRight. I always forget that we have an implicit
repr(x)in the REPL. So, probably nothing to do here?Although note that it's still not all unicorns and rainbows 😉.
>>> x = Decimal('1e1000000') # add more zeroes >>> i = int(x) # succeeds, but takes "a long time"
This is a consequence of that Decimal<->int conversions are still quadratic time. A report on that was open for years, but not long ago closed as "not planned". It would be best if (the external)
libmpdecdid the heavy lifting to use a sub-quadratic approach, but short of that nobody has been both willing and able to put "the smarts" into CPython's wrapping of such conversions.Reacted by Stan UlbrychThis returns almost instantly on my computer.
$ ./python -m timeit -n1 -r1 -s "from decimal import Decimal; x = Decimal('1e1_000_000')" "int(x)" 1 loop, best of 1: 262 msec per loopAdding more zeroes can making it taking "a long time".
$ ./python -m timeit -n1 -r1 -s "from decimal import Decimal; x = Decimal('1e10_000_000')" "int(x)" 1 loop, best of 1: 10.4 sec per loopString to integer conversion takes comparable time
$ ./python -X int_max_str_digits=0 -m timeit -n1 -r1 -s "from decimal import Decimal; x = '1'+'0'*10_000_000" "int(x)" 1 loop, best of 1: 13.6 sec per loopso I do not think that we can optimize Decimal to integer conversion by utilizing Decimal to string to integer conversion.
@serhiy-storchaka, believe it or not, 10 million digits is still on the small side(!) for
O()behavior of str->int to become apparent. Increase it to 100 million digits, and you should find that it takes a lot closer to 10 times (sub-quadratic) longer than 100 times .(quadratic). On my box, about 12 times longer.$ py -X int_max_str_digits=0 -m timeit -n1 -r1 -s "x = '1'+'0'*10_000_000" "int(x)" 1 loop, best of 1: 10.1 sec per loop $ py -X int_max_str_digits=0 -m timeit -n1 -r1 -s "x = '1'+'0'*100_000_000" "int(x)" 1 loop, best of 1: 119 sec per loopIronically enough, CPython's str->int uses
decimalinternally to exploit its advanced (quasi-linear) multiplication.@serhiy-storchaka Those
Decimaltimings are impressive. I get very different performance results on my M4 Pro laptop, off by a couple of orders of magnitude (60 times to 150 times slower than on your machine).mdickinson@MacBook-Pro heron % python -m timeit -n1 -r1 -s "from decimal import Decimal; x = Decimal('1e1_000_000')" "int(x)" 1 loop, best of 1: 15.4 sec per loop mdickinson@MacBook-Pro heron % python -m timeit -n1 -r1 -s "from decimal import Decimal; x = Decimal('1e10_000_000')" "int(x)" 1 loop, best of 1: 1.53e+03 sec per loopBut str-to-int conversion is faster:
mdickinson@MacBook-Pro heron % python -X int_max_str_digits=0 -m timeit -n1 -r1 -s "from decimal import Decimal; x = '1'+'0'*1_000_000" "int(x)" 1 loop, best of 1: 155 msec per loop mdickinson@MacBook-Pro heron % python -X int_max_str_digits=0 -m timeit -n1 -r1 -s "from decimal import Decimal; x = '1'+'0'*10_000_000" "int(x)" 1 loop, best of 1: 6.4 sec per loopI'm not sure how to account for the extreme difference in timings for the
Decimalconversions.mdickinson@MacBook-Pro heron % python -VV Python 3.13.8 (main, Oct 11 2025, 01:55:48) [Clang 17.0.0 (clang-1700.0.13.5)]It seems that libmpdec is not provided on Ubuntu, so I tested pure Python implementation.
Reacted by Mark Dickinson and Tim PetersFrom CPython CI:
# Workaround missing libmpdec-dev on ubuntu 24.04: # https://launchpad.net/~ondrej/+archive/ubuntu/php # https://deb.sury.org/ sudo add-apt-repository ppa:ondrej/php apt-get update apt-get -yq install libmpdec-devI would prefer not to use untrusted PPAs.
It seems that libmpdec is not provided on Ubuntu, so I tested pure Python implementation.
Ah - missed that. Ironically, the pure P:ython implementation can do Decimal->int much faster than when using libmpdec. In this case, offhand it looks like the pure Python run is timing mostly CPython's
10 ** someintspeed, which benefits at least from sub-quadratic Karatsuba multiplication. There's no reason for why libmpdec couldn't do even better than that. But it doesn't now.As mentioned, when libmpdec is available, Python's own str->int does exploit decimal's faster-than-Karatsuba multiplication.
Bug report
Bug description:
Adding many digits to engineering notation causes cpython to work very hard trying to do the conversion, only for it to fail anyway.
But the time it takes goes up when you add extra digits. It appears to be unbounded.
The decimal to int conversion should shortcut to a failure when there is no possibility of generating a real number, e.g. by examining the length of the substring after + or - (in conjunction with sys.set_int_max_str_digits or maybe some other setting).
CPython versions tested on:
3.11, 3.12
Operating systems tested on:
macOS