Repository navigation
Raise ValueError rather of OverflowError in PyLong_AsUnsignedLong() #74020
Description
| BPO | 29834 |
|---|---|
| Nosy | @rhettinger, @serhiy-storchaka, @orenmn |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2017-03-17.08:44:59.699>
labels = ['interpreter-core', 'type-feature', '3.7']
title = 'Raise ValueError rather of OverflowError in PyLong_AsUnsignedLong()'
updated_at = <Date 2017-03-18.09:01:27.125>
user = 'https://github.com/serhiy-storchaka'bugs.python.org fields:
activity = <Date 2017-03-18.09:01:27.125>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2017-03-17.08:44:59.699>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 29834
keywords = []
message_count = 5.0
messages = ['289748', '289752', '289796', '289797', '289798']
nosy_count = 3.0
nosy_names = ['rhettinger', 'serhiy.storchaka', 'Oren Milman']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue29834'
versions = ['Python 3.7']
Activity
serhiy-storchaka commentedon Mar 17, 2017
OverflowError is raised when Python integer doesn't fit in C integer type due to platform limitations. Different platforms have different limits. But in PyLong_AsUnsignedLong() only the upper limit is platform-depended. Negative integers always are not accepted. PyLong_AsUnsignedLong() is used for values that can be only non-negative. I think that ValueError is more appropriate in this case than OverflowError.
orenmn commentedon Mar 17, 2017
note that there are functions that rely on the fact that
PyLong_AsUnsignedLong currently raises OverflowError for both cases.
such functions would probably use
PyErr_ExceptionMatches(PyExc_OverflowError)
or something like
PyErr_GivenExceptionMatches(err, PyExc_OverflowError)
_Py_Uid_Converter() (in Modules/posixmodule.c) is an example, but
ISTM there aren't many such functions.
however, this is only in cpython's C code. I don't know how many
functions in cpython's python code rely on this.
also, maybe there is a lot of user code that rely on this?
rhettinger commentedon Mar 18, 2017
Strong -1 on this. For zero benefit, this breaks everything (within Python and third-party code) that ever relied on the documented behavior , https://docs.python.org/3/c-api/long.html#c.PyLong_AsUnsignedLong .
The cause perfectly fits the definition of an OverflowError:
The time to challenge API design decisions is when they are created, not after they've been published and relied upon for over decade.
Also, we really don't everyone who writes cross-platform code and tests to have to catch both exceptions because they don't know which version is being run. This creates yet another barrier to upgrading Python and as far as I can tell isn't solving any reported user problem. Instead, it is second-guessing design decisions made long ago.
rhettinger commentedon Mar 18, 2017
Looking back in time, this API isn't as old as I thought. The other concerns about breaking a published API still stand.
serhiy-storchaka commentedon Mar 18, 2017
_Py_Uid_Converter() in Modules/posixmodule.c is not an example because it calls PyLong_AsUnsignedLong() only for positive integers > LONG_MAX.
PyLong_AsUnsignedLong() is used not much. The motivation of this issue was that if use PyLong_AsUnsignedLong() for converting non-negative by its nature values ValueError can be more appropriate since this limitation is not platform or implementation dependent. Strictly speaking raising OverflowError for negative values doesn't fits the definition of an OverflowError, since the result is not large at all.
I was going to investigate all usages of PyLong_AsUnsignedLong() and if in majority of them ValueError is appropriate and desirable, changing the exception type at that level can make the implementation simpler.
serhiy-storchaka commentedon Aug 29, 2023
Argument Clinic converters
unsigned_char(bitwise=False),unsigned_short(bitwise=False),unsigned_int(bitwise=False),unsigned_long(bitwise=False),unsigned_long_long(bitwise=False), andsize_traise ValueError for negative integer. It is impossible to implement them using the limited C API.Many code converts Python integer to signed C type and then raise ValueError if it is negative (see for example handling
maxlenindeque()). So, negative value can cause either OverflowError or ValueError, depending on platform.serhiy-storchaka commentedon Jun 27, 2024
New C API
PyLong_AsNativeBytes()raises ValueError if the input is negative and thePy_ASNATIVEBYTES_REJECT_NEGATIVEflag was set. As well as the derived C APIPyLong_ToUInt64()etc.pythongh-74020: Raise ValueError for negative values converted to uns…
serhiy-storchaka commentedon Apr 28, 2025
See also #132987. Since many helper functions raise ValueError instead of OverflowError for negative values, adding support for the special
__index__()method often results in this change. The further along, the more such changes will occur, and many of them will remain unnoticed if there were no specific tests.skirpichev commentedon Apr 29, 2025
I'm with @rhettinger here that changing this will be a huge compatibility break. I suggest to do this with a deprecation period.
But the
ValueErrorseems to be more acceptable here and allows more fine-tuned error reporting.On another hand, changing new API (introduced in 3.13 or even in 3.14) should be considered too. The
OverflowErrorisn't too bad in this context.serhiy-storchaka commentedon Apr 29, 2025
What is is a meaning of it, technically?
skirpichev commentedon Apr 29, 2025
We will warn people first, instead of just breaking code. I.e. now people are not expect
ValueErrorfromint_to_bytes()(with correct arguments).serhiy-storchaka commentedon Apr 29, 2025
How can we do this?
skirpichev commentedon Apr 29, 2025
Emit a warning together with raising an
OverflowErrorerror. Then - after a deprecation period - change error type.serhiy-storchaka commentedon Apr 30, 2025
That would create more problems. For example, look in
Modules/_struct.c. The code replacesOverflowError(andValueError) withstruct.error. Now the user will get a warning aboutOverflowError/ValueErrorwith raisingstruct.error-- instead of completely invisible change.picnixz commentedon Apr 30, 2025
Maybe have an exception class inheriting from ValueError and OverflowError?
skirpichev commentedon Apr 30, 2025
Yes, I see. The problem is that C-API has no functions to filter out warnings. Maybe it should?
Any suggestion on naming?
picnixz commentedon Apr 30, 2025
I'm tempted of having a DomainError but I fear it'll clash with existing ones (the name is likely not that rare but I may be mistaken). Also it wouldn't solve the issue when someone handles separately ValueError and OverflowError
erlend-aasland commentedon May 11, 2025
There's also the possibility of just deprecating
OverflowErrorin What's New/NEWS, and then changing it after x years.