Repository navigation
Enable -fstrict-overflow #96821
Description
At the moment we compile releases with -fwrapv which makes the code a bit safer, but disables certain optimizations. From the GCC docs:
This option instructs the compiler to assume that signed arithmetic overflow of addition, subtraction and multiplication wraps around using twos-complement representation. This flag enables some optimizations and disables others.
My experiments with running sanitisers seem to suggest that we are nearly already ready for -fno-wrapv (or -fstrict-overflow in general). Doing so could lead to quite a few speedups, but we would need to be more careful with the code we write.
It might be worthwhile to get a few benchmarks.
(To be extra precise, we give -fwrapv for clang and gcc for any build that doesn't get --with-pydebug.)
Pitch
My plan right now is to adapt the build system so that only the modules that need it are build with -fwrapv, and the rest can be build with -fstrict-overflow.
We already have config machinery that can add specific CFLAGS for specific modules only.
Perhaps the whole thing can be gated behind a configure flag, like --with-strict-overflow.
If everything goes well, and this improves performance we can consider adding this functionality to one of the standard optimization options.
We can also work on making more modules -fstrict-overflow safe.
Previous discussion
@markshannon @ericsnowcurrently
Brought up on faster-cpython/ideas#458 and inspired by #96678
Some previous issues around -fwrapv:
- https://bugs.python.org/issue11149
- https://bugs.python.org/issue1621
- https://bugs.python.org/issue1608
I'm sure there are more.
Progress so far
As far as is currently known, the three remaining modules that rely on defined integer overflow are fixed by:
_struct: gh-96735: Fix undefined behaviour in struct unpacking functions #96739audioop: gh-96821: Fix undefined behaviour inaudioop.c#96923_ctypes: gh-96821: Fix undefined behaviour in_ctypes/cfield.c#96925
Activity
pythongh-96821: Mark modules that need -fno-strict-overflow
ericsnowcurrently commentedon Sep 15, 2022
This is definitely worth exploring!
ericsnowcurrently commentedon Sep 16, 2022
What do you mean by that?
More concretely:
ericsnowcurrently commentedon Sep 16, 2022
Also, do clang and msvc have something similar?
ericsnowcurrently commentedon Sep 16, 2022
In gh-96823 I asked about detecting compatibility for the compile flag. This is probably the better place to discuss that.
Basically, how can we detect that a C file can be built with
-fstrict-overflowor that it can't?matthiasgoergens commentedon Sep 16, 2022
I ran clang's undefined behaviour sanitiser and address sanitiser over the test suite to detect the modules that need defined overflow.
Of course, strictly speaking a detection this way can only show the presence of the need for defined overflow, but it cannot show the absence of such a need.
But that's no different from any other bug or undefined behaviour, and we just rely on the additional assumption that our test suite has enough coverage.
Small complicating detail: when you use the conpiler flag to make signed integer overflow into defined behaviour, the sanitisers no longer complain about it.
So if you want to check if the flag is still necessary for a module, you need to temporarily turn it off.
matthiasgoergens commentedon Sep 16, 2022
Clang definitely has this flag. We actually use clang for the sanitisers.
I don't know about msvc. There's no trace of anything like this in their docs https://docs.microsoft.com/en-us/cpp/build/reference/compiler-options-listed-by-category?view=msvc-170#code-generation
But we can just stick to the current situation here:
Only gcc and clang explicitly get flags about overflow, and for the other compilers we just hope and pray.
(The situation will actually improve for the other compilers, when we make all modules strict-overflow safe.)
For the actual implementation, I'll probably make
configurecheck for the flags directly, instead of checking for the name and version of the compiler.matthiasgoergens commentedon Sep 17, 2022
I had a look at PEP 7. The relevant section is https://peps.python.org/pep-0007/#c-dialect
The PEP says to use C11 and doesn't mention any deviation from that for overflow. So our code should already avoid signed integer overflow. At this point that's more of an aspiration than reality, though.
So to some extent, we could describe the effort here as part of implementing PEP 7.
Strictly speaking we don't need to change anything, but I think pragmatically we should make a note that up to 3.11 signed integer overflow was tacitly tolerated, but that beginning from 3.12 we are sticking to the standard. I created a draft PR python/peps#2796 for how that could look like. (Please suggest better wording, if you have ideas.)
(Additionally, I also notice that when I am building I am getting a few warnings here or there, but the PEP suggests that we shouldn't be getting any warnings. Perhaps I'll spend a bit of time fixing the code (or suppressing these warnings in known instances that we can't or won't fix.))
matthiasgoergens commentedon Sep 17, 2022
Another thing: whatever policy we decide on, we should probably fix that
--with-pydebugseemingly arbitrarily gives you a different policy.matthiasgoergens commentedon Sep 17, 2022
I just found some slightly bad news in the GCC docs:
If I understand that right, specifying
-fwrapvor-fno-strict-overflowfor one C source file might essentially be enough to preclude optimizations for (almost?) everything.So we might need to be completely clean before we can expect to get a performance benefit?
That's where we want to get to eventually anyway, but it would have been nice if a piecemeal approach worked.
Update: in experiments with clang and its undefined-behaviour sanitizer, the piecemeal approach seems to work. So at least for diagnostics it's good enough! I don't know about optimizations.
matthiasgoergens commentedon Sep 17, 2022
I extended #96823 to incorporate what I wrote above.
pythongh-96821: Assert for demonstrating undefined behaviour
pythongh-96821: Fix undefined behaviour in `audioop.c`
3 remaining items
gh-96821: Fix undefined behaviour in `_testcapimodule.c` (GH-96915)
gh-96821: Fix undefined behaviour in `_testcapimodule.c` (GH-96915) (G…
gh-96821: Fix undefined behaviour in `_testcapimodule.c` (GH-96915) (G…
matthiasgoergens commentedon Sep 25, 2022
@mdickinson Could you please have a look at the other PRs mentioned in this issue?
There's only two left, and then we can turn on strict overflow for (hopefully) extra performance.
I went with implementation defined behaviour for those two PRs. But I'm happy to adopt your clever method for non-implementation defined behaviour, if you prefer that.
matthiasgoergens commentedon Sep 26, 2022
I ran pyperformance benchmarks on a branch that had all the patches applied and
-fstrict-overflowenabled. Overall, we are getting about 1% speedup on GCC:Benchmark hidden because not significant (16): scimark_sparse_mat_mult, regex_compile, sqlalchemy_imperative, html5lib, unpickle_list, pickle, chameleon, pickle_list, hexiom, genshi_xml, 2to3, richards, sympy_str, pickle_dict, unpack_sequence, sqlite_synth
gh-96821: Fix undefined behaviour in `audioop.c` (#96923)
pythongh-96821: Fix undefined behaviour in `audioop.c` (python#96923)
gh-96821: Fix undefined behaviour in `_testcapimodule.c` (GH-96915) (G…
gh-96821: Add config option `--with-strict-overflow` (#96823)
hauntsaninja commentedon Mar 4, 2023
I've fixed and merged #96823 . A follow up improvement could be to check for the compiler switch only if
--with-strict-overflowwas given, as suggested by Erlendpythongh-96821: Add config option `--with-strict-overflow` (python#96823
Merge branch 'main' into nodeaddict
Merge branch 'main' into inlinecomp2
StanFromIreland commentedon Oct 5, 2025
I wrote #139595 for this refactor.