Repository navigation
test_embed fails if Python is built with LTO and LLVM clang on macOS #110313
Description
Bug report
Bug description:
# Next, run the profile task to generate the profile information.
LLVM_PROFILE_FILE="code-%p.profclangr" ./python.exe -m test --pgo --timeout=
Raised RLIMIT_NOFILE: 256 -> 1024
0:00:00 load avg: 3.49 Run 44 tests sequentially
0:00:00 load avg: 3.49 [ 1/44] test_array
0:00:01 load avg: 3.49 [ 2/44] test_base64
0:00:01 load avg: 3.49 [ 3/44] test_binascii -- test_base64 failed (env changed)
0:00:01 load avg: 3.49 [ 4/44] test_binop
0:00:01 load avg: 3.49 [ 5/44] test_bisect
0:00:01 load avg: 3.49 [ 6/44] test_bytes
0:00:05 load avg: 3.37 [ 7/44] test_bz2 -- test_bytes failed (env changed)
0:00:07 load avg: 3.37 [ 8/44] test_cmath
0:00:07 load avg: 3.34 [ 9/44] test_codecs
0:00:09 load avg: 3.34 [10/44] test_collections
0:00:10 load avg: 3.34 [11/44] test_complex
0:00:10 load avg: 3.34 [12/44] test_dataclasses
0:00:11 load avg: 3.34 [13/44] test_datetime
0:00:16 load avg: 3.39 [14/44] test_decimal
-------------------------------------------------------- NOTICE --------------------------------------------------------
test_decimal may generate "malloc can't allocate region"
warnings on macOS systems. This behavior is known. Do not
report a bug unless tests are also failing.
See https://github.com/python/cpython/issues/85100
------------------------------------------------------------------------------------------------------------------------
./Modules/_decimal/libmpdec/context.c:57: warning: mpd_setminalloc: ignoring request to set MPD_MINALLOC a second time
python.exe(67804,0x7fffa28f43c0) malloc: *** mach_vm_map(size=842105263157895168) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
python.exe(67804,0x7fffa28f43c0) malloc: *** mach_vm_map(size=842105263157895168) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
python.exe(67804,0x7fffa28f43c0) malloc: *** mach_vm_map(size=421052631578947584) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
python.exe(67804,0x7fffa28f43c0) malloc: *** mach_vm_map(size=421052631578947584) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
0:00:22 load avg: 3.28 [15/44] test_difflib
0:00:23 load avg: 3.10 [16/44] test_embed
test test_embed failed
0:00:28 load avg: 3.17 [17/44] test_float -- test_embed failed (41 failures)
0:00:28 load avg: 3.17 [18/44] test_fstring
0:00:31 load avg: 3.17 [19/44] test_functools
./Modules/_decimal/libmpdec/context.c:57: warning: mpd_setminalloc: ignoring request to set MPD_MINALLOC a second time
0:00:31 load avg: 3.17 [20/44] test_generators
0:00:32 load avg: 3.17 [21/44] test_hashlib
0:00:33 load avg: 3.08 [22/44] test_heapq
0:00:34 load avg: 3.08 [23/44] test_int
0:00:34 load avg: 3.08 [24/44] test_itertools
0:00:38 load avg: 2.99 [25/44] test_json
0:00:41 load avg: 2.99 [26/44] test_long -- test_json failed (env changed)
0:00:44 load avg: 2.83 [27/44] test_lzma
0:00:45 load avg: 2.83 [28/44] test_math
0:00:48 load avg: 2.84 [29/44] test_memoryview
0:00:49 load avg: 2.84 [30/44] test_operator
0:00:49 load avg: 2.84 [31/44] test_ordered_dict
0:00:51 load avg: 2.84 [32/44] test_patma
0:00:51 load avg: 2.84 [33/44] test_pickle
0:00:58 load avg: 2.63 [34/44] test_pprint
0:00:58 load avg: 2.63 [35/44] test_re
0:01:00 load avg: 2.63 [36/44] test_set -- test_re failed (env changed)
0:01:05 load avg: 2.50 [37/44] test_sqlite3
0:01:06 load avg: 2.50 [38/44] test_statistics -- test_sqlite3 failed (env changed)
0:01:08 load avg: 2.38 [39/44] test_str
0:01:11 load avg: 2.38 [40/44] test_struct -- test_str failed (env changed)
0:01:12 load avg: 2.38 [41/44] test_tabnanny -- test_struct failed (env changed)
0:01:13 load avg: 2.51 [42/44] test_time -- test_tabnanny failed (env changed)
0:01:15 load avg: 2.51 [43/44] test_xml_etree
0:01:16 load avg: 2.51 [44/44] test_xml_etree_c
Total duration: 1 min 18 sec
Total tests: run=8,985 failures=41 skipped=192
Total test files: run=44/44 failed=1 env_changed=8
Result: FAILURE
make: *** [Makefile:800: profile-run-stamp] Error 2
This is on macOS 10.12.6, building with LLVM / Clang 17.0.1. The system otherwise has Python 3.11.6 installed.
CPython versions tested on:
CPython main branch
Operating systems tested on:
macOS
Activity
debohman commentedon Oct 3, 2023
debohman commentedon Oct 3, 2023
Building Python 3.12.0 on the same system with the same toolchain results in the following:
I assume that this is not a fatal error?
mdboom commentedon Oct 9, 2023
We are running into this bug on the CPython benchmarking infrastructure, too.
git bisectsays the first bad commit is 6ab6040.vstinner commentedon Oct 9, 2023
Does test_decimal fail if you run it alone? How did you built Python?
debohman commentedon Oct 9, 2023
Here is the result of running
test_embedby hand:Details
debohman commentedon Oct 9, 2023
And
test_decimal:Details
debohman commentedon Oct 9, 2023
To be clear: this is top of tree
main.debohman commentedon Oct 9, 2023
To answer the question about how I am building Python:
debohman commentedon Oct 10, 2023
It is not clear why
_PyExc_ValueErroris not found when loading_opcode.cpython-313-darwin.so. The symbol is present inpython.exe:debohman commentedon Oct 10, 2023
Something must have changed in the runtime module loading in
main.vstinner commentedon Oct 10, 2023
SyntaxError: (unicode error) \N escapes not supported (can't load unicodedata module)It's syntax that Python cannot locate the unicodedata shared library. Can you import it if you run Python manually? Was it built?
vstinner commentedon Oct 10, 2023
Can the issue be reproduced on other platforms than macOS?
debohman commentedon Oct 10, 2023
debohman commentedon Oct 10, 2023
Perhaps @mdboom can comment about what platform he is using in his comment above.
38 remaining items
debohman commentedon Oct 11, 2023
The stock linker is being used, the rest of the toolchain is the newer llvm / clang.
mdboom commentedon Oct 11, 2023
It's macOS 13.6
I'm building CPython as follows:
mdboom commentedon Oct 11, 2023
I can confirm that backing out 3e3a7da also resolves things for me.
pythongh-110313: Revert "pythongh-90108: Disable LTO on _freeze_modul…
vstinner commentedon Oct 11, 2023
I wrote PR #110720 to revert my "NoLTO" change which introduced the bug.
I don't understand exactly why/how Programs/_testembed is miscompiled on macOS using LLVM 16 or 17, but it's bad: we should not miscompile Python. My change was just to optimize "build Python" command faster (run
make), it should not miscompile Python.It would be nice if someone can dig into the issue, but I don't have the bandwidth for that. I cannot reproduce the issue on Linux, and I don't have access to any macOS machine.
gh-110313: Revert "gh-90108: Disable LTO on _freeze_module and _teste… (
[-]test_embed fails if Python is built with LTO=full and LLVM clang 17 on macOS[/-][+]test_embed fails if Python is built with LTO and LLVM clang on macOS[/+]debohman commentedon Oct 11, 2023
If I may hazard a guess as to why this is failing, we seem to be using objects built with LTO and linking them to create
Programs/_testembedwithout LTO. Perhaps if you wish to to build_testembedwithout LTO, then you need a set of object files built without any LTO. Note that everything works fine when building Python without any LTO.In the case which fails, the generated
_testembedcontains no symbols, so when it tries to runtime load a module, it fails because it cannot resolve the undefined symbols in the module.vstinner commentedon Oct 12, 2023
I suppose that the best way to build with LTO only once instead of 3 times (freeze, testembed, libpython) is to use
./configure --enable-shared. So most object files are only linked once into libpython, and then libpython is reused in freeze and testembed.If I recall correctly, GCC implementation of LTO is very different and so the NOLTO hack to reduce build time when enable-shared is not used still works. Maybe we just use -flto=full in NOLTO on macOS (when cland is used).
debohman commentedon Nov 6, 2023
Shall we close this issue? I have successfully built
3.13.0a1and currentmainwith both PGO and LTO.debohman commentedon Nov 6, 2023
I am also using the latest Clang
17.0.4.vstinner commentedon Nov 6, 2023
Ok, I close the issue.
pythongh-110313: Revert "pythongh-90108: Disable LTO on _freeze_modul…
Abhaygarg656 commentedon Oct 14, 2024