Repository navigation
Read and write of __class__ in two threads causes crash #120198
Description
Activity
Is this a free threaded build?
Is this a free threaded build?
Both builds (default and free-threaded).
Reacted by Kirill PodoprigoraFWIW I cannot repro locally on my own machine, but it crashes the CI.
FWIW I cannot repro locally on my own machine, but it crashes the CI.
If I run this code, no crash occurs:
import threading NTHREADS = 6 def test___class___modification(): class Foo: pass class Bar: pass thing = Foo() def work(): foo = thing for _ in range(10000): foo.__class__ = Bar type(foo) foo.__class__ = Foo type(foo) threads = [] for i in range(NTHREADS): thread = threading.Thread(target=work) thread.start() threads.append(thread) for thread in threads: thread.join() test___class___modification()
But if I apply your changes from the #120195 and run
test_free_threading.test_typeit will crash:./python.exe -m test -v test_free_threading -m test_type == CPython 3.14.0a0 (heads/main:6b606522ca, Jun 7 2024, 08:22:12) [Clang 15.0.0 (clang-1500.3.9.4)] == macOS-14.5-arm64-arm-64bit-Mach-O little-endian == Python build: debug == cwd: /Users/admin/Projects/cpython/build/test_python_worker_71431æ == CPU count: 8 == encodings: locale=UTF-8 FS=utf-8 == resources: all test resources are disabled, use -u option to unskip tests Using random seed: 3124489832 0:00:00 load avg: 3.31 Run 1 test sequentially in a single process 0:00:00 load avg: 3.31 [1/1] test_free_threading test___class___modification (test.test_free_threading.test_type.TestType.test___class___modification) ... ok test_attr_cache (test.test_free_threading.test_type.TestType.test_attr_cache) ... ok test_attr_cache_consistency (test.test_free_threading.test_type.TestType.test_attr_cache_consistency) ... ok test_attr_cache_consistency_subclass (test.test_free_threading.test_type.TestType.test_attr_cache_consistency_subclass) ... ok ---------------------------------------------------------------------- Ran 4 tests in 10.804s OK Python/gc.c:94: gc_decref: Assertion "gc_get_refs(g) > 0" failed: refcount is too small Enable tracemalloc to get the memory block allocation traceback object address : 0x133073a30 object refcount : 1 object type : 0x10323a798 object type name: type object repr : <class 'test.test_free_threading.test_type.TestType.test___class___modification.<locals>.Bar'> Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed Python runtime state: initialized Current thread 0x00000001f61d0c00 (most recent call first): Garbage-collecting File "/Users/admin/Projects/cpython/Lib/test/support/__init__.py", line 814 in gc_collect File "/Users/admin/Projects/cpython/Lib/test/libregrtest/single.py", line 144 in _load_run_test File "/Users/admin/Projects/cpython/Lib/test/libregrtest/single.py", line 181 in _runtest_env_changed_exc File "/Users/admin/Projects/cpython/Lib/test/libregrtest/single.py", line 281 in _runtest File "/Users/admin/Projects/cpython/Lib/test/libregrtest/single.py", line 310 in run_single_test File "/Users/admin/Projects/cpython/Lib/test/libregrtest/main.py", line 363 in run_test File "/Users/admin/Projects/cpython/Lib/test/libregrtest/main.py", line 397 in run_tests_sequentially File "/Users/admin/Projects/cpython/Lib/test/libregrtest/main.py", line 541 in _run_tests File "/Users/admin/Projects/cpython/Lib/test/libregrtest/main.py", line 576 in run_tests File "/Users/admin/Projects/cpython/Lib/test/libregrtest/main.py", line 739 in main File "/Users/admin/Projects/cpython/Lib/test/libregrtest/main.py", line 747 in main File "/Users/admin/Projects/cpython/Lib/test/__main__.py", line 2 in <module> File "/Users/admin/Projects/cpython/Lib/runpy.py", line 88 in _run_code File "/Users/admin/Projects/cpython/Lib/runpy.py", line 198 in _run_module_as_main Extension modules: _testinternalcapi, _testcapi (total: 2) zsh: abort ./python.exe -m test -v test_free_threading -m test_type
Oh yeah I get it only in the test too. Thanks!
Well I'm not sure how to fix this. So it's up for grabs for anyone to take!
I can reproduce on Linux/x86_64, would you mind I take a try?
Reacted by Ken Jin@Zheaoli Yeah sure go ahead!
Seems the bugs root cause here https://github.com/python/cpython/blob/main/Lib/test/libregrtest/setup.py#L73-L77
Can reproduced by the code following below
import threading import sys NTHREADS = 20 def _test_audit_hook(name, args): # print(name, args) pass sys.addaudithook(_test_audit_hook) class A: @classmethod def test___class___modification(self): class Foo: pass class Bar: pass thing = Foo() def work(): foo = thing for _ in range(100000): foo.__class__ = Bar type(foo) foo.__class__ = Foo type(foo) threads = [] for i in range(NTHREADS): thread = threading.Thread(target=work) thread.start() threads.append(thread) for thread in threads: thread.join() A.test___class___modification()
Let me find more detail about the race condition here
Reacted by Ken JinThis bug is also in Python 3.9 ~ now
Update status:
I may need some help here.
For now, I have observed some interesting detail
- The crash will disappear if we add extra code like(print) in the hook function
- Comment https://github.com/python/cpython/blob/main/Python/sysmodule.c#L282 this line, and the crash will disappear
- the crash will disappear too if I use watchpoint in the gdb to watch the object ob_refcnt filed changed event.....
- This bug exist in origin code for PEP 578
I think this is a race condition issue for the sys_audit. But I may need some help to dive deeper into this issue...
Reacted by Weizhen Wang, Ken Jin and Kirill PodoprigoraFTR, I got a crash with a
print(1)in the audit hook and only 1000 runs and 4 threads (not a constant crash but still with the same error). I'll investigate on my side.@Zheaoli it might not be a threading issue. I wrapped the function you sent with
_PyEval_StopTheWorld(is);_PyEval_StartTheWorld(is);and it still crashes. Usually that's strong enough to prevent any races, so I'm thinking the threads might be exposing something else.BTW, I checked the
object_set_classcode and couldn't find any issues there. This is where it's calling the audit hooks.. If I comment that out, it passes too. So this suggests there's probably indeed an issue with audit.Line 6540 in 34f5ae6
So this suggests there's probably indeed an issue with audit.
Yep, https://github.com/python/cpython/blob/main/Python/sysmodule.c#L282 the crash because of here...
Reacted by Ken Jin3 remaining items
- added a commit that references this issue
on Jun 11, 2024 It seems that this started a reference leak on the 3.13 buildbots, see e.g. https://buildbot.python.org/all/#/builders/1396/builds/138
- added a commit that references this issue
on Jun 12, 2024 I only see a test failure on a single macOS nogil buildbot, and all the rest are fine? And I'd expect this new test to expose threading issues, because that's what it's for.
The fix has certainly not introduced a reference leak, so if anything is going to get reverted, it should just be the test.
I just fixed the problem on free threading, so the problem is gone.
The refleak buildbot was failing not because of an actual refleak but it ran the test long enough to always segfault. So I fixed it on the free threading buildbot as well.
- added a commit that references this issue
on Jul 12, 2024
Crash report
What happened?
See #120195 for crashing CI.
Apply diff from there and run
CPython versions tested on:
CPython main branch
Operating systems tested on:
No response
Output from running 'python -VV' on the command line:
No response
Linked PRs