Skip to content

Data race in set iterator length_hint under no-gil #144356

Open
Open
Data race in set iterator length_hint under no-gil#144356
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)topic-free-threadingtype-bugAn unexpected behavior, bug, or error

Description

@hyongtao-code
hyongtao-code
opened on Jan 31, 2026

Bug report

Build info

$ cd cpython-tsan/bin/
$ ./python3.15
Python 3.15.0a5+ free-threading build (heads/main:96e4cd698a, Jan 31 2026, 11:45:37) [Clang 18.1.3 (1ubuntu1)] on linux
Type "help", "copyright", "credits" or "license" for more information.

Bug description

Under Py_GIL_DISABLED, setiter_len() reads so->used non-atomically while concurrent set mutations update it atomically.

This triggers a data race detectable by TSAN when one thread repeatedly calls iterator.__length_hint__() while another mutates the same set

Python script

import threading, time

s = set(range(2000))
it = iter(s)
stop = threading.Event()

def reader():
    while not stop.is_set():
        it.__length_hint__()

def writer():
    i = 0
    while not stop.is_set():
        s.add(i)
        s.discard(i - 1)
        i += 1

threads = [threading.Thread(target=reader) for _ in range(4)]
threads.append(threading.Thread(target=writer))

for t in threads: t.start()
time.sleep(3)
stop.set()
for t in threads: t.join()

TSAN result

Details 
$ TSAN_OPTIONS="halt_on_error=1:history_size=7:report_signal_unsafe=0:verbosity=1" ./python3.15 ./tsan_setiter_len_race.py 
==1703386==WARNING: ThreadSanitizer: memory layout is incompatible, possibly due to high-entropy ASLR.
Re-execing with fixed virtual address space.
N.B. reducing ASLR entropy is preferable.
==1703386==Installed the sigaction for signal 11
==1703386==Installed the sigaction for signal 7
==1703386==Installed the sigaction for signal 8
***** Running under ThreadSanitizer v3 (pid 1703386) *****
==================
WARNING: ThreadSanitizer: data race (pid=1703386)
  Atomic write of size 8 at 0x7fffb4395ce8 by thread T5:
    #0 _Py_atomic_store_ssize_relaxed /home/hyongtao/worksapce/cpython/./Include/cpython/pyatomic_gcc.h:513:3 (python3.15+0x30378c) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #1 set_discard_entry /home/hyongtao/worksapce/cpython/Objects/setobject.c:594:5 (python3.15+0x30378c)
    #2 set_discard_key /home/hyongtao/worksapce/cpython/Objects/setobject.c:630:12 (python3.15+0x30378c)
    #3 set_discard_impl /home/hyongtao/worksapce/cpython/Objects/setobject.c:2616:10 (python3.15+0x30b3c3) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #4 set_discard /home/hyongtao/worksapce/cpython/Objects/clinic/setobject.c.h:502:20 (python3.15+0x30b3c3)
    #5 _PyEval_EvalFrameDefault /home/hyongtao/worksapce/cpython/Python/generated_cases.c.h:4033:35 (python3.15+0x42abe4) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #6 _PyEval_EvalFrame /home/hyongtao/worksapce/cpython/./Include/internal/pycore_ceval.h:118:16 (python3.15+0x41f890) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #7 _PyEval_Vector /home/hyongtao/worksapce/cpython/Python/ceval.c:2092:12 (python3.15+0x41f890)
    #8 _PyFunction_Vectorcall /home/hyongtao/worksapce/cpython/Objects/call.c (python3.15+0x20b85f) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #9 _PyObject_VectorcallTstate /home/hyongtao/worksapce/cpython/./Include/internal/pycore_call.h:136:11 (python3.15+0x20ff36) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #10 method_vectorcall /home/hyongtao/worksapce/cpython/Objects/classobject.c:73:20 (python3.15+0x20ff36)
    #11 _PyObject_VectorcallTstate /home/hyongtao/worksapce/cpython/./Include/internal/pycore_call.h:136:11 (python3.15+0x472d31) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #12 context_run /home/hyongtao/worksapce/cpython/Python/context.c:727:29 (python3.15+0x472d31)
    #13 method_vectorcall_FASTCALL_KEYWORDS /home/hyongtao/worksapce/cpython/Objects/descrobject.c:421:24 (python3.15+0x224ab7) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #14 _PyObject_VectorcallTstate /home/hyongtao/worksapce/cpython/./Include/internal/pycore_call.h:136:11 (python3.15+0x20b1e3) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #15 PyObject_Vectorcall /home/hyongtao/worksapce/cpython/Objects/call.c:327:12 (python3.15+0x20b1e3)
    #16 _Py_VectorCallInstrumentation_StackRefSteal /home/hyongtao/worksapce/cpython/Python/ceval.c:762:11 (python3.15+0x4204bc) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #17 _PyEval_EvalFrameDefault /home/hyongtao/worksapce/cpython/Python/generated_cases.c.h:1809:35 (python3.15+0x426376) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #18 _PyEval_EvalFrame /home/hyongtao/worksapce/cpython/./Include/internal/pycore_ceval.h:118:16 (python3.15+0x41f890) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #19 _PyEval_Vector /home/hyongtao/worksapce/cpython/Python/ceval.c:2092:12 (python3.15+0x41f890)
    #20 _PyFunction_Vectorcall /home/hyongtao/worksapce/cpython/Objects/call.c (python3.15+0x20b85f) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #21 _PyObject_VectorcallTstate /home/hyongtao/worksapce/cpython/./Include/internal/pycore_call.h:136:11 (python3.15+0x20ff36) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #22 method_vectorcall /home/hyongtao/worksapce/cpython/Objects/classobject.c:73:20 (python3.15+0x20ff36)
    #23 _PyVectorcall_Call /home/hyongtao/worksapce/cpython/Objects/call.c:273:16 (python3.15+0x20b4eb) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #24 _PyObject_Call /home/hyongtao/worksapce/cpython/Objects/call.c:348:16 (python3.15+0x20b4eb)
    #25 PyObject_Call /home/hyongtao/worksapce/cpython/Objects/call.c:373:12 (python3.15+0x20b555) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #26 thread_run /home/hyongtao/worksapce/cpython/./Modules/_threadmodule.c:387:21 (python3.15+0x5f12e2) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #27 pythread_wrapper /home/hyongtao/worksapce/cpython/Python/thread_pthread.h:234:5 (python3.15+0x522317) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)

  Previous read of size 8 at 0x7fffb4395ce8 by thread T3:
    #0 setiter_len /home/hyongtao/worksapce/cpython/Objects/setobject.c:1059:58 (python3.15+0x304ce9) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #1 _PyEval_EvalFrameDefault /home/hyongtao/worksapce/cpython/Python/generated_cases.c.h:3935:35 (python3.15+0x42a9d4) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #2 _PyEval_EvalFrame /home/hyongtao/worksapce/cpython/./Include/internal/pycore_ceval.h:118:16 (python3.15+0x41f890) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #3 _PyEval_Vector /home/hyongtao/worksapce/cpython/Python/ceval.c:2092:12 (python3.15+0x41f890)
    #4 _PyFunction_Vectorcall /home/hyongtao/worksapce/cpython/Objects/call.c (python3.15+0x20b85f) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #5 _PyObject_VectorcallTstate /home/hyongtao/worksapce/cpython/./Include/internal/pycore_call.h:136:11 (python3.15+0x20ff36) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #6 method_vectorcall /home/hyongtao/worksapce/cpython/Objects/classobject.c:73:20 (python3.15+0x20ff36)
    #7 _PyObject_VectorcallTstate /home/hyongtao/worksapce/cpython/./Include/internal/pycore_call.h:136:11 (python3.15+0x472d31) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #8 context_run /home/hyongtao/worksapce/cpython/Python/context.c:727:29 (python3.15+0x472d31)
    #9 method_vectorcall_FASTCALL_KEYWORDS /home/hyongtao/worksapce/cpython/Objects/descrobject.c:421:24 (python3.15+0x224ab7) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #10 _PyObject_VectorcallTstate /home/hyongtao/worksapce/cpython/./Include/internal/pycore_call.h:136:11 (python3.15+0x20b1e3) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #11 PyObject_Vectorcall /home/hyongtao/worksapce/cpython/Objects/call.c:327:12 (python3.15+0x20b1e3)
    #12 _Py_VectorCallInstrumentation_StackRefSteal /home/hyongtao/worksapce/cpython/Python/ceval.c:762:11 (python3.15+0x4204bc) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #13 _PyEval_EvalFrameDefault /home/hyongtao/worksapce/cpython/Python/generated_cases.c.h:1809:35 (python3.15+0x426376) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #14 _PyEval_EvalFrame /home/hyongtao/worksapce/cpython/./Include/internal/pycore_ceval.h:118:16 (python3.15+0x41f890) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #15 _PyEval_Vector /home/hyongtao/worksapce/cpython/Python/ceval.c:2092:12 (python3.15+0x41f890)
    #16 _PyFunction_Vectorcall /home/hyongtao/worksapce/cpython/Objects/call.c (python3.15+0x20b85f) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #17 _PyObject_VectorcallTstate /home/hyongtao/worksapce/cpython/./Include/internal/pycore_call.h:136:11 (python3.15+0x20ff36) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #18 method_vectorcall /home/hyongtao/worksapce/cpython/Objects/classobject.c:73:20 (python3.15+0x20ff36)
    #19 _PyVectorcall_Call /home/hyongtao/worksapce/cpython/Objects/call.c:273:16 (python3.15+0x20b4eb) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #20 _PyObject_Call /home/hyongtao/worksapce/cpython/Objects/call.c:348:16 (python3.15+0x20b4eb)
    #21 PyObject_Call /home/hyongtao/worksapce/cpython/Objects/call.c:373:12 (python3.15+0x20b555) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #22 thread_run /home/hyongtao/worksapce/cpython/./Modules/_threadmodule.c:387:21 (python3.15+0x5f12e2) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #23 pythread_wrapper /home/hyongtao/worksapce/cpython/Python/thread_pthread.h:234:5 (python3.15+0x522317) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)

  Thread T5 'Thread-5 (write' (tid=1703392, running) created by main thread at:
    #0 pthread_create <null> (python3.15+0xf421f) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #1 do_start_joinable_thread /home/hyongtao/worksapce/cpython/Python/thread_pthread.h:281:14 (python3.15+0x521424) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #2 PyThread_start_joinable_thread /home/hyongtao/worksapce/cpython/Python/thread_pthread.h:323:9 (python3.15+0x52123a) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #3 ThreadHandle_start /home/hyongtao/worksapce/cpython/./Modules/_threadmodule.c:474:9 (python3.15+0x5f109a) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #4 do_start_new_thread /home/hyongtao/worksapce/cpython/./Modules/_threadmodule.c:1920:9 (python3.15+0x5f0b3f) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #5 thread_PyThread_start_joinable_thread /home/hyongtao/worksapce/cpython/./Modules/_threadmodule.c:2043:14 (python3.15+0x5efc01) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #6 cfunction_call /home/hyongtao/worksapce/cpython/Objects/methodobject.c:564:18 (python3.15+0x2bab87) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #7 _PyObject_MakeTpCall /home/hyongtao/worksapce/cpython/Objects/call.c:242:18 (python3.15+0x20a68f) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #8 _PyObject_VectorcallTstate /home/hyongtao/worksapce/cpython/./Include/internal/pycore_call.h:134:16 (python3.15+0x20b2a7) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #9 PyObject_Vectorcall /home/hyongtao/worksapce/cpython/Objects/call.c:327:12 (python3.15+0x20b2a7)
    #10 _Py_VectorCall_StackRefSteal /home/hyongtao/worksapce/cpython/Python/ceval.c:720:11 (python3.15+0x41fce9) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #11 _PyEval_EvalFrameDefault /home/hyongtao/worksapce/cpython/Python/generated_cases.c.h:3387:35 (python3.15+0x429c79) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #12 _PyEval_EvalFrame /home/hyongtao/worksapce/cpython/./Include/internal/pycore_ceval.h:118:16 (python3.15+0x41f3b0) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #13 _PyEval_Vector /home/hyongtao/worksapce/cpython/Python/ceval.c:2092:12 (python3.15+0x41f3b0)
    #14 PyEval_EvalCode /home/hyongtao/worksapce/cpython/Python/ceval.c:673:21 (python3.15+0x41f3b0)
    #15 run_eval_code_obj /home/hyongtao/worksapce/cpython/Python/pythonrun.c:1366:12 (python3.15+0x4fe967) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #16 run_mod /home/hyongtao/worksapce/cpython/Python/pythonrun.c:1469:19 (python3.15+0x4fe967)
    #17 pyrun_file /home/hyongtao/worksapce/cpython/Python/pythonrun.c:1294:15 (python3.15+0x4f9d13) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #18 _PyRun_SimpleFileObject /home/hyongtao/worksapce/cpython/Python/pythonrun.c:518:13 (python3.15+0x4f9d13)
    #19 _PyRun_AnyFileObject /home/hyongtao/worksapce/cpython/Python/pythonrun.c:81:15 (python3.15+0x4f9498) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #20 pymain_run_file_obj /home/hyongtao/worksapce/cpython/Modules/main.c:410:15 (python3.15+0x53bedf) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #21 pymain_run_file /home/hyongtao/worksapce/cpython/Modules/main.c:429:15 (python3.15+0x53bedf)
    #22 pymain_run_python /home/hyongtao/worksapce/cpython/Modules/main.c:691:21 (python3.15+0x53b20b) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #23 Py_RunMain /home/hyongtao/worksapce/cpython/Modules/main.c:772:5 (python3.15+0x53b20b)
    #24 pymain_main /home/hyongtao/worksapce/cpython/Modules/main.c:802:12 (python3.15+0x53b778) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #25 Py_BytesMain /home/hyongtao/worksapce/cpython/Modules/main.c:826:12 (python3.15+0x53b7fb) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #26 main /home/hyongtao/worksapce/cpython/./Programs/python.c:15:12 (python3.15+0x17282b) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)

  Thread T3 'Thread-3 (reade' (tid=1703390, running) created by main thread at:
    #0 pthread_create <null> (python3.15+0xf421f) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #1 do_start_joinable_thread /home/hyongtao/worksapce/cpython/Python/thread_pthread.h:281:14 (python3.15+0x521424) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #2 PyThread_start_joinable_thread /home/hyongtao/worksapce/cpython/Python/thread_pthread.h:323:9 (python3.15+0x52123a) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #3 ThreadHandle_start /home/hyongtao/worksapce/cpython/./Modules/_threadmodule.c:474:9 (python3.15+0x5f109a) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #4 do_start_new_thread /home/hyongtao/worksapce/cpython/./Modules/_threadmodule.c:1920:9 (python3.15+0x5f0b3f) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #5 thread_PyThread_start_joinable_thread /home/hyongtao/worksapce/cpython/./Modules/_threadmodule.c:2043:14 (python3.15+0x5efc01) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #6 cfunction_call /home/hyongtao/worksapce/cpython/Objects/methodobject.c:564:18 (python3.15+0x2bab87) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #7 _PyObject_MakeTpCall /home/hyongtao/worksapce/cpython/Objects/call.c:242:18 (python3.15+0x20a68f) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #8 _PyObject_VectorcallTstate /home/hyongtao/worksapce/cpython/./Include/internal/pycore_call.h:134:16 (python3.15+0x20b2a7) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #9 PyObject_Vectorcall /home/hyongtao/worksapce/cpython/Objects/call.c:327:12 (python3.15+0x20b2a7)
    #10 _Py_VectorCall_StackRefSteal /home/hyongtao/worksapce/cpython/Python/ceval.c:720:11 (python3.15+0x41fce9) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #11 _PyEval_EvalFrameDefault /home/hyongtao/worksapce/cpython/Python/generated_cases.c.h:3387:35 (python3.15+0x429c79) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #12 _PyEval_EvalFrame /home/hyongtao/worksapce/cpython/./Include/internal/pycore_ceval.h:118:16 (python3.15+0x41f3b0) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #13 _PyEval_Vector /home/hyongtao/worksapce/cpython/Python/ceval.c:2092:12 (python3.15+0x41f3b0)
    #14 PyEval_EvalCode /home/hyongtao/worksapce/cpython/Python/ceval.c:673:21 (python3.15+0x41f3b0)
    #15 run_eval_code_obj /home/hyongtao/worksapce/cpython/Python/pythonrun.c:1366:12 (python3.15+0x4fe967) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #16 run_mod /home/hyongtao/worksapce/cpython/Python/pythonrun.c:1469:19 (python3.15+0x4fe967)
    #17 pyrun_file /home/hyongtao/worksapce/cpython/Python/pythonrun.c:1294:15 (python3.15+0x4f9d13) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #18 _PyRun_SimpleFileObject /home/hyongtao/worksapce/cpython/Python/pythonrun.c:518:13 (python3.15+0x4f9d13)
    #19 _PyRun_AnyFileObject /home/hyongtao/worksapce/cpython/Python/pythonrun.c:81:15 (python3.15+0x4f9498) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #20 pymain_run_file_obj /home/hyongtao/worksapce/cpython/Modules/main.c:410:15 (python3.15+0x53bedf) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #21 pymain_run_file /home/hyongtao/worksapce/cpython/Modules/main.c:429:15 (python3.15+0x53bedf)
    #22 pymain_run_python /home/hyongtao/worksapce/cpython/Modules/main.c:691:21 (python3.15+0x53b20b) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #23 Py_RunMain /home/hyongtao/worksapce/cpython/Modules/main.c:772:5 (python3.15+0x53b20b)
    #24 pymain_main /home/hyongtao/worksapce/cpython/Modules/main.c:802:12 (python3.15+0x53b778) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #25 Py_BytesMain /home/hyongtao/worksapce/cpython/Modules/main.c:826:12 (python3.15+0x53b7fb) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)
    #26 main /home/hyongtao/worksapce/cpython/./Programs/python.c:15:12 (python3.15+0x17282b) (BuildId: 00135b3b874b2f13c2b74122623dcba7b1b9ce4a)

SUMMARY: ThreadSanitizer: data race /home/hyongtao/worksapce/cpython/./Include/cpython/pyatomic_gcc.h:513:3 in _Py_atomic_store_ssize_relaxed
==================

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

Activity

  1. hyongtao-code
    added
    type-bugAn unexpected behavior, bug, or error
    on Jan 31, 2026
  2. aisk
    added
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    topic-free-threading
    on Jan 31, 2026

  3. devdanzin commented on Jul 19, 2026

    @devdanzin
    devdanzin
    on Jul 19, 2026
    Member

    fusil --tsan independently hit this, via a different trigger than the report's __length_hint__-vs-set-mutation script: sharing a single set iterator across threads (no set mutation at all). setiter_iternext reads si->si_pos inside Py_BEGIN_CRITICAL_SECTION(so) (setobject.c:1117) but writes it back outside the section (si->si_pos = i+1, :1128), and si->len-- is likewise unsynchronized — so two threads advancing the same iterator race on its private cursor. setiter_len reading si->len races the same write. The section is keyed on the set so, so it doesn't serialize concurrent use of one iterator.

    import operator, threading

NTHREADS = 8
barrier = threading.Barrier(NTHREADS)

def worker(it):
    barrier.wait()
    for _ in range(20000):
        try:
            next(it)                    # setiter_iternext: si->si_pos / si->len advance
        except StopIteration:
            pass
        operator.length_hint(it, 0)     # setiter_len: reads si->len

for _ in range(300):
    shared = iter(set(range(4096)))     # ONE shared iterator, no set mutation
    ts = [threading.Thread(target=worker, args=(shared,)) for _ in range(NTHREADS)]
    for t in ts: t.start()
    for t in ts: t.join()

    Exit 66 under TSan; setiter_iternext:1117 (read si->si_pos) vs :1128 (write), plus the setiter_len | setiter_iternext (si->len) face. This is exactly the "concurrent use of the same iterator" case PR #144357 addresses by switching to Py_BEGIN_CRITICAL_SECTION2(self, so) and moving the cursor writes inside the section — so it's a useful corroboration of that expanded scope. Confirmed on current main (3.16.0a0), debug + release TSan builds.

    (Found by fusil --tsan, a ThreadSanitizer fuzzer; draft and reproducer by Claude Code, minimized and reviewed by hand.)

  4. devdanzin commented on Jul 20, 2026

    @devdanzin
    devdanzin
    on Jul 20, 2026
    Member

    This isn't only a __length_hint__ data race — the same setiter_iternext path double-frees the set and crashes. On a free-threaded build, setiter_iternext (Objects/setobject.c) reads so = si->si_set unguarded, takes only the set's critical section around the table scan, and then on exhaustion runs — outside that section:

        if (key == NULL) {
        si->si_set = NULL;      /* :1130 */
        Py_DECREF(so);          /* :1131  drop the iterator's one owning ref to the set */
        return NULL;
    }

    Two threads advancing the same set iterator to exhaustion both read the same non-NULL so and both Py_DECREF(so) → the set's refcount underflows → use-after-free.

    Reproducer:

    import threading

NT = 8
def newit():
    return iter(set(range(32)))
cell = [newit()]
def worker():
    for _ in range(200_000):
        it = cell[0]
        try:
            next(it)
        except StopIteration:
            cell[0] = newit()
        except Exception:
            pass
ts = [threading.Thread(target=worker) for _ in range(NT)]
for t in ts: t.start()
for t in ts: t.join()

    PYTHON_GIL=0 ./python repro.py:

    • debug free-threaded build → Objects/setobject.c:1131: _Py_NegativeRefcount: object has negative ref count, ~8/8 within seconds.
    • release -O0 free-threaded build (no sanitizer) → SIGSEGV / SIGABRT, core dumped, 6/6.

    The realistic form uses a long-lived shared frozenset (e.g. a module-level constant, refcount > 1): the underflow is silent until the next gc.collect() reports Python/gc_free_threading.c:999 update_refs: Assertion "refcount >= 0" failed on a frozenset with a wild refcount (~2^60), or a later access UAFs.

    PR #144357 fixes this — widening to Py_BEGIN_CRITICAL_SECTION2(self, so) and, under Py_GIL_DISABLED, making exhaustion sticky via si_pos = -1 and removing the si_set = NULL; Py_DECREF(so) from iternext (dropping the set ref only in dealloc) is exactly what closes the double-DECREF. It'd be good to land it — this is a memory-safety bug, not just a value race. (This is the set sibling of the dict-iterator double-free just filed as gh-154130.)

    (Found by fusil --tsan, a ThreadSanitizer fuzzer; crash confirmed by re-running the reproducer without a sanitizer on plain free-threaded builds. Comment + reproducer by Claude Code, reviewed by hand.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    interpreter-core(Objects, Python, Grammar, and Parser dirs)topic-free-threadingtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions