Repository navigation
Free-threaded CPython can starve a thread reattaching during repeated stop-the-world GC #151518
Description
Bug report
While hardening free-threaded Python support in numba-cuda, we found a workload that could wedge when a stress worker repeatedly called gc.collect() in a tight loop while other threads were doing CUDA dispatch work. Removing only the tight manual-GC worker made the workload pass.
I reduced this to a no-third-party CPython reproducer. The reduced case is strongest on a debug free-threaded CPython main build, where the extra free-threaded GC validation work makes the stop-the-world window long enough to reproduce reliably.
import faulthandler
import gc
import importlib
import os
import sys
import threading
import time
faulthandler.dump_traceback_later(25, exit=True)
modules = (
"abc", "argparse", "collections", "contextlib",
"decimal", "enum", "functools", "heapq",
"importlib", "inspect", "itertools", "json",
"math", "operator", "random", "re",
"statistics", "threading", "types", "weakref",
)
for name in modules:
importlib.import_module(name)
print("python", sys.version, flush=True)
print("abiflags", getattr(sys, "abiflags", ""), flush=True)
print("gil", getattr(sys, "_is_gil_enabled", lambda: None)(), flush=True)
print("debug", hasattr(sys, "gettotalrefcount"), flush=True)
print("pid", os.getpid(), flush=True)
stop = threading.Event()
count = 0
def collect():
global count
while not stop.is_set():
gc.collect()
count += 1
thread = threading.Thread(target=collect, name="gc-collector")
started = time.monotonic()
thread.start()
time.sleep(5.0)
stop.set()
thread.join(10.0)
elapsed = time.monotonic() - started
print(f"alive={thread.is_alive()} count={count} elapsed={elapsed:.6f}",
flush=True)
raise SystemExit(1 if thread.is_alive() else 0)Build used for the reproducer:
./configure --with-pydebug --disable-gil --without-ensurepip
make -jObserved on CPython main debug/free-threaded:
Python 3.16.0a0 free-threading build
abiflags td
GIL disabled
debug True
I reproduced this on a B200 Linux system with 224 CPUs at the host level. I have not yet reproduced it on a smaller local workstation. The test process was running under a Slurm allocation that reported 28 CPUs available to the process and 224 CPUs on the system.
The script regularly times out via the faulthandler watchdog. The Python-level traceback shows the GC worker in gc.collect() and the main thread apparently still at time.sleep(). A native stack shows the more precise state:
gc-collector:
validate_refcounts() / validate_gc_objects()
gc_visit_heaps()
deduce_unreachable_heap()
gc_collect_internal()
_PyGC_Collect()
gc.collect()
main thread:
_PyParkingLot_Park()
tstate_wait_attach()
_PyThreadState_Attach()
PyEval_RestoreThread()
pysleep()
time.sleep()
At the time of the stall, GDB showed the interpreter stop-the-world state as:
requested = true
world_stopped = true
thread_countdown = 0
requester = GC worker thread state
main thread state = _Py_THREAD_SUSPENDED
The suspected issue is that after start_the_world() moves suspended threads back to detached and unparks them, a thread running a tight gc.collect() loop can immediately request another stop-the-world pause and re-suspend a just-unparked thread before that thread gets to attach. The result is starvation of a thread trying to return from a detached operation such as time.sleep().
This was not reproduced in my minimized matrix on:
- a non-debug free-threaded CPython
mainbuild - conda-forge Python 3.14.6t
However, the original larger stress workload was first encountered while testing Python 3.14t free-threaded package support, and the reduced CPython main debug build gives a clean way to expose the progress bug.
I have a local CPython patch that tracks thread states waiting to attach after a stop-the-world suspension and prevents a subsequent stop-the-world requester from immediately re-suspending those attach waiters. With that patch:
- the minimized reproducer passes
- the broader pure-Python GC stress matrix passes
test_free_threading.test_gcpasses under debug and non-debug free-threaded buildstest_free_threading.test_gc test_threading -vpasses under the debug free-threaded build
I will prepare a PR with the fix and regression test once this issue exists.
CPython versions tested on:
- 3.14
- 3.16
- CPython main branch
Operating systems tested on:
- Linux
Activity
zangjiucheng commentedon Jun 22, 2026
Hey, this seems like an interesting example of a thread starvation problem.
I take some time to dive into this problem, and based on my understanding, the root cause is a race in
start_the_world/park_detached_threads: a thread woken fromSUSPENDED→DETACHEDcan be re-parked before it gets a chance to re-attach, because the nextgc.collect()call fires immediately, andpark_detached_threadssees it as plainDETACHED.I have some personal suggestions on this problem fix, maybe introduce
DETACHED_WAKING (4)as a transient state:SUSPENDED→DETACHED_WAKINGinstead ofDETACHED.park_detached_threadsseesDETACHED_WAKING→ sets_PY_EVAL_PLEASE_STOP_BITon the eval breaker without parking. The thread gets a chance to attach first, then suspends cooperatively via_Py_HandlePending().tstate_try_attachnow acceptsDETACHED_WAKING→ATTACHEDin addition toDETACHED→ATTACHED, soPy_END_ALLOW_THREADSsucceeds immediately after a wakeup.tpn commentedon Jul 1, 2026
Thanks — yes, that is the release/re-park race I am seeing. I considered
representing the wakeup entirely in the thread-state value.
The complication with mapping every resumed
SUSPENDEDtstate toDETACHED_WAKINGis that some tstates remain detached because their OS threadis still blocked in sleep or I/O and is not yet trying to attach. If the next
STW refuses to park one of those tstates, it can wait until that operation
returns; an eval-breaker bit cannot run while the thread remains detached.
PR #152826 instead distinguishes tstates parked from
DETACHED, then marks awaiter only when
tstate_wait_attach()actually observes that state. A laterSTW skips only active attach waiters. Passive detached tstates remain
parkable, and the normal uncontended attach path remains the existing CAS. A
state-only representation of an active waiter could also work, but it would
need another state and transition rather than marking every woken tstate.
nascheme commentedon Jul 3, 2026
@colesbury
pythongh-151518: Avoid STW starvation of attaching threads
pythongh-151518: Track attach waiters in the thread state
pythongh-151518: Simplify the GC fairness regression
pythongh-151518: Exercise STW fairness directly
pythongh-151518: Avoid STW starvation of attaching threads
pythongh-151518: Track attach waiters in the thread state
pythongh-151518: Simplify the GC fairness regression
pythongh-151518: Exercise STW fairness directly
pythongh-151518: Preserve attach waiters when resuming BRC suspensions
pythongh-151518: Strengthen the STW attach fairness regression
pythongh-151518: Restrict STW test pauses to free-threaded builds
pythongh-151518: Wake only registered attach waiters
gh-151518: Avoid STW starvation of attaching threads (#152826)
[3.14] pythongh-151518: Avoid STW starvation of attaching threads