Deadlock at shutdown with stop-the-world and daemon threads

@pablogsal

Bug report

Bug description:

Reported by @pablogsal / @godlygeek from memray

Stack trace:
https://gist.github.com/pablogsal/513fa8b0c29cda852ce11c86ce3b1345

We have two threads, the main thread (M) and a daemon thread (D). The main thread starts _Py_Finalize() and performs a global stop the world. The daemon thread is disabling profiling and so tries to performa a stop-the-world specific to it's interpreter:

cpython/Python/pystate.c

Lines 2256 to 2267 in 9745976

    
           static void 
        
           stop_the_world(struct _stoptheworld_state *stw) 
        
           { 
        
               _PyRuntimeState *runtime = &_PyRuntime; 
        
               PyMutex_Lock(&stw->mutex); 
        
               if (stw->is_global) { 
        
                   _PyRWMutex_Lock(&runtime->stoptheworld_mutex); 
        
               } 
        
               else { 
        
                   _PyRWMutex_RLock(&runtime->stoptheworld_mutex); 
        
               }

M: _PyEval_StopTheWorldAll():
M: acquires runtime->stoptheworld->mutex
M: acquires RW lock runtime->stoptheworld_mutex in W (exclusive) mode
M: ... waits on threads
D: _PyEval_StopTheWorld(interp):
D: acquires interp->stoptheworld->mutex
D: ... blocks trying to acquire runtime->stoptheworld_mutex in R mode. Later, the daemon thread will hang in _PyThreadState_HangThread() when trying to re-attach it's thread state.
M: _PyEval_StopTheWorldAll() finishes, marks the interpreter as finalizing
M: ...
M: calls _PyGC_CollectNoFail() which tries to run _PyEval_StopTheWorld(interp)
M: ... blocks trying to acquire interp->stoptheworld->mutex, which is still held by the daemon thread!

Deadlock! Summary:

The daemon thread holds interp->stoptheworld->mutex and is hanging because the interpreter is shutting down.

The main thread is trying to perform the shutdown procedure, including calling the GC a few times, which requires interp->stoptheworld->mutex.

Fix???

Release the previously acquired interp->stoptheworld->mutex when hanging the thread if necessary? Crosses a bunch of abstraction barriers, which is messy and tricky

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

In Memray's specific case, that thread wasn't meant to be a daemon thread at all (it unintentionally inherited the daemonness of its parent thread), and the fix is explicitly making it a non-daemon thread.

Deadlock repro:

import threading
import sys

NUM_THREADS = 10

b = threading.Barrier(NUM_THREADS + 1)

def run_in_bg():
    b.wait()
    while True:
        sys.setprofile(None)

for _ in range(NUM_THREADS):
    t = threading.Thread(target=run_in_bg, daemon=True)
    t.start()

b.wait()
print("Exiting...")

@colesbury

Hi @colesbury ,

I want to have a try at this issue. 😊

Thanks for your reproducing code sample.

I have got the data race stack with TSAN after running this script and quitting the interpreter.

Best Regards,
Edward

@LindaSummer

@LindaSummer - thanks, but I have a particular way in mind that I'd like to address this.

pythongh-137433: Fix deadlock with stop-the-world and daemon threads

Merge branch 'main' into pythongh-137433-stop-the-world-daemon-deadlock

gh-137433: Fix deadlock with stop-the-world and daemon threads (gh-13…

pythongh-137433: Fix deadlock with stop-the-world and daemon threads (p…

[3.14] gh-137433: Fix deadlock with stop-the-world and daemon threads (…

Navigation Menu

Repository navigation

Deadlock at shutdown with stop-the-world and daemon threads #137433

Bug report

Bug description:

Fix???

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	static void
	stop_the_world(struct _stoptheworld_state *stw)
	{
	_PyRuntimeState *runtime = &_PyRuntime;

	PyMutex_Lock(&stw->mutex);
	if (stw->is_global) {
	_PyRWMutex_Lock(&runtime->stoptheworld_mutex);
	}
	else {
	_PyRWMutex_RLock(&runtime->stoptheworld_mutex);
	}

Deadlock at shutdown with stop-the-world and daemon threads #137433

Description

Bug report

Bug description:

Fix???

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

godlygeek commented on Aug 5, 2025

colesbury commented on Aug 12, 2025

LindaSummer commented on Aug 13, 2025

colesbury commented on Aug 13, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions