Some pre-finalization callbacks can create other callbacks

@ericsnowcurrently

Bug report

Bug description:

Currently, "pre-finalization" (callbacks executed while the interpreter is still fully intact) looks like this:

Lines 2023 to 2038 in 642e5df

    
               wait_for_thread_shutdown(tstate); 
        
               // Make any remaining pending calls. 
        
               _Py_FinishPendingCalls(tstate); 
        
               /* The interpreter is still entirely intact at this point, and the 
        
                * exit funcs may be relying on that.  In particular, if some thread 
        
                * or exit func is still waiting to do an import, the import machinery 
        
                * expects Py_IsInitialized() to return true.  So don't say the 
        
                * runtime is uninitialized until after the exit funcs have run. 
        
                * Note that Threading.py uses an exit func to do a join on all the 
        
                * threads created thru it, so this also protects pending imports in 
        
                * the threads created via Threading. 
        
                */ 
        
               _PyAtExit_Call(tstate->interp);

Threads are joined first, then pending calls are executed, and finally atexit callbacks are executed. The problem is that any of these three can create one another, such as a pending call creating a thread, or an atexit handler scheduling a pending call. The best way to demonstrate this is by creating a new thread inside of an atexit callback:

import atexit
import threading
import time


def run():
    print(24)
    time.sleep(1)
    print(42)

@atexit.register
def start_thread():
    threading.Thread(target=run).start()

On 3.13+ (and probably earlier), you'll get one of these three outputs:

24 is printed.
Nothing is printed.
A fatal error occurs due to the stdout lock being unavailable.

This is because the thread created is incorrectly non-daemon, because the call to threading._shutdown (which should join it) has already happened.

@ericsnowcurrently, I discussed this with you at the PyCon sprints. I think we came to the agreement that the best way to fix this is by looping until all three of them execute no callbacks.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

Yeah, this is what we discussed. Thanks for remembering and opening this issue. I have some thoughts, but first I'll take a look at your PR.

gh-136003: Execute pre-finalization callbacks in a loop (GH-136004)

gh-136003: Skip non-daemon threads when exceptions occur during final…

The test test_pending_call_creates_thread_subinterpreter has a ~~reference leak~~.
edit: a file descriptor leak. I'll send a fix.

gh-136003: Close file descriptors in test (GH-139225)

@ZeroIntensity

Sorry, @ZeroIntensity, I should have warned you about closing the file descriptors.

@ZeroIntensity

@ZeroIntensity, I do want to circle back on why os.pipe() wasn't working in the test when we tried it that way.

I should have warned you about closing the file descriptors.

No worries. For the future, it would be nice if regrtest was able to detect unclosed FDs automatically without running in refleak mode.

I do want to circle back on why os.pipe() wasn't working in the test when we tried it that way.

I'll experiment with it more later today. Maybe my laptop has some weird setting that prevents IPC with pipe?

@ericsnowcurrently

Ah, @ericsnowcurrently, we need to use pass_fds for the child process to inherit the fd:

import subprocess
import sys
import os

r, w = os.pipe()
script = f"""if True:
    import os
    os.write({w}, b'a')
"""
subprocess.run([sys.executable, '-c', script], pass_fds=[w])
print(os.read(r, 1))

You may have been thinking of multiprocessing.Pipe, which seems to do this for you.

@vstinner

For the future, it would be nice if regrtest was able to detect unclosed FDs automatically without running in refleak mode.

That would be worth investigating. I would not be surprised if @vstinner or others have given this some consideration, but, even then, perhaps it's time to revisit it.

Checking for FD leaks in regular regrtest mode may slow down the code. At least, it's good that we have the refleak mode to check for FD leaks!

Navigation Menu

Repository navigation

Some pre-finalization callbacks can create other callbacks #136003

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	wait_for_thread_shutdown(tstate);

	// Make any remaining pending calls.
	_Py_FinishPendingCalls(tstate);

	/* The interpreter is still entirely intact at this point, and the
	* exit funcs may be relying on that. In particular, if some thread
	* or exit func is still waiting to do an import, the import machinery
	* expects Py_IsInitialized() to return true. So don't say the
	* runtime is uninitialized until after the exit funcs have run.
	* Note that Threading.py uses an exit func to do a join on all the
	* threads created thru it, so this also protects pending imports in
	* the threads created via Threading.
	*/

	_PyAtExit_Call(tstate->interp);

Some pre-finalization callbacks can create other callbacks #136003

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

ericsnowcurrently commented on Jul 3, 2025

encukou commented on Sep 22, 2025

ericsnowcurrently commented on Sep 22, 2025

ericsnowcurrently commented on Sep 22, 2025

ZeroIntensity commented on Sep 22, 2025

ZeroIntensity commented on Sep 22, 2025

ericsnowcurrently commented on Sep 23, 2025

vstinner commented on Sep 23, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions