Multithreaded subinterpreters can be running during finalization

@ericsnowcurrently

Crash report

What happened?

(Found while investigating a fix for #125920.)

Subinterpreters that are running in a different thread can still be running upon the finalization of the main interpreter, depending on what the thread was doing. Take the following script:

from threading import Thread
import _interpreters


def silly():
    interp = _interpreters.create()
    _interpreters.run_string(interp, "import time; time.sleep(100)")


if __name__ == "__main__":
    Thread(target=silly).start()

On 3.13 and 3.14, killing the process with CTRL+C results in an assertion failure:

python: Python/pylifecycle.c:2480: finalize_subinterpreters: Assertion `!_PyInterpreterState_IsRunningMain(interp)' failed.

However, if the thread is daemon, then the interpreter fully segfaults:

from threading import Thread
import _interpreters


def silly():
    interp = _interpreters.create()
    _interpreters.run_string(interp, "import time; time.sleep(100)")


if __name__ == "__main__":
    Thread(target=silly, daemon=True).start()

I'm going to investigate possible fixes.

cc @ericsnowcurrently

CPython versions tested on:

3.13, CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

No response

Linked PRs

Using .join() will not trigger an assertion error. For the reason, please see the PR submitted by Eric and me.

>>> def silly():
...     interp = _interpreters.create()
...     _interpreters.run_string(interp, "import time; time.sleep(100)")
...
>>> if __name__ == "__main__":
...     Thread(target=silly).start()
...
>>>
>>> Thread(target=silly).start()
>>> Thread(target=silly).start()
>>> quit


^CTraceback (most recent call last):
  File "/root/cpython/Lib/threading.py", line 1540, in _shutdown
    _thread_shutdown()
KeyboardInterrupt:
python: Python/pylifecycle.c:2463: finalize_subinterpreters: Assertion `!_PyInterpreterState_IsRunningMain(interp)' failed.

>>> def silly():
...     interp = _interpreters.create()
...     _interpreters.run_string(interp, "import time; time.sleep(100)")
...
>>> t = Thread(target=silly)
>>> t.start()
>>> t.join()
^CTraceback (most recent call last):
  File "<python-input-14>", line 1, in <module>
    t.join()
    ~~~~~~^^
  File "/root/cpython/Lib/threading.py", line 1092, in join
    self._handle.join(timeout)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^
KeyboardInterrupt

Yes, I know. The issue is that CTRL+C'ing causes the thread to not finish joining, meaning that the thread state is still marked as running. I'm working on a fix to properly clean up thread states before the interpreter shuts down using pending calls, but daemon threads will probably have to get addressed elsewhere (there's a data race, I think).

If you want, you can insert a signal catcher to catch SIGINT and do error handling on it. And in the error handling iterate over the list of threads and handle them

That would result in a thread safety nightmare if the thread was still running :)

#126026 addresses non-daemon threads. It's a bit hacky, though, because I have to manually mess with the status of the thread state.

I think #128640 will fix this as well.

gh-126016: Remove bad assertion in `PyThreadState_Clear` (GH-139158)

Closing this for now. Ideally we'll get to a point where we can add the assertion back, but that can be tracked elsewhere.

gh-126016: Fix flaky test by allowing the SIGINT return code (GH-139219)

Navigation Menu

Repository navigation

Multithreaded subinterpreters can be running during finalization #126016

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Multithreaded subinterpreters can be running during finalization #126016

Description

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Linked PRs

Activity

rruuaanng commented on Oct 27, 2024

ZeroIntensity commented on Oct 27, 2024

rruuaanng commented on Oct 27, 2024

ZeroIntensity commented on Oct 27, 2024

ZeroIntensity commented on Oct 27, 2024

ZeroIntensity commented on Jan 11, 2025

ZeroIntensity commented on Sep 19, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions