Skip to content

Subinterpreters don't properly clean up threads #128639

Closed
Closed
Subinterpreters don't properly clean up threads#128639
Labels
3.13only security fixes3.14bugs and security fixesextension-modulesC modules in the Modules dirtopic-subinterpreterstype-crashA hard crash of the interpreter, possibly with a core dump

Description

@ZeroIntensity
ZeroIntensity
opened on Jan 8, 2025

Crash report

What happened?

I found this issue a little while back, but I'm finally getting around to fixing it.

Currently, subinterpreter finalization assumes that there's only one thread left. So, any remaining threads--daemon or non-daemon--crash the interpreter upon finalizing:

import _interpreters

interp = _interpreters.create()
source = """
import threading
import time

def hello():
    time.sleep(1)

t = threading.Thread(target=hello)
t.start()
"""
_interpreters.run_string(interp, source)

This results in an assertion failure on my end:

python: Python/pystate.c:1969: tstate_activate: Assertion `!tstate->_status.bound_gilstate || tstate == gilstate_tss_get((tstate->interp->runtime))' failed.

So, there's a few things that need to get fixed:

  • The finalization process shouldn't assume that there's one thread. Instead, it should just dive straight into Py_EndInterpreter, which will properly clean things up.
  • It also shouldn't try to manually delete threads via attaching to them and then calling PyThreadState_Delete. That should also happen in Py_EndInterpreter, after all threads have finished.
  • Subinterpreter finalization itself happens way too late. _PyRuntimeState_SetFinalizing has been set, meaning that all threads will be already blocked, and thus cannot shutdown via threading._shutdown, resulting in a deadlock. finalize_subinterpreters should be called before that happens.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

No response

Linked PRs

Activity

  1. ZeroIntensity
    added
    extension-modulesC modules in the Modules dir
    type-crashA hard crash of the interpreter, possibly with a core dump
    topic-subinterpreters
    3.13only security fixes
    3.14bugs and security fixes
    on Jan 8, 2025
  2. github-project-automation
    added this to Subinterpreterson Jan 8, 2025
  3. github-project-automation
    moved this to Todo in Subinterpreterson Jan 8, 2025

  4. ericsnowcurrently commented on May 19, 2025

    @ericsnowcurrently
    ericsnowcurrently
    on May 19, 2025
    Member

    FWIW, I thought I had created an issue about cleaning up lingering thread states automatically. Would you mind double-checking and closing it, if so?

  5. ZeroIntensity
    marked Lingering Subinterpreter Subthread Keeps Program From Exiting #113146 as a duplicate of this issue on May 19, 2025
  6. ericsnowcurrently
    added a commit that references this issue on May 19, 2025

    gh-128639: Don't assume one thread in subinterpreter finalization (gh…

    9859791
  7. miss-islington
    added a commit that references this issue on May 19, 2025

    pythongh-128639: Don't assume one thread in subinterpreter finalizati…

    76bcc46
  8. ZeroIntensity
    added a commit that references this issue on May 19, 2025

    Revert "pythongh-128639: Don't assume one thread in subinterpreter fi…

    79ed70e
  9. ericsnowcurrently
    added a commit that references this issue on May 19, 2025

    Revert "gh-128639: Don't assume one thread in subinterpreter finaliza…

    27bd082

  10. ZeroIntensity commented on May 21, 2025

    @ZeroIntensity
    ZeroIntensity
    on May 21, 2025
    MemberAuthor

    For those who aren't me or Eric: we had to revert the fix because it breaks the test suite when run with --single-process. I'm still trying to figure out why.

  11. ZeroIntensity commented on May 24, 2025

    @ZeroIntensity
    ZeroIntensity
    on May 24, 2025
    MemberAuthor

    @ericsnowcurrently I finally figured out the buildbot failures. This is an awesome bug.

    Here's a repro that crashes with my patch applied:

    import _interpreters
import types
import time

interp = _interpreters.create(
    types.SimpleNamespace(
        use_main_obmalloc=False,
        allow_fork=False,
        allow_exec=False,
        allow_threads=True,
        allow_daemon_threads=True,
        check_multi_interp_extensions=True,
        gil='own',
    )
)
_interpreters.exec(interp, f"""if True:
    import threading
    import time

    def task():
        time.sleep(1)

    threads = [threading.Thread(target=task, daemon=True) for _ in range(3)]
    for t in threads:
        t.start()
    """)
_interpreters.destroy(interp)
time.sleep(2)

    The problem is that something subtly changed with daemon thread shutdown in-between writing my patch and merging it. So, after the time.sleep() call, the thread would still be alive and crash. Unfortunately for me, I randomly chose 100 seconds as the sleep time, so if the process exited within those 100 seconds, nothing would crash. I'll put up the fixed version.

  12. ZeroIntensity
    added a commit that references this issue on May 24, 2025

    Reapply "pythongh-128639: Don't assume one thread in subinterpreter f…

    b14aa73

  13. sergey-miryanov commented on Jul 8, 2025

    @sergey-miryanov
    sergey-miryanov
    on Jul 8, 2025
    Contributor

    I have a following repro:

    from concurrent import interpreters
import _interpreters

def func(*args, **kwargs):
    # pass # crash on main, ok on #134606
    assert False # crash on both

def clean_up_interpreters():
    for interp in interpreters.list_all():
        if interp.id == 0:  # main
            continue
        try:
            interp.close()
        except _interpreters.InterpreterError:
            pass  # already destroyed


def test():
    interp = interpreters.create()
    interp.call_in_thread(func)

try:
    for _ in range(100):
        test()
finally:
    clean_up_interpreters()

    I tested it on main and on #134606 (with merged main). func with assert False crashes with following stack trace:

     	python315.dll!PyDict_GetItemWithError(_object * op, _object * key) Line 2450	C
 	[Inline Frame] python315.dll!get_type_attr_as_size(_typeobject *) Line 30	C
 	[Inline Frame] python315.dll!PyStructSequence_New(_typeobject *) Line 64	C
 	python315.dll!make_unraisable_hook_args(_ts * tstate, _object * exc_type, _object * exc_value, _object * exc_tb, _object * err_msg, _object * obj) Line 1412	C
 	python315.dll!format_unraisable_v(const char * format, char * va, _object * obj) Line 1670	C
 	python315.dll!PyErr_FormatUnraisable(const char * format, ...) Line 1746	C
>	python315.dll!Py_EndInterpreter(_ts * tstate) Line 2449	C
 	python315.dll!_PyXI_EndInterpreter(_is * interp, _ts * tstate, _ts * * p_save_tstate) Line 3285	C
 	python315.dll!interp_destroy(_object * self, _object * args, _object * kwds) Line 946	C
 	python315.dll!cfunction_call(_object * func, _object * args, _object * kwargs) Line 567	C

  14. ZeroIntensity commented on Jul 8, 2025

    @ZeroIntensity
    ZeroIntensity
    on Jul 8, 2025
    MemberAuthor

    No idea yet. Let's wait until we get thread cleanup fixed and then we can investigate that one in a new issue.

  15. ZeroIntensity
    marked concurrent.interpreters - buggy code leads to a crash during finalization #136568 as a duplicate of this issue on Jul 12, 2025
  16. Pranjal095
    added 2 commits that reference this issue on Jul 12, 2025

    pythongh-128639: Don't assume one thread in subinterpreter finalizati…

    29d735a

    Revert "pythongh-128639: Don't assume one thread in subinterpreter fi…

    521fb74
  17. taegyunkim
    added 2 commits that reference this issue on Aug 4, 2025

    pythongh-128639: Don't assume one thread in subinterpreter finalizati…

    c546d00

    Revert "pythongh-128639: Don't assume one thread in subinterpreter fi…

    c92b809
  18. ZeroIntensity
    added a commit that references this issue on Sep 17, 2025

    gh-128639: Don't assume one thread in subinterpreter finalization wit…

    a648813
  19. miss-islington
    added a commit that references this issue on Sep 17, 2025

    pythongh-128639: Don't assume one thread in subinterpreter finalizati…

    7b8c082
  20. ZeroIntensity
    closed this as completedon Sep 18, 2025
  21. github-project-automation
    moved this from Todo to Done in Subinterpreterson Sep 18, 2025
  22. ZeroIntensity
    added a commit that references this issue on Oct 7, 2025

    [3.14] gh-128639: Don't assume one thread in subinterpreter finalizat…

    cec4ddf

  23. sergey-miryanov commented on Nov 4, 2025

    @sergey-miryanov
    sergey-miryanov
    on Nov 4, 2025
    Contributor

    FTR, repro from #128639 (comment) seems to have been fixed. It produces:

    Exception in thread Thread-100 (_call):
Traceback (most recent call last):
  File "X:\Work\cpython\main\Lib\threading.py", line 1075, in _bootstrap_inner
    self._context.run(self.run)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "X:\Work\cpython\main\Lib\threading.py", line 1017, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "X:\Work\cpython\main\Lib\concurrent\interpreters\__init__.py", line 222, in _call
    raise ExecutionFailed(excinfo)
concurrent.interpreters.ExecutionFailed: AssertionError:

Uncaught in the interpreter:

Traceback (most recent call last):
  File "X:\Work\cpython\issues\test.py", line 6, in func
    assert False # crash on both
           ^^^^^
AssertionError

    Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.13only security fixes3.14bugs and security fixesextension-modulesC modules in the Modules dirtopic-subinterpreterstype-crashA hard crash of the interpreter, possibly with a core dump

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions