Creating an ungodly amount of sub interpreters in a short amount of time causes memory debug assertions.

Bug report

Bug description:

Hello. While working on a small joke program, I found a possible memory corruption issue (it could also be a threading issue?) when using the Python C API in a debug only build to quickly create, execute python code, and then destroy 463 sub interpreters. Before I post the code sample and the debug output I'm using a somewhat unique build environment for a Windows developer.

clang++ 18.1.7 for x86_64-pc-windows-msvc
Visual Studio Build Tools 2022 17.11.0
CMake 3.30
Python 3.12.5
ninja 1.12.1

When running the code sample I've attached at the bottom of this post, I am unable to get the exact same output each time, though the traceback does fire in the same location (Due to the size of the traceback I've not attached it, as it's about 10 MB of text for each thread). Additionally, I sometimes have to run the executable several times to get the error to occur. Lastly, release builds do not exhibit any thread crashes or issues as the debug assertions never fire or execute.

The error output seems to also halt in some cases, either because of stack failure or some other issue I was unable to determine and seemed to possibly be outside the scope of the issue presented here. I have the entire error output from one execution where I was able to save the output.

Debug memory block at address p=00000267272D6030: API 'p'
    16718206241729413120 bytes originally requested
    The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):
        at p-7: 0xa0 *** OUCH
        at p-6: 0x00 *** OUCH
        at p-5: 0x00Hello, world! From Thread 31
Hello, world! From Thread 87
 *** OUCH
        at p-4: 0xfd
Hello, world! From Thread 43
Hello, world! From Thread 279
Generating thread state 314
        at p-3: 0xfd
        at p-2: 0xfd
Hello, world! From Thread 168
Generating thread state 315
        at p-1: 0xfd
    Because memory is corrupted at the start, the count of bytes requested
       may be bogus, and checking the trailing pad bytes may segfault.
Generating thread state 316
Generating thread state 317
    The 8 pad bytes at tail=E8030267272D6030 are

The output cut off after this, as the entire program crashed, taking my terminal with it 😅

You'll find the MRE code below. I've also added a minimal version of CMakeLists.txt file I used so anyone can recreate the build with the code below (Any warnings, or additional settings I have do not affect whether the error occurs or not). The code appears to breaks inside of _PyObject_DebugDumpAddress, based on what debugging I was able to do with WinDbg.

Important

std::jthread calls .join() on destruction, so all threads auto-join once the std::vector goes out of scope.

Additionally this code exhibits the same behavior regardless of whether it is a thread_local or declared within the lambda passed to std::thread

main.cxx

#include <vector>
#include <thread>
#include <cstdlib>
#include <print>

#include <Python.h>

namespace {

static thread_local inline PyThreadState* state = nullptr;
static inline constexpr auto MAX_STATES = 463; 
static inline constexpr auto config = PyInterpreterConfig {
  .use_main_obmalloc = 0,
  .allow_fork = 0,
  .allow_exec = 0,
  .allow_threads = 0,
  .allow_daemon_threads = 0,
  .check_multi_interp_extensions = 1,
  .gil = PyInterpreterConfig_OWN_GIL,
};

} /* nameless namespace */

void execute () {
  std::vector<std::jthread> tasks { };
  tasks.reserve(MAX_STATES);
  for (auto count = 0zu; count < tasks.capacity(); count++) {
    std::println("Generating thread state {}", count);
    tasks.emplace_back([count] {
      if (auto status = Py_NewInterpreterFromConfig(&state, &config); PyStatus_IsError(status)) {
        std::println("Failed to initialize thread state {}", count);
        return;
      }
      auto text = std::format(R"(print("Hello, world! From Thread {}"))", count);
      auto globals = PyDict_New();
      auto code = Py_CompileString(text.data(), __FILE__, Py_eval_input);
      auto result = PyEval_EvalCode(code, globals, globals);
      Py_DecRef(result);
      Py_DecRef(code);
      Py_DecRef(globals);
      Py_EndInterpreter(state);
      state = nullptr;
    });
  }
}

int main() {
  PyConfig config {};
  PyConfig_InitIsolatedConfig(&config);
  if (auto status = Py_InitializeFromConfig(&config); PyStatus_IsError(status)) {
    std::println("Failed to initialize with isolated config: {}", status.err_msg);
    return EXIT_FAILURE;
  }
  PyConfig_Clear(&config);
  execute();
  Py_Finalize();
}

CMakeLists.txt

cmake_minimum_required(VERSION 3.30)
project(463-interpreters LANGUAGES C CXX)

find_package(Python 3.12 REQUIRED COMPONENTS Development.Embed)

set(CMAKE_MSVC_RUNTIME_LIBRARY "MultiThreaded$<$<CONFIG:Debug>:Debug>")

add_executable(${PROJECT_NAME})
target_sources(${PROJECT_NAME} PRIVATE main.cxx)
target_compile_features(${PROJECT_NAME} PRIVATE cxx_std_23)
target_precompile_headers(${PROJECT_NAME} PRIVATE <Python.h>)
target_link_libraries(${PROJECT_NAME} PRIVATE Python::Python)

Command to build + run

$ cmake -Bbuild -S. -G "Ninja"
$ cmake --build build && .\build\463-interpreters.exe

CPython versions tested on:

3.12

Operating systems tested on:

Windows

Can you reproduce this behaviour on Linux? or MacOS? Or what happens if you sleep a bit before spawning the sub-interpreters? I'm not an expert in threads but I'd say that interpreters are spawning too fast for the OS to follow up =/ Can you check whether the same occurs with HEAD instead of 3.12.5? (maybe there are some bugfixes that are not yet available)

At a glance, this could possibly be related to a memory allocation failure or something similar.

  auto globals = PyDict_New();
  auto code = Py_CompileString(text.data(), __FILE__, Py_eval_input);
  auto result = PyEval_EvalCode(code, globals, globals);

These lines need a NULL check if an error occurred, otherwise there will be a NULL pointer dereference in the call to Py_DecRef.

At a glance, this could possibly be related to a memory allocation failure or something similar.
  auto globals = PyDict_New();
  auto code = Py_CompileString(text.data(), __FILE__, Py_eval_input);
  auto result = PyEval_EvalCode(code, globals, globals);
These lines need a NULL check if an error occurred, otherwise there will be a NULL pointer dereference in the call to Py_DecRef.

Py_DecRef is the function version of Py_XDECREF. Though I can see the confusion between Py_DecRef and Py_DECREF. If there was a nullptr, we'd have a different set of errors, and the debug memory assertions wouldn't be complaining that all memory was not written with the FORBIDDENBYTE value.

Can you reproduce this behaviour on Linux? or MacOS?
I will work on getting the Linux/macOS behaviors checked when I can. My WSL2 environment is woefully out of date 😅. macOS might be more difficult, as I'm unaware of how the _d version of the Python.framework might be linked against.

Or what happens if you sleep a bit before spawning the sub-interpreters?

Sleeping might fix it, but might also hide the issue. The goal here was just to run a bunch of interpreters, not sleep for the sake of working around possible synchronization issues

I'm not an expert in threads but I'd say that interpreters are spawning too fast for the OS to follow up =/

This is not how threads or operating systems work. There is a synchronization issue or an ABA problem with the debug memory assertions in the C API's internals

Can you check whether the same occurs with HEAD instead of 3.12.5? (maybe there are some bugfixes that are not yet available)

I will also work on giving this a go.

Py_DecRef is the function version of Py_XDECREF

Oh! TIL. I still would suggest adding a NULL check -- an error indicator could be set, which would break subsequent calls to PyErr_Occurred.

The error is outside of the Python C API as one might usually interact with it. i.e., only on subinterpreter initialization (i.e., creating a PyThreadState) or on subinterpreter destruction (Py_EndInterpreter) does the error occur (typically the latter). PyErr_Occurred is not being called anywhere in the MRE, nor in the actual traceback for the failure.

Right, but it could cause memory corruption somewhere, and cause the error somewhere else down the line. Regardless, I wasn't able to reproduce this on Linux, so this might be Windows-specific.

@picnixz

@picnixz I can confirm the behavior still exists on Windows on HEAD. Same errors can occur

Right, but it could cause memory corruption somewhere, and cause the error somewhere else down the line. Regardless, I wasn't able to reproduce this on Linux, so this might be Windows-specific.

PyErr_Occurred only performs a read on the current thread state object. If an error is present when it shouldn't be that still points to a memory corruption error occurring between the destruction and initialization of a PyThreadState.

@ZeroIntensity

Pleased to back up @ZeroIntensity and say that this appears to be a Windows specific issue. I was unable to replicate this memory assertion error on Linux.

However, when executing this code under TSan, I got the following data race error that seems to indicate the same kind of memory corruption COULD be taking place.

WARNING: ThreadSanitizer: data race (pid=11259)
  Write of size 8 at 0x7f24e35e16c0 by thread T10:
    #0 qsort_r <null> (463-interpreters+0x9bc2e) (BuildId: 4e94833395fedac62b3c4caa6921ff713e25cff1)
    #1 qsort <null> (463-interpreters+0x9bec7) (BuildId: 4e94833395fedac62b3c4caa6921ff713e25cff1)
    #2 setup_confname_table /usr/src/python3.12-3.12.3-1ubuntu0.1/build-shared/../Modules/posixmodule.c:13543:5 (libpython3.12.so.1.0+0x3b6946) (BuildId: e2e0fbc52bf8cb7daf99595e16454e44217040f4)
    #3 setup_confname_tables /usr/src/python3.12-3.12.3-1ubuntu0.1/build-shared/../Modules/posixmodule.c:13572:9 (libpython3.12.so.1.0+0x3b6946)
    #4 posixmodule_exec /usr/src/python3.12-3.12.3-1ubuntu0.1/build-shared/../Modules/posixmodule.c:16905:9 (libpython3.12.so.1.0+0x3b6946)
    ...
  Previous write of size 8 at 0x7f24e35e16c0 by thread T18:
    #0 qsort_r <null> (463-interpreters+0x9bc2e) (BuildId: 4e94833395fedac62b3c4caa6921ff713e25cff1)
    #1 qsort <null> (463-interpreters+0x9bec7) (BuildId: 4e94833395fedac62b3c4caa6921ff713e25cff1)
    #2 setup_confname_table /usr/src/python3.12-3.12.3-1ubuntu0.1/build-shared/../Modules/posixmodule.c:13543:5 (libpython3.12.so.1.0+0x3b6946) (BuildId: e2e0fbc52bf8cb7daf99595e16454e44217040f4)
    #3 setup_confname_tables /usr/src/python3.12-3.12.3-1ubuntu0.1/build-shared/../Modules/posixmodule.c:13572:9 (libpython3.12.so.1.0+0x3b6946)
    #4 posixmodule_exec /usr/src/python3.12-3.12.3-1ubuntu0.1/build-shared/../Modules/posixmodule.c:16905:9 (libpython3.12.so.1.0+0x3b6946)

  Location is global '??' at 0x7f24e2e85000 (libpython3.12.so.1.0+0x75c880)

This only happened 3 times, and around the 42 thread generation step on average.

However, when executing this code under TSan, I got the following data race error that seems to indicate the same kind of memory corruption COULD be taking place.

A quick look suggests this TSan failure is a local data-race to posixmodule.c: There appears to be no protection from multiple threads importing posixmodule simultaneously, and hence multiple threads trying to simultaneously qsort in-place one of the three static tables.

This module is used on Windows, but I'm not sure if any of the macros enabling those qsort calls (HAVE_FPATHCONF, HAVE_PATHCONF, HAVE_CONFSTR, or HAVE_SYSCONF) are enabled in the Windows build. My guess would be no...

I don't know if overlapping qsorts on the same array will actually corrupt memory, but I doubt there's any guarantee it won't. (qsort_r doesn't help, that makes the comparison function thread-safe, not the array being sorted, according to the manpage.)

So probably a separate issue, and a more-focused/reliable MRE could be made for it.

I'm going to focus on the allocator issue for the time being, there's a lot of uncharted water in this, I wouldn't be surprised if like, 4 new issues are needed 😅

(If you would like to author PRs for docs change and whatnot, that would be great! You've clearly put much more time into this issue than I have.)

Ok, regarding the allocation issue, I'm worried about breaking code here, because theoretically someone could be relying on all threads being affected by modifying the Raw domain. Speculatively, we could modify what allocator PyMem_Raw* reaches for depending on the current thread state -- for example, if it's NULL, then it's not safe to try and call whatever Python has registered, and should just try and go for malloc. But even that could break code (think if someone allocates something without the GIL, such as in a Py_BEGIN_ALLOW_THREADS block, and then tries to free that memory after the GIL has been re-acquired), and wouldn't play well with debuggers.

I guess a fix could be to try and make the uses of _PyMem_SetDefaultAllocator thread-local somehow, but that's going to be a PITA.

I couldn't think of any way to make _PyMem_SetDefaultAllocator generally useable; I suspect our best bet would be to make it assert that it's only called at safe times: Before/during initialisation and finalisation of the main interpreter. And we can also note that PyMem_Raw* is not threadsafe during those times.

Although that's both pretty messy, somewhat complicated, and may break existing non-crashing code.

There's probably a similar reproduction case (in release builds only) even without the _PyMem_SetDefaultAllocator issue:

(UNTESTED)

#include <thread>

#include <Python.h>

int main()
{
    auto pre_config = PyPreConfig {};
    PyPreConfig_InitIsolatedConfig(&pre_config);
    if (auto status = Py_PreInitialize(&pre_config); PyStatus_Exception(status))
    {
        Py_ExitStatusException(status);
    }

    auto a = PyMem_RawMalloc(100);
    PyMem_SetupDebugHooks();
    PyMem_RawFree(a);

    return 0;
}

This is currently valid: As I recall the only valid place to call PyMem_SetupDebugHooks() (or PyMem_SetAllocator() when not a trivial wrapper for existing allocator) is between Py_PreInitialize and Py_Initialize.

However, this will reproduce the same bug, as PyMem_SetupDebugHooks() will change the allocate\free function pointer and a block allocated under the normal allocator will be free'd by the debug allocator.

So we can possibly fix both holes by noting that PyMem_Raw* is not thread-safe before the main interpreter's Py_InitializeFromConfig, and must not be called before any use of PyMem_SetAllocator which does not meet the stronger version of PyMem_SetAllocator's contract I proposed in the second paragraph of #123134 (comment):

any newly-set allocator after init needs to be able to free allocations created by the allocator it wraps

We would also need to extend the case of "Python has finish initializing" to also include "or PyMem_Raw* APIs have been called".

PyMem_SetupDebugHooks() does not currently meet this requirement, so it must not be called after PyMem_RawMalloc has been called, making this repro case invalid.

A stronger change that closes this second hole would be to deprecate-remove PyMem_SetupDebugHooks() in favour of the existing PyPreConfig parameter (like we did for Py_SetProgramName), plus a public API that can wrap a given allocator with debug hooks and the inverse unwrapping API, which could also be used to fix the small race condition in _PyMem_SetDefaultAllocator, i.e. the "new wrinkle".

Speaking as a user here: I've given up on the subinterpreter API -- for my specific use case, which involves spawning ~8 python threads for parallel processing of a bioinformatics workload (the solution is embedded in Go) -- subinterpreters weren't ready at all.

I couldn't get them to work except when importing a pthreads library in Go (I and another engineer couldn't get them to work inside Go goroutines)
Even after all the Cgo/pthreads magic in Go the solution was 2 times slower than spawning separate Python processes and communicating with them via stdin/stdout
I couldn't find a way to finalise the main python interpreter that wouldn't segfault. I decided this wasn't good enough for production.

Obviously these were unchartered waters, but my reading of the subinterpreter API was that this was ready for multithreaded use. It's perhaps worth updating the documentation:

Using Py_NewInterpreterFromConfig() you can create a sub-interpreter that is completely isolated from other interpreters, including having its own GIL. The most important benefit of this isolation is that such an interpreter can execute Python code without being blocked by other interpreters or blocking any others. Thus a single Python process can truly take advantage of multiple CPU cores when running Python code.

I couldn't think of any way to make _PyMem_SetDefaultAllocator generally useable; I suspect our best bet would be to make it assert that it's only called at safe times: Before/during initialisation and finalisation of the main interpreter. And we can also note that PyMem_Raw* is not threadsafe during those times.

My idea was to make internal uses of _PyMem_SetDefaultAllocator only affect the current thread, so functions that need to use the raw allocator wouldn't be breaking thread safety. But if there are issues with PyMem_SetupDebugHooks as well, then that wouldn't work. I was able to confirm your reproducer with PyMem_SetupDebugHooks.

A deprecation in favor of a config option is probably better, and then I guess updating thread safety notes about PyMem_Raw* is the best way to go.

Obviously these were unchartered waters, but my reading of the subinterpreter API was that this was ready for multithreaded use. It's perhaps worth updating the documentation:

It is ready, in the sense that we have all the necessary subinterpreter isolation ready, but my understanding of the problem is that we sort of don't have public APIs to really take advantage of it yet. I think my solution with manually attaching a thread state for the main interpreter and then switching to the subinterpreter was the valid way to do it--it's just not very pretty.

Specifically, which parts of subinterpreters "did not work", and what needs to be documented better? As far as I can see, most of the people that know about subinterpreters are the ones working on them, so it would be good to get some feedback from users on what needs to be made clearer.

@tbbie

I posted a C-only version of my pthreads code earlier on in this thread, but it must have been deleted (the thread got quite unwieldy so maybe somebody tried to clean it up and deleted that post). I wrote the code using the C API documentation as a starting point I believe @tbbie was commenting on it.

My personal question would have been: is that code correct?

I think having a pthreads template (which is effectively a multithreading standard for C) with works.

I've tried to dig out that code and it's something like this (the idea is run a tight loop from each subinterpreter to test for potential segfaults, etc...). I believe that in this particular instance I was able to call Py_Finalise (I wasn't able to call that in Cgo without segfault):

#ifndef SUBINTERPRETERS_C
#define SUBINTERPRETERS_C

#include <Python.h>
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>

// Define the number of threads to create
#define NUM_THREADS 4
// Define the upper limit for the range in our Python calculation
#define RANGE_LIMIT 10000000

// Configure the Python interpreter
PyInterpreterConfig config = {
    .check_multi_interp_extensions = 1,  // Check for multi-interpreter compatible extensions
    .gil = PyInterpreterConfig_OWN_GIL,  // Each interpreter gets its own GIL
};

// Define a struct to hold the result of each thread's execution
typedef struct {
    int error_code;
    char error_message[256];
} ThreadResult;

// Declare a global mutex to protect interpreter creation
//pthread_mutex_t interpreter_mutex = PTHREAD_MUTEX_INITIALIZER;

// Function that each thread will run
void *run_python_code(void *arg) {
    // Cast the argument to ThreadResult pointer
    ThreadResult *result = (ThreadResult *)arg;
    // Initialize the result
    result->error_code = 0;
    result->error_message[0] = '\0';

    PyThreadState *tstate = NULL;

    // Lock the mutex before creating a new interpreter
    //pthread_mutex_lock(&interpreter_mutex);
    
    // Create a new Python interpreter
    PyStatus status = Py_NewInterpreterFromConfig(&tstate, &config);
    
    // Unlock the mutex after creating the interpreter
    //pthread_mutex_unlock(&interpreter_mutex);

    // Check if interpreter creation was successful
    if (PyStatus_Exception(status)) {
        result->error_code = 1;
        snprintf(result->error_message, sizeof(result->error_message), 
                 "Couldn't spawn new interpreter");
        return NULL;
    }

    // Declare Python objects we'll use
    PyObject *pLocal, *pSum, *pRange, *pIter, *pItem;
    
    // Create a new dictionary for local variables
    pLocal = PyDict_New();
    // Create a Python long object initialized to 0
    pSum = PyLong_FromLong(0);
    // Add 'sum' to the local dictionary
    PyDict_SetItemString(pLocal, "sum", pSum);
    
    // Create a range object
    pRange = PyObject_CallFunction(PyObject_GetAttrString(PyImport_AddModule("builtins"), "range"), "l", RANGE_LIMIT);
    // Get an iterator from the range
    pIter = PyObject_GetIter(pRange);
    
    // Iterate over the range
    while ((pItem = PyIter_Next(pIter))) {
        // Add the current item to the sum
        PyObject *pNewSum = PyNumber_Add(pSum, pItem);
        Py_DECREF(pSum);
        pSum = pNewSum;
        Py_DECREF(pItem);
    }
    
    // Print the final sum
    PyObject_Print(pSum, stdout, 0);
    printf("\n");
    
    // Clean up Python objects
    Py_DECREF(pLocal);
    Py_DECREF(pSum);
    Py_DECREF(pRange);
    Py_DECREF(pIter);

    // End the interpreter
    Py_EndInterpreter(tstate);
    return NULL;
}

int main() {
    // Initialize the Python interpreter
    Py_Initialize();

    // Declare arrays to hold thread IDs and results
    pthread_t threads[NUM_THREADS];
    ThreadResult results[NUM_THREADS];
    int num_threads_created = 0;

    // Create threads
    for (int i = 0; i < NUM_THREADS; i++) {
        int ret = pthread_create(&threads[i], NULL, run_python_code, &results[i]);
        if (ret) {
            fprintf(stderr, "Error creating thread %d: %d\n", i, ret);
            break;
        } else {
            printf("Successfully created thread %d:\n", i);
        }
        num_threads_created++;
    }

    // Wait for all created threads to finish
    for (int i = 0; i < num_threads_created; i++) {
        pthread_join(threads[i], NULL);

        // Print any errors that occurred in the thread
        if (results[i].error_code) {
            fprintf(stderr, "Thread %d error: %s\n", i, results[i].error_message);
        }
        printf("Thread %d completed with error code: %d\n", i, results[i].error_code);
    }
    
    // Destroy the mutex
    //pthread_mutex_destroy(&interpreter_mutex);

    // Finalize the Python interpreter
    Py_Finalize();
    return 0;
}

#endif

At a glance, the bug there is calling Py_NewInterpreterFromConfig without the GIL. See what I did earlier in the thread with PyThreadState_New to get around that.

As I said earlier, I've essentially moved on from this at this stage, and I'm using OS-level processes to manage the interaction between my Go and Python code, and it's been 2x faster than working with subinterpreters and much easier to work with. Personally I'm happy to leave it at that.

But again, speaking from the user perspective, I think having a working example in the documentation would have been helpful at the time (and by this I mean a full example: starting the threads, doing calculation and cleaning up, holding the GIL correctly, etc...). And I think the example should use the most standard multithreading library, like pthreads. I realise that the example will be quite long, and perhaps it will need to be linked from the documentation, but I think as it stands, it's essentially impossible to just read the documentation and come up with a correct example of multithreaded subinterpreter code.

@tbbie

I posted a C-only version of my pthreads code earlier on in this thread, but it must have been deleted (the thread got quite unwieldy so maybe somebody tried to clean it up and deleted that post). I wrote the code using the C API documentation as a starting point I believe @tbbie was commenting on it.

GitHub folds the middle of long issues, clicking this link should show it.

What appears to be the necessary dance for holding the GIL to create an interpreter on the new thread is shown in #123134 (comment), everything from auto tstate = PyThreadState_New(interp); to PyThreadState_Delete(tstate);. The first three lines are what the threading module's ThreadHandle_Start and thread_run do, the rest of the stuff after Py_NewInterpreterFromConfig is basically to clean up the temporary thread state because Py_NewInterpreterFromConfig has created a new one for our separate-GIL interpreter. We've approximately open-coded PyGILState_Ensure for an explicit interpreter (the default is the main interpreter anyway) and and PyGILState_Release for an explicit thread state, but without a bunch of safety rails for nested calls.

So I believe something like this would work, but is not supported:

                auto gilstate = PyGILState_Ensure();
                assert(gilstate == PyGILState_UNLOCKED);
                auto main_interpreter_state = PyThreadState_GetCurrent();

                if (auto status = Py_NewInterpreterFromConfig(&state, &config); PyStatus_IsError(status))
                {
                    std::println("Failed to initialize thread state {}", count);
                    // NOTE: Need to clean up the thread state we created earlier.
                    // See https://github.com/python/cpython/blob/main/Python/crossinterp.c#L1833-L1841
                    return;
                }

                // Swap back to the old thread state, clear it, and then switch back to our new state.
                auto new_thread_state = PyThreadState_Swap(main_interpreter_state);
                assert(new_thread_state == state);
                PyGILState_Release(gilstate);

                auto temp_thread_state = PyThreadState_Swap(state);
                assert(temp_thread_state == NULL);

                // Do stuff with your new interpreter

I've just realised Py_EndInterpreter probably doesn't clean up the thread state, we probably should end with something like this, but PyThreadState_Clear depends on holding the GIL, so does that need to be done before Py_EndInterpreter? (Or per below, maybe that cleanup is done by Py_EndInterpreter already...)

Exploring more:

As it happens, the new module for exposing interpreter creation to Python drops the GIL before calling Py_NewInterpreterFromConfig so maybe holding the GIL or even having an active thread state is not necessary for Py_NewInterpreterFromConfig after all? This code-flow also throws away the new interpreter's thread state (except in a unit test, which is actually quite close to our use-case here), and then creates a new thread state on-demand for executing code.

Anyway, as I just linked, there's a smaller and safer C API prefixed with _PyXI_ which wraps up a bunch of this stuff but with a more-structured approach (and not public, I think; hopefully it will become so in future), and that's then exposed to Python code by the in-tree _interpreters module, and that's wrapped by the interpreters-PEP-734 module on pypi. In this new API, _PyXI_Enter and _PyXI_Exit are roughly analogous to PyGILState_Ensure and PyGILState_Release but for interpreters, and simplify a bunch of this stuff.

I also note that _PyXI_EndInterpreter also does not clear up a temporary thread state used for Py_EndInterpreter (nor does the open-coded version in the unit tests), so maybe Py_EndInterpreter does already clear up the thread state for the interpreter, and I just didn't notice the code to do that.

Yes, _PyXI* are Eric's magic functions that he's using to develop PEP-734, I think. We can't use them yet :)

As it happens, the new module for exposing interpreter creation to Python drops the GIL before calling Py_NewInterpreterFromConfig so maybe holding the GIL or even having an active thread state is not necessary for Py_NewInterpreterFromConfig after all?

It might be a bug. It's all private, so I doubt it's had the kind of stress testing like what we're doing here. Crossinterpreter stuff is all undocumented at the moment. If you read the discussion in my PR about the GIL and subinterpreters, the behavior that you see right now isn't necessarily intended, because it's untested and support is explicitly disallowed.

If you read the discussion in my PR about the GIL and subinterpreters, the behavior that you see right now isn't necessarily intended, because it's untested and support is explicitly disallowed.

I was only noting that if one puts up the code we got working here side-by-side with the implementation of PyGILState_Ensure and PyGILState_Release (and with the threading module) one can see that they're all doing the same operations, as a reassurance that we've ended up with the right way of getting a GIL established in a new thread and cleaning up afterwards. That's not really specific to sub-interpreters, it just happens that we create a sub-interpreter between the two calls, and then switch back to the thread state we had before that happened.

I'm aware that PyGILState_* is not supported in combination with sub-interpreters. The reason I described _PyXI_Enter as "PyGILState_Ensure for interpreters" is because it does exactly the same thing as PyGILState_Ensure except it takes an interpreter state as an argument, while PyGILState_Ensure uses the runtime's "gilstate.autoInterpreterState" pointer, which is always the main interpreter.

It might be a bug. It's all private, so I doubt it's had the kind of stress testing like what we're doing here.

To the best of my knowledge, at no point did taking the GIL or not before Py_NewInterpreterFromConfig affect the results of any repro case we've had here. So the stress testing we've done indicates that it's not required, even though the docs of Py_NewInterpreterFromConfig state that it is. (They also at one point say it's not required, but the overall vibe seems to be that it is formally required, so that's probably an oversight or an abstraction-break.)

That's also what I understood from reading the function, since before the point where it drops the GIL internally anyway, it does nothing but GIL-unneeded actions as far as I recall. The problems we saw inside Py_NewInterpreterFromConfig in 3.12 occur later in the function, when the new GIL has already been established (or the old GIL retaken), and are not related to the act of interpreter creation itself.

Anyway, it may become technically required in the future, and an API that is formally more restricted than its implementation is the correct way 'round IMHO.

I also don't want future discussions of this issue to get bogged down in unrelated API discussions, so being formally correct in repro cases is desirable.

@TBBle

Ok, @TBBle, I think it's time to get some PRs down. I don't know if there's anything that can be done about the PyMem_SetupDebugHooks problem, but we can fix the issues with _PyMem_SetDefaultAllocator internally by making it only affect the current thread, or possibly switch the internal uses that need the true system malloc over to _PyMem_RawMalloc, as that can't be hooked. Would you like to deal with updating the documentation for the allocator contract?

I'm pretty sure that #130287 will fix this.

Navigation Menu

Repository navigation

Creating an ungodly amount of sub interpreters in a short amount of time causes memory debug assertions. #123134

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

67 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Creating an ungodly amount of sub interpreters in a short amount of time causes memory debug assertions. #123134

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Activity

picnixz commented on Aug 19, 2024

ZeroIntensity commented on Aug 19, 2024

bruxisma commented on Aug 19, 2024

bruxisma commented on Aug 19, 2024

ZeroIntensity commented on Aug 19, 2024

bruxisma commented on Aug 19, 2024

ZeroIntensity commented on Aug 19, 2024

bruxisma commented on Aug 20, 2024

bruxisma commented on Aug 20, 2024

TBBle commented on Aug 20, 2024

67 remaining items

ZeroIntensity commented on Oct 9, 2024

ZeroIntensity commented on Oct 13, 2024

TBBle commented on Oct 13, 2024

picrin commented on Oct 13, 2024

ZeroIntensity commented on Oct 13, 2024

picrin commented on Oct 13, 2024

ZeroIntensity commented on Oct 13, 2024

picrin commented on Oct 13, 2024

TBBle commented on Oct 14, 2024

ZeroIntensity commented on Oct 14, 2024

TBBle commented on Oct 14, 2024

ZeroIntensity commented on Oct 30, 2024

ZeroIntensity commented on Feb 19, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions