Minor InterpreterPoolExecutor Issues

Bug report

Bug description:

__main__ handling isn't quite right
non-pickleable-but-shareable objects fail
common code with test.support.interpreters should be shared

This has gh-132775 as a prerequisite.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

I ran into this with the following example:

import argparse
import concurrent.futures
import random
import statistics
import time


EXECUTORS = {
    "thread": concurrent.futures.ThreadPoolExecutor,
    "process": concurrent.futures.ProcessPoolExecutor,
    "interpreter": concurrent.futures.InterpreterPoolExecutor,
}


def pi(trials: int) -> float:
    Ncirc = 0
    ran = random.Random()

    for _ in range(trials):
        x = ran.uniform(-1, 1)
        y = ran.uniform(-1, 1)

        test = x * x + y * y
        if test <= 1:
            Ncirc += 1

    return 4.0 * (Ncirc / trials)


def pi_in_threads(threads: int, trials: int, executor: str) -> float:
    if threads == 0:
        return pi(trials)
    Executor = EXECUTORS[executor]
    with Executor(max_workers=threads) as executor:
        return statistics.mean(executor.map(pi, [trials // threads] * threads))


def main() -> None:
    parser = argparse.ArgumentParser(color=True, suggest_on_error=True)
    parser.add_argument("--threads", type=int, default=0)
    parser.add_argument("--trials", type=int, default=10_000_000)
    parser.add_argument("--executor", choices=EXECUTORS, default="thread")
    args = parser.parse_args()

    start = time.monotonic()
    π = pi_in_threads(args.threads, args.trials, args.executor)
    stop = time.monotonic()

    print(f"{args.trials} trials, {π = }, {stop - start:.4} s")


if __name__ == "__main__":
    main()

It's unable to run with --executor interpreter because it hits oncurrent.futures.interpreter.ExecutionFailed: AttributeError: module '__main__' has no attribute 'pi'. The other two work. I'd assume this might be similar to how ProcessPoolExectuor handles __main__. Good to see there's already an issue, though, just dropping the script in case it's helpful.

The syncobj parameter in the PEP isn't really explained, and it doesn't even seem to be present:

Traceback (most recent call last):
  File "/Users/henryschreiner/git/software/pybind11/concint.py", line 6, in <module>
    tasks = concurrent.interpreters.create_queue(syncobj=True)
TypeError: create() got an unexpected keyword argument 'syncobj'

I'm currently just sticking an integer into a queue, and in main interpreter I'm getting concurrent.interpreters._queues.UNBOUND out. I was wondering if this was related, but it's not even present AFAICT. I see unbounditems= in the code, which maybe is a new name? Is the plan to update the PEP, or just rely on CPython docs?

Is there some way to copy the item so it can be accessed after the interpreter is destroyed? This is my demo code:

Code:

import concurrent.interpreters
import concurrent.futures
import contextlib
import inspect

tasks = concurrent.interpreters.create_queue()

with (
    contextlib.closing(concurrent.interpreters.create()) as interp1,
    contextlib.closing(concurrent.interpreters.create()) as interp2
):

    with concurrent.futures.ThreadPoolExecutor() as pool:
        for interp in [interp1, interp2]:
            interp.prepare_main(tasks=tasks)
            pool.submit(interp.exec, inspect.cleandoc("""
                import time

                time.sleep(5)
                tasks.put(1)
                """
            ))

tasks.put(2)
print(tasks.get_nowait(), tasks.get_nowait(), tasks.get_nowait())

Indenting the final two lines fixes it, but curious if there's a way to get copies so I don't have to worry about the lifetimes, especially for things like integers.

gh-133485: Use interpreters.Interpreter in InterpreterPoolExecutor (g…

pythongh-133485: Use interpreters.Interpreter in InterpreterPoolExecu…

It's unable to run with --executor interpreter because it hits oncurrent.futures.interpreter.ExecutionFailed: AttributeError: module 'main' has no attribute 'pi'. The other two work. I'd assume this might be similar to how ProcessPoolExectuor handles main. Good to see there's already an issue, though, just dropping the script in case it's helpful.

FYI, I just landed the key change here, which will be in beta 4. It should take care of the discrepancy between InterpreterPoolExecutor and the others.

The syncobj parameter in the PEP isn't really explained, and it doesn't even seem to be present:

Yeah, it's gone. I forgot to remove it from the PEP. I'll fix that.

Is there some way to copy the item so it can be accessed after the interpreter is destroyed? This is my demo code:

Objects belong to interpreters and an object cannot outlive the interpreter that created it.

[3.14] gh-133485: Use interpreters.Interpreter in InterpreterPoolExec…

The issues I knew of have been addressed. If there are any other specific inconsistencies (or bugs) then feel free to open a new issue.

I've updated PEP 734: python/peps#4467.

pythongh-133485: Use interpreters.Interpreter in InterpreterPoolExecu…

Navigation Menu

Repository navigation

Minor InterpreterPoolExecutor Issues #133485

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Minor InterpreterPoolExecutor Issues #133485

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

henryiii commented on Jun 18, 2025

henryiii commented on Jun 18, 2025

ericsnowcurrently commented on Jun 19, 2025

ericsnowcurrently commented on Jun 19, 2025

ericsnowcurrently commented on Jun 19, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions