Repository navigation
InterpreterPoolExecutor Failure When __name__ == '__main__': Idiom Not Used #136659
Description
Bug report
Bug description:
From @corona10:
import concurrent.futures
from concurrent.futures import InterpreterPoolExecutor, ThreadPoolExecutor
def fib(n):
if n < 2:
return n
return fib(n-1) + fib(n-2)
with InterpreterPoolExecutor(max_workers=5) as executor:
futures = {executor.submit(fib, n): n for n in range(10)}
for future in concurrent.futures.as_completed(futures):
n = futures[future]
data = future.result()
print(f"fib({n}): {data}")This fails with:
Exception: AttributeError: module '__main__' has no attribute 'fib'
The above exception was the direct cause of the following exception:
concurrent.interpreters.NotShareableError: object could not be unpickled
The above exception was the direct cause of the following exception:
...
The failure goes away if the "executable" part of the script is wrapped in an if __name__ == '__main__': block.
This is because of how Interpreter.call() sends functions between interpreters. It mostly pickles them. On the other side, functions are looked up by name in the defining module. For the __main__ module, this is a problem.
The original script only runs in the main interpreter, so the function will not be found in the other interpreter's __main__ module. We must run the script in the other interpreter, but without polluting __main__ and avoiding executing anything but the desired function definition. We partially solve this by calling runpy.run_path() with a different module name than __main__.
multiprocessing faces a similar situation. See the "Safe importing of main module" entry in the https://docs.python.org/3/library/multiprocessing.html#the-spawn-and-forkserver-start-methods section.
The solution here is probably the same: document that users should apply if __name__ == '__main__':. We may also be able to accommodate this in the code somewhat, since we know when we are running the script in the other interpreter. However, there's a limit to how much we can do so without introducing nontrivial complexity.
CPython versions tested on:
CPython main branch
Operating systems tested on:
No response
Metadata
Metadata
Assignees
Projects
- Status
- Status
Activity
StanFromIreland commentedon Jul 14, 2025
Does this supersede #136603?
Tapeline commentedon Jul 14, 2025
I think subinterpreters are trickier than that, consider this example:You can make it work just by moving the import into the function itself.
Docs on multiprocessing you are referring to, say:
That seems false with subinterpreters. We should define a preferred way of handling these cases (maybe there are even more of them).P.s. there's an issue #136603 regarding this problem.
Upd:
if __name__ == "__main__"solves this problem too, there was just an issue with my tests.[-]InterpreterPoolExecutor Failure When "__name__ == '__main__:' Idiom Not Used[/-][+]InterpreterPoolExecutor Failure When "`__name__` == '`__main__`:" Idiom Not Used[/+][-]InterpreterPoolExecutor Failure When "`__name__` == '`__main__`:" Idiom Not Used[/-][+]InterpreterPoolExecutor Failure When `__name__ == __main__:` Idiom Not Used[/+][-]InterpreterPoolExecutor Failure When `__name__ == __main__:` Idiom Not Used[/-][+]InterpreterPoolExecutor Failure When `__name__ == '__main__'`: Idiom Not Used[/+]corona10 commentedon Jul 14, 2025
Interesting, it's fine at the main branch as you said but when I execute the same script at 3.14b3 which is installed by uv it's fails :(
corona10 commentedon Jul 14, 2025
But let me test with 3.14 branch right now.
corona10 commentedon Jul 14, 2025
Ah okay.
So seems like it was fixed between 3.14b3 and 3.14b4. I met this issue even with that installed by UV 3.14 (currently 3.14b3), so I thought that this was overall a failure to execute, but it looks not now.
corona10 commentedon Jul 14, 2025
I think a good approach for 3.14 would be to update the documentation or improve the error message to suggest using the
__name__ == '__main__'idiom to reduce confusion.ericsnowcurrently commentedon Jul 14, 2025
This is a duplicate of #136603. Sorry I didn't see it first. Looks the other was already closed so we can continue here.