ProcessPoolExecutor raises exception or hangs the process when shutdown(wait=False) and worker throws exception

Bug report

Bug description:

Below code demonstrates two problems that occur when
(a) worker process throws an exception, and
(b) shutdown(wait=False) is used

When max_workers < number of submitted tasks, raises exception TypeError: object of type 'NoneType' has no len()
When max_workers > number of submitted tasks, raises exception TypeError: object of type 'NoneType' has no len() AND then program hangs

There is no problem when shutdown(wait=True) is used or if the worker processes exit cleanly.

import concurrent.futures
import multiprocessing as mp
import time


def task(n: int) -> int:
    if n == 2:
        raise Exception("Not gonna do it")
    else:
        time.sleep(n)
        return n


def main() -> None:
    # max_workers=2 results in raised exception and program exits
    # max_workers=4 results in raised exception and program hangs
    #
    executor = concurrent.futures.ProcessPoolExecutor(
        max_workers=2, mp_context=mp.get_context("forkserver"), max_tasks_per_child=1
    )
    f1 = executor.submit(task, 1)
    f2 = executor.submit(task, 2)
    f3 = executor.submit(task, 3)
    result = 0
    try:
        result += f1.result()
        result += f2.result()
        result += f3.result()
        print(f"Result = {result}")
    except Exception as e:
        print(f"Exception while getting result : {e}")

    executor.shutdown(wait=False)


if __name__ == "__main__":
    main()

CPython versions tested on:

3.12

Operating systems tested on:

Linux, macOS

Linked PRs

After deeping in code, it seems to me that the bug is the run method of _ExecutorManagerThread.
When shutdown with wait set to False is called, this method always tries to ajust the process count.

cpython/Lib/concurrent/futures/process.py

Lines 361 to 363 in 8b26b23

    
           if process_exited: 
        
               with self.shutdown_lock: 
        
                   executor._adjust_process_count()

These lines should be replace as below:

         if process_exited:
              if not self.is_shutting_down():
                  with self.shutdown_lock:
                      executor._adjust_process_count()

This modification fixes the 2 cases

gh-132969 fix ProcessPoolExecutor exception and hang

Shutdown wait=False resets the ProcessPoolExecutor's state, hence returning from _adjust_process_counts() when the state has been reset fixes both issues. I submitted a PR #133220 with that fix and a couple of unittests to validate the fix.

p.s. my PR is on the 3.12 branch, and this is my first time contributing to cpython, so am not sure what steps (if any) I need to follow for porting the fix to 3.13 and main.

p.s. my PR is on the 3.12 branch, and this is my first time contributing to cpython, so am not sure what steps (if any) I need to follow for porting the fix to 3.13 and main.

Resubmitted a new pull request on main since the 3.12 branch is closed for bug fixes.

Deferred re: #133222 (comment).

pythongh-132969: Fix error/hang when shutdown(wait=False) and task ex…

[3.14] gh-132969: Fix error/hang when shutdown(wait=False) and task e…

[3.13] gh-132969: Fix error/hang when shutdown(wait=False) and task e…

@ogbiggles

Backported. Thank you @ogbiggles & and all reviewers!

pythongh-132969: Fix error/hang when shutdown(wait=False) and task ex…

gh-132969: update ACKS file (gh-133222) (#136144)

pythongh-132969: update ACKS file (pythongh-133222) (python#136144)

pythongh-132969: Fix error/hang when shutdown(wait=False) and task ex…

pythongh-132969: update ACKS file (pythongh-133222) (python#136144)

pythongh-132969: Fix error/hang when shutdown(wait=False) and task ex…

pythongh-132969: update ACKS file (pythongh-133222) (python#136144)

Navigation Menu

Repository navigation

`ProcessPoolExecutor` raises exception or hangs the process when `shutdown(wait=False)` and worker throws exception #132969

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

4 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	if process_exited:
	with self.shutdown_lock:
	executor._adjust_process_count()

ProcessPoolExecutor raises exception or hangs the process when shutdown(wait=False) and worker throws exception #132969

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

YvesDup commented on Apr 30, 2025

ogbiggles commented on Apr 30, 2025

ogbiggles commented on May 1, 2025

hugovk commented on May 5, 2025

4 remaining items

encukou commented on Jun 16, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`ProcessPoolExecutor` raises exception or hangs the process when `shutdown(wait=False)` and worker throws exception #132969