Skip to content

Nested multiprocessing leads to AttributeError: is_fork_ctx with forkserver or spawn methods #108520

Closed
Closed
Nested multiprocessing leads to AttributeError: is_fork_ctx with forkserver or spawn methods#108520
Labels
topic-multiprocessingtype-bugAn unexpected behavior, bug, or error

Description

@bwoodsend
bwoodsend
opened on Aug 26, 2023

Bug report

Checklist

  • I am confident this is a bug in CPython, not a bug in a third-party project
  • I have searched the CPython issue tracker,
    and am confident this bug has not been reported before

CPython versions tested on:

3.11

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.11.5 (main, Aug 26 2023, 00:26:34) [GCC 12.2.1 20220924]

A clear and concise description of the bug:

Using nested multiprocessing (i.e. spawn a child process inside a child process) is broken as of Python 3.11.5, leading to an attribute error.

Process Process-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/io/pyi_multiprocessing_nested_process.py", line 15, in process_function
    process.start()
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/lib/python3.11/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "/usr/local/lib/python3.11/multiprocessing/synchronize.py", line 106, in __getstate__
    if self.is_fork_ctx:
       ^^^^^^^^^^^^^^^^
AttributeError: 'Lock' object has no attribute 'is_fork_ctx'
Results: [1]

Minimal code example below. Invoke with argument fork, forkserver or spawn. fork will work. forkserver and spawn will both raise the above error. All three variants work with Python 3.11.4.

import sys
import multiprocessing


def nested_process_function(queue):
    print("Running nested sub-process!")
    queue.put(2)


def process_function(queue):
    print("Running sub-process!")
    queue.put(1)

    process = multiprocessing.Process(target=nested_process_function, args=(queue,))
    process.start()
    process.join()


def main(start_method):
    multiprocessing.set_start_method(start_method)
    queue = multiprocessing.Queue()

    process = multiprocessing.Process(target=process_function, args=(queue,))
    process.start()
    process.join()

    results = []
    while not queue.empty():
        results.append(queue.get())

    print(f"Results: {results}")
    assert results == [1, 2]


if __name__ == '__main__':
    if len(sys.argv) != 2:
        raise SystemExit(f"Usage: {sys.argv[0]} fork|forkserver|spawn")

    main(sys.argv[1])

I believe that the source of this regression is 34ef75d which adds the attribute is_fork_ctx to multiprocessing.Lock() but doesn't update the pickle methods (__getstate__() and __setstate__()) so after being serialised and deserialised, the Lock() object looses that attribute.

The following patch, adding is_fork_ctx to the pickle methods, makes the above work again.

diff --git a/Lib/multiprocessing/synchronize.py b/Lib/multiprocessing/synchronize.py
index 2328d33212..9c5c2aada6 100644
--- a/Lib/multiprocessing/synchronize.py
+++ b/Lib/multiprocessing/synchronize.py
@@ -109,10 +109,11 @@ def __getstate__(self):
                                    'not supported. Please use the same context to create '
                                    'multiprocessing objects and Process.')
             h = sl.handle
-        return (h, sl.kind, sl.maxvalue, sl.name)
+        return (h, sl.kind, sl.maxvalue, sl.name, self.is_fork_ctx)
 
     def __setstate__(self, state):
-        self._semlock = _multiprocessing.SemLock._rebuild(*state)
+        self._semlock = _multiprocessing.SemLock._rebuild(*state[:4])
+        self.is_fork_ctx = state[4]
         util.debug('recreated blocker with handle %r' % state[0])
         self._make_methods()
### Tasks

Linked PRs

Activity

  1. bwoodsend
    added
    type-bugAn unexpected behavior, bug, or error
    on Aug 26, 2023
  2. AlexWaygood
    added
    topic-multiprocessing
    on Aug 26, 2023

  3. terryjreedy commented on Aug 26, 2023

    @terryjreedy
    terryjreedy
    on Aug 26, 2023
    Member

    @pitrou

  4. albanD commented on Aug 28, 2023

    @albanD
    albanD
    on Aug 28, 2023
    Contributor

    Sorry about that, proposed a fix at #108568

  5. carbocation commented on Aug 30, 2023

    @carbocation
    carbocation
    on Aug 30, 2023

    Just to add confirmation to what has already been documented here and to document for others using the gcloud cli and finding this in search, I'm on python3.11.5 as well (Python 3.11.5 (main, Aug 24 2023, 15:09:45) [Clang 14.0.3 (clang-1403.0.22.14.1)]) and am encountering this error with the gcloud cli tools.

    A search for the following error message brought me here:

    $ gcloud storage cp gs://bucket/file ./ --verbosity debug
DEBUG: Running [gcloud.storage.cp] with arguments: [--verbosity: "debug", DESTINATION: "./", SOURCE:1: "['gs://bucket/file']"]
DEBUG: Starting new HTTPS connection (1): storage.googleapis.com:443
DEBUG: https://storage.googleapis.com:443 "GET /storage/v1/b/bucket/o/data%2Ffile?alt=json&projection=noAcl HTTP/1.1" 200 896
Copying gs://bucket/file to file://./file
INFO: Using high CPU count, single-file workload config.
⠛ERROR: gcloud crashed (AttributeError): 'Lock' object has no attribute 'is_fork_ctx'

    This happens regardless of which gcloud version I upgrade or downgrade to:

    $ gcloud version
Google Cloud SDK 444.0.0
beta 2023.08.22
bq 2.0.97
core 2023.08.22
gcloud-crc32c 1.0.0
gsutil 5.25

    And I can successfully execute the command if I tell gcloud to use an older python version before 3.11.5, e.g.:

    $ CLOUDSDK_PYTHON=python3.10 gcloud storage cp gs://bucket/file ./
# (executes just fine)
  6. pitrou
    added a commit that references this issue on Aug 30, 2023

    gh-108520: Fix bad fork detection in nested multiprocessing use case (#…

    add8d45
  7. miss-islington
    added a commit that references this issue on Aug 30, 2023

    pythongh-108520: Fix bad fork detection in nested multiprocessing use…

    c62507f
  8. carljm
    added a commit that references this issue on Aug 30, 2023

    Merge branch 'main' into icompexc

    37377e6
  9. pitrou
    added a commit that references this issue on Aug 30, 2023

    [3.11] gh-108520: Fix bad fork detection in nested multiprocessing us…

    385b195
  10. Yhg1s
    added a commit that references this issue on Aug 30, 2023

    [3.12] gh-108520: Fix bad fork detection in nested multiprocessing us…

    320d398

  11. bwoodsend commented on Sep 2, 2023

    @bwoodsend
    bwoodsend
    on Sep 2, 2023
    Author

    The fix appears to be merged and backported. Shouldn't this be closed?

  12. pitrou commented on Sep 2, 2023

    @pitrou
    pitrou
    on Sep 2, 2023
    Member

    Hmm, interesting. I thought that the auto-merge feature would also auto-close the issue, but it didn't.

  13. pitrou
    closed this as completedon Sep 2, 2023

  14. samos123 commented on Sep 16, 2023

    @samos123
    samos123
    on Sep 16, 2023

    It doesn't seem like a fix has been released for Mac yet. I'm still hitting this issue after running brew upgrade since it's still using python 3.11.5. I did confirm the workaround provided by @carbocation works correctly.

  15. bwoodsend commented on Sep 16, 2023

    @bwoodsend
    bwoodsend
    on Sep 16, 2023
    Author

    Correct, the fix version will be 3.11.6 which doesn't exist yet.

  16. carbocation commented on Oct 3, 2023

    @carbocation
    carbocation
    on Oct 3, 2023

    looks like python3.11.6 was released today. I haven't tested the fix since I'm waiting for brew to support it (to keep my installations manageable).

  17. nfonrose commented on Oct 15, 2023

    @nfonrose
    nfonrose
    on Oct 15, 2023

    I can confirm that the problem no longer occurs with python 3.11.6 (tested with Google Cloud SDK 446.0.1 on MacOS 13.4). I had the issue with python 3.11.5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    topic-multiprocessingtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions