Repository navigation
zipfile with multiprocessing: zipfile.BadZipFile #83544
Description
Activity
maxime-lemonnier commented
on Jan 16, 2020 maxime-lemonniermannequinMannequinAuthorzipfile sometimes throws zipfile.BadZipFile when opening the same zip file from multiple processes
see attached file to reproduce the error. You'll need a zipfile with multiple files in it to reproduce.
maxime-lemonnier commented
on Jan 16, 2020 maxime-lemonniermannequinMannequinAuthorHere's my console output:
python3 test_filesource.py lock file access_mode = file, nb processes = 1, res = 110289, 0.08039402961730957 ms/frame file access_mode = file, nb processes = 4, res = 110289, 0.32297492027282715 ms/frame lock access_mode = lock, nb processes = 4, res = 110289, 0.2950408458709717 ms/frame lock multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker result = (True, func(*args, **kwds)) File "/path/to/script/test_filesource.py", line 64, in read_small return fs_small[i%len(fs_small)][42] File "/path/to/script/test_filesource.py", line 55, in __getitem__ data_bytes = self.access( lambda archive: archive.read(member)) File "/path/to/script/test_filesource.py", line 27, in access_lock return f(self.archive) File "/path/to/script/test_filesource.py", line 55, in <lambda> data_bytes = self.access( lambda archive: archive.read(member)) File "/usr/lib/python3.6/zipfile.py", line 1337, in read with self.open(name, "r", pwd) as fp: File "/usr/lib/python3.6/zipfile.py", line 1419, in open % (zinfo.orig_filename, fname)) zipfile.BadZipFile: File name in directory '00000005.pkl' and header b'00000004.pkl' differ. """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/path/to/script/test_filesource.py", line 90, in <module> f(4, "lock") #crash File "/path/to/script/test_filesource.py", line 81, in f for i in pool.imap_unordered(read_small, frames): File "/usr/lib/python3.6/multiprocessing/pool.py", line 735, in next raise value zipfile.BadZipFile: File name in directory '00000005.pkl' and header b'00000004.pkl' differ.
It appears you are attempting to use the same already opened zipfile instance from multiple different processes at once. This won't even work with multiprocessing start methods other than "fork" as the other processes would not inherit the open file descriptors and the zipfile instance would fail to pickle/unpickle to go to the other process.
In the "fork"ing case, an existing zipfile instance already has an open file and is thus not safe for use from those processes as they'll all be attempting to use the same underlying inherited open file descriptor at once.
zipfileis not going to try to prevent this.Recommendation: Open a new zipfile instance from within each of the processes. parallel reading will work fine then as they no longer share any state.
Other recommendation: Do not use the "fork" multiprocessing start method. use "forkserver" or "spawn". you'll have far fewer problems. (the default start method on posix systems will change to one of those in the future)
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields: