zipfile may raise EOFError too early

I am using the ziprofs.py on Ubuntu. When I run this in Python-3.11.1 it works very well. With Python
3.12.0a4+ there are problems. In both cases, the same ziprofs.py and fuse.py files are used.
The only difference is the python version.

Unfortunately, I was unable to demonstrate problems using Linux
commands like wc, gzip, md5sum. Sofar, I only see this problem with proprietary closed source software.
It appears with different files.

When this EOFError occurs in
def read(self,path,size,offset,fh) of ziprofs.py
then the value of parameter "size" was 131072 and that of "offset" was 75755520.
The size of the file is 525537280 Bytes.
Strikingly, the low value of offset suggests, that the EOF is not yet riched.

Many thanks
C

File "/local/filesystem/git/ZipROFS/ziprofs.py", line 258, in read
    return f.read(size)
           ^^^^^^^^^^^^
  File "/local/python/2023_02_cpython-main/Lib/zipfile/__init__.py", line 948, in read
    data = self._read1(n)
           ^^^^^^^^^^^^^^
  File "/local/python/2023_02_cpython-main/Lib/zipfile/__init__.py", line 1018, in _read1
    data = self._read2(n)
           ^^^^^^^^^^^^^^
  File "/local/python/2023_02_cpython-main/Lib/zipfile/__init__.py", line 1052, in _read2
    raise EOFError
EOFError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/local/filesystem/git/ZipROFS/fuse.py", line 744, in _wrapper
    return func(*args, **kwargs) or 0
           ^^^^^^^^^^^^^^^^^^^^^
  File "/local/filesystem/git/ZipROFS/fuse.py", line 860, in read
    ret = self.operations('read', self._decode_optional_path(path), size, offset, fh)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/local/filesystem/git/ZipROFS/ziprofs.py", line 127, in __call__
    return super().__call__(op, self.root + path, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/local/filesystem/git/ZipROFS/fuse.py", line 1097, in __call__
    return getattr(self, op)(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/local/filesystem/git/ZipROFS/ziprofs.py", line 261, in read
    exit(999)
  File "<frozen _sitebuiltins>", line 26, in __call__
SystemExit: 999

@serhiy-storchaka

@serhiy-storchaka, @gpshead (as zipfile experts that were seen this week)

@sobolevn

It was working with erlend-aasland committed on Jul 22, 2022 f9b3706 and before
and failed since @sobolevn sobolevn committed on Oct 4, 2022 4f380db

This was validated by
git clone https://github.com/python/cpython.git
git checkout 4f380db
rm python; find . -name '*.o' -delete; ./configure; sed -i 's| -O3 | |1' Makefile && make

However without clone I see a carry over from previous checkouts and results are different depending what was checked out before.

Is there a more efficient way to build a python for a particular checkout?
Then I could further narrow it down.

Take a look at https://git-scm.com/docs/git-bisect

Thanks, I'll check it out.

…

On Wed, Feb 15, 2023 at 12:22 PM Nikita Sobolev ***@***.***> wrote: Take a look at https://git-scm.com/docs/git-bisect — Reply to this email directly, view it on GitHub <#101911 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASRZU6AM54BTPGD7JLNRWKLWXS35PANCNFSM6AAAAAAU3ZZ57E> . You are receiving this because you authored the thread.Message ID: ***@***.***>

I narrowed the regression down to below cpython commit. This is yours. This was easy, as this error showed up fast and reliable. However, there is a second one which shows up sporadically and inconstant. This is more time consuming and git bisect may take longer. It needs reading many files before the error is triggered. Python-3.11.1 is working well. Currently a git bisect script is running and hopefully finds the respective commit. Best Christoph commit 330f1d5 BAD1 Author: JuniorJPDJ ***@***.***> Date: Sun Aug 7 01:21:23 2022 +0200 gh-88339: enable fast seeking of uncompressed unencrypted zipfile.ZipExtFile (GH-27737) Avoid reading all of the intermediate data in uncompressed items in a zip file when the user seeks forward. Contributed by: @JuniorJPDJ On Thu, Feb 16, 2023 at 2:14 PM christoph gille ***@***.***> wrote:

…

Thanks, I'll check it out. On Wed, Feb 15, 2023 at 12:22 PM Nikita Sobolev ***@***.***> wrote: > Take a look at https://git-scm.com/docs/git-bisect > > — > Reply to this email directly, view it on GitHub > <#101911 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ASRZU6AM54BTPGD7JLNRWKLWXS35PANCNFSM6AAAAAAU3ZZ57E> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >

I found the other commit since when occasional, inconstant problems occur when using Ziprofs:
I contacted the commiter via Email. The problem cannot be reproduced with simple shell commands.
It occurs eventually when processing a large number of files by a prorietory software.

commit 26a162b BAD
Author: Jan Wolski jan@wolski.fi
Date: Sun May 15 17:49:19 2022 +0300
gh-89668: Optimize ZipFile file header processing algorithm to avoid unneeded IO(gh-25966)

commit 90e7230 GOOD
Author: Victor Stinner vstinner@python.org
Date: Sun May 15 11:19:52 2022 +0200
gh-92781: Avoid mixing declarations and code in C API (#92783)
Avoid mixing declarations and code in the C API to fix the compiler
warning: "ISO C90 forbids mixed declarations and code"
[-Werror=declaration-after-statement].

Thanks again for the great trick with git bisect. I found the other commit since when occasional, inconstant problems occur when using Ziprofs: I contacted the commiter via Email. The problem cannot be reproduced with simple shell commands. It occurs eventually when processing a large number of files by a prorietory software. That means there are two regressions which prevent using the recent cpython version. commit 26a162b **BAD** Author: Jan Wolski ***@***.***> Date: Sun May 15 17:49:19 2022 +0300 gh-89668: Optimize ZipFile file header processing algorithm to avoid unneeded IO(gh-25966) commit 90e7230 **GOOD** Author: Victor Stinner ***@***.***> Date: Sun May 15 11:19:52 2022 +0200 gh-92781: Avoid mixing declarations and code in C API (#92783) Avoid mixing declarations and code in the C API to fix the compiler warning: "ISO C90 forbids mixed declarations and code" [-Werror=declaration-after-statement]. On Sat, Feb 18, 2023 at 8:07 AM christoph gille ***@***.***> wrote:

…

I narrowed the regression down to below cpython commit. This is yours. This was easy, as this error showed up fast and reliable. However, there is a second one which shows up sporadically and inconstant. This is more time consuming and git bisect may take longer. It needs reading many files before the error is triggered. Python-3.11.1 is working well. Currently a git bisect script is running and hopefully finds the respective commit. Best Christoph commit 330f1d5 BAD1 Author: JuniorJPDJ ***@***.***> Date: Sun Aug 7 01:21:23 2022 +0200 gh-88339: enable fast seeking of uncompressed unencrypted zipfile.ZipExtFile (GH-27737) Avoid reading all of the intermediate data in uncompressed items in a zip file when the user seeks forward. Contributed by: @JuniorJPDJ On Thu, Feb 16, 2023 at 2:14 PM christoph gille < ***@***.***> wrote: > Thanks, I'll check it out. > > On Wed, Feb 15, 2023 at 12:22 PM Nikita Sobolev ***@***.***> > wrote: > >> Take a look at https://git-scm.com/docs/git-bisect >> >> — >> Reply to this email directly, view it on GitHub >> <#101911 (comment)>, >> or unsubscribe >> <https://github.com/notifications/unsubscribe-auth/ASRZU6AM54BTPGD7JLNRWKLWXS35PANCNFSM6AAAAAAU3ZZ57E> >> . >> You are receiving this because you authored the thread.Message ID: >> ***@***.***> >> >

I found out that the issues are very likely due to f.seek(offset).

A multithreaded application is loading zip entries via ziprofs
and causes above described errors.
By setting the thread number to one, the errors are gone.

Looking at the function
def read(self, path, size, offset, fh)
in ziprofs, the seek offset is jumping forth and back,
when several threads are reading from the same zip.

When only one thread is reading at a time,
the zip entry is read nicely sequentially.

zipfile is not threadsafe, that's nothing new
that probably means that locks in ziprofs are not working as intended

This looks like either a problem with ziprofs, a problem with user code, and/a feature request for thread safe zipfile. So I’m going to close this.

If you can reproduce a problem with single threaded code and just zipfile, then please reopen this and attach the reproducer.

If you want to suggest a thread safe zipfile, then the best thing to do is start a discussion on discuss.python.org.

I think I can reproduce this with epy (current master, commit c7a87f3), and Python 3.10.10 (from openSUSE/Tumbleweed packages):

stitny~/K/f/reading_the_books$ epy Twenty\ Four\ Years\ Later-ao3_29298975.epub 
Traceback (most recent call last):
  File "/home/matej/.bin/epy", line 8, in <module>
    sys.exit(main())
  File "/home/matej/archiv/knihovna/repos/epy/src/epy_reader/__main__.py", line 18, in main
    filepath = curses.wrapper(reader.start_reading, filepath)
  File "/usr/lib64/python3.10/curses/__init__.py", line 94, in wrapper
    return func(stdscr, *args, **kwds)
  File "/home/matej/archiv/knihovna/repos/epy/src/epy_reader/reader.py", line 1599, in start_reading
    reading_state_or_ebook = reader.read(reading_state)
  File "/home/matej/archiv/knihovna/repos/epy/src/epy_reader/reader.py", line 847, in read
    text_structure, toc_entries, contents = self.get_current_book_content(reading_state)
  File "/home/matej/archiv/knihovna/repos/epy/src/epy_reader/reader.py", line 797, in get_current_book_content
    content = self.ebook.get_raw_text(content_path)
  File "/home/matej/archiv/knihovna/repos/epy/src/epy_reader/ebooks/epub.py", line 188, in get_raw_text
    content = self.file.open(content_path).read()
  File "/usr/lib64/python3.10/zipfile.py", line 911, in read
    buf += self._read1(self.MAX_N)
  File "/usr/lib64/python3.10/zipfile.py", line 993, in _read1
    data += self._read2(n - len(data))
  File "/usr/lib64/python3.10/zipfile.py", line 1028, in _read2
    raise EOFError
EOFError
stitny~/K/f/reading_the_books$

It doesn’t happen with all EPubs, but quite often (like 50:50?).

(Yes, I checked with other EPub readers that Twenty Four Years Later-ao3_29298975.epub (renamed to *.zip because the GitHub rules for the attachment naming) is correct EPub).

AFAIK, epy is rather simple, certainly single-threaded, application.

I don't think anything looks that weird about the zip file, but do note the current version of epy-reader does use multiprocessing and has several comments that appear to be observed issues decompressing with multiprocessing. Can you try with that turned off?

This commited fix #127856 resolved a couple of bugs with regards to concurrent reading of files in a zip. If it is related to multiprocesssing, it may be worth checking if this bug still exists in newer versions of python

(Un)fortunately, epy got fixed in wustho/epy@ee3d693, so it doesn’t crash any more.

This commited fix #127856 resolved a couple of bugs with regards to concurrent reading of files in a zip. If it is related to multiprocesssing, it may be worth checking if this bug still exists in newer versions of python

I'm pretty sure I'm encountering this error in version 3.12.0: I try to seek a file in an uncompressed zip and then read, raising an EOFError, if I catch that error, do some more seeking and reading from the beginning of the file, then try my original seek and read again, it doesn't fail. This is happening reliably every single time for this script and zip file, at least as I am writing this message.

I installed version 3.13.7 in parallel and ran the same combination, and everything works correctly every time. Then going back to version 3.12.0 it continues to fail reliably.

It seems I can at least workaround the bug by seeking to 0 and reading some data first.

I'm pretty sure I'm encountering this error in version 3.12.0:

Does it reproduce with the latest Python 3.12 version? Also, unless this is a security issue (which I believe at first glance it's not), we won't fix 3.12.

I'm pretty sure I'm encountering this error in version 3.12.0:

Does it reproduce with the latest Python 3.12 version? Also, unless this is a security issue (which I believe at first glance it's not), we won't fix 3.12.

It didn't occur to me to try on latest 3.12, but after trying yes it seems it is fixed. Sorry for the noise here, but maybe it can serve as some evidence that it's safe to close this issue.

Ok, so I'm going to close it as fixed. I don't know when it was fixed though but it might have been reported as part of a different issue. Thanks for checking on your side!

Navigation Menu

Repository navigation

`zipfile` may raise `EOFError` too early #101911

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

zipfile may raise EOFError too early #101911

Description

Activity

arhadthedev commented on Feb 15, 2023

christophgil commented on Feb 15, 2023

sobolevn commented on Feb 15, 2023

christophgil commented on Feb 16, 2023

christophgil commented on Feb 18, 2023

christophgil commented on Feb 20, 2023

christophgil commented on Feb 20, 2023

christophgil commented on Feb 23, 2023

jpruciak commented on Mar 15, 2023

ericvsmith commented on Mar 15, 2023

mcepl commented on Mar 31, 2023

thatch commented on May 20, 2025

danifus commented on May 20, 2025

mcepl commented on May 21, 2025

Ragzouken commented on Sep 22, 2025

Ragzouken commented on Sep 22, 2025

picnixz commented on Sep 22, 2025

Ragzouken commented on Sep 22, 2025

picnixz commented on Sep 22, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`zipfile` may raise `EOFError` too early #101911