Repository navigation
Add support for reading older zip formats in ZipFile #103412
Description
Hi,
ZipFile doesn't currently support reading files from archives that were created with the reduce/shrink/implode zip compression types.
These are older formats that aren't widely used but are still well supported by other standards such as InfoZip.
Some projects still need to have support for these (mine included), and there's no reason for Python not to support these natively, at least for reading files. I'd like to emphasize that this issue is only aiming at adding "read-only" support - not suggesting to add full support for writing as part of this issue - as this has even less utility today and the need to create such archives is pretty rare.
I don't see this as a full-blown "new feature" but rather enhancement of the existing support, I believe it will make the Python zip support closer to "feature complete".
I already created an implementation that works for me locally but it might not conform to the standards/methodologies used across the python eco-system so I do require some help, but this implementation was tested with many old zip files and so far no issue was found, it seems to work pretty well.
The following zip file contains the c libs necessary for adding the required support (including source code) and a patch for ZipFile.
The implementation can easily be adjusted to integrate into the current ZipFile codebase instead of the patching it.
zipfile_old.zip
Credits go to Jason Summers (https://github.com/jsummers) who created the c libraries used as a baseline (and that do the heavy lifting) for this implementation. The libs in the attached zip above are a slightly modified version of the these with the adjustments required to support ZipFile compressors/decompressors.
Original libs:
https://github.com/jsummers/oldunzip
Cheers!
Activity
barneygale commentedon Apr 10, 2023
As I understand it, these formats were only in use for a brief period during the SEA vs PKWARE dispute of the early 90s. They're probably too niche to support in the Python standard library, but perhaps you could publish a package to pypi.org?
distantvale commentedon Apr 10, 2023
Indeed they were only used pre 1994. Sure, I could go for a package but it might break at some point since the patch is a bit hacky - ZipFile doesn't support passing parameters to the compressors, so not ideal.
Still, if that's the best path, I can definitely do that.
ronaldoussoren commentedon Apr 11, 2023
That's something that might be easier to get in: What changes to APIs would be needed to make it easier to support additional compression methods in a library on PyPI? The appnote on zip also lists a number of modern compression algorithms we currently don't support because the algorithm itself isn't in the stdlib (such as xz and zstd).
distantvale commentedon Apr 11, 2023
So it depends on the compression itself but in order to keep it generic, I believe all the fields in the ZipInfo struct cover the current requirements as far as I can tell.
You can look in the attached zip, specifically at _zipfile.py, at the params that are passed to
_decompressor.init- compression type, compressed/uncompressed size, flag bits, I imagine other algorithms might need different parameters, but it's safe to assume ZipInfo fields cover 99% of the scenarios, as all metadata is there.erlend-aasland commentedon Jan 12, 2024
ISTM this is superseded by #113767.
HubertHolin commentedon Aug 11, 2024
Hi
There is at least one, extant, scientific instrument manufacturer which uses as the native file format for the software of one of its, still available, products, a ZipFile, several members of which use the shrink method. So strictly speaking, this is not entirely a legacy software question, it is still also a current software question.
Third-party libraries are indeed a way to solve the problem, but this requires some rather obscure foreknowledge of the nature of the problem one encounters, and is more complicated than having one solution that works "out of the box".
Finally, the OP's POC does work (I only had to adapt for very minor aspects of the OS I use), but it would definitely be better if his proposal (integration within the standard library's ZipFile) were adopted.
picnixz commentedon Dec 2, 2024
I think we should close this one since it's superseded by #113767. Alternatively, if the OP wants to support this feature natively, I suggest opening a discourse thread before: https://discuss.python.org/c/ideas/6.
distantvale commentedon Dec 15, 2024
closing as native support for these is deemd unnecessary.
might create a package at some point if there's more interest for it, for now I have custom support for my personal project.