Repository navigation
Export zlib.{adler,crc}32_combine #134635
Description
Feature or enhancement
Proposal:
This is already provided by zlib, and is useful for computing a crc32 in multiple chunks (not linearly).
What I currently do is use ctypes, but that doesn't work on all environments (like Windows). I don't think we'll need a pure python fallback, just a wrapper of what's in zlib, equivalent of https://github.com/fastzip/fastzip/blob/9019107a6732fab9004e625b075faab96f29265a/fastzip/_crc32_combine.py#L63-L67
Has this already been discussed elsewhere?
This is a minor feature, which does not need previous discussion elsewhere
Links to previous discussion of this feature:
No response
Linked PRs
Metadata
Metadata
Assignees
Projects
- Status
Activity
picnixz commentedon May 24, 2025
cc @emmatyping (or is zstd not supporting this one? I don't know if you now want to take care of compression issues as well in general)
emmatyping commentedon May 24, 2025
@picnixz this is zlib related, not zstandard, but I am hoping to take care of more compression issues going forward!
emmatyping commentedon May 24, 2025
I think this request is reasonable, we probably would want to also add
adler32_combinefor completeness/symmetry.I don't think we should add
crc32_combine_gen/crc32_combine_opwithout additional motivation.picnixz commentedon May 24, 2025
Yeah, I actually misread the issue first :')
Great to hear!
Yes, sure. Note that we could also think about whether to add
binascii.crc32_combineas well (we have a minimal support for CRC32 even without zlib in binascii but I don't know if it would also make sense to exposecrc32_combinein this case). I wouldn't recommend it as a first iteration (and we anyway don't havebinascii.adler32)emmatyping commentedon May 24, 2025
I should note that if one is merely iterating over chunks of data, using the
valueargument tozlib.crc32(i.e. setting an initial checksum) should work to compute a running checksum.However, looking at the fastzip code, they are concurrently calculating the checksums and assembling them afterwards, so they cannot chain calls to
zlib.crc32.picnixz commentedon May 25, 2025
Actually, there was #82868 and #17048 for
adler32_combinebut it was not tested. I'll close the older issue and will add both as part of this specific issue.[-]Expose crc32_combine from zlib[/-][+]Export `zlib.{adler,crc}32_combine`[/+]gh-134635: add `zlib.{adler32,crc32}_combine` to combine checksums (#…
pythongh-134635: add `zlib.{adler32,crc32}_combine` to combine checks…
pythongh-134635: add `zlib.{adler32,crc32}_combine` to combine checks…
pythongh-134635: add `zlib.{adler32,crc32}_combine` to combine checks…