Skip to content

binascii.crc32 is not 64-bit clean when USE_ZLIB_CRC32 #82437

Closed
Closed
binascii.crc32 is not 64-bit clean when USE_ZLIB_CRC32#82437
Assignees
gpshead
Labels
3.10 (EOL)end of life3.11only security fixes3.7 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@zmk
zmk
mannequin
opened on Sep 23, 2019
BPO 38256
Nosy @gpshead, @Savier
PRs
  • bpo-38256: Fix binascii.crc32() when inputs are 4+GiB #32000
  • [3.10] bpo-38256: Fix binascii.crc32 large input. #32013
  • [3.9] bpo-38256: Fix binascii.crc32 large input. #32015
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/gpshead'
closed_at = <Date 2022-03-21.06:37:04.336>
created_at = <Date 2019-09-23.07:48:20.996>
labels = ['type-bug', '3.8', '3.9', '3.10', '3.11', '3.7', 'library']
title = 'binascii.crc32 is not 64-bit clean when USE_ZLIB_CRC32'
updated_at = <Date 2022-03-21.06:37:04.335>
user = 'https://bugs.python.org/zmk'

    bugs.python.org fields:

    activity = <Date 2022-03-21.06:37:04.335>
actor = 'gregory.p.smith'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2022-03-21.06:37:04.336>
closer = 'gregory.p.smith'
components = ['Library (Lib)']
creation = <Date 2019-09-23.07:48:20.996>
creator = 'zmk'
dependencies = []
files = []
hgrepos = []
issue_num = 38256
keywords = ['patch']
message_count = 10.0
messages = ['352992', '352993', '352994', '352995', '415577', '415578', '415628', '415636', '415667', '415668']
nosy_count = 4.0
nosy_names = ['gregory.p.smith', 'nadeem.vawda', 'zmk', 'iamsav']
pr_nums = ['32000', '32013', '32015']
priority = 'normal'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38256'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

    Activity

    1. zmk commented on Sep 23, 2019

      @zmk
      zmkmannequin
      on Sep 23, 2019
      MannequinAuthor

      Following code:
      ----------------------------

      import binascii, zlib
bigdata=memoryview(bytearray((1<<32) + 100))

print(binascii.crc32(bigdata))
crc = binascii.crc32(bigdata[:1000])
crc = binascii.crc32(bigdata[1000:], crc)
print(crc)

print(zlib.crc32(bigdata))
crc = zlib.crc32(bigdata[:1000])
crc = zlib.crc32(bigdata[1000:], crc)
print(crc)

      Outputs with python3.7

      $ python3.7 x.py 
2575877834
2838121701
2838121701
2838121701
    2. zmk
      added
      3.7 (EOL)end of life
      stdlibStandard Library Python modules in the Lib/ directory
      type-bugAn unexpected behavior, bug, or error
      on Sep 23, 2019

    3. Savier commented on Sep 23, 2019

      @Savier
      Saviermannequin
      on Sep 23, 2019
      Mannequin

      I've got
      Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)]
      2838121701
      2838121701
      2838121701
      2838121701

    4. zmk commented on Sep 23, 2019

      @zmk
      zmkmannequin
      on Sep 23, 2019
      MannequinAuthor

      Looking at the code, the bug is dependant on USE_ZLIB_CRC32 define and it's unfixed in master. Cause is zlib crc32 that takes length as integer, zlibmodule has workaround, binascii has not.

    5. zmk commented on Sep 23, 2019

      @zmk
      zmkmannequin
      on Sep 23, 2019
      MannequinAuthor

      Found zlibmodule fix: https://bugs.python.org/issue10276

    6. zmk
      added
      3.8 (EOL)end of life
      3.9 (EOL)end of life
      on Sep 23, 2019

    7. gpshead commented on Mar 19, 2022

      @gpshead
      gpshead
      on Mar 19, 2022
      Member
      $ python3.8
Python 3.8.10 (default, Nov 26 2021, 20:14:08) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import binascii, zlib
>>> bigdata=memoryview(bytearray((1<<32) + 100))
>>> 
>>> print(binascii.crc32(bigdata))
2838121701
>>> crc = binascii.crc32(bigdata[:1000])
>>> crc = binascii.crc32(bigdata[1000:], crc)
>>> print(crc)
2838121701
>>> 
>>> print(zlib.crc32(bigdata))
2838121701
>>> crc = zlib.crc32(bigdata[:1000])
>>> crc = zlib.crc32(bigdata[1000:], crc)
>>> print(crc)
2838121701
    8. gpshead
      removed
      3.8 (EOL)end of life
      3.9 (EOL)end of life
      on Mar 19, 2022
    9. gpshead
      closed this as completedon Mar 19, 2022

    10. gpshead commented on Mar 19, 2022

      @gpshead
      gpshead
      on Mar 19, 2022
      Member

      it depends on the build. USE_ZLIB_CRC32 causes it due to zlib's 32-bitness as noted my marko.

      $ ./python 
Python 3.11.0a6+ (heads/main-dirty:b3f2d4c8ba, Mar 19 2022, 15:32:04) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import binascii, zlib
>>> bigdata=memoryview(bytearray((1<<32) + 100))
>>> 
>>> print(binascii.crc32(bigdata))
2575877834
>>> crc = binascii.crc32(bigdata[:1000])
>>> crc = binascii.crc32(bigdata[1000:], crc)
>>> print(crc)
2838121701
>>> 
>>> print(zlib.crc32(bigdata))
2838121701
>>> crc = zlib.crc32(bigdata[:1000])
>>> crc = zlib.crc32(bigdata[1000:], crc)
>>> print(crc)
2838121701
>>>
    11. gpshead
      added
      3.8 (EOL)end of life
      3.9 (EOL)end of life
      3.10 (EOL)end of life
      3.11only security fixes
      on Mar 19, 2022
    12. gpshead
      reopened this on Mar 19, 2022
    13. gpshead
      changed the title [-]binascii.crc32 is not 64-bit clean[/-] [+]binascii.crc32 is not 64-bit clean when USE_ZLIB_CRC32[/+] on Mar 19, 2022
    14. gpshead
      self-assigned this
      on Mar 19, 2022

    15. gpshead commented on Mar 20, 2022

      @gpshead
      gpshead
      on Mar 20, 2022
      Member

      New changeset 9d1c4d6 by Gregory P. Smith in branch 'main':
      bpo-38256: Fix binascii.crc32() when inputs are 4+GiB (GH-32000)
      9d1c4d6

    16. gpshead commented on Mar 20, 2022

      @gpshead
      gpshead
      on Mar 20, 2022
      Member

      New changeset 4c989e1 by Gregory P. Smith in branch '3.10':
      [3.10] bpo-38256: Fix binascii.crc32 large input. (GH-32000) (GH-32013)
      4c989e1

    17. gpshead commented on Mar 21, 2022

      @gpshead
      gpshead
      on Mar 21, 2022
      Member

      New changeset 58a7e13 by Gregory P. Smith in branch '3.9':
      bpo-38256: Fix binascii.crc32 large input. (GH-32000) (GH-32013) (GH-32015)
      58a7e13

    18. gpshead commented on Mar 21, 2022

      @gpshead
      gpshead
      on Mar 21, 2022
      Member

      3.8 and older are in security fix only mode. 3.9+ have been fixed. realistically this was a rare issue with multiple workarounds.

      The 3.9 and 3.10 fixes were strictly bugfix only. The 3.11 fix included some performance improvements.

    19. gpshead
      closed this as completedon Mar 21, 2022
    20. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    3.10 (EOL)end of life3.11only security fixes3.7 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions