Skip to content

Bump the default pickle protocol in shelve #78385

Closed
Closed
Bump the default pickle protocol in shelve#78385
Labels
3.10 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

Description

@serhiy-storchaka
serhiy-storchaka
opened on Jul 24, 2018
BPO 34204
Nosy @rhettinger, @vstinner, @avassalotti, @ambv, @serhiy-storchaka, @ZackerySpytz, @marco-c
PRs
  • bpo-34204: Use pickle.DEFAULT_PROTOCOL in shelve #19639
  • bpo-34204: Make the shelve library use Pickle's default protocol version instead of defaulting to version 3 #22751
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2020-10-29.09:46:33.380>
created_at = <Date 2018-07-24.09:16:42.390>
labels = ['type-feature', 'library', '3.10']
title = 'Bump the default pickle protocol in shelve'
updated_at = <Date 2020-10-29.09:46:33.379>
user = 'https://github.com/serhiy-storchaka'

    bugs.python.org fields:

    activity = <Date 2020-10-29.09:46:33.379>
actor = 'cheryl.sabella'
assignee = 'none'
closed = True
closed_date = <Date 2020-10-29.09:46:33.380>
closer = 'cheryl.sabella'
components = ['Library (Lib)']
creation = <Date 2018-07-24.09:16:42.390>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 34204
keywords = ['patch']
message_count = 5.0
messages = ['322281', '370137', '379812', '379813', '379861']
nosy_count = 7.0
nosy_names = ['rhettinger', 'vstinner', 'alexandre.vassalotti', 'lukasz.langa', 'serhiy.storchaka', 'ZackerySpytz', 'marco-c']
pr_nums = ['19639', '22751']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue34204'
versions = ['Python 3.10']

    Activity

    1. serhiy-storchaka commented on Jul 24, 2018

      @serhiy-storchaka
      serhiy-storchaka
      on Jul 24, 2018
      MemberAuthor

      The default pickle protocol is 4 now. But shelve still uses the pickle protocol 3. Shouldn't it be bumped? Shouldn't shelve use pickle.DEFAULT_PROTOCOL by default?

      Disadvantages:

      1. This will make shelve files incompatible with Python 3.3 by default.

      2. Protocol 4 adds 9 bytes of overhead in comparison with protocol 3. This can be too large for the shelve containing a lot of small objects. Maybe strip redundant frame header for small pickles?

    2. serhiy-storchaka
      added
      3.8 (EOL)end of life
      stdlibStandard Library Python modules in the Lib/ directory
      type-featureA feature request or enhancement
      on Jul 24, 2018

    3. vstinner commented on May 27, 2020

      @vstinner
      vstinner
      on May 27, 2020
      Member

      I wrote a short script to see the impact of file size depending on the protocol:
      ---

      import shelve
import os.path

print("== Short value ==")
for proto in (0, 1, 2, 3, 4, 5):
    filename = 'shelve-picklev%s' % proto
    with shelve.open(filename, protocol=proto) as db:
        assert db._protocol == proto
        for x in range(1000):
            db[str(x)] = str(x)
    print(f'Protocol {proto}: {os.path.getsize(filename)} bytes')
    os.unlink(filename)
print()
print("== Large value ==")
large_value = [str(x) for x in range(1000)]
for proto in (0, 1, 2, 3, 4, 5):
    filename = 'shelve-picklev%s' % proto
    with shelve.open(filename, protocol=proto) as db:
        assert db._protocol == proto
        for x in range(10):
            db[str(x)] = large_value
    print(f'Protocol {proto}: {os.path.getsize(filename)} bytes')
    os.unlink(filename)

      Output with Python 3.9.0b1 (on Fedora 32):
      ---
      == Short value ==
      Protocol 0: 90112 bytes
      Protocol 1: 94208 bytes
      Protocol 2: 94208 bytes
      Protocol 3: 94208 bytes
      Protocol 4: 94208 bytes
      Protocol 5: 94208 bytes

      == Large value ==
      Protocol 0: 139264 bytes
      Protocol 1: 139264 bytes
      Protocol 2: 139264 bytes
      Protocol 3: 139264 bytes
      Protocol 4: 98304 bytes
      Protocol 5: 98304 bytes
      ---

      For short string values, protocol 0 produces smaller files than protocol 1 and higher.

      For large value, protocol 4 and higher produce smaller files than protocol 3 and lower.

    4. marco-c commented on Oct 28, 2020

      @marco-c
      marco-cmannequin
      on Oct 28, 2020
      Mannequin

      I've opened #22751 to fix this, I know there was already a PR, but it seems to have been abandoned.

    5. ZackerySpytz commented on Oct 28, 2020

      @ZackerySpytz
      ZackerySpytzmannequin
      on Oct 28, 2020
      Mannequin

      It has not been abandoned.

    6. ZackerySpytz
      added
      3.10 (EOL)end of life
      and removed
      3.8 (EOL)end of life
      on Oct 28, 2020

    7. miss-islington commented on Oct 29, 2020

      @miss-islington
      miss-islington
      on Oct 29, 2020
      Contributor

      New changeset df59273 by Zackery Spytz in branch 'master':
      bpo-34204: Use pickle.DEFAULT_PROTOCOL in shelve (GH-19639)
      df59273

    8. csabella
      closed this as completedon Oct 29, 2020
    9. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    10. erlend-aasland
      added this to dbm and shelve issueson Jan 25, 2024
    11. erlend-aasland
      moved this to Done in dbm and shelve issueson Jan 25, 2024
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.10 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

      Projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions