Skip to content

dumbdbm should not commit if in read mode #73033

Closed
Closed
dumbdbm should not commit if in read mode#73033
Assignees
serhiy-storchaka
Labels
3.7 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

Description

@joanthanng
joanthanng
mannequin
opened on Dec 1, 2016
BPO 28847
Nosy @vadmium, @serhiy-storchaka, @joanthanng
PRs
  • [Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
    		•
    Files
  • dbm_dumb_readonly.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2016-12-07.09:12:22.519>
created_at = <Date 2016-12-01.03:50:06.052>
labels = ['3.7', 'type-feature', 'library']
title = 'dumbdbm should not commit if in read mode'
updated_at = <Date 2017-03-31.16:36:22.839>
user = 'https://github.com/joanthanng'

    bugs.python.org fields:

    activity = <Date 2017-03-31.16:36:22.839>
actor = 'dstufft'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2016-12-07.09:12:22.519>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2016-12-01.03:50:06.052>
creator = 'Jonathan Ng'
dependencies = []
files = ['45722']
hgrepos = []
issue_num = 28847
keywords = ['patch']
message_count = 13.0
messages = ['282133', '282158', '282166', '282167', '282170', '282218', '282219', '282264', '282265', '282266', '282269', '282271', '282602']
nosy_count = 4.0
nosy_names = ['python-dev', 'martin.panter', 'serhiy.storchaka', 'Jonathan Ng']
pr_nums = ['552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue28847'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']

    Activity

    1. joanthanng
      added
      type-bugAn unexpected behavior, bug, or error
      on Dec 1, 2016

    2. joanthanng commented on Dec 1, 2016

      @joanthanng
      joanthanngmannequin
      on Dec 1, 2016
      MannequinAuthor

      Or at the very least, if there is an OSError in _update, an error should be raised instead of ignoring this error.

      In the current state of the code, if there was an OSError while reading the dirfile, the dirfile would be overwritten with a blank file when it is closed.

    3. joanthanng
      changed the title [-]dumbdbm should not commit if[/-] [+]dumbdbm should not commit if in read mode[/+] on Dec 1, 2016

    4. serhiy-storchaka commented on Dec 1, 2016

      @serhiy-storchaka
      serhiy-storchaka
      on Dec 1, 2016
      Member

      Do you have concrete example when not ignoring an OSError in _update causes an issue? This is needed for writing tests.

    5. serhiy-storchaka
      added
      stdlibStandard Library Python modules in the Lib/ directory
      on Dec 1, 2016

    6. joanthanng commented on Dec 1, 2016

      @joanthanng
      joanthanngmannequin
      on Dec 1, 2016
      MannequinAuthor

      I'm not sure how to create an OSError.

      But perhaps something like this:

      '''
      from dbm import dumb
      import os

      db = dumb.open('temp', flag='n')
db['foo'] = 'bar'
db.close()

db = dumb.open('temp', flag='r')
print(len(db.keys()))
db.close

os.rename('temp.dir', 'temp_.dir') # simulates OSError
db = dumb.open('temp', flag='r')
os.rename('temp_.dir', 'temp.dir')
db.close()

db = dumb.open('temp', flag='r')
assert len(db.keys()) > 0
'''

    7. serhiy-storchaka commented on Dec 1, 2016

      @serhiy-storchaka
      serhiy-storchaka
      on Dec 1, 2016
      Member

      This example is too artificial.

      But there is a real issue: opening read-only files in read mode. Currently this causes a PermissionError on closing.

      For backward compatibility flags 'r' and 'w' are ignored. I.e. opening with 'r' and 'w' creates a file if it is not existing, and opening with 'r' allows modifying the database. Since 3.6 this emits deprecation warnings (bpo-21708). In future versions this will be an error.

      Proposed patch makes two changes:

      1. The index file no longer written if the database was not modified. This increases performance and adds a support of read-only files.

      2. A deprecation warning is raised when the index file is absent in 'r' and 'w' modes. In future versions this will be an error.

      May be the first change can be backported.

    8. serhiy-storchaka
      added
      3.7 (EOL)end of life
      type-featureA feature request or enhancement
      and removed
      type-bugAn unexpected behavior, bug, or error
      on Dec 1, 2016

    9. joanthanng commented on Dec 1, 2016

      @joanthanng
      joanthanngmannequin
      on Dec 1, 2016
      MannequinAuthor

      #1 makes sense to be backported.

      On Thu, Dec 1, 2016 at 8:41 PM, Serhiy Storchaka <report@bugs.python.org>
      wrote:

      Serhiy Storchaka added the comment:

      This example is too artificial.

      But there is a real issue: opening read-only files in read mode. Currently
      this causes a PermissionError on closing.

      For backward compatibility flags 'r' and 'w' are ignored. I.e. opening
      with 'r' and 'w' creates a file if it is not existing, and opening with 'r'
      allows modifying the database. Since 3.6 this emits deprecation warnings
      (bpo-21708). In future versions this will be an error.

      Proposed patch makes two changes:

      1. The index file no longer written if the database was not modified. This
        increases performance and adds a support of read-only files.

      2. A deprecation warning is raised when the index file is absent in 'r'
        and 'w' modes. In future versions this will be an error.

      May be the first change can be backported.

      ----------
      keywords: +patch
      stage: -> patch review
      type: behavior -> enhancement
      versions: +Python 3.7
      Added file: http://bugs.python.org/file45722/dbm_dumb_readonly.patch

      Python tracker <report@bugs.python.org>
      <http://bugs.python.org/issue28847\>

    10. serhiy-storchaka
      self-assigned this
      on Dec 2, 2016

    11. python-dev commented on Dec 2, 2016

      @python-dev
      python-devmannequin
      on Dec 2, 2016
      Mannequin

      New changeset 0516f54491cb by Serhiy Storchaka in branch '2.7':
      Issue bpo-28847: dubmdbm no longer writes the index file in when it is not
      https://hg.python.org/cpython/rev/0516f54491cb

    12. serhiy-storchaka commented on Dec 2, 2016

      @serhiy-storchaka
      serhiy-storchaka
      on Dec 2, 2016
      Member

      The first part is committed in 2.7. I'll commit it in 3.5-3.7 after releasing 3.6.0.

    13. vadmium commented on Dec 3, 2016

      @vadmium
      vadmium
      on Dec 3, 2016
      Member

      Serhiy: The Windows buildbots are having trouble removing read-only files. Maybe restore the write mode for the end of the test, or fix support.rmtree()? See <https://docs.python.org/3/library/shutil.html#rmtree-example\>.

    14. python-dev commented on Dec 3, 2016

      @python-dev
      python-devmannequin
      on Dec 3, 2016
      Mannequin

      New changeset dc7c86de9e13 by Martin Panter in branch '2.7':
      Issue bpo-28847: Fix spelling
      https://hg.python.org/cpython/rev/dc7c86de9e13

    15. vadmium commented on Dec 3, 2016

      @vadmium
      vadmium
      on Dec 3, 2016
      Member

      http://buildbot.python.org/all/builders/AMD64%20Windows8%202.7/builds/9/steps/test/logs/stdio
      ======================================================================
      ERROR: test_readonly_files (test.test_dumbdbm.DumbDBMTestCase)
      ----------------------------------------------------------------------

      Traceback (most recent call last):
  File "D:\buildarea\2.7.bolen-windows8\build\lib\test\test_dumbdbm.py", line 190, in test_readonly_files
    test_support.rmtree(dir)
  File "D:\buildarea\2.7.bolen-windows8\build\lib\test\test_support.py", line 289, in rmtree
    _rmtree(path)
  File "D:\buildarea\2.7.bolen-windows8\build\lib\test\test_support.py", line 245, in _rmtree
    _waitfor(_rmtree_inner, path, waitall=True)
  File "D:\buildarea\2.7.bolen-windows8\build\lib\test\test_support.py", line 199, in _waitfor
    func(pathname)
  File "D:\buildarea\2.7.bolen-windows8\build\lib\test\test_support.py", line 244, in _rmtree_inner
    _force_run(path, os.unlink, fullname)
  File "D:\buildarea\2.7.bolen-windows8\build\lib\test\test_support.py", line 194, in _force_run
    return func(*args)
WindowsError: [Error 5] Access is denied: '@test_3796_tmp\\db.dat'

----------------------------------------------------------------------
'test_dumbdbm' left behind directory '@test_3796_tmp'
. . .
test test_dumbdbm crashed -- <type 'exceptions.WindowsError'>: [Error 5] Access is denied: '@test_3796_tmp\\db.dat'
Traceback (most recent call last):
  File "D:\buildarea\2.7.bolen-windows8\build\PCbuild\..\Lib\test\regrtest.py", line 948, in runtest_inner
    test_time = time.time() - start_time
  File "D:\buildarea\2.7.bolen-windows8\build\PCbuild\..\Lib\test\regrtest.py", line 899, in __exit__
    restore(original)
  File "D:\buildarea\2.7.bolen-windows8\build\PCbuild\..\Lib\test\regrtest.py", line 876, in restore_files
    test_support.rmtree(fn)
. . .
'test_dumbdbm' left behind directory '@test_3796_tmp' and it couldn't be removed: [Error 5] Access is denied: '@test_3796_tmp\\db.dat'
Traceback (most recent call last):
  File "D:\buildarea\2.7.bolen-windows8\build\PCbuild\..\Lib\test\regrtest.py", line 1679, in <module>
    main()
  File "D:\buildarea\2.7.bolen-windows8\build\lib\contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "D:\buildarea\2.7.bolen-windows8\build\lib\test\test_support.py", line 765, in temp_cwd
    rmtree(name)

    16. python-dev commented on Dec 3, 2016

      @python-dev
      python-devmannequin
      on Dec 3, 2016
      Mannequin

      New changeset cb4a892e9b66 by Serhiy Storchaka in branch '2.7':
      Try to fix test.test_support.rmtree() on Windows for fixing bpo-28847 tests.
      https://hg.python.org/cpython/rev/cb4a892e9b66

    17. serhiy-storchaka commented on Dec 3, 2016

      @serhiy-storchaka
      serhiy-storchaka
      on Dec 3, 2016
      Member

      Thanks Martin for pointing on this. Now buildbots are green.

    18. python-dev commented on Dec 7, 2016

      @python-dev
      python-devmannequin
      on Dec 7, 2016
      Mannequin

      New changeset 0a74bc7ba462 by Serhiy Storchaka in branch '3.5':
      Issue bpo-28847: dbm.dumb now supports reading read-only files and no longer
      https://hg.python.org/cpython/rev/0a74bc7ba462

      New changeset 0c532bd28539 by Serhiy Storchaka in branch '3.6':
      Issue bpo-28847: dbm.dumb now supports reading read-only files and no longer
      https://hg.python.org/cpython/rev/0c532bd28539

      New changeset 2f59be67830c by Serhiy Storchaka in branch 'default':
      Issue bpo-28847: dbm.dumb now supports reading read-only files and no longer
      https://hg.python.org/cpython/rev/2f59be67830c

      New changeset a10361dfbf64 by Serhiy Storchaka in branch 'default':
      Issue bpo-28847: A deprecation warning is now emitted if the index file is missed
      https://hg.python.org/cpython/rev/a10361dfbf64

    19. serhiy-storchaka
      closed this as completedon Dec 7, 2016
    20. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    21. erlend-aasland
      added this to dbm and shelve issueson Jan 25, 2024
    22. erlend-aasland
      moved this to Done in dbm and shelve issueson Jan 25, 2024
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    3.7 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions