Skip to content

TarFile.getmember cannot work on tar sourced directory over 100 characters #91387

Closed
Closed
TarFile.getmember cannot work on tar sourced directory over 100 characters#91387
Labels
3.10 (EOL)end of life3.11only security fixes3.9 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@cfernald
cfernald
mannequin
opened on Apr 5, 2022
BPO 47231
Nosy @bitdancer, @ethanfurman, @serhiy-storchaka, @akulakov, @cfernald
Files
  • tarfile_repro.py: Example of tarfile bug using arm gcc compiler tarfile
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2022-04-05.18:15:20.584>
labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
title = 'TarFile.getmember cannot work on tar sourced directory over 100 characters'
updated_at = <Date 2022-04-05.22:06:10.413>
user = 'https://github.com/cfernald'

    bugs.python.org fields:

    activity = <Date 2022-04-05.22:06:10.413>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2022-04-05.18:15:20.584>
creator = 'cfernald'
dependencies = []
files = ['50721']
hgrepos = []
issue_num = 47231
keywords = []
message_count = 2.0
messages = ['416793', '416796']
nosy_count = 5.0
nosy_names = ['r.david.murray', 'ethan.furman', 'serhiy.storchaka', 'andrei.avk', 'cfernald']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue47231'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

    Activity

    1. cfernald commented on Apr 5, 2022

      @cfernald
      cfernaldmannequin
      on Apr 5, 2022
      MannequinAuthor

      A fix was made to unify handling of the trailing slash in TarFile.getmember related to https://bugs.python.org/issue21987. This change fixed the <100 character case, but made it so directories over 100 character which come from a tar file can no longer be accessed through getmember, even if returned from getnames. This appears to be because internal to tarfile, member names still include the trailing slash on directories over 100 characters but getmember will always remove the trailing slash from the provided name so the comparison will always fail.

      A simple example of this is as follows using 3.10.4.

      1. Download: https://developer.arm.com/-/media/Files/downloads/gnu-a/10.3-2021.07/binrel/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu.tar.xz

      2. place attached python script in same directory

      3. run on 3.8.2 -> fails to get stripped version of path (line 16)

      4. run on 3.10.4 -> fails to get even unstripped version (line 12 & 16)

    2. cfernald
      added
      3.9 (EOL)end of life
      3.10 (EOL)end of life
      3.11only security fixes
      stdlibStandard Library Python modules in the Lib/ directory
      type-bugAn unexpected behavior, bug, or error
      on Apr 5, 2022

    3. ethanfurman commented on Apr 5, 2022

      @ethanfurman
      ethanfurman
      on Apr 5, 2022
      Member

      Nosied others from bpo-21987.

    4. ezio-melotti
      transferred this issue fromon Apr 10, 2022

    5. cfernald commented on Apr 11, 2022

      @cfernald
      cfernald
      on Apr 11, 2022
      ContributorAuthor

      Introduced Pull request 32423 to resolve this issue.

    6. zooba commented on Apr 27, 2022

      @zooba
      zooba
      on Apr 27, 2022
      Member

      Is there a logical reason to eagerly strip trailing slashes rather than only doing it for comparisons? (There might be, I just don't see it mentioned here, and I don't know the file format well enough to know if it's okay to lose that info.)

    7. 4 remaining items

    8. miss-islington
      added a commit that references this issue on Jun 17, 2022

      pythongh-91387: Strip trailing slash from tarfile longname directories (

      10cc1fc

    9. tiran commented on Jun 18, 2022

      @tiran
      tiran
      on Jun 18, 2022
      Member

      The new test is causing issues on wasm32-wasi.

      ======================================================================
ERROR: test_longname_directory (test.test_tarfile.GNUReadTest.test_longname_directory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Lib/test/test_tarfile.py", line 1034, in test_longname_directory
    os.rmdir(longdir)
    ^^^^^^^^^^^^^^^^^
OSError: [Errno 28] Invalid argument: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/'
======================================================================
ERROR: test_longname_directory (test.test_tarfile.PaxReadTest.test_longname_directory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Lib/test/test_tarfile.py", line 1034, in test_longname_directory
    os.rmdir(longdir)
    ^^^^^^^^^^^^^^^^^
OSError: [Errno 28] Invalid argument: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/'
----------------------------------------------------------------------
Ran 504 tests in 0.927s

    10. tiran commented on Jun 18, 2022

      @tiran
      tiran
      on Jun 18, 2022
      Member

      The rmdir syscall on WASI does not like the trailing slash.

    11. tiran
      added a commit that references this issue on Jun 18, 2022

      pythongh-91387: Fix tarfile test on WASI

      438d89c
    12. ethanfurman
      added 2 commits that reference this issue on Jun 18, 2022

      gh-91387: Fix tarfile test on WASI (GH-93984)

      dd78aae

      [3.11] pythongh-91387: Fix tarfile test on WASI (pythonGH-93984)

      ec54cd5

    13. cfernald commented on Jun 20, 2022

      @cfernald
      cfernald
      on Jun 20, 2022
      ContributorAuthor

      Thanks @ethanfurman ! sorry for the noise @tiran .

    14. miss-islington
      added 2 commits that reference this issue on Jun 21, 2022

      gh-91387: Strip trailing slash from tarfile longname directories (GH-…

      f6d777c

      gh-91387: Strip trailing slash from tarfile longname directories (GH-…

      46d0e1c
    15. tiran
      added 2 commits that reference this issue on Jun 21, 2022

      [3.11] pythongh-91387: Fix tarfile test on WASI (pythonGH-93984)

      1cf777f

      [3.11] gh-91387: Fix tarfile test on WASI (GH-93984) (#94074)

      b1b0656
    16. serhiy-storchaka
      added a commit that references this issue on Jun 26, 2022

      Revert 30935 defer bpo45162 to 312 (#94285)

      1ffde3e

    17. hauntsaninja commented on Oct 11, 2022

      @hauntsaninja
      hauntsaninja
      on Oct 11, 2022
      Contributor

      Thanks, looks like this has been completed

    18. hauntsaninja
      closed this as completedon Oct 11, 2022
    19. hauntsaninja
      moved this to Done in Tarfile issues 🗄️on Oct 11, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.10 (EOL)end of life3.11only security fixes3.9 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

      Projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions