Skip to content

Add tarfile.TarPath #89812

Open
Open
Add tarfile.TarPath#89812
Labels
3.11only security fixesstdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

Description

@FFY00
FFY00
opened on Oct 28, 2021
BPO 45649
Nosy @jaraco, @ethanfurman, @FFY00, @barneygale

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-10-28.15:59:21.866>
labels = ['type-feature', 'library', '3.11']
title = 'Add tarinfo.Path'
updated_at = <Date 2022-01-02.00:01:26.196>
user = 'https://github.com/FFY00'

bugs.python.org fields:

activity = <Date 2022-01-02.00:01:26.196>
actor = 'barneygale'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-10-28.15:59:21.866>
creator = 'FFY00'
dependencies = []
files = []
hgrepos = []
issue_num = 45649
keywords = []
message_count = 6.0
messages = ['405194', '405206', '405210', '409479', '409481', '409482']
nosy_count = 4.0
nosy_names = ['jaraco', 'ethan.furman', 'FFY00', 'barneygale']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue45649'
versions = ['Python 3.11']

Linked PRs

Activity

  1. FFY00 commented on Oct 28, 2021

    @FFY00
    FFY00
    on Oct 28, 2021
    MemberAuthor

    It would be helpful to have a pathlib-compatible object in tarfile, similarly to zipfile.Path.

  2. FFY00
    added
    3.11only security fixes
    stdlibStandard Library Python modules in the Lib/ directory
    type-featureA feature request or enhancement
    on Oct 28, 2021

  3. jaraco commented on Oct 28, 2021

    @jaraco
    jaraco
    on Oct 28, 2021
    Member

    I vaguely recall exploring this concept and finding that tarfiles don’t supply the requisite interface because they’re not random access. I’m only 10% confident in that recollection, so worth exploring.

  4. FFY00 commented on Oct 28, 2021

    @FFY00
    FFY00
    on Oct 28, 2021
    MemberAuthor

    That is good to know. This isn't very high on my priority list, but I will try to explore when I have some time.

  5. barneygale commented on Jan 1, 2022

    @barneygale
    barneygalemannequin
    on Jan 1, 2022
    Mannequin

    It's possible to do, but will be a little slow due to the nature of tar files. They're a big linked list of files, so you need to do a bunch of reads/seeks from the start to the end to enumerate all files.

    I'd ask that we try to get bpo-24132 solved first. That would let us write:

        # tarfile.py
    class Path(pathlib.AbstractPath):
        def iterdir(self):
            ...
        def stat(self):
            ...

    We'd fill in a smallish number of abstract methods to get a full Path-compatible class with read_text(), is_symlink() etc methods.

  6. jaraco commented on Jan 1, 2022

    @jaraco
    jaraco
    on Jan 1, 2022
    Member

    I'd recommend not to block on bpo-24132. It's not obvious to me that subclassing would be valuable. It depends on how it's implemented, but in my experience, zipfile.Path doesn't and cannot implement the full interface of pathlib.Path. Instead zipfile.Path attempts to implement a protocol. At the time, the protocol was undefined, but now there exists importlib.resources.abc.Traversable (https://docs.python.org/3/library/importlib.html#importlib.abc.Traversable), the interface needed by importlib.resources. I'd honestly just create a standalone class, see if it can implement Traversable, and only then consider if it should implement a more complicated interface (such as something with symlink support or perhaps even later subclassing from pathlib.Path).

  7. barneygale commented on Jan 2, 2022

    @barneygale
    barneygalemannequin
    on Jan 2, 2022
    Mannequin

    If you're only aiming for Traversable compatibility, sure.

    The original bug description asks for something that's pathlib-compatible and similar to zipfile.Path, which goes beyond the Traversable interface in attempting to emulate pathlib.Path.

    The pathlib.Path interface is a good one - I see no reason it can't apply to zip and tar archives in full. Methods of Path objects already raise NotImplementedError if operations aren't supported (e.g. creating symlinks)

    Some prototyping from a couple years back, including a tar path implementation: https://github.com/barneygale/pathlab/tree/master/pathlab

  8. ezio-melotti
    transferred this issue fromon Apr 10, 2022
  9. serhiy-storchaka
    added this to Tarfile issues 🗄️on May 20, 2022

  10. encukou commented on May 3, 2023

    @encukou
    encukou
    on May 3, 2023
    Member

    Anyone implementing this should be aware that tar members don't have unique names.
    A path should generally refer to the last member of the given name (which would overwrite previous ones when extracting -- unless the previous members do shenanigans with symlinks/directories, but API for accessing individual members isn't expected to take that into account).
    TarFile API like getmember will take care of most of this, but not all: for example, a “listdir” operation should probably de-duplicate its result.

  11. 30 remaining items

  12. barneygale
    added a commit that references this issue on Jul 1, 2023

    GH-89812: Improve test for `pathlib.Path.stat()` (GH-106064)

    c2622a0
  13. manosriram
    added a commit that references this issue on Jul 1, 2023

    pythonGH-89812: Simplify creation of symlinks in pathlib tests. (pyth…

    17f4315
  14. barneygale
    added a commit that references this issue on Jul 3, 2023

    Merge branch 'main' into pythongh-89812-omgtarpath

    e4daac9

  15. barneygale commented on Jul 3, 2023

    @barneygale
    barneygale
    on Jul 3, 2023
    Contributor

    PR available that adds tarfile.TarPath! #106337

  16. barneygale commented on Jul 9, 2023

    @barneygale
    barneygale
    on Jul 9, 2023
    Contributor

    What should tarfile.TarPath do with symlinks in archives? In the PR above they're followed in the same circumstances as in pathlib.PosixPath, but that might not be desirable/safe default. We could add an initialiser argument that disabled following symlinks entirely - either omit them or represent them as regular files, somehow?

    If the answer isn't clear then I'd like to incubate this in a PyPI project for a while first

    @encukou I saw you were working on tarfile recently. Pinging in case this interests you!

  17. encukou commented on Jul 10, 2023

    @encukou
    encukou
    on Jul 10, 2023
    Member

    IMO, symlinks to files that are in the archive should continue to work.
    Perhaps relative symlinks to files within the archive root should work too, though these become dangerous if there are any directory symlinks pointing outside.
    In general, if an archive contains directory symlinks, you can only get the final destination of a file by extracting the archive (or by doing a very elaborate dry run, for which tarfile isn't suited).

    It's very much not clear-cut. A tarball is, essentially, a collection of procedural instructions that allow merging contents with an existing filesystem in ”interesting” ways, while TarPath would like to work with a final tree structure.
    IMO, to answer questions like this, we first need a good conceptual model of what TarPath represents.

    A PyPI project that handles the simple cases sounds like a good start.

  18. barneygale commented on Jul 12, 2023

    @barneygale
    barneygale
    on Jul 12, 2023
    Contributor

    Thank you, @encukou! I'll move the TarPath implementation in #106337 into an experimental PyPI package then, leaving just the (private) pathlib._VirtualPath in that PR.

  19. barneygale
    added a commit that references this issue on Jul 12, 2023

    Merge branch 'main' into pythongh-89812-omgtarpath

    e3f2509

  20. encukou commented on Jul 13, 2023

    @encukou
    encukou
    on Jul 13, 2023
    Member

    we first need a conceptual model of what TarPath represents

    FWIW, I won't make one any time soon -- I'm taking a break from tarfile after a few months of dealing with it.
    But if you want to bounce ideas off me, I'm here.

  21. barneygale
    added a commit that references this issue on Jul 19, 2023

    Merge branch 'main' into pythongh-89812-omgtarpath

    42fe91a
  22. barneygale
    added 2 commits that reference this issue on Aug 28, 2023

    Merge branch 'main' into pythongh-89812-omgtarpath

    6318eb7

    Merge branch 'main' into pythongh-89812-omgtarpath

    9b6377a
  23. barneygale
    added 2 commits that reference this issue on Sep 26, 2023

    Merge branch 'main' into pythongh-89812-omgtarpath

    b945cf8

    GH-89812: Add `pathlib._PathBase` (#106337)

    89966a6
  24. Glyphack
    added a commit that references this issue on Sep 2, 2024

    pythonGH-89812: Add `pathlib._PathBase` (python#106337)

    9e32fba
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.11only security fixesstdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions