Can't open orphan path on bare importlib.resources.files() under zipapp

Bug report

Bug description:

Context: In Python 3.12 the function importlib.resources.files() was updated: https://docs.python.org/3/library/importlib.resources.html#importlib.resources.files

Changed in version 3.12: package parameter was renamed to anchor. anchor can now be a non-package module and if omitted will default to the caller’s module. package is still accepted for compatibility but will raise a DeprecationWarning. Consider passing the anchor positionally or using importlib_resources >= 5.10 for a compatible interface on older Pythons.

The Issue: When using importlib.resources.files(), I am able to open and read a text file included as package data in my package. However, if I zip up my package with zipapp and run the .pyz file, it is unable to read the text file.
If I use importlib.resources.files("mypkg") it will work in both cases.

Minimal example:

Create a project directory named mypkg with the src layout.

Create the following files.

mypkg/src/mypkg/a.py

import importlib.resources

def main():
    x = importlib.resources.files() / "data.txt"
    print(x.read_text())

mypkg/src/mypkg/data.txt

datadata

In pyproject.toml make sure to set up the entry script and include the data file.

mypkg/pyproject.toml

[project.scripts]
myscript = "mypkg.a:main"

[tool.setuptools.package-data]
mypkg = ["*.txt"]

Install the package into a fresh Python (virtual) environment with pip.
Run myscript and verify it prints datadata.

Now back in the mypkg project root folder, we will generate a .pyz file with zipapp and run the program from the .pyz file.

Run pip install . --target buildtemp --upgrade && python3 -m zipapp buildtemp --main mypkg.a:main -p '/usr/bin/env python3' -o mypkg.pyz in the terminal.

Run ./mypkg.pyz. This results in an error:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/win/code/sandbox/python/mypkg/./mypkg.pyz/__main__.py", line 3, in <module>
  File "/home/win/code/sandbox/python/mypkg/./mypkg.pyz/mypkg/a.py", line 6, in main
  File "/home/win/.local/share/mise/installs/python/3.12.4/lib/python3.12/importlib/resources/abc.py", line 89, in read_text
    with self.open(encoding=encoding) as strm:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/win/.local/share/mise/installs/python/3.12.4/lib/python3.12/importlib/resources/_adapters.py", line 139, in open
    raise FileNotFoundError("Can't open orphan path")
FileNotFoundError: Can't open orphan path

Note that if you change x = importlib.resources.files() / "data.txt" to x = importlib.resources.files("mypkg") / "data.txt", it will work when running the .pyz file.

CPython versions tested on:

3.12

Operating systems tested on:

Linux, macOS

Linked PRs

Pin irc library version to 20.4.0

@jaraco

ping @jaraco in case you missed this one

The code that handles the module inference is here:

cpython/Lib/importlib/resources/_common.py

Lines 75 to 104 in 6ae942f

    
           @functools.singledispatch 
        
           def resolve(cand: Optional[Anchor]) -> types.ModuleType: 
        
               return cast(types.ModuleType, cand) 
        
           @resolve.register 
        
           def _(cand: str) -> types.ModuleType: 
        
               return importlib.import_module(cand) 
        
           @resolve.register 
        
           def _(cand: None) -> types.ModuleType: 
        
               return resolve(_infer_caller().f_globals['__name__']) 
        
           def _infer_caller(): 
        
               """ 
        
               Walk the stack and find the frame of the first caller not in this module. 
        
               """ 
        
               def is_this_file(frame_info): 
        
                   return frame_info.filename == __file__ 
        
               def is_wrapper(frame_info): 
        
                   return frame_info.function == 'wrapper' 
        
               not_this_file = itertools.filterfalse(is_this_file, inspect.stack()) 
        
               # also exclude 'wrapper' due to singledispatch in the call stack 
        
               callers = itertools.filterfalse(is_wrapper, not_this_file) 
        
               return next(callers).frame

Next step is to determine what expectation is missed when running under zipapp.

I was able to replicate the issue without going through the wheel build process:

 draft 🐚 mkdir -p buildtemp/mypkg
 draft 🐚 cat > buildtemp/mypkg/a.py
import importlib.resources


def main():
    x = importlib.resources.files() / "data.txt"
    print(x.read_text())
 draft 🐚 echo datadata > buildtemp/mypkg/data.txt
 draft 🐚 py -m zipapp buildtemp --main mypkg.a:main -p '/usr/bin/env py' -o mypkg.pyz
 draft 🐚 ./mypkg.pyz
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/jaraco/draft/./mypkg.pyz/__main__.py", line 3, in <module>
    mypkg.a.main()
    ~~~~~~~~~~~~^^
  File "/Users/jaraco/draft/./mypkg.pyz/mypkg/a.py", line 6, in main
    print(x.read_text())
          ~~~~~~~~~~~^^
  File "/opt/python/lib/python3.13/importlib/resources/abc.py", line 89, in read_text
    with self.open(encoding=encoding) as strm:
         ~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.13/importlib/resources/_adapters.py", line 139, in open
    raise FileNotFoundError("Can't open orphan path")
FileNotFoundError: Can't open orphan path

The issue is that CompatibilityFiles is being activated for loading resources.

 draft 🐚 py -m pdb mypkg.pyz
> /Users/jaraco/draft/mypkg.pyz/__main__.py(2)<module>()
-> import mypkg.a
(Pdb) c
Traceback (most recent call last):
  File "/opt/python/lib/python3.13/pdb.py", line 2435, in main
    pdb._run(target)
    ~~~~~~~~^^^^^^^^
  File "/opt/python/lib/python3.13/pdb.py", line 2192, in _run
    self.run(target.code)
    ~~~~~~~~^^^^^^^^^^^^^
  File "/opt/python/lib/python3.13/bdb.py", line 651, in run
    exec(cmd, globals, locals)
    ~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jaraco/draft/mypkg.pyz/__main__.py", line 3, in <module>
    mypkg.a.main()
    ~~~~~~~~~~~~^^
  File "/Users/jaraco/draft/mypkg.pyz/mypkg/a.py", line 6, in main
    print(x.read_text())
          ~~~~~~~~~~~^^
  File "/opt/python/lib/python3.13/importlib/resources/abc.py", line 89, in read_text
    with self.open(encoding=encoding) as strm:
         ~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.13/importlib/resources/_adapters.py", line 139, in open
    raise FileNotFoundError("Can't open orphan path")
FileNotFoundError: Can't open orphan path
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /opt/python/lib/python3.13/importlib/resources/_adapters.py(139)open()
-> raise FileNotFoundError("Can't open orphan path")
(Pdb) u
> /opt/python/lib/python3.13/importlib/resources/abc.py(89)read_text()
-> with self.open(encoding=encoding) as strm:
(Pdb) u
> /Users/jaraco/draft/mypkg.pyz/mypkg/a.py(6)main()
-> print(x.read_text())
(Pdb) x
<importlib.resources._adapters.CompatibilityFiles.OrphanPath object at 0x41f4c584f90>

CompatibilityFiles gets employed when there's no proper resource provider found. I'll figure out why.

It's clear that the bare call to files() is getting CompatibiltyFiles when the call with an anchor returns a proper MultiplexedPath:

(Pdb) importlib.resources.files()
<importlib.resources._adapters.CompatibilityFiles.SpecPath object at 0x41f4c595a10>
(Pdb) importlib.resources.files('mypkg')
MultiplexedPath('/Users/jaraco/draft/mypkg.pyz/mypkg/')

By changing the main() function to emit the inferred caller's name,

def main():
    print(importlib.resources._common._infer_caller().f_globals['__name__'])

I can confirm that the inferred name is correct when it emits mypkg.a.

Interestingly, if I then replace the main function with a static 'mypkg.a', the reported error occurs.

def main():
    x = importlib.resources.files('mypkg.a') / "data.txt"
    print(x.read_text())

So the issue seems to be that files('mypkg') works as expected but files('mypkg.a') does not, and it's only coincidental that files() calls files('mypkg.a').

importlib.resources has a behavior that when given a module will resolve resources in the package of that module.

It seems that the zipimporter as attached to the module is not resolving a resource reader:

(Pdb) importlib.import_module('mypkg.a').__spec__.loader.get_resource_reader('mypkg.a')
(Pdb) importlib.import_module('mypkg.a').__spec__.loader.get_resource_reader('mypkg')
(Pdb) importlib.import_module('mypkg').__spec__.loader.get_resource_reader('mypkg')
<importlib.resources.readers.NamespaceReader object at 0x3df2e585410>

I'm slightly surprised to see NamespaceReader and not a ZipReader.

It seems the issue traces to the logic in get_resource_reader:

cpython/Lib/zipimport.py

Lines 258 to 270 in 6ae942f

    
               def get_resource_reader(self, fullname): 
        
                   """Return the ResourceReader for a package in a zip file. 
        
                   If 'fullname' is a package within the zip file, return the 
        
                   'ResourceReader' object for the package.  Otherwise return None. 
        
                   """ 
        
                   try: 
        
                       if not self.is_package(fullname): 
        
                           return None 
        
                   except ZipImportError: 
        
                       return None 
        
                   from importlib.readers import ZipReader 
        
                   return ZipReader(self, fullname)

When checking loader.is_package for mypkg.a, it returns False:

(Pdb) loader = importlib.import_module('mypkg.a').__spec__.loader
(Pdb) loader.is_package('mypkg.a')
False

In the early days of importlib.resources, all resources needed to be relative to a package (and not a module). Later, that restriction was relaxed to allow resources to be loaded relative to a module.

But it seems that get_resource_reader didn't get the corresponding treatment.

I've confirmed the same issue occurs when using importlib_resources:

 draft [1] 🐚 cat buildtemp/mypkg/a.py
import importlib_resources


def main():
    x = importlib_resources.files('mypkg.a') / "data.txt"
    print(x.read_text())
 draft 🐚 pip install -t buildtemp importlib_resources
Collecting importlib_resources
  Using cached importlib_resources-6.4.0-py3-none-any.whl.metadata (3.9 kB)
Using cached importlib_resources-6.4.0-py3-none-any.whl (38 kB)
Installing collected packages: importlib_resources
Successfully installed importlib_resources-6.4.0
 draft 🐚 py -m zipapp buildtemp --main mypkg.a:main -p '/usr/bin/env py' -o mypkg.pyz
 draft 🐚 ./mypkg.pyz
Traceback (most recent call last):
...
FileNotFoundError: /Users/jaraco/draft/./mypkg.pyz/mypkg/a/data.txt

I suspect the resolution is going to be update get_resource_reader to accept a module.

But first things first, let's figure out why this use-case isn't covered by the tests.

As a check, if I remove the is_package check:

 cpython main 🐚 git diff
diff --git a/Lib/zipimport.py b/Lib/zipimport.py
index f2724dd0268..c0a4abcd95d 100644
--- a/Lib/zipimport.py
+++ b/Lib/zipimport.py
@@ -256,16 +256,8 @@ def load_module(self, fullname):
 
 
     def get_resource_reader(self, fullname):
-        """Return the ResourceReader for a package in a zip file.
-
-        If 'fullname' is a package within the zip file, return the
-        'ResourceReader' object for the package.  Otherwise return None.
+        """Return the ResourceReader for a module in a zip file.
         """
-        try:
-            if not self.is_package(fullname):
-                return None
-        except ZipImportError:
-            return None
         from importlib.readers import ZipReader
         return ZipReader(self, fullname)

The routine then fails, but with a different error:

 draft 🐚 ./mypkg.pyz
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/jaraco/draft/./mypkg.pyz/__main__.py", line 3, in <module>
    mypkg.a.main()
    ~~~~~~~~~~~~^^
  File "/Users/jaraco/draft/./mypkg.pyz/mypkg/a.py", line 6, in main
    print(x.read_text())
          ~~~~~~~~~~~^^
  File "/Users/jaraco/code/python/cpython/Lib/zipfile/_path/__init__.py", line 426, in read_text
    with self.open('r', encoding, *args, **kwargs) as strm:
         ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jaraco/code/python/cpython/Lib/zipfile/_path/__init__.py", line 391, in open
    raise FileNotFoundError(self)
FileNotFoundError: /Users/jaraco/draft/./mypkg.pyz/mypkg/a/data.txt

That's encouraging, suggesting that the ZipReader is getting involved, but that it still doesn't honor resources adjacent to modules.

I started working on a test, seeking to extend these tests to zip files, but realized that the current test structure isn't easily amenable to extension to zip. In python/importlib_resources@3faf336, I've done some refactoring in order to leverage the zip fixtures for this purpose.

Expand ModuleFiles and ImplicitContextFiles tests to include zip files.

chore(deps): Bump importlib-resources from 6.4.0 to 6.4.3 (#1971)

Bump importlib-resources from 6.1.1 to 6.4.3 (#8806)

Bump importlib-resources from 6.4.0 to 6.4.3 (#398)

Bump importlib-resources from 6.1.1 to 6.4.4 (#8915)

Bump importlib-resources from 5.13.0 to 6.4.4 (#91)

gh-121735: Fix module-adjacent references in zip files (#123037)

pythongh-121735: Fix module-adjacent references in zip files (pythonG…

[3.12] pythongh-121735: Fix module-adjacent references in zip files (p…

[3.12] pythongh-121735: Fix module-adjacent references in zip files

[3.12] gh-121735: Fix module-adjacent references in zip files (gh-123037

[3.13] gh-121735: Fix module-adjacent references in zip files (GH-123037

PRs are merged; closing.

Navigation Menu

Repository navigation

Can't open orphan path on bare importlib.resources.files() under zipapp #121735

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

16 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	@functools.singledispatch
	def resolve(cand: Optional[Anchor]) -> types.ModuleType:
	return cast(types.ModuleType, cand)


	@resolve.register
	def _(cand: str) -> types.ModuleType:
	return importlib.import_module(cand)


	@resolve.register
	def _(cand: None) -> types.ModuleType:
	return resolve(_infer_caller().f_globals['__name__'])


	def _infer_caller():
	"""
	Walk the stack and find the frame of the first caller not in this module.
	"""

	def is_this_file(frame_info):
	return frame_info.filename == __file__

	def is_wrapper(frame_info):
	return frame_info.function == 'wrapper'

	not_this_file = itertools.filterfalse(is_this_file, inspect.stack())
	# also exclude 'wrapper' due to singledispatch in the call stack
	callers = itertools.filterfalse(is_wrapper, not_this_file)
	return next(callers).frame

	def get_resource_reader(self, fullname):
	"""Return the ResourceReader for a package in a zip file.

	If 'fullname' is a package within the zip file, return the
	'ResourceReader' object for the package. Otherwise return None.
	"""
	try:
	if not self.is_package(fullname):
	return None
	except ZipImportError:
	return None
	from importlib.readers import ZipReader
	return ZipReader(self, fullname)

Can't open orphan path on bare importlib.resources.files() under zipapp #121735

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

wimglenn commented on Aug 14, 2024

jaraco commented on Aug 14, 2024

jaraco commented on Aug 14, 2024

jaraco commented on Aug 14, 2024

jaraco commented on Aug 14, 2024

jaraco commented on Aug 14, 2024

jaraco commented on Aug 14, 2024

jaraco commented on Aug 14, 2024

16 remaining items

jaraco commented on Dec 24, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions