importlib.resources.files() doesn't work correctly when importlib library is compiled #123085
Description
Bug report
Bug description:
If importlib library is compiled (in particular _common.py), then any package, that uses bare files() to get its resources, breaks, because _infer_caller returns wrong frame (as mentioned in #123037 (comment), __file__ in
Reproducer:
import importlib
import os
import pathlib
import py_compile
import shutil
import sys
import tempfile
import textwrap
def compile(tempdir):
target_dir = pathlib.Path(tempdir) / 'cimportlib'
souce_dir = pathlib.Path(importlib.__file__).parent
shutil.copytree(souce_dir, target_dir, ignore=lambda *_: ['__pycache__'])
for dirpath, _, filenames in os.walk(target_dir):
for filename in filenames:
if filename != "_common.py":
continue
source_path = pathlib.Path(dirpath) / filename
cfile = source_path.with_suffix('.pyc')
py_compile.compile(source_path, cfile)
pathlib.Path.unlink(source_path)
def create_package(tempdir):
package_dir = pathlib.Path(tempdir) / 'somepkg'
package_dir.mkdir()
contents = {
"__init__.py": textwrap.dedent(
"""
import cimportlib.resources as res
val = res.files().joinpath('resource.txt').read_text(encoding='utf-8')
"""
),
"resource.txt": "data",
}
for file, content in contents.items():
path = pathlib.Path(package_dir) / file
path.write_text(content)
def main():
with tempfile.TemporaryDirectory() as tempdir:
compile(tempdir)
create_package(tempdir)
sys.path.insert(0, str(tempdir))
print(importlib.import_module('somepkg').val)
if __name__ == "__main__":
raise SystemExit(main())Expectation:
dataActual outcome:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpngnuw441/cimportlib/resources/resource.txt'It may be important for frozen python applications, as they don't include source code
CPython versions tested on:
3.12, CPython main branch
Operating systems tested on:
Linux, Windows
Linked PRs
- gh-123085: Fix issue in inferred caller when resources package has no source #123102
- [3.12] gh-123085: Fix issue in inferred caller when resources package has no source (GH-123102) #124021
- [3.13] gh-123085: Fix issue in inferred caller when resource package has no source (GH-123102) #124024
- gh-123085: _compile_importlib: Avoid copying sources before compilation #124131
- [3.12] gh-123085: _compile_importlib: Avoid copying sources before compilation (GH-124131) #128581
- [3.12] gh-123085: Remove double 'import os' added by PR #124021 #128600
- [3.13] gh-123085: _compile_importlib: Avoid copying sources before compilation (GH-124131) #137914
Activity
[-]importlib.files() doesn't work correctly when importlib library is compiled[/-][+]importlib.resources.files() doesn't work correctly when importlib library is compiled[/+]jaraco commentedon Aug 17, 2024
Did you encounter this issue in practice, or did you theorize the concern it while reviewing the code?
pythongh-123085: Fix issue in inferred caller when resources package …
Gatsik commentedon Aug 17, 2024
I encountered it in practice
chore(deps): Bump importlib-resources from 6.4.0 to 6.4.3 (#1971)
Bump importlib-resources from 6.1.1 to 6.4.3 (#8806)
Bump importlib-resources from 6.4.0 to 6.4.3 (#398)
Bump importlib-resources from 6.1.1 to 6.4.4 (#8915)
Bump importlib-resources from 5.13.0 to 6.4.4 (#91)
pythongh-123085: Fix issue in inferred caller when resources package …
5 remaining items
[3.13] pythongh-123085: Fix issue in inferred caller when resource pa…
encukou commentedon Sep 13, 2024
Some buildbots started failing when #123102 was merged, see:
I'll probably not have to investigate this week, but, my first guess would be that shutil.copytree copied some security metadata that says “this shouldn't be in
/tmp”.jaraco commentedon Sep 13, 2024
In the first log, I see:
The error is the same on the other buildbot. Both are Fedora, so perhaps that's a factor.
It's failing on this open call:
cpython/Lib/importlib/_bootstrap_external.py
Lines 206 to 207 in bbb36c0
It's conceivable the copy,
cpython/Lib/test/test_importlib/resources/test_files.py
Line 141 in bbb36c0
while it excludes
__pycache__could still be copying extant.pycfiles and those are failing to write atomically. Although, that's probably not it, because the failure is happening when writing out the temp file.I'm not familiar with any OS that has such behaviors. I'd not have considered it. I did read up a bit on the
noexecmount option. At first blush, I wouldn't expect such an option to conflict with_os.O_EXCL | _os.O_CREAT | _os.O_WRONLY.I don't think I'll make much more progress on this without a local reproducer. Do you think this error can be reproduced in a Docker container? In the meantime, is the buildbot breakage worth putting in a check to allow these tests to skip or xfail when they cannot write out the compiled files?
encukou commentedon Sep 17, 2024
I couldn't reproduce it locally. It might need some specific VM settings :(
First let me try an implementation that avoids the initial copy, which should be a bit faster too: #124131
gh-123085: _compile_importlib: Avoid copying sources before compilati…
pythongh-123085: _compile_importlib: Avoid copying sources before com…
[3.12] gh-123085: Fix issue in inferred caller when resources package…
pythongh-123085: _compile_importlib: Avoid copying sources before com…
[3.12] gh-123085: _compile_importlib: Avoid copying sources before co…
pythongh-123085: Remove double 'import os' added by PR python#124021
[3.12] gh-123085: Remove double 'import os' added by PR #124021 (#128600
godlygeek commentedon Aug 14, 2025
I've also encountered this issue in practice. It causes a failure when the stdlib is zipped, and
fails with
[3.13] gh-123085: Fix issue in inferred caller when resource package …
pythongh-123085: _compile_importlib: Avoid copying sources before com…
[3.13] gh-123085: _compile_importlib: Avoid copying sources before co…