Repository navigation
Freeze the encodings module. #89816
Description
Activity
Currently we freeze all the modules imported during runtime initialization, except for the encodings module. It has a lot of submodules and this results in a lot of extra noise in builds. We hadn't frozen it yet because we were still ironing out changes related to frozen modules and the extra noise was a pain. We also waited because we weren't sure if we should freeze all the submodules or just the most likely ones to be used during startup. In the case of the latter, we were also blocked on having __path__ set on the package.
At this point there are no blockers. So we should freeze the encodings modules with either all submodules or the most commonly used subset.
encodings is a package. I think you first have to check whether mixing
frozen and non-frozen submodules are even supported. I've never tried
having only part of a package frozen.Freezing the whole package certainly works.
On Thu, Oct 28, 2021 at 12:15 PM Marc-Andre Lemburg
<report@bugs.python.org> wrote:encodings is a package. I think you first have to check whether mixing
frozen and non-frozen submodules are even supported. I've never tried
having only part of a package frozen.It works as long as __path__ is set properly, which it is now. FWIW,
I tested freezing only some of the submodules a while back and it
worked fine. That was using a different branch that I never merged
but it should be fine with the different change that got merged. Of
course, we'd need to verify that if we went that route.I just tested partially freezing the package, and it seems to working fine :)
On 30.10.2021 17:54, Filipe Laíns wrote:
I just tested partially freezing the package, and it seems to working fine :)
FWIW: I think it's best not bother and simply freeze the whole thing.It's mostly char mappings which compress well and there's a benefit
in sharing these using mmap (which the OS does for you with static
C data).I have already opened up the PR, but I can change if desired.
Eric, I have a simple reproducer for the issue:
This works:
$ LC_ALL=en_US.utf-8 TESTPATH=$(pwd)/Lib:$(pwd)/build/lib.linux-x86_64-3.11 ./Programs/_testembed test_init_setpath_config
This fails because it cannot load ISO-8859-1 / latin-1 codec
$ LC_ALL=en_US.latin1 TESTPATH=$(pwd)/Lib:$(pwd)/build/lib.linux-x86_64-3.11 ./Programs/_testembed test_init_setpath_config Python path configuration: PYTHONHOME = (not set) PYTHONPATH = (not set) program name = 'conf_program_name' isolated = 0 environment = 1 user site = 1 import site = 1 is in build tree = 0 stdlib dir = '' sys._base_executable = 'conf_executable' sys.base_prefix = '' sys.base_exec_prefix = '' sys.platlibdir = 'lib' sys.executable = 'conf_executable' sys.prefix = '' sys.exec_prefix = '' sys.path = [ '/home/heimes/dev/python/cpython/Lib', '/home/heimes/dev/python/cpython/build/lib.linux-x86_64-3.11', ] Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding Python runtime state: core initialized LookupError: unknown encoding: ISO-8859-1
Current thread 0x00007f9c42be6740 (most recent call first):
<no Python frame>With this patch I'm seeing that encodings.__path__ is not absolute and that __spec__ has an empty submodule_search_locations.
--- a/Lib/encodings/__init__.py +++ b/Lib/encodings/__init__.py @@ -98,9 +98,12 @@ def search_function(encoding): # module with side-effects that is not in the 'encodings' package. mod = __import__('encodings.' + modname, fromlist=_import_tail, level=0) - except ImportError: + except ImportError as e: # ImportError may occur because 'encodings.(modname)' does not exist, # or because it imports a name that does not exist (see mbcs and oem) + sys.stderr.write(f"exception: {e}\n") + sys.stderr.write(f"encodings.__path__: {__path__}\n") + sys.stderr.write(f"encodings.__spec__: {__spec__}\n") pass else: break
$ LC_ALL=en_US.latin1 TESTPATH=$(pwd)/Lib:$(pwd)/build/lib.linux-x86_64-3.11 ./Programs/_testembed test_init_setpath_config exception: No module named 'encodings.latin_1' encodings.__path__: ['encodings'] encodings.__spec__: ModuleSpec(name='encodings', loader=<class '_frozen_importlib.FrozenImporter'>, origin='frozen', submodule_search_locations=[]) exception: No module named 'encodings.iso_8859_1' encodings.__path__: ['encodings'] encodings.__spec__: ModuleSpec(name='encodings', loader=<class '_frozen_importlib.FrozenImporter'>, origin='frozen', submodule_search_locations=[])
It should have this search location:
>>> import encodings >>> encodings.__spec__ ModuleSpec(name='encodings', loader=<class '_frozen_importlib.FrozenImporter'>, origin='frozen', submodule_search_locations=['/home/heimes/dev/python/cpython/Lib/encodings'])
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields: