Freeze all modules imported during startup.

@malemburg

BPO	45020
Nosy	@malemburg, @gvanrossum, @warsaw, @brettcannon, @nascheme, @rhettinger, @terryjreedy, @gpshead, @ronaldoussoren, @ncoghlan, @vstinner, @larryhastings, @tiran, @tiran, @tiran, @methane, @markshannon, @ericsnowcurrently, @indygreg, @lysnikolaou, @pablogsal, @miss-islington, @brandtbucher, @isidentical, @shihai1991, @FFY00, @softsol solutions
PRs	bpo-45020: Freeze the modules imported during startup. #28107 bpo-45020: Add -X frozen_modules=[on\|off] to explicitly control use of frozen modules. #28320 bpo-45020: Freeze some of the modules imported during startup. #28335 bpo-45020: Don't test IDLE with frozen module. #28344 [3.10] bpo-45020: Don't test IDLE with frozen module. (GH-28344) #28345 [3.9] bpo-45020: Don't test IDLE with frozen module. (GH-28344) #28346 bpo-45020: Drop the frozen .h files from the repo. #28375 bpo-45020: Revert "Drop the frozen .h files from the repo. (gh-28375)" #28380 bpo-45020: Drop the frozen .h files from the repo. #28392 bpo-45020: Freeze os, site, and codecs. #28398 bpo-45020: Fix build out of source tree #28410 bpo-45020: Fix some corner cases for frozen module generation. #28538 bpo-45020: Make "make all" output less noisy. #28554 bpo-45020: Use a common recipe for freezing modules. #28583 bpo-45020: Default to using frozen modules only on non-debug builds. #28590 bpo-45020: Do not freeze <pkg>/__init__.py twice. #28635 bpo-45020: Add more test cases for frozen modules. #28664 Also install __phello__, in addition to __hello__. #28665 bpo-45020: Identify which frozen modules are actually aliases. #28655 bpo-45020: Default to using frozen modules unless running from source tree. #28940 bpo-45020: Add tests for the -X "frozen_modules" option. #28997 bpo-45020: Fix strict-prototypes warning (GH-29755) #29755 bpo-45020: Fix strict-prototypes warning (GH-29755) #29755 bpo-45020: Fix strict-prototypes warning (GH-29755) #29755
Dependencies	bpo-45186: Marshal output isn't completely deterministic. bpo-45188: De-couple the Windows builds from freezing modules.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericsnowcurrently'
closed_at = <Date 2021-10-28.19:52:39.815>
created_at = <Date 2021-08-26.19:23:58.261>
labels = ['interpreter-core', 'type-feature', '3.11']
title = 'Freeze all modules imported during startup.'
updated_at = <Date 2021-11-24.19:01:47.807>
user = 'https://github.com/ericsnowcurrently'

bugs.python.org fields:

activity = <Date 2021-11-24.19:01:47.807>
actor = 'christian.heimes'
assignee = 'eric.snow'
closed = True
closed_date = <Date 2021-10-28.19:52:39.815>
closer = 'eric.snow'
components = ['Parser']
creation = <Date 2021-08-26.19:23:58.261>
creator = 'eric.snow'
dependencies = ['45186', '45188']
files = []
hgrepos = []
issue_num = 45020
keywords = ['patch']
message_count = 96.0
messages = ['400370', '400371', '400372', '400373', '400374', '400381', '400386', '400394', '400421', '400422', '400427', '400447', '400449', '400450', '400454', '400455', '400459', '400460', '400461', '400469', '400507', '400508', '400510', '400629', '400636', '400638', '400639', '400660', '400664', '400665', '400666', '400667', '400675', '400766', '400769', '400808', '400855', '400856', '401027', '401040', '401734', '401740', '401805', '401811', '401813', '401814', '401848', '401865', '401870', '401905', '401911', '401917', '401918', '401919', '401921', '401954', '401966', '401988', '402017', '402020', '402070', '402080', '402100', '402101', '402103', '402113', '402116', '402118', '402119', '402151', '402356', '402463', '402464', '402476', '402587', '402609', '402629', '402633', '402634', '402898', '402993', '403024', '403025', '403255', '403256', '403323', '403324', '404111', '404164', '404257', '404344', '405002', '405203', '405244', '406143', '406948']
nosy_count = 25.0
nosy_names = ['lemburg', 'gvanrossum', 'barry', 'brett.cannon', 'nascheme', 'rhettinger', 'terry.reedy', 'gregory.p.smith', 'ronaldoussoren', 'ncoghlan', 'vstinner', 'larry', 'christian.heimes', 'christian.heimes', 'christian.heimes', 'methane', 'Mark.Shannon', 'eric.snow', 'indygreg', 'lys.nikolaou', 'pablogsal', 'miss-islington', 'brandtbucher', 'BTaskaya', 'shihai1991', 'FFY00', 'santhu_reddy12']
pr_nums = ['28107', '28320', '28335', '28344', '28345', '28346', '28375', '28380', '28392', '28398', '28410', '28538', '28554', '28583', '28590', '28635', '28664', '28665', '28655', '28940', '28997', '29755', '29755', '29755']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue45020'
versions = ['Python 3.11']

Currently we freeze the 3 main import-related modules into the python binary (along with one test module). This allows us to bootstrap the import machinery from Python modules. It also means we get better performance importing those modules.

If we freeze modules that are likely to be used during execution then we get even better startup times. I'll be putting up a PR that does so, freezing all the modules that are imported during startup. This could also be done for any stdlib modules that are commonly imported.

(also see bpo-45019 and faster-cpython/ideas#82)

FYI, with my branch I'm getting a 15% improvement to startup for "./python -c pass".

I'm aware of two potentially problematic consequences to this change:

making changes to those source files for the modules will not be reflected during execution until after "make" is run
tricks to inject hooks ASAP (e.g. coverage.py swaps the encodings module) may lose their entry point

For the former, I'm not sure there's a way around it. We may consider the inconvenience worth it in order to get the performance benefits.

For the latter, the obvious solution is to introduce a startup hook (e.g. on the CLI) like we've talked about doing for years. (I wasn't able to find previous discussions on that topic after a quick search.)

Not sure whether you are aware, but the PyRun project I'm maintaining
does that and goes beyond this by freezing almost the complete stdlib
and statically linking most C extensions into a single binary:

https://www.egenix.com/products/python/PyRun/

Startup is indeed better, but not as much as you might think.
You do save stat calls and can share resources across processes.

The big time consumer is turning marshal'ed code objects back
into Python objects, though. If that could be made faster by
e.g. using a more efficient storage format such as one which is
platform dependent, it'd be a much bigger win than the freezing
approach.

The big time consumer is turning marshal'ed code objects back
into Python objects, though. If that could be made faster by
e.g. using a more efficient storage format such as one which is
platform dependent, it'd be a much bigger win than the freezing
approach.

That's something Guido has been exploring. :)

See: faster-cpython/ideas#84 (and others)

For the latter, the obvious solution is to introduce a startup hook

I'm not sure why I said "obvious". Sorry about that.

I noticed nedbat un-nosied himself. Probably he didn't realize you were
calling him out because it's possible this would affect coverage.py?

Whoa. os.path is not always an alias for posixpath, is it?

Nope. On Windows, os.path is "ntpath".

On Tue, Oct 5, 2021 at 11:31 AM Guido van Rossum <report@bugs.python.org> wrote:

Whoa. os.path is not always an alias for posixpath, is it?

Steve brought this to my attention a couple weeks ago. Bottom line:
the frozen module entry is only there for checks, not for actual
import, but should probably be removed regardless. See
https://bugs.python.org/issue45272.

New changeset b9cdd0f by Eric Snow in branch 'main':
bpo-45020: Default to using frozen modules unless running from source tree. (gh-28940)
b9cdd0f

could changes related to this be the cause of https://bugs.python.org/issue45506 ?

out of tree builds in main usually cannot pass key tests today. they often hang or blow up with strange exceptions.

Is #73126 only for UNIX?

I built on Windows with default options (PCbuild\build.bat) and it looks like the frozen modules are used by default even though I am running in the source directory. (I put a printf() call in unmarshal_frozen_code().)

I also put a printf() in is_dev_env() and found that it returns 0 on this check:

    /* If dirname() is the same for both then it is a dev build. */
    if (len != _Py_find_basename(stdlib)) {
        return 0;
    }

I assume that's because the binary (in my case at least) is at PCbuild\amd64\python.exe which is not the same as my current directory (which is the repo root).

On Mon, Oct 18, 2021 at 7:14 PM Guido van Rossum <report@bugs.python.org> wrote:

Is #73126 only for UNIX?

It wasn't meant to be. :(

I built on Windows with default options (PCbuild\build.bat) and it looks like the frozen modules are used by default even though I am running in the source directory. (I put a printf() call in unmarshal_frozen_code().)

I'll look into this.

New changeset 6afb285 by Eric Snow in branch 'main':
bpo-45020: Add tests for the -X "frozen_modules" option. (gh-28997)
6afb285

On Mon, Oct 18, 2021 at 7:14 PM Guido van Rossum <report@bugs.python.org> wrote:

Is #73126 only for UNIX?
I built on Windows with default options (PCbuild\build.bat) and it looks like the frozen modules are used by default even though I am running in the source directory. (I put a printf() call in unmarshal_frozen_code().)

FYI, I opened https://bugs.python.org/issue45651 for sorting this out.

I consider this done. There is some lingering follow-up work, for which I've created a number of issues:

https://bugs.python.org/issue45396 - -X frozen_modules=off ignores custom frozen modules
https://bugs.python.org/issue45395 - custom modules override frozen stdlib modules
https://bugs.python.org/issue45651 - frozen_modules default not right on Windows
https://bugs.python.org/issue45272 - "os.path" should be in the list of frozen modules
https://bugs.python.org/issue45653 - freeze encodings
https://bugs.python.org/issue45654 - freeze runpy
https://bugs.python.org/issue45660 - freeze argparse
https://bugs.python.org/issue45661 - freeze other stdlib modules
https://bugs.python.org/issue45273 - os-specific modules frozen unnecessarily?
https://bugs.python.org/issue45096 - update Tools/freeze
https://bugs.python.org/issue45379 - improving frozen-related errors
Add information about frozen stdlib modules. devguide#756 - add devguide info about frozen modules

New changeset 1cbaa50 by Guido van Rossum in branch 'main':
bpo-45696: Deep-freeze selected modules (GH-29118)
1cbaa50

New changeset 5c4b19e by Christian Heimes in branch 'main':
bpo-45020: Fix strict-prototypes warning (GH-29755)
5c4b19e

Navigation Menu

Repository navigation

Freeze all modules imported during startup. #89183

89 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Freeze all modules imported during startup. #89183

Description

Activity

ericsnowcurrently commented on Aug 26, 2021

ericsnowcurrently commented on Aug 26, 2021

ericsnowcurrently commented on Aug 26, 2021

malemburg commented on Aug 26, 2021

ericsnowcurrently commented on Aug 26, 2021

ericsnowcurrently commented on Aug 26, 2021

gvanrossum commented on Aug 27, 2021

89 remaining items

gvanrossum commented on Oct 5, 2021

larryhastings commented on Oct 6, 2021

ericsnowcurrently commented on Oct 6, 2021

ericsnowcurrently commented on Oct 16, 2021

gpshead commented on Oct 18, 2021

gvanrossum commented on Oct 19, 2021

ericsnowcurrently commented on Oct 19, 2021

ericsnowcurrently commented on Oct 25, 2021

ericsnowcurrently commented on Oct 28, 2021

ericsnowcurrently commented on Oct 28, 2021

gvanrossum commented on Nov 11, 2021

tiran commented on Nov 24, 2021

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions