Repository navigation
Remove deep-freezing of code objects and modules. #108716
Description
Deep-freezing, as implemented has a number of problems:
- It is slow to build
- It does not fit into the normal build system,
makewill not recreate deep-frozen modules - Deep-freezing only makes sense if the objects are immutable, but code objects are not (at the VM level)
- It is a bit slower
- It gets in the way of other optimizations, notable faster loading from
pycfiles.
So let's remove it.
But let's keep the ability to deep-freeze objects.
The ability to deep-freeze objects into the executable is valuable, we already do it with strings, and deep-freezing whole immutable object graphs like the sys and builtin modules could improve start up considerably.
Activity
GH-108716: Turn off deep-freezing of code objects. (GH-108722)
malemburg commentedon Sep 8, 2023
Out of interest and unrelated to the other arguments you brought up: Why do you believe that removing deep-freeze for code objects slows down startup time ?
If I look at https://github.com/faster-cpython/benchmarking-public/blob/main/results/bm-20230827-3.13.0a0-1ad9bed/bm-20230827-linux-x86_64-faster%252dcpython-no_deep_freeze_3-3.13.0a0-1ad9bed-vs-3.11.0.md which measures removal of deep-freezing code objects against 3.11 which introduced this, the startup time has gone up by 18%.
The other differences and overall speedup shown in that benchmark are most likely not related to the deep-freeze change.
Background: I'm looking into using deep freeze for eGenix PyRun, which freezes (not deep freezes) most of the stdlib. I haven't done experiments yet (still trying to get things ported to 3.11, which is proving harder than usual due to the many CPython internal changes), so it'd be good to know whether it's worth starting into that adventure.
Thanks.
markshannon commentedon Sep 8, 2023
The main advantage of freezing is having to avoid I/O finding and reading .pyc files.
Because code objects are now mutable at the VM level, having them deep frozen adds a fair bit of complexity to initialization and finalization.
We are looking to deep freeze more objects, though. Strings, and tuples of strings are the obvious candidates.
erlend-aasland commentedon Sep 8, 2023
Is there more work to be done, or can we close this?
markshannon commentedon Sep 8, 2023
There is more to be done with deep freezing in general, but this specific task is done.
malemburg commentedon Sep 8, 2023
Actually, no 🙂 That's the main point of freezing modules using the freeze tool: you avoid I/O and reading PYC files by placing the byte code into a C array.
The purpose and intent of deep-freeze was to additionally avoid having to create the code objects from the byte code and put the ready to use objects into memory (again as a C array). So you skip the "read the byte code step" to create the objects and go straight to the objects. Well, at least that's what I had in mind when I discussed this with Guido some time ago.
Ok, mutable objects are probably not well stored in static C arrays 😉
I'd add bytes to that list, since the byte code in code objects is stored as a bytes object, AFAIK.
In any case, if this logic will change again for 3.13, it's probably better to wait some more before experimenting with deep-freezing the stdlib as I had planned for eGenix PyRun in one of the next releases.
Thanks.
pythongh-108716: make regen-global-objects no longer builds deepfreeze.c
pythongh-108716: make regen-global-objects no longer builds deepfreeze.c
gh-108716: make regen-global-objects no longer builds deepfreeze.c (#…
gh-108716: Cleanup remaining deepfreeze infrastructure (#116919)
pythongh-108716: Cleanup remaining deepfreeze infrastructure (python#…
gh-108716:: Remove _PyStaticCode_Init/Fini (#117141)
pythongh-108716: Cleanup remaining deepfreeze infrastructure (python#…
pythongh-108716:: Remove _PyStaticCode_Init/Fini (python#117141)
Squashed commit of the following:
Squashed commit of the following:
pythongh-108716: Cleanup remaining deepfreeze infrastructure (python#…
pythongh-108716:: Remove _PyStaticCode_Init/Fini (python#117141)
pythongh-108716: make regen-global-objects no longer builds deepfreez…