Skip to content

Reintroduce bootstrap_python for freezing #90108

Closed
Closed
Reintroduce bootstrap_python for freezing#90108
Assignees
tiran
Labels
3.11only security fixesbuildThe build process and cross-buildtype-bugAn unexpected behavior, bug, or error

Description

@tiran
tiran
opened on Dec 1, 2021
BPO 45950
Nosy @gvanrossum, @ronaldoussoren, @tiran, @ned-deily, @ericsnowcurrently, @pablogsal
PRs
  • bpo-45950: Introduce Bootstrap Python again #29859
  • bpo-45950: Fix macOS framework builds of _bootstrap_python (GH-29936) #29936
  • bpo-45950: Try --enable-framework=yes without bootstrap_python fix #29939
  • bpo-45950: Try --enable-framework=yes without bootstrap_python fix #29939
    		•
    Dependencies
  • bpo-45696: "Deep-freeze": skip the marshal step by generating C code
  • bpo-45873: Stop using bootstrap_python for deep-freeze in UNIX builds
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/tiran'
closed_at = None
created_at = <Date 2021-12-01.16:26:37.476>
labels = ['type-bug', 'build', '3.11']
title = 'Reintroduce bootstrap_python for freezing'
updated_at = <Date 2021-12-07.02:25:38.567>
user = 'https://github.com/tiran'

    bugs.python.org fields:

    activity = <Date 2021-12-07.02:25:38.567>
actor = 'ned.deily'
assignee = 'christian.heimes'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2021-12-01.16:26:37.476>
creator = 'christian.heimes'
dependencies = ['45696', '45873']
files = []
hgrepos = []
issue_num = 45950
keywords = ['patch']
message_count = 11.0
messages = ['407467', '407592', '407776', '407777', '407779', '407782', '407790', '407801', '407827', '407830', '407889']
nosy_count = 6.0
nosy_names = ['gvanrossum', 'ronaldoussoren', 'christian.heimes', 'ned.deily', 'eric.snow', 'pablogsal']
pr_nums = ['29859', '29936', '29939', '29939']
priority = None
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45950'
versions = ['Python 3.11']

    Linked PRs

    Activity

    1. tiran commented on Dec 1, 2021

      @tiran
      tiran
      on Dec 1, 2021
      MemberAuthor

      bpo-45696 introduced a _bootstrap_python interpreter, which was used to create frozen and deepfrozen module files. bpo-45873 dropped the _bootstrap_python interpreter again. Instead Python used an existing Python installation to bootstrap the build.

      After some internal discussion we agreed that the approach is flawed and puts too much of a burden on users. Users have to install a fairly recent Python interpreter in order to build 3.11. This makes it harder to build Python on platforms with no or a too old Python version. It also complicated things for container builds and pyenv.

      Let's re-introduce a _bootstrap_python interpreter. Cross compiling will still require a build Python interpreter.

    2. tiran
      added
      3.11only security fixes
      buildThe build process and cross-build
      type-bugAn unexpected behavior, bug, or error
      on Dec 1, 2021
    3. tiran
      assigned
      tiran
      on Dec 1, 2021

    4. tiran commented on Dec 3, 2021

      @tiran
      tiran
      on Dec 3, 2021
      MemberAuthor

      New changeset 84ca123 by Christian Heimes in branch 'main':
      bpo-45950: Introduce Bootstrap Python again (bpo-29859)
      84ca123

    5. ned-deily commented on Dec 6, 2021

      @ned-deily
      ned-deily
      on Dec 6, 2021
      Member

      Framework builds on macOS now fail with a bus error when trying to run the newly built _bootstrap_python. A simplified example:

      $ ./configure --enable-framework=$PWD/Library/Frameworks
checking for git... found
checking build system type... x86_64-apple-darwin20.6.0
checking host system type... x86_64-apple-darwin20.6.0
checking for Python interpreter freezing... ./_bootstrap_python
checking for python3.11... no
checking for python3.10... no
checking for python3.9... no
checking for python3.8... no
checking for python3.7... no
checking for python3.6... no
checking for python3... python3
checking Python for regen version... Python 3.8.9
[...]
creating Modules/Setup.local
creating Makefile

      pkg-config is missing. Some dependencies may not be detected correctly.

      If you want a release build with all stable optimizations active (PGO, etc),
      please run ./configure --enable-optimizations

      $ make
[...]
gcc     -o _bootstrap_python Modules/getbuildinfo.o Parser/token.o  Parser/pegen.o Parser/pegen_errors.o Parser/action_helpers.o Parser/parser.o Parser/string_parser.o Parser/peg_api.o Parser/myreadline.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genericaliasobject.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/unionobject.o Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/Python-tokenize.o Python/asdl.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/frame.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/hamt.o Python/hashtable.o Python/import.o Python/importdl.o Python/initconfig.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/specialize.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/suggestions.o Python/dynload_shlib.o    Modules/config.o Modules/main.o Modules/gcmodule.o Modules/atexitmodule.o  Modules/faulthandler.o  Modules/posixmodule.o  Modules/signalmodule.o  Modules/_tracemalloc.o  Modules/_codecsmodule.o  Modules/_collectionsmodule.o  Modules/errnomodule.o  Modules/_io/_iomodule.o Modules/_io/iobase.o Modules/_io/fileio.o Modules/_io/bytesio.o Modules/_io/bufferedio.o Modules/_io/textio.o Modules/_io/stringio.o  Modules/itertoolsmodule.o  Modules/_sre.o  Modules/_threadmodule.o  Modules/timemodule.o  Modules/_weakref.o  Modules/_abc.o  Modules/_functoolsmodule.o  Modules/_localemodule.o  Modules/_operator.o  Modules/_stat.o  Modules/symtablemodule.o  Modules/pwdmodule.o  Modules/xxsubtype.o \
		Programs/_bootstrap_python.o Modules/getpath.o -ldl  -framework CoreFoundation
./_bootstrap_python ./Tools/scripts/deepfreeze.py Python/frozen_modules/importlib._bootstrap.h -m importlib._bootstrap -o Python/deepfreeze/importlib._bootstrap.c
make: *** [Python/deepfreeze/importlib._bootstrap.c] Bus error: 10
$

      --enable-framework causes a number of changes in interpreter building and initialization. Since, as it stands, _bootstrap_python executable isn't and doesn't need to be built as a framework, there is probably some path confusion in getpath.c or elsewhere.

      Prior to these changes (at 3.11.0a2 at least), it was still possible to run a framework build executable in the build directory through some careful use of build variables in Makefile (like how "make test" works). Without delving deeper into it at the moment, I'm not sure what to suggest. It may be simpler to just do a clean separate non-framework build to provide _bootstrap_python. And/or possibly re-allow the use of --with-build-python to specify a user-supplied build python. I wonder if there are certain configurations on other platforms where there might be similar problems with the current approach.

      I'm marking this as a "release blocker" since it is preventing building the usual python.org macOS binary installer for 3.11.0a3. Since it is still early in the release cycle for 3.11, the release manager might reasonably decide to not delay the release for a fix and thus skip having a macOS installer for a3.

    6. ned-deily
      added
      release-blocker
      on Dec 6, 2021

    7. tiran commented on Dec 6, 2021

      @tiran
      tiran
      on Dec 6, 2021
      MemberAuthor

      Ned, does this change solve the issue with framework builds?

      $ git diff
diff --git a/Makefile.pre.in b/Makefile.pre.in
index 8e6e553554d..2068db30855 100644
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -106,7 +106,7 @@ PY_CFLAGS_NODIST=$(CONFIGURE_CFLAGS_NODIST) $(CFLAGS_NODIST) -I$(srcdir)/Include
 PY_CPPFLAGS=   $(BASECPPFLAGS) -I. -I$(srcdir)/Include $(CONFIGURE_CPPFLAGS) $(CPPFLAGS)
 PY_LDFLAGS=    $(CONFIGURE_LDFLAGS) $(LDFLAGS)
 PY_LDFLAGS_NODIST=$(CONFIGURE_LDFLAGS_NODIST) $(LDFLAGS_NODIST)
-PY_LDFLAGS_NOLTO=$(PY_LDFLAGS) $(CONFIGURE_LDFLAGS_NOLTO) $(LDFLAGS_NODIST)
+PY_LDFLAGS_NOLTO=$(PY_LDFLAGS_NODIST) $(CONFIGURE_LDFLAGS_NOLTO) $(LDFLAGS_NODIST)
 NO_AS_NEEDED=  @NO_AS_NEEDED@
 CCSHARED=      @CCSHARED@
 # LINKFORSHARED are the flags passed to the $(CC) command that links

    8. ned-deily commented on Dec 6, 2021

      @ned-deily
      ned-deily
      on Dec 6, 2021
      Member

      Ned, does this change solve the issue with framework builds?

      No. I would guess at a minimum getpath.c for _bootstrap_python might need to be built without WITH_NEXT_FRAMEWORK defined. But I'm out of time for today and don't expect to have much time available in the immediate future.

    9. 3 remaining items

    10. tiran commented on Dec 6, 2021

      @tiran
      tiran
      on Dec 6, 2021
      MemberAuthor

      New changeset 612e59b by Christian Heimes in branch 'main':
      bpo-45950: Fix macOS framework builds of _bootstrap_python (GH-29936)
      612e59b

    11. tiran commented on Dec 6, 2021

      @tiran
      tiran
      on Dec 6, 2021
      MemberAuthor

      Ned, I have implemented a fix for _bootstrap_python for framework builds and re-enabled --with-build-python for all builds. Previously I disabled the option for standard builds, because --with-build-python can cause build issues when a user runs ./configure --with-build-python with an older 3.11 alpha build.

    12. ned-deily commented on Dec 7, 2021

      @ned-deily
      ned-deily
      on Dec 7, 2021
      Member

      I've now verified that the _bootstrap_python fix for framework builds works - removing "release blocker" status. Thanks, Christian!

    13. ned-deily
      removed
      release-blocker
      on Dec 7, 2021
    14. ezio-melotti
      transferred this issue fromon Apr 10, 2022

    15. kumaraditya303 commented on May 9, 2022

      @kumaraditya303
      kumaraditya303
      on May 9, 2022
      Contributor

      The issue has been fixed, I am closing the issue.

    16. kumaraditya303
      closed this as completedon May 9, 2022
    17. vstinner
      added 2 commits that reference this issue on Sep 19, 2023

      pythongh-90108: Disable LTO on _freeze_module and _testembed

      69c34bd

      gh-90108: Disable LTO on _freeze_module and _testembed (#109581)

      3e3a7da
    18. csm10495
      added a commit that references this issue on Sep 28, 2023

      pythongh-90108: Disable LTO on _freeze_module and _testembed (python#…

      1ab8940
    19. vstinner
      added 2 commits that reference this issue on Oct 11, 2023

      pythongh-110313: Revert "pythongh-90108: Disable LTO on _freeze_modul…

      43950a6

      gh-110313: Revert "gh-90108: Disable LTO on _freeze_module and _teste… (

      467abfe
    20. Glyphack
      added 2 commits that reference this issue on Sep 2, 2024

      pythongh-90108: Disable LTO on _freeze_module and _testembed (python#…

      1716b8d

      pythongh-110313: Revert "pythongh-90108: Disable LTO on _freeze_modul…

      47af056
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    3.11only security fixesbuildThe build process and cross-buildtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions