Skip to content

Add more details about the Python build in sys.version #100086

Closed
Closed
Add more details about the Python build in sys.version#100086
Labels
type-bugAn unexpected behavior, bug, or error

Description

@vstinner
vstinner
opened on Dec 7, 2022

Currently, it's not easy to guess how Python was built just by looking at python -VV output or sys.version.

I propose to enhance sys.version to include more details about how Python was configured for the build:

  • debug or release build
  • "shared" or "framework" build
  • assertions enabled or not
  • address/memory/undefined behavior sanitizer
  • etc.

It should help me to more quickly identify if a buildbot was built in debug mode or release mode.

It should help to see how Python was optimized: with or without LTO and PGO optimizations?

It should help to quickly have an idea of which ABI is used: the "pystats" ABI is different than the "debug" than the "release" ABI.

Linked PRs

Activity

  1. vstinner
    added
    type-bugAn unexpected behavior, bug, or error
    on Dec 7, 2022

  2. ned-deily commented on Dec 7, 2022

    @ned-deily
    ned-deily
    on Dec 7, 2022
    Member

    I don't think this change should be made. It potentially affects all users of Python, for example, the REPL users, but is of no benefit to 99% of them. All of this information is readily available via multiple other mechanisms, in particular, via python3 -m test.pythoninfo or via sysconfig.get_config_vars(). And as implemented in the PR, the information is truncated when displayed due to the string format limitation in getversion.c and can make the header unnecessarily long:

    Python 3.12.0a3+ (heads/py_build_str-dirty:94b93efeed, Dec 7 2022, 16:23:18, release framework bu) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin

  3. vstinner commented on Dec 7, 2022

    @vstinner
    vstinner
    on Dec 7, 2022
    MemberAuthor

    One motivation for me is that many users don't use Python binaries from python.org or from their Linux packaging system, but build their own binary with their custom recipe and it's not trivial to guess how the binary was built. Was it properly optimized with LTO+PGO? Was it built in debug mode?

    For a long time, the most popular Docker recipe to build Python used ./configure && make: no LTO, no PGO.

    Today, I cannot recall if the macOS binary is optimized with LTO or not.

    Recently, @JulienPalard shared his recipe to build a custom Python with few lines of shell code: compile-python.sh. I was surprised to see --with-pydebug in the middle of the script: why building Python in debug mode? Some users may use it without paying attention to this configure flag.

    in particular, via python3 -m test.pythoninfo or via sysconfig.get_config_vars()

    Well, these APIs are not great :-(

    test.pythoninfo is designed to be used on the command line, it doesn't have a simple API to query info about the current Python.

    sysconfig.get_config_vars() is hard to use: you have to guess which variable is the best to get compiler flags: make your choice in this long list, it can depend on how Python was built, in case of doubt you might have to check multiple variables. Moreover, once you have compiler options, you have to parse this string which is not trivial (I hate parsing shell). For LTO, do you have to check compiler flags or linker flags, hum?

  4. vstinner commented on Dec 7, 2022

    @vstinner
    vstinner
    on Dec 7, 2022
    MemberAuthor

    It potentially affects all users of Python, for example, the REPL users, but is of no benefit to 99% of them.

    Ah? I'm not the 99%, I build Python on a daily basis and work on Python built in various ways. I know that :-)

    I saw this "debug build" vs "release build" as an useful information, but maybe it should be omitted for the common case: "release build" string? What about "lto+pgo": is it useful for most users?

    On my Fedora 37, sys.version is Python 3.11.0 (main, Oct 24 2022, 00:00:00) [GCC 12.2.1 20220819 (Red Hat 12.2.1-2)] on linux.

    String length:

    • Python version: 8% (6 characters)
    • space + Build info: 39% (30 characters)
    • space + compiler name and version: 53% (41 characters)
    • Total: 77 characters

    To make this string shorter, the compiler part should be shorter: [GCC 12.2.1 20220819 (Red Hat 12.2.1-2)] on linux.

  5. vstinner commented on Dec 7, 2022

    @vstinner
    vstinner
    on Dec 7, 2022
    MemberAuthor

    At the beginning, I started by modifying test.libregrtest to add some "build details" in the first lines of the test runner:

    == CPython 3.12.0a2+ (heads/main-dirty:9dc787ea96, Dec 6 2022, 11:51:12) [GCC 9.4.0]
== Linux-5.15.0-1019-azure-x86_64-with-glibc2.31 little-endian
== cwd: /opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/build_oot/host/build/test_python_829492æ
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
Using random seed 7151121
0:00:00 load avg: 7.01 Run tests in parallel using 2 child processes (timeout: 15 min, worker timeout: 20 min)

    These lines don't say if Python was built in debug mode or not. The test.pythonfo data is not far in CIs, but it's annoying to have to reach the test.pythoninfo step, unfold it, scroll (this is especially painful in the buildbot web UI), just to check if Python was built in debug mode or not.

    The first line is coming from the code:

            print("==", platform.python_implementation(), *sys.version.split())

    Since sys.version is used, I thought that maybe putting the info directly in sys.version can benefit to more users.

  6. vstinner commented on Dec 7, 2022

    @vstinner
    vstinner
    on Dec 7, 2022
    MemberAuthor

    About the REPL, python2 and pypy put the info on two lines which makes it more readable IMO:

    $ python2
Python 2.7.18 (default, Aug 22 2022, 00:00:00) 
[GCC 12.2.1 20220819 (Red Hat 12.2.1-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

    and

    $ pypy3
Python 3.9.12 (2fc6706f5902, Oct 12 2022, 21:40:58)
[PyPy 7.3.9 with GCC 12.2.1 20220819 (Red Hat 12.2.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>>

    The single line format (sys.version) is also used by the python -VV command:

    $ python3 -VV
Python 3.11.0 (main, Oct 24 2022, 00:00:00) [GCC 12.2.1 20220819 (Red Hat 12.2.1-2)]

  7. vstinner commented on Dec 7, 2022

    @vstinner
    vstinner
    on Dec 7, 2022
    MemberAuthor

    I wrote PR #100093 which only changes the Python test runner (libregrtest), so it doesn't change the Python REPL nor python -VV. It doesn't change sys.version, so the platform module doesn't need to be updated.

  8. vstinner
    added a commit that references this issue on Dec 8, 2022

    gh-100086: Add build info to test.libregrtest (#100093)

    3c89202

  9. vstinner commented on Dec 8, 2022

    @vstinner
    vstinner
    on Dec 8, 2022
    MemberAuthor

    Fixed by: 3c89202

  10. vstinner
    closed this as completedon Dec 8, 2022
  11. vstinner
    added 2 commits that reference this issue on Sep 2, 2023

    pythongh-100086: Add build info to test.libregrtest (python#100093)

    c5bfed5

    [3.11] gh-108822: Backport libregrtest changes from the main branch (#…

    79f7a4c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions