Python3.13 performance Issue with python.org macOS installers on ARM Macs

Bug description:

Python3.13rc1 seems to be about 20-25% slower than Python3.12.4.
Platform: M1 Mac running MacOS Sonoma 14.5

I first noticed the issue when running this repository where Python3.13rc1 ran on average 20-25% slower than Python3.12.4

With help from the Python discord server, we were able to create this minimal reproducible example:

from time import process_time

class Lexer:
    def __init__(self, data):
        self.data = data
        self.index = 0

    def detect(self, text):
        i = 0
        for char in text:
            if self.peek(i) != char:
                return False
            i += 1
        return True

    def peek(self, amt=1):
        if self.index + amt < len(self.data):
            return self.data[self.index + amt]
        else:
            return None

text = "return text + indent * ' ' + '}'\n" * 200
needle = "return text + indent * ' ' + '}'\n" * 199 + "Nope\n" 
lexer = Lexer(text)
start = process_time() 
for x in range(10_000):
    lexer.detect(needle)
print(process_time() - start)

These were the results across many tests:
3.12.4: 7.742122s
3.13rc1: 8.309326s

The performance issue was not reproducible on other Operating Systems.

CPython versions tested on:

3.11.1, 3.12.4, 3.13

CPython Options:

For my testing, I used the official Python3.13rc1 installer from Python.org, which has the following configuration:

OPT:

3.13rc1: -DNDEBUG -g -O3 -Wall
3.12.4: -DNDEBUG -g -O3 -Wall

CONFIG_ARGS:

3.13rc1: '--enable-framework' '--with-framework-name=Python' '--enable-universalsdk=/' '--with-universal-archs=universal2' '--enable-optimizations' '--with-lto' '--without-ensurepip' '--with-system-libmpdec' '--with-openssl=/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local' 'LIBLZMA_CFLAGS=-I/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/include' 'LIBLZMA_LIBS=-L/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/lib -llzma' 'LIBMPDEC_CFLAGS=-I/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/include' 'LIBMPDEC_LIBS=-L/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/lib -lmpdec -lm' 'LIBSQLITE3_CFLAGS=-I/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/include' 'LIBSQLITE3_LIBS=-L/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/lib -lsqlite3' 'TCLTK_CFLAGS=-I/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/include' 'TCLTK_LIBS=-L/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/lib -ltcl -ltk' 'CC=clang'
3.12.4: '-C' '--enable-framework' '--enable-universalsdk=/' '--with-universal-archs=universal2' '--with-computed-gotos' '--without-ensurepip' '--with-openssl=/tmp/_py/libraries/usr/local' '--enable-optimizations' '--with-lto' 'TCLTK_CFLAGS=-I/tmp/_py/libraries/usr/local/include' 'TCLTK_LIBS=-ltcl8.6 -ltk8.6' 'LDFLAGS=-g' 'CFLAGS=-g' 'CC=clang'

Operating systems tested on:

Linux, macOS, Windows

@devdanzin

Me and @devdanzin have figured out that this is only on ARM, Windows and Linux x86 seemed to be faster based on the benchmarks.

Thanks for this report. Across a broad range of benchmarks measuring on the Faster CPython benchmarking machine, we see between a 3% and 8% speedup for 3.13.0b4 vs. 3.12.0 on a MacOS ARM machine.

That doesn't mean that this particular example hasn't found a narrow regression case. In our own benchmarking, some specific benchmarks are in fact slower, but only telco is that much slower (which is a known regression in the decimal module). Maybe some line profiling of 3.12 vs. 3.13 would be helpful here, or some pystats results.

I also noticed that you have --with-computed-gotos for 3.12, but not 3.13 here. That may not be relevant since it's on by default anyway (and I'm not sure it even applies for clang), but we should make sure all other variables are equal.

I just tried to reproduce the above result on our benchmarking M1 Mac, also running macOS 14.5, and got different results:

3.12.0: 7.471261
3.13.0rc1: 5.986742

I suspect this comes down to different configurations between the builds. It looks like 3.12.0 on your machine was built on a different machine? Try building each in exactly the same environment and see if you can still reproduce.

I just tried to reproduce the above result on our benchmarking M1 Mac, also running macOS 14.5, and got different results:
3.12.0: 7.471261
3.13.0rc1: 5.986742
I suspect this comes down to different configurations between the builds. It looks like 3.12.0 on your machine was built on a different machine? Try building each in exactly the same environment and see if you can still reproduce.

I am using 3.12.4, not 3.12.0.
I have tested 3.11, 3.12.1, 3.12.3, all from Python.org at different points in time.
I have also tested many different betas of 3.13, and the results have been consistent.

All the versions installed on my machine are the default ones from Python.org.

@ned-deily

I also can't reproduce with local optimized builds on an M2 Pro running Sonoma 14.5 (3 runs each):

v3.12.4:          5.718666, 6.035776, 5.577658
v3.13.0rc1:       4.933225, 5.214851, 4.837962
v3.13.0rc1 (JIT): 4.746776, 4.747096, 4.764923

However, I can reproduce with the python.org downloads:

3.12.4:           5.684782, 5.808816, 5.734691
3.13.0rc1:        6.673098, 6.930707, 7.369320

So the issue seems to be with the releases themselves. @ned-deily, anything different with macOS releases this time around?

So, this is kinda strange. Consider this reproducer:

import time

s = "X" * 100_000_000

start = time.perf_counter()
for _ in s:
    pass
duration = time.perf_counter() - start
print(f"global: {duration:.3f}s")

def foo():
    for _ in s:
        pass

start = time.perf_counter()
foo()
duration = time.perf_counter() - start
print(f"function: {duration:.3f}s")

With the 3.12.4 python.org download, this prints:

global: 1.237s
function: 0.364s

On 3.13.0rc1:

global: 0.930s
function: 0.680s

So the loop over the string is faster at module level on 3.13, but faster at function level in 3.12. And 3.13 is faster on both if I replace "X" * 100_000_000 with range(100_000_000).

The issue could be some sort of messed-up (or skipped) PGO with the release build. The global version has lots of FOR_ITER -> STORE_NAME pairs while the function version has lots of FOR_ITER -> STORE_FAST pairs. That's really the only difference that I can see. I'd expect PGO to make the latter very fast, at the expense of the former.

But at this point I'm just guessing.

However, I can reproduce with the python.org downloads:

Just dittoing this. I agree, there may be something that changed in the build environment between releases.

@hg0428

I can reproduce a similar slower performance between the python.org 3.12.4 and 3.13.0rc1 builds on an M3 MacBook Pro with macOS 14.6. However, using the same python.org installers on an Intel i9 iMac also with macOS 14.6 shows a performance improvement between 3.12.4 and 3.13.0rc1. These are both universal2 builds meaning they should be running native arm or x86_64 code on each. And, interestingly, if I force running the universal2 builds in x86_64 mode using Rosetta2 emulation on the same M3 (by using python3.1x-intel64), I see performance differences similar to running natively on the Intel Mac, that is 3.13.0rc1 is somewhat faster.

There are a number of differences in the build environments used for 3.12.4 and for 3.13.0rc1, including the compiler versions and deployment targets. I agree that something with optimizations seems a likely culprit. I will dig into it and report back.

Thanks for the report and the simple reproducer, @hg0428!

A quick update: I've run the test case on the full range of python.org macOS installers from 3.13.0a5 through 3.13.0rc1 on both Apple Silicon (M3, arm64) and Intel (i9) Macs and it appears that rc1 on Apple Silicon is the outlier. I also ran the test case on all of the macOS installers where we also provided a free-threading build, that is, 3.13.0b2 through 3.13.0rc1, again, on both CPU types; there, the test-case performance differences roughly mirror those of the the normal (GIL) builds except that the rc1 free-threading version does not exhibit the performance degradation that the normal version does on Apple Silicon. So, the good news at this point is that this particular performance issue appears to be isolated but there is more work to be done to pin it down further. More tomorrow.

Extensive testing with the microbenchmarks provided in this issue and in #122832 confirm that (1) there was a significant performance degradation for at least some test cases with the 3.13.0rc1 provided by the python.org installer particularly on Apple Silicon Macs and that (2) that performance degradation appears to be due to the use of the ThinLTO (Link Time Optimization) Python build option (--with-lto defaulting to --with-lto=thin) with recent Apple-supplied LLVM/Clang build tools. As noted earlier in this issue, the degradation was seen with very simple builds of the test case(s) when --with-lto was included. It was subsequently determined through extensive testing that using the original monolithic LTO build option (--with-lto=full) provides a substantial performance boost for these two benchmarks and, in all test configurations shown here, the three build configurations using --with-lto=full always provided the best performance with these microbenchmarks, in some cases, dramatically better.

Since performance can vary greatly due to many factors, it is unreasonable to expect the same level of improvement in more extensive real-world benchmarks but it seems likely that an overall improvement of some magnitude may be seen with most benchmarks. From reading a bit about ThinLTO, it seems one of its biggest advantages was expected to be providing faster incremental builds rather than doing the monolithic optimization for each build. For the python.org installer case, that isn't an advantage since we always build the binary components of the installer from scratch for each release. And the documentation implies that, in most cases, the monolithic LTO would be expected to provide better run-time performance anyway. So all of that seems to indicate that full LTO is the right choice for the installer build in any case.

Moving forward, with the imminent release of 3.13.0rc2, we will use --with-lto=full to build the macOS installer for it and will run these microbenchmarks again before release. It would be great if more extensive benchmarks could be run with the released installer and results reported here to this issue. I will leave it open for that purpose for the immediate future.

While there are still unanswered questions about exactly what it is about ThinLTO that causes the performance degradation in these environments, I am not planning to delve deeper into it since it's not something I have particular expertise in and it seems that using full LTO is a perfectly satisfactory solution for the macOS installer. Feel free to dive in yourself!

Thanks again to everyone who has helped out with reporting and investigation this!

Appended here is a summary of the more pertinent test results while investigating this issue:

Test Cases

test_122580 is the test case provided above in this issue.
test_122832 is the var_access_benchmark referenced in Significant performance degradation in the macOS build #122832._

Apple Silicon (MacBook Pro 14-inch, Nov 2023, M3, 16GB)

Summary

  test_122580 (version = 3.13.0rc1+-xcode) ran
    1.00 ± 0.02 times faster than test_122580 (version = 3.13.0rc1-lto-full)
    1.04 ± 0.02 times faster than test_122580 (version = 3.13.0rc1+-xcode-beta)
    1.11 ± 0.04 times faster than test_122580 (version = 3.13.0rc1-no-lto)
    1.16 ± 0.02 times faster than test_122580 (version = 3.12.5)
    1.17 ± 0.04 times faster than test_122580 (version = 3.13.0b4)
    1.18 ± 0.02 times faster than test_122580 (version = 3.12.4)
    1.20 ± 0.03 times faster than test_122580 (version = 3.13.0b1)
    1.61 ± 0.04 times faster than test_122580 (version = 3.13.0rc1)

  test_122832 (version = 3.13.0rc1+-xcode-beta) ran
    1.02 ± 0.02 times faster than test_122832 (version = 3.13.0rc1-lto-full)
    1.03 ± 0.01 times faster than test_122832 (version = 3.13.0rc1+-xcode)
    1.07 ± 0.01 times faster than test_122832 (version = 3.12.4)
    1.08 ± 0.01 times faster than test_122832 (version = 3.12.5)
    1.12 ± 0.03 times faster than test_122832 (version = 3.13.0b4)
    1.17 ± 0.01 times faster than test_122832 (version = 3.13.0rc1-no-lto)
    1.28 ± 0.02 times faster than test_122832 (version = 3.13.0b1)
    1.79 ± 0.02 times faster than test_122832 (version = 3.13.0rc1)

Command	Mean [s]	Min [s]	Max [s]	Relative
`test_122580 (version = 3.12.4)`	4.851 ± 0.039	4.797	4.894	1.18 ± 0.02
`test_122580 (version = 3.12.5)`	4.778 ± 0.030	4.739	4.813	1.16 ± 0.02
`test_122580 (version = 3.13.0b1)`	4.954 ± 0.089	4.822	5.053	1.20 ± 0.03
`test_122580 (version = 3.13.0b4)`	4.840 ± 0.141	4.673	4.954	1.17 ± 0.04
`test_122580 (version = 3.13.0rc1)`	6.627 ± 0.138	6.403	6.751	1.61 ± 0.04
`test_122580 (version = 3.13.0rc1-no-lto)`	4.577 ± 0.142	4.385	4.740	1.11 ± 0.04
`test_122580 (version = 3.13.0rc1-lto-full)`	4.128 ± 0.076	4.032	4.225	1.00 ± 0.02
`test_122580 (version = 3.13.0rc1+-xcode)`	4.121 ± 0.066	4.006	4.169	1.00
`test_122580 (version = 3.13.0rc1+-xcode-beta)`	4.304 ± 0.073	4.199	4.372	1.04 ± 0.02

Command	Mean [s]	Min [s]	Max [s]	Relative
`test_122832 (version = 3.12.4)`	16.645 ± 0.059	16.588	16.744	1.07 ± 0.01
`test_122832 (version = 3.12.5)`	16.843 ± 0.064	16.772	16.919	1.08 ± 0.01
`test_122832 (version = 3.13.0b1)`	19.956 ± 0.248	19.749	20.364	1.28 ± 0.02
`test_122832 (version = 3.13.0b4)`	17.402 ± 0.370	17.173	18.058	1.12 ± 0.03
`test_122832 (version = 3.13.0rc1)`	27.907 ± 0.206	27.669	28.202	1.79 ± 0.02
`test_122832 (version = 3.13.0rc1-no-lto)`	18.313 ± 0.161	18.128	18.558	1.17 ± 0.01
`test_122832 (version = 3.13.0rc1-lto-full)`	15.988 ± 0.195	15.834	16.294	1.02 ± 0.02
`test_122832 (version = 3.13.0rc1+-xcode)`	16.057 ± 0.075	15.960	16.160	1.03 ± 0.01
`test_122832 (version = 3.13.0rc1+-xcode-beta)`	15.599 ± 0.139	15.355	15.692	1.00

Intel (iMac Retina 5K 27-inch, 2019, 3.6 GHz 8-Core Intel Core i9, 40GB)

Summary

  test_122580 (version = 3.13.0rc1+-xcode-beta) ran
    1.08 ± 0.03 times faster than test_122580 (version = 3.13.0rc1+-xcode)
    1.08 ± 0.02 times faster than test_122580 (version = 3.13.0rc1-lto-full)
    1.19 ± 0.02 times faster than test_122580 (version = 3.13.0rc1-no-lto)
    1.20 ± 0.02 times faster than test_122580 (version = 3.13.0b4)
    1.24 ± 0.02 times faster than test_122580 (version = 3.13.0rc1)
    1.43 ± 0.08 times faster than test_122580 (version = 3.13.0b1)
    1.46 ± 0.04 times faster than test_122580 (version = 3.12.4)
    1.47 ± 0.03 times faster than test_122580 (version = 3.12.5)

  test_122832 (version = 3.13.0rc1+-xcode-beta) ran
    1.08 ± 0.04 times faster than test_122832 (version = 3.13.0rc1+-xcode)
    1.11 ± 0.02 times faster than test_122832 (version = 3.13.0rc1-lto-full)
    1.13 ± 0.01 times faster than test_122832 (version = 3.12.5)
    1.13 ± 0.01 times faster than test_122832 (version = 3.12.4)
    1.16 ± 0.02 times faster than test_122832 (version = 3.13.0rc1-no-lto)
    1.20 ± 0.02 times faster than test_122832 (version = 3.13.0b4)
    1.26 ± 0.05 times faster than test_122832 (version = 3.13.0rc1)
    1.35 ± 0.02 times faster than test_122832 (version = 3.13.0b1)

Command	Mean [s]	Min [s]	Max [s]	Relative
`test_122580 (version = 3.12.4)`	10.535 ± 0.193	10.293	10.765	1.46 ± 0.04
`test_122580 (version = 3.12.5)`	10.630 ± 0.088	10.540	10.753	1.47 ± 0.03
`test_122580 (version = 3.13.0b1)`	10.328 ± 0.563	9.811	10.947	1.43 ± 0.08
`test_122580 (version = 3.13.0b4)`	8.681 ± 0.093	8.606	8.832	1.20 ± 0.02
`test_122580 (version = 3.13.0rc1)`	8.943 ± 0.060	8.867	9.013	1.24 ± 0.02
`test_122580 (version = 3.13.0rc1-no-lto)`	8.576 ± 0.055	8.497	8.631	1.19 ± 0.02
`test_122580 (version = 3.13.0rc1-lto-full)`	7.777 ± 0.130	7.613	7.905	1.08 ± 0.02
`test_122580 (version = 3.13.0rc1+-xcode)`	7.772 ± 0.136	7.655	8.003	1.08 ± 0.03
`test_122580 (version = 3.13.0rc1+-xcode-beta)`	7.210 ± 0.113	7.088	7.359	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`test_122832 (version = 3.12.4)`	33.019 ± 0.182	32.798	33.217	1.13 ± 0.01
`test_122832 (version = 3.12.5)`	32.950 ± 0.186	32.651	33.158	1.13 ± 0.01
`test_122832 (version = 3.13.0b1)`	39.386 ± 0.470	38.945	39.939	1.35 ± 0.02
`test_122832 (version = 3.13.0b4)`	34.851 ± 0.509	34.294	35.525	1.20 ± 0.02
`test_122832 (version = 3.13.0rc1)`	36.653 ± 1.359	35.594	39.035	1.26 ± 0.05
`test_122832 (version = 3.13.0rc1-no-lto)`	33.851 ± 0.478	33.435	34.554	1.16 ± 0.02
`test_122832 (version = 3.13.0rc1-lto-full)`	32.255 ± 0.505	31.864	33.100	1.11 ± 0.02
`test_122832 (version = 3.13.0rc1+-xcode)`	31.530 ± 1.057	30.654	32.988	1.08 ± 0.04
`test_122832 (version = 3.13.0rc1+-xcode-beta)`	29.104 ± 0.263	28.850	29.499	1.00

Build configurations

Python Version	Apple Clang	Build System	Deployment Target	--with-lto=
3.12.4	clang-1300.0.29.30	x86_64-apple-darwin20.6.0	10.9	thin
3.12.5	clang-1300.0.29.30	x86_64-apple-darwin20.6.0	10.9	thin
3.13.0b1	clang-1300.0.29.30	x86_64-apple-darwin20.6.0	10.9	thin
3.13.0b4	clang-1500.3.9.4	aarch64-apple-darwin23.5.0	10.13	thin
3.13.0rc1	clang-1500.3.9.4	aarch64-apple-darwin23.5.0	10.13	thin
3.13.0rc1-no-lto	clang-1500.3.9.4	aarch64-apple-darwin23.6.0	10.13	no
3.13.0rc1-lto-full	clang-1500.3.9.4	aarch64-apple-darwin23.6.0	10.13	full
3.13.0rc1+-xcode	clang-1500.3.9.4	aarch64-apple-darwin23.6.0	10.13	full
3.13.0rc1+-xcode-beta	clang beta	aarch64-apple-darwin23.6.0	10.13	full

@ned-deily

Thanks for all your hard work on this, @ned-deily. This finding seems reasonable. I've put the rc2 release date on my calendar and will run it against the full pyperformance suite then to confirm.

@ned-deily

Thanks @ned-deily. I wonder, should the default for --with-lto be full instead of thin? I just assumed that was already the case (and I'm not sure what our benchmarking infrastructure assumes).

In general, if I'm passing --with-lto, I'm expecting the build to take a while (and be as fast as possible in return).

What's the default LTO (full or thin) for the Linux builds?

IIUC, thin LTO is only supported on llvm/clang, so our gcc-based Linux builds don't use it.

The default for --with-lto is platform-dependent and compiler-dependent. configure.ac has a bunch of code to deal with it (see, for example, starting here). There has also been a fair amount of discussion about lto options (for example, here and here). If somebody wants to revisit that, it would be better to do it in another issue; I'd like to keep this focused on the macOS installer.

Just tested 3.13rc2 and the performance issues are solved. Thank you.

Just tested 3.13rc2 and the performance issues are solved. Thank you.

Same (on our usual macOS lab machine).

@rhettinger

closing as fixed, thanks for testing @rhettinger & @mdboom !
please re-open if there's anything left to do here.

FWIW, 3.13.0rc3 differences noted: #124567 (comment)

Navigation Menu

Repository navigation

Python3.13 performance Issue with python.org macOS installers on ARM Macs #122580

Bug description:

CPython versions tested on:

CPython Options:

OPT:

CONFIG_ARGS:

Operating systems tested on:

20 remaining items

Test Cases

Apple Silicon (MacBook Pro 14-inch, Nov 2023, M3, 16GB)

Summary

Intel (iMac Retina 5K 27-inch, 2019, 3.6 GHz 8-Core Intel Core i9, 40GB)

Summary

Build configurations

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Python3.13 performance Issue with python.org macOS installers on ARM Macs #122580

Description

Bug description:

CPython versions tested on:

CPython Options:

OPT:

CONFIG_ARGS:

Operating systems tested on:

Activity

ZeroIntensity commented on Aug 1, 2024

mdboom commented on Aug 1, 2024

mdboom commented on Aug 1, 2024

hg0428 commented on Aug 1, 2024

brandtbucher commented on Aug 1, 2024

brandtbucher commented on Aug 1, 2024

brandtbucher commented on Aug 1, 2024

mdboom commented on Aug 1, 2024

ned-deily commented on Aug 1, 2024

ned-deily commented on Aug 2, 2024

20 remaining items

ned-deily commented on Sep 3, 2024

Test Cases

Apple Silicon (MacBook Pro 14-inch, Nov 2023, M3, 16GB)

Summary

Intel (iMac Retina 5K 27-inch, 2019, 3.6 GHz 8-Core Intel Core i9, 40GB)

Summary

Build configurations

mdboom commented on Sep 3, 2024

brandtbucher commented on Sep 3, 2024

itamaro commented on Sep 3, 2024

mdboom commented on Sep 3, 2024

ned-deily commented on Sep 3, 2024

rhettinger commented on Sep 7, 2024

mdboom commented on Sep 9, 2024

itamaro commented on Sep 9, 2024

ned-deily commented on Oct 1, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions