Repository navigation
Vendored mimalloc references macOS preprocessor symbols that no longer exist #118072
Description
Bug report
Bug description:
The CPython sources include a vendored copy of https://github.com/microsoft/mimalloc.
This was added 5 months ago as part of #109914. the commit comment says it is version "v2.12". That doesn't appear to be a mimalloc release, but v2.1.2 does exist, and was the current stable release as of November 2023.
However, that code references TARGET_IOS_IPHONE and TARGET_IOS_SIMULATOR. These symbols were deprecated in iOS 15 (released ~2 years ago). These references have been updated in the dev branch of mimalloc (along with many other changes), but there isn't an updated stable mimalloc release at time of writing.
We have also made a number of other changes to the vendored version (about 18 commits, by my count). It's not clear if these mirror changes that have been made upstream, or if we have effectively forked mimalloc at this point.
The code still appears to work at present, but there's a risk it might not in future when the symbol deprecations are finalised. There may also be other updates and bugfixes in mimalloc that we're not taking advantage of.
This was discovered during an audit of TARGET_OS symbol usage following the report of #117886 and #117891.
CPython versions tested on:
CPython main branch
Operating systems tested on:
macOS
Activity
freakboy3742 commentedon Apr 18, 2024
Tagging @DinoV, @colesbury and @vstinner as they appear to be most active in making modifications to the vendored mimalloc code. I'm happy to submit a patch updating the immediate macOS/iOS preprocessor symbol problem; but I don't know if there are any larger plans to coordinate with upstream mimalloc changes.
colesbury commentedon Apr 19, 2024
The relevant upstream branch is
dev-slice(v2.x development branch): https://github.com/microsoft/mimalloc/tree/dev-slice.But the preprocessor symbols look like they're used in the same places upstream as in CPython:
dev: https://github.com/microsoft/mimalloc/blob/8f7d1e9a41bb0182166aac6a8d4d8b00f60ed032/src/prim/unix/prim.c#L45
dev-slice: https://github.com/microsoft/mimalloc/blob/f199b888b47f77261aac9b63b612e77ff3fbd880/src/prim/unix/prim.c#L45
What should we replace them with?
We should make the necessary changes in CPython's copy as well as file an issue and/or PR upstream.
We're tracking differences from upstream here: #113141. We'll work on getting PRs upstream for the differences sometime after beta 1.
freakboy3742 commentedon Apr 19, 2024
Clearly I need stronger coffee. I don't know what I was looking at, but I swear I saw an updated version.
If I'm reading the intention of the code right (i.e., include
mach/vm_statistics.hon macOS, but not iOS et al), replacing:with
should work. That catches the native explicit "This is macOS" symbol present in current macOS SDKs, with a fallback to "well this must be macOS" if the SDK is old enough that it doesn't define the
TARGET_OS_*symbols.freakboy3742 commentedon Apr 19, 2024
I've opened microsoft/mimalloc#879 to track the issue upstream.
colesbury commentedon May 16, 2024
@freakboy3742 - is this fixed?
freakboy3742 commentedon May 18, 2024
@colesbury It looks like it's been fixed upstream; but the upstream changes haven't been merged into the vendored copy in CPython's sources.