Repository navigation
ctypes CDLL search path issue on MacOS #88130
Description
| BPO | 43964 |
|---|---|
| Nosy | @ronaldoussoren, @ned-deily, @vlazzarini |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2021-04-28.08:39:49.501>
labels = ['OS-mac', '3.9', '3.10', '3.11']
title = 'ctypes CDLL search path issue on MacOS'
updated_at = <Date 2021-06-26.19:26:05.420>
user = 'https://github.com/vlazzarini'bugs.python.org fields:
activity = <Date 2021-06-26.19:26:05.420>
actor = 'ccabrera'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['macOS']
creation = <Date 2021-04-28.08:39:49.501>
creator = 'Victor.Lazzarini'
dependencies = []
files = []
hgrepos = []
issue_num = 43964
keywords = []
message_count = 5.0
messages = ['392173', '394592', '394782', '394788', '396555']
nosy_count = 4.0
nosy_names = ['ronaldoussoren', 'ned.deily', 'Victor.Lazzarini', 'ccabrera']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue43964'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']
Activity
vlazzarini commentedon Apr 28, 2021
With Python 3.9.4 ctypes.CDLL does not appear to find framework libraries installed in /Library/Frameworks.
--- With Python 3.6.5:
--- With Python 3.9.4
This happens with all frameworks I have installed in /Library/Frameworks.
The full path seems to work OK:
>>> ctypes.CDLL('/Library/Frameworks/CUDA.framework/CUDA') <CDLL '/Library/Frameworks/CUDA.framework/CUDA', handle 7fa358404180 at 0x7fa3580eeaf0> >>>but that's suboptimal as in MacOS you might have frameworks installed elsewhere.
vlazzarini commentedon May 27, 2021
I have looked at this closely and it appears Python 3.6.5 has a search path for libraries and frameworks that is missing in the newer versions (3.7 onwards).
I can load libraries from /usr/local/lib and ~/lib without any difficulties in 3.6.5, just by passing the file name. In later versions this does not work, you need to pass a full path. Furthermore if these libraries have dependencies which are set with an rpath, the loading fails because the dependencies are not found. None of this is a problem in 3.6.5. This needs to be resolved because packages that are based on ctypes may be broken in versions >= 3.7
This is the case of Csound and ctcsound, for instance.
ned-deily commentedon May 30, 2021
Thanks for the report! I've spent some time investigating it and the story behind it turns out to be a bit complicated, so bear with me. It's all tied in to Apple's attempts to improve the security of macOS.
As of macOS 10.15 Catalina, Apple introduced new requirements for downloadable installer packages, like those we provide for macOS on python.org; in order for such packages to be installed with the macOS installer, they would now have to be "notarized" by Apple. The notarization process is somewhat similar in concept to the process that an app has to go through to be submitted to the Mac App Store but with less stringent requirements. In particular, the installer package is automatically inspected to ensure that all executables are codesigned, are linked with the more-secure "hardened runtime", and do not request certain less-secure entitlements. Although originally announced for enforcement starting with the release of Catalina in the fall of 2019, Apple delayed the enforcement until February 2020 to give application developers more time to modify their packages to meet the new requirements. (See, for example, https://developer.apple.com/news/?id=12232019a).
The first python.org macOS installers that conformed to the new requirements and were notarized were for the 3.8.2 and 3.7.7 releases, staring in 2020-02. In those first releases, we used two entitlements:
com.apple.security.cs.disable-library-validation
com.apple.security.cs.disable-executable-page-protection
Some issues were reported (like bpo-40198) when using those first releases, so we added two additional entitlements for subsequent releases:
com.apple.security.automation.apple-events
com.apple.security.cs.allow-dyld-environment-variables
While we didn't realize it until your issue, that did have an effect on ctype's behavior when trying to find shared libraries and frameworks. Using the hardened runtime, as now required for notarization, causes the system dlopen() interface, which ctypes uses, to no longer search relative paths. The dlopen man page (for macOS 11, at least) includes this warning:
Note: If the main executable is a set[ug]id binary or codesigned with
entitlements, then all environment variables are ignored, and only a
full path can be used.
After some experimentation, it looks like that statement isn't exactly correct at least on macOS 11 Big Sur: unprefixed paths can still be used to find libraries and frameworks in system locations, i.e. /usr/lib and /System/Library/Frameworks. But it is true that, when using the hardened runtime, dlopen() no longer searches the user-controlled paths /usr/local/lib or /Library/Frameworks by default. And there apparently is no way for a notarized executable to revert to the previous behavior by adding other entitlements (https://developer.apple.com/forums/thread/666066).
With the allow-dyld-environment-variables entitlement included after the initial releases, it *is* now possible to change this behavior externally, if necessary, by explicitly setting the DYLD_LIBRARY_PATH and/or DYLD_FRAMEWORK_PATH environment variables (see man 1 dyld).
To demonstrate using a third-party library in /usr/local/lib (similar results with third-party frameworks in /Library/Frameworks):
# ------ python.org 3.7.6, not codesigned ------
# ------ python.org 3.7.7, first notarized release ------
# ------ python.org 3.7.8 and beyond including 3.9.x ------
It is rather unfortunate that, when including the allow-dyld-environment-variables entitlement, you now receive a less helpful error message "image not found" rather than "file system relative paths not allowed in hardened programs".
So, what to do?
That would mean that python.org installer packages could not be notarized and thus would no longer be installable without going through various onerous steps to override Gatekeeper protections (and would presumably also reduce somewhat system security). Since the primary target of python.org installers is for inexpert users, this option seems unacceptable.
So something like: if the path passed to ctypes is not absolute and the initial call to dlopen() fails with an "image not found" error, try again by making an absolute path by prepending '/usr/local/lib' and/or '/Library/Frameworks' and retrying. I assume we could make that work somehow but the question is should we? That seems very kludgey and would be re-opening a potential security hole that Apple clearly thinks is significant and is trying to discourage.
In particular, document this change in behavior for codesigned installations using the hardened runtime (like python.org installers) and suggest avoid using relative paths on macOS or, if necessary, have the DYLD_LIBRARY_PATH or DYLD_FRAMEWORK_PATH environment variables set appropriately when launching the interpreter. (I did a quick experiment and it seemed that it was not possible to change dlopen()'s behavior by setting those variables from inside the running interpreter process by using OS.ENVIRON, an unsurprising result.)
It seems to me that changing the ctypes documentation (and adding a changelog blurb), option 3, is the least bad of the available options. One would hope that this does not affect *too* many third-party packages and user applications. Obviously it does some :(
@ronald, what do you think?
4 remaining items
vlazzarini commentedon May 31, 2021
Hi Ned,
thanks for your detailed response. I have to say that the LD_LIBRARY_PATH partially worked in some cases, the library got loaded. However in cases where there were dependencies, further issues appeared particularly when the link (as revealed by otool -L) was prefixed by '@rpath'. I was able to fix these by editing these paths (which is possible in my use case), but we just need to note that it may cause difficulties for others.
We also found a solution to avoid needing to set the environment var. We use the following code
instead of just passing the library. That works well, and perhaps you may recommend it in the docs.
ccabrera commentedon Jun 26, 2021
Re option 3: relying on
DYLD_LIBRARY_PATHorDYLD_FRAMEWORK_PATHisn't a great solution either because of SIP. It's can get impractical to use in many standard Makefile-based build systems, for example. (cf. qmk/qmk_cli#60, Homebrew/discussions#1737) I imagine it can get quite vexing to set these variables but find them to not have the effect you expected them to.One workaround for the example above would be to compile your own
makeprogram (or use one from a package manager) so that SIP doesn't sanitise your environment, but I imagine that's not an ideal solution either.