ctypes.util.find_library() should return full pathname instead of filename in linux #65241
Closed as not planned
Description
| BPO | 21042 |
|---|---|
| Nosy | @berkerpeksag, @vadmium |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2014-03-23.22:47:15.768>
labels = ['ctypes', 'type-feature']
title = 'ctypes.util.find_library() should return full pathname instead of filename in linux'
updated_at = <Date 2021-12-09.14:33:36.731>
user = 'https://bugs.python.org/HernanGrecco'bugs.python.org fields:
activity = <Date 2021-12-09.14:33:36.731>
actor = 'Charles Coulombe'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['ctypes']
creation = <Date 2014-03-23.22:47:15.768>
creator = 'Hernan.Grecco'
dependencies = []
files = ['41789', '41835', '41866', '41992', '42003']
hgrepos = []
issue_num = 21042
keywords = ['patch']
message_count = 21.0
messages = ['214647', '259470', '259528', '259725', '259776', '259924', '260258', '260579', '260595', '260636', '260650', '260672', '260948', '260951', '261107', '261228', '261393', '261400', '261844', '261858', '408127']
nosy_count = 6.0
nosy_names = ['python-dev', 'berker.peksag', 'martin.panter', 'Hernan.Grecco', 'beng94', 'Charles Coulombe']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue21042'
versions = []
Activity
HernanGrecco commentedon Mar 23, 2014
In Windows and OSX,
find_libraryreturns the full pathname of the library file. But on Linux, it returns just the filename. Is there a reason for this difference?For consistency, it would be better to return the full pathname in all cases. It is easy to get the filename from the full pathname, but not the other way around.
beng94 commentedon Feb 3, 2016
Added a small patch that solves this issue on Ubuntu 15.10.
Produces output like:
/lib/x86_64-linux-gnu/libm.so.6
/lib/x86_64-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libbz2.so.1.0
I'd be glad to add some test cases if someone can give me some tips on how to do that.
vadmium commentedon Feb 4, 2016
IMO this should be treated as a new feature for the next release. But consistently returning the path sounds good to me if there is no good reason not to.
Left a question on the review. I think you also need to update the documentation, and since this is changing documented behaviour it probably needs a What’s New entry.
For tests, I would try doing something like find_library("c"), and ensuring that the result is an absolute path. We may have to end up skipping the test for platforms like Windows where this is not expected to work. Look through the files in /Lib/ctypes/test/ for a good place for it to live (if there isn’t already a test there you can modify).
beng94 commentedon Feb 6, 2016
Added a new patch, as Martin pointed out, I put back the ABI matching. The regex looks quite ugly, because it has to match \n\t. To be exact, it has to match something like this: "/lib/x86_64-linux-gnu/libc.so.6\n\tlibbz2.so.1.0 (libc6,x86-64)".
I updated the docs, although I don't know what should I write for the version, please someone help me with that.
For testing, there's a test function in this module, I updated that.
vadmium commentedon Feb 7, 2016
The ABI matching looks wrong to me. If I am looking for a 32-bit library, won’t it incorrectly catch the wrong path in the following “ldconfig -p” output:
'\tlibm.so.6 (libc6,x86-64, OS ABI: Linux 2.6.32) => /usr/lib/libm.so.6\n'
'\tlibm.so.6 (libc6, OS ABI: Linux 2.6.32) => /usr/lib32/libm.so.6\n'
Perhaps the abi_type check needs to be moved in front of the path name extraction.
For the version, I would put 3.6. Since this changes documented behaviour and has the potential to break compatibilty, it is best not to change it in a bug fix release. (3.5 has already been released.)
The problem with the test() function in ctypes.util is that it is not run by the main Python regression test suite. The tests under ctypes/test/ are run by the test suite.
beng94 commentedon Feb 9, 2016
I fixed the ABI matching, it was a stupid mistake, thanks for pointing it out :) I think now it works as expected.
I really don't find a place for testing. Maybe a new test file could be added, but I think the testing code for find_library wouldn't be more than 10 lines. Do you have any suggestions?
Thanks Martin for all your patience :)
vadmium commentedon Feb 14, 2016
I think the new regular expression will still find the wrong library in my libm example above. In 32-bit mode, it will be only looking to match \(libc6.*\). Since my example has the 64-bit line first, that one will match first. (I haven’t actually tested this, but I think I compiled 32-bit Python once before just by specifying CC="gcc -m32". Sorry to keep poking holes in your regular expression :)
Do you know if there is documentation for the “ldconfig -p” output format, or do we just have to go on what we see? If so, I would change it to ensure the ABI type string is either followed by a comma and space ", " or a closing bracket ")". A comma on its own, or other letters, is not a match.
I did a search for “find_library”, and the most likely place is /Lib/ctypes/test/test_find.py. You could probably get away with just adding a new method like Test_OpenGL_libs.test_path(). On my computer I have the GL and GLU libraries (but not gle), so I guess that these libraries are fairly common (plus it already has Windows and OS X versions to test).
beng94 commentedon Feb 20, 2016
What do you think about this regex?
'(lib%s\.[^\\s]+\s\(%s(?:\)|,\s.*\))\s=>\s.*)' % (re.escape(name), abi_type))
It works on 64 bit, just like before, but I could not test it on 32 bit. I'll add tests soon.
I looked for documentation on ldconfig, but could not find anything useful.
vadmium commentedon Feb 21, 2016
Tamás, it might be a good idea for you to sign a contributor agreement <https://www.python.org/psf/contrib/contrib-form/\>.
I compiled Python in 32-bit mode and tried your v2 patch out, which found the wrong library as I predicted. Then I tried your new regex and it picked out the correct line. I had to edit it to get it to extract just the filename from the line:
r'lib%s\.[^\\s]+\s\(%s(?:,\s.*)?\)\s=>\s(.*)' % (re.escape(name), abi_type)
I factored out the closing bracket from the comma + space bit, moved the group brackets to the end to extract the filename, and made it a raw string.
Without the patch, in 32-bit mode it will find 64-bit-only libraries:
With my edited regex:
beng94 commentedon Feb 21, 2016
I've added a new method to Test_OpenGL_libs as you suggested. I check whether find_library returns an absolute path. Note that I didn't distinguish different systems, as according to the docs, only Linux systems return the file name, other systems return the absolute path. (https://docs.python.org/3.5/library/ctypes.html#ctypes-reference) An other thing to note, that I introduced some code duplication as I use the same code snippet from setUpClass method to figure out the correct parameters to find_library.
The patch uses the same regex as you gave.
By the way, what do I have to do to compile CPython on a 64 bit system in 32 bit mode? I tried ./configure CC="gcc -m32" but it gave me an error. Is it the correct way?
Also, I signed contributor agreement.
vadmium commentedon Feb 22, 2016
I left a suggestion about the duplication in the code review.
I set CC with “configure” like you said. I also had to run “make clean” to get rid of the old 64-bit stuff. But it might depend on the GCC that you have installed. On Arch Linux, I have gcc-multilib, which supports 64-bit and 32-bit targets.
beng94 commentedon Feb 22, 2016
Updated the patch to remove the code duplication, now it stores the values that are calculated in the setUpClass method. It was a good and simple idea, I should have come up with it... :)
I'm pretty sure I got the errors during configuration because of the gcc version I have.
vadmium commentedon Feb 27, 2016
Thanks, this looks pretty good to me. I just need to remember to write a What’s New entry.
1 remaining item
beng94 commentedon Mar 2, 2016
Is there anything else that I can do for this issue?
vadmium commentedon Mar 6, 2016
No I think this is ready Tamás. I have been away, but it is on my list of things to catch up on. I won’t add any What’s New entry.
python-dev commentedon Mar 9, 2016
New changeset 3092cf163eb4 by Martin Panter in branch 'default':
Issue bpo-21042: Return full path in ctypes.util.find_library() on Linux
https://hg.python.org/cpython/rev/3092cf163eb4
vadmium commentedon Mar 9, 2016
It looks like the ldconfig parsing isn’t working for some ABIs. See the following buildbot failures:
I presume there are other flags in the ABI string, perhaps like (libc6,hard-float) on ARM. The code that produces these strings seems to be here: <https://sourceware.org/git/gitweb.cgi?p=glibc.git;a=blob;f=elf/cache.c;h=fbee172#l72\>.
Looking closer at the find_library() implementation, I also realize it is not correct to say an absolute path is always returned. If the ldconfig check fails, it falls back to _get_soname(). I think we have the following options:
python-dev commentedon Mar 16, 2016
New changeset 811ec2860dc4 by Martin Panter in branch 'default':
Issue bpo-21042: Revert Linux find_library() to return just filename
https://hg.python.org/cpython/rev/811ec2860dc4
vadmium commentedon Mar 16, 2016
I reverted the change until we can come up with something more consistent.
CharlesCoulombe commentedon Dec 9, 2021
Any update on this issue?
This would be helpful to HPC systems that don't have libraries installed in standard place, and to standardize find_library as well!
jackjansen commentedon May 2, 2022
I would like this feature too (possibly optionally, based on an extra parameter
require_full_pathdefaulting to false).On MacOS and Windows I can influence where
find_librarysearches by settingsys.environ["DYLD_LIBRARY_PATH"]or"PATH"before calling onfind_library, but on Linux this doesn't work:find_librarydoes find the library, but the dynamic loader has cached the value of theLD_LIBRARY_PATHon program startup, so it doesn't see my runtime changes, so it can't find the unqualified library name.mara004 commentedon Dec 20, 2023
I also think it would be useful to have the full path info available to avoid uncertainty as to which file was loaded.
encukou commentedon Feb 23, 2026
I'm proposing to soft-deprecate
ctypes.util.find_library: https://discuss.python.org/t/106232encukou commentedon Mar 20, 2026
It is now soft-deprecated. It cannot be made to work correctly and consistently on all platforms, and any change runs a chance to break whatever workarounds people already have in place.
I'm prioritizing stability over new features, and encourage solving this outside ctypes itself.