Repository navigation
Statically linked sqlite3 fails checks due to missing -lm #128321
Description
Bug report
Bug description:
On Linux, if I build a static version of sqlite3 (e.g., ./configure --disable-shared && make from a sqlite3 tarball)
You can see that ./configure for CPython fails the SQLite checks (added back in #30016):
$ ./configure 2>&1 | grep sqlite
checking for sqlite3 >= 3.15.2... yes
checking for sqlite3.h... yes
checking for sqlite3_bind_double in -lsqlite3... no
checking for sqlite3_column_decltype in -lsqlite3... no
checking for sqlite3_column_double in -lsqlite3... no
checking for sqlite3_complete in -lsqlite3... no
checking for sqlite3_progress_handler in -lsqlite3... no
checking for sqlite3_result_double in -lsqlite3... no
checking for sqlite3_set_authorizer in -lsqlite3... no
checking for sqlite3_trace_v2 in -lsqlite3... no
checking for sqlite3_trace in -lsqlite3... no
checking for sqlite3_value_double in -lsqlite3... no
checking for sqlite3_load_extension in -lsqlite3... no
checking for sqlite3_serialize in -lsqlite3... no
checking for --enable-loadable-sqlite-extensions... no
And in the config.log there are a bunch of failures due to undefined references to functions in libm, e.g.:
configure:15469: checking for sqlite3_bind_double in -lsqlite3
configure:15492: gcc -o conftest -I/usr/local/include -I$(srcdir)/Modules/_sqlite -L/usr/local/lib -lsqlite3 -L/usr/local/lib conftest.c -lsqlite3 -L/usr/local/lib -lsqlite3 -ldl >&5
/usr/bin/ld: /usr/local/lib/libsqlite3.a(sqlite3.o): in function `logFunc':
/repo/sqlite-autoconf-3470100/sqlite3.c:131765:(.text+0x26858): undefined reference to `log2'
/usr/bin/ld: /repo/sqlite-autoconf-3470100/sqlite3.c:131768:(.text+0x268d4): undefined reference to `log'
/usr/bin/ld: /repo/sqlite-autoconf-3470100/sqlite3.c:131762:(.text+0x268ec): undefined reference to `log10'
/usr/bin/ld: /repo/sqlite-autoconf-3470100/sqlite3.c:131750:(.text+0x2691c): undefined reference to `log'
CPython is determing SQLite's requirements via pkg-config if it can at:
Lines 4204 to 4209 in 492b224
And -lm is included in the pkg-config for the static sqlite3 build, but it's in the private libs and requires the --static flag to be included in the --libs output:
$ pkg-config --static --libs sqlite3
-L/usr/local/lib -lsqlite3 -lm
$ pkg-config --libs sqlite3
-L/usr/local/lib -lsqlite3
However, PKG_CHECK_MODULES doesn't pass this flag. There's a PKG_CHECK_MODULES_STATIC macro which does, but CPython doesn't seem to use this yet. The alternative is set the flag manually at configure time:
PKG_CONFIG="$PKG_CONFIG --static"
which changes all the PKG_CHECK_MODULES calls to include the --static flag. However, there's a missing export for LIBS which prevents this from resolving the issue:
$ git rev-parse HEAD
f9a5a3a3ef34e63dc197156e9a5f57842859ca04
$ export PKG_CONFIG="pkg-config --static"
$ ./configure 2>&1 | grep sqlite
checking for sqlite3 >= 3.15.2... yes
checking for sqlite3.h... yes
checking for sqlite3_bind_double in -lsqlite3... no
checking for sqlite3_column_decltype in -lsqlite3... no
checking for sqlite3_column_double in -lsqlite3... no
checking for sqlite3_complete in -lsqlite3... no
checking for sqlite3_progress_handler in -lsqlite3... no
checking for sqlite3_result_double in -lsqlite3... no
checking for sqlite3_set_authorizer in -lsqlite3... no
checking for sqlite3_trace_v2 in -lsqlite3... no
checking for sqlite3_trace in -lsqlite3... no
checking for sqlite3_value_double in -lsqlite3... no
checking for sqlite3_load_extension in -lsqlite3... no
checking for sqlite3_serialize in -lsqlite3... no
checking for --enable-loadable-sqlite-extensions... no
checking for stdlib extension module _sqlite3... missing
^C
This is "solved" with the patch:
diff --git a/configure.ac b/configure.ac
index badb19d5589..5ed147f5482 100644
--- a/configure.ac
+++ b/configure.ac
@@ -4221,6 +4221,7 @@ dnl bpo-45774/GH-29507: The CPP check in AC_CHECK_HEADER can fail on FreeBSD,
dnl hence CPPFLAGS instead of CFLAGS.
CPPFLAGS="$CPPFLAGS $LIBSQLITE3_CFLAGS"
LDFLAGS="$LIBSQLITE3_LIBS $LDFLAGS"
+ LIBS="$LIBSQLITE3_LIBS $LIBS"
AC_CHECK_HEADER([sqlite3.h], [
have_sqlite3=yesAs demonstrated by:
$ export PKG_CONFIG="pkg-config --static"
$ ./configure 2>&1 | grep sqlite
checking for sqlite3 >= 3.15.2... yes
checking for sqlite3.h... yes
checking for sqlite3_bind_double in -lsqlite3... yes
checking for sqlite3_column_decltype in -lsqlite3... yes
checking for sqlite3_column_double in -lsqlite3... yes
checking for sqlite3_complete in -lsqlite3... yes
checking for sqlite3_progress_handler in -lsqlite3... yes
checking for sqlite3_result_double in -lsqlite3... yes
checking for sqlite3_set_authorizer in -lsqlite3... yes
checking for sqlite3_trace_v2 in -lsqlite3... yes
checking for sqlite3_value_double in -lsqlite3... yes
checking for sqlite3_load_extension in -lsqlite3... yes
checking for sqlite3_serialize in -lsqlite3... yes
checking for --enable-loadable-sqlite-extensions... no
For context, this was reported in astral-sh/python-build-standalone#449 and some of the findings here are repeated there.
I've posted the patch in #128322
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Activity
zanieb commentedon Dec 29, 2024
cc @erlend-aasland — I understand you're the expert in this area.
zanieb commentedon Dec 29, 2024
In addition to #128322, we may also want to include
-lmatcpython/configure.ac
Line 4207 in 492b224
If the
pkg-configcheck fails, we fallback to including these flags. However, without-lmall of our subsequent checks will fail! Happy to post a pull request for that too.eli-schwartz commentedon Dec 29, 2024
This will result in over-linking when sqlite3 is used as a shared library (no pkg-config --static).
zanieb commentedon Dec 30, 2024
Ah true — we could just include it during the checks but it seems like a bit of a pain.
gh-128321: Set LIBS instead of LDFLAGS when checking sqlite3 requirem…
pythongh-128321: Set LIBS instead of LDFLAGS when checking sqlite3 re…
pythongh-128321: Set LIBS instead of LDFLAGS when checking sqlite3 re…
[3.12] gh-128321: Set LIBS instead of LDFLAGS when checking sqlite3 r…
[3.13] gh-128321: Set LIBS instead of LDFLAGS when checking sqlite3 r…
pythongh-128321: Set LIBS instead of LDFLAGS when checking sqlite3 re…
solvingj commentedon Feb 12, 2025
@zanieb Thank you for finding and fixing this. It's a huge help.
I'm probably missing something, and if so please forgive me.
Since
cpythonwon't be usingpkgconfig --staticany time soon, is it therefor impossible for anyone to compile sqlite3 in statically without patchingconfigurethemselves just to append-lmto theLIBS, and just to successfully satisfy the check? If so, does the overlinking represent a risk for the sqlite symbol check that is worth leaving it in this state, which is basically just broken for this use case?eli-schwartz commentedon Feb 12, 2025
Typically what happens is people who are interested in a static build export
PKG_CONFIG="pkg-config --static"to ensure that all dependencies are detected statically. That will include-lmand also-lz(the latter wasn't mentioned here and it is needed if you build a dynamic python that is statically linked to the sqlite library).So everything works fine using pkg-config, the question is what to do about the fallback.
For a fallback, what you'd typically do is perform a series of compile checks, to see if you can link successfully to sqlite on its own and upon failure check again to see if you can link successfully to sqlite when adding
-lm -lz.I have no objections to the "series of compile checks". I don't think it's fully appropriate in terms of general theory to simply add it unconditionally though. I'm also fine with only supporting pkg-config at all, for my own personal use, but that's another story. :)
solvingj commentedon Feb 12, 2025
I see. It seems to me the project is designed to support static/dynamic choices (and all configuration choices for that matter), on a per-dependency basis. I definitely see it being "very common" that people who want "some deps" compiled in statically want "all deps" statically. On the other hand, it's also "not uncommon" for people to want a subset of deps static. Regardless, it doesn't seem appropriate even in principle to have an "all static or nothing static" support story, just because pkgconfig makes the "all static" case easy. So many other configurations and mechanisms in cpython are in place to make it flexible in this way.