Built Python prefers system packaged libpython when installed; local install should be built with rpath? #784

Hello,

If you happen to pull in the libpython package from the base distro into a container with the same version of Python, the built python will end up using the libpython shared-library from the system-installed package; e.g.

$ sudo docker run -it --network=host python:3.9-bullseye /bin/bash

# ldd /usr/local/bin/python3.9
	linux-vdso.so.1 (0x0000ffff9cc6a000)
	libpython3.9.so.1.0 => /usr/local/lib/libpython3.9.so.1.0 (0x0000ffff9c89a000)
        ...

# apt-get update
...

# apt-get install -y python3.9-dev
...
The following additional packages will be installed:
  libpython3.9 libpython3.9-dev
The following NEW packages will be installed:
  libpython3.9 libpython3.9-dev python3.9-dev
...

# ldd /usr/local/bin/python3.9
	linux-vdso.so.1 (0x0000ffff82c39000)
	libpython3.9.so.1.0 => /usr/lib/aarch64-linux-gnu/libpython3.9.so.1.0 (0x0000ffff82663000)
        ...

AFAICS, --enable-shared as been there basically forever (4faf7f1). From prior comments (#777 (comment)) I believe the intent here is for the /usr/local/bin/python to be strongly isolated from any packaged python.

One option to avoid this might to be to build twice, a bit like the Debian package does, so that /usr/local/bin/python3.X is self-contained, and then copy back the .so library in a second step?

The other option seems to be to add rpath linker flags to ensure that the /usr/local/bin version finds its version first. The binary could be patched post-build; I tried that, e.g. (in the same container as above)

# apt-get install patchelf
...

# patchelf --set-rpath '$ORIGIN/../lib' /usr/local/bin/python3.9

# ldd /usr/local/bin/python3.9
	linux-vdso.so.1 (0x0000ffff873dc000)
	libpython3.9.so.1.0 => /usr/local/bin/../lib/libpython3.9.so.1.0 (0x0000ffff86ffd000)
        ...

Probably that would take some thought on what other bits need patching like that.

I think the other option is to build with LDFLAGS=-Wl,-rpath='$ORIGIN/../lib'? I feel like that should ensure that everything looks relative to itself for libraries.

For full context on this -- I actually found this with Python 3.11 containers, where we had enabled unstable repos since we needed some updated packages in the container. It seems that unstable has had 3.11 packages committed recently, and we managed to pull in libpython3.11. Suddenly /usr/local/bin/python3.11 had a sys.path that only had dist-packages (not site-packages). This led me to finding that the built interpreter was using the .so from the packaged Python.