Skip to content

Importing ctypes results in extra open libffi file descriptor since 3.14 #135893

Open
Open
Importing ctypes results in extra open libffi file descriptor since 3.14#135893
Labels
extension-modulesC modules in the Modules dirtopic-ctypestype-bugAn unexpected behavior, bug, or error

Description

@daandemeyer
daandemeyer
opened on Jun 24, 2025

Bug report

Bug description:

On python 3.13 (Fedora), importing ctypes results in the following output from lsfd:

python3 893377 daandemeyer    mem r-x---  REG   0:37     0  304862113 /usr/lib64/libffi.so.8.1.4
python3 893377 daandemeyer    mem r-----  REG   0:37     0  304862113 /usr/lib64/libffi.so.8.1.4
python3 893377 daandemeyer    mem r-----  REG   0:37     0  304862113 /usr/lib64/libffi.so.8.1.4
python3 893377 daandemeyer    mem rw----  REG   0:37     0  304862113 /usr/lib64/libffi.so.8.1.4

On python 3.14 (Fedora), importing ctypes results in the following output from lsfd:

python3 892407 daandemeyer      5 r-----  REG  0:238  1342       7553 /usr/lib64/libffi.so.8.2.0

This ended up breaking https://github.com/systemd/mkosi because it relied on the fact that no new regular file descriptors were opened by importing ctypes (on python 3.14 the file descriptor shows up in /proc/self/fd whereas on python 3.13 the file descriptors are memory maps which don't show up in /proc/self/fd). I searched through the commit list and couldn't find the commit that introduced this change. It also didn't seem to be documented in the release notes. Can anyone point me to the commit introducing this change and the reasoning behind it?

CPython versions tested on:

3.14

Operating systems tested on:

Linux

Activity

  1. daandemeyer
    added
    type-bugAn unexpected behavior, bug, or error
    on Jun 24, 2025
  2. picnixz
    added
    topic-ctypes
    extension-modulesC modules in the Modules dir
    on Jun 24, 2025

  3. behrmann commented on Jun 24, 2025

    @behrmann
    behrmann
    on Jun 24, 2025

    This can be reproduced outside of Fedora as well with uv run -p python3.14 python and then running lsfd --pid on the interpreter process before and after importing ctypes.

    With python 3.12 and 3.13 the diff of the lsfd output before and after importing ctypes is empty, with 3.14 the diff is

    --- py314_witout	2025-06-24 18:12:47.689673532 +0200
+++ /dev/fd/63	2025-06-24 18:12:59.767811969 +0200
@@ -50,6 +50,7 @@
 python3.14 30153 user    mem r-----  REG btrfs:37     0   22340483 /usr/lib/libpthread.so.0
 python3.14 30153 user    mem r-----  REG btrfs:37     0   22340483 /usr/lib/libpthread.so.0
 python3.14 30153 user    mem rw----  REG btrfs:37     0   22340483 /usr/lib/libpthread.so.0
+python3.14 30153 user    mem r-x---  REG btrfs:58     0   29265231 /home/user/.local/share/uv/python/cpython-3.14.0b2-linux-x86_64-gnu/bin/python3.14
 python3.14 30153 user    shm r-----  REG btrfs:37     0   22341176 /usr/lib/gconv/gconv-modules.cache
 python3.14 30153 user    mem r-----  REG btrfs:37     0   22340447 /usr/lib/ld-linux-x86-64.so.2
 python3.14 30153 user    mem r-x---  REG btrfs:37     0   22340447 /usr/lib/ld-linux-x86-64.so.2
@@ -61,3 +62,4 @@
 python3.14 30153 user      2 rw----  CHR    pts:1    38          4 /dev/pts/1
 python3.14 30153 user      3 rw----  CHR    pts:1    38          4 /dev/pts/1
 python3.14 30153 user      4 rw----  CHR    pts:1    38          4 /dev/pts/1
+python3.14 30153 user      5 r-----  REG btrfs:58   755   29265231 /home/user/.local/share/uv/python/cpython-3.14.0b2-linux-x86_64-gnu/bin/python3.14

    Looking at python-build-standalone they do seem to use the same libffi version except for musl, which could rule out a change in libffi.

  4. vstinner commented on Jun 24, 2025

    @vstinner
    vstinner
    on Jun 24, 2025
    Member

    This new behavior comes from the commit 04d4aac of the issue gh-128485:

    commit 04d4aacaacbe26146e9b5ff6b0043f86f3c543d4
Author: Kumar Aditya <kumaraditya@python.org>
Date:   Mon Mar 24 18:14:25 2025 +0530

    gh-128485: ensure that dlmalloc initializes itself at import time in ctypes  (#131633)

    cc @kumaraditya303

  5. vstinner commented on Jun 24, 2025

    @vstinner
    vstinner
    on Jun 24, 2025
    Member

    This ended up breaking https://github.com/systemd/mkosi because it relied on the fact that no new regular file descriptors were opened by importing ctypes (on python 3.14 the file descriptor shows up in /proc/self/fd whereas on python 3.13 the file descriptors are memory maps which don't show up in /proc/self/fd)

    Can't you update mkosi for this ctypes change?

  6. daandemeyer commented on Jun 25, 2025

    @daandemeyer
    daandemeyer
    on Jun 25, 2025
    ContributorAuthor

    This ended up breaking https://github.com/systemd/mkosi because it relied on the fact that no new regular file descriptors were opened by importing ctypes (on python 3.14 the file descriptor shows up in /proc/self/fd whereas on python 3.13 the file descriptors are memory maps which don't show up in /proc/self/fd)

    Can't you update mkosi for this ctypes change?

    I can but there's nothing preventing cpython from changing this again in the future then. I'm also still confused why there needs to be a regular file descriptor open to libffi, doesn't it just need to be mmapped?

  7. daandemeyer commented on Jun 25, 2025

    @daandemeyer
    daandemeyer
    on Jun 25, 2025
    ContributorAuthor

    To add some more details, the offending function in libffi which opens the file descriptor without closing it is ffi_tramp_get_libffi(). Since we don't make use of closures in mkosi, it should be OK for us to close the file descriptor behind libffi's back.

  8. vstinner commented on Jun 25, 2025

    @vstinner
    vstinner
    on Jun 25, 2025
    Member

    Example to reproduce the issue:

    $ ./python
>>> import os
>>> os.system(f"ls -l /proc/{os.getpid()}/fd/")
total 0
lrwx------ 1 vstinner vstinner 64 25 juin  11:43 0 -> /dev/pts/0
lrwx------ 1 vstinner vstinner 64 25 juin  11:43 1 -> /dev/pts/0
lrwx------ 1 vstinner vstinner 64 25 juin  11:43 2 -> /dev/pts/0
lrwx------ 1 vstinner vstinner 64 25 juin  11:43 3 -> /dev/pts/0
lrwx------ 1 vstinner vstinner 64 25 juin  11:43 4 -> /dev/pts/0
0

# Importing _ctypes creates a new file descriptor (fd 5)
>>> import _ctypes
>>> os.system(f"ls -l /proc/{os.getpid()}/fd/")
total 0
lrwx------ 1 vstinner vstinner 64 25 juin  11:43 0 -> /dev/pts/0
lrwx------ 1 vstinner vstinner 64 25 juin  11:43 1 -> /dev/pts/0
lrwx------ 1 vstinner vstinner 64 25 juin  11:43 2 -> /dev/pts/0
lrwx------ 1 vstinner vstinner 64 25 juin  11:43 3 -> /dev/pts/0
lrwx------ 1 vstinner vstinner 64 25 juin  11:43 4 -> /dev/pts/0
lr-x------ 1 vstinner vstinner 64 25 juin  11:43 5 -> /usr/lib64/libffi.so.8.1.4
0

  9. daandemeyer commented on Jun 25, 2025

    @daandemeyer
    daandemeyer
    on Jun 25, 2025
    ContributorAuthor

    To add some more details, the offending function in libffi which opens the file descriptor without closing it is ffi_tramp_get_libffi(). Since we don't make use of closures in mkosi, it should be OK for us to close the file descriptor behind libffi's back.

    Actually given @behrmann's example above, doing this is very hard, since it's not trivial to detect which file descriptor we need to close.

  10. kumaraditya303 commented on Jun 25, 2025

    @kumaraditya303
    kumaraditya303
    on Jun 25, 2025
    Contributor

    Example to reproduce the issue:

    I am not able to reproduce this on main:

    ❯ ./python
Python 3.15.0a0 (heads/main:15c6d63fe6f, Jun 25 2025, 15:33:31) [Clang 18.1.3 (1ubuntu1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.system(f"ls -l /proc/{os.getpid()}/fd/")
total 0
lrwx------ 1 realkumaraditya realkumaraditya 64 Jun 25 16:05 0 -> /dev/pts/0
lrwx------ 1 realkumaraditya realkumaraditya 64 Jun 25 16:05 1 -> /dev/pts/0
lrwx------ 1 realkumaraditya realkumaraditya 64 Jun 25 16:05 2 -> /dev/pts/0
lrwx------ 1 realkumaraditya realkumaraditya 64 Jun 25 16:05 3 -> /dev/pts/0
lrwx------ 1 realkumaraditya realkumaraditya 64 Jun 25 16:05 4 -> /dev/pts/0
0
>>> import _ctypes
>>> os.system(f"ls -l /proc/{os.getpid()}/fd/")
total 0
lrwx------ 1 realkumaraditya realkumaraditya 64 Jun 25 16:05 0 -> /dev/pts/0
lrwx------ 1 realkumaraditya realkumaraditya 64 Jun 25 16:05 1 -> /dev/pts/0
lrwx------ 1 realkumaraditya realkumaraditya 64 Jun 25 16:05 2 -> /dev/pts/0
lrwx------ 1 realkumaraditya realkumaraditya 64 Jun 25 16:05 3 -> /dev/pts/0
lrwx------ 1 realkumaraditya realkumaraditya 64 Jun 25 16:05 4 -> /dev/pts/0
0
>>>
  11. daandemeyer
    added a commit that references this issue on Jun 25, 2025

    sandbox: Work around extra file descriptor opened by importing ctypes…

    68b3ecc

  12. vstinner commented on Jun 25, 2025

    @vstinner
    vstinner
    on Jun 25, 2025
    Member

    I am not able to reproduce this on main

    Oh. It may depend on the libffi version. I'm using libffi-3.4.6-5.fc42.x86_64 on Fedora 42.

  13. kumaraditya303 commented on Jun 25, 2025

    @kumaraditya303
    kumaraditya303
    on Jun 25, 2025
    Contributor

    Oh. It may depend on the libffi version. I'm using libffi-3.4.6-5.fc42.x86_64 on Fedora 42.

    Okay, I think the best fix is just ignore any file descriptors created by ctypes as the behavior is caused by the underlying libffi library not by the _ctypes extension.

  14. daandemeyer commented on Jun 25, 2025

    @daandemeyer
    daandemeyer
    on Jun 25, 2025
    ContributorAuthor

    Oh. It may depend on the libffi version. I'm using libffi-3.4.6-5.fc42.x86_64 on Fedora 42.

    Okay, I think the best fix is just ignore any file descriptors created by ctypes as the behavior is caused by the underlying libffi library not by the _ctypes extension.

    We cannot ignore any file descriptors because systemd's file descriptor packing protocol depends on being able to move file descriptors to 3, 4, 5, 6, ... . When importing ctypes, libffi's file descriptors squats on file descriptor number 3 so we have to close it to be able to implement the file descriptor packing protocol properly. We're slightly lucky for now that this doesn't break any functionality of ctypes but there is no guarantee that it will stay that way. I would prefer an officially supported way to prevent libffi's file descriptor from being opened like it was before python 3.14.

  15. kumaraditya303 commented on Jun 25, 2025

    @kumaraditya303
    kumaraditya303
    on Jun 25, 2025
    Contributor

    I would prefer an officially supported way to prevent libffi's file descriptor from being opened like it was before python 3.14.

    I think you should open an issue upstream as this happens in libffi, the code change in 3.14 is necessary to ensure thread safety of dlmalloc allocator, you can see its rationale on the corresponding issue.

  16. daandemeyer
    added a commit that references this issue on Jun 25, 2025

    sandbox: Work around extra file descriptor opened by importing ctypes…

    694c8d2

  17. behrmann commented on Jun 25, 2025

    @behrmann
    behrmann
    on Jun 25, 2025

    @kumaraditya303 can you tell us the libffi version you were using in the above version where you were unable to reproduce this?

  18. kumaraditya303 commented on Jun 25, 2025

    @kumaraditya303
    kumaraditya303
    on Jun 25, 2025
    Contributor

    can you tell us the libffi version you were using in the above version where you were unable to reproduce this?

    I tested on libffi 3.4.6 on ubuntu linux.

  19. behrmann commented on Jun 25, 2025

    @behrmann
    behrmann
    on Jun 25, 2025

    That's curious then, since that's the same upstream version @vstinner reports on Fedora. Knowing this I just build main I too get

    Python 3.15.0a0 (heads/main:e5f03b94b6d, Jun 25 2025, 15:04:46) [GCC 15.1.1 20250425] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.getpid()
40249
>>> os.system(f"ls -l /proc/{os.getpid()}/fd/")
total 0
lrwx------ 1 drako drako 64 Jun 25 15:09 0 -> /dev/pts/1
lrwx------ 1 drako drako 64 Jun 25 15:09 1 -> /dev/pts/1
lrwx------ 1 drako drako 64 Jun 25 15:09 2 -> /dev/pts/1
lrwx------ 1 drako drako 64 Jun 25 15:09 3 -> /dev/pts/1
lrwx------ 1 drako drako 64 Jun 25 15:09 4 -> /dev/pts/1
0
>>> import ctypes
>>> os.system(f"ls -l /proc/{os.getpid()}/fd/")
total 0
lrwx------ 1 drako drako 64 Jun 25 15:09 0 -> /dev/pts/1
lrwx------ 1 drako drako 64 Jun 25 15:09 1 -> /dev/pts/1
lrwx------ 1 drako drako 64 Jun 25 15:09 2 -> /dev/pts/1
lrwx------ 1 drako drako 64 Jun 25 15:09 3 -> /dev/pts/1
lrwx------ 1 drako drako 64 Jun 25 15:09 4 -> /dev/pts/1
0

    with libffi 3.5.0-1 on Arch.

  20. behrmann commented on Jun 25, 2025

    @behrmann
    behrmann
    on Jun 25, 2025

    Rebuilding with current 3.14.0b3 and libffi 3.5.0 also can't reproduce this problem.

  21. behrmann commented on Jun 25, 2025

    @behrmann
    behrmann
    on Jun 25, 2025

    It seems like this working on Ubuntu and Arch should be disregarded, since libffi there is built with the --disable-exec-static-tramp option, which was enabled in Fedora for 3.4.4.

    So this should be reproducible on any sufficiently new libffi (one including libffi/libffi@9ba5592) built with support for static trampolines, which should be the default. Though older, this surfaced now because of the combination of the fix for free threading and the option being reenabled in some distros (it was disabled for a while because it broke some other users of libffi).

    Props to @bluca for remembering libffi/libffi@9ba5592

  22. bluca commented on Jun 25, 2025

    @bluca
    bluca
    on Jun 25, 2025

    To reproduce the problem one needs libffi v3.4.0 or greated, with static exec trampolines enabled (ie, this patch: libffi/libffi@9ba5592 ). A lot of distros disabled that when it was introduced due to some regressions in GNOME, which have been fixed some time ago, but for example debian/ubuntu haven't re-enabled the feature yet.

  23. daandemeyer commented on Jun 26, 2025

    @daandemeyer
    daandemeyer
    on Jun 26, 2025
    ContributorAuthor

    Note that currently the tramp file descriptor does not have O_CLOEXEC set and is currently leaked into all child processes that do not take care to close open file descriptors: libffi/libffi#926

  24. vstinner commented on Jun 19, 2026

    @vstinner
    vstinner
    on Jun 19, 2026
    Member

    Note that currently the tramp file descriptor does not have O_CLOEXEC set and is currently leaked into all child processes that do not take care to close open file descriptors: libffi/libffi#926

    It has been fixed in libffi v3.5.2: libffi/libffi@6067118.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    extension-modulesC modules in the Modules dirtopic-ctypestype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions