ctypes thread safety auditing (and fixing)

@encukou

Feature or enhancement

This is a tracking issue for all thread safety problems related to ctypes. I'll be working on this, but others can feel free to give me some help.

First of all, we need to find where the thread safety problems are.

### Auditing
- [x] Audit `_ctypes.c`
- [ ] Audit `_ctypes_test.c` (and probably the generated file too)
- [ ] Audit `callbacks.c`
- [ ] Audit `callproc.c`
- [ ] Audit `cfield.c`
- [x] Audit `malloc_closure.c`
- [ ] Audit `stgdict.c`

I'll be tracking the issues that get found when auditing here. The plan is to just create a new issue and link to it for each new problem instead of flooding this issue with PRs.

Generally, the workflow for fixes should follow most of the rules from #116738, but I suspect we'll need recursive mutexes for quite a few things related to callbacks, because it's difficult to tell what might be re-entrant, and we can't use critical sections for arbitrary function pointers.

### Known Issues
- [ ] https://github.com/python/cpython/issues/127946
- [ ] https://github.com/python/cpython/issues/128182
- [ ] https://github.com/python/cpython/issues/128485
- [ ] https://github.com/python/cpython/issues/128567
- [ ] https://github.com/python/cpython/issues/128570
- [ ] https://github.com/python/cpython/issues/131974

cc @encukou, as the ctypes genius, and @colesbury as the free-threading mastermind.

Linked PRs

Thank you! I can review the ctypes parts :)

Heads-up: I'm currently working in cfield.c/PyCField*; focus elsewhere to avoid conflicts.

Got it, thanks! I'll probably just move to callbacks.c next.

I discussed this with Peter and moving forward I'll be taking over the thread safety work and continue it.

I investigated the thread safety of ctypes and here's the (incomplete) list of issues:

Race in _ctypes_alloc_callback and libffi under free-threading #128485: race in initialization of dlmalloc in _ctypes_alloc_callback. This is fixed by allocating a small memory block and freeing it in module initialization to force initialization of the allocator and is now thread safe with gh-128485: ensure that dlmalloc initializes itself at import time in ctypes #131633.
the freelist in malloc_closure.c is not thread safe in free-threading. This can be fixed by adding lock around the alloc and free functions in free-threading in gh-127945: add locking to malloc closure in free-threading #131662.
concurrent calls to PyCStructUnionType_update_stginfo needs to be made thread safe as it modifies the type's data. This can be fixed by adding critical section around reading and writing to stginfo data in free-threading as using just mutex can lead to re-entrancy deadlocks.
the initialization and concurrent modifications of stginfo is not thread safe
the module level functions such as resize are not thread safe and need to acquire critical section of the object being resized.

I think there are also races on StgInfo.flags, in particular with the DICTFLAG_FINAL bit.

cpython/Modules/_ctypes/_ctypes.c

Lines 718 to 719 in 4596666

    
           info->flags &= ~DICTFLAG_FINAL; /* clear the 'final' flag in the subclass info */ 
        
           baseinfo->flags |= DICTFLAG_FINAL; /* set the 'final' flag in the baseclass info */

I think there are also races on StgInfo.flags, in particular with the DICTFLAG_FINAL bit.

Yes, I am covering that under "the initialization and concurrent modifications of stginfo is not thread safe".

This is different and more important than concurrent modification or initialization of stginfo, and I think it should be one of the early things we should focus on. Concurrent modifications of ctypes Structures is not an important use case.

However, creating separate instances of the same structure is an important use case:

from ctypes import Structure, c_int
from concurrent.futures import ThreadPoolExecutor

class POINT(Structure):
    _fields_ = [("x", c_int), ("y", c_int)]

with ThreadPoolExecutor() as executor:
    futures = [executor.submit(POINT, i, i) for i in range(1000)]
    results = [future.result() for future in futures]

This needs to be both efficient and thread-safe.

gh-127945: add locking to malloc closure in free-threading (#131662)

gh-127945: fix thread safety of ctypes state (#131710)

gh-127945: fix thread safety of creating instances of ctypes structur…

gh-127945: fix thread safety and add lock held assertions to paramfun…

gh-127945: move initialization of field desc to module exec in ctypes (…

gh-127945: change `_ctypes_test.c` static globals to thread local (#1…

gh-127945: fix critical sections around ctypes array (#132646)

gh-127945: skip more tests in ctypes when using parallel threads (#13…

gh-127945: add lock held assertions in ctypes arrays (#132720)

gh-127945: add `test_ctypes` to free-threading TSAN CI (#132727)

Can this issue be closed now as all linked PRs have been merged? Is there anything else that needs to be done?

gh-127945: Update What's New in Python 3.14 for free-threaded ctypes (#…

pythongh-127945: Update What's New in Python 3.14 for free-threaded c…

[3.14] gh-127945: Update What's New in Python 3.14 for free-threaded …

pythongh-127945: Update What's New in Python 3.14 for free-threaded c…

Navigation Menu

Repository navigation

`ctypes` thread safety auditing (and fixing) #127945

Feature or enhancement

Linked PRs

5 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	info->flags &= ~DICTFLAG_FINAL; /* clear the 'final' flag in the subclass info */
	baseinfo->flags \|= DICTFLAG_FINAL; /* set the 'final' flag in the baseclass info */

ctypes thread safety auditing (and fixing) #127945

Description

Feature or enhancement

Linked PRs

Activity

encukou commented on Jan 10, 2025

ZeroIntensity commented on Jan 10, 2025

kumaraditya303 commented on Mar 20, 2025

kumaraditya303 commented on Mar 24, 2025

colesbury commented on Mar 24, 2025

kumaraditya303 commented on Mar 24, 2025

colesbury commented on Mar 24, 2025

5 remaining items

sergey-miryanov commented on May 15, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`ctypes` thread safety auditing (and fixing) #127945