You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I think backporting PyMutex would be nontrivial, since it relies on integration with mimalloc AIUI.
Instead, you could follow what we did in NumPy and explicitly wrap PyThread_type_lock and PyMutex in a compatibility macro, or similar. numpy/numpy#27011
I'm not sure if it makes sense to put something like that in pythoncapi-compat.
I was hopping that it would have been possible to have a generic way to implement it here, maybe using PyThread_type_lock internally.
But after playing for a while with your suggestion (which was also Emma's), I'm starting to wonder if it's possible at all, as PyMutex API lacks the init & free methods.
But my knowledge about both C and the CPython C API is limited, so I would be happy if someone proves me wrong!
In the meanwhile, I will work following the suggestion previously mentioned. Thanks!
PyMutex doesn't need to be heap-allocated, so you can leave the allocate and free functions as stubs. The docs for PyMutex say you need to initialize like it like e.g.
PyMutex mut = {0};
But that's the only special thing you need to handle.
I was thinking if it was possible, as a compatibility layer to PyMutex, to implement PyMutex using PyThread_type_lock for older versions of Python not having PyMutex.
Like you mentioned, since the initialization of PyMutex in code that would use it does not call any function, I don't see where the call to PyThread_allocate_lock can be hooked up in the compatibility layer.
Couldn't the compat function just not do anything to allocate and free the object in the PyMutex branch that 3.13 and newer use?
For example, in NpyString_free_allocator in NumPy, there's an ifdef so that on older python versions we free the PyThread_type_lock mutex that hangs off the allocator but on newer versions we do nothing because PyMutex doesn't need to be free'd.
There's a similar ifdef in NpyString_new_allocator. You can grep static_string.c in NumPy for PY_VERSION_HEX to find all the version-dependent code. It turns out that StringDType was the only place where we were using PyThread_type_lock before we added free-threaded support and we're only using PyMutex elsewhere in code paths that are only unsafe under the free-threaded build.
PyMutex was added to Python 3.13 and it's part of the public C API since Python 3.13. You don't need pythoncapi-compat on Python 3.13 and newer.
Backporting the whole feature to Python 3.12 would be too much work (add too much code, complicated to maintain).
I suggest closing this issue. You have to add two different code paths depending on the Python version: PyMutex for Python 3.13 and newer, or PyThread_allocate_lock() on Python 3.12 and older.g
By the way, I'm working on python/cpython#134747 : "Change PyThread_allocate_lock() implementation to PyMutex" on Python 3.15 :-)
I understand that backporting PyMutex for Python 3.12 and below is not something you want to support in this repository. Instead, users should make the necessary changes in their code to use either PyMutex or PyThread_allocate_lock based on the Python version.
Activity
ngoldbaum commentedon May 29, 2025
I think backporting PyMutex would be nontrivial, since it relies on integration with mimalloc AIUI.
Instead, you could follow what we did in NumPy and explicitly wrap PyThread_type_lock and PyMutex in a compatibility macro, or similar. numpy/numpy#27011
I'm not sure if it makes sense to put something like that in pythoncapi-compat.
Rogdham commentedon May 29, 2025
I was hopping that it would have been possible to have a generic way to implement it here, maybe using
PyThread_type_lockinternally.But after playing for a while with your suggestion (which was also Emma's), I'm starting to wonder if it's possible at all, as
PyMutexAPI lacks the init & free methods.But my knowledge about both C and the CPython C API is limited, so I would be happy if someone proves me wrong!
In the meanwhile, I will work following the suggestion previously mentioned. Thanks!
ngoldbaum commentedon May 29, 2025
PyMutex doesn't need to be heap-allocated, so you can leave the allocate and free functions as stubs. The docs for PyMutex say you need to initialize like it like e.g.
But that's the only special thing you need to handle.
Rogdham commentedon May 29, 2025
Sorry, I was not clear at all in my last message.
I was thinking if it was possible, as a compatibility layer to
PyMutex, to implementPyMutexusingPyThread_type_lockfor older versions of Python not havingPyMutex.Like you mentioned, since the initialization of
PyMutexin code that would use it does not call any function, I don't see where the call toPyThread_allocate_lockcan be hooked up in the compatibility layer.Likewise for calling
PyThread_free_lock.ngoldbaum commentedon May 29, 2025
Couldn't the compat function just not do anything to allocate and free the object in the
PyMutexbranch that 3.13 and newer use?For example, in
NpyString_free_allocatorin NumPy, there's an ifdef so that on older python versions we free thePyThread_type_lockmutex that hangs off the allocator but on newer versions we do nothing becausePyMutexdoesn't need to be free'd.https://github.com/numpy/numpy/blob/3f1b4571cedefe505a281a2ed5c865f1e6e35e63/numpy/_core/src/multiarray/stringdtype/static_string.c#L269-L284
There's a similar ifdef in
NpyString_new_allocator. You can grepstatic_string.cin NumPy forPY_VERSION_HEXto find all the version-dependent code. It turns out that StringDType was the only place where we were usingPyThread_type_lockbefore we added free-threaded support and we're only usingPyMutexelsewhere in code paths that are only unsafe under the free-threaded build.Does that make sense?
vstinner commentedon May 29, 2025
PyMutexwas added to Python 3.13 and it's part of the public C API since Python 3.13. You don't need pythoncapi-compat on Python 3.13 and newer.Backporting the whole feature to Python 3.12 would be too much work (add too much code, complicated to maintain).
I suggest closing this issue. You have to add two different code paths depending on the Python version:
PyMutexfor Python 3.13 and newer, orPyThread_allocate_lock()on Python 3.12 and older.gBy the way, I'm working on python/cpython#134747 : "Change PyThread_allocate_lock() implementation to PyMutex" on Python 3.15 :-)
Rogdham commentedon May 29, 2025
I understand that backporting
PyMutexfor Python 3.12 and below is not something you want to support in this repository. Instead, users should make the necessary changes in their code to use eitherPyMutexorPyThread_allocate_lockbased on the Python version.Thank you for the pointers!