Repository navigation
"mem" and "object" Allocators are No Longer Protected by the GIL #105766
Description
Once we moved to per-interpreter GIL, the promises in the docs no longer hold:
...where the allocation must be performed with the GIL held.
It's still fine for pymalloc, but any custom, non-wrapping "mem"/"object" allocators would need to be updated to be thread-safe or per-interpreter.
I have a PR up that does an okay job of adapting such allocators: gh-105619. However, it penalizes use of such an allocator in subinterpreters that have their own GIL.
Honestly, I'm leaning toward documenting that such allocators must be thread-safe or per-interpreter. From what I understand, the documented guarantees (in the docs and in PEP 445) are more about representing what pymalloc needs than what custom allocators need.
Perhaps the biggest question is: what projects would be impacted? I haven't had a chance yet to search for projects that use custom mem/object allocators that aren't thread-safe. I suspect there aren't more than two or three.
Activity
encukou commentedon Jun 14, 2023
Can we add a flag, so that e.g.
PyMem_SetAllocator(PYMEM_DOMAIN_MEM | PyMEM_THREADSAFE, ...)would setPYMEM_DOMAIN_MEMwithout the wrappers?ericsnowcurrently commentedon Jul 26, 2023
python-dev thread: https://discuss.python.org/t/what-to-do-about-custom-allocators-and-the-gil/30327
ericsnowcurrently commentedon Jul 28, 2023
I'm going to add a note in the docs that custom allocators (that have their own state), including hooks/wrappers, must be thread-safe.
We could also qualify that to only when subinterpreters are used, but I'm not sure that matters. If we did qualify it that way, it may also make sense to add the flag @encukou suggested above (and its inverse).
gh-105766: Document that Custom Allocators Must Be Thread-Safe (gh-10…
pythongh-105766: Document that Custom Allocators Must Be Thread-Safe (p…
[3.12] gh-105766: Document that Custom Allocators Must Be Thread-Safe (…