Repository navigation
Fail PyGILState_Ensure() If Finalizing? #124622
Description
Feature or enhancement
Proposal:
Once the runtime or current interpreter starts finalizing, we currently exit or stall the current thread as soon as it tries to acquire the GIL, via PyThread_exit_thread() (or PyThread_hang_thread() after gh-105805). That includes in PyGILState_Ensure(). However, in the case of the latter it might make more sense (i.e. friendlier) for the call to fail instead of exiting/hanging the current thread.
FWIW, this is a hypothetical situation I noticed while looking at pystate.c. Also, the same scenario would be affected by gh-124619.
CC @gpshead
Has this already been discussed elsewhere?
No response given
Links to previous discussion of this feature:
No response
Activity
gpshead commentedon Sep 26, 2024
A challenge in doing this is that
PyGILState_Ensurereturns anenum PyGILState_STATEvalue (defined in Include/pystate.h). That only contains LOCKED and UNLOCKED states today, so callers of the function in existing compiled Python C API code will never expect other values. We'd need to know if they were compiled against a recent enough Python version in order to return a new enum value representing an error because the interpreter is being finalized thus no GIL acquisition could possibly succeed.gpshead commentedon Sep 26, 2024
I think a new GIL acquisition and release C API would be needed. The way the existing ones get used in existing C code is not amenible to suddenly bolting an error state onto; none of the existing C code is written that way. After the call they always just assume they have the GIL and can proceed. The API was designed as "it'll block and only return once it has the GIL" without any other option.
ZeroIntensity commentedon Sep 27, 2024
This is something that seems fatal-error worthy rather than a new
PyGILState_STATEvalue -- there's nothing much a dev can do if they want to call the C API during a finalizing interpreter.That's been loosely discussed recently (in the context of subinterpreters, rather than the GIL), see #123728 (comment)
rruuaanng commentedon Oct 31, 2024
If you don't mind, I will sort out the requirements under discussion and publish some issue in the near future.
gpshead commentedon Feb 2, 2025
Tying threads together, a new
PyGILState_EnsureOrSafelyFail()is suggested for this issue in https://discuss.python.org/t/safely-using-the-c-api-when-python-might-shut-down/78850 as allowing modern extension code to actually be able to detect failure would be a good thing.ZeroIntensity commentedon Feb 3, 2025
I don't think it's a good idea to add new things with the term "GIL" in them. I'd prefer a totally new API design (e.g.,
PyThreadState_Ensure) that takes an explicit interpreter rather than an implicit one.ZeroIntensity commentedon Feb 4, 2025
Ok, I've been playing around with implementing this. I think we need to very clearly define two things:
Py_Finalizeassumes that there's only one thread left, so there's no mechanism in place for getting rid of new thread states. (I do have an open PR that changes this for subinterpreters, but not the main interpreter yet.)pythongh-124622: Add PyGILState_EnsureOrFail() function
pythongh-124622: Add PyGILState_EnsureOrFail() function
vstinner commentedon Feb 5, 2025
I created #129688 to add
PyGILState_EnsureOrFail()function.pythongh-124622: Add PyThreadState_Ensure() function
pythongh-124622: Add PyThreadState_Ensure() function
vstinner commentedon Feb 14, 2025
See also comments on a previous similar attempt: #123728