Repository navigation
Free-threading QSBR delayed free mem fails in high thread turnover environment #130794
Description
Crash report
What happened?
When cycling a lot of threads and modifying objects which use PyMem_FreeDelayed() in those threads the queues of memory blocks to free build up without ever being freed until the process runs out of memory, unless something else causes them to be freed. For example in the reproducer below commenting in the gc.collect() or the for loop will cause the delayed blocks to be freed correctly. Note that this is an extreme case.
Reproducer:
import gc
import threading
def fupmem(b, l):
b.wait()
l *= 2
def check(funcs, *args):
barrier = threading.Barrier(len(funcs))
thrds = []
for func in funcs:
thrd = threading.Thread(target=func, args=(barrier, *args))
thrds.append(thrd)
thrd.start()
for thrd in thrds:
thrd.join()
if __name__ == "__main__":
count = 0
while True:
print(count := count + 1)
check([fupmem] * 10, [None] * 256)
# gc.collect()
# for i in range(1000):
# l = []
# del lCPython versions tested on:
3.14
Operating systems tested on:
Linux
Output from running 'python -VV' on the command line:
Python 3.14.0a5+ experimental free-threading build (heads/main:b3c18bfd828, Mar 3 2025, 09:13:46) [GCC 11.4.0]
Activity
tom-pytel commentedon Mar 3, 2025
ping @colesbury
colesbury commentedon Mar 3, 2025
Thanks for the bug report. Typically, each thread periodically processes their queue of memory blocks to be freed. This happens every 254 calls to
_PyMem_FreeDelayed(). The problems is that:_PyMem_FreeDelayed(), so they don't call_PyMem_ProcessDelayed._PyMem_ProcessDelayedThe fix is relatively straightforward: when the thread exits and merges its queue into the shared per-interpreter queue, it should also take the opportunity to process the shared queue.
pythongh-130794: Process interpreter QSBR queue in _PyMem_AbandonDela…
Merge branch 'main' into pythongh-130794-process-queue
gh-130794: Process interpreter QSBR queue in _PyMem_AbandonDelayed. (g…
[3.13] pythongh-130794: Process interpreter QSBR queue in _PyMem_Aban…
[3.13] gh-130794: Process interpreter QSBR queue in _PyMem_AbandonDel…
kumaraditya303 commentedon Mar 11, 2025
Closing as it was fixed by #130808 and backported.