Repository navigation
threading.enumerate(): reentrant call during a GC collection hangs #88588
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2021-06-16.09:44:01.520>
created_at = <Date 2021-06-14.21:26:28.622>
labels = ['library', '3.9', '3.10', '3.11']
title = 'threading.enumerate(): reentrant call during a GC collection hangs'
updated_at = <Date 2021-06-16.09:44:01.519>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2021-06-16.09:44:01.519>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-06-16.09:44:01.520>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2021-06-14.21:26:28.622>
creator = 'vstinner'
dependencies = []
files = ['50110']
hgrepos = []
issue_num = 44422
keywords = ['patch']
message_count = 8.0
messages = ['395851', '395852', '395880', '395881', '395883', '395884', '395885', '395914']
nosy_count = 2.0
nosy_names = ['vstinner', 'miss-islington']
pr_nums = ['26727', '26737', '26738', '26741']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue44422'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']
Activity
vstinner commentedon Jun 14, 2021
The threading.enumerate() code is simple:
---
values(), list() and list+list operations can call trigger a GC collection depending on the GC thresholds, created Python objects, etc.
During a GC collection, a Python object destructor (del) can again call threading.enumerate().
Problem: _active_limbo_lock is not re-entrant lock and so the second call hangs.
In practice, this issue was seen in OpenStack which uses eventlet, when a destructor uses the logging module. The logging module uses threading.current_thread(), but this function is monkey-patched by the eventlet module.
Extract of the eventlet function:
---
https://github.com/eventlet/eventlet/blob/master/eventlet/green/threading.py
Attached PR makes _active_limbo_lock re-entrant to avoid this issue. I'm not sure if it's ok or not, I would appreciate to get a review and your feedback ;-)
Attached file triggers the threading.enumerate() hang on re-entrant call.
vstinner commentedon Jun 14, 2021
There are different ways to fix this issue:
(A) The problem is that functions other than threading.enumerate()
also rely on this lock, like threading.active_count(). I would prefer to not have to rewrite "half" of threading.py in C. Using a RLock is less intrusive.
(B) This is a simple and reliable option. But gc.disable() is process-wide: it affects all Python threads, and so it might have surprising side effects. Some code might rely on the current exact GC behavior. I would prefer to not disable the GC temporarily.
vstinner commentedon Jun 15, 2021
New changeset 243fd01 by Victor Stinner in branch 'main':
bpo-44422: Fix threading.enumerate() reentrant call (GH-26727)
243fd01
miss-islington commentedon Jun 15, 2021
New changeset c3b776f by Miss Islington (bot) in branch '3.10':
bpo-44422: Fix threading.enumerate() reentrant call (GH-26727)
c3b776f
vstinner commentedon Jun 15, 2021
New changeset 8fe57aa by Miss Islington (bot) in branch '3.9':
bpo-44422: Fix threading.enumerate() reentrant call (GH-26727) (GH-26738)
8fe57aa
vstinner commentedon Jun 15, 2021
Ok, the deadlock on reentrant call to threading.enumerate() is now fixed in 3.9, 3.10 and main (future 3.11) branches. I close the issue. Thanks for the reviews, as usual :-)
vstinner commentedon Jun 15, 2021
Oh, I reopen the issue for PR 26741.
vstinner commentedon Jun 16, 2021
New changeset 0729694 by Victor Stinner in branch 'main':
bpo-44422: threading.Thread reuses the _delete() method (GH-26741)
0729694