Repository navigation
dir() can raise RuntimeError: dictionary changed size during iteration on the free-threaded build #157217
Description
Bug report
Bug description:
I believe there is an issue in the free-threaded build when using dir(). Apologies if this isn't considered an issue. I read https://docs.python.org/3/howto/free-threading-python.html#thread-safety and my understanding of:
Built-in types like dict, list, and set use internal locks to protect against concurrent modifications in ways that behave similarly to the GIL.
is that this is something that could be fixed. Possibly this was missed when implementing #114508.
Calling dir() on a class (or on an instance of it) can fail with RuntimeError: dictionary changed size during iteration if another thread concurrently performs a read that lazily stores something in the class __dict__. The most common such read on 3.14+ is the first access to __annotations__ (e.g. through typing.get_type_hints()), which stores __annotations_cache__ on the class.
Same can happen with e.g. copy.copy(), which will set __slotnames__ via copyreg._slotnames().
MRE
import sys
import threading
import typing
from concurrent.futures import ThreadPoolExecutor
def make_class():
class C:
x: int
# Insert many elements in the class dict so that `dir()` spends more time
# iterating over it (the race also happens without this, just less often):
for i in range(1000):
setattr(C, f'attr_{i}', i)
return C
def main(rounds: int = 100, readers: int = 8) -> None:
failures = 0
for _ in range(rounds):
C = make_class()
barrier = threading.Barrier(readers + 1)
errors = []
def read():
barrier.wait()
try:
dir(C)
except RuntimeError as e:
errors.append(e)
def write():
barrier.wait()
# First access to `C.__annotations__` (here, through `get_type_hints()`) stores
# `__annotations_cache__` in the class `__dict__`.
typing.get_type_hints(C)
with ThreadPoolExecutor(max_workers=readers + 1) as executor:
futures = [executor.submit(read) for _ in range(readers)] + [executor.submit(write)]
for future in futures:
future.result()
failures += len(errors)
if errors and failures == len(errors):
print(f'{type(errors[0]).__name__}: {errors[0]}')
gil = sys._is_gil_enabled()
print(f'{sys.version.split()[0]} ({"GIL" if gil else "free-threaded"}): {failures}/{rounds * readers} dir() calls failed')
if __name__ == '__main__':
main()$ python3.15t mre.py
RuntimeError: dictionary changed size during iteration
3.15.0rc2 (free-threaded): 108/800 dir() calls failed
$ PYTHON_GIL=1 python3.14t mre.py
3.15.0rc2 (GIL): 0/800 dir() calls failed
Analysis
AI analysis pointed me at the following. I'm not knowledgeable to know if this is the actual issue, but can help for initial debugging.
The reader: dir()
type.__dir__() and object.__dir__() collect names with merge_class_dict(), which fetches cls.__dict__ and merges it into a fresh dict with PyDict_Update():
Lines 6965 to 6982 in 894af95
cls.__dict__ is a mappingproxy, not a dict, so dict_merge() doesn't take the fast path (which holds the critical sections of both dicts). It takes the generic path instead, which only holds the critical section of the destination dict:
Lines 4309 to 4324 in 894af95
The generic path gets the keys with PyMapping_Keys(), which for a non-dict calls .keys() and turns the returned dict_keys view into a list by iterating over it:
Lines 2433 to 2468 in 894af95
That iteration over the class __dict__ happens without holding its critical section, and the dict iterator checks the size at every step, so any concurrent insertion into the class __dict__ raises RuntimeError.
The writer: a lazy cache stored on the class by a read
The type.__annotations__ getter stores the evaluated annotations as __annotations_cache__ in the class __dict__ on first access:
Lines 2194 to 2201 in 894af95
If the class has no __annotate__ function, the type.__annotate__ getter (called by the above) also stores __annotate_func__ = None:
Lines 2086 to 2092 in 894af95
Other stdlib operations do the same kind of lazy insertion into a class __dict__, so the issue is not specific to annotations:
-
copy.copy(),copy.deepcopy()and pickling of an instance store__slotnames__on the class, on every version, through_PyType_GetSlotNames()/copyreg._slotnames():Lines 7790 to 7795 in 894af95
Lines 155 to 158 in 894af95
-
The first instantiation of a concrete subclass of a
typing.Protocolreplaces its__init__():Lines 1925 to 1931 in 894af95
Other affected readers
Everything relying on dir() is affected, e.g. inspect.getmembers() and inspect.classify_class_attrs() (which additionally iterate over base.__dict__.items() themselves), or unittest.mock's autospec. dict(vars(cls)) (used e.g. by typing.get_type_hints() for the evaluation locals) goes through the same dict_merge() generic path:
Line 2451 in 894af95
Possible fix
In dict_merge() (or in PyMapping_Keys()), unwrap a mappingproxy whose underlying mapping is a dict and use the locked dict-to-dict path. This would fix dir(), dict(vars(cls)), {**vars(cls)} and everything built on them at once. The pure-Python loops over base.__dict__.items() in inspect and enum.Enum.__dir__ would still need to iterate over a copy.
CPython versions tested on:
3.15
Operating systems tested on:
macOS
Activity
search1ofall-maker commentedon Sep 10, 2026
Analysis & Proposed Fix:
The issue diagnosis is spot on. When dir() calls merge_class_dict(), classdict is a mappingproxy (PyDictProxy_Type). Because PyAnyDict_Check(b) evaluates to false for proxies in dict_merge() (Objects/dictobject.c), CPython falls back to the generic PyMapping_Keys() iteration path, bypassing the critical section on the underlying dictionary.
Proposed Solution:
We can handle mappingproxy objects directly in dict_merge() by unwrapping the proxy to its underlying mapping before performing the critical section lock and merge:
Check if PyDictProxy_Check(b) is true.
Extract the underlying dict via ((PyDictProxyObject *)b)->mapping.
Pass the underlying dict to Py_BEGIN_CRITICAL_SECTION2(a, underlying_dict) and run dict_dict_merge().
This will fix thread-safety for dir(), dict(vars(cls)), and {**vars(cls)} in free-threaded builds without adding per-iteration lock overhead in Python land.
I can submit a PR with this fix and a regression test for test_free_threaded if the core devs agree with this approach.
pythongh-157217: Fix dir() race when merging from a mappingproxy
pythongh-157217: Export _PyDictProxy_GetMapping for dict_merge
pythongh-157217: Add free-threading dir() race test
pythongh-157217: Cover mappingproxy in dict.update tests
pythongh-157217: Cover mappingproxy in dict.update tests
pythongh-157217: Cover mappingproxy in dict.update tests
pythongh-157217: Add mappingproxy accessor for dict_merge
pythongh-157217: Add NEWS blurb for mappingproxy dict_merge fix
pythongh-157217: Add mappingproxy dict() and update tests
pythongh-157217: Add free-threading dir() race test
Add dictobject.c hunk for pythongh-157217
2 remaining items
pythongh-157217: Drop unused mappingproxy helper from internal header
pythongh-157217: Follow-up hunk to unwrap mappingproxy locally
pythongh-157217: Unwrap mappingproxy locally in dict_merge
pythongh-157217: Fix dir() race when merging from a mappingproxy
pythongh-157217: Fix ruff F841/E731 in test_type.py for Lint/prek
pythongh-157217: Fix ruff F841/E731 in test_type.py for Lint/prek
ci: one-shot squash helper for pythongh-157217-fix
pythongh-157217: Fix dir() race when merging from a mappingproxy
Merge python/cpython main into pythongh-157217-fix
Complete pythongh-157217 mappingproxy race fix on current main
Merge branch 'main' into pythongh-157217-fix
dpdani commentedon Sep 25, 2026
This doesn't look like a bug to me. The interpreter raises
RuntimeError: dictionary changed size during iterationbecause that's precisely what happens in the MRE. The fact that in a with-GIL build this cannot be exercised because of the lack of parallelism doesn't make this a bug.Viicos commentedon Sep 25, 2026
So if this isn't a bug/isn't something that should be fixed, what am I misinterpreting in the docs section I mentioned?
dpdani commentedon Sep 28, 2026
From the docs you mention:
Pointing to:
I have no idea why you have an issue with this case specifically, is it a problem that came up as a bug report in a typing library? Or is it just theoretical? The problem of concurrently calling
dir()doesn't look like a big issue to me.Viicos commentedon Sep 28, 2026
I believe this make sense if you are iterating over the items yourself using
items(), butdir()is a builtin, it would feel odd to me to hold an external lock every time one wants to usedir()on a class.It is theoretical (in the sense that I did not encounter the issue directly, but as seen in repro can still happen in practice), which I found when investigating other free-threaded race conditions in Pydantic. I think this is pretty rare in practice.
PyDict_Update()(used bytype.__dir__()) documentation states:So in some way, the thread safety concern is documented (in our case, b would be the
__dict__of thedir()argument). But havingdir()usingPyDict_Update()is an implementation detail, anddir()could very much go through a different variant where it holds a lock for both dicts.