Repository navigation
Race in concurrent list mutation and item retrieval #129069
Description
Bug report
Bug description:
A data race exists in lists when they are concurrently updated and indexed (or otherwise accessed): for slice assignment and other ways of adding or removing multiple items (list.extend, in-place multiplication), memory writes to the newly allocated areas happen with memcpy/memmove, which have no write size/atomicity guarantees (and it's not unlikely that they will perform jagged writes). These reallocations happen with the lock held, but since we're relying on QSBR, concurrent reads of those memory areas are technically possible.
The tests in PR #128798 (test_free_threading.test_iteration), when run under ThreadSanitizer, expose those races. (They are suppressed in Tools/tsan/suppressions_free_threading.txt).
CPython versions tested on:
CPython main branch
Operating systems tested on:
No response
Linked PRs
- gh-129069: make list ass_slice and memory_repeat safe in free-threading #131882
- gh-129069: Fix listobject.c data races due to memmove #142957
- gh-129069: fix more thread safety issues in list #143019
- gh-129069: fix TSAN-0013 from gh-153852 #154516
- [3.14] gh-129069: fix more thread safety issues in
list(GH-143019) #158488
Activity
colesbury commentedon Jan 20, 2025
I'm a bit confused by the "memory writes to the newly allocated areas happen with memcpy/memmove" part of the issue description.
The data races with
memcpyI've seen involve writes to the existing item array. For example, inlist_ass_slice_lock_held, we havememmovecalls withself->ob_itemandrecycleas the destinations. The ones with the newly allocatedrecycleas the destination are okay, but the ones withself->ob_itemare potentially a problem.Yhg1s commentedon Jan 20, 2025
Yes, you're right, 'newly allocated' isn't really relevant. I just meant that the calls that move items around before/after list_resize() are potentially problematic. I have to admit my head hurts every time I try to trace through list_ass_slice_lock_held.
pythongh-129069: make list ass_slice and memory_repeat safe
pythongh-129069: Fix listobject.c data races due to memmove
Merge branch 'main' into pythongh-129069-list-memmove
gh-129069: Fix listobject.c data races due to memmove (gh-142957)
pythongh-129069: Fix listobject.c data races due to memmove (pythongh…
gh-129069: fix more thread safety issues in `list` (#143019)