Repository navigation
marshal.dumps() crashes when a set is modified while it is being marshalled #159185
Description
Crash report
To sort the elements of a set, w_complex_object() reads the size of the
set, allocates a list of that size, then marshals each element on its own
in a loop. Marshalling an element can run Python code (an element's
__buffer__, an audit hook), and that code can change the set. If the set
grows, the loop writes past the end of the list. If it shrinks, the list
is left with NULL items and is then sorted.
import marshal, sys
class Item:
def __hash__(self):
return 0
def __buffer__(self, flags):
if sys.argv[1] == 'grow':
s.update(range(1000, 1400))
else:
s.difference_update(range(1, 60))
return memoryview(b'abc')
s = {Item()}
s.update(range(1, 60))
marshal.dumps(s)On main:
shrink, release build: segmentation fault.grow, release build: no error; writes past the list allocation.grow, debug build:
PyList_SET_ITEM: Assertion 'index < list->allocated' failed.
On the free-threaded build another thread changing the set between the size
read and the critical section has the same effect.
Proposed fix
Stop the loop at the size that was read, and raise
RuntimeError("set changed size during iteration") if the number of
elements seen differs from it, as iterating a set does.
Discovered using Claude while doing a deep dive on pyc compilation determinism.
CPython versions tested on:
CPython main branch
Operating systems tested on:
No response
Output from running 'python -VV' on the command line:
No response
Linked PRs
- gh-159185: Fix marshalling of a set modified while its elements are marshalled #159195
- [3.15] gh-159185: Fix marshalling of a set modified while its elements are marshalled (GH-159195) #159199
- [3.14] gh-159185: Fix marshalling of a set modified while its elements are marshalled (GH-159195) #159200
- gh-159185: Document that PyMarshal_WriteObjectToFile can silently truncate before 3.16 #159201
- [3.15] gh-159185: Document that PyMarshal_WriteObjectToFile can silently truncate before 3.16 (GH-159201) #159202
- [3.14] gh-159185: Document that PyMarshal_WriteObjectToFile can silently truncate before 3.16 (GH-159201) #159203
Activity
gh-159185: Fix marshalling of a set modified while its elements are m…
gpshead commentedon Oct 11, 2026
Automated backports failed because main has gh-156942, which is not on 3.15 or 3.14. On main the marshal writer raises the exception where the error is detected. On the older branches it stores a
WFERR_*code and turns it into an exception at the end, so an exception set inside the loop would be overwritten withValueError("unmarshallable object").The backport therefore follows the branch's own pattern: a new
WFERR_SET_CHANGED_SIZEcode, set where main raises directly, and a new case in the final switch that raisesRuntimeError("set changed size during iteration"). The in-loop check and the test are the same as on main. The marshal.c change is identical on both branches.One behaviour to know about
PyMarshal_WriteObjectToFile()still raises nothing on these branches. 😦 It ignores every marshalling error there; reporting them is gh-155907, which is not backported. With the fix it no longer crashes on a mutated set, but it returns silently after writing a truncated 5-byte file.marshal.dumps(),marshal.dump()andPyMarshal_WriteObjectToString()do raise. The NEWS entry only names dumps and dump, so it is still accurate.... keeping them draft while considering if the above should be backported first so it is a happier situation. [edit: Verdict, not backporting those].
gh-159185: Document that PyMarshal_WriteObjectToFile can silently tru…
[3.14] gh-159185: Fix marshalling of a set modified while its element…
[3.15] gh-159185: Fix marshalling of a set modified while its element…
[3.15] gh-159185: Document that PyMarshal_WriteObjectToFile can silen…
gpshead commentedon Oct 11, 2026
the doc note I added isn't anything I'd call pretty but this shouldn't be a widely used API and it at least reflects the pre-existing reality of its deficiencies.
[3.14] gh-159185: Document that PyMarshal_WriteObjectToFile can silen…