Repository navigation
Heap buffer overflow in _PyTokenizer_ensure_utf8 #144872
Description
Bug report
Bug description:
OSS-Fuzz has found a heap buffer overflow in _PyTokenizer_ensure_utf8. Link to OSS-Fuzz bug report.
The root cause is that valid_utf8() in Parser/tokenizer/helpers.c checks continuation bytes in reverse order thus reader s[expected] before s[1] on these lines:
cpython/Parser/tokenizer/helpers.c
Lines 497 to 499 in 8b7b5a9
When a multi-byte UTF-8 sequence is truncated - such as a 3-byte lead \xEA followed immediately by a null terminator - the backward loop reads past the end of the valid data before encountering the null byte that would stop it.
This is not a security-critical issue.
CPython versions tested on:
CPython main branch
Operating systems tested on:
No response
Activity
pythongh-144872: move NEWS entry to Core_and_Builtins
gh-144872: fix heap buffer overflow `_PyTokenizer_ensure_utf8` (#144807)
pythongh-144872: fix heap buffer overflow `_PyTokenizer_ensure_utf8` (p…
[3.14] gh-144872: fix heap buffer overflow `_PyTokenizer_ensure_utf8` (…
pythongh-144872: fix heap buffer overflow `_PyTokenizer_ensure_utf8` (p…
[3.13] pythongh-144872: fix heap buffer overflow `_PyTokenizer_ensure…
[3.13] gh-144872: fix heap buffer overflow `_PyTokenizer_ensure_utf8` (…
pablogsal commentedon Mar 2, 2026
Thanks a lot for the report @AdamKorcz !
pythongh-144872: fix heap buffer overflow `_PyTokenizer_ensure_utf8` (p…