Repository navigation
\rs introduced after codec decoding cause SystemError #145234
Description
Bug report
Bug description:
The testcase is:
#coding=U7+AA0''resulting in:
$ ./python -c "import py_compile; py_compile.compile('testcase')"
Sorry: SystemError: Parser/string_parser.c:286: bad argument to internal function
I gather this is because +AA0 decodes to \r in UTF-7, however, _PyTokenizer_translate_newlines is called before the codec decoding so it slips by and blows up (at least it apologizes😆). I think it could be fixed by calling _PyTokenizer_translate_newlines again after decoding if a \r is introduced, WDYT?
CPython versions tested on:
CPython main branch
Operating systems tested on:
No response
Activity
pythongh-145234: Fix SystemError in parser when \r is introduced afte…
pythongh-145234: Normalize decoded CR in string tokenizer
pythongh-145234: Add patch attribution in NEWS entry
pythongh-145234: keep tokenizer source pointer after UTF-8 translation
gh-145234: Normalize decoded CR in string tokenizer (#145281)
pythongh-145234: Normalize decoded CR in string tokenizer (pythonGH-1…
[3.14] gh-145234: Normalize decoded CR in string tokenizer (GH-145281) (
[3.13] pythongh-145234: Normalize decoded CR in string tokenizer (pyt…
[3.13] gh-145234: Normalize decoded CR in string tokenizer (GH-145281) (
pablogsal commentedon Feb 28, 2026
Thanks for the report @StanFromIreland !
pythongh-145234: Normalize decoded CR in string tokenizer (python#145281
pythongh-145234: Normalize decoded CR in string tokenizer (python#145281