Repository navigation
Error in AST recursion depth tracking change of gh-95185 #106905
Description
Bug report
The change in 0047447 seems to miss the recusrion depth adjustment in case of an error. As an example for some of the generated code:
PyObject*
ast2obj_mod(struct ast_state *state, void* _o)
{
mod_ty o = (mod_ty)_o;
PyObject *result = NULL, *value = NULL;
PyTypeObject *tp;
if (!o) {
Py_RETURN_NONE;
}
if (++state->recursion_depth > state->recursion_limit) {
PyErr_SetString(PyExc_RecursionError,
"maximum recursion depth exceeded during ast construction");
return 0;
}
switch (o->kind) {
case Module_kind:
tp = (PyTypeObject *)state->Module_type;
result = PyType_GenericNew(tp, NULL, NULL);
if (!result) goto failed;
value = ast2obj_list(state, (asdl_seq*)o->v.Module.body, ast2obj_stmt);
if (!value) goto failed;
if (PyObject_SetAttr(result, state->body, value) == -1)
goto failed;
...
}
state->recursion_depth--;
return result;
failed:
Py_XDECREF(value);
Py_XDECREF(result);
return NULL;
}
Note that the failed code path is missing the state->recursion_depth--; statement.
I found this as I'm trying to track down where spurious SystemError: AST constructor recursion depth mismatch errors in Python 3.11 are coming from. E.g.
File "/env/lib/python3.11/site-packages/bloscpack/numpy_io.py", line 358, in unpack_ndarray_from_bytes
return unpack_ndarray(source)
^^^^^^^^^^^^^^^^^
File "/env/lib/python3.11/site-packages/bloscpack/numpy_io.py", line 305, in unpack_ndarray
sink = PlainNumpySink(source.metadata)
^^^^^^^^^^^^^^^^^
File "/env/lib/python3.11/site-packages/bloscpack/numpy_io.py", line 136, in __init__
dtype_ = ast.literal_eval(metadata['dtype'])
^^^^^^^^^^^^^^^^^
File "/env/lib/python3.11/ast.py", line 64, in literal_eval
node_or_string = parse(node_or_string.lstrip(" \t"), mode='eval')
^^^^^^^^^^^^^^^^^
File "/env/lib/python3.11/ast.py", line 50, in parse
return compile(source, filename, mode, flags,
^^^^^^^^^^^^^^^^^
SystemError: AST constructor recursion depth mismatch (before=48, after=47)
Your environment
Reproduced in Python 3.11 but the code in main looks the same.
Linked PRs
- gh-106905: avoid incorrect SystemError about recursion depth mismatch #106906
- [3.12] gh-106905: avoid incorrect SystemError about recursion depth mismatch (GH-106906) #112032
- [3.11] gh-106905: Avoid incorrect SystemError about recursion depth mismatch (#106906) #112849
- gh-106905: Use separate structs to track recursion depth in each PyAST_mod2obj call. #113035
- [3.12] gh-106905: Use separate structs to track recursion depth in each PyAST_mod2obj call. (GH-113035) #113472
- [3.11] gh-106905: Use separate structs to track recursion depth in each PyAST_mod2obj call. (GH-113035) (GH-113472) #113476
Activity
romuald commentedon Jul 24, 2023
I can confirm this issue.
It causes a bunch of INTERNALERROR in some specific case with py.test when it tries to report errors traces
Reverting 0047447 locally solves the problem (actual error is properly reported)
(edit: typo)
MaZderMind commentedon Sep 22, 2023
Might this be the root-cause for sqlalchemy/mako#378, pytest-dev/pytest#10874, home-assistant/core#95364, bieniu/ha-shellies-discovery#448 and other projects reporting
SystemError: AST constructor recursion depth mismatchsince python 3.11? If so, while the issue might not happen too often, it still hits lots and projects and people.malthe commentedon Nov 13, 2023
@serhiy-storchaka do you perhaps have some information to help this issue along?
gh-106905: avoid incorrect SystemError about recursion depth mismatch (…
pythongh-106905: avoid incorrect SystemError about recursion depth mi…
yilei commentedon Dec 7, 2023
Thanks @mmohrhard for the fix and @pablogsal for reviewing and merging the fix.
Any thoughts on what conditions this bug will likely happen? We are also seeing the "recursion depth mismatch" error but only in production. So we are trying to construct a reproducible example so we could be more confident that bringing #106906 would fix our issues in production.
yilei commentedon Dec 7, 2023
[3.11] pythongh-106905: Avoid incorrect SystemError about recursion d…
gpshead commentedon Dec 7, 2023
Agreed that a regression test for this would be good to have.
[-]Error in recursion depth tracking change of gh-95185[/-][+]Error in Python-ast recursion depth tracking change of gh-95185[/+][-]Error in Python-ast recursion depth tracking change of gh-95185[/-][+]Error in AST recursion depth tracking change of gh-95185[/+]8 remaining items
gpshead commentedon Dec 13, 2023
I've confirmed that the draft PR #113035 backported to 3.11 fixes the problem on a machine that immediately reproduces the problem with the above test code otherwise. 3.12 "got lucky" with that GC change and likely avoided this bug being possible (or at least far more rare if there are any other paths that could similarly enable it).
yilei is also testing the PR on our 3.11 runtime at work to ensure it fixes our users problems.
yilei commentedon Dec 15, 2023
After patching our 3.11 runtime two days ago at work, I can confirm #113035 fixed the issue for us.
gh-106905: Use separate structs to track recursion depth in each PyAS…
[3.12] pythongh-106905: Use separate structs to track recursion depth…
[3.12] gh-106905: Use separate structs to track recursion depth in ea…
[3.11] [3.12] pythongh-106905: Use separate structs to track recursio…
[3.11] gh-106905: Use separate structs to track recursion depth in ea…
pythongh-106905: Use separate structs to track recursion depth in eac…
pythongh-106905: Use separate structs to track recursion depth in eac…
pythongh-106905: avoid incorrect SystemError about recursion depth mi…
pythongh-106905: Use separate structs to track recursion depth in eac…
gh-106905: fix AST parser race condition
pythongh-106905: avoid incorrect SystemError about recursion depth mi…
pythongh-106905: Use separate structs to track recursion depth in eac…