Repository navigation
Grammar differences between Doc grammar and Pegen grammar for Python #100292
Description
This is a question about the difference between the grammars at:
- https://docs.python.org/3/reference/grammar.html, and
- https://github.com/python/cpython/blob/9d2dcbbccdf4207249665b58c8bd28430c4f7afd/Grammar/python.gram
I am scraping and refactoring the Pegen grammar for Python at https://github.com/python/cpython/blob/9d2dcbbccdf4207249665b58c8bd28430c4f7afd/Grammar/python.gram mechanically. After removing the strings for lookahead, I performed a diff between the grammar in the Doc at https://docs.python.org/3/reference/grammar.html and the refactored grammar. It should be the same. But, it is not.
I noticed that some lookahead expressions are deleted for a rule in the Doc grammar, but others are not.
For example, in the Doc grammar, the simple_stmt rule does not have the &e lookahead expressions:
simple_stmt:
| assignment
| star_expressions
| return_stmt
| import_stmt
| raise_stmt
| 'pass'
| del_stmt
| yield_stmt
| assert_stmt
| 'break'
| 'continue'
| global_stmt
| nonlocal_stmt
Similarly, for the compound_stmt rule, the &-lookahead expressions have been deleted from the Doc grammar:
compound_stmt:
| function_def
| if_stmt
| class_def
| with_stmt
| for_stmt
| try_stmt
| while_stmt
| match_stmt
Of course, the simple_stmt rule in the .gram file contains the & lookahead expressions:
simple_stmt[stmt_ty] (memo):
| assignment
| e=star_expressions { _PyAST_Expr(e, EXTRA) }
| &'return' return_stmt
| &('import' | 'from') import_stmt
| &'raise' raise_stmt
| 'pass' { _PyAST_Pass(EXTRA) }
| &'del' del_stmt
| &'yield' yield_stmt
| &'assert' assert_stmt
| 'break' { _PyAST_Break(EXTRA) }
| 'continue' { _PyAST_Continue(EXTRA) }
| &'global' global_stmt
| &'nonlocal' nonlocal_stmt
So far, this all makes sense: remove the lookahead because it is not needed in the CFG.
But, if we now look further into the Doc grammar, we see that some rules still have the lookahead listed:
del_stmt:
| 'del' del_targets &(';' | NEWLINE)
...
slash_no_default:
| param_no_default+ '/' ','
| param_no_default+ '/' &')'
slash_with_default:
| param_no_default* param_with_default+ '/' ','
| param_no_default* param_with_default+ '/' &')'
Can someone clarify why some lookahead expressions are deleted and others are not in the Doc grammar?
Activity
puneeth072003 commentedon Dec 25, 2022
I'd like to work on this issue
StanFromIreland commentedon Aug 22, 2025
Closing as duplicate of the up-to-date issue where work is ongoing.
cc @encukou