Repository navigation
3.12: tokenize adds a newline when it is not there #105259
Description
Activity
@lysnikolaou can you take a look at this?
I think this is due to this line:
Line 1309 in e6373c0
in combination with this line:cpython/Python/Python-tokenize.c
Line 249 in e6373c0
But the real question is what is going to break if we cover the first line with
if (!tok->extra_tokens)and how we can fix it.~@lysnikolaou The problem is that we are always adding a new line at the end of the line buffer so we cannot distinguish if the last one is there for real or not (also, we always hardcode
\nforNEWLINEtokens which we should not do for the last one).The problem is that we are always adding a new line at the end of the line buffer so we cannot distinguish if the last one is there for real or not
Do we? I thought you had removed this in #104850, but I guess I misunderstood that PR. I'll have a look.
Do we? I thought you had removed this in #104850, but I guess I misunderstood that PR. I'll have a look.
That was "reverted" because apparently that was required to keep backward compatibility IIRC. But this is also a different kind of "new line": the tokenize module needs new lines at the end of every logical line to correctly emit the last tokens. Just comment out this part and see what fails (locations of last tokens are wrong as well as the some implicit dedents are missing):
Line 1309 in e6373c0
Notice that just removing that is not enough to fix this: we would also need to emit the last artificial newline if the input doesn't contain it.
Yeah, I can wait.
- added a commit that references this issue
on Jun 6, 2023 - moved this from Todo to Done in Release and Deferred blockers 🚫
on Jun 6, 2023
Bug report
tokenize adds a concrete newline character when it is not present. this breaks any sort of roundtrip source generation and pycodestyle's end-of-file checker
here's an example of a file containing a single byte (generated via
echo -n 1 > t.py)Your environment
cc @pablogsal
Linked PRs