Skip to content

The lines in tokens from tokenize.generate_tokens incorrectly indicate multiple lines. #104972

Closed
Closed
The lines in tokens from tokenize.generate_tokens incorrectly indicate multiple lines.#104972
Labels
type-bugAn unexpected behavior, bug, or error

Description

@nedbat
nedbat
opened on May 26, 2023

The line attribute in tokens returned by tokenize.generate_tokens incorrectly indicate multiple lines. The tokens should have an invariant that using the .start and .end attributes to index into the .line attribute will produce the .string attribute.

tokbug.py:

import io
import sys
import tokenize

SOURCE = r"""
a + \
b
"""

print(sys.version)
readline = io.StringIO(SOURCE).readline
for tok in tokenize.generate_tokens(readline):
    correct = (tok.string) == (tok.line[tok.start[1]: tok.end[1]])
    print(tok, "" if correct else "<*****!!!")

Run with 3.12.0a7:

% /usr/local/pyenv/pyenv/versions/3.12.0a7/bin/python tokbug.py
3.12.0a7 (main, Apr  5 2023, 05:51:58) [Clang 14.0.3 (clang-1403.0.22.14.1)]
TokenInfo(type=62 (NL), string='\n', start=(1, 0), end=(1, 1), line='\n')
TokenInfo(type=1 (NAME), string='a', start=(2, 0), end=(2, 1), line='a + \\\n')
TokenInfo(type=54 (OP), string='+', start=(2, 2), end=(2, 3), line='a + \\\n')
TokenInfo(type=1 (NAME), string='b', start=(3, 0), end=(3, 1), line='b\n')
TokenInfo(type=4 (NEWLINE), string='\n', start=(3, 1), end=(3, 2), line='b\n')
TokenInfo(type=0 (ENDMARKER), string='', start=(4, 0), end=(4, 0), line='')

Run with 3.12.0b1:

% /usr/local/pyenv/pyenv/versions/3.12.0b1/bin/python tokbug.py
3.12.0b1 (main, May 23 2023, 16:19:59) [Clang 14.0.3 (clang-1403.0.22.14.1)]
TokenInfo(type=65 (NL), string='\n', start=(1, 0), end=(1, 1), line='\n')
TokenInfo(type=1 (NAME), string='a', start=(2, 0), end=(2, 1), line='a + \\\n')
TokenInfo(type=55 (OP), string='+', start=(2, 2), end=(2, 3), line='a + \\\n')
TokenInfo(type=1 (NAME), string='b', start=(3, 0), end=(3, 1), line='a + \\\nb\n') <*****!!!
TokenInfo(type=4 (NEWLINE), string='\n', start=(3, 1), end=(3, 2), line='a + \\\nb\n') <*****!!!
TokenInfo(type=0 (ENDMARKER), string='', start=(4, 0), end=(4, 0), line='')

Related to #104825? cc @pablogsal

Linked PRs

Activity

  1. nedbat
    added
    type-bugAn unexpected behavior, bug, or error
    on May 26, 2023

  2. nedbat commented on May 26, 2023

    @nedbat
    nedbat
    on May 26, 2023
    MemberAuthor

    The tip of 3.12 shows one more style of change, due to omitting the newlines from .line:

    3.12.0b1+ (heads/3.12:6324458bef, May 26 2023, 06:25:21) [Clang 14.0.3 (clang-1403.0.22.14.1)]
TokenInfo(type=65 (NL), string='\n', start=(1, 0), end=(1, 1), line='') <*****!!!
TokenInfo(type=1 (NAME), string='a', start=(2, 0), end=(2, 1), line='a + \\')
TokenInfo(type=55 (OP), string='+', start=(2, 2), end=(2, 3), line='a + \\')
TokenInfo(type=1 (NAME), string='b', start=(3, 0), end=(3, 1), line='a + \\\nb') <*****!!!
TokenInfo(type=4 (NEWLINE), string='\n', start=(3, 1), end=(3, 2), line='a + \\\nb') <*****!!!
TokenInfo(type=0 (ENDMARKER), string='', start=(4, 0), end=(4, 0), line='')

  3. pablogsal commented on May 26, 2023

    @pablogsal
    pablogsal
    on May 26, 2023
    Member

    CC: @mgmacias95

  4. pablogsal
    added 5 commits that reference this issue on May 26, 2023

    pythongh-104972: Ensure that line attributes in tokens in the tokeniz…

    c381380

    pythongh-104972: Ensure that line attributes in tokens in the tokeniz…

    0b23e0f

    pythongh-104972: Ensure that line attributes in tokens in the tokeniz…

    d477678

    pythongh-104972: Ensure that line attributes in tokens in the tokeniz…

    3d2f429

    gh-104972: Ensure that line attributes in tokens in the tokenize modu…

    3fdb55c
  5. miss-islington
    added a commit that references this issue on May 26, 2023

    pythongh-104972: Ensure that line attributes in tokens in the tokeniz…

    25cf9f2

  6. pablogsal commented on May 26, 2023

    @pablogsal
    pablogsal
    on May 26, 2023
    Member

    @nedbat can you check with main now?

  7. pablogsal
    closed this as completedon May 26, 2023
  8. terryjreedy
    added a commit that references this issue on May 26, 2023

    [3.12] gh-104972: Ensure that line attributes in tokens in the tokeni…

    8ca2957

  9. terryjreedy commented on May 26, 2023

    @terryjreedy
    terryjreedy
    on May 26, 2023
    Member

    If you set automerge on the backport, the usual test-hypothesis failure disabled it. I merged it.

  10. nedbat commented on May 26, 2023

    @nedbat
    nedbat
    on May 26, 2023
    MemberAuthor

    @pablogsal Gorgeous! Thanks for the quick turnaround.

  11. pablogsal commented on May 26, 2023

    @pablogsal
    pablogsal
    on May 26, 2023
    Member

    If you set automerge on the backport, the usual test-hypothesis failure disabled it. I merged it.

    Thanks a lot @terryjreedy !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions