From 7e7b3b7fe1826a4b41e41624bc34ed98fd02ef2c Mon Sep 17 00:00:00 2001 From: johnslavik Date: Wed, 21 Oct 2026 06:50:39 -0700 Subject: [PATCH 1/5] Document tokenize.TokenInfo and its attributes --- Doc/library/tokenize.rst | 57 ++++++++++++++++++++++++++-------------- 1 file changed, 38 insertions(+), 19 deletions(-) diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index 2eea51734fde03c..af69a63236fc984 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -16,8 +16,8 @@ colorizers for on-screen displays. To simplify token stream handling, all :ref:`operator ` and :ref:`delimiter ` tokens and :data:`Ellipsis` are returned using the generic :data:`~token.OP` token type. The exact -type can be determined by checking the ``exact_type`` property on the -:term:`named tuple` returned from :func:`tokenize.tokenize`. +type can be determined by checking the :attr:`TokenInfo.exact_type` property on +the :class:`TokenInfo` returned from :func:`tokenize.tokenize`. .. warning:: @@ -40,26 +40,11 @@ The primary entry point is a :term:`generator`: :meth:`io.IOBase.readline` method of file objects. Each call to the function should return one line of input as bytes. - The generator produces 5-tuples with these members: the token type; the - token string; a 2-tuple ``(srow, scol)`` of ints specifying the row and - column where the token begins in the source; a 2-tuple ``(erow, ecol)`` of - ints specifying the row and column where the token ends in the source; and - the line on which the token was found. The line passed (the last tuple item) - is the *physical* line. The 5 tuple is returned as a :term:`named tuple` - with the field names: - ``type string start end line``. - - The returned :term:`named tuple` has an additional property named - ``exact_type`` that contains the exact operator type for - :data:`~token.OP` tokens. For all other token types ``exact_type`` - equals the named tuple ``type`` field. + The generator produces :class:`TokenInfo` objects. .. versionchanged:: 3.1 Added support for named tuples. - .. versionchanged:: 3.3 - Added support for ``exact_type``. - :func:`.tokenize` determines the source encoding of the file by looking for a UTF-8 BOM or encoding cookie, according to :pep:`263`. @@ -71,9 +56,43 @@ The primary entry point is a :term:`generator`: a single line of input. However, :func:`generate_tokens` expects *readline* to return a str object rather than bytes. - The result is an iterator yielding named tuples, exactly like + The result is an iterator yielding :class:`TokenInfo` objects, exactly like :func:`.tokenize`. It does not yield an :data:`~token.ENCODING` token. +.. class:: TokenInfo(type, string, start, end, line) + + A :term:`named tuple` with five fields describing a token: + + .. attribute:: type + + The token type. + + .. attribute:: string + + The token string. + + .. attribute:: start + + A 2-tuple ``(srow, scol)`` of ints specifying the row and column where + the token begins in the source. + + .. attribute:: end + + A 2-tuple ``(erow, ecol)`` of ints specifying the row and column where + the token ends in the source. + + .. attribute:: line + + The physical line or lines on which the token was found. + + .. attribute:: exact_type + + The exact operator type for :data:`~token.OP` tokens. For all other + token types, this equals :attr:`type`. + + .. versionadded:: 3.3 + + All constants from the :mod:`token` module are also exported from :mod:`!tokenize`. From d12f71442b6f53e26dd56cb2702bf847a0897580 Mon Sep 17 00:00:00 2001 From: johnslavik Date: Wed, 21 Oct 2026 07:14:44 -0700 Subject: [PATCH 2/5] Clarify TokenInfo field semantics and positions --- Doc/library/tokenize.rst | 53 ++++++++++++++++++++++++++++++++-------- 1 file changed, 43 insertions(+), 10 deletions(-) diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index af69a63236fc984..82c496926b9d60d 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -61,34 +61,67 @@ The primary entry point is a :term:`generator`: .. class:: TokenInfo(type, string, start, end, line) - A :term:`named tuple` with five fields describing a token: + A :term:`named tuple` describing a token. It has five fields and an + additional :attr:`exact_type` property. .. attribute:: type - The token type. + An integer identifying the token type, as defined by the constants in + :mod:`token`, such as :data:`~token.NAME` and :data:`~token.STRING`. + Operators and delimiters have type :data:`~token.OP`; use + :attr:`exact_type` for their specific token type. .. attribute:: string - The token string. + The token text as a :class:`str`, usually as it appears in the source. + For :data:`~token.STRING` tokens, this includes the quotes, any prefix, + and escape sequences, without evaluating them. + + For an :data:`~token.ENCODING` token, this is the encoding name. + Tokens inserted without corresponding source text can have an empty + string. In :data:`~token.FSTRING_MIDDLE` and + :data:`~token.TSTRING_MIDDLE` tokens, doubled braces are replaced by + single braces. .. attribute:: start - A 2-tuple ``(srow, scol)`` of ints specifying the row and column where - the token begins in the source. + A 2-tuple ``(srow, scol)`` of ints specifying the inclusive start + position of the token in the source. Rows are numbered from 1 and + columns from 0. Columns count Unicode code points; a tab counts as + one column, without expanding it to a tab stop. + + For an :data:`~token.ENCODING` token, this is ``(0, 0)``. .. attribute:: end - A 2-tuple ``(erow, ecol)`` of ints specifying the row and column where - the token ends in the source. + A 2-tuple ``(erow, ecol)`` of ints specifying the exclusive end + position of the token, using the same coordinate conventions as + :attr:`start`. For a token spanning multiple lines, *erow* is the + final row and *ecol* is the column on that row. + + For example, the name ``spam`` at the beginning of the first line + has :attr:`start` equal to ``(1, 0)`` and :attr:`end` equal to ``(1, 4)``. + + For an :data:`~token.ENCODING` token, this is ``(0, 0)``. + A :data:`~token.NEWLINE` or :data:`~token.NL` token inserted at the end + of input without a final newline can have an empty :attr:`string` + and an end column one beyond the source text. .. attribute:: line - The physical line or lines on which the token was found. + A :class:`str` containing the full physical source line on which + the token was found, including surrounding text and the line terminator + if present. For a token spanning multiple lines, this can contain + several physical lines. It is empty for :data:`~token.ENCODING` and + :data:`~token.ENDMARKER` tokens. .. attribute:: exact_type - The exact operator type for :data:`~token.OP` tokens. For all other - token types, this equals :attr:`type`. + An integer identifying the specific operator type for + :data:`~token.OP` tokens, as defined by the constants in :mod:`token`. + For all other token types, this equals :attr:`type`. + For example, a ``+`` token has :attr:`type` equal to :data:`~token.OP` + and :attr:`exact_type` equal to :data:`~token.PLUS`. .. versionadded:: 3.3 From 94140f176fe35a4bc4594beaebd08f8c4d8cc6dc Mon Sep 17 00:00:00 2001 From: johnslavik Date: Wed, 21 Oct 2026 10:47:47 -0700 Subject: [PATCH 3/5] Address review --- Doc/library/tokenize.rst | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index 82c496926b9d60d..0cb1bf479bad1b7 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -3,7 +3,7 @@ .. module:: tokenize :synopsis: Lexical scanner for Python source code. - +` **Source code:** :source:`Lib/tokenize.py` -------------- @@ -43,7 +43,7 @@ The primary entry point is a :term:`generator`: The generator produces :class:`TokenInfo` objects. .. versionchanged:: 3.1 - Added support for named tuples. + Changed to yield :class:`TokenInfo` objects instead of plain tuples. :func:`.tokenize` determines the source encoding of the file by looking for a UTF-8 BOM or encoding cookie, according to :pep:`263`. @@ -73,7 +73,7 @@ The primary entry point is a :term:`generator`: .. attribute:: string - The token text as a :class:`str`, usually as it appears in the source. + The token text as a :class:`str`. For :data:`~token.STRING` tokens, this includes the quotes, any prefix, and escape sequences, without evaluating them. @@ -95,24 +95,24 @@ The primary entry point is a :term:`generator`: .. attribute:: end A 2-tuple ``(erow, ecol)`` of ints specifying the exclusive end - position of the token, using the same coordinate conventions as - :attr:`start`. For a token spanning multiple lines, *erow* is the - final row and *ecol* is the column on that row. + position of the token. Rows are numbered from 1 and columns from 0. + For a token spanning multiple lines, *erow* is the final row and + *ecol* is the column on that row. For example, the name ``spam`` at the beginning of the first line has :attr:`start` equal to ``(1, 0)`` and :attr:`end` equal to ``(1, 4)``. For an :data:`~token.ENCODING` token, this is ``(0, 0)``. A :data:`~token.NEWLINE` or :data:`~token.NL` token inserted at the end - of input without a final newline can have an empty :attr:`string` - and an end column one beyond the source text. + of input without a final newline can have an end column one beyond + the source text. .. attribute:: line - A :class:`str` containing the full physical source line on which + A :class:`str` containing the full source line on which the token was found, including surrounding text and the line terminator if present. For a token spanning multiple lines, this can contain - several physical lines. It is empty for :data:`~token.ENCODING` and + several source lines. It is empty for :data:`~token.ENCODING` and :data:`~token.ENDMARKER` tokens. .. attribute:: exact_type @@ -125,6 +125,8 @@ The primary entry point is a :term:`generator`: .. versionadded:: 3.3 + .. versionadded:: 3.1 + All constants from the :mod:`token` module are also exported from :mod:`!tokenize`. From 35e19fc0c30da95fb3fc006613e435d2bd52f1f3 Mon Sep 17 00:00:00 2001 From: johnslavik Date: Wed, 21 Oct 2026 10:55:52 -0700 Subject: [PATCH 4/5] Simplify the exact token type description --- Doc/library/tokenize.rst | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index 0cb1bf479bad1b7..a3bada030d8327a 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -3,7 +3,7 @@ .. module:: tokenize :synopsis: Lexical scanner for Python source code. -` + **Source code:** :source:`Lib/tokenize.py` -------------- @@ -15,9 +15,8 @@ colorizers for on-screen displays. To simplify token stream handling, all :ref:`operator ` and :ref:`delimiter ` tokens and :data:`Ellipsis` are returned using -the generic :data:`~token.OP` token type. The exact -type can be determined by checking the :attr:`TokenInfo.exact_type` property on -the :class:`TokenInfo` returned from :func:`tokenize.tokenize`. +the generic :data:`~token.OP` token type. +The :attr:`TokenInfo.exact_type` property gives the exact token type. .. warning:: From 7ceb205a33e07da7c0a91d37f5a9bb130e90666c Mon Sep 17 00:00:00 2001 From: johnslavik Date: Wed, 21 Oct 2026 13:08:30 -0700 Subject: [PATCH 5/5] Move TokenInfo version note below its description --- Doc/library/tokenize.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index a3bada030d8327a..2d2cb28640d1976 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -63,6 +63,8 @@ The primary entry point is a :term:`generator`: A :term:`named tuple` describing a token. It has five fields and an additional :attr:`exact_type` property. + .. versionadded:: 3.1 + .. attribute:: type An integer identifying the token type, as defined by the constants in @@ -124,8 +126,6 @@ The primary entry point is a :term:`generator`: .. versionadded:: 3.3 - .. versionadded:: 3.1 - All constants from the :mod:`token` module are also exported from :mod:`!tokenize`.