Repository navigation
Base the tokenizer API on source offsets #153569
Description
The tokenizer currently stores many source positions as pointers into buffers that can move. Reallocating the input means rebasing all these pointers, and missing one is very easy, especially with incremental input and f-strings.
I think tokenizer positions should be offsets into the decoded source instead:
typedef Py_ssize_t TokenizerOffset;
typedef struct {
TokenizerOffset start;
TokenizerOffset end;
} TokenizerSpan;The main ideas would be:
- One source object owns the decoded text.
- The cursor only stores its current offset and line boundaries.
- Tokens, errors and f-string state use offset spans instead of pointers.
- pegen and
_tokenizeask the source for a view or copy of a span. - Sequential tokenization keeps the current line in the cursor, while uncommon line lookups can use a small sparse index.
- Normal parsing can keep one contiguous buffer, while
tokenize(readline)can eventually use reclaimable chunks.
For incremental tokenization, new decoded input would be appended only when the cursor needs more data. Existing offsets would remain valid even if the underlying storage moves. Once tokenize has returned copied token and line strings, old input could be discarded when no active token, cursor, f-string frame or error still refers to it.
This is very nice because it separates source storage from tokenizer state, removes pointer rebasing, and means consumers no longer need to access tokenizer internals.
Linked PRs
- gh-153569: Split tokenizer lexer into focused files #153585
- gh-153569: Add tokenizer source and cursor primitives #153587
- gh-153569: Replace tokenizer input readers #156472
- gh-153569: return token spans and unify decoded source storage #156482
- gh-153569: centralize formatted-string state and source spans #156484
- gh-153569: hide tokenizer state behind token and source operations #156654
- gh-153569: consolidate tokenizer input, layout, and diagnostics #157055
- gh-153569: Simplify tokenizer state #158124
- gh-153569: Simplify tokenizer source views and reader ownership #158849
Activity
Arbaaz123676 commentedon Jul 11, 2026
Hi @pablogsal, I'd like to work on this issue. Can you please assign it to me?
pablogsal commentedon Jul 11, 2026
Thanks for the help @Arbaaz123676! Unfortunately this is something I am working on right now as a series of commits and is very delicate work. If you want to help reviewing the PRs that would be good!
gh-153569: Split tokenizer lexer into focused files (#153585)
pythongh-153569: Add tokenizer source and cursor primitives
pythongh-153569: Add tokenizer source and cursor primitives
fixup! pythongh-153569: Add tokenizer source and cursor primitives
fixup! pythongh-153569: Add tokenizer source and cursor primitives
fixup! pythongh-153569: Add tokenizer source and cursor primitives
pythongh-153569: Add tokenizer source and cursor primitives
fixup! pythongh-153569: Add tokenizer source and cursor primitives
fixup! pythongh-153569: Add tokenizer source and cursor primitives
gh-153569: Add tokenizer source and cursor primitives (#153587)
pythongh-153569: preserve unterminated interactive input
221 remaining items
gh-153569: Simplify tokenizer state (#158124)
pythongh-153569: Consolidate tokenizer source access and token views
pythongh-153569: Move tokenizer source access into the source API
pythongh-153569: Use logical spans in tokenizer views and column calc…
pythongh-153569: Move tokenizer input state and encoding ownership in…
pythongh-153569: Keep tokenizer source scans and copies on offsets an…
pythongh-153569: Move tokenizer source access into the source API
pythongh-153569: Use logical spans in tokenizer views and column calc…
pythongh-153569: Move tokenizer input state and encoding ownership in…
pythongh-153569: Keep tokenizer source scans and copies on offsets an…
pythongh-153569: Remove redundant tokenizer views and simplify reader…
pythongh-153569: Remove redundant tokenizer views and simplify reader…
pythongh-153569: Reuse the incremental decoder factory helper
pythongh-153569: Export incremental decoder helper for parser generat…