Skip to content

In IDLE colorizer, replace re with tokenizer #140347

Open
Open
In IDLE colorizer, replace re with tokenizer#140347
Labels
topic-IDLEtype-featureA feature request or enhancement

Description

@terryjreedy
terryjreedy
opened on Oct 20, 2025

The current IDLE colorizer was designed when the python grammar was restricted to a context-free subtype. Parsing with RE's was sufficient or closely so to pick out substrings to be color tagged. The current grammar's sometime context-dependence make this more difficult or even impossible.

A few versions ago, the new C tokenizer used to compile Python code was exposed as a function in the tokenizer module. It replaced a python-coded tokenizer that did not always match the old C tokenizer and that must have been much slower. I assume that is was not seen as suitable for IDLE colorizer. However, the current exposed C tokenizer is being used to power the new PyREPL colorizer. The initial (draft) patch copies portions of that colorizer.

My main concerns are compatibility and speed. Some initial questions:

  1. The syntax categories in the REPL colorizer are not the same as IDLE's (listed on the Highlights settings page). What are they?
  2. IDLE colorizes 10000 lines editor contents as well as single interactive lines. Will the PR colorizer do the same, with similar speed?
  3. Is REPL colorizer stable? Are there open bug reports? (Should be labelled topic-repl.) The code is private and undocumented, I presume intentionally, and I do not expect that issues only relevant to IDLE would be welcome.
  4. In the IDLE Issues project, colorizer issues are in Highlights section. Does this fix any, or appear to make fixes easier?

Linked PRs

Activity

  1. terryjreedy
    added this to IDLE Issueson Oct 20, 2025
  2. terryjreedy
    added
    stdlibStandard Library Python modules in the Lib/ directory
    topic-IDLE
    on Oct 20, 2025
  3. StanFromIreland
    added
    type-featureA feature request or enhancement
    on Oct 20, 2025

  4. StanFromIreland commented on Oct 20, 2025

    @StanFromIreland
    StanFromIreland
    on Oct 20, 2025
    Member

    Starting from the bottom.

    In the IDLE Issues project, colorizer issues are in Highlights section. Does this fix any, or appear to make fixes easier?

    The remainder of the issues in the project are not directly related to the actual colorizer logic, and there should be no change in how they would be implemented.

    IDLE colorizes 10000 lines editor contents as well as single interactive lines. Will the PR colorizer do the same, with similar speed?

    The PR should be faster than the regexes, although I have not yet done proper benchmarking.

    The syntax categories in the REPL colorizer are not the same as IDLE's (listed on the Highlights settings page). What are they?

    They can be modified to match, and in the PR it does.

    The last one requires more research, for which I do not have time right now.

  5. StanFromIreland commented on Oct 20, 2025

    @StanFromIreland
    StanFromIreland
    on Oct 20, 2025
    Member

    My main issue with the implementation is that the old colorizer would continue on errornous syntax, whereas tokenizing may fail. The repl colorizer has a fallback, but it does not always succeed. Trying to complicate the logic more is not worth it IMO, I have meddled with it a little and introduced a second line-by-line attempt, though that also causes problems (I will try fix these, the main issue with line-by-line is context, e.g. are we in a triple quoted string, is lost).
    I think we would need to accept that on errornous syntax, there sometimes may be no highlighting at all.

  6. TheLizzard commented on Oct 22, 2025

    @TheLizzard
    TheLizzard
    on Oct 22, 2025

    Mostly but it fails on cases like this:

    Image

    That is the only bug left with the regex colorizer and it cannot be fixed without very complicated python logic or recursive regexes (which python's re doesn't provide as far as I know).

    The PR should be faster than the regexes

    It will probably be much faster but unfortunately it won't really speed up IDLE's colorization. IDLE spends ~75% of the time adding the tags to the tkinter.Text widget after they have been computed. tkinter is a major bottleneck but can be sped up

  7. StanFromIreland
    marked this as blocked on Dec 14, 2025
  8. terryjreedy
    removed
    stdlibStandard Library Python modules in the Lib/ directory
    on Jan 15, 2026

  9. StanFromIreland commented on Jun 14, 2026

    @StanFromIreland
    StanFromIreland
    on Jun 14, 2026
    Member

    Another case, #151471, would (I presume, since it works correctly in the repl) have been avoided if using a tokenizer.

  10. serhiy-storchaka commented on Jul 1, 2026

    @serhiy-storchaka
    serhiy-storchaka
    on Jul 1, 2026
    Member

    I built a Stage-1 spike: swap IDLE's ColorDelegator._add_tags_in_section to get spans from _pyrepl.utils.gen_colors, keeping IDLE's incremental engine (SYNC-bounded recolorize_main). Findings:

    The glue is small. Only that one method changes. IDLE's SYNC point (a newline outside strings) is exactly gen_colors's "valid start of a code block" requirement, so the incremental machinery stays as-is. Categories are compatible — pyrepl is a superset (it colors builtins too); only number/op are extra. On valid code the spike is strictly more correct: match/case/type soft keywords and f-strings the regex can't do.

    The real blocker is error recovery. gen_colors only recovers from unterminated strings, so any other TokenError drops everything after it:

    from _pyrepl.utils import gen_colors
src = "ur'x'\nmatch point:\n    case 1: pass\n"   # ur'x' is an invalid prefix
print(list(gen_colors(src)))   # -> []  — the valid match/case below is lost too

    IDLE amplifies this: recolorize_main accumulates large sections, so one bad token can blank a dozen lines, where the old regex tolerated it.

    Where the fix belongs: in the shared gen_colors — on TokenError, restart tokenizing at the next line (generalizing the string recovery) — so PyREPL and IDLE both benefit instead of IDLE reintroducing regex logic. Open UX question for IDLE: on an un-tokenizable line, prefer no highlight, a regex fallback, or skip-and-continue?

    Happy to turn the spike into a draft PR if that direction sounds right.

  11. terryjreedy commented on Jul 9, 2026

    @terryjreedy
    terryjreedy
    on Jul 9, 2026
    MemberAuthor

    What is the difference between 'no highlight' and 'skip-and-continue'?

    On entry of ur" or 'ru" into IDLE, the valid r" or u" is highlighted and the invalid prefix not, making the error immediately visible. In REPL, there is no highlight and therefore no contrast, though user could note the string highlight absence. Interactively, a newline will likely follow soon, generating a SyntaxError with message. IDLE also used the colorizer in the editor, and a compile may be a long time away.

    Unlike REPL, IDLE Shell/Editors have a status bar, where a message like 'line has error' could be displayed.

    How can tokenizing restart on the next line in the middle of a multiline string?

    It seems to me that colorizing needs a more an error-tolerant tokenizing than compiling does. Perhaps it should be 2-stage. The first stage detects code, string (or perhaps 1-line and n-line strings and f-string variants) , and comment blocks. IDLE could use a simplified RE for this. The second stage only applies to code blocks, and IDLE could use gen_colors for that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    topic-IDLEtype-featureA feature request or enhancement

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions