Repository navigation
untokenize() round-trip fails for t-strings (with only type + string) #134582
Description
Bug report
Bug description:
Found when investigating test.test_tokenize.TestRoundtrip.test_random_files failing on #134577
def test(code):
tokens = list(tokenize.tokenize(iter([code]).__next__))
from5 = tokenize.untokenize(tokens)
print("from5 ", from5, eval(from5))
tokens2 = [tok[:2] for tok in tokens]
from2 = tokenize.untokenize(tokens2)
print("from2 ", from2, eval(from2))>>> test(b't"{ {} }"')
from5 b't"{ {} }"' Template(strings=('', ''), interpolations=(Interpolation({}, ' {}', None, ''),))
from2 b't"{{}}"' Template(strings=('{}',), interpolations=())
>>> test(b'f"{ {} }"')
from5 b'f"{ {} }"' {}
from2 b'f"{ {} }"' {}From what I understand, untokenize should round-trip correctly even with only the type and string of tokens.
CPython versions tested on:
CPython main branch
Operating systems tested on:
macOS
Activity
gh-134582: Fix t-strings untokenize() roundtrip removing space betwee…
pythongh-134582: Fix t-strings untokenize() roundtrip removing space …
[3.14] gh-134582: Fix t-strings untokenize() roundtrip removing space…
pythongh-134582: Fix t-strings untokenize() roundtrip removing space …
pythongh-134582: Fix t-strings untokenize() roundtrip removing space …
pythongh-134582: Fix t-strings untokenize() roundtrip removing space …