Skip to content

3.11.0b1 regression: re.template removed without a deprecation period #92728

Closed
Closed
3.11.0b1 regression: re.template removed without a deprecation period#92728
Assignees
serhiy-storchaka
Labels
3.11only security fixes3.12only security fixesrelease-blockertopic-regextype-bugAn unexpected behavior, bug, or error

Description

@hroncok
hroncok
opened on May 12, 2022

Bug report

In Python 3.10, the re.template function exists and is not deprecated:

>>> import re
>>> re.template('', re.I)
re.compile('', re.TEMPLATE|re.IGNORECASE)

In Python 3.11.0b1 this function is gone:

>>> import re
>>> re.template('', re.I)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 're' has no attribute 'template'

This was removed in b09184bf05 without a depreciation period.
Despite being undocumented, projects actually use this function, see e.g. rpm-software-management/dnf#1827

Please restore this function and deprecate it in 3.11 and 3.12 if you want to remove it in 3.13. (That is how I understand PEP 387, section Making Incompatible Changes.)

Your environment

  • CPython versions tested on: 3.11.0b1
  • Operating system and architecture: All

Activity

  1. hroncok
    added
    type-bugAn unexpected behavior, bug, or error
    on May 12, 2022
  2. hroncok
    added a commit that references this issue on May 12, 2022

    Don't use undocumented re.template()

    ac91dc3

  3. hroncok commented on May 12, 2022

    @hroncok
    hroncok
    on May 12, 2022
    ContributorAuthor

    I intend to send a PR later today.

    However, I am not sure if I can revert the removal + deprecate after the feature freeze, or if I must only revert the removal in 3.11 and deprecate it in 3.12+. Will start with a PR to main anyway.

    EDIT: I won't send a PR before there is a consensus.

  4. AlexWaygood
    added
    topic-regex
    3.11only security fixes
    3.12only security fixes
    on May 12, 2022
  5. AlexWaygood
    assigned
    serhiy-storchaka
    on May 12, 2022

  6. vstinner commented on May 12, 2022

    @vstinner
    vstinner
    on May 12, 2022
    Member

    Removal discussed at #91367

  7. vstinner commented on May 12, 2022

    @vstinner
    vstinner
    on May 12, 2022
    Member

    The discussion started in this larger issue about re "reorganization": #91308

  8. vstinner commented on May 12, 2022

    @vstinner
    vstinner
    on May 12, 2022
    Member

    I don't understand the purpose of the re.template() function and the re.TEMPLATE flag. It doesn't seem to disable backtracking:

    >>> re.compile(r"((a)b\2)", re.IGNORECASE).match("aba").groups()
('aba', 'a')
>>> re.template(r"((a)b\2)", re.IGNORECASE).match("aba").groups()
('aba', 'a')

    The only thing which seems to be changed is that repetition is not supported by TEMPLATE:

    >>> re.template(r"a+")
re.error: internal: unsupported template operator MAX_REPEAT
>>> re.template(r"a*")
re.error: internal: unsupported template operator MAX_REPEAT

  9. vstinner commented on May 12, 2022

    @vstinner
    vstinner
    on May 12, 2022
    Member

    Code search for "TEMPLATE" in re/sre source code files of Python 3.10:

    Lib/re.py:    TEMPLATE = T = sre_compile.SRE_FLAG_TEMPLATE # disable backtracking
Lib/sre_compile.py:            if flags & SRE_FLAG_TEMPLATE:
Lib/sre_constants.py:SRE_FLAG_TEMPLATE = 1 # template mode (disable backtracking)
Lib/sre_constants.py:        f.write("#define SRE_FLAG_TEMPLATE %d\n" % SRE_FLAG_TEMPLATE)
Lib/sre_parse.py:    "t": SRE_FLAG_TEMPLATE,
Lib/sre_parse.py:GLOBAL_FLAGS = SRE_FLAG_DEBUG | SRE_FLAG_TEMPLATE
Modules/sre_constants.h:#define SRE_FLAG_TEMPLATE 1

  10. vstinner commented on May 12, 2022

    @vstinner
    vstinner
    on May 12, 2022
    Member

    cc @serhiy-storchaka

  11. hroncok commented on May 12, 2022

    @hroncok
    hroncok
    on May 12, 2022
    ContributorAuthor

    To be honest, I don't understand the purpose of the re.template() function either and I think that deprecating and removing it is a good thing.

  12. vstinner commented on May 12, 2022

    @vstinner
    vstinner
    on May 12, 2022
    Member

    If we decide to not revert the change, I suggest to document the removal in What's New in Python 3.11 and explain how to port impacted applications. For example, replace re.template() with re.compile().

  13. hroncok commented on May 12, 2022

    @hroncok
    hroncok
    on May 12, 2022
    ContributorAuthor

    If we decide to not revert the change, I suggest to document the removal in What's New in Python 3.11 and explain how to port impacted applications. For example, replace re.template() with re.compile().

    I suppose we should do that even if we "just" deprecate it.

  14. 9 remaining items

  15. hugovk commented on May 12, 2022

    @hugovk
    hugovk
    on May 12, 2022
    Member

    I don't find any "re\.template\( in the top PypI 5,000 packages.

    GitHub's new search tool finds only 19 Python files matching /\bre\.template\(/, many of which are forks or duplicates.

    1. https://github.com/elyscape/dnf/blob/5965627a81be59f6dbb9ff2362be1aba8f897237/dnf/cli/term.py#L284
    2. https://github.com/venaxyt/mysterium/blob/8cac9b1bbdb882873a0e260bfb8bfd273bcfcbc3/modules/re.py#L99
    3. https://github.com/containerbuildsystem/osbs-client/blob/f719759af18ef9f3bb45ee4411f80a9580723e31/make_mock_json.py#L142
    4. https://github.com/th3r00t/pyShelf/blob/aef30f3dcab4014d16a0f65388e92b6310f673a4/src/backend/lib/config.py#L61
    5. https://github.com/shoptime/trex/blob/75fea0e2f79a6be25917011456bbfef31aebdbaf/support/mongoengine.py#L20
    6. https://github.com/brinkqiang/dnf/blob/6517f47c94bdf2da2b1ce11605babee4d8cb8756/dnf/cli/term.py#L291
    7. https://github.com/tizenorg/platform.upstream.yum/blob/40014cdf8aae2f0a8c3eccd7f908e6ed1f5ddd65/output.py#L268
    8. https://github.com/lovaya/Ryven/blob/a11e361528d982a9dd3c489dd536f8b05ffd56e1/Ryven/packages/auto_generated/re/nodes.py#L355
    9. https://github.com/Distrotech/yum/blob/f4e54aeed297158c563828aa3ebb93d0c8ce7e38/output.py#L334
    10. https://github.com/rpm-software-management/yum/blob/4ed25525ee4781907bd204018c27f44948ed83fe/output.py#L334
    11. https://github.com/TheApacheCats/yum/blob/5c9c8db7b380457f71482a5ed28cd5fdb653a37b/output.py#L334
    12. https://github.com/timgates42/KomodoEdit/blob/980b5b00afda0144dd204c19de175b1a32ff12a8/src/codeintel/test2/test_python.py#L2220
    13. https://github.com/ryanuber/dnf/blob/fd8c108be591d52da6758214cb2b32a54d1afc31/dnf/cli/output.py#L334
    14. https://github.com/itmightbemo/KomodoEdit/blob/7f125d5140994e213ee892afac0210d7fa293699/src/codeintel/test2/test_python.py#L2182
    15. https://github.com/fatherlinux/containers-deep-dive/blob/76b3dbeb21c84d9536dc47660e958da520815b9e/containers201/rootfs/usr/share/yum-cli/output.py#L334
    16. https://github.com/cgwalters/dnf/blob/b81feda59ab79bc8a1ade4a54ee4b35789f2b75b/dnf/cli/term.py#L276
    17. https://github.com/SublimeCodeIntel/CodeIntel/blob/5a84f609c55972297119c2d7bc9d045222ee6d93/codeintel/test2/test_python.py#L2230
    18. https://github.com/alamatic/komodoedit/blob/19b97c35c59f1d232fa51bb0c96cb4cf5cc86248/src/codeintel/test2/test_python.py#L2082
    19. https://github.com/rmccue/codeintel/blob/5a6d47672c35d5fb1911312a8e3973284abd165d/test2/test_python.py#L2165

  16. DanielNoord commented on May 13, 2022

    @DanielNoord
    DanielNoord
    on May 13, 2022
    Contributor

    I have no opinion on this particular removal/deprecation, but just quickly want to point to PEP8 (https://peps.python.org/pep-0008/#public-and-internal-interfaces) and it's directive that:
    To better support introspection, modules should explicitly declare the names in their public API using the __all__ attribute. Setting __all__ to an empty list indicates that the module has no public API.. Previously, template was in __all__ which is also pre-faced with a # public symbols comment.

    Perhaps there are reasons why this does not hold true for CPython, but it would be good if that could be documented somewhere. As a package maintainer I have recently been struggling to correctly determine what is and isn't public in a codebase that does not currently document this. So guidelines such as the one in the PEP and general agreement on those guidelines would be helpful for me and the wider ecosystem I think.

  17. serhiy-storchaka
    added
    release-blocker
    on May 22, 2022

  18. hroncok commented on May 24, 2022

    @hroncok
    hroncok
    on May 24, 2022
    ContributorAuthor

    But if you think that it is worth, I am not opposing to reverting #32300 and adding a deprecation warning instead.

    Yes, I believe that would be in the spirit of the policy. I will try to draft a PR shortly and then I will leave it for the others to decide if it's worth it.

  19. serhiy-storchaka commented on May 24, 2022

    @serhiy-storchaka
    serhiy-storchaka
    on May 24, 2022
    Member

    How long the deprecation period should be? Would the only 3.11 be enough?

  20. hroncok commented on May 24, 2022

    @hroncok
    hroncok
    on May 24, 2022
    ContributorAuthor

    The policy speaks of 2 releases.

  21. hroncok
    added a commit that references this issue on May 24, 2022

    Revert "bpo-47211: Remove function re.template() and flag re.TEMPLATE (…

    2e8f218

  22. hroncok commented on May 24, 2022

    @hroncok
    hroncok
    on May 24, 2022
    ContributorAuthor

    I've opened #93161

  23. encukou commented on May 24, 2022

    @encukou
    encukou
    on May 24, 2022
    Member

    @rhettinger:

    Undocumented functions do not require a deprecation period.

    I disagree.
    Is this documented somewhere, or is it tribal knowledge (which should be documented, now that we have PEP-387 written)?

  24. serhiy-storchaka
    added a commit that references this issue on May 25, 2022

    gh-92728: Restore re.template, but deprecate it (GH-93161)

    16a7e4a
  25. miss-islington
    added 2 commits that reference this issue on May 25, 2022

    pythongh-92728: Restore re.template, but deprecate it (pythonGH-93161)

    f191b17

    gh-92728: Restore re.template, but deprecate it (GH-93161)

    74b205b

  26. serhiy-storchaka commented on May 25, 2022

    @serhiy-storchaka
    serhiy-storchaka
    on May 25, 2022
    Member

    Thank you @hugovk for your investigation and PR.

  27. serhiy-storchaka
    closed this as completedon May 25, 2022

  28. hugovk commented on May 25, 2022

    @hugovk
    hugovk
    on May 25, 2022
    Member

    Thank you @hroncok for the PR ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

3.11only security fixes3.12only security fixesrelease-blockertopic-regextype-bugAn unexpected behavior, bug, or error

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions