Skip to content

test_import.test_concurrency leaks ref on Windows #106176

Closed
Closed
test_import.test_concurrency leaks ref on Windows#106176
Labels
3.12only security fixes3.13only security fixesOS-windowsrelease-blockertopic-importlibtype-bugAn unexpected behavior, bug, or error

Description

@sunmy2019
sunmy2019
opened on Jun 28, 2023

On Python 3.12, test_import.test_concurrency() leaks on Windows:

vstinner@WIN C:\victor\python\3.12>python -m test test_import -m test_concurrency -R 3:3
(...)
test_import leaked [22, 36, 28] references, sum=86

bisected to #94504

Potentially Related: #104702

See #102251 (comment) #104796 (comment)

Linked PRs

Activity

  1. sunmy2019
    added
    type-bugAn unexpected behavior, bug, or error
    OS-windows
    on Jun 28, 2023

  2. vstinner commented on Jun 28, 2023

    @vstinner
    vstinner
    on Jun 28, 2023
    Member

    This leak can be seen on the "AMD64 Windows11 Refleaks 3.12" buildbot for example: https://buildbot.python.org/all/#/builders/1103/builds/36

    I reproduced it locally.

  3. vstinner commented on Jun 28, 2023

    @vstinner
    vstinner
    on Jun 28, 2023
    Member

    cc @exarkun @brettcannon

  4. Eclips4 commented on Jun 28, 2023

    @Eclips4
    Eclips4
    on Jun 28, 2023
    Member

    On Python 3.12, test_import.test_concurrency() leaks on Windows:

    Did you mean that this test leaks only on 3.12?
    I can still reproduce it on current main branch (3.13).

    More info 
    ./python -m test test_import -m test_concurrency -R 3:3
Running debug|x64 interpreter...
0:00:00 Run tests sequentially
0:00:00 [1/1] test_import
beginning 6 repetitions
123456
......
test_import leaked [40, 40, 40] references, sum=120
test_import leaked [40, 40, 40] memory blocks, sum=120
test_import failed (reference leak)

== Tests result: FAILURE ==

1 test failed:
    test_import

Total duration: 7.0 sec
Tests result: FAILURE

  5. sunmy2019 commented on Jun 28, 2023

    @sunmy2019
    sunmy2019
    on Jun 28, 2023
    MemberAuthor

    Did you mean that this test leaks only on 3.12?

    No. It also affects 3.12

  6. AlexWaygood
    added
    3.12only security fixes
    3.13only security fixes
    on Jun 28, 2023

  7. exarkun commented on Jun 28, 2023

    @exarkun
    exarkun
    on Jun 28, 2023
    Contributor

    I'm not familiar with the leak checking system and I'm not likely to have much time to learn about it in the foreseeable future.

  8. Eclips4
    added a commit that references this issue on Jun 28, 2023

    pythongh-106176: Fix reference leak in importlib/_bootstrap.py

    5328e32
  9. brettcannon
    added
    topic-importlib
    on Jun 28, 2023

  10. brettcannon commented on Jun 28, 2023

    @brettcannon
    brettcannon
    on Jun 28, 2023
    Member

    @exarkun it just means some things are not getting cleaned up.

  11. exarkun commented on Jun 29, 2023

    @exarkun
    exarkun
    on Jun 29, 2023
    Contributor

    Just trying to signal my likelihood of pitching on here: low.

  12. vstinner commented on Jul 23, 2023

    @vstinner
    vstinner
    on Jul 23, 2023
    Member

    Oops, I just reported a duplicate of this bug: #107086 (closed).

    I also found the same commit with git bisect: commit 3325f05

    commit 3325f054e33b318aa56b74472f76a56b8afc0510
Author: Jean-Paul Calderone <exarkun@twistedmatrix.com>
Date:   Fri Jan 20 19:00:39 2023 -0500

    gh-91351: Fix some bugs in importlib handling of re-entrant imports (GH-94504)

    Co-authored-by: Brett Cannon <brett@python.org>

  13. vstinner commented on Jul 23, 2023

    @vstinner
    vstinner
    on Jul 23, 2023
    Member

    A fix was proposed: PR #106207.

  14. erlend-aasland commented on Aug 1, 2023

    @erlend-aasland
    erlend-aasland
    on Aug 1, 2023
    Contributor

    @Yhg1s: IMO, this is also a release blocker (in my opinion, any ref leak is a release blocker).

  15. Yhg1s commented on Aug 1, 2023

    @Yhg1s
    Yhg1s
    on Aug 1, 2023
    Member

    (Yes, this looks like a real leak, not an artifact of how the refleak checker reruns things, so it should be a release blocker.)

  16. erlend-aasland
    added
    release-blocker
    on Aug 1, 2023
  17. ezio-melotti
    added this to Release and Deferred blockers 🚫on Aug 1, 2023
  18. github-project-automation
    moved this to Todo in Release and Deferred blockers 🚫on Aug 1, 2023

  19. brettcannon commented on Aug 2, 2023

    @brettcannon
    brettcannon
    on Aug 2, 2023
    Member

    We could revert the fix that leads to the leak to unblock the release as I'm not sure if anyone has come up with a good solution to avoid the memory leak and the bugs that get fixed are not fundamental to import (although it would still be nice to have the bugs fixed).

  20. brettcannon
    added a commit that references this issue on Aug 25, 2023

    pythonGH-106176: Fix reference leak when importing across multiple th…

    50e4c5d

  21. brettcannon commented on Aug 25, 2023

    @brettcannon
    brettcannon
    on Aug 25, 2023
    Member

    I created #108497 to try out an idea that @Eclips4 had for using weakrefs.

  22. vstinner
    added a commit that references this issue on Aug 29, 2023

    GH-106176, GH-104702: Fix reference leak when importing across multip…

    5f85b44
  23. miss-islington
    added a commit that references this issue on Aug 29, 2023

    pythonGH-106176, pythonGH-104702: Fix reference leak when importing a…

    d2f0afd
  24. Yhg1s
    added a commit that references this issue on Aug 29, 2023

    [3.12] GH-106176, GH-104702: Fix reference leak when importing across…

    c942fae
  25. Eclips4
    closed this as completedon Aug 29, 2023
  26. github-project-automation
    moved this from Todo to Done in Release and Deferred blockers 🚫on Aug 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.12only security fixes3.13only security fixesOS-windowsrelease-blockertopic-importlibtype-bugAn unexpected behavior, bug, or error

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions