Skip to content

Occasionally multiprocessing.test.*.test_threads can take 10 minutes #113205

Closed
Closed
Occasionally multiprocessing.test.*.test_threads can take 10 minutes#113205
Assignees
encukou
Labels
3.11only security fixes3.12only security fixestestsTests in the Lib/test dirtopic-multiprocessingtype-bugAn unexpected behavior, bug, or error

Description

@gpshead
gpshead
opened on Dec 16, 2023

Bug report

Bug description:

See the logs in https://buildbot.python.org/all/#/builders/1138/builds/498

10 slowest tests:
- test.test_multiprocessing_spawn.test_threads: 10 min 14 sec
- test.test_multiprocessing_fork.test_threads: 10 min 13 sec
- test_imaplib: 1 min 31 sec
- test_signal: 1 min 18 sec
- test.test_multiprocessing_spawn.test_processes: 52.6 sec
- test.test_concurrent_futures.test_wait: 48.6 sec
- test_socket: 40.7 sec
- test.test_multiprocessing_forkserver.test_processes: 40.1 sec
- test_io: 37.4 sec
- test_codecs: 37.3 sec

Normally those two .test_threads sub-suites in test_multiprocessing do not make the top 10 slowest at all.

Something odd is going on and occasionally making them sit around for a very long time before apparently succeeding anyways. I've seen it multiple time in the rpi buildbot logs. I expect it'll show up on some other bots as well but I haven't gone hunting.

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

Activity

  1. gpshead
    added
    type-bugAn unexpected behavior, bug, or error
    testsTests in the Lib/test dir
    on Dec 16, 2023
  2. AlexWaygood
    added
    topic-multiprocessing
    on Dec 17, 2023
  3. gpshead
    added
    3.11only security fixes
    3.12only security fixes
    on Dec 19, 2023

  4. gpshead commented on Dec 19, 2023

    @gpshead
    gpshead
    on Dec 19, 2023
    MemberAuthor

    Notably I'm not seeing this on the main 3.x branch builds, only 3.12 and 3.11. https://buildbot.python.org/all/#/workers/25

  5. encukou commented on Jan 17, 2024

    @encukou
    encukou
    on Jan 17, 2024
    Member

    It appears on Windows too, and there the refleak buildbot sometimes fails with a timeout.

    I will investigate.

  6. encukou commented on Jan 17, 2024

    @encukou
    encukou
    on Jan 17, 2024
    Member

    WithThreadsTestPool.test_terminate takes LONG_TIMEOUT (5 minutes) in about 1.5% cases (on my machine).

    It happens when the first task starts executing before termination. Unlike processes, threads can't be safely forcefully stopped, so the expected behaviour of ThreadPool.terminate is to wait for in-progress tasks to finish.
    But the test is designed for process pools, and it uses very long tasks.

    On main, the test is now skipped for threads entirely. We can do better: for threads, use a shorter task, but still make sure the API works.

  7. encukou
    added a commit that references this issue on Jan 17, 2024

    pythongh-113205: test_multiprocessing.test_terminate: Test the API wo…

    f34cfdb
  8. gpshead
    added a commit that references this issue on Jan 18, 2024

    gh-113205: test_multiprocessing.test_terminate: Test the API on threa…

    c1db960
  9. miss-islington
    added 2 commits that reference this issue on Jan 18, 2024

    pythongh-113205: test_multiprocessing.test_terminate: Test the API on…

    09cd0dc

    pythongh-113205: test_multiprocessing.test_terminate: Test the API on…

    db5921b

  10. gpshead commented on Jan 18, 2024

    @gpshead
    gpshead
    on Jan 18, 2024
    MemberAuthor

    The merged PR in main has a few buildbot failures. is this perhaps related: https://buildbot.python.org/all/#/builders/345/builds/6859/steps/5/logs/stdio

  11. gpshead
    assigned
    encukou
    on Jan 18, 2024

  12. encukou commented on Jan 18, 2024

    @encukou
    encukou
    on Jan 18, 2024
    Member

    The PR changed one specific case, so it's unrelated.
    But I'll look at those failures anyway, with higher priority than backporting this fix.

  13. 5 remaining items

  14. serhiy-storchaka commented on Jan 18, 2024

    @serhiy-storchaka
    serhiy-storchaka
    on Jan 18, 2024
    Member

    It is disturbing that it only happens in 1.5% cases. It means that in majority of cases terminate() is called when no tasks was even started. It is perhaps not for what this test was purposed (by running slow tasks which take "forever" to complete).

  15. serhiy-storchaka
    reopened this on Jan 18, 2024
  16. serhiy-storchaka
    added a commit that references this issue on Jan 18, 2024

    pythongh-113205: test_multiprocessing.test_terminate: Give tasks a ch…

    ce35e5a
  17. kulikjak
    added a commit that references this issue on Jan 22, 2024

    pythongh-113205: test_multiprocessing.test_terminate: Test the API on…

    d03347f
  18. serhiy-storchaka
    added a commit that references this issue on Jan 24, 2024

    gh-113205: test_multiprocessing.test_terminate: Give tasks a chance t…

    ce75b4c
  19. miss-islington
    added 2 commits that reference this issue on Jan 24, 2024

    pythongh-113205: test_multiprocessing.test_terminate: Give tasks a ch…

    0c8555a

    pythongh-113205: test_multiprocessing.test_terminate: Give tasks a ch…

    1ef99a6
  20. serhiy-storchaka
    closed this as completedon Jan 24, 2024
  21. serhiy-storchaka
    added 2 commits that reference this issue on Jan 24, 2024

    [3.11] gh-113205: test_multiprocessing.test_terminate: Give tasks a c…

    b39a314

    [3.12] gh-113205: test_multiprocessing.test_terminate: Give tasks a c…

    03f8f77

  22. encukou commented on Jan 24, 2024

    @encukou
    encukou
    on Jan 24, 2024
    Member

    We could also go all the way and let the worker signal its start using a event or queue, but sleep is good enough.

  23. aisk
    added 2 commits that reference this issue on Feb 11, 2024

    pythongh-113205: test_multiprocessing.test_terminate: Test the API on…

    2d366e1

    pythongh-113205: test_multiprocessing.test_terminate: Give tasks a ch…

    7d7a297
  24. Glyphack
    added 2 commits that reference this issue on Sep 2, 2024

    pythongh-113205: test_multiprocessing.test_terminate: Test the API on…

    ddf3788

    pythongh-113205: test_multiprocessing.test_terminate: Give tasks a ch…

    825dede
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

3.11only security fixes3.12only security fixestestsTests in the Lib/test dirtopic-multiprocessingtype-bugAn unexpected behavior, bug, or error

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions