Split long running test suites up into smaller sub suites to reduce our regrtest long tail on multi-core systems.

The slowest tests of the Python test suite are:

test_concurrent_futures
test_multiprocessing_spawn
test_peg_generator
test_tools.test_freeze tests

They each tend take multiple minutes to run. The bulk of our others take <10 seconds. They occupy a lot of wall time as a long tail in a normal make test or regrtest run on a typical parallel run multi-core systems.

This bug originally proposed to skip them unless the "cpu" resource is enabled (it's disabled by default). That was deemed appropriate for test_peg_generator and test_tools.test_freeze which are either not platform specific or rarely needed in CI. (PR #108386) - This PR reduced the total test duration between 3 and 5 minutes.

Linked PRs

pythongh-108388: test_concurrent_futures requires cpu

pythongh-108388: regrtest runs slowest tests first

pythongh-108388: regrtest splits test_asyncio package

pythongh-108388: Split test_multiprocessing_spawn

gh-108388: regrtest splits test_asyncio package (#108393)

pythongh-108388: regrtest splits test_asyncio package (pythonGH-108393)

gh-108388: Split test_multiprocessing_spawn (#108396)

pythongh-108388: Convert test_concurrent_futures to package

I experience new failures of test_concurrent_futures on Windows on my unrelated PR: https://github.com/python/cpython/actions/runs/5972558288/job/16203298171?pr=108456

Failing tests:

     test.test_concurrent_futures.test_deadlock
     test.test_concurrent_futures.test_shutdown

I experience new failures of test_concurrent_futures on Windows on my unrelated PR

Sadly, the issue is known for at least one month: see issue #107219.

Logs:

  File "D:\a\cpython\cpython\Lib\test\test_concurrent_futures\test_deadlock.py", line 236 in test_crash_big_data

It's this test which hangs sometimes on Windows.

[3.12] gh-108388: Split test_multiprocessing_spawn (GH-108396) (#108442)

[3.12] gh-108388: Convert test_concurrent_futures to package (#108401) (

pythongh-108388: regrtest splits test_asyncio package (python#108393)

[3.11] gh-108822: Backport libregrtest changes from the main branch (#…

This feature has been implemented in the main branch, and backported to the 3.12 branch. I don't think that it's worth it to backport it to the 3.11 branch.

I splitted these test packages:

test_concurrent_futures
test_multiprocessing_fork
test_multiprocessing_forkserver
test.test_multiprocessing_spawn

Example:

$ ./python -m test test_concurrent_futures --list-tests
test_concurrent_futures.test_as_completed
test_concurrent_futures.test_deadlock
test_concurrent_futures.test_future
test_concurrent_futures.test_init
test_concurrent_futures.test_process_pool
test_concurrent_futures.test_shutdown
test_concurrent_futures.test_thread_pool
test_concurrent_futures.test_wait

I close my issue.

pythongh-108388: Split test_multiprocessing_spawn (pythonGH-108396)

[3.11] gh-108388: Split test_multiprocessing_spawn (GH-108396) (#109688)

[3.11] gh-108388: Convert test_concurrent_futures to package (#108401) (

Navigation Menu

Repository navigation

Split long running test suites up into smaller sub suites to reduce our regrtest long tail on multi-core systems. #108388

Linked PRs

12 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Split long running test suites up into smaller sub suites to reduce our regrtest long tail on multi-core systems. #108388

Description

Linked PRs

Activity

12 remaining items

sobolevn commented on Aug 25, 2023

vstinner commented on Aug 25, 2023

vstinner commented on Sep 13, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions