libregrtest: add option to run every test case in separate process

@vstinner

Feature or enhancement

Proposal:

libregrtest can run every test module in a separate process, and can even run several test modules in parallel. It has options to simply output all test modules or all test cases.

I propose to add an option to run every test case in a separate process. Test cases from the same module should still run sequentially, because they can use the same resource (for example create a server at fixed port).

This will slow down the tests several times, but it is not for regular running in the CI, it is for manual running once every months or for manual running selected test module while writing new tests or debugging test flakiness. It can help to solve the following issues:

Order dependency. Some tests are only passed because they run in specific order, and the order of tests in a module is the same, only the order of test modules is randomized. If run the test in a separate process, it is always the first, and cannot depend on others.
Environment changes and leaks. Some tests change the environment (e.g. leak temporary files or do not wait for terminating a thread or a subprocess), and it is not noticed because subsequent tests have cleaned after it. If run the test in a separate process, it is always the last, and nobody cleans after it.

cc @vstinner @gpshead

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Linked PRs

For now I can do this by using the following command:

PYTHON=./python; $PYTHON -m test -uall --list-tests 2>/dev/null | while read -r test
do
    $PYTHON -m test -uall --list-cases $test 2>/dev/null | xargs -n1 $PYTHON -m test -vuall $test -m
done

But it is slow (because it does not parallel different test modules), the output is difficult to analyze, and it cannot be run on Windows.

@gpshead

See also closed PR gh-99637 by @gpshead.

@serhiy-storchaka

hii @serhiy-storchaka i'm planning to go with this, please let me know if you have any pointers for me. currently going through the mentioned PR.

Look at the implementation of --list-tests and --list-cases options. What is common and what is different. Then look how running tests in multiple processes is implemented (option -j). Make it running tests one-case-per-process instead of one-module-per-process.

This can be an easy task, but it can also be difficult. Only undertake this if you are confident in your abilities.

gh-109817: Add --single-process-per-case option to libregrtest (GH-15…

@serhiy-storchaka

@serhiy-storchaka Are we left with something in this or any improvement, please feel free to ping me . If this needs work

gh-109817: Do not copy all test cases into the worker process (GH-155713

Running the whole test suite with --single-process-per-case found five pre-existing test bugs:

test_sysconfig leaves the sysconfig prefix cache modified #155726 -- test_sysconfig leaves the sysconfig prefix cache modified
test_import leaves files in the current directory #155731 -- test_import leaves files in the current directory
test_repr_lock leaves a dangling process #155732 -- test_repr_lock leaves a dangling process
test_attribute_completion fails because of a lazy internal import #155727 -- test_attribute_completion fails because of a lazy internal import
test_track_literals depends on a garbage collection #155728 -- test_track_literals depends on a garbage collection

The first three modify the environment, but this is not detected when the environment is checked once per test file, because other tests in the same file clean up after them.

It also made the lists of failed and skipped tests unreadable, because test case identifiers are longer than module names (GH-155714).

pythongh-109817: Add --single-process-per-case option to libregrtest (p…

pythongh-109817: Do not copy all test cases into the worker process (p…

Navigation Menu

Repository navigation

libregrtest: add option to run every test case in separate process #109817

Feature or enhancement

Proposal:

Has this already been discussed elsewhere?

Links to previous discussion of this feature:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

libregrtest: add option to run every test case in separate process #109817

Description

Feature or enhancement

Proposal:

Has this already been discussed elsewhere?

Links to previous discussion of this feature:

Linked PRs

Activity

serhiy-storchaka commented on Sep 25, 2023

vstinner commented on Sep 25, 2023

Aniketsy commented on Jun 5, 2026

serhiy-storchaka commented on Jun 6, 2026

Aniketsy commented on Aug 13, 2026

serhiy-storchaka commented on Aug 13, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions