Repository navigation
Allow regrtest to receive a file with test (and subtests) to ignore #82138
Description
| BPO | 37957 |
|---|---|
| Nosy | @vstinner, @serhiy-storchaka, @pablogsal |
| PRs |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2019-08-27.00:03:54.483>
labels = ['type-feature', 'tests', '3.9']
title = 'Allow regrtest to receive a file with test (and subtests) to ignore'
updated_at = <Date 2020-04-14.23:05:24.243>
user = 'https://github.com/pablogsal'bugs.python.org fields:
activity = <Date 2020-04-14.23:05:24.243>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2019-08-27.00:03:54.483>
creator = 'pablogsal'
dependencies = []
files = []
hgrepos = []
issue_num = 37957
keywords = ['patch']
message_count = 11.0
messages = ['350584', '350601', '350606', '350607', '350617', '350626', '357006', '357015', '357016', '366411', '366460']
nosy_count = 3.0
nosy_names = ['vstinner', 'serhiy.storchaka', 'pablogsal']
pr_nums = ['16989']
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue37957'
versions = ['Python 3.9']
Activity
pablogsal commentedon Aug 27, 2019
When building Python in some uncommon platforms (I am looking at you Solaris and AIX) there are some known tests that will fail. Right now, regrtest has the ability to ignore entire tests using the -x option and to receive a filter file using the --matchfile filter.
The problem with the --matchfile option is that it receives a file with patterns to accept and when you want to ignore a couple of tests and subtests, is too cumbersome to lists ALL tests that are not the ones that you want to accept.
The problem with -x is that is not easy to ignore just a subtests that fail and the whole tests needs to be ignored.
So I suggest to add a new command line option similar to --matchfile but the other way around. Another possibility is allowing to reverse the meaning of the matchfile argument, but I find that a bit more confusing.
vstinner commentedon Aug 27, 2019
I'm fine with adding an --excludefile=filename option to regrtest. Do you want to try to implement it? A test can be added in test_regrtest.
serhiy-storchaka commentedon Aug 27, 2019
Tests that fail on some platforms can be marked to skip in code.
For example:
vstinner commentedon Aug 27, 2019
We provide best-effort support for AIX, but I consider that Solaris is no longer supported, and so I would prefer to avoid code specific to Soliars in the Python source code. Except if Pablo wants to provide again best-effort support for Solaris?
https://pythondev.readthedocs.io/platforms.html
pablogsal commentedon Aug 27, 2019
One of the use cases of this feature is for people to package Python at distributions or companies and want to run the test suite without modifying the code or applying patches.
vstinner commentedon Aug 27, 2019
FYI in the past, Fedora and RHEL packages skipped multiple tests using downstream patches. Hopefully, we managed to reenable almost all tests.
Currently in Fedora, the python3 package use regrtest -x to skip some tests:
# Run the upstream test suite
# test_gdb skipped on s390x:
# https://bugzilla.redhat.com/show_bug.cgi?id=1678277
# test_gdb skipped everywhere:
# https://bugzilla.redhat.com/show_bug.cgi?id=1734327
# test_distutils
# distutils.tests.test_bdist_rpm tests fail when bootstraping the Python
# package: rpmbuild requires /usr/bin/pythonX.Y to be installed
LD_LIBRARY_PATH=$ConfDir $ConfDir/python -m test.regrtest \
-wW --slowest -j0 \
%if %{with bootstrap}
-x test_distutils \
%endif
-x test_gdb \
%ifarch %{mips64}
-x test_ctypes \
%endif
https://src.fedoraproject.org/rpms/python3/blob/master/f/python3.spec
We could only skip some test methods, but it's simpler for us to skip a whole test file.
pablogsal commentedon Nov 19, 2019
New changeset e0cd8aa by Pablo Galindo in branch 'master':
bpo-37957: Allow regrtest to receive a file with test (and subtests) to ignore (GH-16989)
e0cd8aa
vstinner commentedon Nov 20, 2019
I'm trying to keep regrtest in sync between 3.7, 3.8 and master branches. Maybe backport this change to 3.7 and 3.8 branches once buildbots validated the change? I'm keeping them in sync mostly to make bugfixes easier, but also to make my life easier when I debug issues in 3.7 or 3.8 branch.
pablogsal commentedon Nov 20, 2019
Ok, I reopen until the backports
vstinner commentedon Apr 14, 2020
New changeset 67b8a1f by Victor Stinner in branch '3.8':
[3.8] Update libregrtest from master (GH-19516)
67b8a1f
vstinner commentedon Apr 14, 2020
New changeset b894b66 by Victor Stinner in branch '3.7':
Update libregrtest from master (GH-19517)
b894b66