Always use unittest for collecting tests in regrtests

@terryjreedy

BPO	45229
Nosy	@terryjreedy, @ambv, @ambv, @zware, @serhiy-storchaka, @pablogsal, @miss-islington
PRs	bpo-45229: Remove test_main in many tests #28405 bpo-45229: Fix setUpModule in test_ssl #28454 [3.10] bpo-45229: Remove test_main in many tests (GH-28405) #28455 [3.9] bpo-45229: Remove test_main in many tests (GH-28405) #28456 bpo-45229: Make pickle tests discoverable #28467 bpo-45229: Use doctest.DocTestSuite instead of run_doctest #28468 [3.10] bpo-45229: Make pickle tests discoverable (GH-28467) #28478 [3.9] bpo-45229: Make pickle tests discoverable (GH-28467) #28479 bpo-45229: Make datetime tests discoverable #28615 bpo-45229: Make tkinter tests discoverable #28637 [3.10] bpo-45229: Make datetime tests discoverable (GH-28615) #28639 [3.9] bpo-45229: Make datetime tests discoverable (GH-28615). #28645 bpo-45229: Revert "Make datetime tests discoverable" #28650 [3.9] Revert "[3.9] bpo-45229: Make datetime tests discoverable (GH-28615) (GH-28645)" #28660 [3.9] Revert "[3.9] Revert "[3.9] bpo-45229: Make datetime tests discoverable (GH-28615) (GH-28645)"" #28666 Revert "bpo-45229: Revert "Make datetime tests discoverable"" #28667 [3.10] bpo-45229: Make datetime tests discoverable (GH-28615) #28668 [3.10] bpo-45229: Make tkinter tests discoverable (GH-28637) #28927 [3.9] bpo-45229: Make tkinter tests discoverable (GH-28637) #28929 bpo-45229: Make doctest tests discoverable #28986 bpo-45229: Make test_http_cookiejar discoverable #29004 [3.10] bpo-45229: Make test_http_cookiejar discoverable (GH-29004) #29006 [3.9] bpo-45229: Make test_http_cookiejar discoverable (GH-29004) #29007 [3.10] bpo-45229: Make doctest tests discoverable (GH-28986) #29095 [3.9] bpo-45229: Make doctest tests discoverable (GH-28986) #29096
Dependencies	bpo-40058: Running test_datetime twice fails with: module 'datetime' has no attribute '_divide_and_round' bpo-40173: test.support.import_fresh_module fails to correctly block submodules when fresh is specified bpo-45185: test.test_ssl.TestEnumerations is not run bpo-45187: Some tests in test_socket are not run bpo-45500: Rewrite test_dbm bpo-45502: Fix test_shelve and make it discoverable

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = None
created_at = <Date 2021-09-17.07:31:12.126>
labels = ['type-feature', 'tests', '3.9', '3.10', '3.11']
title = 'Always use unittest for collecting tests in regrtests'
updated_at = <Date 2021-10-20.16:52:45.847>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2021-10-20.16:52:45.847>
actor = 'lukasz.langa'
assignee = 'serhiy.storchaka'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2021-09-17.07:31:12.126>
creator = 'serhiy.storchaka'
dependencies = ['40058', '40173', '45185', '45187', '45500', '45502']
files = []
hgrepos = []
issue_num = 45229
keywords = ['patch']
message_count = 33.0
messages = ['402006', '402009', '402053', '402076', '402154', '402156', '402168', '402195', '402214', '402228', '402245', '402254', '402255', '402903', '402945', '402952', '402953', '402956', '402959', '402974', '402982', '402996', '402999', '403000', '403846', '403853', '403855', '404137', '404142', '404143', '404466', '404495', '404496']
nosy_count = 7.0
nosy_names = ['terry.reedy', 'lukasz.langa', 'lukasz.langa', 'zach.ware', 'serhiy.storchaka', 'pablogsal', 'miss-islington', 'jerrylikerice1s']
pr_nums = ['28405', '28454', '28455', '28456', '28467', '28468', '28478', '28479', '28615', '28637', '28639', '28645', '28650', '28660', '28666', '28667', '28668', '28927', '28929', '28986', '29004', '29006', '29007', '29095', '29096']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue45229'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

Currently regrtest supports two ways of collecting and running tests in module.

If the module contains the "test_main" function, regrtest just calls it. This function usually calls run_unittest() with a list of test classes, composed manually, it can also run doctests with run_doctest(), generate list of test classes dynamically, and run some code before and after running tests.
Otherwise regrtest uses unittest for for collecting tests.

The disadvantage of the former way is that new test classes should be added manually to the list. If you forget to do this, new tests will not be run. See for example bpo-45185 and bpo-45187. Not runned tests can hide bugs.

So it would be better to eliminate human factor and detect tests automatically.

PR 28405 contains simple changes for 60 test files. It does not cover files which use complex code for generating list of test classes and running doctests (because there are subtle differences between running doctests with run_doctest() and via DocTestSuite). They are left for the following PRs.

Navigation Menu

Repository navigation

Always use unittest for collecting tests in regrtests #89392

Linked PRs

67 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Always use unittest for collecting tests in regrtests #89392

Description

Linked PRs

Activity

serhiy-storchaka commented on Sep 17, 2021

serhiy-storchaka commented on Sep 17, 2021

zware commented on Sep 17, 2021

67 remaining items

serhiy-storchaka commented on Sep 9, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions