Support parametrized tests in unittest

@terryjreedy

BPO	7897
Nosy	@terryjreedy, @ncoghlan, @pitrou, @ezio-melotti, @merwok, @bitdancer, @voidspace, @cjerdonek, @kynan, @Julian, @ericsnowcurrently, @vadmium, @zware, @phmc, @moormaster

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/voidspace'
closed_at = None
created_at = <Date 2010-02-10.02:21:20.694>
labels = ['type-feature', 'tests']
title = 'Support parametrized tests in unittest'
updated_at = <Date 2019-10-28.20:47:39.885>
user = 'https://bugs.python.org/fperez'

bugs.python.org fields:

activity = <Date 2019-10-28.20:47:39.885>
actor = 'pconnell'
assignee = 'michael.foord'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2010-02-10.02:21:20.694>
creator = 'fperez'
dependencies = []
files = []
hgrepos = []
issue_num = 7897
keywords = []
message_count = 60.0
messages = ['99149', '99215', '99216', '99217', '99218', '99222', '99223', '99224', '99226', '99243', '102667', '102740', '102741', '102750', '102754', '105508', '105520', '105554', '113113', '140729', '140730', '140732', '140733', '140735', '140736', '140737', '140738', '140739', '140740', '140766', '140771', '140772', '140776', '140777', '140785', '140804', '140809', '140811', '140818', '140837', '140838', '140843', '140844', '147236', '153450', '153452', '161979', '163809', '164511', '164512', '164967', '181507', '181859', '196378', '196382', '196425', '224457', '224492', '224501', '352882']
nosy_count = 26.0
nosy_names = ['terry.reedy', 'spiv', 'exarkun', 'ncoghlan', 'pitrou', 'ezio.melotti', 'eric.araujo', 'r.david.murray', 'michael.foord', 'hpk', 'fperez', 'chris.jerdonek', 'Yaroslav.Halchenko', 'santoso.wijaya', 'nchauvat', 'kynan', 'Julian', 'abingham', 'eric.snow', 'martin.panter', 'zach.ware', 'borja.ruiz', 'bfroehle', 'pconnell', 'Mikl\xc3\xb3s.Fazekas', 'moormaster']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue7897'
versions = ['Python 3.6']

IPython has unittest-based parametric testing (something nose has but
which produces effectively undebuggable tests, while this approach
gives perfectly debuggable ones). The code lives here for 2.x and
3.x:

http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/testing/_paramtestpy2.py

http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/testing/_paramtestpy3.py

we import them into our public decorators module for actual use:

http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/testing/decorators.py

Simple tests showing them in action are here:

http://bazaar.launchpad.net/%7Eipython-dev/ipython/trunk/annotate/head%3A/IPython/testing/tests/test_decorators.py#L45

The code is all BSD and we'd be more than happy to see it used
upstream; the less we have to carry the better.

If there is interest in this code, I'm happy to sign a PSF contributor agreement, the code is mostly my authorship. I received help for the 3.x version on the Testing in Python mailing list, I would handle asking for permission on-list if there is interest in including this.

I'm not sure what this brings. It is easy to write a loop iterating over test data.
What parametric testing could bring is precise progress and error display (displaying the parameters for each run), but this doesn't seem to be the case here, since you are not making the parameters available to unittest. Your examples just boil down to writing "yield is_smaller(x, y)" instead of "self.assertLess(x, y)"...

With paramterized tests *all* the tests are run and *all* failures reported. With testing in a loop the tests stop at the first failure.

By the way - I have no opinion on whether or not using yield is the right way to support parameterized tests. It may be better for the test method to take arguments, and be decorated as a parameterized test, with the decorator providing the parameters. When I come to look at it I will look at how py.test and nose do it and solicit advice on the Testing in Python list. We had a useful discussion there previously that would be good to refer to.

With paramterized tests *all* the tests are run and *all* failures reported. With testing in a loop the tests stop at the first failure.

+1 to this justification. Parameterized tests are a big win over a simple for loop in a test.

(However, I haven't looked at the IPython code at all, and Antoine's objection seemed to have something in particular to do with the IPython code?)

It may be better for the test method to take arguments, and
be decorated as a parameterized test, with the decorator
providing the parameters.

+1 on something like this. That's also how NUnit supports parameterized tests.

> With paramterized tests *all* the tests are run and *all* failures reported. With testing in a loop the tests stop at the first failure.

+1 to this justification. Parameterized tests are a big win over a
simple for loop in a test.

Ah, thank you. Looks better indeed.

(However, I haven't looked at the IPython code at all, and Antoine's
objection seemed to have something in particular to do with the
IPython code?)

No, it has to do that you need to be able to distinguish the different
runs of your parameterized test (the same way you distinguish between
different test methods by their names, or line numbers, in the test
report).

If I have 500 runs in my parameterized (*) test, and the only failure
reported is that "2 is not greater than 3", I don't know where to start
looking because the traceback won't give me the information of *which*
parameters were currently in use.

(*) (this is horrible to type)

Antoine: the failure message would include a repr of the parameters used in the particular test that failed. So you can tell which test failed and with what parameters.

Something else I think it would be nice to consider is what the id() (and shortDescription(), heh) of the resulting tests will be.

It would be great if the id were sufficient to identify a particular test *and* data combination.

In trial, we're trying to use test ids to support distributed test running. If the above property holds for parameterized tests, then we'll be able to automatically distribute them to different processes or hosts to be run.

I should probably have clarified better our reasons for using this type of code. The first is the one Michael pointed out, where such parametric tests all execute; it's very common in scientific computing to have algorithms that only fail for certain values, so it's important to identify these points of failure easily while still running the entire test suite.

The second is that the approach nose uses produces on failure the nose stack, not the stack of the test. Nose consumes the test generators at test discovery time, and then simply calls the stored assertions at test execution time. If a test fails, you see a nose traceback which is effectively useless for debugging and with which using --pdb for interactive debugging doesn't help much (all you can do is print the values, as your own stack is gone). This code, in contrast, evaluates the full test at execution time, so a failure can be inspected 'live'. In practice this makes an enormous difference in a test suite being actively useful for ongoing development where changes may send you into debugging often.

I hope this helps clarify the intent of the code better, I'd be happy to provide further details.

In PyMVPA we have our little decorator as an alternative to Fernando's generators, and which is closer, I think, to what Michael was wishing for:
@sweepargs

https://github.com/yarikoptic/PyMVPA/blob/master/mvpa/testing/sweepargs.py

NB it has some minor PyMVPA specificity which could be easily wiped out, and since it was at most 4 eyes looking at it and it bears "evolutionary" changes, it is far from being the cleanest/best piece of code, BUT:

it is very easy to use, just decorate a test method/function and give an argument which to vary within the function call, e.g smth like

@sweepargs(arg=range(5))
def test_sweepargs_demo(arg):
    ok_(arg < 5)
    ok_(arg < 3)
    ok_(arg < 2)

For nose/unittest it would still look like a single test

if failures occur, sweepargs groups failures by the type/location of the failures and spits out a backtrace for one of failures + summary (instead of detailed backtraces for each failure) specifying which arguments lead to what error... here is the output for example above:

$> nosetests -s test_sweepargs_demo.py
F
======================================================================
FAIL: mvpa.tests.test_sweepargs_demo.test_sweepargs_demo
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.5/nose/case.py", line 183, in runTest
    self.test(*self.arg)
  File "/usr/lib/pymodules/python2.5/nose/util.py", line 630, in newfunc
    return func(*arg, **kw)
  File "/home/yoh/proj/pymvpa/pymvpa/mvpa/tests/test_sweepargs_demo.py", line 11, in test_sweepargs_demo
    ok_(arg < 2)
  File "/usr/lib/pymodules/python2.5/nose/tools.py", line 25, in ok_
    assert expr, msg
AssertionError: 
 Different scenarios lead to failures of unittest test_sweepargs_demo (specific tracebacks are below):
  File "/home/yoh/proj/pymvpa/pymvpa/mvpa/tests/test_sweepargs_demo.py", line 10, in test_sweepargs_demo
    ok_(arg < 3)
    File "/usr/lib/pymodules/python2.5/nose/tools.py", line 25, in ok_
    assert expr, msg
  on
    arg=3 
    arg=4

File "/home/yoh/proj/pymvpa/pymvpa/mvpa/tests/test_sweepargs_demo.py", line 11, in test_sweepargs_demo
ok_(arg < 2)
File "/usr/lib/pymodules/python2.5/nose/tools.py", line 25, in ok_
assert expr, msg
on
arg=2

----------------------------------------------------------------------
Ran 1 test in 0.003s

FAILED (failures=1)

obviousely multiple decorators could be attached to the same test, to test on all combinations of more than 1 argument but output atm is a bit cryptic ;-)

Hey Yarick,

On Thu, Apr 8, 2010 at 18:53, Yaroslav Halchenko <report@bugs.python.org> w=
rote:

In PyMVPA we have our little decorator as an alternative to Fernando's ge=
nerators, =A0and which is closer, I think, to what Michael was wishing for:
@sweepargs

https://github.com/yarikoptic/PyMVPA/blob/master/mvpa/testing/sweepargs.py

NB it has some minor PyMVPA specificity which could be easily wiped out, =
and since it was at most 4 eyes looking at it and it bears "evolutionary" c=
hanges, it is far from being the cleanest/best piece of code, BUT:

it is very easy to use, just decorate a test method/function and give a=
n argument which to vary within the function call, e.g smth like

@sweepargs(arg=3Drange(5))
def test_sweepargs_demo(arg):
=A0 =A0ok_(arg < 5)
=A0 =A0ok_(arg < 3)
=A0 =A0ok_(arg < 2)

For nose/unittest it would still look like a single test

Thanks for the post; I obviously defer to Michael on the final
decision, but I *really* would like a solution that reports an
'argument sweep' as multiple tests, not as one. They are truly
multiple tests (since they can pass/fail independently), so I think
they should be treated as such.

On the other hand, your code does have nifty features that could be
used as well, so perhaps the best of both can be used in the end.

Cheers,

f

Fernando, I agree... somewhat ;-)

At some point (whenever everything works fine and no unittests fail) I wanted to merry sweepargs to nose and make it spit out a dot (or animate a spinning wheel ;)) for every passed unittest, so instead of 300 dots I got a picturesque field of thousands dots and Ss and also saw how many were skipped for some parametrizations. But I became "Not sure" of such feature since field became quite large and hard to "grasp" visually although it gave me better idea indeed of what was the total number of "testings" were done and skipped. So may be it would be helpful to separate notions of tests and testings and provide user ability to control the level of verbosity (1 -- tests, 2 -- testings, 3 -- verbose listing of testings (test(parametrization)))

But I blessed sweepargs every time whenever something goes nuts and a test starts failing for (nearly) all parametrization at the same point. And that is where I really enjoy the concise summary.
Also I observe that often an ERROR bug reveals itself through multiple tests. So, may be it would be worth developing a generic 'summary' output which would collect all tracebacks and then groups them by the location of the actual failure and tests/testings which hit it?

subtests don't satisfy my use cases. You can't run an individual subtest by name, and I find that to be a very important thing to be able to do during development and debugging. At the moment at least I'm fine with just having my parameterize decorator in the email module, so I'm not motivated to move this forward right now. I would like to come back to it eventually, though.

Right, subtests are about improving reporting without adding selectivity.
Explicitly parameterized tests require more structural changes to tests,
but give the selectivity that subtests don't.

Is there any possibility of getting this into 3.5? If it helps I've always got time on my hands so if nothing else I could do testing on Windows 8.1.

I don't believe the "how to request specific parameters" aspect has been
clearly addressed so far, and the subTest() API in Python 3.4 already
allows subtests that aren't individually addressable.

Any new API (if any) should likely also be based on the subtest feature,
and I don't think the community has enough experience with that yet to be
confident in how best to integrate it with parameterised testing.

I agree with Nick. There is a potential use case for parameterized tests as well as sub tests, but it's not something we're going to rush into.

@parameterized

+1 for the feature

Subtests make the test results of all asserts visible at test execution time but decrease the readability of a test:

@parameterized([2,4,6])
def test_method_whenCalled_returnsNone(self, a):
# 1) arrange
something = Something()

    # 2) act
    result = something.method(a)

    # 3) assert
    self.assertIsNone(result)

When using subtests the phases of 1) arrange, 2) act, 3) assert are not clearly separated, the unit test contains logic and two additional indentation levels that could be avoided with parameterized tests:

def test_method_whenCalled_returnsNone(self, a):
    # 1) arrange
    something = Something()

    for a in [2,4,6]:
        with self.subTest(a=a):
            # 2) act
            result = something.method(a)

            # 3) assert
            self.assertIsNone(result)

The unittest-enhancing absl.testing.parameterized module is another third-party implementation of this that fits in better with unittest as is. https://github.com/abseil/abseil-py/blob/main/absl/testing/parameterized.py

There are two ways to parametrize test method.

Use a simple decorator which runs the test method with different arguments as subtests. This is easy, it saves 1-2 levels of indentation per parameter, but it have limitations. They are not truly different tests, you cannot run only a subset of tests. Added helper to test.support for parameterizing tests #135120 implements such private decorator in test.support, later it can be added to unittest.
Real parameterization should generate separate test for each compination of parameter values. It requires more deep changes in unittest. But it has advantages -- each test is independent, with its own set-up/tear-down, and we may implement selecting tests by parameter values, not only by name.

(2) can be done via a class decorator. I implemented this with a terrible API in test_email/__init__/parameterize, and I have an implementation I'm much happier with here: https://github.com/bitdancer/parameterizabletests. I haven't touched that code for a long time, but a closely related version of it has been in use continuously in a private repo, so the code hasn't bit rotted.

I would prefer this feature be implemented by unittest TestCase, since decorating a whole class to parameterize only some test methods feels a bit weird. I don't think the changes to unittest itself would be significant even to do it as integrated rather than a class decorator; my code isn't that complex. IMO the biggest problem might be bikeshedding the API ;)

If there is interest I will work on a PR.

Navigation Menu

Repository navigation

Support parametrized tests in unittest #52145

42 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Support parametrized tests in unittest #52145

Description

Activity

fperez commented on Feb 10, 2010

pitrou commented on Feb 11, 2010

voidspace commented on Feb 11, 2010

voidspace commented on Feb 11, 2010

exarkun commented on Feb 11, 2010

briancurtin commented on Feb 11, 2010

pitrou commented on Feb 11, 2010

voidspace commented on Feb 11, 2010

exarkun commented on Feb 11, 2010

fperez commented on Feb 11, 2010

YaroslavHalchenko commented on Apr 9, 2010

fperez commented on Apr 9, 2010

YaroslavHalchenko commented on Apr 9, 2010

42 remaining items

bitdancer commented on Aug 28, 2013

ncoghlan commented on Aug 28, 2013

BreamoreBoy commented on Jul 31, 2014

ncoghlan commented on Aug 1, 2014

voidspace commented on Aug 1, 2014

moormaster commented on Sep 20, 2019

ginneapig commented on Jan 20, 2023

carljm commented on Jan 20, 2023

gpshead commented on Jun 13, 2023

serhiy-storchaka commented on Jun 4, 2025

bitdancer commented on Jun 5, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions