You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
unittest runner thinks failing test is not run #17561
Running tests (unittest): /home/jacob/projects/libreriscv/simd_signal/test_reproducer.py
Running tests: [object Object]
Total number of tests expected to run: 1
Total number of tests run: 1
Total number of tests passed: 0
Total number of tests failed: 0
Total number of tests failed with errors: 0
Total number of tests skipped: 0
Finished running tests!
> Test run finished at 9/28/2021, 5:56:12 PM <
Output for Python Test Log in the Output panel
test (test_reproducer.Tests) ...
======================================================================
FAIL: test (test_reproducer.Tests) (<subtest>)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jacob/projects/libreriscv/simd_signal/test_reproducer.py", line 7, in test
assert False
AssertionError
----------------------------------------------------------------------
Ran 1 test in 0.000s
FAILED (failures=1)
Our test adapter for unittest, when it was written was not designed to handle unittest subtests. This is primarily due to unittest discovery cannot detect them before running the test. when subtests are used, the results are generated slightly differently from when they are not used. So we currently don't have a way to match the before subtest state.
We are in the process of re-writing the test adapter. With the redesign we will take unittest sub tests into consideration.
But it won't make it into up coming release. I will share the build from this PR when it is done here so you can try it out. For the code that yo shared the result should look like this:
Richard Coppin (@coppinri) There are limitations to this. We can only detect sub-test at run time, unittest discovery does not know about sub-tests. The subtest nodes you see above are only reporting nodes, you can specifically select them and run.
Activity
coppinri commentedon Sep 29, 2021
This seems to be the behaviour particular to subtests.
Worse though, if the test passed previously, but now the test assertion fails, the UI still shows it as passed.

karthiknadig commentedon Sep 30, 2021
Our test adapter for
unittest, when it was written was not designed to handleunittestsubtests. This is primarily due tounittestdiscovery cannot detect them before running the test. when subtests are used, the results are generated slightly differently from when they are not used. So we currently don't have a way to match the before subtest state.We are in the process of re-writing the test adapter. With the redesign we will take unittest sub tests into consideration.
karthiknadig commentedon Sep 30, 2021
Related #17242
karthiknadig commentedon Sep 30, 2021
Jacob Lifshay (@programmerjake) I made a PR (#17578) that addresses this issue in some form:

But it won't make it into up coming release. I will share the build from this PR when it is done here so you can try it out. For the code that yo shared the result should look like this:

coppinri commentedon Oct 1, 2021
Wow, I was only expecting it to update the UI on a failed run, but if it's able to delve into- and report on- each subtest that would be great.
karthiknadig commentedon Oct 1, 2021
Richard Coppin (@coppinri) There are limitations to this. We can only detect sub-test at run time,
unittestdiscovery does not know about sub-tests. The subtest nodes you see above are only reporting nodes, you can specifically select them and run.if you want to give this a try, you will need insiders version of VS Code (v 1.61.0-insider) and this build of python extensions https://github.com/microsoft/vscode-python/suites/3925208299/artifacts/98213954