Skip to content

pytest tests marked as passed or failed if test name contains upper letters #6695

Closed

Description

Hello,

I've encountered a little bug recently in the "Tests" tab. I am not 100% sure that the bug is related to vscode or to the python extension. According to the answer on vscode repo, it is due to the extension, that's why I'm reposting here.

Environment data

  • VS Code version: 1.34.0
  • Extension version (available under the Extensions sidebar): 2019.6.24221
  • OS and version: Windows_NT x64 6.1.7601
  • Python version (& distribution if applicable, e.g. Anaconda): 3.6.8 64 bit
  • Type of virtual environment used : conda
  • Relevant/affected Python packages and their versions: user interface of the test tab, pytest
  • Jedi or Language Server? "python.jediEnabled" = True

My file settings.json is :

{
    "python.testing.pytestArgs": [
        "tests"
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true
}

Bug description :

When a test file contains an upper letter, it is not displayed properly in the test tab. After running if passed or failed, it is displayed with a "grey question mark".

Steps to Reproduce:

  1. Create an empty project (note : use a conda environment where pytest is installed)
  2. Create a "tests" folder
    2.1 Create a test_working.py file
import pytest
def test_this_one_works():
    assert 1==1

2.2 Create a test_NotWorking.py file with the same code

import pytest
def test_this_one_works():
    assert 1==1

In the Test tab of vscode, run Discover Tests
If everything ran properly, you should see the following architecture :

  • (?) tests
    • (?) test_notworking.py (with lower case!!!)
      • (?) test_this_one_does_not_work
    • (?) test_working.py
      • (?) test_this_one_works

In the Test tab of vscode, run Run All Tests.

Actual & expected behaviour

-(green tick ok) tests
- (?) test_notworking.py <- BUG - EXPECTED BEHAVIOUR : should have a green tick and have name in upper case
- (?) test_this_one_does_not_work <- BUG - EXPECTED BEHAVIOUR : should have a green tick and have name in upper case
- (green tick ok) test_working.py (with lower case!!!)
- (green tick ok) test_this_one_works

example_img

Logs

The tests seem to be ran whatever is displayed (the bug only concerns the user interface) : indeed in the bottom line (where my python version and my environment are shown), I see an icon "two tests passed". When i run pytest tests in the command line, everything is fine.

Python Test log

python C:\Users\F400411\.vscode\extensions\ms-python.python-2019.6.24221\pythonFiles\testing_tools\run_adapter.py discover pytest -- -s --cache-clear tests
============================= test session starts =============================
platform win32 -- Python 3.6.8, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: c:\Users\F400411\Documents\1_Datascience\3_MISSIONS\bug_test_vscode
collected 2 items

tests\test_NotWorking.py .                                               [ 50%]
tests\test_working.py .                                                  [100%]

- generated xml file: C:\Users\F400411\AppData\Local\Temp\tmp-5180zo59mydLbZ4a.xml -
========================== 2 passed in 0.03 seconds ===========================
python C:\Users\F400411\.vscode\extensions\ms-python.python-2019.6.24221\pythonFiles\testing_tools\run_adapter.py discover pytest -- -s --cache-clear tests
============================= test session starts =============================
platform win32 -- Python 3.6.8, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: c:\Users\F400411\Documents\1_Datascience\3_MISSIONS\bug_test_vscode
collected 2 items

tests\test_NotWorking.py .                                               [ 50%]
tests\test_working.py .                                                  [100%]

- generated xml file: C:\Users\F400411\AppData\Local\Temp\tmp-5180SM9GGMumKvVy.xml -
========================== 2 passed in 0.02 seconds ===========================

Conclusion :

This bugs is not critical at all since it only concerns the UI, but in my old projects I can't see the tests in the UI (while it used to work perfectly fine few weeks ago). I definitely do not want to change the names of the scripts (the tests scripts are a mirror of the scripts names of my project).

Any hope this would be easy to fix?

Many thanks for reading and any further help !
Galileo

Activity

  1. ghost removed
    triage-neededNeeds assignment to the proper sub-team
    on Jul 25, 2019
  2. removed their assignment
    on Jul 25, 2019
  3. karrtikr commented on Jul 25, 2019

    @karrtikr

    Thanks for reporting this Yolan Honoré-Rougé (@Galileo-Galilei)

  4. PhilMacKay commented on Jul 26, 2019

    @PhilMacKay

    This is very likely related to #6410 except in the current issue, upper case letters are in the file name instead of the directory name.

  5. marsfan commented on Jul 28, 2019

    @marsfan

    Philippe MacKay (@PhilMacKay) Don Jayamanne (@DonJayamanne) I just stumbled across this trying to fix it on my computer. It is definitely (in my opnion) an extension of #6410.

    In settings.json if I put
    "python.testing.pytestArgs": [ "tests" ],
    to set up pytest, it correctly displays the markers as a red x or a green check.
    But if I put
    "python.testing.pytestArgs": [ "Tests" ],`
    in settings .json, the results are all the grey questionmark, even after running.

    Hereis the python test log from the a passed test run with all lowercase folder name in settings.json

    platform win32 -- Python 3.7.4, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
    rootdir: d:\Documents\GitHub\Data-Processing
    collected 1 item
    
    tests\test_dataextraction.py .                                           [100%]
    
    - generated xml file: C:\Users\Gabe\AppData\Local\Temp\tmp-143848g0BcuWYtuLN.xml -
    ========================== 1 passed in 0.07 seconds ===========================
    

    And here is the failed, where the folder is listed as Tests

    platform win32 -- Python 3.7.4, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
    rootdir: d:\Documents\GitHub\Data-Processing
    collected 1 item
    
    Tests\test_dataextraction.py .                                           [100%]
    
    - generated xml file: C:\Users\Gabe\AppData\Local\Temp\tmp-14384m2leEWBHKABE.xml -
    ========================== 1 passed in 0.08 seconds ===========================
    

    For me it appears that the actual folder name does not matter, just the capitalization in settings.json

  6. PhilMacKay commented on Aug 7, 2019

    @PhilMacKay

    Gabe R. (@marsfan) I've tried this on my side and it doesn't solve my issue.
    Have you tried to see what happens when the python file itself has capital letters in it? What about if it was located inside a sub-folder with a capital letter in it?

    Edit: After extensive tests, I have been able to reproduce your issue. Using a Pytest version below 5.0.0 and Python 3.7.3, I get the exact same behaviour as you. It enables me to run tests :D
    For a complete fix, we need to change the "NORMCASE" method in ~.vscode\extensions\ms-python.python-2019.8.29288\pythonFiles\testing_tools\adapter\pytest_discovery.py in class TestCollector. I changed NORMCASE = staticmethod(os.path.normcase) to NORMCASE = staticmethod(os.path.normpath) and it solved the problem!

  7. ericsnowcurrently commented on Aug 13, 2019

    @ericsnowcurrently

    This should be resolved now due to #6781 and #6877. Those changes are already in the insiders build of the extension and will be part of the September release.

  8. ericsnowcurrently commented on Aug 14, 2019

    @ericsnowcurrently

    Philippe MacKay (@PhilMacKay) indicated that this is resolved. See #6820 (comment)

  9. locked as resolved and limited conversation to collaborators on Aug 21, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area-testingbugIssue identified by VS Code Team member as probable bugregressionBug didn't exist in a previous release

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions