Skip to content

Test Discovery Fails - Reports as PyTest not installed  #6347

Closed

Description

@drcrook1

Testing was working yesterday; I signed off around 4:45pm EST, got up this morning and encountered issues discovering and importing the packages from vscode. To me it appears as though the root of the workspace is not being added to the path.

Example: executing python -m pytest ETC from c:/projects/customers/CUSTOMERX/PROJECTX/ adds that to the path, specifying ./tests/ searches ./tests for tests and I can import the high level package from there. I feel as if the break is likely in this area.

Environment data

  • VS Code version: XXX
    Version: 1.35.1 (user setup)
    Commit: c7d83e57cd18f18026a8162d042843bda1bcf21f
    Date: 2019-06-12T14:30:02.622Z
    Electron: 3.1.8
    Chrome: 66.0.3359.181
    Node.js: 10.2.0
    V8: 6.6.346.32
    OS: Windows_NT x64 10.0.17134

  • Extension version (available under the Extensions sidebar): 2019.6.22090 (25 June 2019)

  • Python version (& distribution if applicable, e.g. Anaconda): Anaconda 2019.03 for Windows python version 3.6.8

  • Type of virtual environment used (N/A | venv | virtualenv | conda | ...): N/A base

  • Relevant/affected Python packages and their versions: pytest

  • Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info How to update the language server to the latest stable version #3977): Jedi enabled (output indicates it started jedi python language engine)

Expected behaviour

Discover Unit Tests - for example below works:

c:\projects\customers\CUSTOMERX\PROJECTX>python -m pytest --doctest-modules --junitxml=./tests/junit/test-results.xml --cov=./PACKAGEX/ ./tests/ --cov-report=xml:./tests/junit/coverage.xml --cov-report=html:./tests/junit/cov_html
================================================= test session starts =================================================
platform win32 -- Python 3.6.8, pytest-4.6.2, py-1.8.0, pluggy-0.12.0
rootdir: c:\projects\customers\CUSTOMERX\PACKAGEX, inifile: setup.cfg
plugins: arraydiff-0.3, cov-2.7.1, doctestplus-0.3.0, openfiles-0.3.2, remotedata-0.3.1
collected 14 items

tests\test_system_files.py ..                                                                                    [ 14%]

Actual behaviour

From the python test log (outputs) - Tried pytest 4.6.2, 4.6.3 and 4.4.1

python C:\Users\DrCrook\.vscode\extensions\ms-python.python-2019.6.22090\pythonFiles\testing_tools\run_adapter.py discover pytest -- -s --cache-clear c:\projects\customers\CUSTOMERX\PROJECTX
Test Discovery failed: 
Error: ============================= test session starts =============================
platform win32 -- Python 3.6.8, pytest-4.4.1, py-1.8.0, pluggy-0.12.0
rootdir: c:\projects\customers\CUSTOMERX\PROJECTX, inifile: setup.cfg
plugins: arraydiff-0.3, cov-2.7.1, doctestplus-0.3.0, openfiles-0.3.2, remotedata-0.3.1
collected 0 items / 9 errors

=================================== ERRORS ====================================
_________________ ERROR collecting tests/test_system_files.py _________________
ImportError while importing test module 'c:\projects\customers\CUSTOMERX\PROJECTX\tests\test_system_files.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests\test_system_files.py:6: in <module>
    from PACKAGEX.result import Result
E   ModuleNotFoundError: No module named 'PACKAGEX'

My folder structure:

/root
   /PACKAGEX
        /submoduleX
        /submoduleY
   /tests
        /helpers
        /test_submoduleX
        /test_submoduleY

.vscode/settings.json

{
    "python.testing.pytestArgs": [
    "--verbose",
    "--rootdir",
    "${workspaceFolder}",
    "${workspaceFolder}/tests"
    ],
    "python.testing.cwd": "${workspaceFolder}",
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.pythonPath": "C:\\ProgramData\\Anaconda3\\python.exe"
   }

Activity

  1. added
    triage-neededNeeds assignment to the proper sub-team
    bugIssue identified by VS Code Team member as probable bug
    on Jun 26, 2019
  2. changed the title [-]Test Discovery Fails - Reports as PyTest not installed - More Problems Too[/-] [+]Test Discovery Fails - Reports as PyTest not installed [/+] on Jun 26, 2019
  3. ghost removed
    triage-neededNeeds assignment to the proper sub-team
    on Jun 26, 2019
  4. drcrook1 commented on Jun 26, 2019

    @drcrook1
    Author

    Additionally, this is new and maybe related. Ctrl + Shift + P -> Python: Configure Tests yeilds:
    image

    could be related:
    #5619

  5. drcrook1 commented on Jun 26, 2019

    @drcrook1
    Author

    Verified my alternate work computer has the same problem. All machines I have across all python projects are failing on this as of today.

  6. bchamand commented on Jun 27, 2019

    @bchamand

    I'm in the same situation of you except I'm in MacOS 10.14.5. I directly downgrade the version of the extension to go back to the previous version.

  7. jahan01 commented on Jun 27, 2019

    @jahan01

    +1, I am on Mac 10.14, presumably started after last update of python extension

  8. lukasz-migas commented on Jun 27, 2019

    @lukasz-migas

    Same problem here. Downgrading to the previous version (2019.5.18875) fixes the problem.

  9. drcrook1 commented on Jun 27, 2019

    @drcrook1
    Author

    downgrading the extension works for me as well. version (2019.5.18875)

  10. kimadeline commented on Jun 27, 2019

    @kimadeline

    Hello David Crook (@drcrook1) , Benjamin Chamand (@bchamand) , jahan01 and Lukasz Migas (@lukasz-migas) 👋 thank you for reporting this issue and confirming that downgrading the extension works. If you could verify the following while I comb through the extension that'd be great 🔎:

    • you said that python -m pytest ./tests works, can you try running pytest ./tests/?
    • are there any __init__.py files in your test folders? Something that would look like:
    root/
       mypackage/
          __init__.py
          submoduleA/
             __init__.py
             fileA.py
          submoduleB/
             __init__.py
             fileB.py
       tests/
          __init__.py
          test_subA/
             __init__.py
             test_A.py
          test_subB/
             __init__.py
             test_B.py
    
  11. lukasz-migas commented on Jun 27, 2019

    @lukasz-migas

    Kim-Adeline Miguel (@kimadeline) python -m pytest works fine but pytest ./tests does not. My project is structured mostly like that although I keep the tests folders within each submodule. Each folder and subfolder has its own __init__.py.

  12. kimadeline commented on Jun 27, 2019

    @kimadeline

    The failing discovery command is pytest -s --cache-clear --collect-only /path/to/root/folder being run in the root folder.

    We changed how pytest discovers tests in 2019.6.22090:
    ✋ We used to parse some of pytest's output to know which files contained tests, and then loop through these files one by one. However, it was unreliable and discouraged.
    👉 Now we leave the entire discovery up to pytest.

    Assuming you have this structure (notice the absence of __init__.py files in tests/ and its subfolders):

    root/
       mypackage/
          __init__.py
          submoduleA/
             __init__.py
             fileA.py
          submoduleB/
             __init__.py
             fileB.py
       tests/
          test_subA/
             test_A.py
          test_subB/
             test_B.py
    

    There is a difference in running python -m pytest [...] and just pytest [...], because the former will add the current directory to sys.path, which in return will make mypackage importable, while the latter doesn't. Because we are now executing a script to call pytest with our own discovery plug-in, we are no longer able to use python -m pytest for that implicit change to sys.path.

    By adding __init__.py files to your tests folders, pytest will import them as packages, add the root directory to sys.path, and make mypackage visible.

    In other words because we are doing the "right" thing by directly plugging into pytest to do discovery, pytest's sys.path is different and so it's resulting in pytest finding things differently from how we used to do it (although it's the same as if you had specified python.testing.pytestPath). If people specifically would like us to consider adding '.' to sys.path, please feel free to open a new issue for the feature request (and people can 👍 it to show that they support the idea).

  13. 5 remaining items

  14. ghost added
    triage-neededNeeds assignment to the proper sub-team
    on Jun 27, 2019
  15. ericsnowcurrently commented on Jun 27, 2019

    @ericsnowcurrently

    The solution will be to fix run_adapter.py to not replace sys.path[0], since that blows away the value set by the interpreter: the workspace folder. However, we still need that new value in sys.path, so we should instead insert it at position 1.

  16. KsenoJiva commented on Jul 1, 2019

    @KsenoJiva

    I noticed that the TestExplorer shows all Paths as lowercase. Therefore it could not find folders and modules within. In your case PACKAGEX might be uppercase too. Might there be a connection?

  17. DonJayamanne commented on Jul 17, 2019

    @DonJayamanne

    Dup of #5919

  18. locked as resolved and limited conversation to collaborators on Jul 24, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-testingbugIssue identified by VS Code Team member as probable bugimportantIssue identified as high-priorityregressionBug didn't exist in a previous release

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions