Skip to content

Test suite skips failing tests when setUp[Class] fails #90681

Closed
Closed
Test suite skips failing tests when setUp[Class] fails#90681
Labels
3.10 (EOL)end of life3.11only security fixes3.9 (EOL)end of lifetestsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

Description

@sobolevn
sobolevn
opened on Jan 25, 2022
BPO 46523
Nosy @gvanrossum, @ezio-melotti, @sobolevn, @Fidget-Spinner
PRs
  • bpo-46523: fix tests rerun when setUp[Class|Module] fails #30895
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2022-01-25.15:42:34.695>
labels = ['type-bug', 'tests', '3.9', '3.10', '3.11']
title = 'Test suite skips failing tests when setUp[Class] fails'
updated_at = <Date 2022-01-25.20:48:08.548>
user = 'https://github.com/sobolevn'

    bugs.python.org fields:

    activity = <Date 2022-01-25.20:48:08.548>
actor = 'sobolevn'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2022-01-25.15:42:34.695>
creator = 'sobolevn'
dependencies = []
files = []
hgrepos = []
issue_num = 46523
keywords = ['patch']
message_count = 2.0
messages = ['411620', '411625']
nosy_count = 4.0
nosy_names = ['gvanrossum', 'ezio.melotti', 'sobolevn', 'kj']
pr_nums = ['30895']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue46523'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

    Activity

    1. sobolevn commented on Jan 25, 2022

      @sobolevn
      sobolevn
      on Jan 25, 2022
      MemberAuthor

      Here's what happened. We had an error in test_typing.py, which was silently ignored.

       ======================================================================
ERROR: setUpClass (test.test_typing.NewTypeTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\cpython\cpython\lib\test\test_typing.py", line 3917, in setUpClass
    UserId = NewType('UserId', int)
NameError: name 'NewType' is not defined

----------------------------------------------------------------------
Ran 396 tests in 0.085s

FAILED (errors=1, skipped=1)
test test_typing failed

      Link: https://github.com/python/cpython/runs/4902363883?check_suite_focus=true

      But, later the suite runner tried to rerun it:

      0:09:12 load avg: 6.37 Re-running failed tests in verbose mode
0:09:12 load avg: 6.37 Re-running test_typing in verbose mode (matching: setUpClass)

1 re-run test:
    test_typing

1 test run no tests:
    test_typing

      And since nothing matched setUpClass - no tests were executed and the CI went green instead of red.

      What can we do?

      1. Only schedule real test_ item to be rerun, fail for everything else
      2. Convert setupClass failure into the whole class rerun
      3. Other options?

      I would like to work on this, when we will decide which way is best.

    2. sobolevn
      added
      3.9 (EOL)end of life
      3.10 (EOL)end of life
      3.11only security fixes
      testsTests in the Lib/test dir
      type-bugAn unexpected behavior, bug, or error
      on Jan 25, 2022

    3. Fidget-Spinner commented on Jan 25, 2022

      @Fidget-Spinner
      Fidget-Spinner
      on Jan 25, 2022
      Member

      I vote for option 2.

    4. Fidget-Spinner
      changed the title [-]Test suite skips failing tests[/-] [+]Test suite skips failing tests when setUp[Class] fails[/+] on Jan 25, 2022
    5. Fidget-Spinner
      changed the title [-]Test suite skips failing tests[/-] [+]Test suite skips failing tests when setUp[Class] fails[/+] on Jan 25, 2022
    6. ezio-melotti
      transferred this issue fromon Apr 10, 2022

    7. furkanonder commented on May 8, 2023

      @furkanonder
      furkanonder
      on May 8, 2023
      Contributor

      @sobolevn The issue seems to have been solved. We can close the issue.

    8. sobolevn
      closed this as completedon May 9, 2023
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.10 (EOL)end of life3.11only security fixes3.9 (EOL)end of lifetestsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions