Skip to content

test_idle test_configdialog.HighPageTest failing on macOS #134770

Open
Open
test_idle test_configdialog.HighPageTest failing on macOS#134770
Labels
OS-mactopic-IDLEtype-bugAn unexpected behavior, bug, or error

Description

@emmatyping
emmatyping
opened on May 27, 2025

Bug report

Saw these failure on a normal macOS 13 runner:

======================================================================
ERROR: test_set_theme_type (idlelib.idle_test.test_configdialog.HighPageTest.test_set_theme_type)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/cpython/cpython/Lib/idlelib/idle_test/test_configdialog.py", line 295, in setUp
    d.set_highlight_target.called = 0
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'method' object has no attribute 'called' and no __dict__ for setting new attributes

======================================================================
ERROR: test_theme_source (idlelib.idle_test.test_configdialog.HighPageTest.test_theme_source)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/cpython/cpython/Lib/idlelib/idle_test/test_configdialog.py", line 295, in setUp
    d.set_highlight_target.called = 0
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'method' object has no attribute 'called' and no __dict__ for setting new attributes

======================================================================
ERROR: tearDownClass (idlelib.idle_test.test_configdialog.HighPageTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/cpython/cpython/Lib/idlelib/idle_test/test_configdialog.py", line 284, in tearDownClass
    del d.set_highlight_target, d.set_color_sample
        ^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'HighPage' object has no attribute 'set_highlight_target'. Did you mean: 'highlight_target'?

======================================================================
FAIL: test_delete_custom (idlelib.idle_test.test_configdialog.HighPageTest.test_delete_custom)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/cpython/cpython/Lib/idlelib/idle_test/test_configdialog.py", line 748, in test_delete_custom
    eq(d.custom_theme_on.state(), ())
    ~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Tuples differ: ('background',) != ()

First tuple contains 1 additional elements.
First extra element 0:
'background'

- ('background',)
+ ()

<snip>

Example failed run:
https://github.com/python/cpython/actions/runs/15265517831/job/42930405567

Activity

  1. emmatyping
    added
    type-bugAn unexpected behavior, bug, or error
    stdlibStandard Library Python modules in the Lib/ directory
    topic-IDLE
    on May 27, 2025

  2. hugovk commented on May 27, 2025

    @hugovk
    hugovk
    on May 27, 2025
    Member

    I restarted it and it passed this time: https://github.com/python/cpython/actions/runs/15265517831

  3. emmatyping commented on May 27, 2025

    @emmatyping
    emmatyping
    on May 27, 2025
    MemberAuthor

    Thanks! That's even more confusing though :(

  4. StanFromIreland commented on May 27, 2025

    @StanFromIreland
    StanFromIreland
    on May 27, 2025
    Member

    I have seen this pop up several times over a few months.

    cc expert @terryjreedy

  5. terryjreedy
    added this to IDLE Issueson May 27, 2025
  6. terryjreedy
    changed the title [-]test_idle failing on macOS[/-] [+]test_idle test_configdialog.HighPageTest failing on macOS[/+] on May 27, 2025
  7. terryjreedy
    added
    OS-mac
    on May 27, 2025

  8. terryjreedy commented on May 27, 2025

    @terryjreedy
    terryjreedy
    on May 27, 2025
    Member

    Unittest separately reports Errors (unanticipated exceptions) and Failures (assertXXX not true). For test_configdialog.HighTest, there were 4 Failures and 3 Errors (in that order, the opposite of the report order). Including the 1 fail reported above, the failures are summarized as follows, in the order the tests methods are run.

    1. test_delete_custom 748: Tuples differ: ('background',) != ()
    2. test_load_theme_cfg 310: Tuples differ: ('disabled', 'background') != ('disabled',)
    3. test_on_new_color_set 571: '#de0000' != '#3f7cae'
    4. test_set_highlight_target 644: Tuples differ: ('background',) != ('disabled', 'selected')

    I think 1. and 3. are some sort of very rare timing issues, but I am not sure how to fix. Perhaps such occur more often macOS 13 than macOS 10, the version when we wrote these tests. Beside the main issue, 1. adds custom mocks which are not deleted when the assert fails. I will fix this possible omission, in all IDLE test files, in a separate issue. I think this cleanup failure is the cause of 2. and maybe 4.

    Failure 4. must have caused the 3 AttributeErrors by removing but not restoring the mock with the 'called' attribute. In the first,
    d is an instance of idlelib.configdialog.HighPage.
    set_highlight_target is an instance of idlelib.idle_test.mock_idle.Func mocking the method of the same name.
    called is an attribute of the mock set in __init__ and incremented in __called and reset in setUp.

    This is the first report of such errors that I have seen in the last few years. macOS 13 seems flakier than some earlier versions. Are recent failures only on macOS? To fix the primary failures, I need more reports to to get a better idea of what fails. For instance, are some asserts failing multiple times, or are they more random. Links to failure runs, like the one above, are enough for me to make summaries like in this comment. (Easier after cascade failures are removed.)

  9. ned-deily commented on May 28, 2025

    @ned-deily
    ned-deily
    on May 28, 2025
    Member

    FWIW, I have never seen this particular kind of behavior of test_idle on any version of macOS when doing installer sanity tests but those tests always run the test suite in the same order and run GUI-related tests, including test_idle, as separate test runs so there would be no interference from other tests or from random ordering of tests.

  10. StanFromIreland commented on May 28, 2025

    @StanFromIreland
    StanFromIreland
    on May 28, 2025
    Member

    See this run from March (I hope the logs don't disappear), though it seems to have some slightly different fails.

  11. sobolevn commented on Jun 25, 2025

    @sobolevn
    sobolevn
    on Jun 25, 2025
    Member

    Similar error happened today: https://github.com/python/cpython/actions/runs/15870466272/job/44745627978?pr=135840

     ======================================================================
ERROR: test_source_selected (idlelib.idle_test.test_configdialog.HelpSourceTest.test_source_selected)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/cpython/cpython/Lib/idlelib/idle_test/test_configdialog.py", line 1355, in test_source_selected
    fr.set.called = fr.upc.called = 0
    ^^^^^^^^^^^^^
AttributeError: 'method' object has no attribute 'called' and no __dict__ for setting new attributes

======================================================================
ERROR: tearDownClass (idlelib.idle_test.test_configdialog.HelpSourceTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/cpython/cpython/Lib/idlelib/idle_test/test_configdialog.py", line 1322, in tearDownClass
    del frame.set, frame.set_add_delete_state
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'HelpFrame' object has no attribute 'set_add_delete_state'

======================================================================
ERROR: tearDownClass (idlelib.idle_test.test_configdialog.KeysPageTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/cpython/cpython/Lib/idlelib/idle_test/test_configdialog.py", line 788, in tearDownClass
    del page.set_keys_type, page.load_keys_list
        ^^^^^^^^^^^^^^^^^^
AttributeError: 'KeysPage' object has no attribute 'set_keys_type'

======================================================================
FAIL: test_set_add_delete_state (idlelib.idle_test.test_configdialog.HelpSourceTest.test_set_add_delete_state)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/cpython/cpython/Lib/idlelib/idle_test/test_configdialog.py", line 1374, in test_set_add_delete_state
    eq(fr.button_helplist_edit.state(), ('disabled',))
    ~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Tuples differ: ('disabled', 'background') != ('disabled',)

First tuple contains 1 additional elements.
First extra element 1:
'background'

- ('disabled', 'background')
+ ('disabled',)

======================================================================
FAIL: test_on_bindingslist_select (idlelib.idle_test.test_configdialog.KeysPageTest.test_on_bindingslist_select)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/cpython/cpython/Lib/idlelib/idle_test/test_configdialog.py", line 1046, in test_on_bindingslist_select
    self.assertEqual(d.button_new_keys.state(), ())
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Tuples differ: ('background',) != ()

First tuple contains 1 additional elements.
First extra element 0:
'background'

- ('background',)
+ ()

======================================================================
FAIL: test_set_keys_type (idlelib.idle_test.test_configdialog.KeysPageTest.test_set_keys_type)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/cpython/cpython/Lib/idlelib/idle_test/test_configdialog.py", line 940, in test_set_keys_type
    eq(d.button_delete_custom_keys.state(), ('disabled',))
    ~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Tuples differ: ('disabled', 'background') != ('disabled',)

First tuple contains 1 additional elements.
First extra element 1:
'background'

- ('disabled', 'background')
+ ('disabled',)

----------------------------------------------------------------------
Ran 623 tests in 26.533s

FAILED (failures=3, errors=3, skipped=4)
test test_idle failed

  12. emmatyping commented on Aug 7, 2025

    @emmatyping
    emmatyping
    on Aug 7, 2025
    MemberAuthor

    Another instance here: https://github.com/python/cpython/actions/runs/16791013574/job/47552476050?pr=137496

  13. terryjreedy
    removed
    stdlibStandard Library Python modules in the Lib/ directory
    on Jan 15, 2026

  14. terryjreedy commented on Feb 2, 2026

    @terryjreedy
    terryjreedy
    on Feb 2, 2026
    Member

    Old test logs disappear, so the actual error reports of variable occasional failures need to be copied here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    OS-mactopic-IDLEtype-bugAn unexpected behavior, bug, or error

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions