test_stress_delivery_dependent() of test_signal randomly fails on AMD64 Debian root 3.6/3.x

@pitrou

BPO	30849
Nosy	@pitrou, @vstinner, @zware, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/pitrou'
closed_at = <Date 2021-05-26.22:47:39.082>
created_at = <Date 2017-07-04.15:16:51.076>
labels = ['tests']
title = 'test_stress_delivery_dependent() of test_signal randomly fails on AMD64 Debian root 3.6/3.x'
updated_at = <Date 2021-09-08.12:47:26.384>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-09-08.12:47:26.384>
actor = 'vstinner'
assignee = 'pitrou'
closed = True
closed_date = <Date 2021-05-26.22:47:39.082>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2017-07-04.15:16:51.076>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 30849
keywords = []
message_count = 15.0
messages = ['297674', '297732', '299948', '299985', '299986', '300017', '300181', '300182', '300550', '318313', '320891', '320900', '321019', '394492', '401380']
nosy_count = 4.0
nosy_names = ['pitrou', 'vstinner', 'zach.ware', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue30849'
versions = ['Python 3.6']

Antoine: I told you that such stess-test is going to require work to polish it. So here is another example of failure. Enjoy :-)

http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.6/builds/534/steps/test/logs/stdio

== CPython 3.6.2rc1+ (heads/3.6:580cd5c, Jul 5 2017, 00:47:42) [GCC 4.9.2]
== Linux-3.16.0-4-amd64-x86_64-with-debian-8.7 little-endian
...
== CPU count: 1
...
Run tests in parallel using 2 child processes
0:11:40 load avg: 1.29 [383/405/1] test_signal failed -- running: test_datetime (38 sec)
...
======================================================================
FAIL: test_stress_delivery_dependent (test.test_signal.StressTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/root/buildarea/3.6.angelico-debian-amd64/build/Lib/test/test_signal.py", line 1053, in test_stress_delivery_dependent
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 4602 != 10000 : Some signals were lost

======================================================================
FAIL: test_stress_delivery_simultaneous (test.test_signal.StressTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/root/buildarea/3.6.angelico-debian-amd64/build/Lib/test/test_signal.py", line 1086, in test_stress_delivery_simultaneous
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 8206 != 10000 : Some signals were lost

Another fail. The test pass when run again later.

http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.6/builds/538/steps/test/logs/stdio

======================================================================
FAIL: test_stress_delivery_dependent (test.test_signal.StressTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/root/buildarea/3.6.angelico-debian-amd64/build/Lib/test/test_signal.py", line 1053, in test_stress_delivery_dependent
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 4073 != 10000 : Some signals were lost

Hey Antoine, as expected, your test fails randomly on the CI :-(

Another failure:

http://buildbot.python.org/all/builders/x86%20Gentoo%20Non-Debug%20with%20X%203.x/builds/829/steps/test/logs/stdio

test_stress_delivery_dependent (test.test_signal.StressTest) ... Timeout (0:15:00)!
Thread 0xb74bd700 (most recent call first):
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_signal.py", line 973 in measure_itimer_resolution
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_signal.py", line 985 in decide_itimer_count
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_signal.py", line 1001 in test_stress_delivery_dependent
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/case.py", line 615 in run
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/case.py", line 663 in __call__
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/suite.py", line 122 in run
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/suite.py", line 84 in __call__
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/suite.py", line 122 in run
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/suite.py", line 84 in __call__
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/suite.py", line 122 in run
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/suite.py", line 84 in __call__
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/runner.py", line 176 in run
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/support/init.py", line 1896 in _run_suite
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/support/init.py", line 1940 in run_unittest
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/libregrtest/runtest.py", line 168 in test_runner
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/libregrtest/runtest.py", line 169 in runtest_inner
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/libregrtest/runtest.py", line 137 in runtest
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/libregrtest/main.py", line 290 in rerun_failed_tests
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/libregrtest/main.py", line 540 in _main
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/libregrtest/main.py", line 510 in main
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/libregrtest/main.py", line 585 in main
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/main.py", line 2 in <module>
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/runpy.py", line 85 in _run_code
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/runpy.py", line 193 in _run_module_as_main

Hmm. Perhaps someone can give me temporary shell access to one of those buildbots?

Antoine: "Hmm. Perhaps someone can give me temporary shell access to one of those buildbots?"

Contact directly the owners:

AMD64 Debian root 3.6: Chris Angelico <rosuav@gmail.com>
x86 Gentoo Non-Debug with X 3.x: Zachary Ware <zachary.ware+bb@gmail.com>

Hmm. Perhaps someone can give me temporary shell access to one of those buildbots?

Check your @python.org inbox.

Victor, Zachary, http://buildbot.python.org/all/builders/x86%20Gentoo%20Non-Debug%20with%20X%203.x/builds/830 corresponds to a very old changeset (more than one month old) before itimer() was fixed in 729780a.

Oh, Zach's Gentoo buildbot was offline for a long time. Maybe it ran tests
on old commits? What about the first messages of this issue on the other
buildbot? Maybe it's also outdated? I am fine with closing the issue if you
fixed a bug in the meanwhile. Don't worry, I will reopen it if it comes
back :-D

test_stress_delivery_dependent() failed on AMD64 Debian root 3.x too, it's not specific to 3.6:

http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.x/builds/1192/steps/test/logs/stdio

http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.x/builds/1191/steps/test/logs/stdio

Recent failure on AMD64 Debian root 3.7:

http://buildbot.python.org/all/#/builders/127/builds/361

0:03:33 load avg: 1.38 [140/415/1] test_signal failed

======================================================================
FAIL: test_stress_delivery_dependent (test.test_signal.StressTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/root/buildarea/3.7.angelico-debian-amd64/build/Lib/test/test_signal.py", line 1116, in test_stress_delivery_dependent
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 3586 != 10000 : Some signals were lost

======================================================================
FAIL: test_stress_delivery_simultaneous (test.test_signal.StressTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/root/buildarea/3.7.angelico-debian-amd64/build/Lib/test/test_signal.py", line 1149, in test_stress_delivery_simultaneous
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 8884 != 10000 : Some signals were lost

I can reproduce the failure on multiple OSes in VirtaulBox when run test_signal in parallel with test_regrtest:

./python -m test -j2 test_signal  test_regrtest test_regrtest test_regrtest
Run tests in parallel using 2 child processes
0:00:29 load avg: 4.24 [1/4] test_regrtest passed
0:00:58 load avg: 4.96 [2/4] test_regrtest passed -- running: test_signal (58 sec 728 ms)
0:01:00 load avg: 4.96 [3/4/1] test_signal failed
test test_signal failed -- Traceback (most recent call last):
  File "/home/serhiy/py/cpython3.7/Lib/test/test_signal.py", line 1116, in test_stress_delivery_dependent
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 5073 != 10000 : Some signals were lost

But can't reproduce on hardware.

After increasing the number of CPUs in the virtual machine the failure no longer reproduced. It is reproducible only with a single CPU.

After increasing the number of CPUs in the virtual machine the failure no longer reproduced. It is reproducible only with a single CPU.

The test just failed on AMD64 Debian root 3.x and according to test.pythoninfo, this machine has a single CPU:

os.cpu_count: 1

http://buildbot.python.org/all/#/builders/27/builds/1247

...
== CPU count: 1
== encodings: locale=UTF-8, FS=utf-8
Using random seed 6423496
Run tests in parallel using 2 child processes
0:00:00 load avg: 0.71 [ 1/417] test_genericclass passed
0:00:00 load avg: 0.71 [ 2/417] test_copyreg passed
0:00:03 load avg: 0.71 [ 3/417] test_os passed
stty: standard input: Inappropriate ioctl for device
running: test_tokenize (32 sec 627 ms), test_signal (30 sec 11 ms)
0:00:51 load avg: 1.32 [ 4/417] test_tokenize passed (50 sec 821 ms) -- running: test_signal (48 sec 506 ms)
0:00:54 load avg: 1.32 [ 5/417/1] test_signal failed
...
test_stress_delivery_dependent (test.test_signal.StressTest) ... detected median itimer() resolution: 0.000044 s.
FAIL
test_stress_delivery_simultaneous (test.test_signal.StressTest) ... detected median itimer() resolution: 0.000032 s.
ok
...
======================================================================
FAIL: test_stress_delivery_dependent (test.test_signal.StressTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_signal.py", line 1156, in test_stress_delivery_dependent
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 3617 != 10000 : Some signals were lost

I didn't see this failure recently, I close the issue.

We also got this error randomly on an internal s390x Red Hat build server:

======================================================================
FAIL: test_stress_delivery_simultaneous (test.test_signal.StressTest)
This test uses simultaneous signal handlers.
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/lib64/python3.9/test/test_signal.py", line 1245, in test_stress_delivery_simultaneous
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 5936 != 10000 : Some signals were lost

I reopen the issue, the test still fails randomly on AMD64 Debian root 3.x: https://buildbot.python.org/all/#/builders/345/builds/5878

See also issue gh-110083: test_signal: test_stress_modifying_handlers() crash with SIGSEGV on GHA macOS (macOS-12.7).

Now that tests are re-run in verbose mode when they fail, the test no longer makes a whole test suite to be marked as failed. I close the issue again, so nobody seems to be eagger to fix the test or the code.

Navigation Menu

Repository navigation

test_stress_delivery_dependent() of test_signal randomly fails on AMD64 Debian root 3.6/3.x #75032

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

test_stress_delivery_dependent() of test_signal randomly fails on AMD64 Debian root 3.6/3.x #75032

Description

Activity

vstinner commented on Jul 4, 2017

vstinner commented on Jul 5, 2017

vstinner commented on Aug 8, 2017

pitrou commented on Aug 9, 2017

vstinner commented on Aug 9, 2017

zware commented on Aug 9, 2017

pitrou commented on Aug 11, 2017

vstinner commented on Aug 11, 2017

vstinner commented on Aug 18, 2017

vstinner commented on May 31, 2018

serhiy-storchaka commented on Jul 2, 2018

serhiy-storchaka commented on Jul 2, 2018

vstinner commented on Jul 4, 2018

vstinner commented on May 26, 2021

vstinner commented on Sep 8, 2021

vstinner commented on Sep 20, 2023

vstinner commented on Sep 29, 2023

vstinner commented on Nov 8, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions