Skip to content

test_gdb failed with 15 min timeout on PPC64LE RHEL8 3.x (LTO + pydebug) #109972

Closed
Closed
test_gdb failed with 15 min timeout on PPC64LE RHEL8 3.x (LTO + pydebug)#109972
Labels
testsTests in the Lib/test dir

Description

@vstinner
vstinner
opened on Sep 27, 2023

PPC64LE RHEL8 3.x:

0:19:19 load avg: 1.37 [463/463/1] test_gdb process crashed (Exit code 1)
Timeout (0:15:00)!
Thread 0x00007fffae664d70 (most recent call first):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/selectors.py", line 398 in select
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/subprocess.py", line 2108 in _communicate
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/subprocess.py", line 1209 in communicate
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 107 in run_gdb
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 224 in get_stack_trace
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 949 in test_pycfunction
  (...)

(...)

0:19:19 load avg: 1.37 Re-running 1 failed tests in verbose mode in subprocesses
0:19:19 load avg: 1.37 Run 1 test in parallel using 1 worker process (timeout: 15 min, worker timeout: 20 min)
0:19:49 load avg: 1.22 running (1): test_gdb (30.0 sec)
0:20:19 load avg: 1.19 running (1): test_gdb (1 min)
0:20:49 load avg: 1.11 running (1): test_gdb (1 min 30 sec)
(...)
0:33:50 load avg: 1.09 running (1): test_gdb (14 min 30 sec)
0:34:20 load avg: 1.14 running (1): test_gdb (15 min)
0:34:20 load avg: 1.14 [1/1/1] test_gdb process crashed (Exit code 1)
Re-running test_gdb in verbose mode
test_NULL_ob_type (test.test_gdb.PrettyPrintTests.test_NULL_ob_type)
Ensure that a PyObject* with NULL ob_type is handled gracefully ... ok
(...)
test_pycfunction (test.test_gdb.PyBtTests.test_pycfunction)
Verify that "py-bt" displays invocations of PyCFunction instances ... Timeout (0:15:00)!
Thread 0x00007fffa14b4d70 (most recent call first):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/selectors.py", line 398 in select
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/subprocess.py", line 2108 in _communicate
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/subprocess.py", line 1209 in communicate
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 107 in run_gdb
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 224 in get_stack_trace
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 949 in test_pycfunction
  (...)

build: https://buildbot.python.org/all/#/builders/559/builds/3951

Linked PRs

Activity

  1. vstinner
    added
    testsTests in the Lib/test dir
    on Sep 27, 2023
  2. vstinner
    changed the title [-]test_gdb failed with 15 min timeout on AMD64 RHEL8 Refleaks 3.x[/-] [+]test_gdb failed with 15 min timeout on PPC64LE RHEL8 3.x[/+] on Sep 27, 2023

  3. vstinner commented on Sep 27, 2023

    @vstinner
    vstinner
    on Sep 27, 2023
    MemberAuthor

    Ah, it's a LTO build.

    configure + make:

    ../configure --prefix '$(PWD)/target' --with-pydebug --with-platlibdir=lib64 --enable-ipv6 --enable-shared --with-computed-gotos=yes --with-dbmliborder=gdbm:ndbm:bdb --enable-loadable-sqlite-extensions --with-ssl-default-suites=openssl --without-static-libpython --with-lto
(...)
gcc -pthread -c -fno-strict-overflow -Wsign-compare -g -Og -Wall    -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I../Include/internal -IObjects -IInclude -IPython -I. -I../Include   -fPIC -DPy_BUILD_CORE -o Programs/python.o ../Programs/python.c
(...)
gcc -pthread   -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g  -Xlinker -export-dynamic -o python Programs/python.o -L. -lpython3.13d -lpthread -ldl  -lutil                        -lm

    test.pythoninfo:

    libregrtests.build_info: debug shared LTO

CC.version: gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-18)

sysconfig[PY_CFLAGS]: -fno-strict-overflow -Wsign-compare -g -Og -Wall
sysconfig[PY_CFLAGS_NODIST]: -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I../Include/internal
sysconfig[PY_STDMODULE_CFLAGS]: -fno-strict-overflow -Wsign-compare -g -Og -Wall -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I../Include/internal -IObjects -IInclude -IPython -I. -I../Include -fPIC

sysconfig[PY_LDFLAGS_NODIST]: -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g
sysconfig[PY_CORE_LDFLAGS]: -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g

  4. vstinner commented on Sep 27, 2023

    @vstinner
    vstinner
    on Sep 27, 2023
    MemberAuthor

    On this buildbot, if Python is built with ./configure --with-pydebug and gcc -Og, test_gdb completes in 4 min 30 sec and looks reliable:

    [vstinner@rh-power-vm13 cpython]$ ./python -m test test_gdb -u all -v
(...)
Total duration: 4 min 30 sec
Total tests: run=46 skipped=1
Total test files: run=1/1
Result: SUCCESS
  5. vstinner
    changed the title [-]test_gdb failed with 15 min timeout on PPC64LE RHEL8 3.x[/-] [+]test_gdb failed with 15 min timeout on PPC64LE RHEL8 3.x (LTO + pydebug)[/+] on Sep 27, 2023

  6. vstinner commented on Sep 27, 2023

    @vstinner
    vstinner
    on Sep 27, 2023
    MemberAuthor

    When Python is built with ./configure --with-pydebug --with-lto && make -j10, well, test_gdb is ... way slower: 19 min 45 sec.

    test_gdb passed in 19 min 45 sec

== Tests result: SUCCESS ==

1 test OK.

Total duration: 19 min 45 sec
Total tests: run=46 skipped=1
Total test files: run=1/1
Result: SUCCESS

  7. vstinner commented on Sep 27, 2023

    @vstinner
    vstinner
    on Sep 27, 2023
    MemberAuthor

    Python built with ./configure --with-pydebug CFLAGS="-O0" and gcc -O0:

    test_gdb passed in 4 min 12 sec

== Tests result: SUCCESS ==

1 test OK.

Total duration: 4 min 12 sec
Total tests: run=46 skipped=1
Total test files: run=1/1
Result: SUCCESS

  8. vstinner commented on Sep 27, 2023

    @vstinner
    vstinner
    on Sep 27, 2023
    MemberAuthor

    The test runs 130 gdb commands in total. Average:

    • gcc -O0: 1.9 sec / command
    • gcc -Og: 2.1 sec / command
    • gcc -Og -flto: 9.1 sec / command

  9. vstinner commented on Sep 27, 2023

    @vstinner
    vstinner
    on Sep 27, 2023
    MemberAuthor

    Number of gdb commands per test case:

    PrettyPrintTests: 62

PyBtTests: 58

PyListTests: 6
PyLocalsTests: 5
PyPrintTests: 7
StackNavigationTests: 7
  10. vstinner
    added a commit that references this issue on Sep 27, 2023

    pythongh-109972: Split test_gdb into a package of 3 tests

    c8fe6f6

  11. vstinner commented on Sep 27, 2023

    @vstinner
    vstinner
    on Sep 27, 2023
    MemberAuthor

    I wrote PR #109977 to split test_gdb into a package of 3 tests so these 3 tests can be run in parallel.

  12. vstinner
    added 3 commits that reference this issue on Sep 28, 2023

    pythongh-109972: Split test_gdb into a package of 3 tests

    754d551

    pythongh-109972: Split test_gdb into a package of 3 tests

    534bdad

    pythongh-109972: Split test_gdb into a package of 3 tests

    f695d74

  13. vstinner commented on Sep 28, 2023

    @vstinner
    vstinner
    on Sep 28, 2023
    MemberAuthor

    The test runs 130 gdb commands in total. Average: gcc -Og -flto: 9.1 sec / command

    Well, gdb takes 3.9 seconds just to load symbols from the Python ELF binary. Time measured with (before, I removed manually python-gdb.py):

    time gdb -batch --init-eval-command "add-auto-load-safe-path $PWD" -args ./python

  14. 8 remaining items

  15. vstinner
    added 2 commits that reference this issue on Sep 28, 2023

    pythongh-109972: Enhance test_gdb

    18029d6

    gh-109972: Enhance test_gdb (#110026)

    757cbd4
  16. vstinner
    closed this as completedon Sep 28, 2023
  17. csm10495
    added 2 commits that reference this issue on Sep 29, 2023

    pythongh-109972: Split test_gdb.py into test_gdb package (python#109977)

    c476688

    pythongh-109972: Enhance test_gdb (python#110026)

    30f6589
  18. vstinner
    added 6 commits that reference this issue on Oct 4, 2023

    pythongh-109972: Split test_gdb.py into test_gdb package (python#109977)

    3ee646f

    [3.12] gh-109972: Split test_gdb.py into test_gdb package (#109977) (#…

    e7a61d3

    [3.12] pythongh-109972: Split test_gdb.py into test_gdb package (pyth…

    8dbf9c0

    [3.11] gh-109972: Split test_gdb.py into test_gdb package (#109977) (… (

    6c98c73

    pythongh-109972: Enhance test_gdb (python#110026)

    b660dcb

    [3.12] gh-109972: Enhance test_gdb (#110026) (#110351)

    bbce8bd
  19. miss-islington
    added a commit that references this issue on Oct 4, 2023

    [3.12] pythongh-109972: Enhance test_gdb (pythonGH-110026) (pythonGH-…

    50ec4bb
  20. vstinner
    added a commit that references this issue on Oct 4, 2023

    [3.11] [3.12] gh-109972: Enhance test_gdb (GH-110026) (GH-110351) (#1…

    aa8d3db
  21. Glyphack
    added 2 commits that reference this issue on Sep 2, 2024

    pythongh-109972: Split test_gdb.py into test_gdb package (python#109977)

    a359ff6

    pythongh-109972: Enhance test_gdb (python#110026)

    93af2ba
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testsTests in the Lib/test dir

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions