Skip to content

Some error paths do not set Exceptions in _testexternalinspection #130052

Closed
Closed
Some error paths do not set Exceptions in _testexternalinspection#130052
Labels
extension-modulesC modules in the Modules dirtype-bugAn unexpected behavior, bug, or error

Description

@sergey-miryanov
sergey-miryanov
opened on Feb 12, 2025

Bug report

Bug description:

Following @colesbury findings at #130035 (comment), I did a manual inspection of _testexternalinspection and found a few more cases where Exception should be set.

Will sent a PR soon.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

Activity

  1. sergey-miryanov
    added
    type-bugAn unexpected behavior, bug, or error
    on Feb 12, 2025
  2. encukou
    added
    extension-modulesC modules in the Modules dir
    on Feb 13, 2025
  3. sergey-miryanov
    added 3 commits that reference this issue on Feb 13, 2025

    Merge branch 'main' into pythongh-130052-test-external-inspection-set…

    9679c08

    Merge branch 'main' into pythongh-130052-test-external-inspection-set…

    4830892

    Merge branch 'main' into pythongh-130052-test-external-inspection-set…

    8bf3c22
  4. vstinner
    added a commit that references this issue on Feb 20, 2025

    gh-130052: Fix some exceptions on error paths in _testexternalinspect…

    69426fc
  5. colesbury
    marked Python 3.14.0a5: test_external_inspection fails on freethreading-debug build on s390x Linux #130035 as a duplicate of this issue on Feb 25, 2025
  6. sergey-miryanov
    closed this as completedon Mar 25, 2025

  7. befeleme commented on Apr 10, 2025

    @befeleme
    befeleme
    on Apr 10, 2025
    Contributor

    I see the same issue happening with a7 on aarch64 with freethreading-debug build:

    test_remote_stack_trace (test.test_external_inspection.TestGetStackTrace.test_remote_stack_trace) ... python: /builddir/build/BUILD/Python-3.14.0a7/Objects/call.c:342: _PyObject_Call: Assertion `!_PyErr_Occurred(tstate)' failed.
Fatal Python error: Aborted
<Cannot show all threads while the GIL is disabled>
Stack (most recent call first):
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/test_external_inspection.py", line 63 in test_remote_stack_trace
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/unittest/case.py", line 606 in _callTestMethod
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/unittest/case.py", line 660 in run
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/unittest/case.py", line 716 in __call__
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/unittest/suite.py", line 122 in run
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/unittest/suite.py", line 84 in __call__
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/unittest/suite.py", line 122 in run
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/unittest/suite.py", line 84 in __call__
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/unittest/runner.py", line 259 in run
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 84 in _run_suite
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 42 in run_unittest
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 162 in test_func
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 118 in regrtest_runner
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 165 in _load_run_test
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 210 in _runtest_env_changed_exc
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 319 in _runtest
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 348 in run_single_test
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/libregrtest/worker.py", line 92 in worker_process
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/libregrtest/worker.py", line 127 in main
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/libregrtest/worker.py", line 131 in <module>
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/runpy.py", line 88 in _run_code
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/runpy.py", line 198 in _run_module_as_main
Extension modules: _testinternalcapi, _testexternalinspection (total: 2)

  8. sergey-miryanov commented on Apr 10, 2025

    @sergey-miryanov
    sergey-miryanov
    on Apr 10, 2025
    ContributorAuthor

    @befeleme Can you rerun this test with following patch?

    diff --git a/Lib/test/test_external_inspection.py b/Lib/test/test_external_inspection.py
index 2ab48a4778b..b71f65fed32 100644
--- a/Lib/test/test_external_inspection.py
+++ b/Lib/test/test_external_inspection.py
@@ -60,6 +60,7 @@ def foo():
                 with open(fifo, "r") as fifo_file:
                     response = fifo_file.read()
                 self.assertEqual(response, "ready")
+                print('test_remote_stack', p, p.pid, type(p.pid))
                 stack_trace = get_stack_trace(p.pid)
             except PermissionError:
                 self.skipTest("Insufficient permissions to read the stack trace")

    I want to make sure that the get_stack_trace from _testexternalinspection is actually called (I just checked the code again and don't see anything suspicious).

  9. vstinner commented on Apr 15, 2025

    @vstinner
    vstinner
    on Apr 15, 2025
    Member

    On Fedora Rawhide x86-64, I get the following unrelated errors:

    ERROR: test_async_remote_stack_trace (test.test_external_inspection.TestGetStackTrace.test_async_remote_stack_trace) (task_factory_variant='asyncio.new_event_loop')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.14t/test/test_external_inspection.py", line 146, in test_async_remote_stack_trace
    stack_trace = get_async_stack_trace(p.pid)
OSError: [Errno 14] Bad address

======================================================================
ERROR: test_async_staggered_race_remote_stack_trace (test.test_external_inspection.TestGetStackTrace.test_async_staggered_race_remote_stack_trace)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.14t/test/test_external_inspection.py", line 334, in test_async_staggered_race_remote_stack_trace
    stack_trace = get_async_stack_trace(p.pid)
OSError: [Errno 14] Bad address

======================================================================
ERROR: test_self_trace (test.test_external_inspection.TestGetStackTrace.test_self_trace)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.14t/test/test_external_inspection.py", line 357, in test_self_trace
    stack_trace = get_stack_trace(os.getpid())
OSError: [Errno 14] Bad address

    On Fedora 41 AArch64 with Python built with ./configure --disable-gil CFLAGS="-O0" && make -j10, the test pass with success: ./python -m test test_external_inspection -v.

  10. vstinner commented on Apr 15, 2025

    @vstinner
    vstinner
    on Apr 15, 2025
    Member

    Ah, Fedora builds Python with LTO. I recall that test_external_inspection had issues with LTO (on Fedora). I re-ran tests with LTO on x86-64 and AArch64 and I still fail to reproduce the failure.

  11. befeleme commented on Apr 16, 2025

    @befeleme
    befeleme
    on Apr 16, 2025
    Contributor

    Testing again with the patch, a similar failure happens on s390x in another test: test_async_gather_remote_stack_trace.
    On aarch64 the build was this time successful. :(

    0:24:30 load avg: 2.73 [1/1/1] test_external_inspection worker non-zero exit code (Exit code -6 (SIGABRT))
Re-running test_external_inspection in verbose mode (matching: )
test_async_gather_remote_stack_trace (test.test_external_inspection.TestGetStackTrace.test_async_gather_remote_stack_trace) ... python: /builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Objects/call.c:342: _PyObject_Call: Assertion `!_PyErr_Occurred(tstate)' failed.
Fatal Python error: Aborted
<Cannot show all threads while the GIL is disabled>
Stack (most recent call first):
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/test_external_inspection.py", line 272 in test_async_gather_remote_stack_trace
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/unittest/case.py", line 606 in _callTestMethod
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/unittest/case.py", line 660 in run
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/unittest/case.py", line 716 in __call__
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/unittest/suite.py", line 122 in run
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/unittest/suite.py", line 84 in __call__
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/unittest/suite.py", line 122 in run
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/unittest/suite.py", line 84 in __call__
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/unittest/runner.py", line 259 in run
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 84 in _run_suite
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 42 in run_unittest
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 162 in test_func
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 118 in regrtest_runner
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 165 in _load_run_test
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 210 in _runtest_env_changed_exc
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 319 in _runtest
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 348 in run_single_test
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/libregrtest/worker.py", line 92 in worker_process
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/libregrtest/worker.py", line 127 in main
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/libregrtest/worker.py", line 131 in <module>
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/runpy.py", line 88 in _run_code
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/runpy.py", line 198 in _run_module_as_main
Extension modules: _testinternalcapi, _testexternalinspection (total: 2)
1 test failed again:
    test_external_inspection
== Tests result: FAILURE then FAILURE ==

  12. vstinner commented on Apr 16, 2025

    @vstinner
    vstinner
    on Apr 16, 2025
    Member

    Ok, I reproduced the error on s390x using the Fedora package python3.14-freethreading-debug.s390x.

    The error comes from search_map_for_section() at:

    402     exit:
403         if (close(fd) != 0) {
404             PyErr_SetFromErrno(PyExc_OSError);
405         }

    The problem is that the exit label is called with an exception set, and close(-1) tries to raise a new exception. In this case, the assertion fails.

    I'm working on a fix: #132594

  13. vstinner
    added 2 commits that reference this issue on Apr 16, 2025

    pythongh-130052: Fix search_map_for_section() error handling

    8c71d19

    gh-130052: Fix search_map_for_section() error handling (#132594)

    014c7f9
  14. miss-islington
    added a commit that references this issue on Apr 16, 2025

    pythongh-130052: Fix search_map_for_section() error handling (pythonG…

    0a8a5ee
  15. vstinner
    added a commit that references this issue on Apr 16, 2025

    [3.13] gh-130052: Fix search_map_for_section() error handling (GH-132594

    ff3f658
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    extension-modulesC modules in the Modules dirtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions