Skip to content

test_pdb crashes when run after test_sys_settrace #108976

Closed
Closed
test_pdb crashes when run after test_sys_settrace#108976
Labels
3.12only security fixes3.13only security fixesrelease-blockertestsTests in the Lib/test dirtype-crashA hard crash of the interpreter, possibly with a core dump

Description

@serhiy-storchaka
serhiy-storchaka
opened on Sep 6, 2023

Crash report

Just run two tests sequentially:

$ ./python -m test -v test_sys_settrace test_pdb
...
test_pdb_breakpoints_preserved_across_interactive_sessions (test.test_pdb)
Doctest: test.test_pdb.test_pdb_breakpoints_preserved_across_interactive_sessions ... python: Python/instrumentation.c:1293: _Py_call_instrumentation_instruction: Assertion `next_opcode != 0' failed.
Fatal Python error: Aborted

Current thread 0x00007fba31132740 (most recent call first):
  File "<string>", line 1 in <module>
  File "/home/serhiy/py/cpython/Lib/bdb.py", line 600 in run
  File "/home/serhiy/py/cpython/Lib/pdb.py", line 1962 in run
  File "<doctest test.test_pdb.test_pdb_breakpoints_preserved_across_interactive_sessions[3]>", line 7 in <module>
  File "/home/serhiy/py/cpython/Lib/doctest.py", line 1374 in __run
  File "/home/serhiy/py/cpython/Lib/doctest.py", line 1524 in run
  File "/home/serhiy/py/cpython/Lib/doctest.py", line 2244 in runTest
  File "/home/serhiy/py/cpython/Lib/unittest/case.py", line 589 in _callTestMethod
  File "/home/serhiy/py/cpython/Lib/unittest/case.py", line 634 in run
  File "/home/serhiy/py/cpython/Lib/unittest/case.py", line 690 in __call__
  File "/home/serhiy/py/cpython/Lib/unittest/suite.py", line 122 in run
  File "/home/serhiy/py/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/home/serhiy/py/cpython/Lib/unittest/suite.py", line 122 in run
  File "/home/serhiy/py/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/home/serhiy/py/cpython/Lib/unittest/suite.py", line 122 in run
  File "/home/serhiy/py/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/home/serhiy/py/cpython/Lib/unittest/runner.py", line 240 in run
  File "/home/serhiy/py/cpython/Lib/test/support/__init__.py", line 1142 in _run_suite
  File "/home/serhiy/py/cpython/Lib/test/support/__init__.py", line 1269 in run_unittest
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/runtest.py", line 393 in run_unittest
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/runtest.py", line 447 in test_func
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/runtest.py", line 407 in regrtest_runner
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/runtest.py", line 451 in _load_run_test
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/runtest.py", line 491 in _runtest_env_changed_exc
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/runtest.py", line 348 in _runtest
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/runtest.py", line 375 in runtest
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/main.py", line 462 in run_test
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/main.py", line 495 in run_tests_sequentially
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/main.py", line 632 in run_tests
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/main.py", line 855 in action_run_tests
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/main.py", line 883 in _main
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/main.py", line 822 in main
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/main.py", line 891 in main
  File "/home/serhiy/py/cpython/Lib/test/__main__.py", line 2 in <module>
  File "/home/serhiy/py/cpython/Lib/runpy.py", line 88 in _run_code
  File "/home/serhiy/py/cpython/Lib/runpy.py", line 198 in _run_module_as_main

It is only reproducible in 3.12+.

Linked PRs

Activity

  1. serhiy-storchaka
    added
    testsTests in the Lib/test dir
    type-crashA hard crash of the interpreter, possibly with a core dump
    3.12only security fixes
    3.13only security fixes
    on Sep 6, 2023

  2. serhiy-storchaka commented on Sep 6, 2023

    @serhiy-storchaka
    serhiy-storchaka
    on Sep 6, 2023
    MemberAuthor

    test_pdb only crashes after RaisingTraceFuncTestCase.test_line_event_raises_before_opcode_event or after any of TraceOpcodesTestCase cases from test_sys_settrace.

    $ ./python -m test -v test_sys_settrace test_pdb -m test_pdb_display_command -m test.test_sys_settrace.RaisingTraceFuncTestCase.test_line_event_raises_before_opcode_event
$ ./python -m test -v test_sys_settrace test_pdb -m test_pdb_display_command -m test.test_sys_settrace.TraceOpcodesTestCase.test_01_basic

    These tests also occur in #108982, so this all may be symptoms of the same bug.

  3. gaogaotiantian commented on Sep 6, 2023

    @gaogaotiantian
    gaogaotiantian
    on Sep 6, 2023
    Member

    I'm able to reduce the MRE:

    import sys

sys._getframe().f_trace_opcodes = True

breakpoint()
a = 1

    It will bring up the pdb console, and you can just c to trigger the SegFault.

  4. gaogaotiantian commented on Sep 7, 2023

    @gaogaotiantian
    gaogaotiantian
    on Sep 7, 2023
    Member

    The crash is a bit tricky, and affects more than sys.settrace - sys.monitoring also suffers from it.

    The fundamental reason was when INSTRUCTION and LINE events are both set, and INSTRUMENTATION is turned off in LINE event callback, the system got confused. In INSTRUMENTED_LINE, it will try to get the next opcode and would be fed a wrong one - INSTRUMENTED_INSTRUCTION by the current design. However, the instrumentation is already stripped, which would eventually cause issues that lead to a SegFault.

    I have a fix which simply checks if this case happens when opcode is requested from INSTRUMENTED_LINE and use the original opcode for the instruction.

    This might not be the best way to fix it, but it's clean and simple. If @markshannon has a better idea, I can do something different.

  5. markshannon
    added
    release-blocker
    on Sep 8, 2023
  6. ezio-melotti
    added this to Release and Deferred blockers 🚫on Sep 8, 2023
  7. github-project-automation
    moved this to Todo in Release and Deferred blockers 🚫on Sep 8, 2023
  8. markshannon
    added 2 commits that reference this issue on Sep 11, 2023

    GH-108976. Keep monitoring data structures valid during de-optimizati…

    4a69301

    pythonGH-108976. Keep monitoring data structures valid during de-opti…

    646b7ce
  9. Yhg1s
    added a commit that references this issue on Sep 12, 2023

    [3.12] GH-108976. Keep monitoring data structures valid during de-opt…

    3efe7bc

  10. markshannon commented on Sep 12, 2023

    @markshannon
    markshannon
    on Sep 12, 2023
    Member

    This is fixed now, both in main and 3.12

  11. markshannon
    closed this as completedon Sep 12, 2023
  12. github-project-automation
    moved this from Todo to Done in Release and Deferred blockers 🚫on Sep 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.12only security fixes3.13only security fixesrelease-blockertestsTests in the Lib/test dirtype-crashA hard crash of the interpreter, possibly with a core dump

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions