Skip to content

Test suite crashes when run single process --with-pydebug enabled #121832

Closed
#122340
Closed
Test suite crashes when run single process --with-pydebug enabled#121832
#122340
Assignees
ericsnowcurrentlymarkshannon
Labels
3.13only security fixes3.14bugs and security fixestype-crashA hard crash of the interpreter, possibly with a core dump

Description

@freakboy3742
freakboy3742
opened on Jul 16, 2024

Crash report

What happened?

The make testios testing target crashes when compiled --with-pydebug enabled (as is done in CI).

The error is a SIGABRT, raised during test_types; however, running test_types by itself isn't enough to reproduce the problem.

The error is raised on L8476 of typeobject.c:

    assert(self->tp_version_tag != 0);

The same failure can be manufactured on macOS with M1 hardware, as long as the test suite is executed as a single process (i.e. python -m test, not the multi-process option enabled by make test).

Full error trace on iOS:
stacktrace.txt

The same problem doesn't appear to occur when debug is not enabled.

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS, iOS

Output from running 'python -VV' on the command line:

3.14.0a0; verified on 2bac2b86, but possibly present back to dc03ce7

Linked PRs

Activity

  1. freakboy3742
    added
    type-crashA hard crash of the interpreter, possibly with a core dump
    3.13only security fixes
    OS-ios
    3.14bugs and security fixes
    on Jul 16, 2024

  2. freakboy3742 commented on Jul 16, 2024

    @freakboy3742
    freakboy3742
    on Jul 16, 2024
    ContributorAuthor

    I've narrowed the cause of the crash to #121602 (which was backported to 3.13 as #121632, so it explains the failures on both branches).

  3. freakboy3742 commented on Jul 16, 2024

    @freakboy3742
    freakboy3742
    on Jul 16, 2024
    ContributorAuthor

    I have confirmed the crash is occurring in the SubInterpreterTest added as part of #121602. If that test is commented out, the test suite passes without error.

  4. ericsnowcurrently commented on Jul 16, 2024

    @ericsnowcurrently
    ericsnowcurrently
    on Jul 16, 2024
    Member

    FYI, I'm pretty sure gh-121636 will fix this.

    The gist is: the change in gh-121602 runtime initialization makes a copy of the (mostly) unmodified PyTypeObject struct for each builtin static type. During finalization we restore that copy, including the tp_version_tag. Clearly a part of that is causing things to get in an inconsistent state on iOS. The fix in gh-121636 restricts the copy/restore to only the "tp" slots (function pointers), restoring the pre-121602 status quo for the rest, including tp_version_tag.

    Is there a way you could verify gh-121636 fixes the iOS build?

    All that said, I'm definitely curious why this would be manifesting only on iOS?

  5. freakboy3742 commented on Jul 16, 2024

    @freakboy3742
    freakboy3742
    on Jul 16, 2024
    ContributorAuthor

    FYI, I'm pretty sure gh-121636 will fix this.
    ...
    Is there a way you could verify gh-121636 fixes the iOS build?

    The iOS buildbot should be sufficient to verify any fix; from the look of it, the current state of #121636 doesn't fix it. The test failure isn't very helpful - it's a hard crash of the emulator, which the test report doesn't show very nicely - but with the test_types SubinterpreterTest commented out, the suite completes and passes on iOS. I've verified the same tests locally, and I get the same failure on #121636.

    All that said, I'm definitely curious why this would be manifesting only on iOS?

    You and me both :-)

    My best guess right now it's not iOS specific, but a function of how the test suite is executed on iOS. iOS is forced to run the entire test suite sequentially in a single process. This isn't (AFAIK) the default mode of operation on any other runner; so it's possible the specific sequence of test execution is causing an odd state to emerge. One of the debugging tasks I'm going to look into today is to try and reproduce this on macOS (the other is to try and find a subset of tests that causes the failure, rather than needing to get to test_types alphabetically... after 20 minutes)

  6. ericsnowcurrently commented on Jul 16, 2024

    @ericsnowcurrently
    ericsnowcurrently
    on Jul 16, 2024
    Member

    Looks like gh-121636 isn't sufficient. I'm going to try a different approach.

  7. ericsnowcurrently commented on Jul 16, 2024

    @ericsnowcurrently
    ericsnowcurrently
    on Jul 16, 2024
    Member

    See gh-121882.

  8. freakboy3742 commented on Jul 16, 2024

    @freakboy3742
    freakboy3742
    on Jul 16, 2024
    ContributorAuthor

    Dang it... looks like gh-121882 doesn't fix the problem either... 😢

  9. freakboy3742 commented on Jul 17, 2024

    @freakboy3742
    freakboy3742
    on Jul 17, 2024
    ContributorAuthor

    Confirming: the test_types test doesn't fail on macOS when the test suite is executed in single process mode, so the problem isn't purely about execution order. Now trying to bisect the test suite to find a smaller subset of tests that fail.

  10. freakboy3742 commented on Jul 17, 2024

    @freakboy3742
    freakboy3742
    on Jul 17, 2024
    ContributorAuthor

    Scratch that - it does crash on macOS - just you just have to remember to enable --with-pydebug.

    @ericsnowcurrently To reproduce:

    ./configure --enable-universalsdk="`xcrun --show-sdk-path`" --with-universal-archs=universal2 --with-pydebug
make -j8
./python.exe -m test

    (Although I also had to patch test_buildin.py::PtyTests.test_input_tty to avoid a lockup due to #116402).

    Key details: Clean macOS build, with pydebug enabled; running as a single process (not multiprocess as make test will do), on an M1 MacBook Pro.

    Test failure:

    ...
0:17:13 load avg: 14.30 [424/478/1] test_types
Assertion failed: (self->tp_version_tag != 0), function init_static_type, file typeobject.c, line 8476.
Fatal Python error: Aborted

Current thread 0x00000001fb35fac0 (most recent call first):
  <no Python frame>

Extension modules: _testinternalcapi, _testcapi, _testlimitedcapi, _testbuffer, _testclinic, _testclinic_limited, _ctypes_test, xxsubtype, _testexternalinspection, _testsinglephase_basic_wrapper, _testsinglephase_basic_copy, _testsinglephase_with_reinit, _testsinglephase_with_state, _test_non_isolated, _test_shared_gil_only (total: 15)
zsh: abort      ./python.exe -m test

    Now trying to narrow down a subset of these tests that will fail.

  11. freakboy3742
    changed the title [-]iOS testbed crashes when `--with-pydebug` enabled[/-] [+]Test suite crashes when run single process `--with-pydebug` enabled[/+] on Jul 17, 2024
  12. freakboy3742
    removed
    OS-ios
    on Jul 17, 2024

  13. 16 remaining items

  14. miss-islington
    added a commit that references this issue on Jul 23, 2024

    pythongh-121832: Skip subinterpreter static type check on iOS to rest…

    9d9ea48

  15. markshannon commented on Jul 23, 2024

    @markshannon
    markshannon
    on Jul 23, 2024
    Member

    @ericsnowcurrently can you make sure to revert #122150 when this is fixed?

  16. markshannon
    assigned
    ericsnowcurrently
    on Jul 23, 2024
  17. freakboy3742
    added a commit that references this issue on Jul 23, 2024

    [3.13] gh-121832: Skip subinterpreter static type check on iOS to res…

    81f9339
  18. markshannon
    added a commit that references this issue on Jul 24, 2024

    GH-121832: Assert that the version number of static builtin types is …

    e55b05f
  19. nohlson
    added 4 commits that reference this issue on Jul 24, 2024

    pythongh-121832: Skip subinterpreter static type check on iOS to rest…

    4a6def1

    pythonGH-121832: Assert that the version number of static builtin typ…

    838bcfb

    pythongh-121832: Skip subinterpreter static type check on iOS to rest…

    263866c

    pythonGH-121832: Assert that the version number of static builtin typ…

    d503137
  20. miss-islington
    added a commit that references this issue on Jul 25, 2024

    pythonGH-121832: Assert that the version number of static builtin typ…

    b9f3e7c
  21. ericsnowcurrently
    added a commit that references this issue on Jul 25, 2024

    [3.13] GH-121832: Assert that the version number of static builtin ty…

    aca41cf
  22. freakboy3742
    added a commit that references this issue on Jul 27, 2024

    gh-121832: Revert test skip introduced by #122150. (#122340)

    863a92f
  23. freakboy3742
    closed this as completedin #122340on Jul 27, 2024
  24. freakboy3742
    added 2 commits that reference this issue on Jul 27, 2024

    [3.13] pythongh-121832: Revert test skip introduced by pythonGH-122150.…

    f6e57db

    [3.13] gh-121832: Revert test skip introduced by GH-122150. (GH-122340)…

    7b35c50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

3.13only security fixes3.14bugs and security fixestype-crashA hard crash of the interpreter, possibly with a core dump

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions