Skip to content

test__xxsubinterpreters: test_already_running() crash randomly on Python built with TraceRefs: invalid object chain #107080

Closed
#107751
Closed
test__xxsubinterpreters: test_already_running() crash randomly on Python built with TraceRefs: invalid object chain#107080
#107751
Assignees
ericsnowcurrently
Labels
3.12only security fixes3.13only security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)topic-subinterpreterstype-bugAn unexpected behavior, bug, or error

Description

@vstinner
vstinner
opened on Jul 22, 2023

AMD64 Arch Linux TraceRefs 3.x: https://buildbot.python.org/all/#/builders/484/builds/3721

Assertion error: Objects/object.c:2235: _Py_ForgetReference: Assertion failed: invalid object chain.

$ ./python -m test -j2 test__xxsubinterpreters test__xxsubinterpreters -m test_already_running -v
(...)
== Python build: debug TraceRefs
(...)
== CPU count: 12
(...)
0:00:00 load avg: 1.19 Run tests in parallel using 2 child processes
0:00:00 load avg: 1.19 [1/2] test__xxsubinterpreters passed
test_already_running (test.test__xxsubinterpreters.RunStringTests.test_already_running) ... ok
(...)

0:00:00 load avg: 1.19 [2/2/1] test__xxsubinterpreters crashed (Exit code -6)
test_already_running (test.test__xxsubinterpreters.RunStringTests.test_already_running) ...

Objects/object.c:2235: _Py_ForgetReference: Assertion failed: invalid object chain
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7f01d0be95a0
object refcount : 0
object type     : 0xa1dc00
object type name: str
object repr     : <refcnt 0 at 0x7f01d0be95a0>

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x00007f01df4a1740 (most recent call first):
  <no Python frame>
(...)

Python built with:

git clean -fdx
./configure --with-pydebug --with-trace-refs CFLAGS="-O0"
make -j14

Linked PRs

Activity

  1. vstinner
    added
    type-bugAn unexpected behavior, bug, or error
    on Jul 22, 2023
  2. vstinner
    changed the title [-]test__xxsubinterpreters: test_already_running() crash randomly on Python built with TraceRefs[/-] [+]test__xxsubinterpreters: test_already_running() crash randomly on Python built with TraceRefs: invalid object chain[/+] on Jul 22, 2023

  3. vstinner commented on Jul 22, 2023

    @vstinner
    vstinner
    on Jul 22, 2023
    MemberAuthor

    The object chain is invalid:

    (gdb) frame
#13 0x000000000055e731 in _Py_ForgetReference (op=(None, None, None)) at Objects/object.c:2235
2235	        _PyObject_ASSERT_FAILED_MSG(op, "invalid object chain");

(gdb) l
2230	    }
2231	
2232	    if (op == &refchain ||
2233	        op->_ob_prev->_ob_next != op || op->_ob_next->_ob_prev != op)
2234	    {
2235	        _PyObject_ASSERT_FAILED_MSG(op, "invalid object chain");
2236	    }
2237	
2238	#ifdef SLOW_UNREF_CHECK
2239	    PyObject *p;

(gdb) p op->_ob_prev->_ob_next != op
$14 = 1

(gdb) p op->_ob_next->_ob_prev != op
$15 = 1

  4. vstinner commented on Jul 22, 2023

    @vstinner
    vstinner
    on Jul 22, 2023
    MemberAuthor

    Similar error in test_interpreters.test_create_many_threaded():

    $ ./python -m test test_interpreters -m test_create_many_threaded -v
(...)
test_create_many_threaded (test.test_interpreters.StressTests.test_create_many_threaded) ...

Objects/object.c:2235: _Py_ForgetReference: Assertion failed: invalid object chain
(...)

  5. vstinner commented on Jul 22, 2023

    @vstinner
    vstinner
    on Jul 22, 2023
    MemberAuthor

    cc @ericsnowcurrently

  6. ericsnowcurrently
    self-assigned this
    on Aug 2, 2023

  7. ericsnowcurrently commented on Aug 2, 2023

    @ericsnowcurrently
    ericsnowcurrently
    on Aug 2, 2023
    Member

    --with-trace-refs builds use a C global variable to store the linked list of objects. The problem is that the variable is used by all interpreters. The solution is to make that variable per-interpreter.

  8. ericsnowcurrently
    added
    3.12only security fixes
    3.13only security fixes
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    topic-subinterpreters
    on Aug 2, 2023
  9. github-project-automation
    added this to Subinterpreterson Aug 2, 2023
  10. github-project-automation
    moved this to Todo in Subinterpreterson Aug 2, 2023
  11. ericsnowcurrently
    added a commit that references this issue on Aug 3, 2023

    gh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpreters (g…

    58ef741
  12. miss-islington
    added a commit that references this issue on Aug 3, 2023

    pythongh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpre…

    b55de7a
  13. Yhg1s
    added a commit that references this issue on Aug 3, 2023

    [3.12] gh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpr…

    58af229
  14. ericsnowcurrently
    closed this as completedon Aug 3, 2023
  15. github-project-automation
    moved this from Todo to Done in Subinterpreterson Aug 3, 2023
  16. ericsnowcurrently
    added a commit that references this issue on Aug 4, 2023

    Revert "[3.12] pythongh-107080: Fix Py_TRACE_REFS Crashes Under Isola…

    b380392
  17. Yhg1s
    added a commit that references this issue on Aug 5, 2023

    [3.12] gh-107630: Revert "[3.12] gh-107080: Fix Py_TRACE_REFS Crashes…

    6e4eec7
  18. ericsnowcurrently
    added a commit that references this issue on Aug 7, 2023

    Unrevert "[3.12] pythongh-107080: Fix Py_TRACE_REFS Crashes Under Iso…

    6f8b84f
  19. serhiy-storchaka
    reopened this on Aug 16, 2023
  20. serhiy-storchaka
    linked a pull request that will close this issue[3.12] gh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpreters #107751on Aug 16, 2023
  21. Yhg1s
    added a commit that references this issue on Aug 16, 2023

    [3.12] gh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpr…

    aa9707d

  22. vstinner commented on Sep 14, 2023

    @vstinner
    vstinner
    on Sep 14, 2023
    MemberAuthor

    serhiy-storchaka reopened this Aug 16, 2023

    @serhiy-storchaka: Hum, is this issue fixed or not?

  23. serhiy-storchaka commented on Sep 14, 2023

    @serhiy-storchaka
    serhiy-storchaka
    on Sep 14, 2023
    Member

    I reopened it only because the backport to 3.12 was not merged yet. I linked the backport PR to this issue and thought that merging it will automatically close the issue, but it did not happen.

  24. serhiy-storchaka
    closed this as completedon Sep 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

3.12only security fixes3.13only security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)topic-subinterpreterstype-bugAn unexpected behavior, bug, or error

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions