JIT: executor->vm_data.valid assertion failure in unlink_executor

Crash report

What happened?

Sorry about the messy and convoluted bug report, I hope something interesting might be gleaned from it.

It's possible to non-deterministically abort a special debug JIT build by running the code below, running from a Python executable from a venv where you installed a package in editable mode. This darn requirement for an install in editable mode stumped me for hours!

For this special build, apply this diff (couldn't get it to work in a normal build by changing loop lengths):

index 454c8dde031..9e21c41421a 100644
--- a/Include/internal/pycore_backoff.h
+++ b/Include/internal/pycore_backoff.h
@@ -99,8 +99,8 @@ backoff_counter_triggers(_Py_BackoffCounter counter)
 // Must be larger than ADAPTIVE_COOLDOWN_VALUE, otherwise when JIT code is
 // invalidated we may construct a new trace before the bytecode has properly
 // re-specialized:
-#define JUMP_BACKWARD_INITIAL_VALUE 4095
-#define JUMP_BACKWARD_INITIAL_BACKOFF 12
+#define JUMP_BACKWARD_INITIAL_VALUE 63
+#define JUMP_BACKWARD_INITIAL_BACKOFF 6
 static inline _Py_BackoffCounter
 initial_jump_backoff_counter(void)
 {
@@ -112,8 +112,8 @@ initial_jump_backoff_counter(void)
  * Must be larger than ADAPTIVE_COOLDOWN_VALUE,
  * otherwise when a side exit warms up we may construct
  * a new trace before the Tier 1 code has properly re-specialized. */
-#define SIDE_EXIT_INITIAL_VALUE 4095
-#define SIDE_EXIT_INITIAL_BACKOFF 12
+#define SIDE_EXIT_INITIAL_VALUE 63
+#define SIDE_EXIT_INITIAL_BACKOFF 6

 static inline _Py_BackoffCounter
 initial_temperature_backoff_counter(void)

This MRE must be invoked like this:

/path/to/venv/with/editable/install/bin/python mre.py

Sorry about the long MRE, this code seems stubbornly hard to reduce. MRE:

from random import random

class C_object_v1:

    def __getitem__(self, item):
        return 5
object_v1 = C_object_v1()


int_v4 = -51452741571812374
float_v6 = 46.1
float_v7 = 77.26

class StatefulLen_stateful_len_object_v12:
    def __init__(self):
        self.len_count = 0

    def __len__(self):
        _len = 0 if self.len_count < 70 else 99
        self.len_count += 1
        return _len
stateful_len_object_v12 = StatefulLen_stateful_len_object_v12()

def uop_harness_f1():
    class StatefulIter_uop_harness_f1_6059:

        def __init__(self):
            self._iter_count = 0
            self._iterable = {1, 2, 3}

        def __iter__(self):
            self._iter_count += 1
            if self._iter_count <= 67:
                return iter((None,))
            return iter(self._iterable)
    uop_harness_f1_6059 = StatefulIter_uop_harness_f1_6059()
    evil_iterable = StatefulIter_uop_harness_f1_6059()
    for _ in range(100):
        tuple(evil_iterable)
    object_v1 = globals()['object_v1']

    class SwappedClass_2950:
        pass
    object_v1.__class__ = SwappedClass_2950
    if object_v1:
        pass
    for i_loop_9595 in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] * 50:
        res_compareopint = int_v4 > int_v4
    if random() < 0.01:
        object_v1.__class__.get_value = lambda *a, **kw: 'patched!'
for i_f1 in range(500):
    try:
        uop_harness_f1()
    except Exception:
        break

Backtrace:

python: Python/optimizer.c:1436: unlink_executor: Assertion `executor->vm_data.valid' failed.

Program received signal SIGABRT, Aborted.
Download failed: Invalid argument.  Continuing without source file ./nptl/./nptl/pthread_kill.c.
__pthread_kill_implementation (threadid=<optimized out>, signo=6, no_tid=0)
    at ./nptl/pthread_kill.c:44
warning: 44     ./nptl/pthread_kill.c: No such file or directory
(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=6, no_tid=0)
    at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (threadid=<optimized out>, signo=6) at ./nptl/pthread_kill.c:89
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6)
    at ./nptl/pthread_kill.c:100
#3  0x00007ffff7c4579e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7c288cd in __GI_abort () at ./stdlib/abort.c:73
#5  0x00007ffff7c28830 in __assert_fail_base (fmt=<optimized out>, assertion=<optimized out>,
    file=<optimized out>, line=<optimized out>, function=<optimized out>) at ./assert/assert.c:118
#6  0x00007ffff7c3be1f in __assert_fail (assertion=<optimized out>, file=<optimized out>,
    line=<optimized out>, function=<optimized out>) at ./assert/assert.c:127
#7  0x00005555558a2603 in unlink_executor (executor=executor@entry=0x555555f44480)
    at Python/optimizer.c:1436
#8  0x00005555558a31a7 in uop_dealloc (op=op@entry=0x555555f44480) at Python/optimizer.c:271
#9  0x00005555556dc978 in _Py_Dealloc (op=0x555555f44480) at Objects/object.c:3206
#10 0x00007ffff71d59f7 in ?? ()
#11 0x00007fffffffd470 in ?? ()
#12 0x00007ffff71d522b in ?? ()
#13 0x00007fffffffd470 in ?? ()
#14 0x00007ffff71d4017 in ?? ()
#15 0x0000555555e60c70 in ?? ()
#16 0x00007ffff71d4000 in ?? ()
#17 0x0000555555cca298 in _PyRuntime ()
#18 0x00007ffff7fac0b8 in ?? ()
#19 0x00007ffff7fac160 in ?? ()
#20 0x00005555557ca1c0 in _PyEval_EvalFrameDefault (tstate=0xcdcdcdcdcdcdcdcd, frame=0x555555cca298 <_PyRuntime+331064>, throwflag=-842150451)
    at Python/generated_cases.c.h:5598
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Here are the outputs from running with PYTHON_OPT_DEBUG=4 and PYTHON_LLTRACE=4, since they are too long I've attached them as files:
opt_debug2.txt
lltrace2.txt

Found using lafleur.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.15.0a0 (heads/main-dirty:6bf1c0ab349, Jul 21 2025, 20:28:08) [GCC 14.2.0]

I can't reproduce this. I installed https://github.com/pypa/sampleproject in editable mode on a JIT with the patch applied, and ran the reproducer in an infinite while loop for some time in bash. No crash.

Please try with https://github.com/devdanzin/lafleur/ installed as editable, with no other packages installed. Seems to do the trick here. Sorry for the bad reproducing steps.

OK, here's a reproducer that mimics the effects of having a package installed as editable. Please let me know whether it works.

import sys
from importlib.machinery import all_suffixes as module_suffixes  # Not sure why, but this unused import seems necessary
from importlib.util import spec_from_file_location

class _EditableFinder:
    @classmethod
    def _find_spec(cls, fullname, candidate_path):
        return spec_from_file_location(fullname, candidate_path.with_suffix("pyc"))

sys.meta_path.append(_EditableFinder)

from random import random

class C_object_v1:
    def __getitem__(self, item):
        return 5
object_v1 = C_object_v1()


int_v4 = -51452741571812374
float_v6 = 46.1
float_v7 = 77.26

class StatefulLen_stateful_len_object_v12:
    def __init__(self):
        self.len_count = 0

    def __len__(self):
        _len = 0 if self.len_count < 70 else 99
        self.len_count += 1
        return _len
stateful_len_object_v12 = StatefulLen_stateful_len_object_v12()

def uop_harness_f1():
    class StatefulIter_uop_harness_f1_6059:

        def __init__(self):
            self._iter_count = 0
            self._iterable = {1, 2, 3}

        def __iter__(self):
            self._iter_count += 1
            if self._iter_count <= 67:
                return iter((None,))
            return iter(self._iterable)
    uop_harness_f1_6059 = StatefulIter_uop_harness_f1_6059()
    evil_iterable = StatefulIter_uop_harness_f1_6059()
    for _ in range(100):
        tuple(evil_iterable)
    object_v1 = globals()['object_v1']

    class SwappedClass_2950:
        pass
    object_v1.__class__ = SwappedClass_2950
    if object_v1:
        pass
    for i_loop_9595 in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] * 50:
        res_compareopint = int_v4 > int_v4
    if random() < 0.01:
        object_v1.__class__.get_value = lambda *a, **kw: 'patched!'
for i_f1 in range(500):
    try:
        uop_harness_f1()
    except Exception:
        break

Reproduced. Thanks!

I'm making some progress on reducing. Here's what I've got so far:

import gc

class C1:
    pass

O = C1()

gc.set_threshold(1)

for i in range(326):
    for j in []:
        pass
    class C2:
        pass
    O.__class__ = C2
    if O:
        pass

I'm working now on bringing the JIT thresholds back up.

BTW the imports were just to get the GC in a ready-to-crash state. gc.set_threshold(1) is usually my go-to for removing that factor.

Higher JIT thresholds aren't reproducing. Highest I can get is cranking up the number of loops from 326 to 12868 to reproduce with thresholds of 127 instead of 63.

Thanks Brandt for the repro. Based on debugging this, I speculate the following is happening:
The code which crashes is

        case _EXIT_TRACE: {
        ....
            if (exit->executor && !exit->executor->vm_data.valid) {
                exit->temperature = initial_temperature_backoff_counter();
                _PyFrame_SetStackPointer(frame, stack_pointer);
                Py_CLEAR(exit->executor);  // THIS CRASHES
                stack_pointer = _PyFrame_GetStackPointer(frame);
            }

This causes assert(executor->vm_data.valid); to fail. The assert is clearly wrong in this case as we only free the exit trace executor when the data is no longer valid.

I think the correct thing is just to remove the assertion. WDYT Brandt?

@devdanzin

@devdanzin can you still repro this on main? Mark recently changed the _EXIT_TRACE mechanism.

I cannot reproduce on main anymore. I can still reproduce on 3.14. However, since the JIT is experimental, I'm reluctant to backport any changes to 3.14, especially when it deals with executor lifetimes.

The fix should be #136411

@devdanzin can you still repro this on main? Mark recently changed the _EXIT_TRACE mechanism.

It repros for me in main if we remove the gc tuning from Brandt's MRE:

class C1:
    pass

O = C1()

for i in range(326):
    for j in []:
        pass
    class C2:
        pass
    O.__class__ = C2
    if O:
        pass

Does this abort for you too?

@devdanzin

@devdanzin I can't reproduce the segfault on main.

What build flags are you using?

I cannot reproduce the segfault either, just the abort. On another machine, it's necessary to add the GC tuning back to get the abort:

import gc
gc.set_threshold(1)

class C1:
    pass

O = C1()

for i in range(326):
    for j in []:
        pass
    class C2:
        pass
    O.__class__ = C2
    if O:
        pass

I'm using the simplest config flags to get a debug JIT build:

./configure --with-pydebug --enable-experimental-jit && make -j 14

Reproduced without the GC tuning on my machine.

@markshannon

The failing assertion is in unlink_executor's assert(executor->vm_data.valid);. I think the assertion is just wrong. I don't see how an executor that is being unlinked should be guaranteed to be valid. Here's what happens:

        case _START_EXECUTOR: {
            PyObject *executor = (PyObject *)CURRENT_OPERAND0();
            #ifndef _Py_JIT
            assert(current_executor == (_PyExecutorObject*)executor);
            #endif
            assert(tstate->jit_exit == NULL || tstate->jit_exit->executor == current_executor);
            tstate->current_executor = (PyObject *)executor;
            if (!current_executor->vm_data.valid) {
                assert(tstate->jit_exit->executor == current_executor);
                assert(tstate->current_executor == executor);
                _PyFrame_SetStackPointer(frame, stack_pointer);
                _PyExecutor_ClearExit(tstate->jit_exit);   <------------ this gets called
                stack_pointer = _PyFrame_GetStackPointer(frame);
                if (true) {
                    UOP_STAT_INC(uopcode, miss);
                    JUMP_TO_JUMP_TARGET();
                }
            }
            break;
        }

If you look above, the branch taken is in !current_executor->vm_data.valid. That means the current executor is not valid, and we clear it. Yet when we clear it, we assert it's valid??? Seems like a buggy assertion. @markshannon WDYT?

Navigation Menu

Repository navigation

JIT: `executor->vm_data.valid` assertion failure in `unlink_executor` #136996

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

JIT: executor->vm_data.valid assertion failure in unlink_executor #136996

Description

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Activity

Fidget-Spinner commented on Jul 22, 2025

devdanzin commented on Jul 22, 2025

devdanzin commented on Jul 22, 2025

Fidget-Spinner commented on Jul 22, 2025

brandtbucher commented on Jul 22, 2025

brandtbucher commented on Jul 22, 2025

brandtbucher commented on Jul 22, 2025

Fidget-Spinner commented on Jul 23, 2025

Fidget-Spinner commented on Sep 1, 2025

Fidget-Spinner commented on Sep 1, 2025

devdanzin commented on Sep 1, 2025

Fidget-Spinner commented on Sep 2, 2025

devdanzin commented on Sep 3, 2025

Fidget-Spinner commented on Sep 3, 2025

Fidget-Spinner commented on Sep 3, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

JIT: `executor->vm_data.valid` assertion failure in `unlink_executor` #136996