Segfault/aborts calling readline.set_completer_delims in threads in a free-threaded build

@vstinner

Crash report

What happened?

Calling difflib._test in threads in a free-threaded build (with PYTHON_GIL=0) will result in aborts or segfaults, apparently related to memory issues:

from threading import Thread
import difflib

for x in range(100):
    Thread(target=difflib._test, args=()).start()

Segfault backtrace:

Thread 92 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffeca7e4640 (LWP 631239)]
0x00007ffff7d3f743 in unlink_chunk (p=p@entry=0x555555f56c00, av=0x7ffff7eb8c80 <main_arena>) at ./malloc/malloc.c:1634
1634    ./malloc/malloc.c: No such file or directory.
(gdb) bt
#0  0x00007ffff7d3f743 in unlink_chunk (p=p@entry=0x555555f56c00, av=0x7ffff7eb8c80 <main_arena>)
    at ./malloc/malloc.c:1634
#1  0x00007ffff7d40d2b in _int_free (av=0x7ffff7eb8c80 <main_arena>, p=0x555555f528a0,
    have_lock=<optimized out>) at ./malloc/malloc.c:4616
#2  0x00007ffff7d43453 in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3391
#3  0x000055555570ed2c in _PyMem_RawFree (_unused_ctx=<optimized out>, ptr=<optimized out>)
    at Objects/obmalloc.c:90
#4  0x0000555555713955 in _PyMem_DebugRawFree (ctx=0x555555cdb8f8 <_PyRuntime+824>, p=0x555555f528c0)
    at Objects/obmalloc.c:2767
#5  0x000055555572c462 in PyMem_RawFree (ptr=ptr@entry=0x555555f528c0) at Objects/obmalloc.c:971
#6  0x00005555559527fc in free_threadstate (tstate=tstate@entry=0x555555f528c0) at Python/pystate.c:1411
#7  0x00005555559549c0 in _PyThreadState_DeleteCurrent (tstate=tstate@entry=0x555555f528c0)
    at Python/pystate.c:1834
#8  0x0000555555a11b74 in thread_run (boot_raw=boot_raw@entry=0x555555f52870)
    at ./Modules/_threadmodule.c:355
#9  0x0000555555974fdb in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:242
#10 0x00007ffff7d32ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007ffff7dc4850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Abort 1 backtrace:

Thread 78 "python" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffed17f2640 (LWP 633651)]
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140732413191744) at ./nptl/pthread_kill.c:44
44      ./nptl/pthread_kill.c: No such file or directory.
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140732413191744) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140732413191744) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140732413191744, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7ce0476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7cc67f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff7d27676 in __libc_message (action=action@entry=do_abort,
    fmt=fmt@entry=0x7ffff7e79b77 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#6  0x00007ffff7d3ecfc in malloc_printerr (
    str=str@entry=0x7ffff7e7c6c0 "free(): unaligned chunk detected in tcache 2") at ./malloc/malloc.c:5664
#7  0x00007ffff7d40e3f in _int_free (av=0x7ffff7eb8c80 <main_arena>, p=0x555555dc79b0, have_lock=0)
    at ./malloc/malloc.c:4471
#8  0x00007ffff7d43453 in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3391
#9  0x00007ffff41d4a2b in readline_set_completer_delims (module=<optimized out>, string=<optimized out>)
    at ./Modules/readline.c:593
#10 0x00005555557021ef in cfunction_vectorcall_O (
    func=<built-in method set_completer_delims of module object at remote 0x2000297b920>,
    args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:523
#11 0x000055555567cc55 in _PyObject_VectorcallTstate (tstate=0x555555f15bc0,
    callable=<built-in method set_completer_delims of module object at remote 0x2000297b920>,
    args=0x7ffed17f15a8, nargsf=9223372036854775809, kwnames=0x0) at ./Include/internal/pycore_call.h:167
#12 0x000055555567cd74 in PyObject_Vectorcall (
    callable=callable@entry=<built-in method set_completer_delims of module object at remote 0x2000297b920>,
 args=args@entry=0x7ffed17f15a8, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at Objects/call.c:327
#13 0x00005555558441f7 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x555555f15bc0,
    frame=<optimized out>, throwflag=throwflag@entry=0) at Python/generated_cases.c.h:955
#14 0x0000555555872978 in _PyEval_EvalFrame (throwflag=0, frame=<optimized out>, tstate=0x555555f15bc0)
    at ./Include/internal/pycore_ceval.h:116

Abort 2 backtrace:

Thread 43 "python" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fff54ff9640 (LWP 636245)]
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140734619424320) at ./nptl/pthread_kill.c:44
44      ./nptl/pthread_kill.c: No such file or directory.
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140734619424320) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140734619424320) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140734619424320, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7ce0476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7cc67f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff7d27676 in __libc_message (action=action@entry=do_abort,
    fmt=fmt@entry=0x7ffff7e79b77 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#6  0x00007ffff7d3ecfc in malloc_printerr (
    str=str@entry=0x7ffff7e7cf20 "tcache_thread_shutdown(): unaligned tcache chunk detected")
    at ./malloc/malloc.c:5664
#7  0x00007ffff7d436c4 in tcache_thread_shutdown () at ./malloc/malloc.c:3224
#8  __malloc_arena_thread_freeres () at ./malloc/arena.c:1003
#9  0x00007ffff7d461ca in __libc_thread_freeres () at ./malloc/thread-freeres.c:44
#10 0x00007ffff7d3294f in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:456
#11 0x00007ffff7dc4850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Found using fusil by @vstinner.

CPython versions tested on:

3.13, CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.14.0a1+ experimental free-threading build (heads/main-dirty:612ac283b81, Nov 16 2024, 01:37:56) [GCC 11.4.0]

Linked PRs

gh-126895: fix readline module in free-threaded build #131208

@ZeroIntensity

The _test function is only doing:

def _test():
    import doctest, difflib
    return doctest.testmod(difflib)

I think the issue is either with doctest or the imports, not with difflib itself.

cc @ZeroIntensity

I believe this is due to doctest using pdb, which seems to cause all sorts of segfaults and aborts when called from threads. Comment the marked lines out for a different crash.

import pdb
import sys
from threading import Thread

stdin = open("/dev/null")
sys.stdin = stdin

def f():
    for x in range(100):
        p = pdb.Pdb()
        p.reset()
        try:               # Comment
            p.set_trace()  # these
        except:            # lines
            pass           # out.
        p.reset()
        p.set_continue()

for x in range(100):
    Thread(target=f, args=()).start()

So maybe the issue is just that pdb isn't free-threading ready?

Edit: here's a collection of errors this causes:

    2908176786326028288 bytes originally requested
    The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):
        at p-7: 0x5b *** OUCH
        at p-6: 0xf0 *** OUCH
        at p-5: 0xf8 *** OUCH
        at p-4: 0xfe *** OUCH
        at p-3: 0x53 *** OUCH
        at p-2: 0x1e *** OUCH
        at p-1: 0x99 *** OUCH
    Because memory is corrupted at the start, the count of bytes requested
       may be bogus, and checking the trailing pad bytes may segfault.
mimalloc: error: thread 0x7fa0154d4640: buffer overflow in heap block 0x2000a010f30 of size 40: write after 40 bytes
    The 8 pad bytes at tail=0x285c417fbab28370 are Aborted

free(): unaligned chunk detected in tcache 2
Aborted

tcache_thread_shutdown(): unaligned tcache chunk detected
Aborted

free(): invalid pointer
Aborted

Segmentation fault

double free or corruption (fasttop)
Aborted

Debug memory block at address p=0x5572ce2f7580: API '�'
    14 bytes originally requested
    The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):
        at p-7: 0xdb *** OUCH
        at p-6: 0x4c *** OUCH
        at p-5: 0x84 *** OUCH
        at p-4: 0x07 *** OUCH
        at p-2: 0xc2 *** OUCH
        at p-1: 0x3a *** OUCH
    Because memory is corrupted at the start, the count of bytes requested
       may be bogus, and checking the trailing pad bytes may segfault.
    The 8 pad bytes at tail=0x5572ce2f758e are not all FORBIDDENBYTE (0xfd):
        at tail+0: 0xdd *** OUCH
        at tail+1: 0xdd *** OUCH
        at tail+2: 0xdd *** OUCH
        at tail+3: 0xdd *** OUCH
        at tail+4: 0xdd *** OUCH
        at tail+5: 0xdd *** OUCH
        at tail+6: 0xdd *** OUCH
        at tail+7: 0xdd *** OUCH
    Data at p: dd dd dd dd dd dd dd dd dd dd dd dd dd dd

Enable tracemalloc to get the memory block allocation traceback

Fatal Python error: _PyMem_DebugRawFree: bad ID: Allocated using API '�', verified using API 'r'
Python runtime state: initialized
Aborted

@gaogaotiantian

@gaogaotiantian Is there a plan to make pdb free-threaded friendly or not? if so, how can we help, if not what should we do for this issue?

I'll wait for Tian's input, but I'm happy to spend a few hours covering pdb in locks if we need to.

pdb is a pure Python module, without any black magic (ctypes, changing the code object, etc.). It should not be possible for it to crash (segfault) CPython, free-threaded build or not. If pdb crashes free-threaded build, that means free-threaded build has a bug, not pdb.

From the code above, the conclusion I can get is pdb triggered a crash, and we don't know why. Protecting pdb with lock might hide the crash, but it does not solve the problem. We should not be able to crash CPython with pure Python code.

Also, pdb does not support multi-threading debugging at this point. It's really slow to move it forward for any relatively new features. That being said, I don't think it's a good time to try to protect it with locks because we need to deal with multithreading in the future.

Does it use some weird frames APIs (e.g. sys._getframe) by any chance? Those aren't thread safe IIRC.

The culprit seems to be readline, which pdb uses:

from threading import Thread
import readline

def f():
    for x in range(100):
        readline.get_completer_delims()
        readline.set_completer_delims(' \t\n`@#%^&*()=+[{]}\\|;:\'",<>?')
        readline.set_completer_delims(' \t\n`@#%^&*()=+[{]}\\|;:\'",<>?')
        readline.get_completer_delims()

for x in range(100):
    Thread(target=f, args=()).start()

Seems to fail with the same errors as the original code.

I would be a little bit more careful when dealing with readline, because it relies on some global variables of the readline library, so we can't simply put everything thread or module local. The variables starting with rl_ are the extern global variables exposed by readline itself. Also, we support libedit which has a slightly different interface. The module is not the most stable one in our stdlib so tread carefully :)

If it has global state, I doubt the functions themself are thread safe. In that case, we probably should just mark it as needing the GIL.

In that case, we probably should just mark it as needing the GIL.

Or add @critical_section or another lock on functions which are known to not be thread-safe.

That wouldn't work for subinterpreters, would it?

pythongh-126895: fix readline module in free-threaded build

gh-126895: Fix readline module in free-threaded build (#131208)

This is fixed in main, but not backported to 3.13.

pythongh-126895: Fix readline module in free-threaded build (python#1…

This seems to have regressed in 3.13 somehow, unless it's a different issue coming from the same code. I'm getting aborts and segfaults with the following code:

import readline
from threading import Thread
from time import sleep

def stress_readline():
    for x in range(10):
        readline.get_completer_delims()
        readline.set_completer_delims(' \t\n`@#%^&*()=+[{]}\\|;:\'",<>?')
        sleep(0.01)
        readline.set_completer_delims(' \t\n`@#%^&*()=+[{]}\\|;:\'",<>?')
        readline.get_completer_delims()


alive = []

for x in range(100):
    alive.append(Thread(target=stress_readline, args=()))

for t in alive:
    t.start()

for t in alive:
    t.join()

Abort backtrace 1:

double free or corruption (fasttop)
double free or corruption (fasttop)

Thread 15 "python" received signal SIGABRT, Aborted.

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=6, no_tid=0) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (threadid=<optimized out>, signo=6) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7c4519e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7c28902 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff7c2976c in __libc_message_impl (fmt=fmt@entry=0x7ffff7ddc8e2 "%s\n") at ../sysdeps/posix/libc_fatal.c:134
#6  0x00007ffff7cae565 in malloc_printerr (str=str@entry=0x7ffff7ddfd00 "double free or corruption (fasttop)") at ./malloc/malloc.c:5772
#7  0x00007ffff7cb097a in _int_free (av=0x7fffb8000030, p=<optimized out>, have_lock=0) at ./malloc/malloc.c:4607
#8  0x00007ffff7cb3641 in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3398
#9  tcache_thread_shutdown () at ./malloc/malloc.c:3234
#10 __malloc_arena_thread_freeres () at ./malloc/arena.c:897
#11 0x00007ffff7cb5bd2 in __libc_thread_freeres () at ./malloc/thread-freeres.c:41
#12 0x00007ffff7ca1c27 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:458
#13 0x00007ffff7d33a4c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Segfault backtrace 1:

double free or corruption (fasttop)

Thread 99 "python" received signal SIGSEGV, Segmentation fault.

#0  0x00007ffff7cb34fa in tcache_thread_shutdown () at ./malloc/malloc.c:3233
#1  __malloc_arena_thread_freeres () at ./malloc/arena.c:897
#2  0x00007ffff7cb5bd2 in __libc_thread_freeres () at ./malloc/thread-freeres.c:41
#3  0x00007ffff7ca1c27 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:458
#4  0x00007ffff7d33a4c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Segfault backtrace 2:

Thread 61 "python" received signal SIGSEGV, Segmentation fault.

#0  0x00007ffff7cb0b1c in _int_free (av=0x7ffeb8000030, p=0x7ffeb8000be0, have_lock=0) at ./malloc/malloc.c:4534
#1  0x00007ffff7cb33ee in __GI___libc_free (mem=0x7ffeb8000bf0) at ./malloc/malloc.c:3398
#2  0x00007ffff7fa9732 in readline_set_completer_delims (module=<optimized out>, string=<optimized out>) at ./Modules/readline.c:599
#3  0x0000555555683902 in cfunction_vectorcall_O (func=<optimized out>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:512
#4  0x00005555556278e9 in _PyObject_VectorcallTstate (tstate=0x555555c71070, callable=0x243067e5910, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
    at ./Include/internal/pycore_call.h:168
#5  PyObject_Vectorcall (callable=callable@entry=0x243067e5910, args=0x7ffeb8000be0, args@entry=0x7ffff45451f0, nargsf=48, kwnames=0x7ffebc0008e0, kwnames@entry=0x0) at Objects/call.c:327
#6  0x0000555555758b12 in _PyEval_EvalFrameDefault (tstate=0x555555c71070, frame=0x7ffff4545188, throwflag=<optimized out>) at Python/generated_cases.c.h:813
#7  0x000055555562a3b4 in _PyObject_VectorcallTstate (tstate=0x555555c71070, callable=0x24306915aa0, args=0x7ffec5ffce28, nargsf=1, kwnames=0x0) at ./Include/internal/pycore_call.h:168
#8  method_vectorcall (method=<optimized out>, args=<optimized out>, nargsf=<optimized out>, kwnames=0x0) at Objects/classobject.c:70
#9  0x0000555555843b9c in thread_run (boot_raw=boot_raw@entry=0x555555c71030) at ./Modules/_threadmodule.c:337
#10 0x00005555557d9526 in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:243
#11 0x00007ffff7ca1e2e in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#12 0x00007ffff7d33a4c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Segfault backtrace 3:

Thread 54 "python" received signal SIGSEGV, Segmentation fault.

0x00007ffff7cb0b1c in _int_free (av=0x7ffe3c000030, p=0x7ffe3c000be0, have_lock=0) at ./malloc/malloc.c:4534

#0  0x00007ffff7cb0b1c in _int_free (av=0x7ffe3c000030, p=0x7ffe3c000be0, have_lock=0) at ./malloc/malloc.c:4534
#1  0x00007ffff7cb33ee in __GI___libc_free (mem=0x7ffe3c000bf0) at ./malloc/malloc.c:3398
#2  0x00005555556d8490 in unicode_encode_locale (unicode=<optimized out>, error_handler=<optimized out>, current_locale=1) at Objects/unicodeobject.c:3721
#3  0x00007ffff7fa96e3 in encode (b=0x5e2523479fe9c920) at ./Modules/readline.c:169
#4  readline_set_completer_delims (module=<optimized out>, string=0x5e2523479fe9c920) at ./Modules/readline.c:589
#5  0x0000555555683902 in cfunction_vectorcall_O (func=<optimized out>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:512
#6  0x00005555556278e9 in _PyObject_VectorcallTstate (tstate=0x555555c51e20, callable=0x510187e5910, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
    at ./Include/internal/pycore_call.h:168
#7  PyObject_Vectorcall (callable=callable@entry=0x510187e5910, args=0x7ffe3c000be0, args@entry=0x7ffff45611f0, nargsf=48, kwnames=0x7ffed80008e0, kwnames@entry=0x0) at Objects/call.c:327
#8  0x0000555555758b12 in _PyEval_EvalFrameDefault (tstate=0x555555c51e20, frame=0x7ffff4561188, throwflag=<optimized out>) at Python/generated_cases.c.h:813
#9  0x000055555562a3b4 in _PyObject_VectorcallTstate (tstate=0x555555c51e20, callable=0x51018915aa0, args=0x7ffee6ffde28, nargsf=1, kwnames=0x0) at ./Include/internal/pycore_call.h:168
#10 method_vectorcall (method=<optimized out>, args=<optimized out>, nargsf=<optimized out>, kwnames=0x0) at Objects/classobject.c:70
#11 0x0000555555843b9c in thread_run (boot_raw=boot_raw@entry=0x555555c51de0) at ./Modules/_threadmodule.c:337
#12 0x00005555557d9526 in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:243
#13 0x00007ffff7ca1e2e in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7d33a4c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

python -VV:

Python 3.13.3+ experimental free-threading build (heads/3.13:83cb89b941b, Apr 19 2025, 03:58:36) [Clang 19.1.7 (++20250114103253+cd708029e0b2-1~exp1~20250114103309.40)]

Oh, never mind, according to #131208 (comment) this wasn't backported to 3.13. Sorry for the noise.

pythongh-153291: Fix data race in readline.get_completer() and get_pr…

gh-153291: Fix data race in readline.get_completer() and get_pre_inpu…

[3.15] gh-153291: Fix data race in readline.get_completer() and get_p…

Navigation Menu

Repository navigation

Segfault/aborts calling `readline.set_completer_delims` in threads in a free-threaded build #126895

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Linked PRs

6 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Segfault/aborts calling readline.set_completer_delims in threads in a free-threaded build #126895

Description

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Linked PRs

Activity

picnixz commented on Nov 16, 2024

devdanzin commented on Nov 16, 2024

picnixz commented on Nov 16, 2024

ZeroIntensity commented on Nov 16, 2024

gaogaotiantian commented on Nov 16, 2024

ZeroIntensity commented on Nov 16, 2024

devdanzin commented on Nov 16, 2024

6 remaining items

gaogaotiantian commented on Nov 16, 2024

ZeroIntensity commented on Nov 16, 2024

vstinner commented on Nov 18, 2024

ZeroIntensity commented on Nov 18, 2024

colesbury commented on Mar 17, 2025

devdanzin commented on Apr 19, 2025

devdanzin commented on Apr 19, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Segfault/aborts calling `readline.set_completer_delims` in threads in a free-threaded build #126895