Skip to content

Race between partial_vectorcall_fallback and _PyVectorcall_FunctionInline under free-threading #128050

Closed
#124584
Closed
Race between partial_vectorcall_fallback and _PyVectorcall_FunctionInline under free-threading#128050
#124584
Assignees
corona10
Labels
3.13only security fixes3.14bugs and security fixestopic-free-threadingtype-bugAn unexpected behavior, bug, or error

Description

@hawkinsp
hawkinsp
opened on Dec 18, 2024

Bug report

Bug description:

Repro:

Run the following Python code under v3.13.1t with thread-sanitizer enabled.

import concurrent.futures
import functools
import threading

def f(a, b):
  return a + b

def closure(b, g):
  b.wait()
  g(77)

num_threads = 8
with concurrent.futures.ThreadPoolExecutor(max_workers=num_threads) as executor:
  for _ in range(10000):
    b = threading.Barrier(num_threads)
    g = functools.partial(f, a=42)
    for _ in range(num_threads):
      executor.submit(functools.partial(closure, b, g))

Get:

==================
WARNING: ThreadSanitizer: data race (pid=130482)
  Read of size 8 at 0x7ff6e09ed1a8 by thread T2:
    #0 _PyVectorcall_FunctionInline /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:129:5 (python3.13+0x1eaf8f) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #1 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:163:12 (python3.13+0x1eaf8f)
    #2 PyObject_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c:327:12 (python3.13+0x1eaf8f)
    #3 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e24fb) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #4 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de62a) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #5 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1811:12 (python3.13+0x3de62a)
    #6 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb61f) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #7 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x571bb2) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #8 partial_vectorcall /usr/local/google/home/phawkins/p/cpython/./Modules/_functoolsmodule.c:252:16 (python3.13+0x571bb2)
    #9 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb293) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #10 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb293)
    #11 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb315) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #12 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e46e2) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #13 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de62a) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #14 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1811:12 (python3.13+0x3de62a)
    #15 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb61f) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #16 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef5ef) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #17 method_vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef5ef)
    #18 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb293) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #19 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb293)
    #20 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb315) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #21 thread_run /usr/local/google/home/phawkins/p/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x564292) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #22 pythread_wrapper /usr/local/google/home/phawkins/p/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bd637) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)

  Previous write of size 8 at 0x7ff6e09ed1a8 by thread T8:
    #0 partial_vectorcall_fallback /usr/local/google/home/phawkins/p/cpython/./Modules/_functoolsmodule.c:224:21 (python3.13+0x571d8b) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #1 partial_vectorcall /usr/local/google/home/phawkins/p/cpython/./Modules/_functoolsmodule.c:238:16 (python3.13+0x571a06) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #2 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eafaa) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #3 PyObject_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c:327:12 (python3.13+0x1eafaa)
    #4 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e24fb) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #5 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de62a) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #6 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1811:12 (python3.13+0x3de62a)
    #7 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb61f) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #8 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x571bb2) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #9 partial_vectorcall /usr/local/google/home/phawkins/p/cpython/./Modules/_functoolsmodule.c:252:16 (python3.13+0x571bb2)
    #10 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb293) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #11 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb293)
    #12 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb315) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #13 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e46e2) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #14 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de62a) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #15 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1811:12 (python3.13+0x3de62a)
    #16 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb61f) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #17 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef5ef) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #18 method_vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef5ef)
    #19 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb293) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #20 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb293)
    #21 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb315) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #22 thread_run /usr/local/google/home/phawkins/p/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x564292) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #23 pythread_wrapper /usr/local/google/home/phawkins/p/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bd637) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)

partial_vectorcall_fallback sets vectorcall to nullptr if it decides to fall back to a non-vectorcall code path:

cpython/Modules/_functoolsmodule.c

Line 366 in b92f101

pto->vectorcall = NULL;

But this races with the code that reads an object's vectorcall in the core function call code:

cpython/Include/internal/pycore_call.h

Line 128 in b92f101

memcpy(&ptr, (char *) callable + offset, sizeof(ptr));

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

Activity

  1. hawkinsp
    added
    type-bugAn unexpected behavior, bug, or error
    on Dec 18, 2024
  2. ZeroIntensity
    added
    topic-free-threading
    type-bugAn unexpected behavior, bug, or error
    3.13only security fixes
    3.14bugs and security fixes
    and removed
    type-bugAn unexpected behavior, bug, or error
    on Dec 18, 2024

  3. ZeroIntensity commented on Dec 18, 2024

    @ZeroIntensity
    ZeroIntensity
    on Dec 18, 2024
    Member

    Looks like a race. Probably somewhat related to #127266, but this isn't a full type slot, it's an offset.

  4. hawkinsp commented on Dec 18, 2024

    @hawkinsp
    hawkinsp
    on Dec 18, 2024
    ContributorAuthor

    Here perhaps the best fix is to implement the missing vectorcall case in partial?

    (Or just skip setting the pointer to nullptr in free-threading mode.)

    It seems like a fairly benign race: all it does is prevent a fast-path from being taken again. But there's no such thing as a benign race.

  5. colesbury commented on Dec 18, 2024

    @colesbury
    colesbury
    on Dec 18, 2024
    Contributor

    I think we should set vectorcall once in the partial constructor, instead of sometimes setting it to NULL lazily when it's invoked.

    I don't think there's a good enough reason to have a different implementation for the default build and free threading build. I don't think this sort of change will have any performance impact.

  6. dg-pb commented on Dec 20, 2024

    @dg-pb
    dg-pb
    on Dec 20, 2024
    Contributor

    This one is waiting for review #124584

    It eliminates dynamic switching as vectorcall can handle all cases.

  7. corona10
    self-assigned this
    on Apr 10, 2025

  8. kumaraditya303 commented on Jul 8, 2025

    @kumaraditya303
    kumaraditya303
    on Jul 8, 2025
    Contributor

    This happens sometimes on CI https://github.com/python/cpython/actions/runs/16065399445/job/45338930486, I think we should add suppression for this until this is fixed on both main and older branches, it is unlikely that it will be fixed for 3.13 now.

  9. kumaraditya303
    closed this as completedin #124584on Jul 9, 2025

  10. dpdani commented on Jan 1, 2026

    @dpdani
    dpdani
    on Jan 1, 2026
    Contributor

    I think I'm intermittently getting this race reported by TSAN in my CI.

    TSAN output 
    ============================= test session starts ==============================
platform linux -- Python 3.14.1, pytest-9.0.1, pluggy-1.6.0
rootdir: /__w/cereggii/cereggii
configfile: pyproject.toml
plugins: reraise-2.1.2
collected 135 items

==================
WARNING: ThreadSanitizer: data race (pid=529)
  Read of size 8 at 0x7f000bbd2e78 by thread T34:
    #0 _PyVectorcall_FunctionInline /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_call.h:130:5 (libpython3.14t.so.1.0+0x12bde1) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #1 _PyObject_VectorcallTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_call.h:164:12 (libpython3.14t.so.1.0+0x12bde1)
    #2 PyObject_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:327:12 (libpython3.14t.so.1.0+0x12bde1)
    #3 _PyEval_EvalFrameDefault /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/generated_cases.c.h:1619:35 (libpython3.14t.so.1.0+0x33b81f) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #4 _PyEval_EvalFrame /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_ceval.h:121:16 (libpython3.14t.so.1.0+0x3366f0) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #5 _PyEval_Vector /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/ceval.c:2083:12 (libpython3.14t.so.1.0+0x3366f0)
    #6 _PyFunction_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c (libpython3.14t.so.1.0+0x12c491) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #7 _PyObject_VectorcallTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_call.h:169:11 (libpython3.14t.so.1.0+0x130dd1) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #8 method_vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/classobject.c:73:20 (libpython3.14t.so.1.0+0x130dd1)
    #9 _PyVectorcall_Call /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:273:16 (libpython3.14t.so.1.0+0x12c117) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #10 _PyObject_Call /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:348:16 (libpython3.14t.so.1.0+0x12c117)
    #11 PyObject_Call /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:373:12 (libpython3.14t.so.1.0+0x12c187) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #12 thread_run /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Modules/_threadmodule.c:359:21 (libpython3.14t.so.1.0+0x501112) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #13 pythread_wrapper /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/thread_pthread.h:242:5 (libpython3.14t.so.1.0+0x43ffe7) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)

  Previous write of size 8 at 0x7f000bbd2e78 by thread T33:
    #0 partial_vectorcall_fallback /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Modules/_functoolsmodule.c:378:21 (libpython3.14t.so.1.0+0x50fd8b) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #1 partial_vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Modules/_functoolsmodule.c:393:16 (libpython3.14t.so.1.0+0x50f7ac) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #2 _PyObject_VectorcallTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_call.h:169:11 (libpython3.14t.so.1.0+0x12bdfc) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #3 PyObject_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:327:12 (libpython3.14t.so.1.0+0x12bdfc)
    #4 _PyEval_EvalFrameDefault /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/generated_cases.c.h:1619:35 (libpython3.14t.so.1.0+0x33b81f) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #5 _PyEval_EvalFrame /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_ceval.h:121:16 (libpython3.14t.so.1.0+0x3366f0) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #6 _PyEval_Vector /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/ceval.c:2083:12 (libpython3.14t.so.1.0+0x3366f0)
    #7 _PyFunction_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c (libpython3.14t.so.1.0+0x12c491) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #8 _PyObject_VectorcallTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_call.h:169:11 (libpython3.14t.so.1.0+0x130dd1) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #9 method_vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/classobject.c:73:20 (libpython3.14t.so.1.0+0x130dd1)
    #10 _PyVectorcall_Call /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:273:16 (libpython3.14t.so.1.0+0x12c117) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #11 _PyObject_Call /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:348:16 (libpython3.14t.so.1.0+0x12c117)
    #12 PyObject_Call /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:373:12 (libpython3.14t.so.1.0+0x12c187) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #13 thread_run /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Modules/_threadmodule.c:359:21 (libpython3.14t.so.1.0+0x501112) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #14 pythread_wrapper /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/thread_pthread.h:242:5 (libpython3.14t.so.1.0+0x43ffe7) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)

  Thread T34 'Thread-31 (wrap' (tid=670, running) created by main thread at:
    #0 pthread_create <null> (python3.14+0x64bde) (BuildId: 7c5c6a0a3514fb58caefa389920bee1b45f9edcc)
    #1 do_start_joinable_thread /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/thread_pthread.h:289:14 (libpython3.14t.so.1.0+0x43ee6a) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #2 PyThread_start_joinable_thread /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/thread_pthread.h:331:9 (libpython3.14t.so.1.0+0x43ec8a) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #3 ThreadHandle_start /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Modules/_threadmodule.c:445:9 (libpython3.14t.so.1.0+0x500ca6) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #4 do_start_new_thread /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Modules/_threadmodule.c:1868:9 (libpython3.14t.so.1.0+0x500ca6)
    #5 thread_PyThread_start_joinable_thread /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Modules/_threadmodule.c:1991:14 (libpython3.14t.so.1.0+0x4ffab1) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #6 cfunction_call /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/methodobject.c:564:18 (libpython3.14t.so.1.0+0x1dad49) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #7 _PyObject_MakeTpCall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:242:18 (libpython3.14t.so.1.0+0x12b29e) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #8 _PyObject_VectorcallTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_call.h:167:16 (libpython3.14t.so.1.0+0x12beba) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #9 PyObject_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:327:12 (libpython3.14t.so.1.0+0x12beba)
    #10 _PyEval_EvalFrameDefault /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/generated_cases.c.h:3227:35 (libpython3.14t.so.1.0+0x341b47) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #11 _PyEval_EvalFrame /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_ceval.h:121:16 (libpython3.14t.so.1.0+0x3366f0) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #12 _PyEval_Vector /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/ceval.c:2083:12 (libpython3.14t.so.1.0+0x3366f0)
    #13 _PyFunction_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c (libpython3.14t.so.1.0+0x12c491) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #14 _PyObject_VectorcallDictTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:146:15 (libpython3.14t.so.1.0+0x12afe0) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #15 _PyObject_Call_Prepend /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:504:24 (libpython3.14t.so.1.0+0x12cad6) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #16 call_method /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/typeobject.c:2937:19 (libpython3.14t.so.1.0+0x24cee8) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #17 slot_tp_call /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/typeobject.c:10244:12 (libpython3.14t.so.1.0+0x24cd21) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #18 _PyObject_MakeTpCall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:242:18 (libpython3.14t.so.1.0+0x12b29e) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #19 _PyObject_VectorcallTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_call.h:167:16 (libpython3.14t.so.1.0+0x12beba) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #20 PyObject_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:327:12 (libpython3.14t.so.1.0+0x12beba)
    #21 _PyEval_EvalFrameDefault /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/generated_cases.c.h:3227:35 (libpython3.14t.so.1.0+0x341b47) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #22 _PyEval_EvalFrame /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_ceval.h:121:16 (libpython3.14t.so.1.0+0x3366f0) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #23 _PyEval_Vector /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/ceval.c:2083:12 (libpython3.14t.so.1.0+0x3366f0)
    #24 _PyFunction_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c (libpython3.14t.so.1.0+0x12c491) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #25 _PyObject_VectorcallDictTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:146:15 (libpython3.14t.so.1.0+0x12afe0) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #26 _PyObject_Call_Prepend /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:504:24 (libpython3.14t.so.1.0+0x12cad6) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #27 call_method /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/typeobject.c:2937:19 (libpython3.14t.so.1.0+0x24cee8) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #28 slot_tp_call /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/typeobject.c:10244:12 (libpython3.14t.so.1.0+0x24cd21) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #29 _PyObject_Call /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:361:18 (libpython3.14t.so.1.0+0x12c0e4) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #30 PyObject_Call /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:373:12 (libpython3.14t.so.1.0+0x12c187) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #31 _PyEval_EvalFrameDefault /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/generated_cases.c.h:2654:32 (libpython3.14t.so.1.0+0x33f61f) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #32 _PyEval_EvalFrame /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_ceval.h:121:16 (libpython3.14t.so.1.0+0x3366f0) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #33 _PyEval_Vector /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/ceval.c:2083:12 (libpython3.14t.so.1.0+0x3366f0)
    #34 _PyFunction_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c (libpython3.14t.so.1.0+0x12c491) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #35 _PyObject_VectorcallDictTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:146:15 (libpython3.14t.so.1.0+0x12afe0) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #36 _PyObject_Call_Prepend /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:504:24 (libpython3.14t.so.1.0+0x12cad6) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #37 call_method /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/typeobject.c:2937:19 (libpython3.14t.so.1.0+0x24cee8) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #38 slot_tp_call /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/typeobject.c:10244:12 (libpython3.14t.so.1.0+0x24cd21) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #39 _PyObject_MakeTpCall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:242:18 (libpython3.14t.so.1.0+0x12b29e) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #40 _PyObject_VectorcallTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_call.h:167:16 (libpython3.14t.so.1.0+0x12beba) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #41 PyObject_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:327:12 (libpython3.14t.so.1.0+0x12beba)
    #42 _PyEval_EvalFrameDefault /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/generated_cases.c.h:3227:35 (libpython3.14t.so.1.0+0x341b47) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #43 _PyEval_EvalFrame /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_ceval.h:121:16 (libpython3.14t.so.1.0+0x3366f0) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #44 _PyEval_Vector /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/ceval.c:2083:12 (libpython3.14t.so.1.0+0x3366f0)
    #45 _PyFunction_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c (libpython3.14t.so.1.0+0x12c491) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #46 _PyObject_VectorcallDictTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:146:15 (libpython3.14t.so.1.0+0x12afe0) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #47 _PyObject_Call_Prepend /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:504:24 (libpython3.14t.so.1.0+0x12cad6) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #48 call_method /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/typeobject.c:2937:19 (libpython3.14t.so.1.0+0x24cee8) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #49 slot_tp_call /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/typeobject.c:10244:12 (libpython3.14t.so.1.0+0x24cd21) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #50 _PyObject_MakeTpCall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:242:18 (libpython3.14t.so.1.0+0x12b29e) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #51 _PyObject_VectorcallTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_call.h:167:16 (libpython3.14t.so.1.0+0x12beba) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #52 PyObject_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:327:12 (libpython3.14t.so.1.0+0x12beba)
    #53 _PyEval_EvalFrameDefault /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/generated_cases.c.h:2959:35 (libpython3.14t.so.1.0+0x3409a5) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #54 _PyEval_EvalFrame /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_ceval.h:121:16 (libpython3.14t.so.1.0+0x3366f0) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #55 _PyEval_Vector /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/ceval.c:2083:12 (libpython3.14t.so.1.0+0x3366f0)
    #56 _PyFunction_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c (libpython3.14t.so.1.0+0x12c491) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #57 _PyObject_VectorcallDictTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:146:15 (libpython3.14t.so.1.0+0x12afe0) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #58 _PyObject_Call_Prepend /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:504:24 (libpython3.14t.so.1.0+0x12cad6) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #59 call_method /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/typeobject.c:2937:19 (libpython3.14t.so.1.0+0x24cee8) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #60 slot_tp_call /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/typeobject.c:10244:12 (libpython3.14t.so.1.0+0x24cd21) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #61 _PyObject_MakeTpCall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:242:18 (libpython3.14t.so.1.0+0x12b29e) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #62 _PyObject_VectorcallTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_call.h:167:16 (libpython3.14t.so.1.0+0x12beba) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #63 PyObject_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:327:12 (libpython3.14t.so.1.0+0x12beba)
    #64 _PyEval_EvalFrameDefault /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/generated_cases.c.h:2959:35 (libpython3.14t.so.1.0+0x3409a5) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #65 _PyEval_EvalFrame /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_ceval.h:121:16 (libpython3.14t.so.1.0+0x33622f) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #66 _PyEval_Vector /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/ceval.c:2083:12 (libpython3.14t.so.1.0+0x33622f)
    #67 PyEval_EvalCode /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/ceval.c:975:21 (libpython3.14t.so.1.0+0x33622f)
    #68 run_eval_code_obj /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/pythonrun.c:1365:12 (libpython3.14t.so.1.0+0x41c734) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #69 run_mod /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/pythonrun.c:1459:19 (libpython3.14t.so.1.0+0x41c734)
    #70 pyrun_file /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/pythonrun.c:1293:15 (libpython3.14t.so.1.0+0x417b12) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #71 _PyRun_SimpleFileObject /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/pythonrun.c:521:13 (libpython3.14t.so.1.0+0x417b12)
    #72 _PyRun_AnyFileObject /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/pythonrun.c:81:15 (libpython3.14t.so.1.0+0x417268) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #73 pymain_run_file_obj /work/.pyenv/sources/3.14.1t/Python-3.14.1/Modules/main.c:410:15 (libpython3.14t.so.1.0+0x45e72f) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #74 pymain_run_file /work/.pyenv/sources/3.14.1t/Python-3.14.1/Modules/main.c:429:15 (libpython3.14t.so.1.0+0x45e72f)
    #75 pymain_run_python /work/.pyenv/sources/3.14.1t/Python-3.14.1/Modules/main.c:694:21 (libpython3.14t.so.1.0+0x45da51) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #76 Py_RunMain /work/.pyenv/sources/3.14.1t/Python-3.14.1/Modules/main.c:775:5 (libpython3.14t.so.1.0+0x45da51)
    #77 pymain_main /work/.pyenv/sources/3.14.1t/Python-3.14.1/Modules/main.c:805:12 (libpython3.14t.so.1.0+0x45dfb8) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #78 Py_BytesMain /work/.pyenv/sources/3.14.1t/Python-3.14.1/Modules/main.c:829:12 (libpython3.14t.so.1.0+0x45e03c) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #79 main /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Programs/python.c:15:12 (python3.14+0xe8c9b) (BuildId: 7c5c6a0a3514fb58caefa389920bee1b45f9edcc)

  Thread T33 'Thread-30 (wrap' (tid=669, running) created by main thread at:
    #0 pthread_create <null> (python3.14+0x64bde) (BuildId: 7c5c6a0a3514fb58caefa389920bee1b45f9edcc)
    #1 do_start_joinable_thread /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/thread_pthread.h:289:14 (libpython3.14t.so.1.0+0x43ee6a) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #2 PyThread_start_joinable_thread /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/thread_pthread.h:331:9 (libpython3.14t.so.1.0+0x43ec8a) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #3 ThreadHandle_start /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Modules/_threadmodule.c:445:9 (libpython3.14t.so.1.0+0x500ca6) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #4 do_start_new_thread /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Modules/_threadmodule.c:1868:9 (libpython3.14t.so.1.0+0x500ca6)
    #5 thread_PyThread_start_joinable_thread /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Modules/_threadmodule.c:1991:14 (libpython3.14t.so.1.0+0x4ffab1) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #6 cfunction_call /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/methodobject.c:564:18 (libpython3.14t.so.1.0+0x1dad49) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #7 _PyObject_MakeTpCall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:242:18 (libpython3.14t.so.1.0+0x12b29e) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #8 _PyObject_VectorcallTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_call.h:167:16 (libpython3.14t.so.1.0+0x12beba) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #9 PyObject_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:327:12 (libpython3.14t.so.1.0+0x12beba)
    #10 _PyEval_EvalFrameDefault /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/generated_cases.c.h:3227:35 (libpython3.14t.so.1.0+0x341b47) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #11 _PyEval_EvalFrame /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_ceval.h:121:16 (libpython3.14t.so.1.0+0x3366f0) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #12 _PyEval_Vector /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/ceval.c:2083:12 (libpython3.14t.so.1.0+0x3366f0)
    #13 _PyFunction_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c (libpython3.14t.so.1.0+0x12c491) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #14 _PyObject_VectorcallDictTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:146:15 (libpython3.14t.so.1.0+0x12afe0) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #15 _PyObject_Call_Prepend /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:504:24 (libpython3.14t.so.1.0+0x12cad6) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #16 call_method /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/typeobject.c:2937:19 (libpython3.14t.so.1.0+0x24cee8) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #17 slot_tp_call /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/typeobject.c:10244:12 (libpython3.14t.so.1.0+0x24cd21) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #18 _PyObject_MakeTpCall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:242:18 (libpython3.14t.so.1.0+0x12b29e) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #19 _PyObject_VectorcallTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_call.h:167:16 (libpython3.14t.so.1.0+0x12beba) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #20 PyObject_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:327:12 (libpython3.14t.so.1.0+0x12beba)
    #21 _PyEval_EvalFrameDefault /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/generated_cases.c.h:3227:35 (libpython3.14t.so.1.0+0x341b47) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #22 _PyEval_EvalFrame /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_ceval.h:121:16 (libpython3.14t.so.1.0+0x3366f0) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #23 _PyEval_Vector /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/ceval.c:2083:12 (libpython3.14t.so.1.0+0x3366f0)
    #24 _PyFunction_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c (libpython3.14t.so.1.0+0x12c491) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #25 _PyObject_VectorcallDictTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:146:15 (libpython3.14t.so.1.0+0x12afe0) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #26 _PyObject_Call_Prepend /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:504:24 (libpython3.14t.so.1.0+0x12cad6) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #27 call_method /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/typeobject.c:2937:19 (libpython3.14t.so.1.0+0x24cee8) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #28 slot_tp_call /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/typeobject.c:10244:12 (libpython3.14t.so.1.0+0x24cd21) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #29 _PyObject_Call /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:361:18 (libpython3.14t.so.1.0+0x12c0e4) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #30 PyObject_Call /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:373:12 (libpython3.14t.so.1.0+0x12c187) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #31 _PyEval_EvalFrameDefault /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/generated_cases.c.h:2654:32 (libpython3.14t.so.1.0+0x33f61f) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #32 _PyEval_EvalFrame /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_ceval.h:121:16 (libpython3.14t.so.1.0+0x3366f0) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #33 _PyEval_Vector /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/ceval.c:2083:12 (libpython3.14t.so.1.0+0x3366f0)
    #34 _PyFunction_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c (libpython3.14t.so.1.0+0x12c491) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #35 _PyObject_VectorcallDictTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:146:15 (libpython3.14t.so.1.0+0x12afe0) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #36 _PyObject_Call_Prepend /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:504:24 (libpython3.14t.so.1.0+0x12cad6) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #37 call_method /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/typeobject.c:2937:19 (libpython3.14t.so.1.0+0x24cee8) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #38 slot_tp_call /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/typeobject.c:10244:12 (libpython3.14t.so.1.0+0x24cd21) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #39 _PyObject_MakeTpCall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:242:18 (libpython3.14t.so.1.0+0x12b29e) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #40 _PyObject_VectorcallTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_call.h:167:16 (libpython3.14t.so.1.0+0x12beba) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #41 PyObject_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:327:12 (libpython3.14t.so.1.0+0x12beba)
    #42 _PyEval_EvalFrameDefault /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/generated_cases.c.h:3227:35 (libpython3.14t.so.1.0+0x341b47) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #43 _PyEval_EvalFrame /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_ceval.h:121:16 (libpython3.14t.so.1.0+0x3366f0) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #44 _PyEval_Vector /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/ceval.c:2083:12 (libpython3.14t.so.1.0+0x3366f0)
    #45 _PyFunction_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c (libpython3.14t.so.1.0+0x12c491) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #46 _PyObject_VectorcallDictTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:146:15 (libpython3.14t.so.1.0+0x12afe0) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #47 _PyObject_Call_Prepend /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:504:24 (libpython3.14t.so.1.0+0x12cad6) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #48 call_method /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/typeobject.c:2937:19 (libpython3.14t.so.1.0+0x24cee8) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #49 slot_tp_call /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/typeobject.c:10244:12 (libpython3.14t.so.1.0+0x24cd21) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #50 _PyObject_MakeTpCall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:242:18 (libpython3.14t.so.1.0+0x12b29e) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #51 _PyObject_VectorcallTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_call.h:167:16 (libpython3.14t.so.1.0+0x12beba) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #52 PyObject_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:327:12 (libpython3.14t.so.1.0+0x12beba)
    #53 _PyEval_EvalFrameDefault /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/generated_cases.c.h:2959:35 (libpython3.14t.so.1.0+0x3409a5) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #54 _PyEval_EvalFrame /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_ceval.h:121:16 (libpython3.14t.so.1.0+0x3366f0) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #55 _PyEval_Vector /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/ceval.c:2083:12 (libpython3.14t.so.1.0+0x3366f0)
    #56 _PyFunction_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c (libpython3.14t.so.1.0+0x12c491) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #57 _PyObject_VectorcallDictTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:146:15 (libpython3.14t.so.1.0+0x12afe0) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #58 _PyObject_Call_Prepend /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:504:24 (libpython3.14t.so.1.0+0x12cad6) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #59 call_method /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/typeobject.c:2937:19 (libpython3.14t.so.1.0+0x24cee8) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #60 slot_tp_call /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/typeobject.c:10244:12 (libpython3.14t.so.1.0+0x24cd21) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #61 _PyObject_MakeTpCall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:242:18 (libpython3.14t.so.1.0+0x12b29e) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #62 _PyObject_VectorcallTstate /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_call.h:167:16 (libpython3.14t.so.1.0+0x12beba) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #63 PyObject_Vectorcall /work/.pyenv/sources/3.14.1t/Python-3.14.1/Objects/call.c:327:12 (libpython3.14t.so.1.0+0x12beba)
    #64 _PyEval_EvalFrameDefault /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/generated_cases.c.h:2959:35 (libpython3.14t.so.1.0+0x3409a5) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #65 _PyEval_EvalFrame /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_ceval.h:121:16 (libpython3.14t.so.1.0+0x33622f) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #66 _PyEval_Vector /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/ceval.c:2083:12 (libpython3.14t.so.1.0+0x33622f)
    #67 PyEval_EvalCode /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/ceval.c:975:21 (libpython3.14t.so.1.0+0x33622f)
    #68 run_eval_code_obj /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/pythonrun.c:1365:12 (libpython3.14t.so.1.0+0x41c734) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #69 run_mod /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/pythonrun.c:1459:19 (libpython3.14t.so.1.0+0x41c734)
    #70 pyrun_file /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/pythonrun.c:1293:15 (libpython3.14t.so.1.0+0x417b12) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #71 _PyRun_SimpleFileObject /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/pythonrun.c:521:13 (libpython3.14t.so.1.0+0x417b12)
    #72 _PyRun_AnyFileObject /work/.pyenv/sources/3.14.1t/Python-3.14.1/Python/pythonrun.c:81:15 (libpython3.14t.so.1.0+0x417268) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #73 pymain_run_file_obj /work/.pyenv/sources/3.14.1t/Python-3.14.1/Modules/main.c:410:15 (libpython3.14t.so.1.0+0x45e72f) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #74 pymain_run_file /work/.pyenv/sources/3.14.1t/Python-3.14.1/Modules/main.c:429:15 (libpython3.14t.so.1.0+0x45e72f)
    #75 pymain_run_python /work/.pyenv/sources/3.14.1t/Python-3.14.1/Modules/main.c:694:21 (libpython3.14t.so.1.0+0x45da51) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #76 Py_RunMain /work/.pyenv/sources/3.14.1t/Python-3.14.1/Modules/main.c:775:5 (libpython3.14t.so.1.0+0x45da51)
    #77 pymain_main /work/.pyenv/sources/3.14.1t/Python-3.14.1/Modules/main.c:805:12 (libpython3.14t.so.1.0+0x45dfb8) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #78 Py_BytesMain /work/.pyenv/sources/3.14.1t/Python-3.14.1/Modules/main.c:829:12 (libpython3.14t.so.1.0+0x45e03c) (BuildId: 3497f775552d873f4ec684582b3069ea1aaa8ce5)
    #79 main /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Programs/python.c:15:12 (python3.14+0xe8c9b) (BuildId: 7c5c6a0a3514fb58caefa389920bee1b45f9edcc)

SUMMARY: ThreadSanitizer: data race /work/.pyenv/sources/3.14.1t/Python-3.14.1/./Include/internal/pycore_call.h:130:5 in _PyVectorcall_FunctionInline
==================
tests/test_atomic_dict.py ...................ss.......................
tests/test_atomic_int.py ..........................................................
tests/test_atomic_ref.py .......
tests/test_basics.py .
tests/test_constants.py ..
tests/test_count_down_latch.py .............
tests/test_testing_thread_set.py .
tests/test_thread_handle.py ..
tests/test_thread_set.py .......

================== 133 passed, 2 skipped in 83.12s (0:01:23) ===================
ThreadSanitizer: reported 1 warnings
Error: Process completed with exit code 66.

    I understand the fix wasn't backported to 3.13 / 3.14, correct? Could the suppression be backported though, so that externally we don't get these false positives?

  11. ngoldbaum commented on Jan 15, 2026

    @ngoldbaum
    ngoldbaum
    on Jan 15, 2026
    Contributor

    I understand the fix wasn't backported to 3.13 / 3.14, correct? Could the suppression be backported though, so that externally we don't get these false positives?

    I think fixing #143880 will fix this.

  12. matthiasgoergens
    added a commit that references this issue on Aug 8, 2026

    Annotate two TSAN suppressions with their removal conditions

    6b8dede
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

3.13only security fixes3.14bugs and security fixestopic-free-threadingtype-bugAn unexpected behavior, bug, or error

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions