Skip to content

segfault on main for "./python.exe -m blurb" #109198

Closed
Closed
segfault on main for "./python.exe -m blurb"#109198

Description

@iritkatriel
iritkatriel
opened on Sep 9, 2023
% ./python.exe -m blurb
zsh: segmentation fault  ./python.exe -m blurb

MacOS, clean build.

Activity

  1. iritkatriel commented on Sep 9, 2023

    @iritkatriel
    iritkatriel
    on Sep 9, 2023
    MemberAuthor
    (lldb) run -m blurb
Process 13631 launched: '/Users/iritkatriel/src/cpython/python.exe' (x86_64)
Process 13631 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x20000000481)
    frame #0: 0x00000001001d467a python.exe`_PyEval_EvalFrameDefault(tstate=0x00000001005fa780, frame=<unavailable>, throwflag=0) at generated_cases.c.h:2481:13 [opt]
   2478                uint16_t index = read_u16(&next_instr[3].cache);
   2479                PyTypeObject *tp = Py_TYPE(owner);
   2480                assert(type_version != 0);
-> 2481                DEOPT_IF(tp->tp_version_tag != type_version, LOAD_ATTR);
   2482                assert(tp->tp_flags & Py_TPFLAGS_MANAGED_DICT);
   2483                PyDictOrValues dorv = *_PyObject_DictOrValuesPointer(owner);
   2484                DEOPT_IF(_PyDictOrValues_IsValues(dorv), LOAD_ATTR);
Target 0: (python.exe) stopped.

  2. chgnrdv commented on Sep 9, 2023

    @chgnrdv
    chgnrdv
    on Sep 9, 2023
    Contributor

    On my Debian attempt to install blurb segfaulted. More specifically, it happens when I try to import pip module.

    Backtrace:

    _PyEval_EvalFrameDefault (tstate=tstate@entry=0x555555bf42f8 <_PyRuntime+508728>, frame=0x7ffff7fc1378, throwflag=throwflag@entry=0) at Python/generated_cases.c.h:2492
2492	            PyTypeObject *tp = Py_TYPE(owner);
(gdb) bt
#0  _PyEval_EvalFrameDefault (tstate=tstate@entry=0x555555bf42f8 <_PyRuntime+508728>, frame=0x7ffff7fc1378, throwflag=throwflag@entry=0)
    at Python/generated_cases.c.h:2492
#1  0x00005555557e6c70 in _PyEval_EvalFrame (throwflag=0, frame=<optimized out>, tstate=0x555555bf42f8 <_PyRuntime+508728>) at ./Include/internal/pycore_ceval.h:107
#2  _PyEval_Vector (tstate=tstate@entry=0x555555bf42f8 <_PyRuntime+508728>, func=func@entry=0x7ffff7506090, locals=locals@entry=0x7ffff778e390, args=args@entry=0x0, 
    argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:1632
#3  0x00005555557e6d13 in PyEval_EvalCode (co=co@entry=0x7ffff74d8ca0, globals=globals@entry=0x7ffff778e390, locals=locals@entry=0x7ffff778e390) at Python/ceval.c:582
#4  0x00005555557ba0b6 in builtin_exec_impl (module=module@entry=0x7ffff7912870, source=0x7ffff74d8ca0, globals=0x7ffff778e390, locals=0x7ffff778e390, closure=0x0)
    at Python/bltinmodule.c:1114
#5  0x00005555557ba273 in builtin_exec (module=0x7ffff7912870, args=<optimized out>, nargs=2, kwnames=<optimized out>) at Python/clinic/bltinmodule.c.h:539
#6  0x00005555556e2521 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7ffff7912ed0, args=0x7ffff74e5288, nargsf=<optimized out>, kwnames=0x0)
    at Objects/methodobject.c:441
#7  0x0000555555678287 in _PyVectorcall_Call (tstate=tstate@entry=0x555555bf42f8 <_PyRuntime+508728>, func=0x5555556e2410 <cfunction_vectorcall_FASTCALL_KEYWORDS>, 
    callable=callable@entry=0x7ffff7912ed0, tuple=tuple@entry=0x7ffff74e5270, kwargs=kwargs@entry=0x7ffff778ee70) at Objects/call.c:273
#8  0x0000555555678633 in _PyObject_Call (tstate=0x555555bf42f8 <_PyRuntime+508728>, callable=callable@entry=0x7ffff7912ed0, args=args@entry=0x7ffff74e5270, 
    kwargs=kwargs@entry=0x7ffff778ee70) at Objects/call.c:348
#9  0x000055555567868c in PyObject_Call (callable=callable@entry=0x7ffff7912ed0, args=args@entry=0x7ffff74e5270, kwargs=kwargs@entry=0x7ffff778ee70)
    at Objects/call.c:373
#10 0x00005555557e3daa in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x555555bf42f8 <_PyRuntime+508728>, frame=0x7ffff7fc12f0, throwflag=throwflag@entry=0)
    at Python/generated_cases.c.h:4698
#11 0x00005555557e6c70 in _PyEval_EvalFrame (throwflag=0, frame=<optimized out>, tstate=0x555555bf42f8 <_PyRuntime+508728>) at ./Include/internal/pycore_ceval.h:107
#12 _PyEval_Vector (tstate=0x555555bf42f8 <_PyRuntime+508728>, func=0x7ffff7961190, locals=locals@entry=0x0, args=0x7fffffffd9e0, argcount=2, kwnames=0x0)
    at Python/ceval.c:1632
#13 0x0000555555675c01 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:413
#14 0x000055555567677a in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fffffffd9e0, callable=0x7ffff7961190, tstate=0x555555bf42f8 <_PyRuntime+508728>)
    at ./Include/internal/pycore_call.h:187
#15 object_vacall (tstate=tstate@entry=0x555555bf42f8 <_PyRuntime+508728>, base=base@entry=0x0, callable=0x7ffff7961190, vargs=vargs@entry=0x7fffffffda50)
    at Objects/call.c:819
#16 0x0000555555676964 in PyObject_CallMethodObjArgs (obj=0x0, name=<optimized out>) at Objects/call.c:880
#17 0x000055555582f4c9 in import_find_and_load (tstate=tstate@entry=0x555555bf42f8 <_PyRuntime+508728>, abs_name=abs_name@entry=0x7ffff77ed210) at Python/import.c:2747
#18 0x0000555555835596 in PyImport_ImportModuleLevelObject (name=name@entry=0x7ffff77ed210, globals=<optimized out>, locals=locals@entry=0x7ffff778e270, 
    fromlist=fromlist@entry=0x555555b58620 <_Py_NoneStruct>, level=0) at Python/import.c:2827
#19 0x00005555557c051c in import_name (tstate=tstate@entry=0x555555bf42f8 <_PyRuntime+508728>, frame=frame@entry=0x7ffff7fc1020, name=0x7ffff77ed210, 
    fromlist=fromlist@entry=0x555555b58620 <_Py_NoneStruct>, level=level@entry=0x555555b7cc38 <_PyRuntime+19576>) at Python/ceval.c:2436
#20 0x00005555557d93a1 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x555555bf42f8 <_PyRuntime+508728>, frame=0x7ffff7fc1020, throwflag=throwflag@entry=0)
    at Python/generated_cases.c.h:2936
#21 0x00005555557e6c70 in _PyEval_EvalFrame (throwflag=0, frame=<optimized out>, tstate=0x555555bf42f8 <_PyRuntime+508728>) at ./Include/internal/pycore_ceval.h:107
#22 _PyEval_Vector (tstate=tstate@entry=0x555555bf42f8 <_PyRuntime+508728>, func=func@entry=0x7ffff77ebad0, locals=locals@entry=0x7ffff778e270, args=args@entry=0x0, 
--Type <RET> for more, q to quit, c to continue without paging--
    argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:1632
#23 0x00005555557e6d13 in PyEval_EvalCode (co=co@entry=0x7ffff749bc40, globals=globals@entry=0x7ffff778e270, locals=locals@entry=0x7ffff778e270) at Python/ceval.c:582
#24 0x0000555555868194 in run_eval_code_obj (tstate=tstate@entry=0x555555bf42f8 <_PyRuntime+508728>, co=co@entry=0x7ffff749bc40, globals=globals@entry=0x7ffff778e270, 
    locals=locals@entry=0x7ffff778e270) at Python/pythonrun.c:1730
#25 0x0000555555868b8e in run_mod (mod=<optimized out>, filename=filename@entry=0x7ffff778b290, globals=0x7ffff778e270, locals=0x7ffff778e270, 
    flags=flags@entry=0x7fffffffe088, arena=arena@entry=0x7ffff77ddfc0) at Python/pythonrun.c:1751
#26 0x0000555555868d4d in PyRun_InteractiveOneObjectEx (fp=fp@entry=0x7ffff7e33980 <_IO_2_1_stdin_>, filename=filename@entry=0x7ffff778b290, 
    flags=flags@entry=0x7fffffffe088) at Python/pythonrun.c:287
#27 0x000055555586bb2d in _PyRun_InteractiveLoopObject (fp=fp@entry=0x7ffff7e33980 <_IO_2_1_stdin_>, filename=filename@entry=0x7ffff778b290, 
    flags=flags@entry=0x7fffffffe088) at Python/pythonrun.c:138
#28 0x000055555586c0ad in _PyRun_AnyFileObject (fp=fp@entry=0x7ffff7e33980 <_IO_2_1_stdin_>, filename=filename@entry=0x7ffff778b290, closeit=closeit@entry=0, 
    flags=flags@entry=0x7fffffffe088) at Python/pythonrun.c:73
#29 0x000055555586c188 in PyRun_AnyFileExFlags (fp=0x7ffff7e33980 <_IO_2_1_stdin_>, filename=filename@entry=0x55555591f667 "<stdin>", closeit=closeit@entry=0, 
    flags=flags@entry=0x7fffffffe088) at Python/pythonrun.c:105
#30 0x0000555555894adf in pymain_run_stdin (config=config@entry=0x555555b8eee0 <_PyRuntime+93984>) at Modules/main.c:521
#31 0x0000555555895525 in pymain_run_python (exitcode=exitcode@entry=0x7fffffffe12c) at Modules/main.c:614
#32 0x000055555589556c in Py_RunMain () at Modules/main.c:689
#33 0x00005555558955c0 in pymain_main (args=args@entry=0x7fffffffe170) at Modules/main.c:719
#34 0x0000555555895635 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:743
#35 0x00005555555cc73e in main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15

  3. vstinner commented on Sep 9, 2023

    @vstinner
    vstinner
    on Sep 9, 2023
    Member

    You may try git bisect to see when the regression was introduced.

  4. iritkatriel commented on Sep 10, 2023

    @iritkatriel
    iritkatriel
    on Sep 10, 2023
    MemberAuthor
    commit 0858328ca2457ae95715eb93e347d5c0547bec6f (HEAD)
Author: Mark Shannon <mark@hotpy.org>
Date:   Thu Sep 7 14:39:03 2023 +0100

    GH-108614: Add `RESUME_CHECK` instruction (GH-108630)

  5. iritkatriel commented on Sep 10, 2023

    @iritkatriel
    iritkatriel
    on Sep 10, 2023
    MemberAuthor

    CC @markshannon

  6. markshannon commented on Sep 10, 2023

    @markshannon
    markshannon
    on Sep 10, 2023
    Member

    I cannot reproduce on WSL, Intel x86-64.

    What platform (including hardware and compiler) does this fail on, and how was Python configured?

    What is the value of owner and tp? (it looks like tp does not point to a valid PyTypeObject)

  7. iritkatriel commented on Sep 10, 2023

    @iritkatriel
    iritkatriel
    on Sep 10, 2023
    MemberAuthor

    MacOs Ventura. 2.6 GHz 6-Core Intel Core i7. Totally standard macOS debug configuration.

    p owner
(PyObject *) $3 = 0x00000000ffffffff

  8. iritkatriel commented on Sep 10, 2023

    @iritkatriel
    iritkatriel
    on Sep 10, 2023
    MemberAuthor
    Screenshot 2023-09-10 at 14 53 08

  9. chgnrdv commented on Sep 10, 2023

    @chgnrdv
    chgnrdv
    on Sep 10, 2023
    Contributor

    Sorry, I fell asleep before my bisect script found bad commit 😄

    @markshannon
    Linux machine 5.10.0-25-amd64 #1 SMP Debian 5.10.191-1 (2023-08-16) x86_64 GNU/Linux
    gcc (Debian 10.2.1-6) 10.2.1 20210110
    Python was configured with --with-pydebug.
    owner is 0xffffffff.

    If it doesn't fail with pip invoked as module with -m, try importing pip._internal or pip._internal.utils - this is where it fails on my installation, according to backtrace.

    UPD:
    My pip version is 23.1.2 Upgrading pip to 23.2.1 through ./python -m ensurepip -U made issue go away.

  10. hugovk commented on Sep 10, 2023

    @hugovk
    hugovk
    on Sep 10, 2023
    Member
    Screenshot 2023-09-10 at 14 53 08

    I'm on macOS Ventura 13.5.2 with Apple M2 chip.

  11. vstinner commented on Sep 10, 2023

    @vstinner
    vstinner
    on Sep 10, 2023
    Member

    I failed to reproduce the issue on Windows, Linux, FreeBSD.

    I built Python in debug mode ./configure --with-pydebug (and ``-d` option on Windows). I created a venv, downgrade pip to 23.1.2, run Python and import pip._internal.utils: no crash. I installed blurb in the venv and ran "env/bin/python -m blurb": no crash.

  12. chgnrdv commented on Sep 11, 2023

    @chgnrdv
    chgnrdv
    on Sep 11, 2023
    Contributor

    After upgrading pip to 23.2.1 and downgrading back to 23.1.2 I wasn't able to reproduce the issue.

    Steps to trigger issue [again] that worked on my machine:

    • uninstall pip if previously installed (I used rm -rf ~/.local/lib/python3.13/site-packages/pip*)
    • checkout a commit before the bad one (d485551)
    • build
    • revert f443b54
    • install pip through ./python -m ensurepip
    • git checkout main
    • build again
    • run ./python -m pip

  13. vstinner commented on Sep 11, 2023

    @vstinner
    vstinner
    on Sep 11, 2023
    Member

    Oh ok, with these commands, I can now trigger the crash on Linux!

    I converted your instructions to a script:

    rm -rf ~/.local/lib/python3.13/site-packages/pip*

# Install old pip 23.1.2 in ~/.local/ using ensurepip
git switch -c install-old-pip d485551c9d1792ff3539eef1d6374bd4c01dcd5d
git revert f443b54a2f14e386a91fe4b09f41a265445008b8
git clean -fdx && ./configure --with-pydebug && make 
./python -m ensurepip

# Build new Python
git checkout main
git clean -fdx && ./configure --with-pydebug && make 

# Trigger crash
./python -m pip list

    I tried to copy ~/.local/lib/python3.13/site-packages/pip/ in the Python directory, but if I do that, the bug goes away!

  14. vstinner commented on Sep 11, 2023

    @vstinner
    vstinner
    on Sep 11, 2023
    Member

    If I remove .pyc files from ~/.local/lib/python3.13/site-packages/pip/, the bug goes away.

    @markshannon: Maybe you just forgot to increase the importlib MAGIC number to invalidate old PYC files?

  15. vstinner commented on Sep 11, 2023

    @vstinner
    vstinner
    on Sep 11, 2023
    Member

    @iritkatriel: Can you try to remove blurb .pyc files (maybe create a copy of the blurb/ directory before, to reproduce the bug later if needed)? Does it fix your issue?

  16. vstinner commented on Sep 11, 2023

    @vstinner
    vstinner
    on Sep 11, 2023
    Member

    The commit 0858328 changes most bytecodes. For example, RETURN_GENERATOR was 48 before and became 48 with this change. Old PYC files are now likely to crash Python.

    In the script before, ./python -m ensurepip compiles .py files to .pyc files with the old Python:

    $ ./python -m ensurepip 
$ find ~/.local/lib/python3.13/site-packages/pip/ -name "*.pyc"
/home/vstinner/.local/lib/python3.13/site-packages/pip/_internal/cli/__pycache__/__init__.cpython-313.pyc
/home/vstinner/.local/lib/python3.13/site-packages/pip/_internal/cli/__pycache__/autocompletion.cpython-313.pyc
/home/vstinner/.local/lib/python3.13/site-packages/pip/_internal/cli/__pycache__/base_command.cpython-313.pyc
/home/vstinner/.local/lib/python3.13/site-packages/pip/_internal/cli/__pycache__/cmdoptions.cpython-313.pyc
(...)

    Fix for this bug:

    diff --git a/Lib/importlib/_bootstrap_external.py b/Lib/importlib/_bootstrap_external.py
index 0717e202ec..7a83a58d08 100644
--- a/Lib/importlib/_bootstrap_external.py
+++ b/Lib/importlib/_bootstrap_external.py
@@ -455,6 +455,7 @@ def _write_atomic(path, data, mode=0o666):
 #     Python 3.13a1 3557 (Make the conversion to boolean in jumps explicit)
 #     Python 3.13a1 3558 (Reorder the stack items for CALL)
 #     Python 3.13a1 3559 (Generate opcode IDs from bytecodes.c)
+#     Python 3.13a1 3560 (Add RESUME_CHECK instruction)
 
 #     Python 3.14 will start with 3600
 
@@ -471,7 +472,7 @@ def _write_atomic(path, data, mode=0o666):
 # Whenever MAGIC_NUMBER is changed, the ranges in the magic_values array
 # in PC/launcher.c must also be updated.
 
-MAGIC_NUMBER = (3559).to_bytes(2, 'little') + b'\r\n'
+MAGIC_NUMBER = (3560).to_bytes(2, 'little') + b'\r\n'
 
 _RAW_MAGIC_NUMBER = int.from_bytes(MAGIC_NUMBER, 'little')  # For import.c

  17. vstinner commented on Sep 11, 2023

    @vstinner
    vstinner
    on Sep 11, 2023
    Member

    I wrote PR #109247 to fix this issue.

  18. vstinner commented on Sep 11, 2023

    @vstinner
    vstinner
    on Sep 11, 2023
    Member

    Fixed by commit 3b2ecbc.

  19. vstinner
    closed this as completedon Sep 11, 2023

  20. vstinner commented on Sep 11, 2023

    @vstinner
    vstinner
    on Sep 11, 2023
    Member

    The relationship between the code and the crash was not obvious at all :-( Thanks @iritkatriel and @chgnrdv for the reproducers, it was really helpful! I'm glad that this issue is now fixed.

  21. vstinner
    reopened this on Sep 11, 2023
  22. vstinner
    closed this as completedon Sep 11, 2023

  23. erlend-aasland commented on Sep 11, 2023

    @erlend-aasland
    erlend-aasland
    on Sep 11, 2023
    Contributor

    Good sleuthing, Victor!

  24. vstinner commented on Sep 11, 2023

    @vstinner
    vstinner
    on Sep 11, 2023
    Member

    Well, @iritkatriel and @chgnrdv did the hard part, work on a reproducer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions