Refresh Screen Provided By curses.wrapper Causes Seg Fault (macOS, xcode 15 Apple supplied ncurses 6.0 breakage)

Crash report

What happened?

MacOS began pushing out updates to XCode Command Line Tools to install 15.0 recently. Upon updating I began having issues with curses. This happens with the Python provided by Apple. I'm not aware of the best way to communicate this issue to Apple, hopefully someone here knows who to ping or is watching.

Save the below as curses-segfault.py:

import curses

def main(stdscr):
    stdscr.refresh()

curses.wrapper(main)

Run the script in zsh in MacOS Terminal via:

/usr/bin/python3 curses-segfault.py

An easy way to see if you got the update is via the terminal by running:

softwareupdate --history

CPython versions tested on:

3.9

Operating systems tested on:

macOS

Output from running 'python -VV' on the command line:

python3 -VV Python 3.9.6 (default, Aug 11 2023, 19:44:49) [Clang 15.0.0 (clang-1500.0.40.1)]

Linked PRs

gh-109617: fix ncurses incompatibility on macOS with Xcode 15 #111258

I've been encountering this on main too after updating (the test_curses tests in our suite seem to trigger it). One really annoying side-effect is that it breaks your terminal after the crash:

% ./python.exe -m test --multiprocess 1 --use curses --verbose test_curses

0:00:00 load avg: 1.27 [1/1/1] test_curses process crashed (Exit code -11)
                                                                          test_has_extended_color_support (test.test_curses.MiscTests.test_has_extended_color_support) ... ok
    test_ncurses_version (test.test_curses.MiscTests.test_ncurses_version) ... ncurses_version = curses.ncurses_version(major=6, minor=0, patch=20150808)
                                                                                                                                                         ok
                                                                                                                                                           test_update_lines_cols (test.test_curses.MiscTests.test_update_lines_cols) ... ok
                                                                   test_alt (test.test_curses.TestAscii.test_alt) ... ok
                                                                                                                        test_ascii (test.test_curses.TestAscii.test_ascii) ... ok
        test_controlnames (test.test_curses.TestAscii.test_controlnames) ... ok
                                                                               test_ctrl (test.test_curses.TestAscii.test_ctrl) ... ok
                                                                                                                                      test_ctypes (test.test_curses.TestAscii.test_ctypes) ... ok
                        test_unctrl (test.test_curses.TestAscii.test_unctrl) ... ok
                                                                                   TERM=xterm-256color
                                                                                                      test_attributes (test.test_curses.TestCurses.test_attributes) ... ok
 test_background (test.test_curses.TestCurses.test_background) ... ok
                                                                     test_beep (test.test_curses.TestCurses.test_beep) ... ok
                                                                                                                             test_borders_and_lines (test.test_curses.TestCurses.test_borders_and_lines) ... ok
                                      test_chgat (test.test_curses.TestCurses.test_chgat) ... ok
                                                                                                test_clear (test.test_curses.TestCurses.test_clear) ... ok
                                                                                                                                                          test_color_attrs (test.test_curses.TestCurses.test_color_attrs) ... ok
                                                       test_color_content (test.test_curses.TestCurses.test_color_content) ... ok
                                                                                                                                 test_create_windows (test.test_curses.TestCurses.test_create_windows) ...

                                 == Tests result: FAILURE ==

                                                            1 test failed:
                                                                              test_curses

                                                                                         Total duration: 198 ms
                                                                                                               Total tests: run=0
                                                                                                                                 Total test files: run=1/1 failed=1
                                                                                                                                                                   Result: FAILURE
         %

I'm not a curses expert, but I'm not sure that this is something we can fix on our end, so this should probably be closed. It might be worth disabling the crashy tests if we can detect the broken curses version, though?

@Yhg1s

CC @Yhg1s as a curses expert.

The affected tests appear to be test_create_windows, test_move_cursor, test_output_character, test_refresh, and test_refresh_control.

FWIW this problem should not arise when using a python from a macOS python.org installer or MacPorts as they supply their own version of ncurses and do not use the system version.

@timway

FWIW this problem should not arise when using a python from a macOS python.org installer or MacPorts as they supply their own version of ncurses and do not use the system version.

I can confirm this. I was only able to reproduce it for @timway when using the system version of python.

Downgrading from Xcode 15.0 (curses.ncurses_version(major=6, minor=0, patch=20150808)) to 14.3 (curses.ncurses_version(major=5, minor=7, patch=20081102)) fixes it for me.

It sounds like they shipped a potentially broken version of ncurses embedded within xcode 15?

We've been building and linking with ncurses >= 6 in the rest of the world since 2015 without this problem...

It is really strange behavior on Apple's part to "upgrade" to providing 6.0 from 2015 in their 2023 toolchain when ncurses 6.4 came out last year.

We're at the point where it'd be fine for CPython to simply drop the ability to build and link against 6.0 in 3.13+ given that no supported Linux distro platform will be using it anymore by the time we release 3.13. (RHEL 8 shipped with ncurses 6.1 making that a reasonable minimum) -- I do not expect or encourage anyone to go through and do that to the _curses module code.

Recommendation: do what we do with python.org builds, never build against apple's ncurses.

If a workaround for this isn't obvious to someone with a macos debugger, it is reasonable to have a configure check blocklist the xcode 15 supplied ncurses and refuse to use it. Spending our time dealing with brazenly outdated libraries embedded in someone elses toolchain that we don't ship our own builds with isn't worthwhile (it'd never end).

Thank you all for looking into this - as next steps I've found the Feedback Assistant and the general feedback form on Apple's website. I'll work to post to those methods and provide any feedback here.

I could also look at writing a small C example and see if I can confirm if the problem is in their ncurses implementation and fully removed from Python or not.

If you want to require a specific version or recommend libedit or ncurses on macOS, you can update the doc: https://docs.python.org/dev/using/configure.html

Issues like this should also be reported to Apple as this is a crash with their copy of Python.

I've filed FB13196764 about this.

if (HAVE_CURSES_IS_PAD_RUNTIME) {
    return is_pad(w);
} else {
    // ... what goes here? ...
}

Don't define is_pad macro if the function is not available. Modules/_cursesmodule.c can already be built without is_pad() function. Example in _curses_window_echochar_impl():

#ifdef py_is_pad                        
    if (py_is_pad(self->win)) {                                          
        return PyCursesCheckERR(pechochar(self->win, ch_ | (attr_t)attr),   
                                "echochar");
    }                                                                   
    else                                                       
#endif                                                                   
        return PyCursesCheckERR(wechochar(self->win, ch_ | (attr_t)attr),
                                "echochar");

Or well, just always return 0?

Modules/_cursesmodule.c can already be built without is_pad() function

Eh, kinda, but not really in a way that solves the problem. It falls back in two different ways, depending on checks done at configure time:

if is_pad is a function, define py_is_pad to use that;
otherwise: if is_pad is not available¹, but window flags are inspectable, it defines py_is_pad to check flags. This is what happens on older Apple SDKs (which work!) and on any system with ncurses < 5.8.
otherwise: don't define py_is_pad at all, and ignore the distinction between windows and pads. This is meant to support some other curses, and is never the case with ncurses.

What you are suggesting is 3, but ncurses doesn't like that, e.g. curs_pad(3x):

   It  is  not  legal  to  call  wrefresh  with  a pad as an argument; the
   routines prefresh or pnoutrefresh should be called instead.

In the example you pasted, this would end up calling wechochar on a pad instead of pechochar, which might not work as expected, or impact performance. Note that this might break in a way that is not covered by test_curses. I don't see tests on pads stuff.

On a personal note, I'm getting confused by this conversation. I suggested a fix and tested it. It took some work to figure out the different cases, and I documented that work in this issue. It's not really complicated, only a bit messy with all the historical cruft. But the comments don't address the proposed solution, and seem to argue that some other unspecified solution should be preferable. I have a bucketful of impostor syndrome to handle, and I'd prefer not to feed it more.

yeah, in the current setup, with ncurses, either is_pad is a function (detected by configure, which would set HAVE_CURSES_IS_PAD) or it's not used at all. There is no path that uses the is_pad macro from ncurses.h, which is legal and documented. ↩

I understand that with Xcode 15.0, the is_pad() function is not available, but win->_flags & _ISPAD can be tested if the NCURSES_OPAQUE macro is set to 0. For me, it's surprising to have to tune NCURSES_OPAQUE macro and to inspect a structure member starting with an underscore.

But apparently, this issue affects many macOS issues, it's annoying, and I should stop nitpicking. So just use NCURSES_OPAQUE=0 on macOS, and it should "just works", no?

Does someone want to propose a PR? Nobody proposed a PR so far, no?

I understand that with Xcode 15.0, the is_pad() function is not available, but win->_flags & _ISPAD can be tested if the NCURSES_OPAQUE macro is set to 0. For me, it's surprising to have to tune NCURSES_OPAQUE macro and to inspect a structure member starting with an underscore.

Nope. "Inspecting a structure member starting with an underscore" is what cursesmodule.c does now and has been doing for the last couple decades. If you built on Mac before Xcode 15, you used that.

There is a convenient and documented is_pad macro, defined in Ncurses public headers for a long while now. Since Apple now includes this version, we can use it.

Does someone want to propose a PR? Nobody proposed a PR so far, no?

I asked for early feedback on a diff a couple times above. I will open a PR in a little bit. Maybe it will reduce the confusion.

Suggestion: let's rename the issue to something that better describes the problem.

Is there a workaround? I want to get back to work on my project.

Is there a workaround? I want to get back to work on my project.

I'm no longer able to replicate the issue with Sonoma 14.2.1 and Command Line Tools for Xcode 15.1.

Alternatively, versions from Python.org shouldn't have the issue.

Edit: It looks like 14.2 got some patches for ncurses behavior (https://support.apple.com/en-us/HT214036). While they don't specifically mention this as an issue it does appear to resolve it for me at least.

Yes, downloading the latest directly from python.org made the problem go away. It would be nice if there were something I could do in code, other than telling customers "you need to install a new version of Python on your system"

gh-109617: fix ncurses incompatibility on macOS with Xcode 15 (#111258)

pythongh-109617: fix ncurses incompatibility on macOS with Xcode 15 (p…

Squashed commit of the following:

FYI, as of the Python 3.14.0 alpha 5 pre-release and the upcoming 3.13.3 and 3.12.10 releases, python.org macOS installers now build and link with ncurses 6.5, the current stable release, and no longer with either ncurses 5.9 or with the Apple-supplied and -patched macOS system ncurses.

Since I think the original issue reported here has been addressed by PR #111258 and possibly by Apple - at least there don't seem to be recent reports of people being bitten by this - let's close this issue as fixed.

Navigation Menu

Repository navigation

Refresh Screen Provided By `curses.wrapper` Causes Seg Fault (macOS, xcode 15 Apple supplied ncurses 6.0 breakage) #109617

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Linked PRs

16 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Refresh Screen Provided By curses.wrapper Causes Seg Fault (macOS, xcode 15 Apple supplied ncurses 6.0 breakage) #109617

Description

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Linked PRs

Activity

brandtbucher commented on Sep 20, 2023

brandtbucher commented on Sep 20, 2023

brandtbucher commented on Sep 20, 2023

ned-deily commented on Sep 20, 2023

cdwrobg commented on Sep 20, 2023

brandtbucher commented on Sep 20, 2023

gpshead commented on Sep 20, 2023

gpshead commented on Sep 20, 2023

timway commented on Sep 20, 2023

vstinner commented on Sep 21, 2023

ronaldoussoren commented on Sep 22, 2023

16 remaining items

vstinner commented on Oct 20, 2023

sorcio commented on Oct 20, 2023

Footnotes

vstinner commented on Oct 20, 2023

sorcio commented on Oct 24, 2023

sorcio commented on Oct 29, 2023

efalk commented on Jan 2, 2024

timway commented on Jan 2, 2024

efalk commented on Jan 2, 2024

ned-deily commented on Apr 7, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Refresh Screen Provided By `curses.wrapper` Causes Seg Fault (macOS, xcode 15 Apple supplied ncurses 6.0 breakage) #109617