Repository navigation
Modifications to support iOS as a cross-compilation target #67858
Description
| BPO | 23670 |
|---|---|
| Nosy | @doko42, @ronaldoussoren, @ncoghlan, @ned-deily, @bitdancer, @freakboy3742, @rovitotv, @Lukasa, @rkuska, @mpaolini, @moreati, @controversial, @Mariatta |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/ned-deily'
closed_at = None
created_at = <Date 2015-03-15.06:25:47.056>
labels = ['type-feature', '3.7', 'build']
title = 'Modifications to support iOS as a cross-compilation target'
updated_at = <Date 2019-09-04.00:08:40.855>
user = 'https://github.com/freakboy3742'bugs.python.org fields:
activity = <Date 2019-09-04.00:08:40.855>
actor = 'josh.r'
assignee = 'ned.deily'
closed = False
closed_date = None
closer = None
components = ['Build', 'Cross-Build']
creation = <Date 2015-03-15.06:25:47.056>
creator = 'freakboy3742'
dependencies = []
files = ['38493', '39207', '39286', '39824', '39862', '41376', '42059', '42060']
hgrepos = ['300']
issue_num = 23670
keywords = ['patch']
message_count = 28.0
messages = ['238125', '238127', '238129', '238133', '238134', '238138', '238156', '238167', '238190', '238191', '238197', '238211', '240307', '241187', '241189', '242040', '242547', '245903', '246266', '256780', '256797', '261093', '261095', '267030', '267064', '267092', '284441', '305628']
nosy_count = 15.0
nosy_names = ['doko', 'ronaldoussoren', 'ncoghlan', 'ned.deily', 'r.david.murray', 'cvrebert', 'freakboy3742', 'Todd.Rovito', 'bkabrda', 'Lukasa', 'rkuska', 'mpaolini', 'Alex.Willmer', 'Luke Taylor', 'Mariatta']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue23670'
versions = ['Python 3.4', 'Python 3.5', 'Python 3.6', 'Python 3.7']
Activity
freakboy3742 commentedon Mar 15, 2015
Proposal: iOS should be a supported platform for Python development.
The attached patch is a first pass at a patch to achieve this. It is a single patch against Python 3.4.2 sources, and requires no pre- or post-configure modifications.
Supporting iOS requires multiple builds - one for each of the hardware platforms that iOS supports (x86_64 simulator, ARMv7 and ARM64). These separate builds must then be merged into a single "fat" framework. The patch contains an iOS directory with a "meta" Makefile that manages this build-and-merge process. See the README in the iOS directory for details on usage.
The patch also introduces a new 'ios' platform type.
A sample XCode project for an iOS app is also provided as part of the patch.
iOS/README contains a couple of other notes about the build and the approach taken.
There are some known problems/limitations with this patch:
It's a patch against 3.4.2, not hg trunk
The code doesn't currently compile for ARMv7. In order to support ARM64, it has been necessary to use an unreleased trunk version of libffi; however, this version is currently broken for ARMv7. Older versions of libffi (including the formal 3.2.1 release) *do* work.
The patch doesn't currently provide any way to run the test suite on the target platform. Testing is currently based on a simple smoke test of some basic features.
So - the patch isn't ready for commit to trunk. I'm presenting it for the purposes of getting feedback on the broad approach while I continue to resolve the known issues.
ncoghlan commentedon Mar 15, 2015
Russell, just checking before I change the issue title: do you mean supporting iOS as a cross-compilation target? "Development" is ambiguous here, as you could mean development *of* CPython, rather than *in* Python, and I assume development *for* iOS largely takes place on x86_64 Mac OS X, Windows and Linux systems.
Slavek, Robert - assuming my understanding of the change is correct, I think would be a very interesting CPython enhancement from a Fedora Workstation perspective.
ned-deily commentedon Mar 15, 2015
As a cross-compilation target. From a first quick look at it, it appears the patch requires a current Mac OS X system to build for iOS; the necessary standard build tools and SDKs for iOS are only available on OS X. These are the same build tools used for OS X builds. We already support universal builds for multiple architectures on OS X directly in one pass; unfortunately, the iOS builds require two different SDKs, one for running on the OS X-based simulator and one for the native platform archs which, I assume, is why Russell has gone for the separate builds for each arch and lipo-ed them together. I don't have an opinion yet about the use of the Setup.local configurations rather than modifying setup.py. I appreciate trying to keep the changes for a patch like this as isolated as possible. But, long term, that might not be the best approach assuming there is eventually agreement to fully support iOS as a standard platform (via cross-compilation). The bulk of the patch is the new version of libffi; presumably that will eventually be released upstream in the standard version of libffi so that having a separate copy wouldn't be required? In any case, as a work-in-progress, the patch is certainly nicely done.
[-]Modifications to support iOS as a development platform[/-][+]Modifications to support iOS as a cross-compilation target[/+]freakboy3742 commentedon Mar 15, 2015
Nick: you are correct - these are changes to support iOS as a cross-compilation target, so you can run your Python code on an iOS device. Apologies for the confusion in nomenclature.
Ned: You're correct that the build needs to be run on an OS/X machine, and needs to be done in multiple passes because of the need for multiple SDKs.
I've taken the approach of using Setup.local for two reasons.
Firstly, iOS requires a "fully embedded" Python, because Apple doesn't approve of the use of dynamic loading in apps. Setup.local was the best way I found to force modules to be statically linked into the main library.
Secondly, because of the nature of the end-deployment, I imagine it's inevitable that serious end-users will want to customise their Python build for specific applications. AppStore submission rules prohibit running code that wasn't shipped with the app (i.e., Apple won't let you do an end-run around their app approval processes), so the set of modules you're actually using will be a known quantity. Since a "full" iOS Python build is 34MB or so, much of that bulk will be for binary support of modules that you aren't using, providing an easily configurable way to prune out unused modules from the built product seemed like a useful facility. There's also the issue of providing support for the standard library modules (like SSL) that have dependencies that can't be easily provided or specified out-of-the-box.
If there's a better/more palatable approach than using Setup.local, I'm happy to look into it.
The libffi code is a direct copy from the current (as of a week or so ago) libffi trunk. There have been some bug fixes applied since 3.2.1 was released in november that are necessary to support ctypes on ARM64 (but, as I flagged in my original comment, introduces bugs for ARMv7). The libffi repo suggests the next version will be 4.0, but I have no idea when that will be finalised. I'm currently trying to get their release plans confirmed.
As with OS/X libffi support, the source code needs to be generated, rather than just "./configure && make"'d. That said, the libffi_darwin code could be re-used, as the x86_64 components are the same for Darwin as the iOS simulator.
ned-deily commentedon Mar 15, 2015
Ack, much to consider. Adding doko as I believe he has been most closely following developments with libffi releases.
ned-deily commentedon Mar 15, 2015
A thought, primarily as a note to myself for further investigation: the current OS X framework build support and, to a lesser extent, universal build support have added a fair amount of special-case cruft to the build process, primarily in configure.ac and the main Makefile. Over the years, the build tools have evolved and gotten smarter (and more general) about SDK support and arch support using xcrun, as they were generalized to support iOS vs OS X builds. But, if it is necessary to do separate compile/link steps to support iOS framework builds because of the multiple SDKs, it *might* make sense to integrate those into the main Makefile and to isolate the framework build steps (for both OS X and iOS) to be more of an add-on step. The potential positive impact might be to simplify the current main Makefile and configure.ac by centralizing much of the framework build specific steps scattered throughout the Makefile *and* it could also simplify and speed up the iOS builds by not duplicating the platform-independent build steps. It might even be a useful abstraction for other non-Apple multiple-arch or cross-compilation builds.
Another factor to consider is support for building third-party extension modules standalone as opposed to building as part of the interpreter itself. It would be nice to be able to just use the traditional Distutils-based process for third-party extension modules. For OS X universal builds, we get that essentially for free because of the tool chain support for multiple archs. To do it automatically across multiple SDKs would require adding smarts to Distutils to do multiple compile/links and lipo merges.
bitdancer commentedon Mar 15, 2015
Also remember that being a supported platform requires buildbots. If I'm reading this right that means two: one for the simulator build, and one that actually runs on the device (once the run-the-test issues is solved). I'm not sure what would need to be done to buildbot to support the scenario of building on one platform and running the tests on another after copying the binary, but I'm sure there must be some way to do it...buildbot is written in Python after all :) And I'm sure that capability will be useful for more than just iOS...with Android we can probably run buildbot on the device eventually, but being able to test a cross compile build there as well will be helpful (once there is one).
freakboy3742 commentedon Mar 16, 2015
Understood that buildbots are required. The subject has come up a couple of times on mobile-sig - however, I haven't got a good answer for exactly what this means in practice. Does build hardware need to be delivered to a specific build farm location, or can it just run in my home office? Does it need to be running 24/7, or just guarantee that at least X builds per week, or Y% of all builds are tested?
In terms of physical specifications - at a minimum, at least two iDevices will be required (an ARMv7 device, like a 5th Gen iPod, and an ARM64 device, like an iPhone 5S), tethered to a Mac that runs the simulator.
In terms of execution - there's definitely ways to remote-control the deployment and execution of apps, so I imagine buildbot integration is in the realm of the possible.
Android will have similar requirements, but Android also has an emulator which would allow testing without the existence of a physical device, which would provide an alternative approach.
ncoghlan commentedon Mar 16, 2015
Rather than trying to do something ourselves on the mobile testing front, we may want to instead explore the free-for-open-source-project offering from Sauce Labs: https://saucelabs.com/opensauce/
Adapting that to run the CPython self-tests as an "app" might be interesting, but they're PyCon sponsors, so they may be amenable to helping to get that work properly if we need to ask for their help.
Getting that set up would also potentially be a useful tool for ensuring PSF web services are working correctly across multiple browsers and client devices. (I'd actually like to get us to the point of being able to do continuous deployment of those services some day)
ncoghlan commentedon Mar 16, 2015
(Some work on the Buildbot site would still be required, but it would be triggering a Sauce Labs run rather than managing the mobile hardware directly)
freakboy3742 commentedon Mar 16, 2015
Nick: Finding a way to get on-device test results is next on my TODO list, once I've got the patch up to date for trunk.
FYI - Updating to trunk is currently blocking on bpo-22625 (which was introduced by the fix for bpo-22359).
I had thought about using a mobile testing farm "as a service", but I didn't know Sauce Labs had one. I'll keep that in mind when I get around to working on the test suite.
11 remaining items
moreati commentedon Mar 2, 2016
I've done my best to rebase Freakboy's patch onto 3.6-dev. The attached applies cleanly, but the testsuite goes into an infinite loop. It's a start at least. At a guess the problem is in Lib/test/libregrtest/ or Lib/test/regrtest.py where the patch changes sys.exit() calls to return statements.
I took the liberty of filtering out whitespace & other cosmetic changes that were in 20151221.diff.
freakboy3742 commentedon Mar 2, 2016
Alex - The usual cause for that problem isn't regrtest (at least, not directly). The cause is one of the tests in the suite spawning a subprocess. Due to the way the test harness works, when the test suite forks/spawns a subprocess (e.g., to run a network server or a multiprocess worker), the fork causes the a second version of the test suite to start, rather than starting the server subprocess.
I've attached a patch for 3.5.1. This includes a patch to work around bpo-22625; the patch is definitely suboptimal, but it lets a build come to completion. What is really needed (as requested on python-dev) is some help working out how to fix bpo-22625 properly.
controversial commentedon Jun 3, 2016
Are you aware of Pythonista? I have no affiliation, but I'm a fan of the app and the community surrounding it. See http://omz-software.com/pythonista/ for details. I'm sure communication with the developer of the app could yield some useful insights.
freakboy3742 commentedon Jun 3, 2016
Yes - I'm aware of Pythonista; the author of that app contracted me to develop the 3.5 patch that is attached to this ticket :-)
controversial commentedon Jun 3, 2016
Ok, great!
On Fri, Jun 3, 2016 at 7:34 AM Luke Taylor <luke@deentaylor.com> wrote:
rovitotv commentedon Jan 2, 2017
Russell, this is excellent work!!!! I am truly amazed that within a couple of hours I had Python built and running for the iOS simulator this is a feat I didn't think was possible. Perspective on me I am very familiar with Linux, C, and Python but know very little about iOS and XCode keep that in mind when reviewing my comments.
MacOS: 10.12.2
Xcode: 8.2.1 (8C1002)
iOS: 10.2 (14C89)
Python source code: 3.5.1
I applied your last patch file name 20160217.diff. The first issue I ran into was clock_settime() is unavailable in iOS so I went into timemodule.c line 164 and commented out the line:
ret = clock_settime((clocked_t)clk_id, &tp);
then added this line
ret = 0;
Next I think a more serious problem was with Python/random.c line 92 "Python/random.c:92:19: error: implicit declaration of function 'getentropy' is invalid in C99 [-Werror,-Wimplicit-function-declaration]"
So I did the same thing as above commented out the lines with the getentropy function and added a line of res = 0. Who needs entropy anyway? I wonder if this problem is related to bpo-29057 (http://bugs.python.org/issue29057).
Next in your README file you list a file called main.c when the actual code is main.m.
Next main.m was not compiling with Xcode so I replaced with this line after looking at your Python-iOS-template project on GitHub, as I am not qualified to make these types of changes:
-line 31 was not compiling so I replaced:
wpython_home = Py_DecodeLocale([python_home UTF8String], NULL);
-line 54 had a similar error so I replaced:
python_argv[0] = Py_DecodeLocale(main_script, NULL);
-line 56 had a similar error so I replaced with this line:
python_argv[i] = Py_DecodeLocale(argv[i], NULL);
-line 89 comment it out to remove the assertion failure, I am not sure what
this line is supposed to do but no assertion failure seems like a better result.
I didn't want to blindly attach a diff as I don't understand the Apple lingo but I wanted to provide feedback in the hope that it helps get this patch committed so iOS becomes a target for CPython in the near future. Thanks!
freakboy3742 commentedon Nov 6, 2017
For those interested, I've started tracking these patches on Github: https://github.com/freakboy3742/cpython
The 3.4, 3.5 and 3.6 branches are tested and compile as of the time of this comment; the master branch has also been updated, in preparation for the release of 3.7.
I haven't submitted pull requests because there are still some issues that Ned raised in discussions at PyCon US.
[-]Modifications to support iOS as a cross-compilation target[/-][+]Terminate bugs[/+][-]Terminate bugs[/-][+]Restore[/+][-]Restore[/-][+]Modifications to support iOS as a cross-compilation target[/+]ned-deily commentedon Apr 20, 2023
After discussion with @freakboy3742, we agreed that much has changed since this issue was opened and so it makes most sense to close this issue in favor of a new one, with PRs targeted for 3.13, that reflect the current approaches and changes needed in cpython, for the most part, simpler changes.