Comments (19)
Awesome, @ephemer! I've been thinking about this same problem. I have some work up on a branch based off of apple/swift master
: https://github.com/modocache/swift/tree/build-script-cross-compile-sdks
Here are my notes:
Currently, the Swift build scripts and CMakeLists files support limited cross-compilation of the stdlib, from OS X x86_64 to:
- iOS {armv7, armv7s, arm64}
- iOS simulator {i386, x86_64}
- tvOS arm64
- tvOS simulator x86_64
- watchOS armv7k
- watchOS simulator i386
(All other targets--Linux armv7, FreeBSD x86_64--can only be compiled on host machines of the same OS and architecture.)
This cross-compilation is achieved using dynamically generated CMake custom targets. Each custom target is added as a dependency upon the main stdlib compilation target.
Each custom target may target a different OS and architecture, so variables like CMAKE_SYSTEM_NAME
cannot be used reliably. Instead, the current system succeeds in cross-compilation through a combination of assumptions (iOS, tvOS, and watchOS are all Darwin systems) and manual configuration (for example, isysroot
is set based on the compilation target here then here).
People more familiar with CMake than I have indicated that this is an anti-pattern, and that scaling this out to support cross-compilation across OS X/Linux/iOS/Android would be "impossible". Instead, the suggestion is to invoke CMake multiple times. This is what projects like LLVM do. When cross-compiling from OS X to multiple other targets (such as all of the iOS/tvOS/etc targets), CMake should be invoked once for each. Doing so would allow us to reference variables like CMAKE_SYSTEM_NAME
, since those would represent the target. Variables like CMAKE_HOST_SYSTEM_NAME
could be used to determine information about the host.
from swift.
Hi @modocache, I like the look of the changes you made. Seems to make breaking apart host and target a lot more intuitive. Have you tested these for OS X/iOS targets?
from swift.
I too am elbow-deep in Swift CMake, but I'm working on adding support for the gold linker in linux. I hope that my changes don't conflict with either of these too much. Do you either of you have an estimate of when you expect these changes to be close to merging in? It sounds like there might be a somewhat delicate three-way merge of CMake changes on the horizon.
from swift.
@ephemer They currently only work for all host compilation, and for cross compilation from OS X to iOS/tvOS/watchOS targets. I tried applying the same patches to this SwiftAndroid fork but encountered some errors which I should have saved somewhere... 😞
configure_sdk_unix
won't crash and burn on OS X, but it sets the SWIFT_SDK_${prefix}_PATH
to /
, which is then used as the isysroot
. To have this work for cross-compilation, I think you'll need to allow users of the build scripts to specify where a valid isysroot
can be found.
configure_sdk_darwin
, on the other hand, almost certainly will crash and burn on Linux--it executes xcrun, which shouldn't be available on Linux.
from swift.
@hpux735 Great to hear! No worries here. As I mention above, I think any patches I land to support cross-compilation will be very different from what I have on my https://github.com/modocache/swift/tree/build-script-cross-compile-sdks branch. I don't have a clear picture of what those will look like yet, and I'm happy to rebase onto your changes.
from swift.
I think going forward I'll end up basing any further patches on modocache's
work, so if you keep that in mind and maybe even rebase onto his branch
when it's building across the supported targets, my changes here should be
relatively painless to put on top.
@modocache do you plan to continue work on this? Would be great if you get
it to a mergeable state
William Dillon [email protected] schrieb am Mo., 18. Jan. 2016 um
20:55:
I too am elbow-deep in Swift CMake, but I'm working on adding support for
the gold linker in linux. I hope that my changes don't conflict with either
of these too much. Do you either of you have an estimate of when you expect
these changes to be close to merging in? It sounds like there might be a
somewhat delicate three-way merge of CMake changes on the horizon.—
Reply to this email directly or view it on GitHub
https://github.com/SwiftAndroid/swift/issues/13#issuecomment-172634644.
from swift.
Excellent, what do you think @modocache? Should I rebase off of you?
from swift.
I am actively working on cross-compiling Swift, but I don't think my work on https://github.com/modocache/swift/tree/build-script-cross-compile-sdks is a good approach, so I wouldn't advise rebasing off of that.
I'll probably make a new branch that is based off of apple/swift master
, and I'll post a comment here when that happens! 👋
from swift.
@modocache the approach in your branch seems more manageable than what's currently in master. I missed what the catch is. Also I'm not sure how much further I'll get without some kind of structured approach like your current branch. Are you suggesting I wait a while before continuing with this? I have a pretty good idea what errors you would have come across with SwiftAndroid btw: I've just spent all afternoon working through them myself
from swift.
@ephemer The "catch" I'm imagining is that manually keeping track of which platform we're targeting will become increasingly unwieldy. When cross-compiling multiple targets, it is untenable. For example, let's say we're cross-compiling from OS X to iOS and Android. CMake provides a single, global LINK_FLAGS
parameter. Should we set -rpath
as part of these linker flags? If we're compiling for both iOS and Android then, as far as I can tell, there is no solution here.
Let me know if I'm misunderstanding something--I'm no CMake wizard.
Still, if you can take my https://github.com/modocache/swift/tree/build-script-cross-compile-sdks branch and build something useful out of it, don't let me stop you! 😄 I'm just not convinced it's the right approach.
from swift.
Ok, I only looked at your code on GitHub and I guess I missed this
intricacy.
It makes sense if we can just use the CMAKE_SYSTEM_NAME as the target name,
no? The problem there is that iOS SDK targets also have versioning. But
logically speaking that should be an exception rather than the rule.
I'm not sure how the LINK_FLAGS situation would be an issue if you're
invoking CMake multiple times. Wasn't that the whole point of your change,
to remove this issue?
Generally the way you'd split the code up seemed good though. I'm going to
try using your version as a base but do without the manual tracking of the
target names.
Brian Gesiak [email protected] schrieb am Mo., 18. Jan. 2016 um
22:09:
@ephemer https://github.com/ephemer The "catch" I'm imagining is that
manually keeping track of which platform we're targeting will become
increasingly unwieldy. When cross-compiling multiple targets, it is
untenable. For example, let's say we're cross-compiling from OS X to iOS
and Android. CMake provides a single, global LINK_FLAGS parameter
https://github.com/apple/swift/blob/6964301977ed0c09a5ea09812093260f99a2f8a6/cmake/modules/AddSwift.cmake#L1691-L1692.
Should we set -rpath as part of these linker flags
https://github.com/apple/swift/blob/6964301977ed0c09a5ea09812093260f99a2f8a6/cmake/modules/AddSwift.cmake#L1640-L1644?
If we're compiling for both iOS and Android then, as far as I can tell,
there is no solution here.Let me know if I'm misunderstanding something--I'm no CMake wizard.
Still, if you can take my
https://github.com/modocache/swift/tree/build-script-cross-compile-sdks
branch and build something useful out of it, don't let me stop you! [image:
😄] I'm just not convinced it's the right approach.—
Reply to this email directly or view it on GitHub
https://github.com/SwiftAndroid/swift/issues/13#issuecomment-172654324.
from swift.
I'm not sure how the LINK_FLAGS situation would be an issue if you're invoking CMake multiple times. Wasn't that the whole point of your change, to remove this issue?
My change still builds all stdlib targets using a single CMake invocation, like apple/swift master
does. I plan on proposing a change to have stdlib targets built via multiple CMake invocations sometime today.
Basically, this is how Swift is built now:
# build-script-impl
# Loop over each deployment target.
# This is cross-compilation for standalone Swift builds--that is,
# this builds LLVM and a Swift compiler executable for the target
# platform. (Note that this is different from the cross-compilation
# in SwiftAndroid, which builds a Swift compiler that runs on the
# host platform, and produces Swift programs that can be run on
# Android.)
for deployment_target in "${NATIVE_TOOLS_DEPLOYMENT_TARGETS[@]}" "${CROSS_TOOLS_DEPLOYMENT_TARGETS[@]}"; do
# Loop over cmark, llvm, swift (and potentially lldb, llbuild, swiftpm, xctest, foundation)
for product in "${PRODUCTS[@]}"; do
# ...
# Add all STDLIB_DEPLOYMENT_TARGETS to the targets built by
# the `cmake --build` invocation below.
build_targets=(all "${SWIFT_STDLIB_TARGETS[@]}")
# ...
# Build all stdlib targets at once.
${DISTCC_PUMP} "${CMAKE}" --build "${build_dir}" $(cmake_config_opt ${product}) -- ${BUILD_ARGS} ${build_targets[@]}
done
done
This is how I'd like to change it:
# build-script-impl
# Loop over each deployment target.
for deployment_target in "${NATIVE_TOOLS_DEPLOYMENT_TARGETS[@]}" "${CROSS_TOOLS_DEPLOYMENT_TARGETS[@]}"; do
# Loop over cmark, llvm, swift (and potentially lldb, llbuild, swiftpm, xctest, foundation)
for product in "${PRODUCTS[@]}"; do
# ...
# Loop over each stdlib target.
for stdlib_target in "${SWIFT_STDLIB_TARGETS[@]}"; do
# ...
# Build the stdlib target.
${DISTCC_PUMP} "${CMAKE}" --build "${build_dir}" $(cmake_config_opt ${product}) -- ${BUILD_ARGS} $stdlib_target}
done
done
done
I may be missing something, though. Definitely let me know how your approach goes, or if you see something awkward about how I plan on doing it! 👍
from swift.
Sounds great to me in theory. Before you get too far here though, I'd like to point you to the discussion on the swift mailing list about this topic, starting here (there are about 6 emails on the topic):
https://lists.swift.org/pipermail/swift-dev/Week-of-Mon-20160118/000870.html
Given that we're building a compiler which then builds other things itself (which is always going to be tricky with this kind of dependency graph), it's possible the multiple invocation approach isn't ideal, and is something we only do when really needed.
Either way, I see the build-script AND the CMakeLists becoming unmanageable very quickly the more we work on this cross-compilation stuff. My knowledge of both python and bash are at the "just getting by" level, but I have a feeling the bash script is already way overblown in complexity. If you're going to make changes to build-script-impl, it may be worth trying to port them into the new modularised swift_build_support python modules..
from swift.
@ephemer Absolutely. Have you seen https://bugs.swift.org/browse/SR-237? I'd love to move as much into Python as possible.
Thanks a ton for the link to the mailing lists! I'll read through them now.
from swift.
From the mailing list discussion:
Unfortunately CMake only supports one C compiler and one linker in one CMake invocation. The "CMake way" for cross-compilation is to invoke CMake once for every part of the build that requires a different set of tools.
Cool, this mirrors my understanding as well. Glad to see I'm not completely out of my depth.
The disadvantage to this approach is that there are many CMake invocations, and many ninja invocations. This is bad for the same reasons why recursive make is bad. [1] So we would like to keep the current scheme for as many targets as possible (OS X native, OS X to iOS cross-compilation, Linux native etc.), and use the multi-CMake scheme only where strictly required.
Interesting. I'm reading the linked article now, to get an idea as to whether we should pursue multiple invocations for OS X-to-Linux cross-compilation only, or whether we should a single CMake invocation no matter what kind of cross-compilation we're doing.
from swift.
@modocache I had seen that bug report but missed that you were driving force behind it! Haha, keep up the good work.
From what I've found out the last couple of days, it seems that it really does make sense to run multiple CMake invocations when you want to use a different compiler/linker toolchain, as is most likely the case with e.g. OSX->Android cross compiling. That's because setting the CMAKE_C_COMPILER variable (which currently happens around build-script-impl:1060) causes all the other compiler settings (flags etc) to reset. This may be manageable but sounds like it could be more trouble than it's worth.
My plan is to try to compiling the OSX swift driver etc with the standard Xcode toolchain and use the Android toolchain to build the stdlib. There is probably a missing piece in linking the two invocations here, which may also prove to bring more pain. We'll see.
See here for more reading:
https://cmake.org/Wiki/CMake_Useful_Variables (see the notes around CMAKE_C_COMPILER)
https://cmake.org/Wiki/CMake_Cross_Compiling
from swift.
I feel like I'm out of my depth and don't seem to be making any progress on this at the moment. It seems like the inbuilt clang would be fine to build the android sources, we just need to tell it to use the android linker. I'm not experienced enough with compiling and linking generally to know how to go about that but it seems doable.
On a side note, I did come across this https://github.com/taka-no-me/android-cmake, which may be of interest. (it uses the multiple invocation paradigm)
from swift.
@froody is continuing to work on this; see the mailing list thread here.
from swift.
Closing this in order to centralize the discussion here: https://bugs.swift.org/browse/SR-1362
from swift.
Related Issues (16)
- Setting up environment HOT 2
- JNI Support HOT 6
- Current status in NDK usage? HOT 6
- 1_stdlib/VarArgs.swift HOT 10
- IRGen/objc_simd.sil HOT 5
- stdlib/FixedPoint.swift.gyb
- libicu data files need to be pushed to device HOT 2
- Problem building on Ubuntu 15.10 HOT 2
- find public domain/LLVM code to replace current third-party code HOT 10
- Lib Bionic doesn't compile HOT 5
- Enums and Protocol conformance causes segfault on android HOT 3
- Get tests running HOT 8
- libicu doesn't link at runtime on android systems with their own icu HOT 13
- Find the correct way to package libicu data files HOT 10
- Can share info about how to start porting to Windows.. HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from swift.