crystax / android-platform-ndk

CrystaX NDK - Native Development Kit for Android

Android Native Development Kit (NDK)

The latest version of this document is available at https://android.googlesource.com/platform/ndk/+/master/README.md.

Note: This document is for developers of the NDK, not developers that use the NDK.

The NDK allows Android application developers to include native code in their Android application packages, compiled as JNI shared libraries.

Other Resources

This doc gives a high level overview of the NDK's build, packaging, and test process. For other use cases, or more in depth documentation, refer to the following sources:

• User documentation is available on the Android Developer website.
• Adding a new NDK API or platform version? Check Generating Sysroots.
• Working on Clang or GCC? See Toolchains.md.
• Discussions related to the Android NDK happen on the android-ndk Google Group.
• File bugs against the NDK at https://github.com/android-ndk/ndk/issues.

Building the NDK

Both Linux and Windows host binaries are built on Linux machines. Windows host binaries are built via MinGW cross compiler. Systems without a working MinGW compiler can use build/tools/build-mingw64-toolchain.sh to generate their own and be added to the PATH for build scripts to discover.

Building binaries for Mac OS X requires at least 10.8.

Target headers and binaries are built on Linux.

Components

The NDK consists of three parts: host binaries, target prebuilts, and others (build system, docs, samples, tests).

Host Binaries

• toolchains/ contains GCC and Clang toolchains.
  • $TOOLCHAIN/config.mk contains ARCH and ABIS this toolchain can handle.
  • $TOOLCHAIN/setup.mk contains toolchain-specific default CFLAGS/LDFLAGS when this toolchain is used.
• prebuilt/$HOST_TAG contains build dependencies and additional tools.
  • make, awk, python, yasm, and for Windows: cmp.exe and echo.exe
  • ndk-depends, ndk-stack and ndk-gdb can also be found here.

Target Headers and Binaries

• platforms/android-$VERSION/arch-$ARCH_NAME/ contains headers and libraries for each API level.
  • The build system sets --sysroot to one of these directories based on user-specified APP_ABI and APP_PLATFORM.
• sources/cxx-stl/$STL contains the headers and libraries for the various C++ STLs.
• prebuilt/android-$ARCH/gdbserver contains gdbserver.

Others

• build/ contains the ndk-build system and scripts to rebuild NDK.
• sources/android and sources/third_party contain modules that can be used in apps (cpufeatures, native_app_glue, etc) via $(call import-module, $MODULE)
• tests/

Prerequisites

• AOSP NDK Repository

  • Check out the branch master-ndk

    repo init -u https://android.googlesource.com/platform/manifest \
        -b master-ndk
    
    # Googlers, use
    repo init -u \
        persistent-https://android.git.corp.google.com/platform/manifest \
        -b master-ndk

• Additional Linux Dependencies (available from apt):

  • bison
  • flex
  • libtool
  • mingw-w64
  • pbzip2 (optional, improves packaging times)
  • texinfo
  • python3 (used for Vulkan validation layer generation)
  • python-lxml (used for Vulkan validation layer generation)

• Mac OS X also requires Xcode.

Host/Target prebuilts

For Linux or Darwin:

$ python checkbuild.py

For Windows, from Linux:

$ python checkbuild.py --system windows  # Or windows64.

checkbuild.py also accepts a variety of other options to speed up local builds, namely --arch and --module.

Packaging

By default, checkbuild.py will also package the NDK and run basic tests. To skip the packaging step, use the --no-package flag. Note that running the tests does require the packaging step.

If you need to re-run just the packaging step without going through a build, packaging is handled by build/tools/package.py.

Testing

Testing is discussed in Testing.md.

android-platform-ndk's Issues

Linking failed for bionic missing methods

I want to build the samba4 project on Android with the crystax-ndk-10.3.2, but bionic missing methods such as 'swab', 'seekdir', 'telldir', 'quotactl' cause linking errors, although I find these methods in the header files. I search the 'libcrystatx.so' file with 'nm' tools, and it really does not contain any of these methods. So which library should I include in my build script? I'm not using an Android.mk file, but I try both standalone and packed ndk toolchains, is there anything wrong?

cmake toolchain not included in normal crystax installer?

I noticed that after extracting CrystaX NDK 10.3.1.1, the cmake/toolchain.cmake file is not present. Is this by design? Or is this CMake addition so new that it hasn't made it into the latest release?

Use Continuous Integration / Continuous Deployment

Free CI tools for open source projects exist:

• Travis Ci (supports Linux & OSX)
• CircleCI (only Linux, can run longer jobs, supports bitbucket)
• Appveyor (only Windows)

These are somewhat easy to set up.
The idea would be to have them compile & run tests on every push to a pull request to find regressions.
When pushing a tag/commit to master, you can have these CI tools push blobs to github's release page or anywhere you want. You would no longer need to publish packages yourself.

CMake: copying shared libcrystax, libgnustl and libboost_* to build- and install-tree (and their semantics)

I was wondering what is the "intended" workflow of using CMake build system and shared runtime libraries (libcrystax.so, libgnustl_shared.so and libboost_*.so ahd others); and, more generally, what are the "intended" semantics of build- and install-trees when using CMake with Android NDK.

Cc: @rpavlik — you are the author of the only Android/CMake toolchain file which at least somehow considers boost, so I'd like to hear your comments as well.

Part 1. Shared runtime libraries and CMake

So far, the CrystaX CMake toolchain file copies libcrystax.so and libgnustl_shared.so into the build-tree, and that's all. This has three problems:

1. Bundled third-party libraries (Boost, at least) are simply left out. A project which uses CMake and shared boost is non-functional after building.
2. This behavior depends on the ${LIBRARY_OUTPUT_PATH} variable which is deprecated and not even set by default. Considering that ${ANDROID_STL} defaults to gnustl_shared and libcrystax.so is available only as a shared library, a project built with CMake with default settings is non-functional after building (!).
3. This contradicts the workflow described below (see below).

Part 2. Build- and install-tree semantics with Android

In the CMake toolchain file I find some rudimentary code that sets ${LIBRARY_OUTPUT_PATH} to something containing libs/${ANDROID_NDK_ABI_NAME}. This means that the authors intended the build-tree to become the final location of the libraries. That is — if the CMake project is correctly placed within the Android project, you can just cmake . && make and the libraries will end up in the correct places for Android packaging. (I hope I am clear here.)

However, I find this pretty unusable. First, any typical cross-compilation effort will include multiple projects which will try to find each other. By default (with existing toolchain file), CMake won't search libraries in ${LIBRARY_OUTPUT_PATH}, but it will search in ${CMAKE_INSTALL_PREFIX}/lib. Same holds for find_package(CONFIG).

Second, almost no projects are written with Android/CMake in mind, so vast majority of existing projects will install their libraries in $prefix/lib.

Hence, to make projects find each other (without extensively patching their build systems), I typically build in a temporary directory and then install to a common prefix, with libraries installed in $prefix/lib and exported configs in $prefix/lib/cmake/Project/ProjectConfig.cmake. Then I symlink $prefix/lib from $android_project/libs/$ABI and run Android packaging.

The described workflow allows to cross-compile any "chains" of projects without altering their build systems (provided that they can build for Linux).

With all this in mind, there should be a way to install the shared runtime libraries alongside the project's library destination, typically ${CMAKE_INSTALL_PREFIX}/lib. I'd expect a parameter of kind ANDROID_SHARED_HELPER_DESTINATION, which, when set, will make the toolchain file install the shared runtime libraries to ${CMAKE_INSTALL_PREFIX}/${ANDROID_SHARED_HELPER_DESTINATION}.

This parameter should be honoured by the toolchain file itself (to install libcrystax.so and libgnustl_shared.so), and custom Find*.cmake modules should be written for all bundled 3rd-party projects (boost, ...) which will chain to real find modules and install the found libraries afterwards (an example of such "chaining" already exists in the tree as FindBoost.cmake).

Comments? Maybe I'm horribly wrong somewhere and using CMake with shared runtime already works well?

how to build third part python module write with c/c++

Hi, I run python with this library: https://github.com/joaoventura/pybridge
And I want to run https://github.com/scrapy/scrapy in Android, can I do that?

scrapy requirement some other module that write with c/c++, how to build them?

Unity3D engine support

Current version of CrystaX is not recognized by Unity Editor as NDK.
I am wondering if this development kit could potentially be used as an alternative toolchain to build Unity projects with IL2CPP - this would simplify porting and development of native player plugins for Android target.

Errors when building libzip with crystax

Building Libzip with crystax fails.

CMake Error at CMakeLists.txt:98 (MESSAGE):
  -- ZLIB version too old, please install at least v1.1.2

CMakeFiles/zip.dir/zip_error_strerror.c.o:zip_error_strerror.c:function zip_error_strerror: error: undefined reference to 'zError'
CMakeFiles/zip.dir/zip_error_to_str.c.o:zip_error_to_str.c:function zip_error_to_str: error: undefined reference to 'zError'
CMakeFiles/zip.dir/zip_filerange_crc.c.o:zip_filerange_crc.c:function _zip_filerange_crc: error: undefined reference to 'crc32'
CMakeFiles/zip.dir/zip_filerange_crc.c.o:zip_filerange_crc.c:function _zip_filerange_crc: error: undefined reference to 'crc32'
CMakeFiles/zip.dir/zip_source_crc.c.o:zip_source_crc.c:function zip_source_crc: error: undefined reference to 'crc32'
CMakeFiles/zip.dir/zip_source_crc.c.o:zip_source_crc.c:function crc_read: error: undefined reference to 'crc32'
CMakeFiles/zip.dir/zip_source_deflate.c.o:zip_source_deflate.c:function compress_read: error: undefined reference to 'deflate'
CMakeFiles/zip.dir/zip_source_deflate.c.o:zip_source_deflate.c:function decompress_read: error: undefined reference to 'inflate'
CMakeFiles/zip.dir/zip_source_deflate.c.o:zip_source_deflate.c:function deflate_compress: error: undefined reference to 'deflateInit2_'
CMakeFiles/zip.dir/zip_source_deflate.c.o:zip_source_deflate.c:function deflate_compress: error: undefined reference to 'deflateEnd'
CMakeFiles/zip.dir/zip_source_deflate.c.o:zip_source_deflate.c:function deflate_decompress: error: undefined reference to 'inflateInit2_'
CMakeFiles/zip.dir/zip_source_deflate.c.o:zip_source_deflate.c:function deflate_decompress: error: undefined reference to 'inflateEnd'
collect2: error: ld returned 1 exit status
lib/CMakeFiles/zip.dir/build.make:2824: recipe for target 'lib/libzip.so' failed
make[2]: *** [lib/libzip.so] Error 1
CMakeFiles/Makefile2:85: recipe for target 'lib/CMakeFiles/zip.dir/all' failed
make[1]: *** [lib/CMakeFiles/zip.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

Steps to reproduce:

export cmake_dir=~/Documents/zone/mid/bin/android/google/sdk/cmake/3.6.3155560
export PATH=${cmake_dir}/bin/:$PATH

export ANDROID_NDK=/home/declan/Documents/zone/mid/lib/android/crystax/crystax-ndk-10.3.2_2017_03_08

cmake \
    -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
    -DBUILD_SHARED_LIBS=ON \
    -DCMAKE_TOOLCHAIN_FILE=${ANDROID_NDK}/cmake/toolchain.cmake \
    -DANDROID_ABI=armeabi-v7a

make -j8

python3.5 ssl problem

I use jni run python, but error...
Unexpected error: (<class 'ImportError'>, ImportError("No module named '_ssl'",), <traceback object at 0xee8c1da0>)

Is CrystaX NDK abandoned?

Is CrystaX NDK abandoned?

10.3.2: No cmake directory (Windows builds)

Hi,

I hope this is the right place to ask. The cmake directory is provided in this repo, however it's not when downloading and extracting a prebuilt release from the homepage. I extracted 10.3.2 and 10.3.1 for Windows, neither does contain the cmake directory. Haven't looked at the Linux archives, tbh.

Is this an issue or is this by design? I need the CMake stuff, so, currently I copy the cmake directory into the prebuilt root directory after cloning this repo.

RELEASE.TXT file doesn't contain the "android equivalent version", foiling external tools

We're using the android-cmake "toolchain file" to build CMake-based software using the NDK. It parses RELEASE.TXT to extract the NDK version. However, since the CrystaX NDK contains custom wording in that file, it can't parse it with the same basic logic as the upstream NDK, resulting in this error: https://github.com/OSVR/android-cmake/blob/master/android.toolchain.cmake#L429 https://gist.github.com/DuFF14/999730c5b4663939bc34

I imagine that part of this is a bug on the android-cmake side: it should be more resilient to the contents of RELEASE.TXT. However, it would be useful to the average user and possibly other scripts if the RELEASE.TXT file contained the value for the "equivalent" upstream release. I (as a human) looked at the change log HTML file which contains the version, but parsing a version out of there is a lot harder than out of the simple format in RELEASE.TXT

Missing platforms directory v10.3.2/10.3.1

The latest and next latest versions of the NDK are missing the platforms directory. This causes the make-standalone-toolchain.sh script to fail.

python3.5m crashes if python35.zip doesn't exist in the lib/ directory

https://github.com/crystax/android-platform-ndk/blob/725337b7d4d90ae9be4952e8dc62835c68fbbad8/build/instruments/build-python/getpath.c.3.x assumes that there is a python35.zip file in the apk lib directory but this isn't necessarily the case. There are good reasons why you would want to load python modules some other way (eg: the android studio gradle plugin won't copy files into lib/ that aren't .so files). Before the getpath patch was introduced you could implement imports yourself (eg: using a sys.meta_path hook), but now starting the interpreter causes a Py_FatalError and a crash.

@vmurashev

error: only position independent executables (PIE) are supported.

When trying to run the python binary for android (in the sources/python/3.5/libs folder) it gives that error message. It looks like those binaries are not built with the required -fPIE -pie arguments.

python so problem

06-24 19:37:45.123 14477-14477/com.jventura.pyapp I/pybridge: File "/data/data/com.jventura.pyapp/assets/python/stdlib.zip/socket.py", line 49, in
06-24 19:37:45.123 14477-14477/com.jventura.pyapp I/pybridge: ImportError: dlopen failed: cannot locate symbol "__cxa_atexit" referenced by "_socket.so"...

06-24 19:35:32.533 14095-14095/com.jventura.pyapp I/pybridge: Traceback (most recent call last):
06-24 19:35:32.533 14095-14095/com.jventura.pyapp I/pybridge: File "", line 1, in
06-24 19:35:32.533 14095-14095/com.jventura.pyapp I/pybridge: File "/data/data/com.jventura.pyapp/assets/python/bootstrap.py", line 9, in
06-24 19:35:32.543 14095-14095/com.jventura.pyapp I/pybridge: import ssl
06-24 19:35:32.543 14095-14095/com.jventura.pyapp I/pybridge: File "/data/data/com.jventura.pyapp/assets/python/stdlib.zip/ssl.py", line 98, in
06-24 19:35:32.543 14095-14095/com.jventura.pyapp I/pybridge: ImportError: dlopen failed: cannot locate symbol "__cxa_atexit" referenced by "_ssl.so"...
06-24 19:35:32.543 14095-14095/com.jventura.pyapp I/pybridge: Call into Python interpreter

In some device run crash, the crash device cpu type is armeabi-v7a.
All so is copy from https://dl.crystax.net/ndk/darwin/919/...

make-standalone-toolchain.sh doesn't work

Trying to make standalone toolchain with crystax-ndk-10.3.2 on OS X. Found two problems:

1. In official NDK I can use bash and python scripts. In your port I see only bash.

2. This command produce error build/tools/make-standalone-toolchain.sh --stl=libc++ --arch=arm64 --install-dir=/opt/android/tool64

    Auto-config: --toolchain=aarch64-linux-android-4.9
    Auto-config: --platform=android-21
    Copying prebuilt binaries...
    Copying sysroot headers and libraries...
    Copying crystax headers and libraries...
    Copying Objective C/C++ runtime (gnustep-libobjc2 headers and libraries)...
    Copying libpng headers and libraries...
    Copying libjpeg headers and libraries...
    Copying libtiff headers and libraries...
    Copying c++ runtime headers and libraries (with libc++abi)...
    ERROR: Can't copy from non-directory: ./sources/cxx-stl/llvm-libc++//libcxx/include
   

arm-linux-androideabi/bin/ld: error: cannot find -lcrystax

Hi,

I'm trying to compile some open source code using CMake.
I'm using the toolchain.cmake in $NDK/cmake/ by simply doing:
cmake -DCMAKE_TOOLCHAIN_FILE=$NDK/cmake/toolchain.cmake -DANDROID_ABI=armeabi-v7a
Sadly it doesn't work. During the CMake configure I get this error:

Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" failed.
Compiler: /Users/admin/Downloads/crystax-ndk-10.3.2/toolchains/arm-linux-androideabi-5/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-g++ 
Build flags: -fexceptions;-frtti;-Wno-psabi;--sysroot=/Users/admin/Downloads/crystax-ndk-10.3.2/platforms/android-16/arch-arm;-funwind-tables;-finline-limit=64;-fsigned-char;-no-canonical-prefixes;-march=armv7-a;-mhard-float;-mfpu=vfpv3-d16;-fdata-sections;-ffunction-sections;-Wa,--noexecstack;
Id flags: 

The output was:
1
/Users/admin/Downloads/crystax-ndk-10.3.2/toolchains/arm-linux-androideabi-5/prebuilt/darwin-x86_64/bin/../lib/gcc/arm-linux-androideabi/5.4/../../../../arm-linux-androideabi/bin/ld: **error: cannot find -lcrystax**
/Users/admin/Downloads/crystax-ndk-10.3.2/toolchains/arm-linux-androideabi-5/prebuilt/darwin-x86_64/bin/../lib/gcc/arm-linux-androideabi/5.4/../../../../arm-linux-androideabi/bin/ld: error: /var/folders/gv/x9c3ckh14bq9c1m_y84k6lz40000gp/T//cc4ceuIb.o uses VFP register arguments, output does not
/Users/admin/Downloads/crystax-ndk-10.3.2/toolchains/arm-linux-androideabi-5/prebuilt/darwin-x86_64/bin/../lib/gcc/arm-linux-androideabi/5.4/../../../../arm-linux-androideabi/bin/ld: error: /Users/admin/Downloads/crystax-ndk-10.3.2/toolchains/arm-linux-androideabi-5/prebuilt/darwin-x86_64/bin/../lib/gcc/arm-linux-androideabi/5.4/armv7-a/hard/libgcc.a(unwind-arm.o) uses VFP register arguments, output does not
/Users/admin/Downloads/crystax-ndk-10.3.2/toolchains/arm-linux-androideabi-5/prebuilt/darwin-x86_64/bin/../lib/gcc/arm-linux-androideabi/5.4/../../../../arm-linux-androideabi/bin/ld: error: /Users/admin/Downloads/crystax-ndk-10.3.2/toolchains/arm-linux-androideabi-5/prebuilt/darwin-x86_64/bin/../lib/gcc/arm-linux-androideabi/5.4/armv7-a/hard/libgcc.a(pr-support.o) uses VFP register arguments, output does not
/Users/admin/Downloads/crystax-ndk-10.3.2/platforms/android-16/arch-arm/usr/lib/crtbegin_dynamic.o:crtbrand.c:function _start: error: undefined reference to '__libc_init'
/Users/admin/Downloads/crystax-ndk-10.3.2/platforms/android-16/arch-arm/usr/lib/crtbegin_dynamic.o:crtbrand.c:function atexit: error: undefined reference to '__cxa_atexit'
/Volumes/HD2/cislave/workspace/ndk-build-all/HOST/darwin/LABEL/ndk-build/toolchain/build/../gcc/gcc-5/libgcc/unwind-arm-common.inc:289: error: undefined reference to 'abort'
/Volumes/HD2/cislave/workspace/ndk-build-all/HOST/darwin/LABEL/ndk-build/toolchain/build/../gcc/gcc-5/libgcc/unwind-arm-common.inc:346: error: undefined reference to 'memcpy'
/Volumes/HD2/cislave/workspace/ndk-build-all/HOST/darwin/LABEL/ndk-build/toolchain/build/../gcc/gcc-5/libgcc/unwind-arm-common.inc:376: error: undefined reference to 'memcpy'
/Volumes/HD2/cislave/workspace/ndk-build-all/HOST/darwin/LABEL/ndk-build/toolchain/build/../gcc/gcc-5/libgcc/unwind-arm-common.inc:505: error: undefined reference to 'abort'
/Volumes/HD2/cislave/workspace/ndk-build-all/HOST/darwin/LABEL/ndk-build/toolchain/build/../gcc/gcc-5/libgcc/config/arm/pr-support.c:384: error: undefined reference to 'abort'

I get the same error either on Linux and OSX. To install crystax NDK I simply extracted the crystax-ndk-10.3.2-b903-darwin-x86_64.tar.xz archive. I'm using version 10.3.2-b903.
I tried to do an export LDFLAGS="-L$NDK/sources/crystax/libs/armeabi-v7a/" sadly it didn't work. :(

CMake toolchain does not allow LLVM STL static/shared to be used

At the moment I am running these parameters:

cmake "$DIR/../../.." -G"Ninja" \
    -DCMAKE_BUILD_TYPE="$configuration" \
    -DCMAKE_TOOLCHAIN_FILE="$ANDROID_NDK/cmake/toolchain.cmake" \
    -DBUILD_TESTING=OFF \
    -DANDROID_ABI=armeabi-v7a \
    -DANDROID_NATIVE_API_LEVEL=android-15 \
    -DANDROID_STL=c++_shared \
    -DANDROID_TOOLCHAIN_VERSION=clang3.7 \
    -DANDROID_TOOLCHAIN_NAME=arm-linux-androideabi

Ignore the ANDROID_STL parameter (this is the part that should work). At the moment, GNU STL is selected even though I picked clang. However, it should allow me to select LLVM STL.

Is this already supported? If not, can you add support for it?