theopolis / build-anywhere

Scripts for building compilers that run anywhere, which build things that run anywhere

./build-anywhere.sh /output/path

These scripts build a toolchain/runtime that runs on almost every Linux distribution. The compilers can produce libraries and executables that also run on almost every Linux distribution.

anywhere (n). x86_64 Linux distributions that include a 2.13 (circa 2011+) or newer glibc.

At a very high level:

• Use Crosstool-NG to build gcc 8.3.0 linked against a glibc 2.13.
• Build an older zlib 1.2.11 to link against.
• Build Clang/LLVM 8.0.0 with the new GCC also linked against a glibc 2.13.
• Kernel headers for Linux 4.7 are included as they have certain bpf macros.
• You can use the clang/gcc compiler anywhere.
• You can use either clang or gcc's compiler runtime; recommend using linking flags to link these statically.
• You can use either libstdc++ or libc++; recommended linking these statically.
• You can use all of the LLVM static analysis and sanitizer frameworks.

This project builds the runtime. You can download pre-built runtimes on the Releases page.

man build-anywhere.sh

build-from-source

$ ./build-anywhere.sh /opt/build

This will create several output files/directories in /opt/build, at the end the important one is x86_64-anywhere-linux-gnu.

$ du -h --max-depth=2 /opt/build/x86_64-anywhere-linux-gnu
44M   x86_64-anywhere-linux-gnu/bin
4.0K  x86_64-anywhere-linux-gnu/include
20K   x86_64-anywhere-linux-gnu/scripts
54M   x86_64-anywhere-linux-gnu/libexec/gcc
54M   x86_64-anywhere-linux-gnu/libexec
4.0M  x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/bin
11M   x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/include
48K   x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/lib64
708M  x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot
4.0K  x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/lib
4.0K  x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/debug-root
718M  x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu
1.5M  x86_64-anywhere-linux-gnu/lib/ldscripts
9.8M  x86_64-anywhere-linux-gnu/lib/gcc
12M   x86_64-anywhere-linux-gnu/lib
1.5M  x86_64-anywhere-linux-gnu/share/licenses
120K  x86_64-anywhere-linux-gnu/share/gcc-8.3.0
1.7M  x86_64-anywhere-linux-gnu/share
829M  x86_64-anywhere-linux-gnu

Remember you can build this once and run it from any directory on any x86_64 Linux created in 2011 or newer.

download-prebuilt

You can also download a prebuilt version, x86_64-anywhere-linux-gnu-VERSION.tar.gz, from the GitHub releases page.

You can untar this and run from any directory. See below for guidance on how to use the toolchain.

Using the anywhere toolchain

Sourcing the ./scripts/anywhere-setup.sh script should set up your environment.

source ./x86_64-anywhere-linux-gnu/scripts/anywhere-setup.sh

Important variables:

SYSROOT=x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot
PATH=$PREFIX/bin:$PATH
CXX=clang++
CC=clang

There are several optional variables you may want to include. If you intend to install into the build-anywhere toolchain, also set the following:

PREFIX=$SYSROOT/usr
PKG_CONFIG_PATH=$PREFIX/lib/pkgconfig
ACLOCAL_PATH=$PREFIX/share/aclocal

(Somewhat) security-enhanced toolchain

Source the ./scripts/anywhere-setup-security.sh script adds extra linker and compiler flags.

source ./x86_64-anywhere-linux-gnu/scripts/anywhere-setup-security.sh

Accuracy

There are some excessive linking options included.

-fuse-ld=lld -Wl,-z,relro,-z,now -pie -l:libc++.a -l:libc++abi.a -l:libunwind.a -lpthread -ldl -lrt -lz -lm

And the compiler flags attempt to make small binaries, with PIC, and without newer ASM.

-march=x86-64 -fPIC -Oz

This is a best-effort solution that covers most bases. Every project's build system is different and may not respect the variable this toolchain sets. Most problems can be resolved by telling autotools, cmake, etc, system about the explicit linking and include paths.

As an example, SleuthKit will still find the system libstdc++.so and OpenSSL needs an explicit --prefix=$PREFIX.

A more-accurate version forces the use of clang, clang's compiler runtime, and LLVM's libc++. This is more accurate because it is harder for build systems to work if they make assumptions (e.g., we did not read their documentation closely). So you either break it or it works, which is better. The side effect of this is about 100kB additional code from static linking compared to gcc's runtime and c++ implementation.

You can also remove the libc dynamic libraries to force anything trying to link them statically.

• $PREFIX/lib/libc++*.so*
• $PREFIX/lib/libunwind*.so*

Now build systems have no choice.

build-anywhere's Issues

Use thin LTO for compiling/linking with security environment

We want to include CFG by default so we need to move towards removing visibility and adding LTO when building.

how to build https://github.com/Juan-Cortez3/pdd ?

I downloaded the tar.xz, extracted to /opt and ran the initialization script.
Then I downloaded a small c file from https://github.com/Juan-Cortez3/pdd and ran ./autogen.sh, configure and make.

It fails like this:

# make
clang -DPACKAGE_NAME=\"pdd\" -DPACKAGE_TARNAME=\"pdd\" -DPACKAGE_VERSION=\"1.0.0\" -DPACKAGE_STRING=\"pdd\ 1.0.0\" -DPACKAGE_BUGREPORT=\"[email protected]\" -DPACKAGE_URL=\"\" -DPACKAGE=\"pdd\" -DVERSION=\"1.0.0\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_PTHREAD_H=1 -I.   --gcc-toolchain=/opt/x86_64-anywhere-linux-gnu --sysroot=/opt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot  --gcc-toolchain=/opt/x86_64-anywhere-linux-gnu --sysroot=/opt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot -march=x86-64 -fPIC -Oz  -MT pdd.o -MD -MP -MF .deps/pdd.Tpo -c -o pdd.o `test -f 'src/pdd.c' || echo './'`src/pdd.c
src/pdd.c:64:37: warning: incompatible pointer types initializing 'int *' with an expression of type 'uint8_t *'
      (aka 'unsigned char *') [-Wincompatible-pointer-types]
     {"verbose", no_argument,       &flag_verbose, 1},
                                    ^~~~~~~~~~~~~
src/pdd.c:86:5: warning: implicit declaration of function 'verify_global_var' is invalid in C99
      [-Wimplicit-function-declaration]
    verify_global_var();
    ^
src/pdd.c:112:15: warning: implicit declaration of function 'pthread_create' is invalid in C99
      [-Wimplicit-function-declaration]
        ret = pthread_create(&tinfo[i].thread_id, NULL, trans_data, &tinfo[i]);
              ^
src/pdd.c:120:15: warning: implicit declaration of function 'pthread_join' is invalid in C99
      [-Wimplicit-function-declaration]
        ret = pthread_join(tinfo[i].thread_id, &res);
              ^
src/pdd.c:266:1: error: conflicting types for 'verify_global_var'
verify_global_var(void)
^
src/pdd.c:86:5: note: previous implicit declaration is here
    verify_global_var();
    ^
src/pdd.c:316:76: warning: incompatible integer to pointer conversion passing 'int' to parameter of type 'int *';
      take the address with & [-Wint-conversion]
    while (-1 != (c_opt = getopt_long(argc, argv, short_opt, long_options, option_index))) {
                                                                           ^~~~~~~~~~~~
                                                                           &
/opt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/include/getopt.h:177:49: note: passing
      argument to parameter '__longind' here
                        const struct option *__longopts, int *__longind)
                                                              ^
src/pdd.c:324:13: error: void function 'scan_args' should not return a value [-Wreturn-type]
            return 0;
            ^      ~
5 warnings and 2 errors generated.
Makefile:384: recipe for target 'pdd.o' failed
make: *** [pdd.o] Error 1

without build-anywhere, it succeeds like this:

gcc -DPACKAGE_NAME=\"pdd\" -DPACKAGE_TARNAME=\"pdd\" -DPACKAGE_VERSION=\"1.0.0\" -DPACKAGE_STRING=\"pdd\ 1.0.0\" -DPACKAGE_BUGREPORT=\"[email protected]\" -DPACKAGE_URL=\"\" -DPACKAGE=\"pdd\" -DVERSION=\"1.0.0\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_PTHREAD_H=1 -I.     -g -O2 -MT pdd.o -MD -MP -MF .deps/pdd.Tpo -c -o pdd.o `test -f 'src/pdd.c' || echo './'`src/pdd.c
src/pdd.c:64:37: warning: initialization from incompatible pointer type [-Wincompatible-pointer-types]
      {"verbose", no_argument,       &flag_verbose, 1},
                                     ^
src/pdd.c:64:37: note: (near initialization for ‘long_options[0].flag’)
src/pdd.c: In function ‘main’:
src/pdd.c:86:5: warning: implicit declaration of function ‘verify_global_var’; did you mean ‘init_global_var’? [-Wimplicit-function-declaration]
     verify_global_var();
     ^~~~~~~~~~~~~~~~~
     init_global_var
src/pdd.c:112:15: warning: implicit declaration of function ‘pthread_create’; did you mean ‘timer_create’? [-Wimplicit-function-declaration]
         ret = pthread_create(&tinfo[i].thread_id, NULL, trans_data, &tinfo[i]);
               ^~~~~~~~~~~~~~
               timer_create
src/pdd.c:120:15: warning: implicit declaration of function ‘pthread_join’ [-Wimplicit-function-declaration]
         ret = pthread_join(tinfo[i].thread_id, &res);
               ^~~~~~~~~~~~
src/pdd.c: At top level:
src/pdd.c:266:1: warning: conflicting types for ‘verify_global_var’
 verify_global_var(void)
 ^~~~~~~~~~~~~~~~~
src/pdd.c:86:5: note: previous implicit declaration of ‘verify_global_var’ was here
     verify_global_var();
     ^~~~~~~~~~~~~~~~~
src/pdd.c: In function ‘scan_args’:
src/pdd.c:316:76: warning: passing argument 5 of ‘getopt_long’ makes pointer from integer without a cast [-Wint-conversion]
     while (-1 != (c_opt = getopt_long(argc, argv, short_opt, long_options, option_index))) {
                                                                            ^~~~~~~~~~~~
In file included from /usr/include/getopt.h:36:0,
                 from src/pdd.c:39:
/usr/include/x86_64-linux-gnu/bits/getopt_ext.h:66:12: note: expected ‘int *’ but argument is of type ‘int’
 extern int getopt_long (int ___argc, char *__getopt_argv_const *___argv,
            ^~~~~~~~~~~
src/pdd.c:324:20: warning: ‘return’ with a value, in function returning void
             return 0;
                    ^
src/pdd.c:303:1: note: declared here
 scan_args(int argc, char *argv[])
 ^~~~~~~~~
mv -f .deps/pdd.Tpo .deps/pdd.Po
gcc  -g -O2   -o pdd pdd.o  -lpthread

will-it-run script to check linkage requirements and x86 extensions

After a binary or library is built we want to check the linkage requirements.

1. run ldd -v and check the glibc version requirement.
2. check a small-list of expected dynamic link requirements (m, rt, pthreads, z)
3. disassemble and search for vector/etc instructions that only exist on newer CPUs (non-deterministic check)

How can i use this tool chain to build ppc64le osquery from x86_64

I am trying to build osquery tool for ppc64le , i tried different toolchain versions but all failing to build
Now I want to try using build-anywhere tool chain from x86_64 , can i do it ? how can I do it ?

Would a cython python app benefit from this/how do i do this?

Hi. I downloaded the precompiled version of build-anywhere earlier today. Then:
source /home/kris/x86_64-anywhere-linux-gnu

Then i made a .c file with cython(embedding the python interpreter):
cython --embed -3 program.py
this depends upon pyqt5 and enum(that again depends on some other standard libs in /usr/lib/python3.7 im not sure how to include).

Then how shall i compile this app making sure i include the right dependencies? I don't care if i have to copy the whole /usr/lib/python3.7 folder and set the sys.path inside the python program to look inside customdir/python3.7 if this is the only option, but first and foremost i just want to compile the app.

Then i will try to copy it to centos6 VM just to see if it runs.

Using x86_64-anywhere-linux-gnu-v5 to build ssdeep-2.14.1, it got build failed result

Platform: X86_64 PC
OS: Ubuntu 18.04 (Usinig OS-release upgrade from 16.04)
x86_64-anywhere-linux-gnu release version: x86_64-anywhere-linux-gnu-v5.tar.xz
ssdeep version: 2.14.1

Step:

1. tar Jxf x86_64-anywhere-linux-gnu-v5.tar.xz
2. cp -a x86_64-anywhere-linux-gnu /tmp/
3. source /tmp/x86_64-anywhere-linux-gnu/scripts/anywhere-setup.sh
4. cd ssdeep-2.14.1/
5. ./configure --disabled-shared -> got below error message
   configure: error: unrecognized option: --disabled-shared' Try ./configure --help' for more information
6. ./configure
7. make -> got build failed.
   InstalledDir: /tmp/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/bin
   "/tmp/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/bin/ld" --sysroot=/tmp/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -shared -o .libs/libfuzzy.so.2.1.0 /tmp/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/bin/../lib64/crti.o crtbeginS.o -L/usr/local/lib -L/tmp/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/bin/../lib64 -L/tmp/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/lib/../lib64 -L/tmp/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/lib/../lib64 -L/tmp/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/bin/../lib -L/tmp/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/lib -L/tmp/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/lib .libs/fuzzy.o .libs/edit_dist.o .libs/find-file-size.o -l:libc++.a -l:libc++abi.a -l:libunwind.a -lpthread -ldl -lrt -lz -lm -z relro -z now -soname libfuzzy.so.2 -L/tmp/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/lib -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed crtendS.o /tmp/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/bin/../lib64/crtn.o
   /tmp/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/bin/ld: cannot find crtbeginS.o: No such file or directory
   clang-8: error: linker command failed with exit code 1 (use -v to see invocation)
   Makefile:490: recipe for target 'libfuzzy.la' failed
   make[1]: *** [libfuzzy.la] Error 1
   make[1]: Leaving directory '/home/sqa/ssdeep-2.14.1'
   Makefile:401: recipe for target 'all' failed
   make: *** [all] Error 2