Build instructions about llvm-mingw

[mstorsjo]

Had a rather hard time finding out how to build this myself, i suggest a small tutorial on how to go about it, whats needed (build chain) what must be avoided (compilers on path) for all the different build tools (docker Msys2 etc.).

If you have concrete suggestions, feel free to suggest. I haven't ran into issues with other compilers on the path while building so I don't know what to say about that?

Building in msys2 is not very much tested as I only have access to windows in a rather slow virtual machine, otherwise I mainly do cross compilation.

In most cases, people shouldn't need to build it themselves though - the prebuilt releases should work fine unless you want to build (even more) bleeding edge things.

Also there seems to be one small bug when doing ./build-all.sh $someprefix,
the wrapper script tries to create a symlink to a non existing widl.exe which is build later.

Oh, indeed. I think I have a fix for it, I'll test it a little and push a bit later.

[mstorsjo]

Also there seems to be one small bug when doing ./build-all.sh $someprefix,
the wrapper script tries to create a symlink to a non existing widl.exe which is build later.

Oh, indeed. I think I have a fix for it, I'll test it a little and push a bit later.

f80d067 (on a separate branch still) should fix this. Do you happen to have the possibility to test it (without too much hassle)? If not I think it should be fine and I'll push it to master.

[revelator]

Build instructions was mostly because im trying to integrate a similar approach into Msys2/MinGW* build chain (pacman). The Msys2 clang package is rather broken because it relies completely on gcc runtimes and acts up is you try to use clangs own runtimes (these are statically built because the shared runtimes cannot be built with MinGW64 it seems). I been trying to build llvm-mingw with Msys2 and it gets pretty far but ultimately fails because of the missing widl, so i tried something similar to your patch which allowed it build beyond that point but it still fails further down the chain. But like with the widl problem it seems that with a few modifications it should be possible to build the full toolchian on Msys2.

Giving the new commit a try, hang on. :)

[mati865]

@revelator clang package in MSYS2 have to be release version (8.0.0 right now) with backports, this repository relies on pinned trunk commits already making it no-go for MSYS2.

That said MSYS2 clang packages are fixable, I had successfuly used it with compiler-rt and heavily patched libcxx+libcxxabi.
Depending what you want to do compiler-rt should work, libcxx probably needs some backports + config fixes in PKGBUILD, lld needs few backports.

[mstorsjo]

@revelator clang package in MSYS2 have to be release version (8.0.0 right now) with backports, this repository relies on pinned trunk commits already making it no-go for MSYS2.

FWIW, I have a branch named llvm-8.0 also, which uses that release version instead. The main differences to current master of llvm-mingw, with pinned llvm trunk versions, is that master uses llvm-objcopy/strip instead of building binutils for that purpose, and that the shared libc++.dll can be built without --export-all-symbols, and a bunch of windows packaging updates that aren't directly related to the llvm 8.0 branch (that I could backport to that branch).

The Msys2 clang package is rather broken because it relies completely on gcc runtimes and acts up is you try to use clangs own runtimes (these are statically built because the shared runtimes cannot be built with MinGW64 it seems)

In which way is this broken? There's no right or wrong in which of compiler_rt+libunwind or libgcc to use. By default, clang uses/assumes libgcc when building code for the mingw target, so it works as a direct drop-in replacement for gcc for compiling individual object files.

(these are statically built because the shared runtimes cannot be built with MinGW64 it seems).

What runtimes can't be built shared? compiler_rt builtins are always linked statically. libcxx can be built shared, but it's not entirely straightforward, especially when building it on top of libunwind/libcxxabi in a pure from-scratch setup, instead of building on top of an existing libgcc based environment.

Other valuable pieces like compiler_rt sanitizers should be buildable I think, but that requires building with clang and linking (both the sanitizers themselves, and the end user projects) with lld.

My llvm-mingw packages are usable in msys2, but one important gotcha to keep in mind is that I've changed a number of defaults (by passing other default parameters in wrappers - the main clang executable can be a plain vanilla clang), so that object files (and static libraries) built with it are essentially ABI incompatible with the msys2/mingw64 ones.

The primary ABI breaks are from using libc++ instead of libstdc++, and for defaulting to UCRT instead of msvcrt.dll (which breaks the ABI of anything that uses nontrivial CRT functions). This works fine for building projects from scratch with all of their dependencies, but one can't use prebuilt msys2 provided mingw library packages.

im trying to integrate a similar approach into Msys2/MinGW* build chain (pacman)

Which parts are you trying to integrate? As clang and lld already exist packaged, most of the rest of what I provide amounts to changing defaults, and if those defaults are changed (changing default C++ library, or changing the default CRT to UCRT) one essentially has to rebuild the whole set of packages, or keep separate package namespaces.

[revelator]

Hey mati, was not aware you also knew about this compiler :).

Well what i was trying to achieve with Msys2 was an out of source build of libcxxabi libunwind and libcxx like it is done here since im having trouble with missing exports in the mingw build versions when using -stdlib=libcxx or any other combinations where clang uses its own runtimes.

Msys2's clang also seems to have trouble finding the gcc unwinders as i get a lot of undefined references to them when building in the Msys2 shell.

Strangely though it usually works when i build in codeblocks or via windows command shell.

But yes the abi incompatibility is a bit of a problem.

[mati865]

I was building everything at once and fixed issues like this with hacks similar to

llvm-mingw/build-libcxx.sh

Lines 226 to 228 in 7edc154

	llvm-ar qcsL \
	$PREFIX/$arch-w64-mingw32/lib/libc++.a \
	$PREFIX/$arch-w64-mingw32/lib/libunwind.a

[revelator]

Ok so i was on the right track about that, i used a similar approach but the insource build was a nogo so im trying to split things up a bit like its done in llvm-mingws build scripts.

[mstorsjo]

Giving the new commit a try, hang on. :)

@revelator Did this commit seem to work?

I was building everything at once and fixed issues like this with hacks similar to

@mati865 FWIW, with current clang versions, it might be possible to get rid of that one as well. IIRC there's a new option similar to -rtlib=compiler_rt, where one can pass -unwindlib=libunwind which will automatically add -lunwind in addition to -lc++. I haven't yet looked into if that would work throughout the bootstrapping process, or if it messes up some step.

[revelator]

yep worked like a charm when building in docker :) still a nogo in Msys2 though, fails on building libunwind, but the tools and wrappers build.

[mstorsjo]

Ok, I'll have to try the full build in msys2 myself then.

[mati865]

@mstorsjo yeah, it's not available in Clang 8 though so MSYS2 cannot use it.

[revelator]

? it is allthough its not linked in by default by clang, should not be the issue here though since libunwind is build with llvm-mingw's toolchain and not the Msys2 one.

[revelator]

Ill do a build in Msys2 so you can see whats going on when it fails.

[mati865]

I should have said more explicitly. -unwindlib is only available in the trunk.

[revelator]

ah ok np then :-)
btw it also builds in the win10 ubuntu wsl environment if i install the packages mentioned in the docker build script, i have a hunch that the failure to build it with msys2 is due to something on my path that interferes with the build since msys2 can do symlinks though only the windows type so it must link to an existing package. I guess this is also the reason why it barfed on creating the widl symlink since it was not yet built.

[mstorsjo]

I tested building on msys2 now, and now I see the issues you've been describing. These have popped up after last time I tried building in msys2, after I've started installing wrappers without the cross triplet prefix as well.

I updated the widl-symlink branch with more fixes and instructions, and with this, I've completed a build. I'll try to rerun it from scratch later to see if it really works as it should at this point.

[revelator]

cool :) i look forward to the results.

[revelator]

'CMake Error: Unable to open check cache file for write. C:/Msys64/src/llvm-mingw/libunwind/build-i686-shared/CMakeFiles/CMakeTmp/CMakeFiles/cmake.check_cache
CMake Error at C:/Msys64/mingw64/share/cmake-3.13/Modules/CheckCSourceCompiles.cmake:98 (try_compile):
Failed to configure test project build system.
Call Stack (most recent call first):
C:/Msys64/mingw64/share/cmake-3.13/Modules/CheckCCompilerFlag.cmake:50 (CHECK_C_SOURCE_COMPILES)
C:/Msys64/mingw64/lib/cmake/llvm/HandleLLVMOptions.cmake:172 (check_c_compiler_flag)
C:/Msys64/mingw64/lib/cmake/llvm/HandleLLVMOptions.cmake:635 (add_flag_if_supported)
CMakeLists.txt:80 (include)
'

This is the error from libunwind.

[mstorsjo]

Odd, I haven't seen that one.

It's called from a system-wide installed llvm cmake file (C:/Msys64/mingw64/lib/cmake/llvm/HandleLLVMOptions.cmake) which might be of a mismatching version. I didn't have a system-wide installed llvm, that's probably why I didn't run into it. I could try to add an option to have it not pick it up maybe...

If you get rid of (temporarily remove/rename) the /msys2/mingw64/bin/llvm-config* (and wipe the libunwind/build-* dir) it might not call that at all.

You could also try debugging the actual error (CMake Error: Unable to open check cache file for write. C:/Msys64/src/llvm-mingw/libunwind/build-i686-shared/CMakeFiles/CMakeTmp/CMakeFiles/cmake.check_cache). Does that directory exist? What happens if you make it? Not sure if it's a proper issue or just something caused by the llvm cmake version mismatch.

[revelator]

Indeed its the clang 8 package from msys2's mingw64 its picking up :S, i could uninstall the package to have another go at it ?, removing llvm-config is not enough though i just get other errors if i do that from cmake. i just done that and it gets much farther. It builds llvm mingw64 headers crt tools libraries compiler-rt but it still fails on libunwind with the cmake.check_cache but with path pointing to the native bootstrap compiler instead.

[mstorsjo]

Indeed its the clang 8 package from msys2's mingw64 its picking up :S, i could uninstall the package to have another go at it ?, removing llvm-config is not enough though i just get other errors if i do that from cmake. i just done that and it gets much farther. It builds llvm mingw64 headers crt tools libraries compiler-rt but it still fails on libunwind

Not sure I understand here. First you ran into issues in compiling libunwind, right? Now you say you get further, but it fails on libunwind? That doesn't sound to me like it's getting any further at all?

After changing these things, did you wipe the build directories under libunwind? Cmake caches a lot of things, so unless you wipe it, it won't notice the change you did.

with the cmake.check_cache but with path pointing to the native bootstrap compiler instead.

Do you mean it's pointing at a cmake path in the newly built llvm-mingw destination? I don't see how that's possible. By default, the llvm installation there doesn't include llvm-config, so it shouldn't be able to pick up cmake files from there.

Anyway, I'm simply unable to reproduce this issue. I even tried installing the mingw-w64-x86_64-clang package to make sure it finds a mismatched llvm-config from there, but despite this, it still builds libunwind just fine.

Can you share the full log from the part where it tries to build libunwind - not only the error messages without the rest of the context?

[mstorsjo]

But for the general problem, that it picks up a system-wide installed llvm-config, I don't unfortunately see a good way of making it ignore it. You could make the llvm installation a full one that contains llvm-config, by running ./build-llvm.sh --full-llvm <target>, which should take precedence over the potentially broken system-wide one. (If you didn't know - you can run the individual stages of the build, and don't need to run the full ./build-all.sh.)

[revelator]

sorry was a typo it was libcxx that failed now.

i think im having to start from scratch with a clean msys2 since mine allready contains various compilers and they seem to interfere with the build progress.

[teepean]

Would it be possible to install MSYS2 with a barebones x86 and/or x86_64 MinGW without GCC and use clang downloaded from here and that way to start from scratch?

[revelator]

I was hoping to do that, it should work if you modify a few things in /etc so that you can start llvm-mingw as you would mingw*. (Msys2 by default only has package builder support for the clang compiler via makepkg-clang). Be sure to avoid any compilers when installing though (mingw* gcc might be ok since it can bootstrap the llvm tools in the llvm-mingw sources which then take over building the rest).

[revelator]

in /etc/profile

modify to look like this

'unset MINGW_MOUNT_POINT
. '/etc/msystem'
case "${MSYSTEM}" in
#add clang here start
CLANG32)
MINGW_MOUNT_POINT="${MINGW_PREFIX}"
PATH="${MINGW_MOUNT_POINT}/bin:${MINGW_MOUNT_POINT}/opt/bin:${MSYS2_PATH}${ORIGINAL_PATH:+:${ORIGINAL_PATH}}"
PKG_CONFIG_PATH="${MINGW_MOUNT_POINT}/lib/pkgconfig:${MINGW_MOUNT_POINT}/opt/lib/pkgconfig:${MINGW_MOUNT_POINT}/share/pkgconfig"
ACLOCAL_PATH="${MINGW_MOUNT_POINT}/share/aclocal:/usr/share/aclocal"
MANPATH="${MINGW_MOUNT_POINT}/local/man:${MINGW_MOUNT_POINT}/share/man:${MANPATH}"
;;
CLANG64)
MINGW_MOUNT_POINT="${MINGW_PREFIX}"
PATH="${MINGW_MOUNT_POINT}/bin:${MINGW_MOUNT_POINT}/opt/bin:${MSYS2_PATH}${ORIGINAL_PATH:+:${ORIGINAL_PATH}}"
PKG_CONFIG_PATH="${MINGW_MOUNT_POINT}/lib/pkgconfig:${MINGW_MOUNT_POINT}/opt/lib/pkgconfig:${MINGW_MOUNT_POINT}/share/pkgconfig"
ACLOCAL_PATH="${MINGW_MOUNT_POINT}/share/aclocal:/usr/share/aclocal"
MANPATH="${MINGW_MOUNT_POINT}/local/man:${MINGW_MOUNT_POINT}/share/man:${MANPATH}"
;;
#add clang here end
MINGW32)
MINGW_MOUNT_POINT="${MINGW_PREFIX}"
PATH="${MINGW_MOUNT_POINT}/bin:${MINGW_MOUNT_POINT}/opt/bin:${MSYS2_PATH}${ORIGINAL_PATH:+:${ORIGINAL_PATH}}"
PKG_CONFIG_PATH="${MINGW_MOUNT_POINT}/lib/pkgconfig:${MINGW_MOUNT_POINT}/opt/lib/pkgconfig:${MINGW_MOUNT_POINT}/share/pkgconfig"
ACLOCAL_PATH="${MINGW_MOUNT_POINT}/share/aclocal:/usr/share/aclocal"
MANPATH="${MINGW_MOUNT_POINT}/local/man:${MINGW_MOUNT_POINT}/share/man:${MANPATH}"
;;
MINGW64)
MINGW_MOUNT_POINT="${MINGW_PREFIX}"
PATH="${MINGW_MOUNT_POINT}/bin:${MINGW_MOUNT_POINT}/opt/bin:${MSYS2_PATH}${ORIGINAL_PATH:+:${ORIGINAL_PATH}}"
PKG_CONFIG_PATH="${MINGW_MOUNT_POINT}/lib/pkgconfig:${MINGW_MOUNT_POINT}/opt/lib/pkgconfig:${MINGW_MOUNT_POINT}/share/pkgconfig"
ACLOCAL_PATH="${MINGW_MOUNT_POINT}/share/aclocal:/usr/share/aclocal"
MANPATH="${MINGW_MOUNT_POINT}/local/man:${MINGW_MOUNT_POINT}/share/man:${MANPATH}"
;;
*)
PATH="${MSYS2_PATH}${ORIGINAL_PATH:+:${ORIGINAL_PATH}}"
PKG_CONFIG_PATH="/usr/lib/pkgconfig:/usr/share/pkgconfig:/lib/pkgconfig"
esac'

if you use the shell launchers from msys2-launcher-git copy out the mingw32.exe mingw32.ini mingw64.exe and mingw64.ini and rename them to clang32 clang64 and then edit the clang32/64 ini files to read like this ->

clang32.ini
#MSYS=winsymlinks:nativestrict
#MSYS=error_start:mingw64/bin/qtcreator.exe|-debug|
#CHERE_INVOKING=1
#PATH=%SystemRoot%\System32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SystemRoot%\System32\WindowsPowerShell\v1.0
MSYSTEM=CLANG32

clang64.ini
#MSYS=winsymlinks:nativestrict
#MSYS=error_start:mingw64/bin/qtcreator.exe|-debug|
#CHERE_INVOKING=1
#PATH=%SystemRoot%\System32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SystemRoot%\System32\WindowsPowerShell\v1.0
MSYSTEM=CLANG64

copy llvm-mingw i686 build to clang32 and llvm-mingw build x86_64 to clang64 and yank them into the Msys2 folder, you can now use the clang*.exe launchers to spawn a build shell using llvm-mingw

[mstorsjo]

FWIW, I pushed the msys2 building fixes to master now, and removed the widl-symlink branch.

[revelator]

Cool :), btw am i right in running the build scripts like this -> ./build-all.sh $PWD/native and then when done ./build-cross-tools.sh $PWD/native /opt/llvm-mingw x86 / x86_64 ?, atleast this works with ubuntu and win 10 wsl, allthough i have to cp -rL /opt/llvm/mingw /mnt/c/ on win10's ubuntu wsl else i get posix symlinks to a lot of tools.

[mstorsjo]

Yes, that's the right way for cross-building a toolchain from linux. Yup, you need to convert symlinks to individual files when moving them to windows. In my case, that's done implicitly when zipping the build results for distribution.

[revelator]

Still having no luck building it with Msys2, but i created two packages ready for use with Msys2 that use the old msvcrt runtime instead of ucrt for compatibility reasons.

https://sourceforge.net/projects/cbadvanced/files/llvm-mingw/clang32.7z/download
https://sourceforge.net/projects/cbadvanced/files/llvm-mingw/clang64.7z/download

unzip directly in Msys2 folder and use makepkg-clang to build packages with them.
You might need to modify some PKGBUILD scripts because they dont have package definitions for clang builds yet.

Also packages built with llvm-mingw besides pure tools cannot be used with mingw* compilers since the libs seem to be incompatible with ld.

I allready built numerous packages with these compilers, libtool does seem to have some gripe with clang though so id stay clear of that for the time being. Also libffi fails to build with clang (this is known by the developer and being looked into).

[mstorsjo]

Still having no luck building it with Msys2

Can you give instructions for reproducing it, based on a new clean install of Msys2 in a VM or similar?

that use the old msvcrt runtime instead of ucrt for compatibility reasons.

Ok, that should work fine, I'm regularly testing that setup as well

Also packages built with llvm-mingw besides pure tools cannot be used with mingw* compilers since the libs seem to be incompatible with ld.

Do you mean static libraries, or the import libraries of DLLs?

I think the situation should be this:

• Static libraries of pure C libraries should work fine in GCC/binutils environments (if not, let me know), when you've switched to msvcrt, otherwise it wouldn't have been the case. Make sure the GCC/binutils environment uses a close enough version of mingw-w64 headers/libs as well though, as there's been a bit of code changes there as well.
• Static libraries of C++ libraries won't work, as they're built with libc++ headers, which can't be linked to libstdc++ in the end
• For dynamic libraries, the import libraries produced by lld are in MS short import format (as used by MSVC). In my (limited) testing, GNU ld (at least recent versions) should be able to link against them.

libtool does seem to have some gripe with clang though so id stay clear of that for the time being

Yes, this is a very known issue.

Libtool fails when linking a DLL that contains C++ code, due to clang using compiler_rt, and linking to it by passing an absolute path to the .a file, instead of passing -L + -l to the linker. Upstream clang are reluctant to changing this (https://reviews.llvm.org/D51440). Upstream libtool have been sent patches to change it, but is unresponsive. (https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27866) A patch for this has been applied in Msys2's libtool (https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-libtool/0011-Pick-up-clang_rt-static-archives-compiler-internal-l.patch).

Libtool also has another issue with using lld, when linking with long argument lists. Libtool prefers using linker scripts for passing arguments to the linker in those cases, but lld hasn't implemented linker script for the COFF linker part. Lld does support normal response files though (and libtool is capable of using them, but uses a lower priority). This is worked around with another patch in Msys2: https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-libtool/0012-Prefer-response-files-over-linker-scripts-for-mingw-.patch

Unfortunately, due to the way libtool is packaged into projects' tarballs, one has to rerun autoreconf in Msys2 (with a recent enough Msys2 package of libtool, added last fall) in order to get a fixed libtool in there.

[revelator]

Static C libraries might work, not at home atm so i cannot check it yet.
I did have problems with the dynamic ones though, but im using an older version 2.29 of binutils because of some problems building QT5 with binutils versions after 2.29, so that might well be the case.

yes noticed that a few extra patches made it into Msys2's Mingw* libtool :) im going to try with this build when i get home.

Personally i much prefer less use of linker scripts, but in some cases they are hard to avoid, so a working libtool would be nice.

[mstorsjo]

Static C libraries might work, not at home atm so i cannot check it yet.
I did have problems with the dynamic ones though, but im using an older version 2.29 of binutils because of some problems building QT5 with binutils versions after 2.29, so that might well be the case.

I don't think the support for the short import lib format is that new, but I'm not sure. As always, with a reproducible issue I could comment and maybe do something, now I can't.

yes noticed that a few extra patches made it into Msys2's Mingw* libtool :) im going to try with this build when i get home.

Personally i much prefer less use of linker scripts, but in some cases they are hard to avoid, so a working libtool would be nice.

Where else do you see linker script used, where it's hard to avoid?

[revelator]

Mostly in some windows ports of posix utilities, though far less nowadays than in some older packages.
You still run into them from time to time though, mostly because working around use of them is more work than it is worth or would require a massive rewrite. Graphwiz comes to mind as one package that relies a good deal on libtool scripting.

[mstorsjo]

I think you're referring to something else. Linker script is not libtool scripts. Linker script is a GNU ld feature for controlling the output binary layout exactly. It's a large invasive feature which lld doesn't implement for the COFF backend. It has nothing to do with libtool.

The only touching point is that libtool used this feature as a mechanism for passing lists of filenames to the linker.

[revelator]

Btw, i tried adding the z3 static checker library dependency, to the clang build but its crashing the compiler when built with clang itself.

Need to investigate what causes this, also not sure if worth it, depends if it works on both 32 and 64 bit builds. Have you experimented with this option yourself ?.

[revelator]

Ah sorry yes, i was refering to libtool scripts.

[mstorsjo]

I haven't tried enabling Z3 so far, no.

[revelator]

took a few tries but i managed to build it with z3 enabled.
I should warn that it was not the most obvious thing to do, simply installing it from ubuntus repos does not work since it needs a windows version for the final compile.

Also after compiling and installing it into the native bootstrapper, you need to run ldconfig or you will get errors about z3 not being able to load shared objects.

[revelator]

Had some time to do a new build using msys2.
One problem i run into is that while i can build the native cross compiler with msys2 mingw64 in a clean env, ld.lld.exe hangs on any configure tests. Hooking it up to gdb i noticed its winpthreads that hangs it in a semaphore call. Seems all bugs with winpthreads are not yet ironed out :S

[mstorsjo]

Yes, there's been some issues there. There were some fixes for that in the last few months in winpthreads though, but I'm not sure if the implementation still is free from all potential deadlocks.

[revelator]

Would seem its not, going to try a build with the old pthreads-w32 to see if that is also affected.

[mati865]

@revelator you can try building newer winpthreads, at least one hang bug has been fixed: msys2/MINGW-packages#5610

[ppigazzini]

Hello everybody, going here from msys2/MINGW-packages#5610 . The new libwinpthread pulled with the latest msys2 update seems to build a bug free Stockfish chess engine. Here are my tests official-stockfish/Stockfish#2305 (comment)

[mati865]

This issue (build instructions) seems fixed.