Embed libclang as a submodule about libclang

[sethfowler]

There is actually another reason to do this, which is that I don't think we'll get documentation on Hackage unless we build cleanly on their servers, and it's doubtful they'll have LLVM 3.3 straight away.

[ghorn]

Are you proposing to build llvm and libclang within the cabal project?

[sethfowler]

Yup.

[ghorn]

Using LibClang and llvm haskell bindings, it's always been a pain for me to deal with the underlying libraries. If LibClang handles it for us there are many benifits:

• distribute matching versions of llvm and LibClang for API compatibility
• ensure LibClang works in ghci for everyone by using the shared library.
• users don't have to build llvm if their system doesn't have it (llvm puts tons of stuff in /usr/local/)

I have 3 concerns:

• Build times will be significantly slower (this is totally fine with me though)
• I would want to make sure that this would not have any conflicts with system or user-installed llvm/clang, especially when using the haskell llvm bindings
• Can you do this in a 100% portable way? Otherwise there should be a configure option to use a user-specified library

This would be huge, though

[sethfowler]

Those are some great points, particularly the one about ghci; I'll be paying attention to getting ghci working as I investigate this.

Regarding your concerns:

• I'm hoping to avoid making build times too slow. We really only need to build libclang, not all of llvm, and since libclang only involves the clang frontend and doesn't do any code generation, it may be possible to build it independently of the outer llvm project. I still need to look into this, though; this really depends on exactly how the llvm devs have structured things.
• There definitely won't be any conflict with the system libclang.
• This won't be 100% portable - in particular, I do not plan to support Windows. AFAIK that's the same situation we have now, though. Regardless, I agree that we need an option to use a user-specified copy of libclang.

[sethfowler]

So I've done this, although things are still a bit rough. This is happening on the new branch llvm3.4 (since we're leaving behind what's available in LLVM 3.3). I've only tested on OS X so far, and there everything is working fine so far. I'm able to use LibClang in ghci!

I'm not closing this yet because I want to make sure things work alright on Linux (or for that matter on any computer other than mine), and there is some cleanup yet to be done.

[sethfowler]

@ghorn and @chetant - if you have a chance to test this branch out, please let me know how things go. (Remember to run git submodule update --init before trying to build.)

[ghorn]

Wow, great job. I'm really excited for this feature.

unfortunately:

ghorn@deb-kerman:~/hslibs/LibClang (llvm3.4)$ cabal build
Building llvm and clang...
llvm[0]: Constructing LLVMBuild project information.
llvm[0]: Installing include files
llvm[0]: Making install directory /home/ghorn/hslibs/LibClang/build/out/include/./llvm-c
llvm[0]: Making install directory /home/ghorn/hslibs/LibClang/build/out/include/./llvm-c/Transforms

........... a gazillion lines of llvm building ...............

llvm[4]: Linking Release+Asserts Shared Library libclang.so
llvm[4]: Building Release+Asserts Archive Library libclang.a
llvm[4]: Installing Release+Asserts Archive Library /home/ghorn/hslibs/LibClang/build/out/lib/libclang.a
/usr/bin/ld: /home/ghorn/hslibs/LibClang/build/Release+Asserts/lib/libclangFrontend.a(ASTUnit.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/home/ghorn/hslibs/LibClang/build/Release+Asserts/lib/libclangFrontend.a: could not read symbols: Bad value
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[4]: *** [/home/ghorn/hslibs/LibClang/build/Release+Asserts/lib/libclang.so] Error 1
make[4]: *** Waiting for unfinished jobs....
make[4]: Leaving directory `/home/ghorn/hslibs/LibClang/build/tools/clang/tools/libclang'
make[3]: *** [install] Error 1
make[3]: Leaving directory `/home/ghorn/hslibs/LibClang/build/tools/clang/tools'
make[2]: *** [install] Error 1
make[2]: Leaving directory `/home/ghorn/hslibs/LibClang/build/tools/clang'
make[1]: *** [clang/.makeinstall] Error 2
make[1]: Leaving directory `/home/ghorn/hslibs/LibClang/build/tools'
make: *** [install] Error 1
Ensuring libclang is available as libclang_static...
Building LibClang-0.2.0...
Preprocessing library LibClang-0.2.0...
Warning: unable to find Foreign.GreenCard along the import path: 
  using built-in defaults instead.
[ 1 of 16] Compiling Clang.Internal.ClangApp ( src/Clang/Internal/ClangApp.hs, dist/build/Clang/Internal/ClangApp.o )
[ 2 of 16] Compiling Clang.Internal.FFI ( dist/build/Clang/Internal/FFI.hs, dist/build/Clang/Internal/FFI.o )
[ 3 of 16] Compiling Clang.TranslationUnit ( src/Clang/TranslationUnit.hs, dist/build/Clang/TranslationUnit.o )
[ 4 of 16] Compiling Clang.Traversal  ( src/Clang/Traversal.hs, dist/build/Clang/Traversal.o )
[ 5 of 16] Compiling Clang.Monad      ( src/Clang/Monad.hs, dist/build/Clang/Monad.o )
[ 6 of 16] Compiling Clang.String     ( src/Clang/String.hs, dist/build/Clang/String.o )
[ 7 of 16] Compiling Clang.Completion ( src/Clang/Completion.hs, dist/build/Clang/Completion.o )
[ 8 of 16] Compiling Clang.Diagnostic ( src/Clang/Diagnostic.hs, dist/build/Clang/Diagnostic.o )
[ 9 of 16] Compiling Clang.Source     ( src/Clang/Source.hs, dist/build/Clang/Source.o )
[10 of 16] Compiling Clang.Token      ( src/Clang/Token.hs, dist/build/Clang/Token.o )
[11 of 16] Compiling Clang.Type       ( src/Clang/Type.hs, dist/build/Clang/Type.o )
[12 of 16] Compiling Clang.File       ( src/Clang/File.hs, dist/build/Clang/File.o )
[13 of 16] Compiling Clang.Debug      ( src/Clang/Debug.hs, dist/build/Clang/Debug.o )
[14 of 16] Compiling Clang.Cursor     ( src/Clang/Cursor.hs, dist/build/Clang/Cursor.o )
[15 of 16] Compiling Clang.CrossReference ( src/Clang/CrossReference.hs, dist/build/Clang/CrossReference.o )
[16 of 16] Compiling Clang            ( src/Clang.hs, dist/build/Clang.o )
[ 1 of 16] Compiling Clang.Internal.ClangApp ( src/Clang/Internal/ClangApp.hs, dist/build/Clang/Internal/ClangApp.p_o )
[ 2 of 16] Compiling Clang.Internal.FFI ( dist/build/Clang/Internal/FFI.hs, dist/build/Clang/Internal/FFI.p_o )
[ 3 of 16] Compiling Clang.TranslationUnit ( src/Clang/TranslationUnit.hs, dist/build/Clang/TranslationUnit.p_o )
[ 4 of 16] Compiling Clang.Traversal  ( src/Clang/Traversal.hs, dist/build/Clang/Traversal.p_o )
[ 5 of 16] Compiling Clang.Monad      ( src/Clang/Monad.hs, dist/build/Clang/Monad.p_o )
[ 6 of 16] Compiling Clang.String     ( src/Clang/String.hs, dist/build/Clang/String.p_o )
[ 7 of 16] Compiling Clang.Completion ( src/Clang/Completion.hs, dist/build/Clang/Completion.p_o )
[ 8 of 16] Compiling Clang.Diagnostic ( src/Clang/Diagnostic.hs, dist/build/Clang/Diagnostic.p_o )
[ 9 of 16] Compiling Clang.Source     ( src/Clang/Source.hs, dist/build/Clang/Source.p_o )
[10 of 16] Compiling Clang.Token      ( src/Clang/Token.hs, dist/build/Clang/Token.p_o )
[11 of 16] Compiling Clang.Type       ( src/Clang/Type.hs, dist/build/Clang/Type.p_o )
[12 of 16] Compiling Clang.File       ( src/Clang/File.hs, dist/build/Clang/File.p_o )
[13 of 16] Compiling Clang.Debug      ( src/Clang/Debug.hs, dist/build/Clang/Debug.p_o )
[14 of 16] Compiling Clang.Cursor     ( src/Clang/Cursor.hs, dist/build/Clang/Cursor.p_o )
[15 of 16] Compiling Clang.CrossReference ( src/Clang/CrossReference.hs, dist/build/Clang/CrossReference.p_o )
[16 of 16] Compiling Clang            ( src/Clang.hs, dist/build/Clang.p_o )

src/Clang/Internal/FFI_stub_ffi.c:3:27:
     fatal error: clang-c/Index.h: No such file or directory
compilation terminated.

[sethfowler]

Thanks for trying the branch out!

Looks like this is the problem:

/usr/bin/ld: /home/ghorn/hslibs/LibClang/build/Release+Asserts/lib/libclangFrontend.a(ASTUnit.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC

Oddly, it seems like it's trying to build a shared library even though I disabled that in the configure flags. I'll look into this. It's probably not a hard thing to fix.

[sethfowler]

Now with Travis CI set up (thanks Greg and Chetan!) it's much easier to test things. And it immediately became obvious that I left some hard-coded paths and some unneeded code in Setup.hs. I've pushed some fixes.

[sethfowler]

And we're green on Travis CI! Nice!

Though there are lots of ways in which the build process can be further improved, I think we should handle them in other issues. I consider the initial embedding of libclang a success! Closing.