abyss7 / dist-clang

Distributed [cross-]compilation for Clang

DistClang is the Clang compiler extension with a client-server infrastructure. It features the distributed cross-platform compilation and the intermediate result caching.

The project consists of 2 executables and a couple of configuration files.

clang is a client part and should reside on the machine where the compilation is invoked. It should replace the invocation of the original compiler.

clangd is a server that has 2 different roles: emitter and absorber.

How to build

First of all do clone with an argument --recurse-submodules and then configure the project:

./build/configure

To build You have to use a recent Clang compiler with C++14 support.

For debugging and local usage

ninja -C out/Debug.gn All
cd out/Debug.gn
ln -s clang clang++

The resulting files clang, clang++ and clangd are located in the out/Debug.gn folder.

Linux DEB and RPM packages

ninja -C out/Release.gn rpm_package deb_package

The resulting packages are:

out/Release.gn/dist-clang_<version>_amd64.deb
out/Release.gn/rpmbuild/RPMS/x86_64/dist-clang-<version>-1.x86_64.rpm

Don't use locally the clang and clangd from the out/Release.gn folder since they are hardcoded to use libraries from /usr/lib/dist-clang folder.

Mac OS X package

ninja -C out/Release.gn pkg_package

The resulting package is out/Release.gn/dist-clang-<version>.pkg

How to configure the emitter

TODO!

How to configure the absorber

TODO!

How to run local compilation

The basics is to make use of dist-clang's clang and clang++ as the compilers. Doing

export CC=/usr/bin/dist-clang/clang CXX=/usr/bin/dist-clang/clang++

possibly should work almost always.

To work properly the dist-clang should know about the real compiler's path and compiler's version.

Use local config file

One way to provide information about a real compiler is to put config file somewhere on the path to the folder where the build is performed. File must be named .distclang and should contain something like this:

path: "third_party/llvm-build/Release+Asserts/bin/clang"
version: "clang version 3.7.0 (trunk 231690)"

Use environment variables

Another way - is to set some env. vars:

export DC_CLANG_PATH="/usr/bin/clang"
export DC_CLANG_VERSION="clang version 3.7.0 (trunk 231690)"

Rely on auto-detect

The last resort is to use dist-clang's auto-detect feature: it tries to find the next clang in the path, that differs from the current binary, i.e. /usr/bin/dist-clang/clang. It's a not recommended and error-prone way, since internally paths are compared as a raw strings - without link resolution, etc.

In any way, if the clang path is provided without version, then the version is carved out of the real clang's output.

dist-clang's Issues

Verify a stored cache objects' integrity

At least by size.

Allow daemon to reload some configuration parts without restart

There should be an investigation about what parts of config are eligible for reload.

The reload should be performed on some signal, i.e. SIGHUP.

Move project from Protocol Buffers to Cap'n'proto

It's supposed to be much faster.

See http://kentonv.github.io/capnproto/ for details.

Investigate Pre-Tokenized Headers

If they may help to speed up the preprocessing step.

See http://clang.llvm.org/docs/PTHInternals.html

Implement store for remote-unfriendly files

There are some preprocessed files, that don't compile on remote side because of warnings like -Wtautological-compare, etc. It would be wise to store hashes for such files and do not attempt to compile them remotely.

Direct cache may not get full dependencies

The problem is with some Clang options that restrict headers in the produced dependency file to the in-project headers only, i.e. it skips system headers.

We need to override such behaviour and store full dependencies.

Implement the load balancer

The balancer should be able to dynamically increase/decrease the number of available local jobs for each daemon - according to the third-party average load on the workstation. Also balancer should be able to dynamically changes the number of available jobs for the remote daemons - to properly balance all workstation in use.

Get from configuration the process run timeouts

It allows to tune timeouts without rebuilding binaries.

Add formatted prefix to log messages

Implement format string with params:

• Time from program start.
• Date and time.
• Thread id.
• Verbosity level.

Two source files with the same content may "collide" in direct cache

Source root can have to identical files in the different directories, which may compile into different object code: for example, if we do #include "file.c" from two different locations, this can mean two different object files must be generated, because contents of file.c from two different locations may not be the same.

When checking direct cache with clangd, we only look at (source file content, command line, version) triple, where command line does not include source file name. And if two source files compile with the same version and flags (which does not look awkward in a typical project), clangd may cache object code from compiling the first file, and then "hit" the cache searching for matching content from the second file, producing wrong object code for it.

I created simple example and attached it. Reproduction is as follows:

malets@ub:/tmp/clang-test$ make clean
rm -f a/file.o a/file.d b/file.o b/file.d lib.so
malets@ub:/tmp/clang-test$ make
cc -c -MD -MF a/file.d -o a/file.o a/file.c
cc -c -MD -MF b/file.d -o b/file.o b/file.c
ld -shared -o lib.so a/file.o b/file.o
malets@ub:/tmp/clang-test$ make clean
rm -f a/file.o a/file.d b/file.o b/file.d lib.so
malets@ub:/tmp/clang-test$ make CC=/usr/bin/dist-clang/clang
/usr/bin/dist-clang/clang -c -MD -MF a/file.d -o a/file.o a/file.c
/usr/bin/dist-clang/clang -c -MD -MF b/file.d -o b/file.o b/file.c
ld -shared -o lib.so a/file.o b/file.o
b/file.o: In function `f':
a/file.c:(.text+0x0): multiple definition of `f'
a/file.o:a/file.c:(.text+0x0): first defined here
make: *** [lib.so] Error 1

Restrict clients on absorber according to their version

Implement some lame check of the client's version to prevent usage of buggy previous versions, or to impose the breakage of backward compatibility.

"Too many opened files" goes as compilation error

The daemon sends this kind of error to client, and the client fails.

Do not replace existing /etc/clangd.conf

Please do not replace existing /etc/clangd.conf when .deb installer is run

Implement Snappy protobuf streams instead of Gzip

The Snappy library proves to be more robust and less consuming than gzip: https://code.google.com/p/snappy/

Do not fail to write entry into malformed SQLite database

In case of free space shortage or any other reason the SQLite file may become malformed. We need to properly handle this situation:

• try to repair,
• drop broken image and recreate it,
• try to write new entry again.

Otherwise, the daemon asserts here: https://github.com/abyss7/dist-clang/blob/master/src/cache/file_cache.cc#L100

Bundle llvm and clang into repository

Requires effective measures for migration to GN build system.

Command '--print-file-name' prints two lines instead of one

This command maybe used to determine the path to some internal files (i.e. to link compiler-rt builtins for ubsan), but when used with dist-clang the client outputs only a filename as a first line. The client should output only a single line provided by real compiler.

Try to move to gflags library

Looks better than tclap.

Solve https://bugzilla.samba.org/show_bug.cgi?id=8424

This bug is about direct cache and its inability to respect the headers that may appear on the include path later.

Implement a fair file cache clean-up

Right now the cache is cleaned in the following way:

• Find the least recent 1-st level folder (folder #1).
• Find the least recent 2-nd level folder (folder #2) inside the folder #1.
• Delete all the least recent files inside the folder #2 until the cache size fits the limit.

Better to make more precise clean-up.

Store actual object and etc. files via client

Right now there is a problem when the build is interrupted. The clang clients are already gone, while the daemon may still produce object files - it's unexpected behavior.

The daemon should produce temporary files and report them to client. If the client is active it should move them to expected paths. Otherwise, daemon should remove temporary files.

Implement framework for custom tools

Refactor the code in the way that it will be easy to implement any custom tool that has the same environment (configs, etc.) as a client or daemon.

Do the cache entry compression in async way

Right now the compression is a part of the method FileCache::DoStore() - it requires to store the whole object inside memory to perform compression. Also, it consumes time of compilation process. If we delay the compression, then we can efficiently copy the object file to the cache, and compress it lately using the memory mappings.

/usr/bin/dist-clang/clang++ -print-prog-name=ld incorrect result

Without setting DC_LOG_LEVELS and DC_LOG_ERROR_MARK env variables client writes

No Clang commands found
Failed to parse driver arguments - see errors above
/usr/bin/ld

to stdout that leads to errors when using within scripts.

Make the cache clean-up more precise

Do remove exactly the least recent used cache entries.

Write README (How to deploy distclang on your servers/clients)

"How to configure the emitter" and "How to configure the absorber" are still empty in README.MD.
So it's hard to use distclang anywhere without good help.

Try to move to Bazel build system

Bazel is "self-hosted" and doesn't require anything to work: unlike GN requires Chromium to build.

Implement connection timeout

The timeout may be set only using non-blocking socket connection and polling the writability of the socket.

Actualize expectations about a compiler path in messages from client

Right now in the code the client won't send a message without a compiler path. Also the client does path replacements inside flags according to the specific compiler path - so it's impossible to send a proper message from client to Emitter without predefined compiler path. We should reflect this logic in daemon's code and tests.

Remove from the cache folder everything unrelated.

Related stuff is a current valid cache files and leveldb folder. Do the clean-up on start-up only.

Allow to print some message on CHECK() or DCHECK() failure

Should look like this:

CHECK(condition) << "condition failed!";

It will be very helpful in diagnostics.

Keep order of compiler arguments after splitting in different lists

Otherwise the #include_next behavior may break in a terrible way.

Auto-generate files from .proto

Use internally-built protobuf compiler.

Integrate with Travis build system

clangd does not correctly exit on SIGTERM and SIGINT in most situations

The cause is the following:

1. Upon receiving a signal, the main thread is joining all the worker threads it has created.
2. File cache has "Cache Resetter Worker" (https://github.com/abyss7/dist-clang/blob/master/src/cache/file_cache.cc#L108), which uses simple thread sleeps without any synchronisation (https://github.com/abyss7/dist-clang/blob/master/src/cache/file_cache.cc#L104).
3. By default, if not overriden in config, sleep duration for this resetter thread is 600 seconds (https://github.com/abyss7/dist-clang/blob/master/src/daemon/configuration.proto#L31).
4. As a result, clangd only exits 10 minutes after receiving termination signal. supervisorctl and other process management tools usually only wait a few seconds before sending SIGKILL, thus clangd always gets killed by this signal and does not exit gracefully.

Implement permissive cache mode

Looks like the direct cache has problems. To debug them it's convenient to compare results of direct cache hit and simple cache hit.

Add 'disabled' option to the client's configuration

It allows to seamlessly and temporarily disable the usage of clangd and run real local compiler.

Collect basic statistics and be able to print them

The list of stats are:

• Количество зря потраченного времени на удалённой стороне
• Максимальное время выполнения локальной компиляции - позволит оценить новый необходимый таймаут
• Количество попаданий в прямой кеш
• Количество попаданий в обычный кеш
• Количество успешных локальных тасков
• Количество успешных удалённых тасков

Store the SQLite file cache index on disk

It will improve the daemon's start time.

Allow to parse Clang args externally

This feature should be turned on in config.

It's required in case the internal command-line parser becomes incompatible with previous Clang versions.

Collect stats about cache hits on each entry