abyss7 / dist-clang Goto Github PK
View Code? Open in Web Editor NEWDistributed [cross-]compilation for Clang
License: GNU General Public License v3.0
Distributed [cross-]compilation for Clang
License: GNU General Public License v3.0
Looks better than tclap.
Implement format string with params:
Related stuff is a current valid cache files and leveldb folder. Do the clean-up on start-up only.
Refactor the code in the way that it will be easy to implement any custom tool that has the same environment (configs, etc.) as a client or daemon.
Right now the cache is cleaned in the following way:
Better to make more precise clean-up.
Right now there is a problem when the build is interrupted. The clang clients are already gone, while the daemon may still produce object files - it's unexpected behavior.
The daemon should produce temporary files and report them to client. If the client is active it should move them to expected paths. Otherwise, daemon should remove temporary files.
In case of free space shortage or any other reason the SQLite file may become malformed. We need to properly handle this situation:
Otherwise, the daemon asserts here: https://github.com/abyss7/dist-clang/blob/master/src/cache/file_cache.cc#L100
The cause is the following:
clangd
only exits 10 minutes after receiving termination signal. supervisorctl
and other process management tools usually only wait a few seconds before sending SIGKILL
, thus clangd
always gets killed by this signal and does not exit gracefully.The list of stats are:
Should look like this:
CHECK(condition) << "condition failed!";
It will be very helpful in diagnostics.
Requires effective measures for migration to GN build system.
Implement some lame check of the client's version to prevent usage of buggy previous versions, or to impose the breakage of backward compatibility.
The Snappy library proves to be more robust and less consuming than gzip: https://code.google.com/p/snappy/
Use internally-built protobuf compiler.
This feature should be turned on in config.
It's required in case the internal command-line parser becomes incompatible with previous Clang versions.
If they may help to speed up the preprocessing step.
There should be an investigation about what parts of config are eligible for reload.
The reload should be performed on some signal, i.e. SIGHUP.
Without setting DC_LOG_LEVELS and DC_LOG_ERROR_MARK env variables client writes
No Clang commands found
Failed to parse driver arguments - see errors above
/usr/bin/ld
to stdout that leads to errors when using within scripts.
There are some preprocessed files, that don't compile on remote side because of warnings like -Wtautological-compare, etc. It would be wise to store hashes for such files and do not attempt to compile them remotely.
It allows to seamlessly and temporarily disable the usage of clangd and run real local compiler.
The problem is with some Clang options that restrict headers in the produced dependency file to the in-project headers only, i.e. it skips system headers.
We need to override such behaviour and store full dependencies.
It allows to tune timeouts without rebuilding binaries.
Do remove exactly the least recent used cache entries.
Otherwise the #include_next
behavior may break in a terrible way.
Looks like the direct cache has problems. To debug them it's convenient to compare results of direct cache hit and simple cache hit.
This command maybe used to determine the path to some internal files (i.e. to link compiler-rt builtins for ubsan), but when used with dist-clang the client outputs only a filename as a first line. The client should output only a single line provided by real compiler.
Bazel is "self-hosted" and doesn't require anything to work: unlike GN requires Chromium to build.
The balancer should be able to dynamically increase/decrease the number of available local jobs for each daemon - according to the third-party average load on the workstation. Also balancer should be able to dynamically changes the number of available jobs for the remote daemons - to properly balance all workstation in use.
This bug is about direct cache and its inability to respect the headers that may appear on the include path later.
Please do not replace existing /etc/clangd.conf when .deb installer is run
Right now the compression is a part of the method FileCache::DoStore() - it requires to store the whole object inside memory to perform compression. Also, it consumes time of compilation process. If we delay the compression, then we can efficiently copy the object file to the cache, and compress it lately using the memory mappings.
Source root can have to identical files in the different directories, which may compile into different object code: for example, if we do #include "file.c"
from two different locations, this can mean two different object files must be generated, because contents of file.c
from two different locations may not be the same.
When checking direct cache with clangd, we only look at (source file content, command line, version) triple, where command line does not include source file name. And if two source files compile with the same version and flags (which does not look awkward in a typical project), clangd may cache object code from compiling the first file, and then "hit" the cache searching for matching content from the second file, producing wrong object code for it.
I created simple example and attached it. Reproduction is as follows:
malets@ub:/tmp/clang-test$ make clean
rm -f a/file.o a/file.d b/file.o b/file.d lib.so
malets@ub:/tmp/clang-test$ make
cc -c -MD -MF a/file.d -o a/file.o a/file.c
cc -c -MD -MF b/file.d -o b/file.o b/file.c
ld -shared -o lib.so a/file.o b/file.o
malets@ub:/tmp/clang-test$ make clean
rm -f a/file.o a/file.d b/file.o b/file.d lib.so
malets@ub:/tmp/clang-test$ make CC=/usr/bin/dist-clang/clang
/usr/bin/dist-clang/clang -c -MD -MF a/file.d -o a/file.o a/file.c
/usr/bin/dist-clang/clang -c -MD -MF b/file.d -o b/file.o b/file.c
ld -shared -o lib.so a/file.o b/file.o
b/file.o: In function `f':
a/file.c:(.text+0x0): multiple definition of `f'
a/file.o:a/file.c:(.text+0x0): first defined here
make: *** [lib.so] Error 1
The timeout may be set only using non-blocking socket connection and polling the writability of the socket.
"How to configure the emitter" and "How to configure the absorber" are still empty in README.MD.
So it's hard to use distclang anywhere without good help.
Right now in the code the client won't send a message without a compiler path. Also the client does path replacements inside flags according to the specific compiler path - so it's impossible to send a proper message from client to Emitter without predefined compiler path. We should reflect this logic in daemon's code and tests.
It's supposed to be much faster.
See http://kentonv.github.io/capnproto/ for details.
It will improve the daemon's start time.
The daemon sends this kind of error to client, and the client fails.
At least by size.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.