trick-17 / clang-build Goto Github PK

It seems coveralls is not able to merge the coverage report of different platforms currently. See also lemurheavy/coveralls-public#613. In the comments codecov is mentioned. I liked coveralls so far, but since we are developing for multiple platforms, we have no chance with coveralls to get proper coverage reports...

pbr has the naming scheme version + commits since version. This can conflict for multiple branches. I also do not see the necessity to push every commit to pypi. I would limit it to the master branch. Most people will anyways only care for tagged releases.

On CIs with recent enough Clang, try to build a small CUDA kernel.
Since clang 9, also OpenCL can be built: https://clang.llvm.org/docs/UsersManual.html#opencl-features

It should suffice to only generate them if the source file already needs to be (re-)built anyways.
If it does not need to be built, it means that neither the source file nor the headers it depends on changed, i.e. it's depfile would not have changed either.

The current code is at

Subprocess functions should not be called with shell=True (see e.g.: https://stackoverflow.com/questions/3172470/actual-meaning-of-shell-true-in-subprocess). This way you do not have to take care to tailor your call string to different platforms, among other benefits.

Specifying folders instead of files is more sensible, but sometimes a complete glob is too much. Therefore it would be good to have the options to

• manually specify additional source files outside of the given folders
• deactivate globbing of folders (per target)

As discussed in #37 the travis CI tests were not running until the latest changes to #32. Now they are running, but python3 cannot find clang++.

1. Before any target is parsed
2. Before the build, i.e. before any target is compiled
3. Before any target is linked
4. After the build has finished

Specifying any Python script to be executed would suffice, as any configuration steps etc. (e.g. to create a header with version information) could be done well in Python.

Currently the program may continue to run for quite a while, trying to build parent targets of the failed target.

I think we should have a general dependency name for one target on another. An executable might depend on a library. But an executable might also depend on a header only target! The build-system should just know what to do:

dependence on header only library:

• Add include files

dependence on library:

• Add include files
• Add linking

so how about

# Build a library
[mylib]
target_type = "sharedlibrary"
[mylib.sources]
include_directories = ["mylib/include"]
source_directories = ["mylib/src"]

[myheaderlib]
target_type = "headerlibrary"
[myheaderlib.sources]
include_directories = ["myheaderlib/include"]

# Build an executable and link the library
[myexe]
output_name = "runExe"
target_type = "executable"
dependencies = ["mylib", "myheaderlib"]
[myexe.sources]
include_directories = ["myexe/include"]
source_directories = ["myexe/src"]
[myexe.flags]
link = ["-DMYLIB_SOME_DEFINE"]

This depends on #49.

Currently all sources are downloaded and dependencies might in some cases not be resolved properly when using subprojects.

The stages should probably look as follows:

1. Resolve the project names
   • currently, subprojects can be given a name and a url. In the future, only directory should be specified
   • if a subproject does not define its own name, the directory specified in the parent project is used (if nested, the deepest folder name)
2. Resolve dependencies
   • All project sources (i.e. all build configs) are present, so try to create the full dependency DAG. If a dependency is not found, an error message is written
   • From the given options, determine which targets should be built
   • Write the DAG into dotfile (with colours to encode which targets are built)
3. Download targets that should be built
   • Check for presence of sources
   • If not present, download
   • If present and cloned by git, check that the version matches and try to checkout the correct version if it doesn't
4. Configure targets that should be built
   • Search for sources
   • ...
   • Determine sources that changed
   • Generate depfile commands and execute them
   • Determine which sources need to be compiled
5. Build
   • Compile
   • Link
6. Bundle
   • bundle dependencies into bin folders
   • create redistributable packages

Note: alternative to the first two steps of the first stage, we could force project names to be specified by the depending project, but this might make problems when specifying dependencies by name in different projects.

I think the list of include folders is mixed up in the readme. The $PATH normally isn't searched for included headers as far as I am aware for example. I would much rather state that all the paths that clang by default searches plus the include folder are paths which are searched through by default by clang-build.

see also here: https://stackoverflow.com/a/11946295/2305545

Pulling the plug just because something unexpected happend is not a calm response and makes error tracking/debugging tough. It also makes it impossible to include that code in anything else knowing that it could just exit. We should replace the occurrences of sys.exit(...) with proper exceptions which we should deal with in the main function.

There is really no reason to have a copy of the Eigen repository in this repo just to use it in a MWE. We should replace it by a dummy library. That should be enough for the unit tests.

See also https://stackoverflow.com/a/49367391/4069571

• The test should be done without a file
• The test should check a few known numbers (89, 03, 11, 14, 17, 20)
• Numbers should be padded with zeros (currently 3 is checked, but 03 is not)

When a build fails the error messages are only printed after the last target has failed or finished. For long build times this means a lot of waiting for nothing.

Logging setup is part of the Environment constructor. Logging formatting setup, however, should not be part of the clang build main function. Changing this will fix the tests logging everything multiple times.

I think the following flags should be added to the defaults:

-fsanitize=address

see https://clang.llvm.org/docs/AddressSanitizer.html

and also

-Wshadow

seem useful flags for writing clean code. The first one can help identify illegal access mistakes and the second one can help finding shadowing bugs like in e.g. nested loops where the loop variables are shadowed in one of the inner loops.

Looking at:

It seems that while the releaseFlags of the dependencies are included, for the target itself only the default release flags are included.

This is classified as an Error, but as it does not stop the build, it can go unnoticed.
What is the desired behaviour, should this simply stop the entire build or should it only stop the build after compiling everything that can be compiled?

Note that often, this is due to a target where the type was not specified and likely the auto-detection did not give the desired "header only" result. In this case, it should not be built as an executable and it would be better to stop everything before building.

I am currently writing the user guide and I noticed that while it is nice to have a simple name for the target and maybe are more complex name for the output file, by default it would be nice if the target name would be taken over if no output_name is given.

[myexe]

should produce myexe.exe on Windows while

[myexe]
output_name = "MyApp-v1.0-test1234"

should of course behave like before.

It turns out that multiple files can be specified per line (see e.g. here), separated by whitespace, while there may appear whitespace inside a file name, which clang will escape with \ .
At the same time, as far as I can tell, clang does not escape backslashes in Windows paths (maybe it's double backslashes).

We need to either/or

• parse the file correctly, i.e. whitespace and \ \n separate file names and \ whitespace and \\ are part of file names
• use the -MV flag so that file names are in quotes, see here

I tried to do the first but failed, as the strings python parsed from the file, when inserted into pathlib, did not yield correct file names. Splitting the string at whitespace did not work, as the \ whitespace was split as well...

This could be done by allowing e.g.

output_name  = "script: myname.py"
[mylib.sources]
include_directories = "script: mypaths.py"

i.e. if the variable is a string and starts with "script:" and ends with ".py", the contents after the "script:" is interpreted as a path (whitespace should be trimmed).
If the file is found, it is executed and the return is assigned to the corresponding variable.
Then, checks should be run to make sure that the variable has the right form (e.g. a string or a list).

This more general setting of variables would for example allow build processes with platform-dependencies.

Nanogui is a good example of nice usage of submodules. We should either recursively clone by default (there may be arguments against this) or provide an easy way to specify that it is required, e.g. git_submodules = True.

A general question: how can we control naturally and easily which targets are built, and have sane defaults?

When using subprojects, it would make sense to not build all targets by default, but only those which are depended on by any target of the root project. Probably, it should be the command line which allows one to build all targets?

Try to build and run OpenMP code accross platforms

The test was commented out on PR #23 with 57ff743, because we don't yet know how we want to solve rpath etc.
The executable cannot be run and the test is therefore useless.

Currently you are using print statements everywhere. However, if you would use functions like the python logging stuff, you could set different output levels. So people who want to integrate your code into their toolchain could silence your output or select how much output they want.

This could simply create a basic project, to make it easier for people to create projects with a sane structure and basic config which can be easily built with clang-build.

The question is whether this makes sense since the simple kind of project, which could be created without any knowledge about the goals of the project, could be built without any project configuration file...

Maybe by adding more options, such as create type=executable name=myexe this would become more useful. But would it really help anyone?

It could be a nice feature to have e.g. clang-build -bcoverage automatically add the required flags for coverage checks.

This may be sensible to have, e.g. for rebuilding only a single test executable.

If a toml file (even a very simple one), sources are expected to be in the src folder. However, without a toml file, clang-build will look directly in the target folder. To me this has several disadvantages:

• more heterogenious project structures (some with src folder, others just having the source files directly in the top directory
• restructuring of the project when adding a toml file later, or breaking the default (both are cumbersome)
• messy code

I also see the advantage that clang-build is more usable for quick tests as no special folder structure is required. However, creating a single 'src' folder is not really effort and so I am not sure how much this point is actually relevant.

On PR #23, a lot of changes were made and the documentation needs to be updated accordingly.

• update Readme with better example of small config file and a full config file
• update Readme and make it more compact
• add docs folder and readthedocs
• add step by step incremental guide to documentation in docs folder
• add note that boost filesystem can be built with this, as a non-trivial example of a real-world application

The command line should be powerful enough that one rarely needs to edit a build config.
Suggested additional arguments:

• clear build folder (for given targets only?)
• clear argument cache

Also, the given arguments should be cached so that they do not need to be entered repeatedly. Is that a sane default?

Maybe the path should also reflect build type and version numbers?
The question is what should happen if no version is known or the build type is Default or wasn't specified...

• For unknown versions, instead of a folder like v1.0.0 maybe vAny?
• For unspecified/default build type, maybe folder Default?

That aside, it would make sense to have paths such as build/release/targetname/1.0.0/ (with subdirectories for external_sources, obj/source_subfolder, lib, bin, include and possibly others).

Furthermore, one could add a package flag to targets, which could be used to gather binaries and includes for a target and place them in corresponding folders, e.g. build/package/targetname/1.0.0/bin. Instead of a package flag, it could also be a build type (I would always want to create the package, but some may only want to do this when a release is published?).

So:

• build
  • release
    • targetname
      • 1.0.0
        • external_sources
        • obj
        • depfiles
        • lib
        • bin
        • include
      • 1.1.0
        • ...
  • debug
    • ...
  • package
    • targetname
      • 1.0.0
      • 1.1.0

This is very important for modularisation and usage of external projects.
Currently, only targets are allowed to be included from external.

The difference with an external project should be that the corresponding directory (irrespective of being downloaded or a (sub)directory) should be searched for a project file, and tried with a default configuration if none is found (the latter would be equivalent to an empty external target).

The project syntax could for example be

[project: myproject]
[myexe]
output_name  = "run"
dependencies = ["somelibrary", "otherlibrary"]

[project: someproject]
# the project file of this project should define the targets "somelibrary" and "otherlibrary"
external    = true
url         = "https://github.com/someteam/someproject"

While the target-based version would download the same project twice:

[myexe]
output_name  = "run"
dependencies = ["Eigen"]

[somelibrary]
target_type = "shared library"
external    = true
url         = "https://github.com/someteam/someproject"
directory = "somelibrary"

[otherlibrary]
target_type = "shared library"
external    = true
url         = "https://github.com/someteam/someproject"
directory = "otherlibrary"

As with targets, the build directory should reflect if there are multiple projects, e.g.
build/<projectname>/<targetname>/<buildtype>/...
whereas if there's only a single project, it should be fine to have
build/<targetname>/<buildtype>/...

Did we already decide on whether we want to identify folders from target names? E.g.:

my_project
├── shared_headers
|   └── header1.hpp
├── my_executable
|   ├── include
|   |   ├── header1.hpp
|   |   └── header2.hpp
|   └── src
|       ├── src1.cpp
|       └── src2.hpp
├── my_lib
|   ├── include
|   |   └── header3.hpp
|   └── src
|       └── src3.hpp
└── clang-build.toml

[shared_headers] # Do we even need this?

[my_lib]
target_type = "shared library"
output_name  = "the_lib"

[my_executable]
dependencies = ["my_lib", "shared_headers"]

It seems to me that we can infer from the input in the toml if the target is external or not. So maybe it is not needed. This would simplify the toml script and be in line with the idea to make everything as simple/smart as possible.

Maybe I have missed this, but how do we link against boost if it is for example installed as a package on linux?

Is this link: https://raw.githubusercontent.com/GPMueller/clang-build-test/master/clang-build.py in the readme still correct?

In the readme it says

# Include an external package/target (i.e. not from this toml file)
[somelib]
external = true
path = "/path/to/sources"

# Build an executable and link the library
[myexe]
[myexe.sources]
include_directories = ["include", "mylib.sources.include_directories"]
source_directories = ["src"]

[myexe.link]
dependencies = ["somelib"]

Shouldn't it read: somelib.sources.include_directories?

Also I am not sure I understand the example. So we have the linker dependency on somelib. Why do we have to explicitly give it the source files again? This seems redundant and error prone (one might forget one or the other and not directly notice it, because there is an entry already that states that there is a dependency).

Path separators shouldn't be hardcoded into your scripts. Although it works most of the times, there are modules like PathLib2 that take away all the hassle with python path separators. E.g. if you are on windows, you have a folder that starts with U and you manipulate those strings (like adding a filename to the end of the path), python will not like that newly created string (see here: https://docs.python.org/3/howto/unicode.html for the example I gave).

The type of a source could be determined from its file ending (a sensible requirement that C and only C files have .c endings).
The object should then be compiled with the appropriate compiler and the C++ default flags should be replaced by C default flags.

We should definitely build a small MWE for

• a C library linked into a C++ executable (maybe https://libgit2.org/ ?)
• C and C++ objects mixed together in one target

Since pypi is supposed to nicely render the *.rst file, we have to make sure it is bug-free. For this several tools can be put in the CI scripts, see also this discussion:
https://stackoverflow.com/questions/16367770/my-rst-readme-is-not-formatted-on-pypi-python-org

This is a nontrivial problem and if our build system can manage it easily somehow it will be a good proof of concept.

On OSX, python 2.7 is used. It seems that python3 is not even installed. See the output of travis for the master branch and #32 . However, the current version of clang-build only supports python3 which is why no tests are run on the OSX platform.

Checking for changed files in terms of last modified is not the best approach (e.g. automatically generated files that might not change in content but are recreated occasionally). Instead we should try to keep track of changes via hashing of file content. At least for source and header files.

Looking at the code we see that the function to generate the dependency files is called after checking the dependency file for updated headers. So the first time the build is run, obviously no dependency file exists which is ok, as it is clear that we need to compile the first time we run clang-build. However, the second time, the dependency file is checked to see if some of the headers changed. But the dependency file might be outdated and some new headers made it into the target. This is not captured. Or am I missing something here?