nickdiego / compiledb

Tool for generating Clang's JSON Compilation Database files for make-based build systems.

Compilation Database Generator

Tool for generating Clang's JSON Compilation Database file for GNU make-based build systems.

It's aimed mainly at non-cmake (cmake already generates compilation database) large codebases. Inspired by projects like YCM-Generator and Bear, but faster (mainly with large projects), since in most cases it doesn't need a clean build (as the mentioned tools do) to generate the compilation database file, to achieve this it uses the make options such as -n/--dry-run and -k/--keep-going to extract the compile commands. Also, it's more cross-compiling friendly than YCM-generator's fake-toolchanin approach.

Installation

# pip install compiledb

• Supports Python 2.x and 3.x (for now, tested only with 2.7 and 3.6 versions)
• For bash completion support, add the content of sh-completion/compiledb.bash file to your .bashrc file, for example.
• ZSH completion coming soon :)

Usage

compiledb provides a make python wrapper script which, besides to execute the make build command, updates the JSON compilation database file corresponding to that build, resulting in a command-line interface similar to Bear.

To generate compile_commands.json file using compiledb's "make wrapper" script, executing Makefile target all:

$ compiledb make

compiledb forwards all the options/arguments passed after make subcommand to GNU Make, so one can, for example, generate compile_commands.json using core/main.mk as main makefile (-f flag), starting the build from build directory (-C flag):

$ compiledb make -f core/main.mk -C build

By default, compiledb make generates the compilation database and runs the actual build command requested (acting as a make wrapper), the build step can be skipped using the -n or --no-build options.

$ compiledb -n make

compiledb base command has been designed so that it can be used to parse compile commands from arbitrary text files (or stdin), assuming it has a build log (ideally generated using make -Bnwk command), and generates the corresponding JSON Compilation database.

For example, to generate the compilation database from build-log.txt file, use the following command.

$ compiledb --parse build-log.txt

or its equivalent:

$ compiledb < build-log.txt

Or even, to pipe make's output and print the compilation database to the standard output:

$ make -Bnwk | compiledb -o-

By default compiledb generates a JSON compilation database in the "arguments" list format. The "command" string format is also supported through the use of the --command-style flag:

$ compiledb --command-style make

Testing / Contributing

I've implemented this tool because I needed to index some AOSP's modules for navigating and studying purposes (after having no satisfatory results with current tools available by the time such as YCM-Generator and Bear). So I've reworked YCM-Generator, which resulted in the initial version of compiledb/parser.py and used successfully to generate compile_commands.json for some AOSP modules in ~1min running in a Docker container and then could use it with some great tools, such as:

• Vim + YouCompleteMe + rtags + chromatica.nvim
• Neovim + LanguageClient-neovim + cquery + deoplete
• Neovim + ALE + ccls

Notice:

• Windows: tested on Windows 10 with cmd, wsl(Ubuntu), mingw32
• Linux: tested only on Arch Linux and Ubuntu 18 so far
• Mac: tested on macOS 10.13 and 10.14

Patches are always welcome :)

License

GNU GPLv3

compiledb's Issues

Support incremental update of compile_commands.json

• If compile_commands.json already exists, update it with the compile commands extract from the input stream. Make this the default behavior
• Add a -f/--overwrite cmd line option to make it possible to make compile_commands.json file to be overwritten

Add basic unit tests

compilation database not generated for OpenWRT

Hi,
I tried to build a using a cross compilation toolchain using make and used compiledb to generate
compilation database, but it kind of hung after the compilation was done for about 10mins. So I Ctrl-C'd
it and got out. I didn't find any json files in my build dir. What do you think could have wrong ? Here is the last output:
make[1]: Leaving directory '/home/sk/openwrt_trunk/attitude-adjustment-38900'
()

Processing build commands from

^Cmake[14748]: *** wait: No child processes. Stop.
make[14748]: *** Waiting for unfinished jobs....
make[14748]: *** wait: No child processes. Stop.
make[14747]: *** wait: No child processes. Stop.
make[14747]: *** Waiting for unfinished jobs....
make[14747]: *** wait: No child processes. Stop.
make[14746]: *** wait: No child processes. Stop.
make[14746]: *** Waiting for unfinished jobs....
make[14746]: *** wait: No child processes. Stop.
make[14745]: *** wait: No child processes. Stop.
make[14745]: *** Waiting for unfinished jobs....
make[14745]: *** wait: No child processes. Stop.
make[14744]: *** wait: No child processes. Stop.
make[14744]: *** Waiting for unfinished jobs....
make[14744]: *** wait: No child processes. Stop.
make[14743]: *** wait: No child processes. Stop.
make[14743]: *** Waiting for unfinished jobs....
make[14743]: *** wait: No child processes. Stop.
make[14742]: *** [/home/sk/openwrt_trunk871/attitude-adjustment-38900/include/verbose.mk] Interrupt
make[14741]: *** [/home/sk/openwrt_trunk871/attitude-adjustment-38900/include/verbose.mk] Interrupt
make[14740]: *** [/home/sk/openwrt_trunk871/attitude-adjustment-38900/include/verbose.mk] Interrupt
make[14739]: *** [/home/sk/openwrt_trunk871/attitude-adjustment-38900/include/verbose.mk] Interr

Does not work with command with wildcard

include ../Makefile.inc

allgen : ${TLPI_LIB}

# compiledb failed to expand *.c
${TLPI_LIB} : *.c ename.c.inc
	${CC} -c -g ${CFLAGS} *.c
	${RM} ${TLPI_LIB}
	${AR} rs ${TLPI_LIB} *.o

ename.c.inc :
	sh Build_ename.sh > ename.c.inc
	echo 1>&2 "ename.c.inc built"

clean :
	${RM} *.o ename.c.inc ${TLPI_LIB}

PS: It seems that bear doesn't have this problem. But to support dry-run, bear needs some hacking.

-n/--no-build with make is ignored in 0.10.0

Trying to run compiledb -n make tries to build even with -n or --no-build.

Have tried on ubuntu 19.04 and Fedora 27, both produce identical results.

Steps to reproduce:

1. pip3 install compiledb==0.9.8
2. compiledb -n make #works
3. pip3 install compiledb==0.10.0
4. compiledb -n make #fails

Here is a cutout where I did this on an intentionally broken project:

atmelfan@gpa-thinkpad:~/Documents/solarbora/projects/src-inverter$ sudo pip3 install compiledb==0.10.0
Collecting compiledb==0.10.0
  Downloading https://files.pythonhosted.org/packages/f1/e9/66676cdfb6665b9b70e0660ea3306976d9f79c742c753d3e6594e60df629/compiledb-0.10.0-py3-none-any.whl
Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from compiledb==0.10.0) (7.0)
Requirement already satisfied: shutilwhich in /usr/local/lib/python3.7/dist-packages (from compiledb==0.10.0) (1.1.0)
Requirement already satisfied: bashlex in /usr/local/lib/python3.7/dist-packages (from compiledb==0.10.0) (0.14)
Installing collected packages: compiledb
  Found existing installation: compiledb 0.9.8
    Uninstalling compiledb-0.9.8:
      Successfully uninstalled compiledb-0.9.8
Successfully installed compiledb-0.10.0
atmelfan@gpa-thinkpad:~/Documents/solarbora/projects/src-inverter$ compiledb -n make
src/main.c: In function ‘main’:
src/main.c:100:7: error: ‘kgomrwaoyrpouiw’ undeclared (first use in this function)
      }kgomrwaoyrpouiw
       ^
src/main.c:100:7: note: each undeclared identifier is reported only once for each function it appears in
src/main.c:101:5: error: expected ‘;’ before ‘}’ token
     }
     ^
make: *** [Makefile:65: build/main.o] Error 1
atmelfan@gpa-thinkpad:~/Documents/solarbora/projects/src-inverter$ sudo pip3 install compiledb==0.9.8
Collecting compiledb==0.9.8
  Downloading https://files.pythonhosted.org/packages/f3/93/cf32ff4c28aa3d22e35ae6f3648abf3d67ffc068d6f946191a7647372857/compiledb-0.9.8-py3-none-any.whl
Requirement already satisfied: bashlex in /usr/local/lib/python3.7/dist-packages (from compiledb==0.9.8) (0.14)
Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from compiledb==0.9.8) (7.0)
Installing collected packages: compiledb
  Found existing installation: compiledb 0.10.0
    Uninstalling compiledb-0.10.0:
      Successfully uninstalled compiledb-0.10.0
Successfully installed compiledb-0.9.8
atmelfan@gpa-thinkpad:~/Documents/solarbora/projects/src-inverter$ compiledb -n make
## Processing build commands from 3
## Loaded compilation database with 6 entries from compile_commands.json
## Writing compilation database with 6 entries to compile_commands.json
## Done.

Doesn't emit subdir.

In the output file, "directory" contains only the project root, not the full path to "file". Then scan-buld does nothing.

I tried to use -e option to skip a directory. (Constructed a regular expression) but for some reason it does not work. I would like to escape parsing one directory which I am not interested in. I would like to skip all the files that have .mk extensions?

Backtick introduced into relative source file paths in compilation database

Similar to this bug, it looks like the relative paths for source files are mangled at the start - the backtick completely breaks things here:

  "'./'`bar-view.cpp"
  ], 
  "file": "'./'`bar-view.cpp"

libtool is probably doing something weird here, as the result is a shared object (XFCE4 panel plugin project).

Version of tool used is f0f79d5 (doesn't look like theres an official version for these scripts).

Slow parsing with automake-generated Makefiles

compiledb make wrapper currently uses the -B / --always-make option to gather compile commands without having to execute an actual clean build. For automake-generated Makefiles, this option leads to very long parsing time, caused by configure script being executed for every make target.

Currently, to work around this issue, one can use a separate temporary build directory (to avoid clean build - useful for huge codebases) and execute make without -B option, for example:

$ export LANG=C.UTF-8
$ mkdir build-tmp && cd build-tmp && ../configure
$ make -nwk | compiledb

(idea) Silent mode

Specifying dash for output file no longer works

After reading your README, I realize that my latest changes break the -o- option. Instead of outputting to stdout, it outputs to a file called -. I'll be working on a PR for this soon. Sorry for the inconvenience.

dir_stack.pop() IndexError: pop from empty list

when i tried to generator compile_commands.json from a makefile build log, it stopped and printed error :

compiledb\parser.py", line 127, in parse_build_log
    dir_stack.pop()
IndexError: pop from empty list

i added a few debug log myself , so the line number may not be accurate but close .
btw, what does the number in "make[3]" , "make[5]" mean ? i think these numbers are not cared by compiledb yet .

Use bashlex to tokenize input lines

Currently bashlex is only being used partially to preprocess the substitution commands (commonly used in autotools generated makefiles). This task is to refactor parsing code to use the bashlex to parse the command lines into a list of CommandNode/WordNode to better extract compilation commands even from complex shell constructions.

Refactor command parser to use OptParse/ArgParse python module

Once the compilation command line and the compiler command is detected we could use python std modules OptParse or ArgParse to parse the command line options, then earning benefits such as simplifying parsing code, reducing the number of bugs, among others.

Next release schedule?

Hi,

the release of 0.9.8 was in 10/2018. We are waiting for a fix that has already been merged to the master to be included in a released version.

Can someone tell when we can expect the next version?

Thanks!

Double pair of backslash and quote

Tested on project xfce4-hardware-monitor-plugin by running both make -nkw | compiledb and compiledb make.

Issue: In compile_commands.json, there are some lines like "-DHARDWARE_MONITOR_GLADEDIR=\"\"/usr/local/share/xfce4-hardware-monitor-plugin/glade/\"\"", which has \"\" at right side.

Expected: "-DHARDWARE_MONITOR_GLADEDIR=\"/usr/local/share/xfce4-hardware-monitor-plugin/glade/\"",

-D option w/o value won't be picked up

The script recognizes switches of the firm - DFOO=bar, but not - DFOO

Include --sysroot=.* in flags_whitelist

Hi

In a test project I have a different sysroot set and discovered that the generated compile_commands file doesn't currently include that flag.
I use the generated file for running clang-tidy and for that to work I need the --sysroot set correctly in compile_commands.
I did a test with adding --sysroot=.* in flags_whitelist myself and that seems to do the trick.

So could you please do that update?

Cheers,

Daniel

c-ipfs cannot generator compiledb correctly

git clone [email protected]:Agorise/c-ipfs.git
cd c-ipfs
compiledb make
it generate a compile_database.json, but it's empty

bear can generate compiledb successfully

Does not produce the same db as bear

Does it ignore assembly file?

(idea/suggestion) quiet/silent flag

Hi,

It would be nice if there were an argument like "-s" to tell compiledb not to print the lines starting with ## or the line with the ().

At the moment I'm using:

compiledb make | grep -v "## *\|()"

when compiling from my terminal, but if I run the build in my editor (Sublime Text 3) the grep fucks up the output (instead of getting real-time printing, it won't print anything until it's done with the whole build).

Regards,
-Falk

compile_commands.json overwritten with empty result if failed or empty result from make

I have a project with many Makefiles in sub folders
I want to produce a single compile_commands.json for use with clang_tidy
Traversing all of the Makefiles and outputting to the same compile_commands should work I think

But when compiledb is given a Makefile which does not produce anything or with an error - then the existing compile_commands.json is overwritten with an empty file - I would expect it to remain same as before.

Of course I could manually filter out those Makefiles, but I would prefer compiledb to work as expected here.

Cheers,

Daniel

Refactor compiledb-parser as a python module

Add bash completion support

add support for a custom make path

Can you please add support for specifying a path to a custom make rather than the one used in system path. This is useful for replacing default make with compiledb so all make commands are automatically redirected to compiledb.
So forexample in a shell script named "make" I want to be able to do compiledb -o commands.json -c /usr/bin/mymake $1 $2 ...
Build systems using custom tools like yocto's bitbake for example I am hoping the above suggestion should work to intercept all make commands.
Kind Regards,
Beri

differenet database generated while using parallel build(-jN)

I noticed different compilation database generated while using parallel build.
I use the following commands to generate build_log.txt

make -w | tee build_log.txt
compiledb < build_log.txt

make -w -j8 | tee build_log.txt
compiledb < build_log.txt

I fetch part of the built log did not generated in compile_commands.json and use both of them run with compiledb.
compiledb generates nothing between squares if build log is parallel one.
but done well with normal one.

part of built log without -jN, build_log_j1.txt

make[2]: Entering directory '/home/user/workspaces/bsp/projects/ats2/app/cli'
cp -f -R man /home/user/workspaces/bsp/projects/ats2/target/mnt/
make[3]: Entering directory '/home/user/workspaces/bsp/projects/ats2/app/cli/src'
armv5-linux-gcc -c -I/home/user/workspaces/bsp/projects/ats2/app/include  -Os -Wall cli.c
armv5-linux-gcc -c -I/home/user/workspaces/bsp/projects/ats2/app/include  -Os -Wall cmd.c
armv5-linux-gcc -o cli cli.o cmd.o -Os -Wall    -L/home/user/workspaces/bsp/projects/ats2/target/usr/lib -lmisc -lperiphery -lcommon -lobj -lreadline
armv5-linux-strip cli
armv5-linux-gcc -c -I/home/user/workspaces/bsp/projects/ats2/app/include  -Os -Wall cliReadFile.c
armv5-linux-gcc -o cliReadFile cliReadFile.o cmd.o -Os -Wall    -L/home/user/workspaces/bsp/projects/ats2/target/usr/lib -lmisc -lperiphery -lcommon -lobj -lreadline
armv5-linux-strip cliReadFile
make[3]: Leaving directory '/home/user/workspaces/bsp/projects/ats2/app/cli/src'
make[2]: Leaving directory '/home/user/workspaces/bsp/projects/ats2/app/cli'
make[2]: Entering directory '/home/user/workspaces/bsp/projects/ats2/app/cli'
make[3]: Entering directory '/home/user/workspaces/bsp/projects/ats2/app/cli/src'
for i in cli cliReadFile ; do \
    cp -f $i /home/user/workspaces/bsp/projects/ats2/target/bin || exit 1; \
done
make[3]: 'install' is up to date.
make[3]: Leaving directory '/home/user/workspaces/bsp/projects/ats2/app/cli/src'
make[2]: Leaving directory '/home/user/workspaces/bsp/projects/ats2/app/cli'

part of built log with -jN, build_log_j8.txt

make[2]: Entering directory '/home/archer/workspaces/bsp/projects/ats2/app/cli'
cp -f -R man /home/archer/workspaces/bsp/projects/ats2/target/mnt/
make[3]: Entering directory '/home/archer/workspaces/bsp/projects/ats2/app/cli'
armv5-linux-gcc -c -I/home/archer/workspaces/bsp/projects/ats2/app/include  -Os -Wall cli.c
armv5-linux-gcc -c -I/home/archer/workspaces/bsp/projects/ats2/app/include  -Os -Wall cmd.c
armv5-linux-gcc -c -I/home/archer/workspaces/bsp/projects/ats2/app/include  -Os -Wall cliReadFile.c
armv5-linux-gcc -o cli cli.o cmd.o -Os -Wall    -L/home/archer/workspaces/bsp/projects/ats2/target/usr/lib -lmisc -lperiphery -lcommon -lobj -lreadline
armv5-linux-gcc -o cliReadFile cliReadFile.o cmd.o -Os -Wall    -L/home/archer/workspaces/bsp/projects/ats2/target/usr/lib -lmisc -lperiphery -lcommon -lobj -lreadline
armv5-linux-strip cli
armv5-linux-strip cliReadFile
make[3]: Leaving directory '/home/archer/workspaces/bsp/projects/ats2/app/cli/src'
make[2]: Leaving directory '/home/archer/workspaces/bsp/projects/ats2/app/cli'
make[2]: Entering directory '/home/archer/workspaces/bsp/projects/ats2/app/cli'
make[3]: Entering directory '/home/archer/workspaces/bsp/projects/ats2/app/cli'
for i in cli cliReadFile ; do \
    cp -f $i /home/archer/workspaces/bsp/projects/ats2/target/bin || exit 1; \
done
make[3]: 'install' is up to date.
make[3]: Leaving directory '/home/archer/workspaces/bsp/projects/ats2/app/cli/src'
make[2]: Leaving directory '/home/archer/workspaces/bsp/projects/ats2/app/cli'

here are the commands i use to generate database

compiledb -p build_log_j1.txt -o j1.json
compiledb -p build_log_j8.txt -o j8.json

two files are basically the same except the order

Parser fails when invalid bash commands are present

Expected behavior: Parser should just ignore those invalid commands
Actual behavior: compiledb fails with a bashlex parsing error and doesn't the compilation db file
Input samples:

checking for gcc... (cached) gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes

Thanks @lyonlh for testing and reporting this.

compiledb not working with autoconf

Hi,

I have 2 projects - one with Makefile and other with autoconf/automake. I'm able to successfully create a database with Makefile but not with automake. It's failing to read local macros generated by autoconf with error "command not found"

AM I missing something?

Thanks!

Handle versioned compiler binaries

The current parser doesn't handle versioned binaries such as clang-5.0, only plain clang.

https://github.com/nickdiego/compiledb/blob/master/compiledb/parser.py#L28:_L29

Would be good if the regular expression could be expanded, to also allow a version suffix ?

https://packages.ubuntu.com/search?keywords=gcc
https://packages.ubuntu.com/search?keywords=clang

Bug on row 67 in parser?

Hi

On row 67 in compiledb-gen-parser, include_path_prefix is given as both second and third argument to parse_flags:
(count, skipped, compile_db) = parse_flags(build_log, include_path_prefix, include_path_prefix)

But second argument to parse_flags is proj_dir:
def parse_flags(build_log, proj_dir, inc_prefix):

So shouldn't row 67 be changed to send in proj_dir as second argument instead?

As it is now I get:
"directory": null,
in all of the generated entries.

Doing the proposed change locally produces the result I expect.

Cheers,

Daniel

compiledb not working with linux kernel

Hi,
when I exec compiledb -n make in Linux kernel source code，It's not working
there is the log
Thanks!

/usr1/jza/linux-5.2.15$ compiledb -n make
Traceback (most recent call last):
  File "/home/jza/.local/bin/compiledb", line 11, in <module>
    sys.exit(cli())
  File "/home/jza/.local/lib/python3.5/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/jza/.local/lib/python3.5/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/jza/.local/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/jza/.local/lib/python3.5/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jza/.local/lib/python3.5/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/jza/.local/lib/python3.5/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/jza/.local/lib/python3.5/site-packages/compiledb/commands/make.py", line 98, in command
    done = generate(**args)
  File "/home/jza/.local/lib/python3.5/site-packages/compiledb/__init__.py", line 106, in generate
    command_style=command_style)
  File "/home/jza/.local/lib/python3.5/site-packages/compiledb/__init__.py", line 54, in generate_json_compdb
    use_full_path=use_full_path, command_style=command_style)
  File "/home/jza/.local/lib/python3.5/site-packages/compiledb/parser.py", line 89, in parse_build_log
    while (line.endswith('\\\n')):
TypeError: endswith first arg must be bytes or a tuple of bytes, not str
jza@dyjza00001:/usr1/jza/linux-5.2.15$ ***
*** Configuration file ".config" not found!
***
*** Please run some configurator (e.g. "make oldconfig" or
*** "make menuconfig" or "make xconfig").
***

compiledb fails with localized make output

The parsing logic appears to be dependent on english-language output. (e.g. "Entering directory ..."). This is not given on localized installations, therefore the parsing will fail, as files are not being correctly found.

To mitigate this, the compiledb make-wrapper should prepend "LC_ALL=C.UTF-8" to the make call, while the documentation should suggest using a supported locale, and a warning should be printed, when compiledb runs in an unsupported locale in --parse/std-in mode.

compiledb incorrectly strip quotes.

Version: using master at commit 88ec08e
Found this while trying to build kernel with compiledb.

> cat a.log
gcc -DKBUILD_MODNAME='"swap"' ./swap.c

> compiledb --parse a.log
## Processing build commands from a.log
## Writing compilation database with 1 entries to compile_commands.json
## Done.

> cat compile_commands.json
[
 {
  "directory": "/tmp",
  "arguments": [
   "gcc",
   "-DKBUILD_MODNAME=swap",
   "./swap.c"
  ],
  "file": "./swap.c"
 }
]

The single quotes in the argument disappear.

Configure Continuous Integration

Restructure as python module

Support skipping compiler flags through a regex command line option

After a recent cleanup in parsing code, the hardcoded logic for ignoring some compiler flags has been completely removed. For most use cases it should be enough to pass the exact build command line, however it may be desirable to skip/ignore some compiler flag to specific use cases (indexing source code, etc..).

A regex-like command line option should work well for this.

Preprocessor directives with spaces are incorrectly parsed

When the command line used to compile a file contains "-D somedefine", the generated compile_commands.json file will contain the string "-D" in the arguments array, instead of "-D somedefine".
If the command line is "-Dsomedefine", it works as expected.

I have checked gcc documentation but I have found no answer to the question of the correct way "-Ddefine" or "-D define", so if you find information that the space is actually forbidden (but tolerated), feel free to close this.

Cannot install compiledb by pip

Under Ubuntu 18.04, cannot install compiledb by pip.

$ pip search compiledb         
compiledb (0.9.0)  - Tool for generating Clang JSON Compilation Database files for
                     make-based build systems.

$ pip install --user compiledb
Collecting compiledb
  Could not find a version that satisfies the requirement compiledb (from versions: 0.9.0.linux-x86_64)
No matching distribution found for compiledb

Support setting AOSP combo from command line option

Currently it tries to use aosp_x86_64-eng by default.

Investigate it better.

TypeError with a project

compiledb fails with xnd-all. Steps to reproduce:

git clone https://github.com/plures/xnd-all.git
cd xnd-all
compiledb -n make install

Output:

## Processing build commands from 3
## Loaded compilation database with 36 entries from compile_commands.json
## Writing compilation database with 36 entries to compile_commands.json
## Done.
Traceback (most recent call last):
  File "/usr/local/bin/compiledb", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/compiledb/commands/make.py", line 103, in command
    pipe.wait()
  File "/usr/local/lib/python3.7/site-packages/compiledb/commands/make.py", line 63, in __exit__
    self.cleanup()
  File "/usr/local/lib/python3.7/site-packages/compiledb/commands/make.py", line 68, in cleanup
    if os.path.isfile(self.path):
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/genericpath.py", line 30, in isfile
    st = os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

examples/compiledb-aosp.sh is broken for AOSP 8.0.0

Hi Nick,
Is there any indication which version of AOSP the compiledb-aosp.sh script work? There seems to be too many issues which breaks the script on AOSP Android 8.0.0.

Support ccache (and other 'wrappers')

Do not assume "compiler" will be the first token in log entries, this should make it work correctly with build using auxiliary tools like ccache.

Commands parsed incorrectly when shell/bash substitution commands are used

I'm currently working on clang static analysis capability with an autotools codebase, so thanks for this tool to plug the compilation database hole!

This is probably an unusual thing, but in my inherited project there are some macros for paths which are speechmark-delimited: HARDWARE_MONITOR_GLADEDIR and PACKAGE_LOCALE_DIR:

libhardwaremonitor_la_CPPFLAGS = \
                        -DHARDWARE_MONITOR_GLADEDIR=\""$(gladedir)/"\" \
                        -DPACKAGE_LOCALE_DIR=\"$(localedir)\" \
                        @LIBXFCE4PANEL_CFLAGS@ \
                        @LIBXFCE4UI_CFLAGS@

in compile_commands.json these result in:

"-DHARDWARE_MONITOR_GLADEDIR=\\\"\"/usr/local/share/xfce4-hardware-monitor-plugin/glade/\"\\\"",
"-DPACKAGE_LOCALE_DIR=\\\"/usr/local/share/locale\\\"",

The resultant clang-tidy call does not consider these macros as defined when they are referenced in code. Doing the following replacements fixes it:

\\\"\" -> \"
\"\\\" -> \"
\\\" -> \"

Saying this, I don't think there is a particular need for speechmark-delimiting paths here since a number of other things in the build system cannot cope with spaces in my experience, so I'll remove that from my project - just noting here in case someone eventually has a good reason for them.

Similar to this bug, it looks like the relative paths for source files are mangled at the start - the backtick completely breaks things here:

  "'./'`bar-view.cpp"
  ], 
  "file": "'./'`bar-view.cpp"

Not sure where this is coming from.

Anyway, after find&replacing, I was able to get clang-tidy analysing the code :) scan-build doesn't work but luckily it isn't needed.

Failure on openssl 1.0.1

When running it on openssl 1.0.1f, there is an error:

##Processing build commands from
make: *** [Makefile] Error 1
make: Failed to remake makefile 'Makefile'.
##Loaded compilation database with 0 entries from compile_commands.json
##Writing compilation database with 0 entries to compile_commands.json

The command I used:
./config
compiledb -n make

Environment: Ubuntu 16.04.5, compiledb is installed with pip.

Error with compiledb-gen-make when no exclude flags specified

When I run compiledb-gen-make without any exclude flags, the program crashes:

## Processing build commands from 'std input'
Traceback (most recent call last):
  File "/home/bart/Tech/code/follow/compiledb-generator/compiledb-gen-parser", line 239, in <module>
    sys.exit(main())
  File "/home/bart/Tech/code/follow/compiledb-generator/compiledb-gen-parser", line 59, in main
    (count, skipped, compile_db) = parse_flags(build_log, proj_dir, include_path_prefix, exclude_list, verbose)
  File "/home/bart/Tech/code/follow/compiledb-generator/compiledb-gen-parser", line 103, in parse_flags
    msg('Error: Regular expression not valid: {}'.format(regex_pattern))
UnboundLocalError: local variable 'regex_pattern' referenced before assignment

The following does work (i.e. an empty exclude list)

compiledb-gen-make -- -e " "

Compiler detection regex will have false positives

The regex for finding the compiler invocation will find non-compilation lines that, for instance, happen to run a command that has something like gcc as an argument , or a switch name (or even part of a switch name).

Not sure how to solve this fully short of parsing shell syntax, but the regex can be tighter than it is now.

Parallel make (when combined with recursive?) results in broken compilation db

Recursive make with parallel build (on project with lots of things being compiled?) is broken. It causes a bunch of errors because the command lines are interleaved with each other. Unclear if the problem reproduces without recursive make but my guess is recursive make + parallel build is the only corner case here.

Seems like --output-sync would be the fix but then I see a bunch of fcntl(): Bad file descriptor lines & at the end I get 0 entries written (& there's no actual compile_commands.json file). It doesn't matter which --output-sync value is used as long as it's not none (e.g. line & recurse have the same issue to varying degrees)

It seems like the correct fix is that compiledb should be overriding MAKE so that recursive make invokes compiledb which puts that compilation DB somewhere else & then at the end all the compilation DBs get merged into 1 file.

Win32 assert should be removed

Hi there,

Just found this awesome project!

I am using a windows dev system and have tested compiledb in the following cases:
Test steps:

1. pip install compiledb
2. Comment out assert not sys.platform.startswith("win32") in site-packages/compiledb/cli.py
3. Call compiledb make from the relevant shell.

Results:

• Windows Subsystem for Linux: Using arm-none-eabi-gcc and makefiles
  • No issues.
  • Generates the compile_commands with the WSL directory style "/mnt/c/' this mean that if you want to use the generated commands in say vscode in windows, vscode cant actually find the files. However this is a limitation with how WSL is implemented and is a side effect of using WSL for anything. It is not any fault of compiledb
• MSYS3 MINGW32 shell: Using xtensa-esp32-elf-gcc and makefiles
  • No issues.
  • Again generates the compile_commands with the directory style "/c/" which cant be used by programs inside windows like vscode
• Windows CMD: Using arm-none-eabi-gcc and makefiles
  • No issues.
  • Generates compile commands that are compatible with windows programs, like vscode

My recommendation is that you should be able to remove the assert not sys.platform.startswith("win32") line from cli.py to enable compiledb to work on all platforms.

Please let me know if you would like me to do any more tests, provideo you with the output compile_commands.json files, or anything else.

Option to produce splitted arguments array (as in pre-0.9.5)

I used compiledb for a while to analyze Makefiles produced by our legacy build system. It was very convenient since compile_commands.json file contained separate field arguments: [] which I could then grep using jq. After recent update 0.9.2 -> 0.9.7 my scripts broke down since now compiledb just copies whole long compile command into a single command field. It would be nice to have an option to add arguments array to the output.

Please add argument to specify input compile database file

This code is assuming that the original compilation database file is called compile_commands.json and is in the same directory where compiledb is executed. Please add a command-line argument to specify the input compilation database file