Some files missing after database creation about codeql-cli-binaries

[mvanotti]

Hi! I've re-ran this test with CodeQL CLI version 2.2.4, and found similar results.

By looking only at c and cc files, I see that there are 1140 missing files from the database, albeit ~700 of them are test files, but they do appear in compile_commands.json.

[alexet]

We crash while extracting that file so it doesn't end up in the database. We can reproduce it and we are working on a fix.

[matt-gretton-dann]

Can you please give the output of codeql --version? There are several different issues going on here and I want to check that I am seeing the same issues you are.

[mvanotti]

Hi Matt,

$ ./codeql/codeql --version
CodeQL command-line toolchain.
Version: 2.0.1.
Copyright (C) 2019 GitHub, Inc.

I'm using the latest released version downloaded from this github repo

[matt-gretton-dann]

Thank you for that info. I have identified several different issues which in our code base which are being triggered by the fuchsia build. I am working on fixes - but unfortunately they are unlikely to land in the next release of CodeQL. I will keep you updated on how the work is going through this ticket.

[mvanotti]

Cool! Let me know if there's anything I can do to help.

[matt-gretton-dann]

We have fixes internally now for a couple of the main issues (lld not recognised as a linker, and the vast majority of extraction failures). These should make it into the next release of the CodeQL CLI Binaries (v2.0.3). I do not have a date for that release yet.

[hmakholm]

(@matt-gretton-dann actually 2.0.3 will probably be a point release after 2.0.2 to fix just a single brown-paper-bag bug, and probably out tomorrow. The one that contains your fixes this will then be 2.0.4).

[mvanotti]

Do you know if the change made it a release? We are at 2.1.0

[mvanotti]

/zircon/tools/zbi/zbi.cc is now being included in my database. Will continue testing to see if there's something else missing.

[mvanotti]

I ran the following query to list all the files:

import cpp

from File f
select f.getAbsolutePath(), ""

I see 8046 different files.

And also gathered all the files from compile_commands.json used during compilation in fuchsia (both in out/default and out/default.zircon). Doing a sort | uniq, I get 9320 different cc files.

This means that there are ~1.3k files that are missing. This gist contains the list of files in the db and in compile_commands.json.

Some examples of missing files (by eye) are some of the autogenerated fidl files, some unit tests, and some random files that I have no idea why are missing.

[Manouchehri]

@mvanotti Are you still running into this?