How to create codeql DB for android? about codeql-cli-binaries

[1orenz0]

same as @NeatMonster, I tried to create a db for libdrm (simple external native project with no dependencies) and codeql cli failed to "see" any source code :

codeql database create -l cpp -c "`pwd`/build/soong/soong_ui.bash --make-mode libdrm" /tmp/aosp-db-cpp
....
[2020-12-23 10:09:46] [build] [100% 711/711] Copy: out/target/product/generic/obj/SHARED_LIBRARIES/libdrm_intermediates/libdrm.so.toc
Finalizing database at /tmp/aosp-db-cpp.
No source code was seen during the build.

I tried the TEMPORARY_PATH_RESTRICTIONS=true trick described in #47 but it does not work anymore since the env var has been deprecated : https://android-review.googlesource.com/q/I2e3709525598030f5c24d12fa966e531dfed33ae=

My guess of maybe what's going wrong :

• clang execution is now sandboxed under a nsjail worker, which may hinders codeql cli interception hooks. I deactivated the sandbox (by rm the nsjail binary) but it didn't solved the issue
• soong - the android build system - redefine "make" and handle paths internally instead of relying on Unix conventions. Codeql cli is not the only binary impacted by it, tools like Bear also don't work anymore.

[alexet]

Can you try using ALLOW_NINJA_ENV=true. I think that should force our environment variables to be propagated?

[hmakholm]

You're giving a --command="m" option, and codeql cannot find any m command to run.

I suspect you're trying to make codeql execute an alias or shell function have set up in your interactive shell session, but it won't be able to see that.

[sweetlie]

Thanks!~

[sweetlie]

I tried to create database for android system with full build commands.
But again, I got the error message:
Finalizing database at /home/sweetlie/development/codeql-home/databases/android_201216_0.
No source code was seen during the build.

my log files are here.
https://github.com/sweetlie/codeql-logs/blob/main/build-tracer.log
https://github.com/sweetlie/codeql-logs/blob/main/database-create-20201217.103725.688.log

Could anyone help me?

[alexet]

I have had a brief look at the logs and it looks to me like this isn't clean build. We can only see compilations that actually happen. Is there a clean command or a clean build command that you can use?

[NeatMonster]

I'm seeing the exact same thing as @sweetlie: No source code was seen during the build.

Here is what I'm doing (building on a Linux server):

repo init --no-repo-verify -u https://android.googlesource.com/platform/manifest -b <branch>
repo sync -qc -j4
source build/envsetup.sh
lunch
codeql database create -l cpp -c "`pwd`/build/soong/soong_ui.bash --make-mode -j1" /tmp/aosp-database

@alexet I started with a fresh AOSP source tree clone and saw the usual Android compilation steps.

Please note that the targeted language is cpp, not java, and that building only "services" yields the same result.

Edit: Still getting No source code was seen during the build. even if I'm targeting Java instead.

[NeatMonster]

@alexet I tried it on libdrm and it worked perfectly. I will do the full build now. Thanks a lot!

[NeatMonster]

Ok, so I think some of the build commands are still not being intercepted:

$ cat log/database-create-20210106.181857.600.log | rg MediaPlayer | rg -v "Scanning|Importing"
2021-01-07 01:18:38] [build] [ 72% 78914/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayerService.cpp
[2021-01-07 01:18:39] [build] [ 72% 78915/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayerClient.cpp
[2021-01-07 01:18:41] [build] [ 72% 78919/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayer.cpp
[2021-01-07 01:18:50] [build] [ 73% 78945/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayerService.cpp [arm]
[2021-01-07 01:18:52] [build] [ 73% 78948/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayerClient.cpp [arm]
[2021-01-07 01:18:55] [build] [ 73% 78954/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayer.cpp [arm]
[2021-01-07 01:19:10] [build] [ 73% 78987/108129] //frameworks/base/media/jni:libmedia_jni clang++ android_media_MediaPlayer.cpp
[2021-01-07 01:19:23] [build] [ 73% 79010/108129] //frameworks/base/media/jni:libmedia_jni clang++ android_media_MediaPlayer.cpp [arm]
[2021-01-07 01:20:02] [build] [ 73% 79070/108129] //frameworks/av/media/libmediaplayerservice:libmediaplayerservice clang++ MediaPlayerFactory.cpp [arm]
[2021-01-07 01:20:13] [build] [ 73% 79085/108129] //frameworks/av/media/libmediaplayerservice:libmediaplayerservice clang++ MediaPlayerService.cpp [arm]
[2021-01-07 01:29:36] [build] [ 73% 79728/108129] //frameworks/wilhelm/src:libwilhelm clang++ android/MediaPlayer_to_android.cpp
[2021-01-07 01:29:42] [build] [ 73% 79736/108129] //frameworks/wilhelm/src:libwilhelm clang++ android/android_GenericMediaPlayer.cpp
[2021-01-07 01:29:48] [build] [ 73% 79749/108129] //frameworks/wilhelm/src:libwilhelm clang++ objects/CMediaPlayer.cpp
[2021-01-07 01:30:15] [build] [ 73% 79795/108129] //frameworks/wilhelm/src:libwilhelm clang++ android/MediaPlayer_to_android.cpp [arm]
[2021-01-07 01:30:20] [build] [ 73% 79804/108129] //frameworks/wilhelm/src:libwilhelm clang++ android/android_GenericMediaPlayer.cpp [arm]
[2021-01-07 01:30:27] [build] [ 73% 79815/108129] //frameworks/wilhelm/src:libwilhelm clang++ objects/CMediaPlayer.cpp [arm]
$ unzip -l src.zip | rg MediaPlayer
     2386  01-08-2021 12:40   home/neat/aosp/frameworks/av/media/libmedia/IMediaPlayerClient.cpp
     6935  01-08-2021 12:40   home/neat/aosp/frameworks/av/media/libmedia/include/media/IMediaPlayer.h
     1426  01-08-2021 12:40   home/neat/aosp/frameworks/av/media/libmedia/include/media/IMediaPlayerClient.h
     3832  01-08-2021 12:40   home/neat/aosp/frameworks/av/media/libmedia/include/media/IMediaPlayerService.h
    13243  01-08-2021 12:40   home/neat/aosp/frameworks/av/media/libmediaplayerservice/include/MediaPlayerInterface.h

For example, IMediaPlayerService.cpp and MediaPlayerService.cpp are not being added to the database.

[pcy190]

Ok, so I think some of the build commands are still not being intercepted:

$ cat log/database-create-20210106.181857.600.log | rg MediaPlayer | rg -v "Scanning|Importing"
2021-01-07 01:18:38] [build] [ 72% 78914/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayerService.cpp
[2021-01-07 01:18:39] [build] [ 72% 78915/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayerClient.cpp
[2021-01-07 01:18:41] [build] [ 72% 78919/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayer.cpp
[2021-01-07 01:18:50] [build] [ 73% 78945/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayerService.cpp [arm]
[2021-01-07 01:18:52] [build] [ 73% 78948/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayerClient.cpp [arm]
[2021-01-07 01:18:55] [build] [ 73% 78954/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayer.cpp [arm]
[2021-01-07 01:19:10] [build] [ 73% 78987/108129] //frameworks/base/media/jni:libmedia_jni clang++ android_media_MediaPlayer.cpp
[2021-01-07 01:19:23] [build] [ 73% 79010/108129] //frameworks/base/media/jni:libmedia_jni clang++ android_media_MediaPlayer.cpp [arm]
[2021-01-07 01:20:02] [build] [ 73% 79070/108129] //frameworks/av/media/libmediaplayerservice:libmediaplayerservice clang++ MediaPlayerFactory.cpp [arm]
[2021-01-07 01:20:13] [build] [ 73% 79085/108129] //frameworks/av/media/libmediaplayerservice:libmediaplayerservice clang++ MediaPlayerService.cpp [arm]
[2021-01-07 01:29:36] [build] [ 73% 79728/108129] //frameworks/wilhelm/src:libwilhelm clang++ android/MediaPlayer_to_android.cpp
[2021-01-07 01:29:42] [build] [ 73% 79736/108129] //frameworks/wilhelm/src:libwilhelm clang++ android/android_GenericMediaPlayer.cpp
[2021-01-07 01:29:48] [build] [ 73% 79749/108129] //frameworks/wilhelm/src:libwilhelm clang++ objects/CMediaPlayer.cpp
[2021-01-07 01:30:15] [build] [ 73% 79795/108129] //frameworks/wilhelm/src:libwilhelm clang++ android/MediaPlayer_to_android.cpp [arm]
[2021-01-07 01:30:20] [build] [ 73% 79804/108129] //frameworks/wilhelm/src:libwilhelm clang++ android/android_GenericMediaPlayer.cpp [arm]
[2021-01-07 01:30:27] [build] [ 73% 79815/108129] //frameworks/wilhelm/src:libwilhelm clang++ objects/CMediaPlayer.cpp [arm]
$ unzip -l src.zip | rg MediaPlayer
     2386  01-08-2021 12:40   home/neat/aosp/frameworks/av/media/libmedia/IMediaPlayerClient.cpp
     6935  01-08-2021 12:40   home/neat/aosp/frameworks/av/media/libmedia/include/media/IMediaPlayer.h
     1426  01-08-2021 12:40   home/neat/aosp/frameworks/av/media/libmedia/include/media/IMediaPlayerClient.h
     3832  01-08-2021 12:40   home/neat/aosp/frameworks/av/media/libmedia/include/media/IMediaPlayerService.h
    13243  01-08-2021 12:40   home/neat/aosp/frameworks/av/media/libmediaplayerservice/include/MediaPlayerInterface.h

For example, IMediaPlayerService.cpp and MediaPlayerService.cpp are not being added to the database.

I have the same problem. I tried it successfully on libdrm but fail when I make-mode with media or any target else. It says that No source code was seen during the build., too. And again, the intercepted code is less than the source c code.

[atorralba]

The following build script has worked for us to extract AOSP 10 and 11 with CodeQL. Take it with a grain of salt!

# set CODEQL_DIST and CODEQL_DB accordingly

ANDROID_ROOT="$(pwd)"
CODEQL_BINARY="$CODEQL_DIST/codeql"
ALT_JAVAC_PATH=$(mktemp -p /tmp codeql-javac.XXXXXXX)
chmod +x "$ALT_JAVAC_PATH"
cat > "$ALT_JAVAC_PATH" <<EOF
#!/bin/bash

# strip arguments with spaces
args=()
for i in "\$@"; do
  if [[ "\$i" =~ " " ]];
  then
    echo -n
  else
    args+=("\$i")
  fi
done
echo
export _JAVA_OPTIONS="-Xmx80000M"
"$CODEQL_BINARY" database trace-command \
  "$CODEQL_DB" \
  "$ANDROID_ROOT/prebuilts/jdk/jdk11/linux-x86/bin/javac" -- \
  "\${args[@]}"
EOF
source build/envsetup.sh 
export ALTERNATE_JAVAC="$ALT_JAVAC_PATH"
mm clean
DISABLE_ARTIFACT_PATH_REQUIREMENTS=true m -j8 droid # set number of threads in -j accordingly

Make sure you set CODEQL_DIST to the location of your CodeQL install, and CODEQL_DB to the path of a new database created with something like codeql database init --language=java --begin-tracing --source-root=(aosp location) aosp_db.