Comments (11)
same as @NeatMonster, I tried to create a db for libdrm (simple external native project with no dependencies) and codeql cli failed to "see" any source code :
codeql database create -l cpp -c "`pwd`/build/soong/soong_ui.bash --make-mode libdrm" /tmp/aosp-db-cpp
....
[2020-12-23 10:09:46] [build] [100% 711/711] Copy: out/target/product/generic/obj/SHARED_LIBRARIES/libdrm_intermediates/libdrm.so.toc
Finalizing database at /tmp/aosp-db-cpp.
No source code was seen during the build.
I tried the TEMPORARY_PATH_RESTRICTIONS=true
trick described in #47 but it does not work anymore since the env var has been deprecated : https://android-review.googlesource.com/q/I2e3709525598030f5c24d12fa966e531dfed33ae=
My guess of maybe what's going wrong :
- clang execution is now sandboxed under a nsjail worker, which may hinders
codeql
cli interception hooks. I deactivated the sandbox (by rm the nsjail binary) but it didn't solved the issue - soong - the android build system - redefine "make" and handle paths internally instead of relying on Unix conventions. Codeql cli is not the only binary impacted by it, tools like Bear also don't work anymore.
from codeql-cli-binaries.
Can you try using ALLOW_NINJA_ENV=true
. I think that should force our environment variables to be propagated?
from codeql-cli-binaries.
You're giving a --command="m"
option, and codeql
cannot find any m
command to run.
I suspect you're trying to make codeql
execute an alias or shell function have set up in your interactive shell session, but it won't be able to see that.
from codeql-cli-binaries.
Thanks!~
from codeql-cli-binaries.
I tried to create database for android system with full build commands.
But again, I got the error message:
Finalizing database at /home/sweetlie/development/codeql-home/databases/android_201216_0.
No source code was seen during the build.
my log files are here.
https://github.com/sweetlie/codeql-logs/blob/main/build-tracer.log
https://github.com/sweetlie/codeql-logs/blob/main/database-create-20201217.103725.688.log
Could anyone help me?
from codeql-cli-binaries.
I have had a brief look at the logs and it looks to me like this isn't clean build. We can only see compilations that actually happen. Is there a clean command or a clean build command that you can use?
from codeql-cli-binaries.
I'm seeing the exact same thing as @sweetlie: No source code was seen during the build.
Here is what I'm doing (building on a Linux server):
repo init --no-repo-verify -u https://android.googlesource.com/platform/manifest -b <branch>
repo sync -qc -j4
source build/envsetup.sh
lunch
codeql database create -l cpp -c "`pwd`/build/soong/soong_ui.bash --make-mode -j1" /tmp/aosp-database
@alexet I started with a fresh AOSP source tree clone and saw the usual Android compilation steps.
Please note that the targeted language is cpp
, not java
, and that building only "services" yields the same result.
Edit: Still getting No source code was seen during the build.
even if I'm targeting Java instead.
from codeql-cli-binaries.
@alexet I tried it on libdrm and it worked perfectly. I will do the full build now. Thanks a lot!
from codeql-cli-binaries.
Ok, so I think some of the build commands are still not being intercepted:
$ cat log/database-create-20210106.181857.600.log | rg MediaPlayer | rg -v "Scanning|Importing"
2021-01-07 01:18:38] [build] [ 72% 78914/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayerService.cpp
[2021-01-07 01:18:39] [build] [ 72% 78915/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayerClient.cpp
[2021-01-07 01:18:41] [build] [ 72% 78919/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayer.cpp
[2021-01-07 01:18:50] [build] [ 73% 78945/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayerService.cpp [arm]
[2021-01-07 01:18:52] [build] [ 73% 78948/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayerClient.cpp [arm]
[2021-01-07 01:18:55] [build] [ 73% 78954/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayer.cpp [arm]
[2021-01-07 01:19:10] [build] [ 73% 78987/108129] //frameworks/base/media/jni:libmedia_jni clang++ android_media_MediaPlayer.cpp
[2021-01-07 01:19:23] [build] [ 73% 79010/108129] //frameworks/base/media/jni:libmedia_jni clang++ android_media_MediaPlayer.cpp [arm]
[2021-01-07 01:20:02] [build] [ 73% 79070/108129] //frameworks/av/media/libmediaplayerservice:libmediaplayerservice clang++ MediaPlayerFactory.cpp [arm]
[2021-01-07 01:20:13] [build] [ 73% 79085/108129] //frameworks/av/media/libmediaplayerservice:libmediaplayerservice clang++ MediaPlayerService.cpp [arm]
[2021-01-07 01:29:36] [build] [ 73% 79728/108129] //frameworks/wilhelm/src:libwilhelm clang++ android/MediaPlayer_to_android.cpp
[2021-01-07 01:29:42] [build] [ 73% 79736/108129] //frameworks/wilhelm/src:libwilhelm clang++ android/android_GenericMediaPlayer.cpp
[2021-01-07 01:29:48] [build] [ 73% 79749/108129] //frameworks/wilhelm/src:libwilhelm clang++ objects/CMediaPlayer.cpp
[2021-01-07 01:30:15] [build] [ 73% 79795/108129] //frameworks/wilhelm/src:libwilhelm clang++ android/MediaPlayer_to_android.cpp [arm]
[2021-01-07 01:30:20] [build] [ 73% 79804/108129] //frameworks/wilhelm/src:libwilhelm clang++ android/android_GenericMediaPlayer.cpp [arm]
[2021-01-07 01:30:27] [build] [ 73% 79815/108129] //frameworks/wilhelm/src:libwilhelm clang++ objects/CMediaPlayer.cpp [arm]
$ unzip -l src.zip | rg MediaPlayer
2386 01-08-2021 12:40 home/neat/aosp/frameworks/av/media/libmedia/IMediaPlayerClient.cpp
6935 01-08-2021 12:40 home/neat/aosp/frameworks/av/media/libmedia/include/media/IMediaPlayer.h
1426 01-08-2021 12:40 home/neat/aosp/frameworks/av/media/libmedia/include/media/IMediaPlayerClient.h
3832 01-08-2021 12:40 home/neat/aosp/frameworks/av/media/libmedia/include/media/IMediaPlayerService.h
13243 01-08-2021 12:40 home/neat/aosp/frameworks/av/media/libmediaplayerservice/include/MediaPlayerInterface.h
For example, IMediaPlayerService.cpp
and MediaPlayerService.cpp
are not being added to the database.
from codeql-cli-binaries.
Ok, so I think some of the build commands are still not being intercepted:
$ cat log/database-create-20210106.181857.600.log | rg MediaPlayer | rg -v "Scanning|Importing" 2021-01-07 01:18:38] [build] [ 72% 78914/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayerService.cpp [2021-01-07 01:18:39] [build] [ 72% 78915/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayerClient.cpp [2021-01-07 01:18:41] [build] [ 72% 78919/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayer.cpp [2021-01-07 01:18:50] [build] [ 73% 78945/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayerService.cpp [arm] [2021-01-07 01:18:52] [build] [ 73% 78948/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayerClient.cpp [arm] [2021-01-07 01:18:55] [build] [ 73% 78954/108129] //frameworks/av/media/libmedia:libmedia clang++ IMediaPlayer.cpp [arm] [2021-01-07 01:19:10] [build] [ 73% 78987/108129] //frameworks/base/media/jni:libmedia_jni clang++ android_media_MediaPlayer.cpp [2021-01-07 01:19:23] [build] [ 73% 79010/108129] //frameworks/base/media/jni:libmedia_jni clang++ android_media_MediaPlayer.cpp [arm] [2021-01-07 01:20:02] [build] [ 73% 79070/108129] //frameworks/av/media/libmediaplayerservice:libmediaplayerservice clang++ MediaPlayerFactory.cpp [arm] [2021-01-07 01:20:13] [build] [ 73% 79085/108129] //frameworks/av/media/libmediaplayerservice:libmediaplayerservice clang++ MediaPlayerService.cpp [arm] [2021-01-07 01:29:36] [build] [ 73% 79728/108129] //frameworks/wilhelm/src:libwilhelm clang++ android/MediaPlayer_to_android.cpp [2021-01-07 01:29:42] [build] [ 73% 79736/108129] //frameworks/wilhelm/src:libwilhelm clang++ android/android_GenericMediaPlayer.cpp [2021-01-07 01:29:48] [build] [ 73% 79749/108129] //frameworks/wilhelm/src:libwilhelm clang++ objects/CMediaPlayer.cpp [2021-01-07 01:30:15] [build] [ 73% 79795/108129] //frameworks/wilhelm/src:libwilhelm clang++ android/MediaPlayer_to_android.cpp [arm] [2021-01-07 01:30:20] [build] [ 73% 79804/108129] //frameworks/wilhelm/src:libwilhelm clang++ android/android_GenericMediaPlayer.cpp [arm] [2021-01-07 01:30:27] [build] [ 73% 79815/108129] //frameworks/wilhelm/src:libwilhelm clang++ objects/CMediaPlayer.cpp [arm] $ unzip -l src.zip | rg MediaPlayer 2386 01-08-2021 12:40 home/neat/aosp/frameworks/av/media/libmedia/IMediaPlayerClient.cpp 6935 01-08-2021 12:40 home/neat/aosp/frameworks/av/media/libmedia/include/media/IMediaPlayer.h 1426 01-08-2021 12:40 home/neat/aosp/frameworks/av/media/libmedia/include/media/IMediaPlayerClient.h 3832 01-08-2021 12:40 home/neat/aosp/frameworks/av/media/libmedia/include/media/IMediaPlayerService.h 13243 01-08-2021 12:40 home/neat/aosp/frameworks/av/media/libmediaplayerservice/include/MediaPlayerInterface.h
For example,
IMediaPlayerService.cpp
andMediaPlayerService.cpp
are not being added to the database.
I have the same problem. I tried it successfully on libdrm but fail when I make-mode with media
or any target else. It says that No source code was seen during the build.
, too. And again, the intercepted code is less than the source c code.
from codeql-cli-binaries.
The following build script has worked for us to extract AOSP 10 and 11 with CodeQL. Take it with a grain of salt!
# set CODEQL_DIST and CODEQL_DB accordingly
ANDROID_ROOT="$(pwd)"
CODEQL_BINARY="$CODEQL_DIST/codeql"
ALT_JAVAC_PATH=$(mktemp -p /tmp codeql-javac.XXXXXXX)
chmod +x "$ALT_JAVAC_PATH"
cat > "$ALT_JAVAC_PATH" <<EOF
#!/bin/bash
# strip arguments with spaces
args=()
for i in "\$@"; do
if [[ "\$i" =~ " " ]];
then
echo -n
else
args+=("\$i")
fi
done
echo
export _JAVA_OPTIONS="-Xmx80000M"
"$CODEQL_BINARY" database trace-command \
"$CODEQL_DB" \
"$ANDROID_ROOT/prebuilts/jdk/jdk11/linux-x86/bin/javac" -- \
"\${args[@]}"
EOF
source build/envsetup.sh
export ALTERNATE_JAVAC="$ALT_JAVAC_PATH"
mm clean
DISABLE_ARTIFACT_PATH_REQUIREMENTS=true m -j8 droid # set number of threads in -j accordingly
Make sure you set CODEQL_DIST
to the location of your CodeQL install, and CODEQL_DB
to the path of a new database created with something like codeql database init --language=java --begin-tracing --source-root=(aosp location) aosp_db
.
from codeql-cli-binaries.
Related Issues (20)
- vs2008 cpp project errror HOT 4
- CLI can't handle '@' in query file path HOT 11
- [Java][maven] "No source code was seen and extracted" HOT 1
- ERROR: Could not detect a suitable build command for the source checkout HOT 2
- Evaluator log summary generation failed due to `NullPointerException` HOT 2
- Cannot create a database on M1 Mac HOT 4
- Missing Source Files from Database for XNU HOT 7
- Xcode iOS simulator build fails when codeql is injected HOT 14
- CodeQL Cpp Library Errors when Publishing CodeQL Pack HOT 2
- GitHub Code Scanning does not recognize CodeQL Analysis Workflow HOT 2
- A fatal error occurred: IOException while executing process with args: [/home/abs4x/CodeQL/cli/tools/linux64/scc, --by-file, --format, json] HOT 5
- Java heap problem
- Option --overwrite does not overwrite database folder created by --db-cluster HOT 3
- Linux arm64 support? HOT 2
- No supported build system detected. HOT 1
- CodeQL enable breaks mono path HOT 2
- Could not resolve library path for /Users/xxx/codeql/codeql-exm/java/ql/examples HOT 2
- Could not resolve module csharp
- typescript-parser-wrapper fails for tsconfig.json path check on Windows HOT 8
- Changelog entry for 2.13.5 is missing HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from codeql-cli-binaries.