Comments (7)
Some of the program flags:
-Xmx15120M --off-heap-ram=15120 --verbosity=progress --threads=30
from codeql-cli-binaries.
By looking at the output of pstree, I can see that there are 30 "pool" threads:
java(124002)-+-{AsyncAppender-W}(124029)
|-{AsynchronousPro}(124146)
|-{AsynchronousPro}(124148)
|-{AsynchronousPro}(124149)
|-{AsynchronousPro}(124150)
|-{AsynchronousPro}(124151)
|-{AsynchronousPro}(124152)
|-{AsynchronousPro}(124153)
|-{AsynchronousPro}(124154)
|-{AsynchronousPro}(124155)
|-{AsynchronousPro}(124158)
|-{AsynchronousPro}(124159)
|-{AsynchronousPro}(124161)
|-{AsynchronousPro}(124162)
|-{AsynchronousPro}(124189)
|-{AsynchronousPro}(124201)
|-{AsynchronousPro}(124202)
|-{AsynchronousPro}(124203)
|-{AsynchronousPro}(124209)
|-{AsynchronousPro}(124212)
|-{AsynchronousPro}(124214)
|-{AsynchronousPro}(124215)
|-{AsynchronousPro}(124216)
|-{C1 CompilerThre}(124018)
|-{C2 CompilerThre}(124017)
|-{Common-Cleaner}(124024)
|-{Finalizer}(124015)
|-{G1 Conc#0}(124010)
|-{G1 Conc#1}(124108)
|-{G1 Conc#2}(124109)
|-{G1 Conc#3}(124110)
|-{G1 Conc#4}(124111)
|-{G1 Conc#5}(124112)
|-{G1 Main Marker}(124009)
|-{G1 Refine#0}(124011)
|-{G1 Refine#10}(124222)
|-{G1 Refine#11}(124223)
|-{G1 Refine#12}(124224)
|-{G1 Refine#13}(124225)
|-{G1 Refine#1}(124164)
|-{G1 Refine#2}(124165)
|-{G1 Refine#3}(124166)
|-{G1 Refine#4}(124167)
|-{G1 Refine#5}(124168)
|-{G1 Refine#6}(124169)
|-{G1 Refine#7}(124183)
|-{G1 Refine#8}(124184)
|-{G1 Refine#9}(124221)
|-{G1 Young RemSet}(124012)
|-{GC Thread#0}(124008)
|-{GC Thread#10}(124039)
|-{GC Thread#11}(124040)
|-{GC Thread#12}(124041)
|-{GC Thread#13}(124042)
|-{GC Thread#14}(124043)
|-{GC Thread#15}(124044)
|-{GC Thread#16}(124045)
|-{GC Thread#17}(124046)
|-{GC Thread#18}(124047)
|-{GC Thread#19}(124048)
|-{GC Thread#1}(124030)
|-{GC Thread#20}(124049)
|-{GC Thread#21}(124050)
|-{GC Thread#22}(124051)
|-{GC Thread#23}(124052)
|-{GC Thread#24}(124053)
|-{GC Thread#2}(124031)
|-{GC Thread#3}(124032)
|-{GC Thread#4}(124033)
|-{GC Thread#5}(124034)
|-{GC Thread#6}(124035)
|-{GC Thread#7}(124036)
|-{GC Thread#8}(124037)
|-{GC Thread#9}(124038)
|-{Reference Handl}(124014)
|-{Service Thread}(124022)
|-{Signal Dispatch}(124016)
|-{Sweeper thread}(124019)
|-{TimeoutThread}(124181)
|-{TimeoutThread}(124182)
|-{TimeoutThread}(124185)
|-{TimeoutThread}(124187)
|-{TimeoutThread}(124188)
|-{TimeoutThread}(124191)
|-{TimeoutThread}(124192)
|-{TimeoutThread}(124193)
|-{TimeoutThread}(124194)
|-{TimeoutThread}(124196)
|-{TimeoutThread}(124197)
|-{TimeoutThread}(124199)
|-{TimeoutThread}(124200)
|-{TimeoutThread}(124220)
|-{TimeoutThread}(124227)
|-{TimeoutThread}(124231)
|-{TimeoutThread}(124233)
|-{TimeoutThread}(124234)
|-{TimeoutThread}(124235)
|-{TimeoutThread}(124236)
|-{TimeoutThread}(124237)
|-{TimeoutThread}(124238)
|-{VM Periodic Tas}(124023)
|-{VM Thread}(124013)
|-{java}(124003)
|-{pool-1-thread-1}(124078)
|-{pool-1-thread-1}(124087)
|-{pool-1-thread-1}(124088)
|-{pool-1-thread-1}(124089)
|-{pool-1-thread-1}(124090)
|-{pool-1-thread-1}(124091)
|-{pool-1-thread-1}(124092)
|-{pool-1-thread-1}(124093)
|-{pool-1-thread-1}(124094)
|-{pool-1-thread-1}(124095)
|-{pool-1-thread-1}(124096)
|-{pool-1-thread-2}(124079)
|-{pool-1-thread-2}(124097)
|-{pool-1-thread-2}(124098)
|-{pool-1-thread-2}(124099)
|-{pool-1-thread-2}(124100)
|-{pool-1-thread-2}(124101)
|-{pool-1-thread-2}(124102)
|-{pool-1-thread-2}(124103)
|-{pool-1-thread-2}(124104)
|-{pool-1-thread-2}(124105)
|-{pool-1-thread-2}(124106)
|-{pool-1-thread-3}(124080)
|-{pool-1-thread-3}(124107)
|-{pool-1-thread-4}(124081)
|-{pool-1-thread-5}(124082)
|-{pool-1-thread-6}(124083)
|-{pool-1-thread-7}(124084)
|-{pool-1-thread-8}(124085)
`-{pool-1-thread-9}(124086)
from codeql-cli-binaries.
(Previously, I ran this same command with 100GB of memory limit, but it wasn't honored, and it ended up exceeding the machine's total memory).
from codeql-cli-binaries.
I left it running overnight, and memory didn't grow beyond 82GB, but it looked like it wasn't making any progress: there were still 6 queries running, but only one core was at 100%. After 18 hours it hadn't finished so I stopped and started it again.
from codeql-cli-binaries.
For legacy reasons, the CodeQL evaluator treats the off-heap RAM limit as advisory -- it will attempt to stay within it, but if necessary will try to use more. The alternative would be to abort evaluation with an out-of-memory error, but introducing this behaviour would break customers who rely on the analysis concluding while using a little bit more memory than requested.
We're considering whether we can gate this behaviour behind a flag, or perhaps make the limit hard-by-default and provide a flag to treat it as soft again.
from codeql-cli-binaries.
Does database create work the same?
The expensive parts of database creation are not affected by what @p0 wrote. Unfortunately we don't have much in the way of knobs to turn to adjust the memory usage in this phase, although limiting the number of threads should indeed relieve some of the pressure.
Another approach you could try would be to increase the heap limit of the JVM that runs codeql
. You can pass a -J-Xmx1234M
option to codeql database create
, where 1234
is a number of megabytes. (Without this, the JVM defaults to fixed fraction of the physical RAM according to the OS).
from codeql-cli-binaries.
@hmakholm I deleted my comment earlier after figuring out it was unrelated myself, but thank you for providing a solution anyway! =D
from codeql-cli-binaries.
Related Issues (20)
- vs2008 cpp project errror HOT 4
- CLI can't handle '@' in query file path HOT 11
- [Java][maven] "No source code was seen and extracted" HOT 1
- ERROR: Could not detect a suitable build command for the source checkout HOT 2
- Evaluator log summary generation failed due to `NullPointerException` HOT 2
- Cannot create a database on M1 Mac HOT 4
- Missing Source Files from Database for XNU HOT 7
- Xcode iOS simulator build fails when codeql is injected HOT 14
- CodeQL Cpp Library Errors when Publishing CodeQL Pack HOT 2
- GitHub Code Scanning does not recognize CodeQL Analysis Workflow HOT 2
- A fatal error occurred: IOException while executing process with args: [/home/abs4x/CodeQL/cli/tools/linux64/scc, --by-file, --format, json] HOT 5
- Java heap problem
- Option --overwrite does not overwrite database folder created by --db-cluster HOT 3
- Linux arm64 support? HOT 2
- No supported build system detected. HOT 1
- CodeQL enable breaks mono path HOT 2
- Could not resolve library path for /Users/xxx/codeql/codeql-exm/java/ql/examples HOT 2
- Could not resolve module csharp
- typescript-parser-wrapper fails for tsconfig.json path check on Windows HOT 8
- Changelog entry for 2.13.5 is missing HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from codeql-cli-binaries.