database analyze not honoring the memory limits. about codeql-cli-binaries

[mvanotti]

Some of the program flags:

-Xmx15120M --off-heap-ram=15120 --verbosity=progress  --threads=30

[mvanotti]

By looking at the output of pstree, I can see that there are 30 "pool" threads:

java(124002)-+-{AsyncAppender-W}(124029)
             |-{AsynchronousPro}(124146)
             |-{AsynchronousPro}(124148)
             |-{AsynchronousPro}(124149)
             |-{AsynchronousPro}(124150)
             |-{AsynchronousPro}(124151)
             |-{AsynchronousPro}(124152)
             |-{AsynchronousPro}(124153)
             |-{AsynchronousPro}(124154)
             |-{AsynchronousPro}(124155)
             |-{AsynchronousPro}(124158)
             |-{AsynchronousPro}(124159)
             |-{AsynchronousPro}(124161)
             |-{AsynchronousPro}(124162)
             |-{AsynchronousPro}(124189)
             |-{AsynchronousPro}(124201)
             |-{AsynchronousPro}(124202)
             |-{AsynchronousPro}(124203)
             |-{AsynchronousPro}(124209)
             |-{AsynchronousPro}(124212)
             |-{AsynchronousPro}(124214)
             |-{AsynchronousPro}(124215)
             |-{AsynchronousPro}(124216)
             |-{C1 CompilerThre}(124018)
             |-{C2 CompilerThre}(124017)
             |-{Common-Cleaner}(124024)
             |-{Finalizer}(124015)
             |-{G1 Conc#0}(124010)
             |-{G1 Conc#1}(124108)
             |-{G1 Conc#2}(124109)
             |-{G1 Conc#3}(124110)
             |-{G1 Conc#4}(124111)
             |-{G1 Conc#5}(124112)
             |-{G1 Main Marker}(124009)
             |-{G1 Refine#0}(124011)
             |-{G1 Refine#10}(124222)
             |-{G1 Refine#11}(124223)
             |-{G1 Refine#12}(124224)
             |-{G1 Refine#13}(124225)
             |-{G1 Refine#1}(124164)
             |-{G1 Refine#2}(124165)
             |-{G1 Refine#3}(124166)
             |-{G1 Refine#4}(124167)
             |-{G1 Refine#5}(124168)
             |-{G1 Refine#6}(124169)
             |-{G1 Refine#7}(124183)
             |-{G1 Refine#8}(124184)
             |-{G1 Refine#9}(124221)
             |-{G1 Young RemSet}(124012)
             |-{GC Thread#0}(124008)
             |-{GC Thread#10}(124039)
             |-{GC Thread#11}(124040)
             |-{GC Thread#12}(124041)
             |-{GC Thread#13}(124042)
             |-{GC Thread#14}(124043)
             |-{GC Thread#15}(124044)
             |-{GC Thread#16}(124045)
             |-{GC Thread#17}(124046)
             |-{GC Thread#18}(124047)
             |-{GC Thread#19}(124048)
             |-{GC Thread#1}(124030)
             |-{GC Thread#20}(124049)
             |-{GC Thread#21}(124050)
             |-{GC Thread#22}(124051)
             |-{GC Thread#23}(124052)
             |-{GC Thread#24}(124053)
             |-{GC Thread#2}(124031)
             |-{GC Thread#3}(124032)
             |-{GC Thread#4}(124033)
             |-{GC Thread#5}(124034)
             |-{GC Thread#6}(124035)
             |-{GC Thread#7}(124036)
             |-{GC Thread#8}(124037)
             |-{GC Thread#9}(124038)
             |-{Reference Handl}(124014)
             |-{Service Thread}(124022)
             |-{Signal Dispatch}(124016)
             |-{Sweeper thread}(124019)
             |-{TimeoutThread}(124181)
             |-{TimeoutThread}(124182)
             |-{TimeoutThread}(124185)
             |-{TimeoutThread}(124187)
             |-{TimeoutThread}(124188)
             |-{TimeoutThread}(124191)
             |-{TimeoutThread}(124192)
             |-{TimeoutThread}(124193)
             |-{TimeoutThread}(124194)
             |-{TimeoutThread}(124196)
             |-{TimeoutThread}(124197)
             |-{TimeoutThread}(124199)
             |-{TimeoutThread}(124200)
             |-{TimeoutThread}(124220)
             |-{TimeoutThread}(124227)
             |-{TimeoutThread}(124231)
             |-{TimeoutThread}(124233)
             |-{TimeoutThread}(124234)
             |-{TimeoutThread}(124235)
             |-{TimeoutThread}(124236)
             |-{TimeoutThread}(124237)
             |-{TimeoutThread}(124238)
             |-{VM Periodic Tas}(124023)
             |-{VM Thread}(124013)
             |-{java}(124003)
             |-{pool-1-thread-1}(124078)
             |-{pool-1-thread-1}(124087)
             |-{pool-1-thread-1}(124088)
             |-{pool-1-thread-1}(124089)
             |-{pool-1-thread-1}(124090)
             |-{pool-1-thread-1}(124091)
             |-{pool-1-thread-1}(124092)
             |-{pool-1-thread-1}(124093)
             |-{pool-1-thread-1}(124094)
             |-{pool-1-thread-1}(124095)
             |-{pool-1-thread-1}(124096)
             |-{pool-1-thread-2}(124079)
             |-{pool-1-thread-2}(124097)
             |-{pool-1-thread-2}(124098)
             |-{pool-1-thread-2}(124099)
             |-{pool-1-thread-2}(124100)
             |-{pool-1-thread-2}(124101)
             |-{pool-1-thread-2}(124102)
             |-{pool-1-thread-2}(124103)
             |-{pool-1-thread-2}(124104)
             |-{pool-1-thread-2}(124105)
             |-{pool-1-thread-2}(124106)
             |-{pool-1-thread-3}(124080)
             |-{pool-1-thread-3}(124107)
             |-{pool-1-thread-4}(124081)
             |-{pool-1-thread-5}(124082)
             |-{pool-1-thread-6}(124083)
             |-{pool-1-thread-7}(124084)
             |-{pool-1-thread-8}(124085)
             `-{pool-1-thread-9}(124086)

[mvanotti]

(Previously, I ran this same command with 100GB of memory limit, but it wasn't honored, and it ended up exceeding the machine's total memory).

[mvanotti]

I left it running overnight, and memory didn't grow beyond 82GB, but it looked like it wasn't making any progress: there were still 6 queries running, but only one core was at 100%. After 18 hours it hadn't finished so I stopped and started it again.

[p0]

For legacy reasons, the CodeQL evaluator treats the off-heap RAM limit as advisory -- it will attempt to stay within it, but if necessary will try to use more. The alternative would be to abort evaluation with an out-of-memory error, but introducing this behaviour would break customers who rely on the analysis concluding while using a little bit more memory than requested.

We're considering whether we can gate this behaviour behind a flag, or perhaps make the limit hard-by-default and provide a flag to treat it as soft again.

[hmakholm]

@Manouchehri:

Does database create work the same?

The expensive parts of database creation are not affected by what @p0 wrote. Unfortunately we don't have much in the way of knobs to turn to adjust the memory usage in this phase, although limiting the number of threads should indeed relieve some of the pressure.

Another approach you could try would be to increase the heap limit of the JVM that runs codeql. You can pass a -J-Xmx1234M option to codeql database create, where 1234 is a number of megabytes. (Without this, the JVM defaults to fixed fraction of the physical RAM according to the OS).

[Manouchehri]

@hmakholm I deleted my comment earlier after figuring out it was unrelated myself, but thank you for providing a solution anyway! =D