ModSecurity 2.7.4 mlogc utilizing 100% CPU about modsecurity

[brenosilva]

Hello,

Maybe you can try use gdb to get some info where is the looping issue ? You can try recompile mgloc with -g option.

Try "attach" gdb command with mlogc pid. Then using "n" and/or "s" commands to steps through the code.

Take a look here: http://httpd.apache.org/dev/debugging.html

The pstack tool can also help us extract info.

Thanks

Breno

[Shuro]

I had the same problem a while ago, the whole website stopped to be accessible. It caused a downtime through bad configured site surveillance with nagios.

dpkg -l apache2*

apache2 2.2.16-6+squeeze11
apache2-mpm-worker 2.2.16-6+squeeze11
apache2-suexec-custom 2.2.16-6+squeeze11
apache2-utils 2.2.16-6+squeeze11
apache2.2-bin 2.2.16-6+squeeze11
apache2.2-common 2.2.16-6+squeeze11

/usr/bin/mlogc -v

ModSecurity Log Collector (mlogc) v2.7.5
APR: compiled="1.4.2"; loaded="1.4.2"
PCRE: compiled="8.2"; loaded="8.02 2010-03-19"
cURL: compiled="7.21.0"; loaded="libcurl/7.21.0 GnuTLS/2.8.6 zlib/1.2.3.4 libidn/1.15"

[Shuro]

Still a problem every now and then. Thats not a big problem with load balancing, but if it is happening with a Single-Webserver project (at 03:00AM), then good night.

[zimmerle]

Hi @Shuro, can you provide more details as suggested on @brenosilva's comment?

[Shuro]

i'm not good in debugging things.

2.8.0 mlogc still segfaults and consumes 100% CPU:

[196862.149968] mlogc[22928]: segfault at 70697a67203a ip 000070697a67203a sp 00007f0fb9eed518 error 14
[196862.149978] mlogc[22922] general protection ip:7f0fc06a2578 sp:7f0fb3ffe550 error:0
[196862.149985]  in libapr-1.so.0.4.2[7f0fc0683000+38000]
[198850.450737] mlogc[5431] general protection ip:7fe39802556b sp:7fe39006d510 error:0 in libapr-1.so.0.4.2[7fe398006000+38000]
[198850.450764] mlogc[5424] general protection ip:7fe39802556b sp:7fe391870500 error:0 in libapr-1.so.0.4.2[7fe398006000+38000]
[198855.859490] mlogc[5433]: segfault at 0 ip 00007f4157c7206c sp 00007fffa35e9e90 error 6 in libpthread-2.11.3.so[7f4157c6b000+17000]

This is 2.8.0 compiled and used on Debian Squeeze. The segfault logged could be the result of 'pkill -9 mlogc`.
After pkill and automatic restart of mlogc the newly started process runs at "normal" CPU usage (far below 100%)
for 1-2 minutes then returns to 100%. The 100% CPU usage alone wouldn't be a problem but Apache then eventually
stops accepting new incoming HTTP connections. Connection requests time out and the only workaround so far:

pkill -9 mlogc && /etc/init.d/apache2 stop && /etc/init.d/apache2 start

[Shuro]

Here we go.

gdb backtrace when mlogc works as it's supposed to be:

% gdb mlogc `pidof mlogc`
[...]
(gdb) bt
#0  0x00007f3034d4d14d in read () from /lib/libpthread.so.0
#1  0x00007f30353f8623 in apr_file_read () from /usr/lib/libapr-1.so.0
#2  0x0000000000405d68 in receive_loop () at mlogc.c:2005
#3  0x000000000040614f in main (argc=<value optimized out>, argv=0x7fff519d5ae8) at mlogc.c:2306

On the same machine shorty afterwards mlogc consumes 100% CPU
and gdb shows this:

% gdb mlogc `pidof mlogc`
GNU gdb (GDB) 7.0.1-debian
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/mlogc...done.
Attaching to program: /usr/bin/mlogc, process 1061
Reading symbols from /usr/lib/libapr-1.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libapr-1.so.0
Reading symbols from /usr/lib/libcurl-gnutls.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libcurl-gnutls.so.4
Reading symbols from /lib/libpcre.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib/libpcre.so.3
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x7f302fe63700 (LWP 4398)]
[New Thread 0x7f3031666700 (LWP 1063)]
[New Thread 0x7f3031e67700 (LWP 1062)]
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libuuid.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libuuid.so.1
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /usr/lib/libidn.so.11...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libidn.so.11
Reading symbols from /usr/lib/liblber-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/liblber-2.4.so.2
Reading symbols from /usr/lib/libldap_r-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libldap_r-2.4.so.2
Reading symbols from /usr/lib/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /usr/lib/libgnutls.so.26...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgnutls.so.26
Reading symbols from /usr/lib/libgcrypt.so.11...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgcrypt.so.11
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /usr/lib/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /usr/lib/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /lib/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libkrb5support.so.0
Reading symbols from /lib/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libkeyutils.so.1
Reading symbols from /usr/lib/libtasn1.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libtasn1.so.3
Reading symbols from /usr/lib/libgpg-error.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgpg-error.so.0
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
0x00007f3034d4cc74 in __lll_lock_wait () from /lib/libpthread.so.0
(gdb) bt
#0  0x00007f3034d4cc74 in __lll_lock_wait () from /lib/libpthread.so.0
#1  0x00007f3034d48179 in _L_lock_953 () from /lib/libpthread.so.0
#2  0x00007f3034d47f9b in pthread_mutex_lock () from /lib/libpthread.so.0
#3  0x00007f30353fd6e8 in apr_thread_mutex_lock () from /usr/lib/libapr-1.so.0
#4  0x00007f30353fdcee in ?? () from /usr/lib/libapr-1.so.0
#5  0x00007f30353fe54a in apr_pool_destroy () from /usr/lib/libapr-1.so.0
#6  0x00007f30353fe448 in apr_pool_destroy () from /usr/lib/libapr-1.so.0
#7  0x0000000000405029 in create_new_worker (lock=0) at mlogc.c:1796
#8  0x0000000000405187 in add_entry (data=<value optimized out>, start_worker=1) at mlogc.c:409
#9  0x0000000000405afa in receive_loop () at mlogc.c:2065
#10 0x000000000040614f in main (argc=<value optimized out>, argv=0x7fff519d5ae8) at mlogc.c:2306

I hope this helps.

Here are the versions from this build:

ModSecurity Log Collector (mlogc) v2.8.0
   APR: compiled="1.4.2"; loaded="1.4.2"
  PCRE: compiled="8.2"; loaded="8.02 2010-03-19"
  cURL: compiled="7.21.0"; loaded="libcurl/7.21.0 GnuTLS/2.8.6 zlib/1.2.3.4 libidn/1.15"

[Shuro]

More:

(gdb) bt
#0  0x00007f48ded453f0 in pthread_mutex_unlock@plt () from /usr/lib/libapr-1.so.0
#1  0x00007f48ded5574a in apr_thread_mutex_unlock () from /usr/lib/libapr-1.so.0
#2  0x00007f48ded564ed in apr_pool_destroy () from /usr/lib/libapr-1.so.0
#3  0x00007f48ded56448 in apr_pool_destroy () from /usr/lib/libapr-1.so.0
#4  0x0000000000405029 in create_new_worker (lock=0) at mlogc.c:1796
#5  0x0000000000405187 in add_entry (data=<value optimized out>, start_worker=1) at mlogc.c:409
#6  0x0000000000405afa in receive_loop () at mlogc.c:2065
#7  0x000000000040614f in main (argc=<value optimized out>, argv=0x7fffcc343628) at mlogc.c:2306


(gdb) bt full
#0  0x00007f48ded453f0 in pthread_mutex_unlock@plt () from /usr/lib/libapr-1.so.0
No symbol table info available.
#1  0x00007f48ded5574a in apr_thread_mutex_unlock () from /usr/lib/libapr-1.so.0
No symbol table info available.
#2  0x00007f48ded564ed in apr_pool_destroy () from /usr/lib/libapr-1.so.0
No symbol table info available.
#3  0x00007f48ded56448 in apr_pool_destroy () from /usr/lib/libapr-1.so.0
No symbol table info available.
#4  0x0000000000405029 in create_new_worker (lock=0) at mlogc.c:1796
        thread = 0x0
#5  0x0000000000405187 in add_entry (data=<value optimized out>, start_worker=1) at mlogc.c:409
No locals.
#6  0x0000000000405afa in receive_loop () at mlogc.c:2065
        rc = <value optimized out>
        fd_stdin = 0x2577df0
        nbytes = 3980
        buf = 0x2567de8 "Connection: Keep-Alive"
        errstr = "\340\061\064\314\377\177\"...
        evnt = <value optimized out>
        curr = 2280
        next = 3980
        done = 0
        drop_next = 0
        buffered_events = 52
        count = 50
        tmp_pool = 0x2578df8
#7  0x000000000040614f in main (argc=<value optimized out>, argv=0x7fffcc343628) at mlogc.c:2306
        opt = 0x25070a0
        rc = <value optimized out>
(gdb)

[celesteking]

Hello dears. I'm hitting the same problem now.

(gdb) bt
#0  0x00002b533e63316f in pthread_mutex_unlock () from /lib64/libpthread.so.0
#1  0x00002b533e8659b0 in apr_thread_mutex_unlock () from /usr/local/apache/lib/libapr-1.so.0
#2  0x00002b533e8668ca in apr_pool_destroy () from /usr/local/apache/lib/libapr-1.so.0
#3  0x00002b533e866834 in apr_pool_destroy () from /usr/local/apache/lib/libapr-1.so.0
#4  0x0000000000404069 in create_new_worker (lock=0) at mlogc.c:1796
#5  0x00000000004043b5 in add_entry (
    data=0x6139468 "xxx.com 137.186.96.90 - - [23/May/2015:10:22:26 --0500] \"GET /favicon.ico HTTP/1.1\" 200 0 \"-\" \"-\" VWCbMthWnAEAAE6@H4AAAAC0 \"-\" /nobody/20150523/20150523-1022/20150523-102226-VWCbMthWnAEAAE"..., 
    start_worker=1) at mlogc.c:409
#6  0x00000000004057a4 in receive_loop (argc=2, argv=0x7fffa78118a8) at mlogc.c:2065
#7  main (argc=2, argv=0x7fffa78118a8) at mlogc.c:2306

This process is ancient as you might see. The newer one doesn't consume much cpu.

4     0 20080     1  29   4 171752  3544 -      RNl  ?        305:09 /usr/local/modsecurity/bin/mlogc /etc/httpd/conf/mlogc.conf
4     0 27756 27755  18   0 171752  3556 pipe_w Sl   ?          0:02 /usr/local/modsecurity/bin/mlogc /etc/httpd/conf/mlogc.conf

# strace -fx -p 20080              
Process 20080 attached with 3 threads - interrupt to quit
[pid 20261] select(0, NULL, NULL, NULL, {0, 5000} <unfinished ...>
[pid 20260] futex(0x60d8fe0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 20261] <... select resumed> )      = 0 (Timeout)
[pid 20261] select(0, NULL, NULL, NULL, {0, 10000}) = 0 (Timeout)
[pid 20261] select(0, NULL, NULL, NULL, {0, 10000}) = 0 (Timeout)
............
[pid 20261] select(0, NULL, NULL, NULL, {0, 10000}) = 0 (Timeout)
[pid 20261] select(0, NULL, NULL, NULL, {0, 10000} <unfinished ...>
Process 20080 detached
Process 20260 detached
Process 20261 detached

strace -fx -p 20080 2>&1 |grep -v 'select(0' produces nothing.

ModSecurity Log Collector (mlogc) v2.8.0
APR: compiled="1.5.1"; loaded="1.5.1"
PCRE: compiled="8.36"; loaded="8.36 2014-09-26"
cURL: compiled="7.38.0"; loaded="libcurl/7.38.0 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5"

[celesteking]

It appears the thread is essentially stuck in:

(gdb) next
811             apr_pool_destroy(pool->child);
1: pool->child = (apr_pool_t *) 0x35585c8
(gdb) 
810         while (pool->child)
1: pool->child = (apr_pool_t *) 0x35585c8
(gdb) 
811             apr_pool_destroy(pool->child);
1: pool->child = (apr_pool_t *) 0x35585c8
(gdb) 
810         while (pool->child)
1: pool->child = (apr_pool_t *) 0x35585c8
(gdb) 
811             apr_pool_destroy(pool->child);
1: pool->child = (apr_pool_t *) 0x35585c8
(gdb) 
810         while (pool->child)
1: pool->child = (apr_pool_t *) 0x35585c8
(gdb) 
811             apr_pool_destroy(pool->child);
1: pool->child = (apr_pool_t *) 0x35585c8
(gdb) 
810         while (pool->child)
1: pool->child = (apr_pool_t *) 0x35585c8

(gdb) p *pool->child
$8 = {parent = 0x3575cf8, child = 0x0, sibling = 0x35585c8, ref = 0x3575d00, cleanups = 0x0, free_cleanups = 0x0, allocator = 0x34c65e0, subprocesses = 0x0, abort_fn = 0, user_data = 0x0, tag = 0x0, active = 0x35585a0, 
  self = 0x35585a0, self_first_avail = 0x3558640 "VWEww9hWnAEAADIFOPUAAADu", pre_cleanups = 0x0}

(gdb) list 811
806
807         /* Destroy the subpools.  The subpools will detach themselve from
808          * this pool thus this loop is safe and easy.
809          */
810         while (pool->child)
811             apr_pool_destroy(pool->child);
812
813         /* Run cleanups */
814         run_cleanups(&pool->cleanups);
815

If you need more info, tell me.

[zimmerle]

I am assuming that it is no longer a problem, therefore I am closing this issue.