Failure when exiting worker process about nginx-module-vts

[vozlt]

Hi, Thanks for reporting.
I'd like to ask some questions without the problem that status variables(active,waiting...) spiked.

1. Are you saying nginx does not properly terminated when running nginx stop or nginx reload?
2. If it(1) doesn't, Does the error(exited on signal 11) frequently appears?

[natnet00]

If I do an nginx reload, everything seems to work as expected. There are no abnormal status codes in the logs.

The "exited on signal 11" error happens a couple of times per day. e.g. 11 times yesterday while handling > 100k requests.

I guess there is something segfaulting?

[vozlt]

Hi, Thanks for reporting.
There has been similar issue #35.
As mentioned issue above, the abnormal values(active,waiting...) probably seem to happen because of the failure to decrease the status variable values(active,waiting...). In other words, cause of problem that the status variables spiked seems to due to the terminating of worker process by signal 11 before running decrease process. So we need to find out why worker process exited on signal 11 to solve the problem.

Some questions

1. What version of the nginx-module-vts do you use?
2. If you disable the nginx-module-vts, Does it also happen in logs(worker process by signal 11)?
3. Please try backtrace(core dump): See the #35 (comment)

Status variables

The status variables used in ngx_http_stub_status_module and nginx-module-vts are a shared memory variables(atomic). Some variables are only increased without decrease as follows:

stats	increase	decrease
accepted	yes	no
handled	yes	no
requests	yes	no
active	yes	yes
reading	yes	yes
writing	yes	yes
waiting	yes	yes

Increase function

(void) ngx_atomic_fetch_add($ngx_stat_variable, 1);

Decrease function

(void) ngx_atomic_fet

[natnet00]

1. I can answer the first question right now:
   If I disable nginx-module-vts, I don't see any "exited on signal 11" error.

2. I've set "max core file size" to unlimited and restarted nginx now. I'll write again as soon as I have a core dump.

[natnet00]

I've got this after entering:
gdb /usr/sbin/nginx /tmp/nginx-coredump-2016-08-19
backtrace full

I guess I need to recompile nginx with certain flags?

Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00000000004dd1f0 in ?? ()
(gdb) backtrace full
#0 0x00000000004dd1f0 in ?? ()
No symbol table info available.
#1 0x000000000047c437 in ?? ()
No symbol table info available.
#2 0x000000000047df32 in ngx_http_free_request ()
No symbol table info available.
#3 0x00000000004a9991 in ngx_http_v2_close_stream ()
No symbol table info available.
#4 0x000000000047e091 in ?? ()
No symbol table info available.
#5 0x000000000047e75a in ?? ()
No symbol table info available.
#6 0x000000000047f4af in ngx_http_finalize_request ()
No symbol table info available.
#7 0x000000000047f6c7 in ngx_http_finalize_request ()
No symbol table info available.
#8 0x000000000048ef15 in ?? ()
No symbol table info available.
#9 0x0000000000491be0 in ?? ()
No symbol table info available.
#10 0x000000000048f068 in ?? ()
No symbol table info available.
#11 0x0000000000469c2c in ?? ()
No symbol table info available.
#12 0x0000000000460434 in ngx_process_events_and_timers ()
No symbol table info available.
#13 0x0000000000467c25 in ?? ()
No symbol table info available.
#14 0x000000000046630f in ngx_spawn_process ()
No symbol table info available.
#15 0x0000000000466ec7 in ?? ()
No symbol table info available.
#16 0x0000000000468c95 in ngx_master_process_cycle ()
No symbol table info available.
#17 0x0000000000441a91 in main ()
No symbol table info available.

[natnet00]

I've recompiled nginx with the (hopefully) right flags and am waiting for a new dump now.

[natnet00]

gdb still saying: "Reading symbols from /usr/sbin/nginx...(no debugging symbols found)...done."

I used the following option in the configure command before compiling nginx:
--with-cc-opt="-DTCP_FASTOPEN=23 -g -O0"
--with-debug
Shouldn't it be compiled with debugging symbols then?

[natnet00]

I know now why the debug symbols were missing. Because I produced a debian package and those symbols were stripped out when creating the package.

Here is the output:

#0  0x0000000000539d73 in ngx_http_vhost_traffic_status_shm_add_upstream (r=0x294b0f0) at /usr/local/src/nginx/build/nginx-module-vts/src/ngx_http_vhost_traffic_status_module.c:2300
        p = 0x294b0f0 "HTTP"
        type = 0
        rc = 140724488337616
        host = 0x33ab7d0
        key = {len = 140724488337552, data = 0x539bda <ngx_http_vhost_traffic_status_shm_add_server+261> "\311\303UH\211\345H\203\354\060H\211}\330H\213E\330H\213@\030H\213\025\311zA"}
        dst = {len = 139782272131072, data = 0x100000002 <error: Cannot access memory at address 0x100000002>}
        i = 52496952
        u = 0x33aafb0
        state = 0xa
        uscf = 0x4a6c2b <ngx_http_finalize_request+1898>
        uscfp = 0x3353828
        umcf = 0xd
#1  0x0000000000537866 in ngx_http_vhost_traffic_status_handler (r=0x294b0f0) at /usr/local/src/nginx/build/nginx-module-vts/src/ngx_http_vhost_traffic_status_module.c:1178
        rc = 0
        ctx = 0x2cc4350
        vtscf = 0x3261fb8
#2  0x00000000004a916b in ngx_http_log_request (r=0x294b0f0) at src/http/ngx_http_request.c:3576
        i = 1
        n = 2
        log_handler = 0x344d768
        cmcf = 0x2949988
#3  0x00000000004a8fd8 in ngx_http_free_request (r=0x294b0f0, rc=0) at src/http/ngx_http_request.c:3523
        log = 0x2c26728
        pool = 0x495b41 <ngx_http_core_run_phases+144>
        linger = {l_onoff = -115115664, l_linger = 32764}
        cln = 0x0
        ctx = 0x6
        clcf = 0x294b0f0
#4  0x00000000004ebaa9 in ngx_http_v2_close_stream (stream=0x294c008, rc=0) at src/http/v2/ngx_http_v2.c:4028
        pool = 0x2c18600
        ev = 0x294b0f0
        fc = 0x2c26580
        node = 0x29760d0
        h2c = 0x295ed50
#5  0x00000000004a8e17 in ngx_http_close_request (r=0x294b0f0, rc=0) at src/http/ngx_http_request.c:3459
        c = 0x2c26580
#6  0x00000000004a6ea7 in ngx_http_finalize_connection (r=0x294b0f0) at src/http/ngx_http_request.c:2549
        clcf = 0x0
#7  0x00000000004a657a in ngx_http_finalize_request (r=0x294b0f0, rc=-4) at src/http/ngx_http_request.c:2308
        c = 0x2c26580
        pr = 0x0
        clcf = 0x325b8f8
#8  0x00000000004a678b in ngx_http_finalize_request (r=0x294b0f0, rc=301) at src/http/ngx_http_request.c:2366
        c = 0x2c26580
        pr = 0x1
        clcf = 0x1600000000
#9  0x00000000004c45ba in ngx_http_upstream_finalize_request (r=0x294b0f0, u=0x33a98f0, rc=301) at src/http/ngx_http_upstream.c:4211
        flush = 46586264
#10 0x00000000004c036b in ngx_http_upstream_intercept_errors (r=0x294b0f0, u=0x33a98f0) at src/http/ngx_http_upstream.c:2406
        status = 301
        i = 0
        h = 0x294b0f0
        err_page = 0x3260b40
        clcf = 0x325b8f8
#11 0x00000000004bfc30 in ngx_http_upstream_process_header (r=0x294b0f0, u=0x33a98f0) at src/http/ngx_http_upstream.c:2198
        n = 370
        rc = 0
        c = 0x2b3f740
#12 0x00000000004bd8aa in ngx_http_upstream_handler (ev=0x2c760d0) at src/http/ngx_http_upstream.c:1126
        c = 0x2c26580
        r = 0x294b0f0
        u = 0x33a98f0
#13 0x0000000000484d1e in ngx_epoll_process_events (cycle=0x29470e0, timer=902, flags=1) at src/event/modules/ngx_epoll_module.c:907
        events = 1
        revents = 8197
        instance = 0
        i = 0
        level = 9980840
        err = 0
        rev = 0x2c760d0
        wev = 0x2c98170
        queue = 0x9625c0 <ngx_event_timer_sentinel>
        c = 0x2b3f740
#14 0x0000000000474015 in ngx_process_events_and_timers (cycle=0x29470e0) at src/event/ngx_event.c:242
        flags = 1
        timer = 902
        delta = 1471674792073
#15 0x0000000000482151 in ngx_worker_process_cycle (cycle=0x29470e0, data=0x0) at src/os/unix/ngx_process_cycle.c:753
        worker = 0
#16 0x000000000047e96c in ngx_spawn_process (cycle=0x29470e0, proc=0x48205c <ngx_worker_process_cycle>, data=0x0, name=0x65fb5b "worker process", respawn=-4) at src/os/unix/ngx_process.c:198
        on = 1
        pid = 0
        s = 9
#17 0x0000000000480f1d in ngx_start_worker_processes (cycle=0x29470e0, n=8, type=-4) at src/os/unix/ngx_process_cycle.c:358
        i = 0
        ch = {command = 1, pid = 0, slot = 0, fd = 0}
#18 0x0000000000480a73 in ngx_master_process_cycle (cycle=0x29470e0) at src/os/unix/ngx_process_cycle.c:243
        title = 0x30db7a0 "master process /usr/sbin/nginx"
        p = 0x30db7be ""
        size = 31
        i = 1
        n = 47
        sigio = 0
        set = {__val = {0 <repeats 16 times>}}
        itv = {it_interval = {tv_sec = 0, tv_usec = 140724488338896}, it_value = {tv_sec = 67, tv_usec = 9667528}}
        live = 1
        delay = 0
        ls = 0x6
        ccf = 0x2948fe8
#19 0x0000000000440bf2 in main (argc=1, argv=0x7ffcf92381e8) at src/core/nginx.c:367
        b = 0x0
        log = 0x95fe40 <ngx_log>
        i = 0
        cycle = 0x29430d0
        init_cycle = {conf_ctx = 0x0, pool = 0x2942170, log = 0x95fe40 <ngx_log>, new_log = {log_level = 0, file = 0x0, connection = 0, disk_full_time = 0, handler = 0x0, data = 0x0, writer = 0x0, wdata = 0x0, action = 0x0, next = 0x0}, log_use_stderr = 0, files = 0x0, free_connections = 0x0, free_connection_n = 0, modules = 0x0, modules_n = 0, modules_used = 0, reusable_connections_queue = {prev = 0x0, next = 0x0}, listening = {elts = 0x2942710, nelts = 6, size = 240, nalloc = 10, pool = 0x2942170}, paths = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, config_dump = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, open_files = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, shared_memory = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, connection_n = 0, files_n = 0, connections = 0x0, read_events = 0x0, write_events = 0x0, old_cycle = 0x0, conf_file = {len = 21, data = 0x65b176 "/etc/nginx/nginx.conf"}, conf_param = {len = 0, data = 0x0}, conf_prefix = {len = 11, data = 0x65b176 "/etc/nginx/nginx.conf"}, prefix = {len = 17, data = 0x65b164 "/usr/local/nginx/"}, lock_file = {len = 0, data = 0x0}, hostname = {len = 0, data = 0x0}}
        cd = 0x659ff0 <__libc_csu_init>
        ccf = 0x2944848

[natnet00]

So this seems to be the problem:
if (u->resolved == NULL) {
uscf = u->conf->upstream;
}

[vozlt]

Thanks for your cooperation and efforts.

1. What version of the nginx-module-vts do you use? latest commit?
2. Can I get the core dump?

[natnet00]

1. Yes, the latest commit - got it yesterday via "git clone https://github.com/vozlt/nginx-module-vts.git"
2. Tried to attach it here, it's 80 MB (too big to attach). I'll provide you a link to the core dump:
   [removed]

[vozlt]

Thanks, I got a your core dump.
It seems that I need to have your os information and nginx debug binary to run the core dump.

1. What system environment do you running?(OS Version, GLIBC Version)
2. Can I get the your nginx debug binary?

[natnet00]

It's a self compiled binary: [deleted]

I'm running Debian jessie.
GLIBC: Debian GLIBC 2.19-18+deb8u4
cat /etc/debian_version: 8.5

[vozlt]

Thanks, I'm not entirely sure, but I have fixed the suspected line in code.
Please test...
Latest commit: f56b5b6

[natnet00]

Thank you, I'm testing it now. Let's see if there are any suspicious exit codes after 24 hours.

[natnet00]

Ok, up to now I haven't seen any "exited on signal 11" log messages since using the latest build.
It's been 21 hours since install time.

I'd say: yes, the bug is fixed ;)

Thank you for the fast fix.

[vozlt]

Thanks to your help.

FYI. debug history.

$ cat /etc/debian_version
8.5

$ ldd --version
ldd (Debian GLIBC 2.19-18+deb8u4) 2.19

$ gdb nginx nginx-coredump-2016-08-20

.
.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000539d73 in ngx_http_vhost_traffic_status_shm_add_upstream (r=0x294b0f0) at /usr/local/src/nginx/build/nginx-module-vts/src/ngx_http_vhost_traffic_status_module.c:2300

warning: Source file is more recent than executable.
2300            uscf = u->conf->upstream;

(gdb) #++++++++++++++++++++++++ backtrace
(gdb) bt
#0  0x0000000000539d73 in ngx_http_vhost_traffic_status_shm_add_upstream (r=0x294b0f0) at /usr/local/src/nginx/build/nginx-module-vts/src/ngx_http_vhost_traffic_status_module.c:2300
#1  0x0000000000537866 in ngx_http_vhost_traffic_status_handler (r=0x294b0f0) at /usr/local/src/nginx/build/nginx-module-vts/src/ngx_http_vhost_traffic_status_module.c:1178
#2  0x00000000004a916b in ngx_http_log_request (r=0x294b0f0) at src/http/ngx_http_request.c:3576
#3  0x00000000004a8fd8 in ngx_http_free_request (r=0x294b0f0, rc=0) at src/http/ngx_http_request.c:3523
#4  0x00000000004ebaa9 in ngx_http_v2_close_stream (stream=0x294c008, rc=0) at src/http/v2/ngx_http_v2.c:4028
#5  0x00000000004a8e17 in ngx_http_close_request (r=0x294b0f0, rc=0) at src/http/ngx_http_request.c:3459
#6  0x00000000004a6ea7 in ngx_http_finalize_connection (r=0x294b0f0) at src/http/ngx_http_request.c:2549
#7  0x00000000004a657a in ngx_http_finalize_request (r=0x294b0f0, rc=-4) at src/http/ngx_http_request.c:2308
#8  0x00000000004a678b in ngx_http_finalize_request (r=0x294b0f0, rc=301) at src/http/ngx_http_request.c:2366
#9  0x00000000004c45ba in ngx_http_upstream_finalize_request (r=0x294b0f0, u=0x33a98f0, rc=301) at src/http/ngx_http_upstream.c:4211
#10 0x00000000004c036b in ngx_http_upstream_intercept_errors (r=0x294b0f0, u=0x33a98f0) at src/http/ngx_http_upstream.c:2406
#11 0x00000000004bfc30 in ngx_http_upstream_process_header (r=0x294b0f0, u=0x33a98f0) at src/http/ngx_http_upstream.c:2198
#12 0x00000000004bd8aa in ngx_http_upstream_handler (ev=0x2c760d0) at src/http/ngx_http_upstream.c:1126
#13 0x0000000000484d1e in ngx_epoll_process_events (cycle=0x29470e0, timer=902, flags=1) at src/event/modules/ngx_epoll_module.c:907
#14 0x0000000000474015 in ngx_process_events_and_timers (cycle=0x29470e0) at src/event/ngx_event.c:242
#15 0x0000000000482151 in ngx_worker_process_cycle (cycle=0x29470e0, data=0x0) at src/os/unix/ngx_process_cycle.c:753
#16 0x000000000047e96c in ngx_spawn_process (cycle=0x29470e0, proc=0x48205c <ngx_worker_process_cycle>, data=0x0, name=0x65fb5b "worker process", respawn=-4) at src/os/unix/ngx_process.c:198
#17 0x0000000000480f1d in ngx_start_worker_processes (cycle=0x29470e0, n=8, type=-4) at src/os/unix/ngx_process_cycle.c:358
#18 0x0000000000480a73 in ngx_master_process_cycle (cycle=0x29470e0) at src/os/unix/ngx_process_cycle.c:243
#19 0x0000000000440bf2 in main (argc=1, argv=0x7ffcf92381e8) at src/core/nginx.c:367

(gdb) #++++++++++++++++++++++++ frame 0
(gdb) f 0
#0  0x0000000000539d73 in ngx_http_vhost_traffic_status_shm_add_upstream (r=0x294b0f0) at /usr/local/src/nginx/build/nginx-module-vts/src/ngx_http_vhost_traffic_status_module.c:2300
2300            uscf = u->conf->upstream;

(gdb) l
2295        }
2296
2297        u = r->upstream;
2298
2299        if (u->resolved == NULL) {
2300            uscf = u->conf->upstream;
2301
2302        } else {
2303            host = &u->resolved->host;
2304

(gdb) p/x u->conf->upstream
Cannot access memory at address 0x0

(gdb) p/x u->conf
$1 = 0x0

(gdb) p/x r->upstream_states
$3 = 0x33aa618

(gdb) p/x r->upstream_states->nelts
$4 = 0x1

(gdb) p/x r->upstream_states->elts
$5 = 0x33aa640

(gdb) p/x r->upstream->state
$6 = 0x0

(gdb) p/x ((ngx_http_upstream_state_t *)r->upstream_states->elts)[0].peer
$7 = 0x33aa710

(gdb) p/d ((ngx_http_upstream_state_t *)r->upstream_states->elts)[0].peer->len
$8 = 15

(gdb) x/15c ((ngx_http_upstream_state_t *)r->upstream_states->elts)[0].peer->data
0x33aa850:      50 '2'  46 '.'  49 '1'  54 '6'  46 '.'  49 '1'  48 '0'  54 '6'
0x33aa858:      46 '.'  49 '1'  55 '7'  54 '6'  58 ':'  56 '8'  48 '0'

(gdb) p/d ((ngx_http_upstream_state_t *)r->upstream_states->elts)[0].status
$10 = 301

(gdb) p/x r->upstream->state
$11 = 0x0

(gdb) #++++++++++++++++++++++++ frame 10
(gdb) f 10
#10 0x00000000004c036b in ngx_http_upstream_intercept_errors (r=0x294b0f0, u=0x33a98f0) at src/http/ngx_http_upstream.c:2406
2406    ngx_http_upstream_test_connect(ngx_connection_t *c)

(gdb) reverse-search ngx_http_upstream_intercept_errors
2333    ngx_http_upstream_intercept_errors(ngx_http_request_t *r,

(gdb) l 2333,2399
2333    ngx_http_upstream_intercept_errors(ngx_http_request_t *r,
2334        ngx_http_upstream_t *u)
2335    {
2336        ngx_int_t                  status;
2337        ngx_uint_t                 i;
2338        ngx_table_elt_t           *h;
2339        ngx_http_err_page_t       *err_page;
2340        ngx_http_core_loc_conf_t  *clcf;
2341
2342        status = u->headers_in.status_n;
.
.
2360        for (i = 0; i < clcf->error_pages->nelts; i++) {
2361
2362            if (err_page[i].status == status) {
.
.
2397                return NGX_OK;
2398            }
2399        }

(gdb) p/d status
$1 = 301

(gdb) p/x u->conf->intercept_errors
$2 = 0x1

(gdb) set $i=0
(gdb) while ($i < clcf->error_pages->nelts)
 >p/d err_page[$i++].status
 >end
$3 = 301
$4 = 302
$5 = 307

(gdb) #++++++++++++++++++++++++ frame 12
(gdb) f 12
#12 0x00000000004bd8aa in ngx_http_upstream_handler (ev=0x2c760d0) at src/http/ngx_http_upstream.c:1126
1126

(gdb) reverse-search ngx_http_upstream_handler
1099    ngx_http_upstream_handler(ngx_event_t *ev)

(gdb) l 1099,1126
1099    ngx_http_upstream_handler(ngx_event_t *ev)
1100    {
1101        ngx_connection_t     *c;
1102        ngx_http_request_t   *r;
1103        ngx_http_upstream_t  *u;
1104
1105        c = ev->data;
1106        r = c->data;
1107
1108        u = r->upstream;
1109        c = r->connection;
1110
1111        ngx_http_set_log_request(c->log, r);
1112
1113        ngx_log_debug2(NGX_LOG_DEBUG_HTTP, c->log, 0,
1114                       "http upstream request: \"%V?%V\"", &r->uri, &r->args);
1115
1116        if (ev->write) {
1117            u->write_event_handler(r, u);
1118
1119        } else {
1120            u->read_event_handler(r, u);
1121        }
1122
1123        ngx_http_run_posted_requests(c);
1124    }
1125
1126

(gdb) p/x u
$16 = 0x33a98f0

(gdb) p/x r->upstream
$17 = 0x33aafb0

(gdb) p/x u->resolved
$18 = 0x33a9cf8

(gdb) p/x r->upstream->resolved
$19 = 0x0

(gdb) p/d u->resolved->host->len
$20 = 14

(gdb) x/14c u->resolved->host->data
0x294d09f:      98 'b'  105 'i' 108 'l' 100 'd' 101 'e' 114 'r' 46 '.'  98 'b'
0x294d0a7:      105 'i' 108 'l' 100 'd' 46 '.'  100 'd' 101 'e'

(gdb) p/x ((ngx_connection_t *)ev->data)->data
$21 = 0x2b40308

(gdb) p/x r
$22 = 0x294b0f0

(gdb) l 1335,1347
1335    static void
1336    ngx_http_upstream_connect(ngx_http_request_t *r, ngx_http_upstream_t *u)
1337    {
1338        ngx_int_t          rc;
1339        ngx_connection_t  *c;
1340
1341        r->connection->log->action = "connecting to upstream";
1342
1343        if (u->state && u->state->response_time) {
1344            u->state->response_time = ngx_current_msec - u->state->response_time;
1345        }
1346
1347        u->state = ngx_array_push(r->upstream_states);

(gdb) p/x r->upstream_states->elts
$23 = 0x33aa640

(gdb) p/x r->upstream->state
$24 = 0x0

(gdb) p/x u->state
$25 = 0x33aa640

(gdb) # I assume that r->upstream_states->elts and r->upstream->state is the same if upstream is running normally.