nginx -s reload FAILED about nginx-module-vts

[vozlt]

Does it appear if you are running except for nginx-module-vts?

[tkgeng]

It does not.

Without vhost_traffic_status_zone; configuration in http{} block all is well.

[vozlt]

When you are running "nginx -s reload" it go through process as follows:
First, It reads pid from pid file which specified nginx.conf such as /var/run/nginx.pid
Finally, It calls kill(pid, 1) function.

I guess that the message seems to be failed to call kill(pid, 1) function because of what pid does not exist.
So you need to confirm pid number from pid file before "nginx -s reload" whether pid is exist or not.

$ vi src/core/ngx_cycle.c

 998 ngx_int_t
 999 ngx_signal_process(ngx_cycle_t *cycle, char *sig)
1000 {
.
.
1025     n = ngx_read_file(&file, buf, NGX_INT64_LEN + 2, 0);
.
.
1047     return ngx_os_signal_process(cycle, sig, pid);
1048 
1049 }

$ vi src/os/unix/ngx_process.c

613 ngx_int_t
614 ngx_os_signal_process(ngx_cycle_t *cycle, char *name, ngx_int_t pid)
615 {
616     ngx_signal_t  *sig;
617 
618     for (sig = signals; sig->signo != 0; sig++) {
619         if (ngx_strcmp(name, sig->name) == 0) {
620             if (kill(pid, sig->signo) != -1) {
621                 return 0;
622             }
623         
624             ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
625                           "kill(%P, %d) failed", pid, sig->signo);
626         }             
627     }
628 
629     return 1;
630 }

[tkgeng]

The truth likes above, so strange a problem ;(

[tkgeng]

Have you tested it and is there the same problem in your environment?

Or maybe the issue only appear with myself because of someting error in my nginx.conf?

[vozlt]

I don't appear that. It does not seem the problem of nginx.conf.
I looked through the "nginx -s reload" process in process management aspect.
It is as follows:

1. The new worker processes are started by master process.
2. It waits for a ngx_msleep(100).
3. The old worker processes are killed by master process.

I don't know exactly why, but I guess maybe it is one of the as follows:

1. The master process is killed by os kernel because of the out of memory or etc...
2. The master process is killed by itself when running in the above 3 phase.

Do you have enough memory to run nginx processes more than 3 or further?
Please check the /var/log/messages and /var/log/nginx/error.log

$ vi src/os/unix/ngx_process_cycle.c

  72 void
  73 ngx_master_process_cycle(ngx_cycle_t *cycle)
  74 {
  .
  .
 139     for ( ;; ) {
  .
  .
 220         if (ngx_reconfigure) {
 221             ngx_reconfigure = 0;
 222 
 223             if (ngx_new_binary) {
 224                 ngx_start_worker_processes(cycle, ccf->worker_processes,
 225                                            NGX_PROCESS_RESPAWN);
 226                 ngx_start_cache_manager_processes(cycle, 0);
 227                 ngx_noaccepting = 0;
 228 
 229                 continue;
 230             }
 231 
 232             ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "reconfiguring");
 233 
 234             cycle = ngx_init_cycle(cycle);
 235             if (cycle == NULL) {
 236                 cycle = (ngx_cycle_t *) ngx_cycle;
 237                 continue;
 238             }
 239 
 240             ngx_cycle = cycle;
 241             ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx,
 242                                                    ngx_core_module);
 243             ngx_start_worker_processes(cycle, ccf->worker_processes,
 244                                        NGX_PROCESS_JUST_RESPAWN);
 245             ngx_start_cache_manager_processes(cycle, 1);
 246 
 247             /* allow new processes to start */
 248             ngx_msleep(100);
 249 
 250             live = 1;
 251             ngx_signal_worker_processes(cycle,
 252                                         ngx_signal_value(NGX_SHUTDOWN_SIGNAL));
 253         }

[tkgeng]

1. Yes, I have enough memory.
2. Unfortunately, the error comes from a shared library, libc, this is the log in /var/log/messages:
   kernel: nginx[26377]: segfault at 6 ip 00007f9d25be2710 sp 00007fffbfb12928 error 4 in libc-2.12.so[7f9d25b59000+18a000]

[tkgeng]

Looks like it's a memory error, this is the strace output when master process was exiting:

[vozlt]

Maybe, It looks like your system problem, not the problem of nginx-module-vts or nginx if it failed to mmap().
Even so, please check the shmmax size(cat /proc/sys/kernel/shmmax) if you are using linux.

$ vi src/os/unix/ngx_shmem.c

 14 ngx_int_t
 15 ngx_shm_alloc(ngx_shm_t *shm)
 16 {
 17     shm->addr = (u_char *) mmap(NULL, shm->size,
 18                                 PROT_READ|PROT_WRITE,
 19                                 MAP_ANON|MAP_SHARED, -1, 0);
 20 
 21     if (shm->addr == MAP_FAILED) {
 22         ngx_log_error(NGX_LOG_ALERT, shm->log, ngx_errno,
 23                       "mmap(MAP_ANON|MAP_SHARED, %uz) failed", shm->size);
 24         return NGX_ERROR;
 25     }
 26 
 27     return NGX_OK;
 28 }

[tkgeng]

return NGX_ERROR;

There were no related errors in Nginx's error log except for kill(pid, 1) failed (3: No such process) .

We are using Linux, this is the cat /proc/sys/kernel/shmmax output:

68719476736

[vozlt]

I wonder a few things.

1. worker_processes in nginx.conf
2. physical memory size
3. system running environment(physical or virtual)
4. Does it appear if you attempt to several retry "nginx restart"?
5. Does it appear if you attempt to do "nginx -s reload" in other system as same configuration?
6. Please test it if mmap() appears segmentation fault as follows:

$ vi mmap-test.c

#include <stdio.h>
#include <sys/mman.h>

int main(int argc, char **argv) {
    char *addr;
    int size = 1024 * 1024 * 8; /* 8M */
    int i, j = 0;

    for (i = 0; i < size * 64; i++) {
        addr = (char *) mmap(NULL, size,
                PROT_READ|PROT_WRITE,
                MAP_ANON|MAP_SHARED, -1, 0);

        if (addr == MAP_FAILED) {
            j++;
        }
    }

    printf(">>> MAP_FAILED(%d)/%d\n", j, size * 64);

    return 0;
}

$ gcc -o mmap-test mmap-test.c
$ ./mmap-test

[tkgeng]

This is my test result:

[root@szq-ops-test-tk ~]# time ./mmap-test
>>> MAP_FAILED(536805396)/536870912

real    0m54.433s
user    0m11.330s
sys     0m43.074s

And strace this process it output mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)

[tkgeng]

I have test all the things what you are care about.

1. three
2. 8GB
3. virtual also physical
4. it does not appear.
5. in my Debian 8 x64 (virtual), it dose not appear.

[vozlt]

I wonder what it is due to the ngx_shm_alloc() in src/os/unix/ngx_shmem.c.
So, please test each as follows:

1. Test limit_conn_zone to size 1m & vhost_traffic_status off and "nginx -s reload" try to 3 more than times.

   http {
       limit_conn_zone $binary_remote_addr zone=addr:8m;
       vhost_traffic_status off
       .
       .
       .
   }

2. Test vhost_traffic_status_zone to size 64k and "nginx -s reload" try to 3 more than times.

   http {
       vhost_traffic_status_zone shared:vhost_traffic_status:64k
       .
       .
       .
   }

3. Test vhost_traffic_status_zone to size 8m and "nginx -s reload" try to 3 more than times.

   http {
       vhost_traffic_status_zone shared:vhost_traffic_status:8m
       .
       .
       .
   }

And let me know the below.

1. Does it appear([alert] kill (25315, 1) failed (3: No such process)) each(1-3)?
2. When did this problem([alert] kill (25315, 1) failed (3: No such process)) happen? (3 or 4 or 5... times)

[tkgeng]

1. It dose not appear.
2. It dose, the tenth time.
3. It dose, the tenth time.

[vozlt]

Okay, thanks.
I was wondering if the master process killed all worker processes through nginx -s reload.
So, I want to check the list of nginx process before and after nginx -s relaod until the alert([alert] kill (25315, 1) failed (3: No such process)) appears.

Essential list of process:

• pid
• ppid
• cmd

Simply as follows:

$ vi nginx-reload.sh

#! /bin/env bash
echo ">>> S $(date +'%s')"
ps -efl | grep nginx: | grep -v grep
nginx -s reload
echo ">>> E $(date +'%s')"
ps -efl | grep nginx: | grep -v grep

$ bash nginx-reload.sh > result.txt
$ bash nginx-reload.sh >> result.txt
.
.
$ bash nginx-reload.sh >> result.txt
[alert] kill (25315, 1) failed (3: No such process)

And let me know the result.txt.

FYI. The alert message will not write output to result.txt because of what the alert message takes file descriptor 2(/dev/stderr).

[tkgeng]

Hi, this is my test result in configuration 3:

result.txt

[vozlt]

$ vi result.txt

.
.
>>> S 1456280881                                                                                                                                                                                                                     
1 S root     30993     1  0  80   0 - 126825 rt_sig 10:27 ?       00:00:00 nginx: master process ./nginx                                                                                                                             
5 S nobody   31124 30993  1  80   0 - 140140 ep_pol 10:28 ?       00:00:00 nginx: worker process                                                                                                                                     
5 S nobody   31125 30993  1  80   0 - 140140 ep_pol 10:28 ?       00:00:00 nginx: worker process                                                                                                                                     
5 S nobody   31126 30993  2  80   0 - 140140 ep_pol 10:28 ?       00:00:00 nginx: worker process                                                                                                                                     
5 S nobody   31127 30993  0  80   0 - 126825 ep_pol 10:28 ?       00:00:00 nginx: cache manager process                                                                                                                              
5 S nobody   31128 30993  0  80   0 - 126825 ep_pol 10:28 ?       00:00:00 nginx: cache loader process                                                                                                                               
>>> E 1456280881                                                                                                                                                                                                                     
5 S nobody   31124     1  1  80   0 - 140140 ep_pol 10:28 ?       00:00:00 nginx: worker process                                                                                                                                     
5 S nobody   31125     1  1  80   0 - 140140 ep_pol 10:28 ?       00:00:00 nginx: worker process                                                                                                                                     
5 S nobody   31126     1  2  80   0 - 140140 ep_pol 10:28 ?       00:00:00 nginx: worker process                                                                                                                                     
5 S nobody   31127     1  0  80   0 - 126825 ep_pol 10:28 ?       00:00:00 nginx: cache manager process                                                                                                                              
5 S nobody   31128     1  0  80   0 - 126825 ep_pol 10:28 ?       00:00:00 nginx: cache loader process
.
.

It seems to occur before the master process kills worker processes.
In other words, it seems to fail before the master process calls fork().
I guess there is a part in this below somewhere.

$ vi src/os/unix/ngx_process_cycle.c

 220         if (ngx_reconfigure) {
 221             ngx_reconfigure = 0;
 222 
 .
 .
 233 
 234             cycle = ngx_init_cycle(cycle);
 235             if (cycle == NULL) {
 236                 cycle = (ngx_cycle_t *) ngx_cycle;
 237                 continue;
 238             }
 239 
 240             ngx_cycle = cycle;
 241             ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx,
 242                                                    ngx_core_module);
 243             ngx_start_worker_processes(cycle, ccf->worker_processes,
 244                                        NGX_PROCESS_JUST_RESPAWN);
 .
 .
 253         }

Could I get your debug log and backtrace, please?
See the nginx debugging.

[vozlt]

Okay. I checked the debug.txt without backtrace.
It seems to fail during the reading /usr/local/nginx/etc/conf.d/upstream.conf.
Is there the vhost_traffic_status_zone config in /usr/local/nginx/etc/conf.d/upstream.conf?
If not, please check nginx -s reload without /usr/local/nginx/etc/conf.d/upstream.conf.
If you do, Does it appear([alert] kill (25315, 1) failed (3: No such process))?

$ vi debut.txt

5397 2016/02/25 15:17:45 [debug] 18649#0: include /usr/local/nginx/etc/conf.d/upstream.conf
.
.
5446 2016/02/25 15:17:45 [debug] 18649#0: malloc: 00007F7B58011ED0:1024                                                                                                                                          
5447 2016/02/25 15:17:45 [debug] 18649#0: malloc: 00007F7B5EF80010:3559937                                                                                                                                       
5448 2016/02/25 15:17:46 [debug] 18720#0: timer delta: 502                                                                                                                                                       
5449 2016/02/25 15:17:46 [debug] 18720#0: worker cycle                                                                                                                                                           
5450 2016/02/25 15:17:46 [debug] 18720#0: accept mutex lock failed: 0                                                                                                                                            
5451 2016/02/25 15:17:46 [debug] 18720#0: epoll timer: 500                                                                                                                                                       
5452 2016/02/25 15:17:46 [debug] 18719#0: timer delta: 502                                                                                                                                                       
5453 2016/02/25 15:17:46 [debug] 18719#0: worker cycle                                                                                                                                                           
5454 2016/02/25 15:17:46 [debug] 18719#0: accept mutex lock failed: 0                                                                                                                                            
5455 2016/02/25 15:17:46 [debug] 18719#0: epoll timer: 500                                                                                                                                                       
5456 2016/02/25 15:17:46 [debug] 18720#0: timer delta: 501                                                                                                                                                       
5457 2016/02/25 15:17:46 [debug] 18720#0: worker cycle                                                                                                                                                           
5458 2016/02/25 15:17:46 [debug] 18720#0: accept mutex lock failed: 0                                                                                                                                            
5459 2016/02/25 15:17:46 [debug] 18720#0: epoll timer: 500                                                                                                                                                       
5460 2016/02/25 15:17:46 [debug] 18719#0: timer delta: 501
.
.

[tkgeng]

No vhost_traffic_status_zone config in /usr/local/nginx/etc/conf.d/upstream.conf.
I remove the upstream.conf and the error also appears
debug.txt

[vozlt]

Is there the vhost_traffic_status_zone in only /usr/local/nginx/etc/conf.d/status.conf?
Or, Is there the vhost_traffic_status_zone in [1-4].2333.com.conf?
In above debug.txt, It seems to fail during the reading /usr/local/nginx/etc/conf.d/4.2333.com.conf.

$ vi debug.txt

2016/02/25 17:48:35 [debug] 6434#0: include /usr/local/nginx/etc/conf.d/1.2333.com.conf
.
2016/02/25 17:48:35 [debug] 6434#0: include /usr/local/nginx/etc/conf.d/2.2333.com.conf
.
2016/02/25 17:48:35 [debug] 6434#0: include /usr/local/nginx/etc/conf.d/3.2333.com.conf
.
2016/02/25 17:48:35 [debug] 6434#0: include /usr/local/nginx/etc/conf.d/4.2333.com.conf
.
2016/02/25 17:48:36 [debug] 6434#0: malloc: 00007FEF246DC010:4729250
2016/02/25 17:48:36 [debug] 6454#0: timer delta: 501
2016/02/25 17:48:36 [debug] 6454#0: worker cycle
2016/02/25 17:48:36 [debug] 6454#0: accept mutex lock failed: 0
2016/02/25 17:48:36 [debug] 6454#0: epoll timer: 500
2016/02/25 17:48:36 [debug] 6456#0: timer delta: 501
2016/02/25 17:48:36 [debug] 6456#0: worker cycle
2016/02/25 17:48:36 [debug] 6456#0: accept mutex lock failed: 0
2016/02/25 17:48:36 [debug] 6456#0: epoll timer: 500
2016/02/25 17:48:36 [debug] 6454#0: timer delta: 501
2016/02/25 17:48:36 [debug] 6454#0: worker cycle
2016/02/25 17:48:36 [debug] 6454#0: accept mutex lock failed: 0
.
.
.

FYI. The nginx-module-vts has been updated(7166277). It also added debug log for this issue.

[tkgeng]

No, vhost_traffic_status_zone is in only /usr/local/nginx/etc/nginx.conf:

http {
    ...
    vhost_traffic_status_zone shared:vhost_traffic_status:8m;
    ...
    location /status {
            access_log off;
            #vhost_traffic_status_filter_by_host on;
            vhost_traffic_status_filter_by_set_key $server_name;
            vhost_traffic_status_display;
            vhost_traffic_status_display_format html;
    }

    ...
    include conf.d/*.conf;
}

I have recompiled Nginx with your module V7166277, and below is the debug log:

debug.txt

I am pleasure to provide anything information about this error 😄

[vozlt]

Thanks.
I have been update(4d74c4a).
I added some debug log and some exception routine.
Please test and give me the debug log.

[tkgeng]

HI, below is the debug log:
debug.txt

[vozlt]

Please let me just check a few things.

1. Does it appear if you exclude the vhost_traffic_status_filter_by_set_key directive?

   http {
   ...
   vhost_traffic_status_zone shared:vhost_traffic_status:8m;
   ...
   location /status {
           access_log off;
           vhost_traffic_status_display;
           vhost_traffic_status_display_format html;
   }
   ...
   include conf.d/*.conf;
   }

2. Does it appear if you set the directive vhost_traffic_status_filter_check_duplicate off?

   http {
   ...
   vhost_traffic_status_zone shared:vhost_traffic_status:8m;
   ...
   location /status {
           access_log off;
           vhost_traffic_status_filter_check_duplicate off;
           vhost_traffic_status_filter_by_set_key $server_name;
           vhost_traffic_status_display;
           vhost_traffic_status_display_format html;
   }
   ...
   include conf.d/*.conf;
   }

Please also let me know process log that contains memory usage when sending debug.txt.
Simply as follows:

$ vi nginx-reload.sh

#! /bin/env bash
process_cmd="ps -eo user,pid,ppid,pcpu,pmem,tty,vsz,rss,size,bsdtime,stime,cmd --sort=+pid,+ppid,+vsz,+rss,+pcpu"
echo ">>> S $(date +'%s') $( date +'%Y/%m/%d %H:%M:%S')"
$process_cmd  | grep nginx: | grep -v grep
nginx -s reload
echo ">>> E $(date +'%s') $( date +'%Y/%m/%d %H:%M:%S')"
$process_cmd  | grep nginx: | grep -v grep

$ bash nginx-reload.sh > result.txt
$ bash nginx-reload.sh >> result.txt
.
.
$ bash nginx-reload.sh >> result.txt
[alert] kill (25315, 1) failed (3: No such process)

Thanks.

[tkgeng]

Aha, good news:

1. It does not appears.
2. It does not appears, too.

Maybe the vhost_traffic_status_filter_by_set_key is the chief culprit.

[vozlt]

Thanks.
The nginx-module-vts has been updated.(a356814).
Please test and let me know results.

[tkgeng]

Hi,
I have recompiled and tested, the result is that the problem solved 🎉🎉

Thanks very much 👍

[vozlt]

Thanks.

FYI. I got a core dump at the end of debug.

$ gdb /usr/sbin/nginx /tmp/core.13728

(gdb) bt
#0  0x00007fec2687b740 in memcpy () from /lib64/libc.so.6
#1  0x00000000004c51eb in ngx_http_vhost_traffic_status_filter_unique (pool=0x2235d90, keys=0x2250a18) at ../nginx-module-vts/src/ngx_http_vhost_traffic_status_module.c:3223
#2  0x00000000004c554d in ngx_http_vhost_traffic_status_merge_loc_conf (cf=0x7ffff5c0f9d0, parent=0x227e948, child=0x22509f0) at ../nginx-module-vts/src/ngx_http_vhost_traffic_status_module.c:4920
#3  0x000000000044a804 in ngx_http_merge_servers (cf=0x7ffff5c0f9d0, cmd=<value optimized out>, conf=<value optimized out>) at src/http/ngx_http.c:595
#4  ngx_http_block (cf=0x7ffff5c0f9d0, cmd=<value optimized out>, conf=<value optimized out>) at src/http/ngx_http.c:268
#5  0x000000000042bf66 in ngx_conf_handler (cf=0x7ffff5c0f9d0, filename=0x2235fa8) at src/core/ngx_conf_file.c:427
#6  ngx_conf_parse (cf=0x7ffff5c0f9d0, filename=0x2235fa8) at src/core/ngx_conf_file.c:283
#7  0x0000000000429526 in ngx_init_cycle (old_cycle=<value optimized out>) at src/core/ngx_cycle.c:274
#8  0x000000000043e23f in ngx_master_process_cycle (cycle=0x21dfc00) at src/os/unix/ngx_process_cycle.c:234
#9  0x000000000041a551 in main (argc=<value optimized out>, argv=<value optimized out>) at src/core/nginx.c:367

(gdb) f 1
#1  0x00000000004c51eb in ngx_http_vhost_traffic_status_filter_unique (pool=0x2235d90, keys=0x2250a18) at ../nginx-module-vts/src/ngx_http_vhost_traffic_status_module.c:3223
3223            ngx_memcpy(p, filters[i].filter_name.value.data,

(gdb) bt full 2
#0  0x00007fec2687b740 in memcpy () from /lib64/libc.so.6
No symbol table info available.
#1  0x00000000004c51eb in ngx_http_vhost_traffic_status_filter_unique (pool=0x2235d90, keys=0x2250a18) at ../nginx-module-vts/src/ngx_http_vhost_traffic_status_module.c:3223
        hash = <value optimized out>
        p = 0x7fec207e601c ""
        key = {len = 36013564, data = 0x7fec207e6010 "$server_name"}
        i = <value optimized out>
        n = 1
        uniqs = 0x2263ce0
        filter_keys = 0x0
        filter = <value optimized out>
        filters = 0x22512f8
        filter_uniqs = <value optimized out>
        __func__ = "ngx_http_vhost_traffic_status_filter_unique"
(More stack frames follow...)

(gdb) l
3218            }
3219
3220            p = key.data;
3221            p = ngx_cpymem(p, filters[i].filter_key.value.data,
3222                           filters[i].filter_key.value.len);
3223            ngx_memcpy(p, filters[i].filter_name.value.data,
3224                       filters[i].filter_name.value.len);
3225            hash = ngx_crc32_short(key.data, key.len);
3226
3227            filter_uniqs = ngx_array_push(uniqs);

(gdb) p/x p
$1 = 0x7fec207e601c

(gdb) p filters[0].filter_name.value.len
$2 = 36013552

(gdb) x filters[0].filter_name.value.data
0x7fec00010106: Cannot access memory at address 0x7fec00010106

[tkgeng]

It has been working fine for a long time since you fixed the bug.