intermittent connection refused about lua-resty-redis

[agentzh]

Did the redis server get restarted at that time?

The most common reason for this socket error (returned from the operating system) is that the remote server is (temporarily) not listening on the unix domain socket.

Regards,
-agentzh

[rkearsley]

it wasn't restarted to my knowledge
I will increase logging level and check further

thanks

[agentzh]

BTW, you may want to increase the backlog setting for the listening socket in the redis server. The redis server hardcodes the backlog value 511 in its source:

$ ack 511 src/
src/anet.c
265:    if (listen(s, 511) == -1) { /* the magic 511 constant is from nginx */

The accept() queue on your redis server side can easily overflow when your redis is really busy at some moments. Ironically, nginx does provide ways to adjust the backlog setting for its listening socket (that is, the backlog parameter for the listen configure directive).

[rkearsley]

excellent I will definitely try that, I really bet that's the problem!
There may be an influx of connections when the redis replies which I cache in ngx.shared.dict expire, which could easily all be at the same time!
Never the less, I have managed to tidy my lua code a lot with regard to redis while trying to find the problem which is always good :)

[agentzh]

BTW, have you enabled the connection pool for redis? See

https://github.com/agentzh/lua-resty-redis#set_keepalive

It could help by saving lots of connect() and close() syscalls.

Also, I'm pondering adding a built-in wait queue to the cosockets to prevent such peaks of new connections when the frontend cache entries are expired.

Another thing you could probably do here is to use a shdict entry to limit concurrent fetches when some cache entry is expired.

[rkearsley]

yeah I'm using set keepalive.. I could increase the keepalive idle timeout to be greater than shdict expire time also I suppose (currently it's less than) which would probably help
I forked redis and set the backlog to -1 which nginx also does apparently, since I'm using FreeBSD, can do this:

The sysctl(3) MIB variable kern.ipc.somaxconn specifies a hard limit on
 backlog; if a value greater than kern.ipc.somaxconn or less than zero is
 specified, backlog is silently forced to kern.ipc.somaxconn.

Will see how it goes like this 👍

[rkearsley]

Hi
This issue was solved. I was using "KEYS" irresponsibly causing the redis worker to be busy for longer than a few seconds and then waiting new connections would build up and overflow the backlog

Many thanks as always!