Retry on 'Redis server went away' about wp-redis

[joshkoenig]

It should be easy to replicate this by lowering the connection timeout:

config set timeout 5

And then having a process that connects, sleeps for 6 seconds, and then tries to interact.

[danielbachhuber]

This is addressed by ee98d09 in #18

[danielbachhuber]

Actually, it might be addressed. If Redis throws a different exception message for timeout, then we'll need to catch that too.

[danielbachhuber]

@joshkoenig I'm having difficulty replicating the timeout exception.

Here's my test script:

salty-wordpress ➜  pantheon.dev  cat test.php
<?php

global $wp_object_cache;
wp_cache_set( 'foo', 'bar' );
var_dump( $wp_object_cache->redis->isConnected() );
$wp_object_cache->cache = array();
var_dump( wp_cache_get( 'foo' ) );
sleep( 8 );
var_dump( $wp_object_cache->redis->isConnected() );
$wp_object_cache->cache = array();
var_dump( wp_cache_get( 'foo' ) );

$wp_object_cache->cache = array(); will reset the internal object cache, so it falls back to hitting Redis for the value.

Here's my test case:

salty-wordpress ➜  pantheon.dev  redis-cli
127.0.0.1:6379> config set timeout 1
OK
127.0.0.1:6379> exit
salty-wordpress ➜  pantheon.dev  wp eval-file test.php
bool(true)
string(3) "bar"
bool(true)
string(3) "bar"

I'd expect isConnected() to return false in the second call.

We're also setting timeout=1 in __construct(): https://github.com/pantheon-systems/wp-redis/blob/master/object-cache.php#L724

Is there something simple I'm doing wrong?

[danielbachhuber]

Oh. From the docs:

Timeouts are not to be considered very precise: Redis avoids to set timer events or to run O(N) algorithms in order to check idle clients, so the check is performed incrementally from time to time. This means that it is possible that while the timeout is set to 10 seconds, the client connection will be closed, for instance, after 12 seconds if many clients are connected at the same time.

[danielbachhuber]

I extended the script timeout to 20 seconds, and then killed the client connection with CLIENT KILL:

salty-wordpress ➜  pantheon.dev  redis-cli
127.0.0.1:6379> CLIENT LIST
addr=127.0.0.1:35639 fd=7 name= age=186 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=client
addr=127.0.0.1:35647 fd=5 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
127.0.0.1:6379> CLIENT KILL 127.0.0.1:35647
OK
127.0.0.1:6379> CLIENT LIST
addr=127.0.0.1:35639 fd=7 name= age=195 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=client
127.0.0.1:6379>

No dice. I think I'm going to punt on writing tests for this. If we can set up a way to reproduce on Pantheon, we can verify the Exception message to make sure we catch it and refresh the connection.

[joshkoenig]

The isConnected() method just tells if PHP "thinks" it's connected. I believe that just checks if a connection was established. The error case comes up when PHP believes this to be true, but turns out it's not true because the connection was closed from the server side.

I'll take a look at reproducing this again. The config set timeout worked for me before, but I didn't turn it into a script.

[danielbachhuber]

I believe that just checks if a connection was established. The error case comes up when PHP believes this to be true, but turns out it's not true because the connection was closed from the server side.

Huh. That's pretty lame if true.

I suppose we could try a get() or exists() on an almost impossible value, and see if an exception is thrown.

Though, on the last line, wp_cache_get() should've thrown an Exception if the connection was invalid. Maybe PhpRedis gracefully restarts the connection now if the connection details still work?

[joshkoenig]

I don't think it gracefully restarts; the failure I was able to do before was to just so a get() after waiting a while for things to timeout.

If you look at where the exception is thrown, it's when the socket communication isn't working no more:

https://github.com/phpredis/phpredis/blob/05e38d590143d044f18c63c5a581c645abd9d4b1/redis.c#L446

[danielbachhuber]

@joshkoenig how do you want to wrap this up?

[joshkoenig]

I guess we'll close it for now given that we've added the handling for the error. Would feel better if we had a solid integration test. I'll make a new issue for this.