RedisConnectionException at each first Connect(), while redis-py and hiredis work fine about stackexchange.redis

[ship2moon]

I dont know writing it here will solve our poblem. Please add a "+1" to the same error:
"It was not possible to connect to the redis server(s); to create a disconnected multiplexer, disable AbortOnConnectFail. UnableToResolvePhysicalConnection on PING at StackExchange.Redis.ConnectionMultiplexer"

[smiket]

I am seeing this issue too. In my case, I have a unit test that spawns multiple threads. Each thread gets a multiplexer (the first thread in will initialize it). That first thread enters ConnectionMultiplexer.ConnectImpl(). In that method, there is this line of code:

var task = Factory.StartNew(() => { return muxer.ReconfigureAsync(true, false, log, null, "connect").Result; });

In normal scenarios, this always works. However, in my multithreaded test, the Task.Factory never ends up calling the inline lambda in the above code. So the task times out, and surfaces as the exception noted in the beginning of this issue. I'm looking at it now, but thought I'd just post to maybe shed some additional info.

Adding onto the above ...
Calling connect the first way (below) produces the error, but calling the second way does not:

return ConnectionMultiplexer.Connect(options)
return Task.Run(async () => await ConnectionMultiplexer.ConnectAsync(options, sw)).Result;

[smiket]

Marc, I have a unit test for you that can reproduce the issue.
Attaching screenshot of error, and unit test below.
In the unit test, if you dial back the number of threads below 30 (on my host) there are no issues.
If you turn it up over 50, the errors begin.

Looks like line 771 in ConnectionMultiplexer (task.wait) is timing out.
Its not that the connection timeout occurs - rather, the task never starts (deadlock / depletion?).
The error materializes as a connection error, but its really the task not starting.
The breakpoint in the lamda on line 767 never gets hit / task never starts.

** Edited the test to force the issue more reliably -
** I know there should be only one multiplexer (unit test news one up each time).
** But I just wanted to force the condition to illustrate thread pool depletion, materializing the error.

Hope this helps, and thanks,
Steve

    [Test]
    public void CanSetAndGetItemsAcrossThreads()
    {
        var opts = ConfigurationOptions.Parse("localhost:6379");
        opts.ConnectTimeout = 10000;
        opts.SyncTimeout = 10000;

        List<Task> tasks = new List<Task>();
        for (int i = 0; i < 100; i++)
        {
            var task = Task.Run(() =>
            {
                using (var conn = ConnectionMultiplexer.Connect(opts, Console.Out))
                {
                    var str = Guid.NewGuid().ToString();
                    var redis = conn.GetDatabase(0);
                    redis.StringSet(str, str);
                    redis.StringSet(str, str);
                }
            });
            tasks.Add(task);
        }

        Task.WaitAll(tasks.ToArray());
        Console.WriteLine("Test Completed");
    }

[hesi726]

Please see http://stackoverflow.com/questions/23871110/stackexchange-redis-failing-to-connect-with-mono-in-mac-os-x ,

And I use the follows package,
https://www.nuget.org/packages/StackExchange.Redis.Mono/

it solved my question.

[kmacmcfarlane]

In my case, the call to Connect(...) fails if it is inside an async delegate. I found this by trying to connect from within Nito's AsyncContext.Run(async () => {}) method. Hope that helps someone else :-)

[venkatperi]

+1

Stackexchange.Redis: 1.0.316.0 (via nuget)
Mono 3.12.0 ((detached/a813491)
Connection String: "192.168.0.40:10001"

StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s); to create a disconnected multiplexer, disable AbortOnConnectFail. UnableToResolvePhysicalConnection on PING
at StackExchange.Redis.ConnectionMultiplexer.ConnectImpl (System.Func`1<StackExchange.Redis.ConnectionMultiplexer>,System.IO.TextWriter) <IL 0x00096, 0x0031b>
at StackExchange.Redis.ConnectionMultiplexer.Connect (string,System.IO.TextWriter) <IL 0x0001a, 0x000e7>

Output of telnet:

telnet 192.168.0.40 10001
Trying 192.168.0.40...
Connected to 192.168.0.40.
Escape character is '^]'.

PING
+PONG

[berkutx]

+1 I have a same error, venkatperi (only under latest mono in linux, under windows w/o mono - all is ok), log:
faulted: UnableToResolvePhysicalConnection on PING

[patroza]

If you are using new Lazy<ConnectionMultiplexer>(initCode) manually or through a third party library (e.g SimpleInjector DI container) then the exception on creation is cached, and therefore you need to restart the web app to fix the issue again.

I'm currently experimenting by calling the following in my Cache manipulation methods (instead of using DI):

        void EnsureConnection() {
            if (_cacheClient != null)
                return;

            lock (_lock) {
                if (_cacheClient != null)
                    return;
                var connectionMultiplexer = ConnectionMultiplexer.Connect(Environments.RedisCacheConnectionString);
                Thread.MemoryBarrier();
                _cacheClient = connectionMultiplexer;
            }
        }

(Unsure at this time if the Multiplexer will automatically restore a connection if it was lost in the meantime, otherwise one could extend it with IsConnected check I suppose. But since I see a ConnectionRestored event i'm guessing it will handle it automatically).

Another option could be to launch the Connect initialization on an ASP initialize-once-per-recycle basis in a static field. Which should try and restart the app on failure.

[smiket]

I changed my test to not launch threads via Task.Run ...
Instead, I am now creating threads manually, and starting them.
The issue goes away - I was depleting the thread pool (as noted earlier in this thread).

[saixiaohui]

I encountered the same issue. I downloaded the Redis code and debugged from my side, I found the code that is causing issue for me.

When the code tries to connect, it starts a new task and waits for either complete or timeout.
var task = Task.Factory.StartNew(() => muxer.ReconfigureAsync(true, false, log, null, "connect").Result);

In order to limit the number of parallel tasks my code can use, I created a TaskFactory and set the TaskScheduler as a limited concurrency level task scheduler. Somehow this task used the same TaskFactory (haven't figured out why) and all the threads were busy on something else, so the task was never scheduled and executed during waiting.

I am guessing most of the issues happened here is because the task was not executed or hasn't completed.

@mgravell , hope this helps.

[rjmooney]

So I ran into this issue as well. After reading through some of the comments and taking a look at the code, I noticed an unnecessary indirection in ConnectionMultiPlexer.ConnectImpl which appears to be causing a deadlock:

// note that task has timeouts internally, so it might take *just over* the regular timeout
// wrap into task to force async execution
var task = Factory.StartNew(() => muxer.ReconfigureAsync(true, false, log, null, "connect").Result);

What's happening here is the code is spawning an async task which waits synchronously on an async task result. This is bad juju. What it appears is happening next is the code blocks the context thread waiting for the wrapper task created with Factory.StartNew to return:

if (!task.Wait(muxer.SyncConnectTimeout(true)))

When the continuation from ReconfigureAsync returns, it waits for the context thread to become available. Since the context thread is blocked waiting for the continuation, it results in deadlock. Stephen Cleary has an excellent description of the pattern on his blog.

The fix, I believe, is simple: spawn the ReconfigureAsync method directly:

// note that task has timeouts internally, so it might take *just over* the regular timeout
var task = muxer.ReconfigureAsync(true, false, log, null, "connect");

if (!task.Wait(muxer.SyncConnectTimeout(true)))

If I had to guess, I'd say ReconfigureAsync was once a synchronous method that was converted to async without updating the ConnectImpl implementation to remove the async wrapping.

The above is conjecture at this point, however in limited testing it appears to solve the problem.

Submitted as pull request #353.

[mgravell]

If you pass in a writer for the "log", what does it say?
On 10 May 2014 00:27, "tw-bert" [email protected] wrote:

Hi Marc,

Version 1.0.289.0 (nuget, .net 4.0).

When I leave my workstation alone for five minutes, after calling
Connect() I get a RedisConnectionException the first time. Immediately
retrying does work with no delay at a 90% success rate. No info is written
to the supplied TextWriter.

var oCli = ConnectionMultiplexer.Connect("srv-flux-01:14144", log);

Complete exception message:

An unhandled exception of type 'StackExchange.Redis.RedisConnectionException' occurred in StackExchange.Redis.dll
Additional information: It was not possible to connect to the redis server(s); to create a disconnected multiplexer, disable AbortOnConnectFail

I connect from Win7 Pro X64 to Redis 2.8.9 (x64) on a Debian machine on
our LAN (same subnet). No hardware firewalls in between, only Windows
firewall.
Redis connections from redis-py or RedisDesktopManager(which uses hiredis)
never fail, and always connect within milliseconds.

I use a simple Windows Forms project, VS2013 Express, for testing.
I use (and need) the .NET Framework 4 Client Profile.

Any ideas?

Kind regards, TW

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/42
.

[tw-bert]

The log stays empty.

[tw-bert]

I did some further testing.

When I first create a BookSleeve.RedisConnection, and immediately after that a StackExchange.Redis.ConnectionMultiplexer , the booksleeve connection is valid (without pause), and the stackexchange.redis connection throws an exception.

I also had a situation where I could not connect until I restarted the exe (or started debugging again).

I also tried a "resolvedns=1" option I found in your sourcecode, but this doesn't seem to have any effect (on resolving this issue).

When the exception is not thrown, the log looks like this:

srv-flux-01:14144

1 unique nodes specified
Requesting tie-break from srv-flux-01:14144 > __Booksleeve_TieBreak...
Allowing endpoints 00:00:01 to respond...
srv-flux-01:14144 returned with success
srv-flux-01:14144 had no tiebreaker set
Single master detected: srv-flux-01:14144
srv-flux-01:14144: Standalone v2.8.9, master; 1 databases; keep-alive: 00:01:00; int: ConnectedEstablished; sub: ConnectedEstablished, 1 active
srv-flux-01:14144: int ops=11, qu=0, qs=1, qc=0, wr=0, sync=9, socks=1; sub ops=3, qu=0, qs=0, qc=0, wr=0, subs=1, sync=3, socks=1
Circular op-count snapshot; int: 0+11=11 (1,10 ops/s; spans 10s); sub: 0+3=3 (0,30 ops/s; spans 10s)
Sync timeouts: 0; fire and forget: 0; last heartbeat: -1s ago
Starting heartbeat...

Kind regards, TW

[mgravell]

The log is never empty. It should always say something. Try passing in a
StringWriter, then write the ToString afterwards.
On 10 May 2014 10:52, "tw-bert" [email protected] wrote:

The log stays empty.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/42#issuecomment-42737620
.

[tw-bert]

Hi Marc, I used a StreamWriter (which I flushed in the finally). I'll try a StringWriter in a few days (can't access my work environment atm) and get back here.

[tw-bert]

Yes, a StringWriter does give back a result:

---------------------------

---------------------------
StackExchange.Redis.RedisConnectionException: It was not possible to connect to
the redis server(s); to create a disconnected multiplexer, disable AbortOnConnectFail

   at StackExchange.Redis.ConnectionMultiplexer.Connect(String configuration,
TextWriter log) in
c:\TeamCity\buildAgent\work\58bc9a6df18a3782\StackExchange.Redis\StackExchange\Redis\
ConnectionMultiplexer.cs:line 738

   at stackexchange_redis_test_three.Form1.button1_Click(Object sender, EventArgs
e) in
e:\work\eclipse\11.3A\amdir_gui\amdir\amber\msroot\msflux\logic_cs_dotnet\stackexchan
ge_redis_test_three\Form1.cs:line 29

srv-flux-01:14144



1 unique nodes specified

Requesting tie-break from srv-flux-01:14144 > __Booksleeve_TieBreak...

Allowing endpoints 00:00:01 to respond...

srv-flux-01:14144 did not respond

srv-flux-01:14144 failed to nominate (WaitingForActivation)

No masters detected

srv-flux-01:14144: Standalone v2.0.0, master; keep-alive: 00:01:00; int:
Connecting; sub: ConnectedEstablished, 1 active; not in use: DidNotRespond

srv-flux-01:14144: int ops=0, qu=4, qs=0, qc=0, wr=0, socks=1; sub ops=3, qu=0,
qs=0, qc=0, wr=0, subs=1, sync=3, socks=1

Circular op-count snapshot; int: 0 (0,00 ops/s; spans 10s); sub: 0+3=3 (0,30
ops/s; spans 10s)

Sync timeouts: 0; fire and forget: 0; last heartbeat: -1s ago

Starting heartbeat...


---------------------------
OK   
---------------------------

When it goes wrong, it keeps going wrong within that process (some static condition it seems).
Same as before: other redis clients have no problems.
When I start a second .NET StackExchange.Redis instance next to the first, the second succeeds, while the first keeps on failing.

I can now reproduce this behaviour by issuing:

ipconfig /release & ipconfig /renew

Sidenote: when I did only the release, I lost my RDP connection, duh. Makes perfect sense lol. By combining the two statements, RDP will be disconnected as well, but at least you can connect again after a few seconds.

If StackExchange.Redis' connect is the first thing I do after renew, it fails with the RedisConnectionException above.

Any idea?

[tw-bert]

Hi Marc, can you reproduce this?

I'm thinking about returning to BookSleeve, which is always an option ofcourse (and we've had very good experiences with BookSleeve). I can't get StackExchange.Redis to work reliably.

Kind regards, TW

[mgravell]

I am still investigating

On 14 May 2014 09:22, tw-bert [email protected] wrote:

Hi Marc, can you reproduce this?

I'm thinking about returning to BookSleeve, which is always an option
ofcourse (and we've had very good experiences with BookSleeve). I can't get
StackExchange.Redis to work reliably.

Kind regards, TW

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/42#issuecomment-43054423
.

Regards,

Marc

[tw-bert]

Great, thanks. And take your time. Give me a nudge if I can help.

[SiqiLu]

I also had a similar issue. Add I used a RetryPolicy to avoid it.

[blemasle]

@SiqiLu How did you configure your retry policy ? I've noticed that with a IIS hosted application, the error keeps coming once its there no matter how hard I try. The only thing to do is restarting the IIS application pool. So I'm surprised that you managed to solve this with just a retry policy.

[tw-bert]

@SiqiLu I second blemasle's question. A workaround like 'restarting the .NET process' does not work for me (because it's somewhere inside a running .NET Gui client).

A self-quote from earlier in this thread:

When it goes wrong, it keeps going wrong within that process (some static condition it seems).
Same as before: other redis clients have no problems.
When I start a second .NET StackExchange.Redis instance next to the first, the second succeeds, while the first keeps on failing.

I can reproduce the exception by issuing:

ipconfig /release & ipconfig /renew

(before the first command of SE.Redis)

Can you (blemasle and SiqiLu) reproduce this?

[mgravell]

I'm still struggling to reliably repro in a way that allows code fix;
haven't given up, but it is just vexing that I'm fighting Socket here..

On 28 May 2014 09:43, tw-bert [email protected] wrote:

@SiqiLu https://github.com/SiqiLu I second blemasle's question. A
workaround like 'restarting the .NET process' does not work for me (because
it's somewhere inside a running .NET Gui client).

A self-quote from earlier in this thread:

When it goes wrong, it keeps going wrong within that process (some static
condition it seems).
Same as before: other redis clients have no problems.
When I start a second .NET StackExchange.Redis instance next to the first,
the second succeeds, while the first keeps on failing.

I can reproduce the exception by issuing:

ipconfig /release & ipconfig /renew

(before the first command of SE.Redis)

Can you (blemasle and SiqiLu) reproduce this?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/42#issuecomment-44379568
.

Regards,

Marc

[tw-bert]

ipconfig /release & ipconfig /renew

does not give a reliable repro for you?

[mgravell]

I am pushing a new build which hopefully should help with this; basically,
it moves some retry logic down into the library. In my tests over a VPN,
simply: sometimes the socket connect doesn't ever complete, and the only
thing to do is to take off and nuke it from orbit.

On 28 May 2014 12:26, tw-bert [email protected] wrote:

ipconfig /release & ipconfig /renew

does not give a reliable repro for you?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/42#issuecomment-44394442
.

Regards,

Marc

[BillMcCrary]

Newest version fixed my issue, thanks!

[blemasle]

I ran accross a "No connection is available for the operation" exception now. This occurs on the first redis operation that is done at the application startup. A refresh (and so re issue the command) seems to fix the problem thought. Any idea ?

[blemasle]

I confirm that I face the exact same problem. It has "simply" been relocated elsewhere.
Before : crashes while establishing the connection
After : crashes on each arbitrary operation requested, eg : "No connection is available to service this operation: ZRANGE my:key".

Retrying the operation keeps crashing, while reloading the pool works most of the time.

[SiqiLu]

Thanks for the newest version. I will try it after returning to my work. Also I will recheck whether my retry policy working.

[SiqiLu]

@blemasle I am in my holiday now without my code.I will retry my solution just after returning.

[SiqiLu]

@blemasle The same as you said, the issue would not be solved by the retrying.

[mgravell]

I've put in some changes that have addressed some connection issues, but I
simply do not have a reliable repro scenario to properly investigate here.
I can't make it fail in the way you describe. I would love to make it work,
but to do that: first I need to see it fail reproducibly.

Is there any detail / context you can add here?

On 20 June 2014 09:02, blemasle [email protected] wrote:

This issue is still reproductible on my side. Any progress on this ? I'm
seriously considering finding another library as this prevent headache-less
production exploitation....

—
Reply to this email directly or view it on GitHub
#42 (comment)
.

Regards,

Marc

[shvez]

i would say that this issue very similar on #38. I would even say, that root of problem is same. I posted there 50 line program which reproroduce 100% issue. this one https://dl.dropboxusercontent.com/u/27459474/StackExchangeTest2.zip

[blemasle]

I do not reproduce the issue on my side with that code. Connection succedded.

[shvez]

Connection fails if you started redis after client.

[blemasle]

Nope. I managed to connect, but it took a while. It seems you have a different issue there.
In our cases, it seems to fail right away (without any delay) until your kill the app hosting the connection.

[shvez]

ok. i will check with last version, thank you for info

[princeoffoods]

Just confirming that shvez's sample program reproduces the problem.

[pardha2002]

Hi me to getting same issue with new version .. 1.0.316.0

below is error i am getting ..
<--18-07-2014 09:39:20:950
resultStackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s); to create a disconnected multiplexer, disable AbortOnConnectFail
at StackExchange.Redis.ConnectionMultiplexer.Connect(ConfigurationOptions configuration, TextWriter log) in c:\TeamCity\buildAgent\work\58bc9a6df18a3782\StackExchange.Redis\StackExchange\Redis\ConnectionMultiplexer.cs:line 766

code:
var configurationOptions = new ConfigurationOptions
{
EndPoints =
{
{"xxx.xxx.x.xxxx", 6379}
},

                 //AbortOnConnectFail = false
            };

            var connectionMultiplexer = ConnectionMultiplexer.Connect(configurationOptions);
            IDatabase db = connectionMultiplexer.GetDatabase();
            //connection transaction
            //ITransaction tran = db.CreateTransaction();
            //IBatch batch = db.CreateBatch(true);

[kieranbenton]

Just another shout to say I am experiencing the same issue (with the latest nuget build). On startup of an app (I've only experienced this from a web app so far) SE.Redis will occasionally get stuck at:

StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s); to create a disconnected multiplexer, disable AbortOnConnectFail
at StackExchange.Redis.ConnectionMultiplexer.Connect(ConfigurationOptions configuration, TextWriter log)

And won't shift until I touch web.config and restart the app, at which point it works fine. This is with a local development install of redis - so networking issues are fairly unlikely.

I can't reproduce it at will - but I do see it fairly often. Is it worth me capturing a memory dump of the process when its stuck like that for you to have a look at @mgravell or perform any other kind of diagnosis for you?

Cheers.

[Plasma]

I just had this happen (recently started using SE.Redis).

3 node cluster (2 web servers, 1 redis server). I could reach the redis server fine from everywhere.

Web server 1 was working fine. Web server 2 could not reach the redis server. A touch of web.config fixed the issue. Also; this was immediately after a deployment that obviously caused the apppool to recycle.

Using NuGet 1.0.322

[Plasma]

I imagined this was some race condition during app startup; I tried throwing connections at a server (and also killing them constantly) in a test project but it consistently did re-connect correctly so I couldn't get it stuck in a permanent broken state.

[Plasma]

This happened today in the morning when I loaded up my development environment (Redis and website on same machine).

I managed to collect some diagnostic information which is interesting.

The endpoint remains in a DidNotRespond state. I did not have a TextWriter log attached to the endpoint (I do now) so I couldn't get extra trace info as to why it was marked DidNotRespond. I will next time.

I notice that it has a RunId set; yet apparently didn't respond - how would that be possible?

EDIT:

I've also attached the Storm Log at the time that shows connection status. Note that Server 2 is supposed to be unresponsive (I didn't have it running).

---

Host: PLASMA

PersistentStorage

• Storm Log -
• Status Log -
  127.0.0.1:6379: Standalone v2.8.9, master; 16 databases; keep-alive: 00:00:10; int: ConnectedEstablished; sub: ConnectedEstablished, 1 active; not in use: DidNotRespond
  127.0.0.1:6379: int ops=15, qu=0, qs=0, qc=0, wr=0, sync=20, socks=2; sub ops=9, qu=0, qs=0, qc=0, wr=0, subs=1, sync=9, socks=1
  Circular op-count snapshot; int: 14+1=15 (0.10 ops/s; spans 10s); sub: 8+1=9 (0.10 ops/s; spans 10s)
  127.0.0.1:6389: Standalone v2.0.0, master; keep-alive: 00:00:10; int: Connecting; sub: Connecting
  127.0.0.1:6389: int ops=0, qu=0, qs=0, qc=0, wr=0, sync=10, socks=50; sub ops=0, qu=0, qs=0, qc=0, wr=0, socks=50
  Circular op-count snapshot; int: 0 (0.00 ops/s; spans 10s); sub: 0 (0.00 ops/s; spans 10s)
  Sync timeouts: 0; fire and forget: 0; last heartbeat: 0s ago

Cache

• Storm Log -
• Status Log -
  127.0.0.1:6379: Standalone v2.8.9, master; 16 databases; keep-alive: 00:00:10; int: ConnectedEstablished; sub: ConnectedEstablished, 1 active; not in use: DidNotRespond
  127.0.0.1:6379: int ops=15, qu=0, qs=0, qc=0, wr=0, sync=20, socks=2; sub ops=9, qu=0, qs=0, qc=0, wr=0, subs=1, sync=9, socks=1
  Circular op-count snapshot; int: 14+1=15 (0.10 ops/s; spans 10s); sub: 8+1=9 (0.10 ops/s; spans 10s)
  127.0.0.1:6389: Standalone v2.0.0, master; keep-alive: 00:00:10; int: Connecting; sub: Connecting
  127.0.0.1:6389: int ops=0, qu=0, qs=0, qc=0, wr=0, sync=10, socks=50; sub ops=0, qu=0, qs=0, qc=0, wr=0, socks=50
  Circular op-count snapshot; int: 0 (0.00 ops/s; spans 10s); sub: 0 (0.00 ops/s; spans 10s)
  Sync timeouts: 0; fire and forget: 0; last heartbeat: 0s ago

Jobs

• Storm Log -
• Status Log -
  127.0.0.1:6379: Standalone v2.8.9, master; 16 databases; keep-alive: 00:00:10; int: ConnectedEstablished; sub: ConnectedEstablished, 1 active; not in use: DidNotRespond
  127.0.0.1:6379: int ops=15, qu=0, qs=0, qc=0, wr=0, sync=20, socks=2; sub ops=9, qu=0, qs=0, qc=0, wr=0, subs=1, sync=9, socks=1
  Circular op-count snapshot; int: 14+1=15 (0.10 ops/s; spans 10s); sub: 8+1=9 (0.10 ops/s; spans 10s)
  127.0.0.1:6389: Standalone v2.0.0, master; keep-alive: 00:00:10; int: Connecting; sub: Connecting
  127.0.0.1:6389: int ops=0, qu=0, qs=0, qc=0, wr=0, sync=10, socks=50; sub ops=0, qu=0, qs=0, qc=0, wr=0, socks=50
  Circular op-count snapshot; int: 0 (0.00 ops/s; spans 10s); sub: 0 (0.00 ops/s; spans 10s)
  Sync timeouts: 0; fire and forget: 0; last heartbeat: 0s ago

[AngCaruso]

Just to pile-on and say I am experiencing the same issue. In my case, our application consists of Redis installed as a windows service, three other windows services, and an ASP.NET MVC web application. All running on the same machine, and all communicate through Redis in various ways. What I am finding since I switched from BookSleeve to StackExchange.Redis a few days ago is that during installation, my services take an extraordinary amount of time to start up, apparently because they are taking quite a long time to attempt to connect to the Redis service. Sometimes they succeed and some times they fail to connect. I see the same exception message noted by other posters above. When they fail, I can sometimes get them to connect by restarting the service. I would say the failure rate is greater than 50% in my case. This is happening on both server 2008 R2 as well as 2012 R2.

For now I am forced to switch back to BookSleeve (which never failed me) since our software is entering Beta testing and reliability is a must. I am hoping this gets resolved soon.

I am using version 1.0.322 which I pulled with NuGet just a few days ago.

[Raven87]

Adding to this as well since we just had this occur last night for the first time. We have two applications running on the same machine (Windows Server 2008 R2) both using the same version of StackExchange.Redis (version 1.0.273, yes not the latest version). The WCF Service application had no problems but our Web API application bombed last night trying to connect to our Redis server right after IIS performed the 29 hour recycle on the WebAPI application.

[blemasle]

That's the exact same case we have @Raven87. What I found until now is to kill the w3wp process and bring it to life again.

Until now, i've never been able to reproduce the issue on a standalone app, but I keep trying to reproduce the issue at a 100% rate as this is becoming more and more problematic...

[shvez]

@Plasma, @Raven87, @blemasle i ask just to be sure. are you talking about last version build 322 or higher?

[blemasle]

@shvez Yes I updated to the latest version yesterday or the day before, can't remember.

[shvez]

@blemasle thank you

[mgravell]

K; I need to read through a lot of "chain" and try to understand what is
going on here...

On 6 August 2014 13:53, Ilya Shvetsov [email protected] wrote:

@blemasle https://github.com/blemasle thank you

—
Reply to this email directly or view it on GitHub
#42 (comment)
.

Regards,

Marc

[blemasle]

It's not really the same case but the resulting behaviour is the same if once started and connected, you shut down the redis server.

You get a (normal in this case) : "No connection is available to service this operation: EXEC"

Maybe this can help you find what's going wrong in case where a connection IS available ?

[Raven87]

After further testing if we call ConnectionMultiplexer.Connect(...) and this fails with a time out (in our case we turned Redis off first) the method throws an exception and fails. We then turn Redis back on and the connect call is made again it still continues to fail despite the fact that it should be able to connect to Redis (which we verified using Redis Desktop Manager). Our work around, that we think works, is to add "abortConnect=false" in the connection configuration which then creates a disconnected multiplexer and once we turn Redis back on this can connect just fine without issue.

@blemasle Maybe this type of work around would work for you as well?

This issue seems to only happen if the very first connect fails. We did encounter other issues with the first scenario as well when we attempted to dispose of the text logger we passed in to the Connect(...) call.

*Note - we did our testing using the latest version of StackExchange.Redis 1.0.322; we only have one Redis server (no shards or fail overs); and this was done in a Web API application using a singleton pattern to create the multiplexer.

[AngCaruso]

@Raven87 The "abortConnect=false" configuration setting did not solve the problem for me.

[tw-bert]

Yes, the problem also persists for us.
We do have an ugly but (so far) reliable workaround: use BookSleeve as well as SE.Redis.
Make a dummy connection using BookSleeve, and immediately after that, make a connection with SE.Redis. BookSleeve seems to initialize the routing in a proper way (see my earlier comments in May).
Disconnect and dispose the BookSleeve connection when you are connected through SE.Redis.

Could be of help to someone, cheers

[Plasma]

I can reproduce "No connection is available to service this operation: EXEC" If I break into the debugger and pause execution for some time (say 1 minute) and then resume the website running; all requests fail with this immediately and doesn't seem to figure out the connection is okay.

[blemasle]

@Plasma I can't reproduce it using your provided steps. I'll try the abortConnect=false trick from @Raven87 but I do not believe in it.

The disconnected multiplexer was a good match with old version where some logic were done on first initialization. Since @mgravell made some changes to that logic as he said (moving it deeper in the lib if I remember correctly), we're seing the issue on each operation, stating that "no connection is available".

So I can't see how the disconnectedMultiplexer could solve this :)

[Plasma]

I notice that the connection check is done via a Task; perhaps thread starvation causes the check to timeout?

Sent from my iPhone

On 7 Aug 2014, at 6:40 pm, blemasle [email protected] wrote:

@Plasma I can't reproduce it using your provided steps. I'll try the abortConnect=false trick from @Raven87 but I do not believe in it.

The disconnected multiplexer was a good match with old version where some logic were done on first initialization. Since @mgravell made some changes to that logic as he said (moving it deeper in the lib if I remember correctly), we're seing the issue on each operation, stating that "no connection is available".

So I can't see how the disconnectedMultiplexer could solve this :)

—
Reply to this email directly or view it on GitHub.

[mgravell]

It is indeed possible that it is related to thread starvation, especially
at the start of an application due to the way that the thread-pool ramps (I
remember sitting down with Lucian for about an hour just talking about the
way this works). I am actively investigating.

On 7 August 2014 10:40, Andrew Armstrong [email protected] wrote:

I notice that the connection check is done via a Task; perhaps thread
starvation causes the check to timeout?

Sent from my iPhone

On 7 Aug 2014, at 6:40 pm, blemasle [email protected] wrote:

@Plasma I can't reproduce it using your provided steps. I'll try the
abortConnect=false trick from @Raven87 but I do not believe in it.

The disconnected multiplexer was a good match with old version where
some logic were done on first initialization. Since @mgravell made some
changes to that logic as he said (moving it deeper in the lib if I remember
correctly), we're seing the issue on each operation, stating that "no
connection is available".

So I can't see how the disconnectedMultiplexer could solve this :)

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
#42 (comment)
.

Regards,

Marc

[Plasma]

This is a test that shows TIMEOUT happening, although I wonder if its the way async/await works in a Console app. https://gist.github.com/Plasma/62f9e3a2010ed1a6e486

[turowicz]

Simillar problems here, redis going up and down, connections aren't being made. The "abortConnect=false" doesn't help.

[noocyte]

I am able to repro this error on my machine now. If I turn of SSL it works just fine... So port 6379 works, but not 6380... I see the same when connecting using a Redis client.

[Vijay27anand]

I am also facing same issue with ASP.net application on Windows 2008 server, but its working fine on a Windows 7 local dev pc.

ConnectionMultiplexer.Connect is throwing this error, on button click event. I have to reset IIS to make it work again. its working for one or two button click event and after that constistently getting this error and IIS reset is required to make it work again and fails again.

StackExchange.Redis.RedisConnectionException: It was not possible to connect to
the redis server(s); to create a disconnected multiplexer, disable
AbortOnConnectFail

Let me know when this issue will be fixed.

[cilliemalan]

We were experiencing the same issue, and what seemed to fix it was using a host name instead of an IP address.

In our case it was working fine on dev machines, but sporadically failed on the server.

[turowicz]

Closing the connection and disposing it before creating new one fixed it for us.

  static readonly Object Lock = new Object();
  static ConnectionMultiplexer _connection;

  static ConnectionMultiplexer Connection
  {
      get
      {
          lock (Lock)
          {
              if (_connection == null || !_connection.IsConnected)
              {
                  //kill old connection
                  if (_connection != null)
                  {
                      _connection.Close(false);
                      _connection.Dispose();
                      _connection = null;
                  }

                  _connection = ConnectionMultiplexer.Connect(_redisConnection);
              }
              return _connection;
          }
      }
  }

[mgravell]

hmmm... sounds like DNS lookup might be at least related; that gives me
something to look into, at least - I'll boost the error specificity in that
area to see if that is indeed the case

On 25 September 2014 14:20, Wojtek Turowicz [email protected]
wrote:

Closing the connection and disposing it before creating new one fixed it
for us.

static ConnectionMultiplexer _connection;

static ConnectionMultiplexer Connection
{
get
{
lock (Lock)
{
if (_connection == null || !_connection.IsConnected)
{
//kill old connection
if (_connection != null)
{
_connection.Close(false);
_connection.Dispose();
_connection = null;
}

              _connection = ConnectionMultiplexer.Connect(_redisConnection);
          }
          return _connection;
      }
  }

}

—
Reply to this email directly or view it on GitHub
#42 (comment)
.

Regards,

Marc

[Vijay27anand]

Turowicz If we have connection as static singleton as mention in the post, for a enterprise application in one IIS worker process we will have only one connection right, so is it good idea to have it as static connection or how to multiple connection from one worker process for concurrency.

[turowicz]

You need only one connection multiplexer. The Connect(...) method returns a new instance of it. This is how I understand the documentation provided in Readme.md file.

[mgravell]

Yes, multiplexers should be aggressively shared and re-used; they should
not be created per-call. Any mechanism that supports that is fine; the
simplest of which is a static field. But there are lots of ways of
implementing it.

On 26 September 2014 09:26, Wojtek Turowicz [email protected]
wrote:

You need only one connection multiplexer. The Connect(...) method returns
a new instance of it. This is how I understand the documentation provided
in Readme.md file.

—
Reply to this email directly or view it on GitHub
#42 (comment)
.

Regards,

Marc

[Vijay27anand]

I have a requirement to have Redis server running in different boxes and application code in different boxes. Based on the session id which is a number we will to route to Redis server 1 or 2 or 3. We are using the static connection multiplexer to reuse the connection but every time we are hitting the first initialized connection since its static. Is there a way that we can have multiple connections to different Redis server and reuse it and based on our session id hash logic identify which Redis server to hit and get the value.

[mgravell]

If you have "n" multiplexers and code that fetches the multiplexer you want
based on the session: sure. Just: get the appropriate multiplexer and
have fun.

Note that when cluster is RTM you can achieve something similar using hash
tags in the key, meaning: everything "/foo/{sessionid}/bar/..." is
guaranteed to be in the same hash bucket, so on the same node. SE.Redis
does not currently include non-cluster client-based sharding, simply
because I don't have the scenarios to properly design, implement, and test
such. Most of the actual shard code exists though, because of cluster.
On 6 Oct 2014 20:25, "Vijay27anand" [email protected] wrote:

I have a requirement to have Redis server running in different boxes and
application code in different boxes. Based on the session id which is a
number we will to route to Redis server 1 or 2 or 3. We are using the
static connection multiplexer to reuse the connection but every time we are
hitting the first initialized connection since its static. Is there a way
that we can have multiple connections to different Redis server and reuse
it and based on our session id hash logic identify which Redis server to
hit and get the value.

—
Reply to this email directly or view it on GitHub
#42 (comment)
.

[razzemans]

@mgravell I have seen this issue 2 times during development and found this thread. I'm using a hostname in the connectionstring and after setting my DNS server to something non-existant I get this error everytime. Well, actually also sometimes a variation on it:

It was not possible to connect to the redis server(s); to create a disconnected multiplexer, disable AbortOnConnectFail. SocketFailure on PING

and sometimes:

It was not possible to connect to the redis server(s); to create a disconnected multiplexer, disable AbortOnConnectFail. UnableToResolvePhysicalConnection on PING

So sounds to me like a DNS hickup when I got this error 2 times during development.

[vladkosarev]

Trying out Redis on Azure and also getting this intermittently -

StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s); to create a disconnected multiplexer, disable AbortOnConnectFail. UnableToResolvePhysicalConnection on PING at StackExchange.Redis.ConnectionMultiplexer

[jijiechen]

same issue here on windows server 2012, azure vm with subnet enabled, in iis hosted site.

Other clients, including telnet clients can connect to this Redis server, but my code fails very often.
Redis Server version 2.8.17; StackExchange.Redis client version 1.0.333.

configurationOptions: 192.168.1.11:6387,keepAlive=180,allowAdmin=True
logs:
It was not possible to connect to the redis server(s); to create a disconnected multiplexer, disable AbortOnConnectFail.

1 unique nodes specified
Requesting tie-break from 192.168.1.11:6387 > __Booksleeve_TieBreak...
Allowing endpoints 00:00:05 to respond...
192.168.1.11:6387 did not respond
192.168.1.11:6387 failed to nominate (WaitingForActivation)
No masters detected
192.168.1.11:6387: Standalone v2.0.0, master; keep-alive: 00:03:00; int: Connecting; sub: ConnectedEstablished, 1 active; not in use: DidNotRespond
192.168.1.11:6387: int ops=0, qu=4, qs=0, qc=0, wr=0, socks=1; sub ops=3, qu=0, qs=0, qc=0, wr=0, subs=1, sync=3, socks=1
Circular op-count snapshot; int: 0 (0.00 ops/s; spans 10s); sub: 0+3=3 (0.30 ops/s; spans 10s)
Sync timeouts: 0; fire and forget: 0; last heartbeat: -1s ago
resetting failing connections to retry...
retrying; attempts left: 2...
1 unique nodes specified
Requesting tie-break from 192.168.1.11:6387 > __Booksleeve_TieBreak...
Allowing endpoints 00:00:05 to respond...
192.168.1.11:6387 did not respond
192.168.1.11:6387 failed to nominate (WaitingForActivation)
No masters detected
192.168.1.11:6387: Standalone v2.0.0, master; keep-alive: 00:03:00; int: Connecting; sub: ConnectedEstablished, 1 active; not in use: DidNotRespond
192.168.1.11:6387: int ops=0, qu=2, qs=0, qc=0, wr=0, async=5, socks=2; sub ops=3, qu=0, qs=0, qc=0, wr=0, subs=1, sync=3, socks=1
Circular op-count snapshot; int: 0 (0.00 ops/s; spans 10s); sub: 0+3=3 (0.30 ops/s; spans 10s)
Sync timeouts: 0; fire and forget: 0; last heartbeat: -1s ago
resetting failing connections to retry...
retrying; attempts left: 1...
1 unique nodes specified
Requesting tie-break from 192.168.1.11:6387 > __Booksleeve_TieBreak...
Allowing endpoints 00:00:05 to respond...
192.168.1.11:6387 did not respond
192.168.1.11:6387 failed to nominate (WaitingForActivation)
No masters detected
192.168.1.11:6387: Standalone v2.0.0, master; keep-alive: 00:03:00; int: Connecting; sub: ConnectedEstablished, 1 active; not in use: DidNotRespond
192.168.1.11:6387: int ops=0, qu=2, qs=0, qc=0, wr=0, async=8, socks=3; sub ops=3, qu=0, qs=0, qc=0, wr=0, subs=1, sync=3, socks=1
Circular op-count snapshot; int: 0 (0.00 ops/s; spans 10s); sub: 0+3=3 (0.30 ops/s; spans 10s)
Sync timeouts: 0; fire and forget: 0; last heartbeat: -1s ago

[turowicz]

I'm getting that error again. The interesting thing is that some of my peer devs are working on the exact same setup and they do not experience this.

Any ideas?

BTW even trying to connect to a localhost redis doesn't work. Meanwhile redis-client connects instantly.

[kohner86]

I'm seeing the same issue intermittently while connecting to an Azure Redis instance. From the Azure monitoring I only have ~40 connections open at any given time.

-Lee

[mralj]

This is happening to me too, I'm using StackExchange.Redis.StrongName ver. 1.0.394, oddly enough, I've noticed that if I use StackExchange.Redis ver. 1.0.394 everything works fine. (but I need StrongName because of RedisSessionStateProvider)

[bhenze]

Has anyone found a definitive solution for the "No connection is available to service this operation: SETEX" issue? I am currently using RE.Redis 1.0.331 and am seeing the same thing in my WebAPI app running in IIS. I have yet to constantly reproduce this issue but when it does happen its always occurs after rebooting our server.

I have 5 servers talking to redis and redis is configured with 1 master and 3 slaves. When I get into this state, about 1 in 5 or so calls actually work and the rest give me the no connection error. Also, most of the time my servers have no issue spinning back up and calling redis. I see this after reboot only about 1 in 10 times. I am just starting to dive into solutions for this issue but wanted to verify if someone has tracked this to the route cause yet.

[andysinclair]

We were experiencing the same issue on one of our installations and to resolve it I replaced the IP address of the Redis server with the hostname (as mentioned earlier in this thread). No idea why this happened as we have the same configuration on lots of machines and have never seen this issue before.

[BrandesEric]

+1 - I'm seeing what looks like the same thing with 1.0.414. It happens only on app startup, and not every time, but almost every time. We're deploying with MSDeploy. I'm going to try adding a ConnectRetry value > 1 and using the Task.Run() method @smiket suggested.

Update: This is still occurring after trying all reasonable mitigations I could find in this thread.

@mgravell I can reproduce this fairly reliably - what can I get you to help you debug it further?

[BrandesEric]

@saixiaohui thanks for the insight! I think you're on to something. We're using a limited concurrency scheduler as well - and the call to create the multiplexer the first time and do the Connect() is actually within a spun-off task.

I ended up creating the multiplexer and doing the Connect() in the static ctor of the class that ultimately spawns off the threads using the LCS, and so far that seems to have solved it. However, i'm not calling it a success yet, since the issue was semi-intermittent.

[dengere]

1.0.481 version, i've got exactly same exception as started this thread subject.

"It was not possible to connect to the redis server(s); to create a disconnected multiplexer, disable AbortOnConnectFail. SocketFailure on PING"

did i missed something?

[linfx]

It was not possible to connect to the redis server(s); to create a disconnected multiplexer, disable AbortOnConnectFail. SocketFailure on PING

[kohner86]

I was never able to come up with a real solution to this problem so I ended up creating a 'WatchDog' timer which performs a write operation every 30 seconds and re-initializes the ConnectionMultiplexer if the operation fails. Seems to be working pretty well, haven't had a single outage caused by this since the timer was put in place.

[aliostad]

Just saw this in production twice.

[DentArthrDent]

My team and I just noticed a correlation between our web servers having connection issues with Redis (No Connections Available / Timeouts) and Redis saving snapshots of itself.

After a little digging I stumbled on this gem on http://redis.io/topics/persistence

For our application we don't need persistence so I'm just disabling it altogether, but might be worth looking into as a potential cause or if it's tied to the difficulty with reproducing the issue.

I'll post again if the errors show again with persistence disabled

[DentArthrDent]

Unfortunately I am still seeing the issue relatively frequently

[MarwaAhmed]

I've the same problem. I'm using Azure redis cache as a session provider:
"Exception": {
"ClassName": "StackExchange.Redis.RedisConnectionException",
"Message": "No connection is available to service this operation: EXISTS xxxx,
"Data": {
"redis-command": "EXISTS xxxx"
},

Where xxxx is an arbitrary key.

[swmech]

Checking on this issue, as we're seeing it in production as well (Azure Redis cache). Has there been progress made on determining a fix? Or is first opening a dummy connection still the accepted workaround?

[dengere]

Do you really have any workaround? We can't use on azure last two months.
We stop to use it.
30 Ara 2015 00:35 tarihinde "swmech" [email protected] yazdı:

Checking on this issue, as we're seeing it in production as well (Azure
Redis cache). Has there been progress made on determining a fix? Or is
opening a dummy connection first still the accepted workaround?

—
Reply to this email directly or view it on GitHub.

[jonranes]

I had the same issue, I implemented the rjmooney change and the problem is not happening anymore.

[aliostad]

👍 @rjmooney Thanks!
@mgravell @NickCraver Guys this is bringing our site down once a day, can you please review and release the fix?

[smiket]

@aliostad please let us know if this resolves your issue after a few days?

[aliostad]

@smiket we are deploying next week, I will keep you posted.

[ronnyek]

Is this code in any nuget releases? I've cloned latest and included and still very much seeing this in dnx on linux. redis-cli works fine, but still cant connect via stackexchange.redis

[aliostad]

@smiket it looks like my problem had nothing to do with this. But still investigating.

[rjmooney]

@ronnyek @aliostad https://www.nuget.org/packages/StackExchange.Redis/1.1.572-alpha. What issue are you each seeing exactly?

[smiket]

@aliostad @rjmooney we solved our problem - but it had nothing to do with the pull request associated with this issue. Our connection errors were caused by what I outlined above (see my previous post). We were seeing the errors every day - and now we have not seen it in two weeks.

If the threadpool is drained then SE redis will not be able to connect. The reason is that it times out waiting for a thread to become available, as it tries to connect. The error indicates a connection issue, but its really thread starvation in our case (not a deadlock).

We took a hard look at our code, and found we were running several simultaneous jobs to load our cache. The jobs kick off at the same time, and each uses parallel loops etc. Lots of simultaneous, parallel loops - drained the pool.

We lightened up on parallelism, decreased the number of jobs running at same time and such. By putting less pressure on the thread pool, we solved the issue. In our case (and I suspect same for a number of other folks here) we were causing the issue. Its just unfortunate the error message is a bit misleading.

Hope it helps!
Steve

[ronnyek]

@rjmooney basically in vnext with I believe that alpha, I was getting this:
An unhandled exception of type 'StackExchange.Redis.RedisConnectionException' occurred in StackExchange.Redis.dll
Additional information: It was not possible to connect to the redis server(s); to create a disconnected multiplexer, disable AbortOnConnectFail on launch

only running on linux.

[danielkornev]

Same thing, although we launch multiple separate instances for a desktop (yikes!)-based WPF 4.5 app.

Issue remains when using both the latest stable, and the latest alpha versions of StackExchange.Redis from NuGet.

Maybe it's a good idea to create own threads instead of using the Managed Thread Pool?

StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s); to create a disconnected multiplexer, disable AbortOnConnectFail. UnableToResolvePhysicalConnection on PING
   at StackExchange.Redis.ConnectionMultiplexer.<ConnectAsync>d__20.MoveNext() in c:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\ConnectionMultiplexer.cs:line 765
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at ZU.Storage.Redis.RedisEnvironmentManagement.<>c__DisplayClass0_0.<<CreateRedisInstance>b__0>d.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at ZU.Storage.Redis.RedisEnvironmentManagement.CreateRedisInstance(String currentFolder, Boolean showRedisInstanceCommandPrompt)