Comments (4)
Using the ConnectionPoolManager for Round Robin ConnectionMultiplexer.
Unfortunately, that's not something we're familiar with or control, but my first guess given your symptoms is that is re-creating multiplexers rather than letting them re-connect internally, likely spiking your socket counts too. I'd need to see the full exceptions (we're getting truncated there) and how long we've been connected to get a better idea. The full exception message will also contain some thread pool and queue data which is rather helpful in diagnosing these.
from stackexchange.redis.
Hi Nick,
thanks for your answer. I can even reproduce this behaviour, when just using one instance of connectionmultiplexer, ignoring the ConnectionPoolManager.
I enabled some more debug information. Here is a full exception:
2024-04-23 17:16:51,582 level=ERROR,class=CustomRedisCacheService,id=4fd044dc-cef5-4b48-976c-bdb0cf17b16d,action=GetValueAsync,"StackExchange.Redis.RedisConnectionException: No connection is active/available to service this operation: TTL+GET Measurement.ApiMeas... It was not possible to connect to the redis server(s). ConnectTimeout, mc: 1/1/0, clientName: 47VMR02(SE.Redis-v2.7.33.41805), PerfCounterHelperkeyHashSlot: 4497, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=241,Free=32526,Min=4,Max=32767), POOL: (Threads=361,QueuedItems=1186,CompletedItems=435500,Timers=35), v: 2.7.33.41805
---> System.AggregateException: One or more errors occurred. (It was not possible to connect to the redis server(s). ConnectTimeout) (It was not possible to connect to the redis server(s). ConnectTimeout) (It was not possible to connect to the redis server(s). ConnectTimeout) (It was not possible to connect to the redis server(s). ConnectTimeout) (It was not possible to connect to the redis server(s). ConnectTimeout) (It was not possible to connect to the redis server(s). ConnectTimeout) (SocketClosed (ReadEndOfStream, last-recv: 0) on 192.168.47.162:6379/Interactive, Idle/MarkProcessed, last: SETEX, origin: ReadFromPipe, outstanding: 413, last-read: 0s ago, last-write: 0s ago, keep-alive: 3600000s, state: ConnectedEstablished, mgr: , in: 0, in-pipe: 0, out-pipe: 108914, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.7.33.41805) (SocketClosed (ReadEndOfStream, last-recv: 0) on 192.168.47.192:6379/Interactive, Idle/MarkProcessed, last: GET, origin: ReadFromPipe, outstanding: 436, last-read: 0s ago, last-write: 0s ago, keep-alive: 3600000s, state: ConnectedEstablished, mgr: , in: 0, in-pipe: 0, out-pipe: 110859, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.7.33.41805) (SocketClosed (ReadEndOfStream, last-recv: 0) on 192.168.47.191:6379/Interactive, Idle/MarkProcessed, last: GET, origin: ReadFromPipe, outstanding: 396, last-read: 0s ago, last-write: 0s ago, keep-alive: 3600000s, state: ConnectedEstablished, mgr: , in: 0, in-pipe: 0, out-pipe: 62341, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.7.33.41805) (SocketClosed (ReadEndOfStream, last-recv: 0) on 192.168.47.195:6379/Interactive, Idle/MarkProcessed, last: INFO, origin: ReadFromPipe, outstanding: 0, last-read: 0s ago, last-write: 27s ago, keep-alive: 3600000s, state: ConnectedEstablished, mgr: , in: 0, in-pipe: 0, out-pipe: 0, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.7.33.41805) (SocketClosed (ReadEndOfStream, last-recv: 0) on 192.168.47.185:6379/Interactive, Idle/MarkProcessed, last: INFO, origin: ReadFromPipe, outstanding: 0, last-read: 0s ago, last-write: 55s ago, keep-alive: 3600000s, state: ConnectedEstablished, mgr: , in: 0, in-pipe: 0, out-pipe: 0, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.7.33.41805) (SocketClosed (ReadEndOfStream, last-recv: 0) on 192.168.47.187:6379/Interactive, Idle/MarkProcessed, last: INFO, origin: ReadFromPipe, outstanding: 0, last-read: 0s ago, last-write: 27s ago, keep-alive: 3600000s, state: ConnectedEstablished, mgr: , in: 0, in-pipe: 0, out-pipe: 0, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.7.33.41805)
---> StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). ConnectTimeout
--- End of inner exception stack trace ---
---> (Inner Exception #1) StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). ConnectTimeout<---
---> (Inner Exception #2) StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). ConnectTimeout<---
---> (Inner Exception #3) StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). ConnectTimeout<---
---> (Inner Exception #4) StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). ConnectTimeout<---
---> (Inner Exception #5) StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). ConnectTimeout<---
---> (Inner Exception #6) StackExchange.Redis.RedisConnectionException: SocketClosed (ReadEndOfStream, last-recv: 0) on 192.168.47.162:6379/Interactive, Idle/MarkProcessed, last: SETEX, origin: ReadFromPipe, outstanding: 413, last-read: 0s ago, last-write: 0s ago, keep-alive: 3600000s, state: ConnectedEstablished, mgr: , in: 0, in-pipe: 0, out-pipe: 108914, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.7.33.41805
at Sma.Sp.Measurement.CustomRedisCacheService.SetValueAsync(CacheKey key, Object value, CacheExpiry expiry) in D:\Development\IguanaBL\BusinessCore\Sma.Sp.Measurement\Code\Sma.Sp.Measurement\CustomRedisCacheService.cs:line 169<---
---> (Inner Exception #7) StackExchange.Redis.RedisConnectionException: SocketClosed (ReadEndOfStream, last-recv: 0) on 192.168.47.192:6379/Interactive, Idle/MarkProcessed, last: GET, origin: ReadFromPipe, outstanding: 436, last-read: 0s ago, last-write: 0s ago, keep-alive: 3600000s, state: ConnectedEstablished, mgr: , in: 0, in-pipe: 0, out-pipe: 110859, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.7.33.41805
at Sma.Sp.Measurement.CustomRedisCacheService.GetValueAsync(CacheKey cacheKey, Type responseType) in D:\Development\IguanaBL\BusinessCore\Sma.Sp.Measurement\Code\Sma.Sp.Measurement\CustomRedisCacheService.cs:line 80<---
---> (Inner Exception #8) StackExchange.Redis.RedisConnectionException: SocketClosed (ReadEndOfStream, last-recv: 0) on 192.168.47.191:6379/Interactive, Idle/MarkProcessed, last: GET, origin: ReadFromPipe, outstanding: 396, last-read: 0s ago, last-write: 0s ago, keep-alive: 3600000s, state: ConnectedEstablished, mgr: , in: 0, in-pipe: 0, out-pipe: 62341, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.7.33.41805
at Sma.Sp.Measurement.CustomRedisCacheService.GetValueAsync(CacheKey cacheKey, Type responseType) in D:\Development\IguanaBL\BusinessCore\Sma.Sp.Measurement\Code\Sma.Sp.Measurement\CustomRedisCacheService.cs:line 80<---
---> (Inner Exception #9) StackExchange.Redis.RedisConnectionException: SocketClosed (ReadEndOfStream, last-recv: 0) on 192.168.47.195:6379/Interactive, Idle/MarkProcessed, last: INFO, origin: ReadFromPipe, outstanding: 0, last-read: 0s ago, last-write: 27s ago, keep-alive: 3600000s, state: ConnectedEstablished, mgr: , in: 0, in-pipe: 0, out-pipe: 0, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.7.33.41805<---
---> (Inner Exception #10) StackExchange.Redis.RedisConnectionException: SocketClosed (ReadEndOfStream, last-recv: 0) on 192.168.47.185:6379/Interactive, Idle/MarkProcessed, last: INFO, origin: ReadFromPipe, outstanding: 0, last-read: 0s ago, last-write: 55s ago, keep-alive: 3600000s, state: ConnectedEstablished, mgr: , in: 0, in-pipe: 0, out-pipe: 0, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.7.33.41805<---
---> (Inner Exception #11) StackExchange.Redis.RedisConnectionException: SocketClosed (ReadEndOfStream, last-recv: 0) on 192.168.47.187:6379/Interactive, Idle/MarkProcessed, last: INFO, origin: ReadFromPipe, outstanding: 0, last-read: 0s ago, last-write: 27s ago, keep-alive: 3600000s, state: ConnectedEstablished, mgr: , in: 0, in-pipe: 0, out-pipe: 0, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.7.33.41805<---
--- End of inner exception stack trace ---
at StackExchange.Redis.ConnectionMultiplexer.ThrowFailed[T](TaskCompletionSource`1 source, Exception unthrownException) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2048
--- End of stack trace from previous location ---
from stackexchange.redis.
worth mentioning, that the same connectionmultiplexer instance worked fine for several minutes before starting with this with answer times around 0-10 ms
from stackexchange.redis.
It looks like you have some extreme thread exhaustion as indicated by WORKER: (Busy=241,Free=32526,Min=4,Max=32767), POOL: (Threads=361,QueuedItems=1186,CompletedItems=435500,Timers=35)
, this will be that tasks for connecting to Redis or completing commands get so far behind in the queue, they timeout before being completed (at the shared threadpool level). I'd suggest taking a dump and see what those hundreds of threads are doing, it's likely something's either locking or a remote call that should be async elsewhere in the application.
from stackexchange.redis.
Related Issues (20)
- ERR wrong number of arguments for 'expire' command - Azure Cache for Redis HOT 4
- Support for Hash Field Expiration commands HOT 1
- How to resume enumerate key from last breakpoint? HOT 2
- When need use await for batch.KeyExpireAsync(key, TimeSpan.FromSeconds(expiry)) and when not? HOT 4
- Is there any way to match keys aligned reg pattern ^[0-9A-Za-z_-]{80}$? HOT 8
- Load test with subscribe/unsubscribe/publish HOT 2
- Support for HSCAN NOVALUES option HOT 1
- Support reading last message from stream
- Support MAXAGE option for CLIENT KILL
- Connection established but 'The specified endpoint is not defined' HOT 1
- What't the possible reason of StackExchange.Redis.RedisTimeoutException?
- Error: No endpoints specified (Parameter 'config'), when connecting from Docker container HOT 5
- Can I pick specific shard of a redis server? HOT 9
- ArgumentException while OutputCache is cached HOT 2
- What's the possible reason leading difference peak in Server Load, Processor Time and CPU usage? HOT 6
- RedisConnectionException in StackExchange.Redis when using ElastiCache Cluster "No connection is available to service this operation" HOT 3
- SortedSetRangeByValueAsync method returns incorrect results when filtering by value range
- SortedSetRank doesn't include the score HOT 1
- Issues setting key using Microsoft.Extensions.Caching.StackExchangeRedis HOT 3
- Redis7.4-rc1 is available
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from stackexchange.redis.