Comments (11)
How is the RStream.readGroup() method called?
ConnectionWatchDog has some logs printed but not clear for me
It looks like this connection in reconnection state. Can you share the full log?
from redisson.
@mrniko
Actually we are using spring library( org.springframework.data.redis.connection.stream) and the logs are too huge but I previously sent the ones which were suspicious.
This is working fine but at times consumer stops receiving messages over stream and nothing helps unless app service restart.
Also, please suggest if this is fine to use or we should use conventional way using a while(true) loop and fetch/poll using RStream continuously.
Below is the consumer side polling =>
private RedisConnectionFactory redisConnectionFactory; // RedissonConnectionFactory autowired via //RedissonAutoConfiguration
StreamMessageListenerContainer listenerContainer = StreamMessageListenerContainer.create(redisConnectionFactory, options);
Subscription subscription = listenerContainer.register(getReadRequest(consumer, streamOffset), listener(channel, consumer, listener));
listenerContainer.start();
private StreamMessageListenerContainer.StreamReadRequest<String> getReadRequest(
Consumer consumer, StreamOffset<String> streamOffset) {
return StreamMessageListenerContainer.StreamReadRequest
.builder(streamOffset)
.consumer(consumer)
.autoAcknowledge(false)
.cancelOnError((err) -> false) // do not stop consuming after error
.build();`
}
private <T> StreamListener listener(String channel, Consumer consumer, RedisStreamListener<T> listener) {
return message -> {
try {
log.info("Acknowledging message: {}", message.getId());
stringRedisTemplate.opsForStream().acknowledge(channel, consumer.getGroup(), message.getId());
log.info("RECEIVED ({}) message: messageId = {}, sequence = {}, data = {}", consumer,
message.getId().getValue(), message.getId().getSequence(), message.getValue());
listener.onReceive((T) message.getValue());
} catch(Exception e) {
log.error("Error while processing message:{}", message);
}
};
}
private <T> StreamMessageListenerContainer.StreamMessageListenerContainerOptions<String, ObjectRecord<String, T>> getOptions(
RedisStreamListener<T> listener) {
return StreamMessageListenerContainer
.StreamMessageListenerContainerOptions
.builder()
.pollTimeout(Duration.ofSeconds(0))
.targetType(listener.getMessageType())
.build();
}
Publisher side code:
class RedisStreamPublisherImpl implements RedisStreamPublisher {
@Resource(type = StringRedisTemplate.class)
private StringRedisTemplate stringRedisTemplate;
@Override
public <T extends Serializable> RecordId publish(String channel, T message) {
RecordId recordId = null;
try {
ObjectRecord<String, T> record = StreamRecords.newRecord()
.ofObject(message)
.withStreamKey(channel);
recordId = stringRedisTemplate.opsForStream().add(record);
if (isNull(recordId)) {
log.warn("STREAMING Error: topic = {}, data = {}, error: {}", channel, message, "RecordId returned as null");
} else {
log.info("STREAMING: recordId = {}, topic = {}, data = {}", recordId.getValue(), channel, message);
}
} catch (Throwable throwable) {
log.error("STREAMING Errored", throwable);
}
return recordId;
}
}
from redisson.
Unable to reproduce using the test below.
@Test
public void test1() throws InterruptedException {
RedisConnectionFactory redisConnectionFactory = new RedissonConnectionFactory(redisson);
StreamMessageListenerContainer listenerContainer = StreamMessageListenerContainer.create(redisConnectionFactory, getOptions());
Consumer consumer = Consumer.from("group", "consumer1");
StreamOffset<String> streamOffset = StreamOffset.create("test", ReadOffset.from(">"));
String channel = "test";
AtomicInteger counter = new AtomicInteger();
Subscription subscription = listenerContainer.register(getReadRequest(consumer, streamOffset),
listener(redisConnectionFactory, channel, consumer, counter));
StringRedisTemplate t1 = new StringRedisTemplate(redisConnectionFactory);
t1.opsForStream().createGroup("test", "group");
listenerContainer.start();
ExecutorService s = Executors.newSingleThreadExecutor();
s.submit(() -> {
for (int i = 0; i < 10; i++) {
StringRedisTemplate stringRedisTemplate = new StringRedisTemplate(redisConnectionFactory);
ObjectRecord<String, String> record = StreamRecords.newRecord()
.ofObject("message")
.withStreamKey(channel);
RecordId recordId = stringRedisTemplate.opsForStream().add(record);
System.out.println("recordId " + recordId);
}
});
Thread.sleep(5000);
Assertions.assertThat(counter.get()).isEqualTo(10);
}
private StreamMessageListenerContainer.StreamMessageListenerContainerOptions<String, ObjectRecord<String, String>> getOptions() {
return StreamMessageListenerContainer
.StreamMessageListenerContainerOptions
.builder()
.pollTimeout(Duration.ofSeconds(1))
.targetType(String.class)
.build();
}
private StreamMessageListenerContainer.StreamReadRequest<String> getReadRequest(
Consumer consumer, StreamOffset<String> streamOffset) {
return StreamMessageListenerContainer.StreamReadRequest
.builder(streamOffset)
.consumer(consumer)
.autoAcknowledge(false)
.cancelOnError((err) -> false) // do not stop consuming after error
.build();
}
private <T> StreamListener listener(RedisConnectionFactory redisConnectionFactory, String channel, Consumer consumer, AtomicInteger counter) {
return message -> {
try {
System.out.println("Acknowledging message: " + message.getId());
StringRedisTemplate stringRedisTemplate = new StringRedisTemplate(redisConnectionFactory);
stringRedisTemplate.opsForStream().acknowledge(channel, consumer.getGroup(), message.getId());
System.out.println("RECEIVED " + consumer + " " + message);
counter.incrementAndGet();
} catch(Exception e) {
e.printStackTrace();
}
};
}
from redisson.
@mrniko
It normally occurs when there are no messages in stream for quite some time say 7-8 hours and then 1 message is pushed.
It is quite intermittent in nature.
Can you suggest me on this please -
Also, please suggest if this is fine to use or we should use conventional way using a while(true) loop and fetch/poll using RStream continuously.
I was wondering if mixing Redisson with spring library can cause this?
It would be helpful if you could send some code snippet to read continuously using RStream.readGroup() approach.
from redisson.
I was wondering if mixing Redisson with spring library can cause this?
No.
It would be helpful if you could send some code snippet to read continuously using RStream.readGroup() approach.
Your code should work fine.
Are there any exceptions in logs prior issue?
from redisson.
I was wondering if mixing Redisson with spring library can cause this?
No.
It would be helpful if you could send some code snippet to read continuously using RStream.readGroup() approach.
Your code should work fine.
Are there any exceptions in logs prior issue?
No exceptions, just some commands were (Completed exceptionally) in the promise and then the ConnectinWatchdog entries which I shared earlier.
from redisson.
I was able to reproduce the issue. Can you please help here.
Steps-
- Use above code which uses StreamMessageListenerContainer for consuming messages.
- Setup sentinel
- Start the service and publish message to stream, we will get that message.
- Now, failover sentinel ( lets say earlier master was node1:6379, now master should be node2:6379)
- Publish message to stream.
- Now we never get this message.
Ot works if we stop the StreamMessageListenerContainer and then call invoke it from start again like below-
RedisConnectionFactory redisConnectionFactory = new RedissonConnectionFactory(redisson);
StreamMessageListenerContainer listenerContainer = StreamMessageListenerContainer.create(redisConnectionFactory, getOptions());
Consumer consumer = Consumer.from("group", "consumer1");
StreamOffset<String> streamOffset = StreamOffset.create("test", ReadOffset.from(">"));
String channel = "test";
AtomicInteger counter = new AtomicInteger();
Subscription subscription = listenerContainer.register(getReadRequest(consumer, streamOffset),
listener(redisConnectionFactory, channel, consumer, counter));
StringRedisTemplate t1 = new StringRedisTemplate(redisConnectionFactory);
t1.opsForStream().createGroup("test", "group");
listenerContainer.start();
from redisson.
Thanks for the test. It helped a lot.
Can you try attached version?
redisson-3.29.1-SNAPSHOT.jar.zip
from redisson.
Hi @mrniko
This works, but we cannot use this version in production.
Can you let us know on the planned release version and date for this fix?
Also, There is 1 more issue I found in RedissonConnectionFactory class.
public RedisSentinelConnection getSentinelConnection() {
if (!this.redisson.getConfig().isSentinelConfig()) {
throw new InvalidDataAccessResourceUsageException("Redisson is not in Sentinel mode");
} else {
SentinelConnectionManager manager = (SentinelConnectionManager)((Redisson)this.redisson).getConnectionManager();
Iterator var2 = manager.getSentinels().iterator();
while(var2.hasNext()) {
RedisClient client = (RedisClient)var2.next();
org.redisson.client.RedisConnection connection = client.connect();
try {
String res = (String)connection.sync(RedisCommands.PING, new Object[0]);
if ("pong".equalsIgnoreCase(res)) {
return new RedissonSentinelConnection(connection);
}
} catch (Exception var6) {
log.warn("Can't connect to " + client, var6);
connection.closeAsync();
}
}
throw new InvalidDataAccessResourceUsageException("Sentinels are not found");
}
}
Issue-
- It always tries to connect to 1 sentinel only which is defined in config based on iterator.If that sentinel process is down, it does not try connecting to others and then returning one.
- The try should be placed right after while block ideally like below.
public RedisSentinelConnection getSentinelConnection() {
if (!this.redisson.getConfig().isSentinelConfig()) {
throw new InvalidDataAccessResourceUsageException("Redisson is not in Sentinel mode");
} else {
SentinelConnectionManager manager = (SentinelConnectionManager)((Redisson)this.redisson).getConnectionManager();
Iterator var2 = manager.getSentinels().iterator();
while(var2.hasNext()) {
try {
RedisClient client = (RedisClient)var2.next();
org.redisson.client.RedisConnection connection = client.connect();
String res = (String)connection.sync(RedisCommands.PING, new Object[0]);
if ("pong".equalsIgnoreCase(res)) {
return new RedissonSentinelConnection(connection);
}
} catch (Exception var6) {
log.warn("Can't connect to " + client, var6);
connection.closeAsync();
}
}
throw new InvalidDataAccessResourceUsageException("Sentinels are not found");
}
}
from redisson.
Hello @inampar,
This works, but we cannot use this version in production.
Can you let us know on the planned release version and date for this fix?
Fixed in 1f86d11 a82bd11 commits included in 3.30.0 version. Which has already bean released.
The try should be placed right after while block ideally like below.
Good point.
from redisson.
Fixed
from redisson.
Related Issues (20)
- In a multi-master multi-slave cluster setup, when a master node goes down, why does the client attempt to connect to the original master node?
- Readmode setting invalid HOT 1
- The RListMultimap key was deleted by mistake HOT 1
- Missing entries in reflect-config.json HOT 1
- Keyspace notifications should be listened only on master nodes HOT 1
- The cluster switches the master/slave nodes, causing an issue HOT 1
- Unable to add/change master node HOT 1
- eval when value is json str ,not execute HOT 1
- Hereβs the issue we encountered with `redisson-3.11.5.jar` HOT 10
- Aws redisCache
- AWS RedisCache Failover requests HOT 5
- Redisson getRateLimiter use expireIfSet get Error HOT 8
- Subscribe timeout: (62000ms). Increase 'subscriptionsPerConnection' and/or 'subscriptionConnectionPoolSize' parameters. HOT 1
- RLocalCachedMap local cache use? HOT 4
- Redisson json set doesn't accept "=" as json string key HOT 3
- LEAK: ByteBuf.release() was not called before it's garbage-collected HOT 1
- fix
- redisson netty consumes high cpu HOT 3
- ((RedisCluster) redissonClient.getRedisNodes(redisNodes)).getMasters() does not appear to track changes to cluster topology, should it?
- RRateLimiter.trySetRate does not reset all rate limiter state despite documentation
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from redisson.