Comments (5)
An update on the above, I restarted this again but this time the indices make sense. The consumer picks up the latest produced index.
[2018-08-03 00:38:32,485] INFO single-cluster-monitor/Current index=94, nextIndex=-1 for partition=4 (com.linkedin.kmf.services.ConsumeService)
[2018-08-03 00:38:32,486] INFO single-cluster-monitor/Current index=94, nextIndex=-1 for partition=7 (com.linkedin.kmf.services.ConsumeService)
[2018-08-03 00:38:32,486] INFO single-cluster-monitor/Current index=94, nextIndex=-1 for partition=0 (com.linkedin.kmf.services.ConsumeService)
[2018-08-03 00:38:32,487] INFO single-cluster-monitor/Current index=94, nextIndex=-1 for partition=6 (com.linkedin.kmf.services.ConsumeService)
[2018-08-03 00:38:32,487] INFO single-cluster-monitor/Current index=94, nextIndex=-1 for partition=12 (com.linkedin.kmf.services.ConsumeService)
[2018-08-03 00:38:32,488] INFO single-cluster-monitor/Current index=94, nextIndex=-1 for partition=8 (com.linkedin.kmf.services.ConsumeService)
[2018-08-03 00:38:32,488] INFO single-cluster-monitor/Current index=94, nextIndex=-1 for partition=15 (com.linkedin.kmf.services.ConsumeService)
[2018-08-03 00:38:32,488] INFO single-cluster-monitor/Current index=94, nextIndex=-1 for partition=14 (com.linkedin.kmf.services.ConsumeService)
[2018-08-03 00:38:32,490] INFO single-cluster-monitor/Current index=94, nextIndex=-1 for partition=3 (com.linkedin.kmf.services.ConsumeService)
[2018-08-03 00:38:32,491] INFO single-cluster-monitor/Current index=94, nextIndex=-1 for partition=11 (com.linkedin.kmf.services.ConsumeService)
[2018-08-03 00:38:32,493] INFO single-cluster-monitor/Current index=94, nextIndex=-1 for partition=5 (com.linkedin.kmf.services.ConsumeService)
[2018-08-03 00:38:32,493] INFO single-cluster-monitor/Current index=94, nextIndex=-1 for partition=13 (com.linkedin.kmf.services.ConsumeService)
[2018-08-03 00:38:32,500] INFO single-cluster-monitor/Current index=94, nextIndex=-1 for partition=1 (com.linkedin.kmf.services.ConsumeService)
[2018-08-03 00:38:32,500] INFO single-cluster-monitor/Current index=94, nextIndex=-1 for partition=9 (com.linkedin.kmf.services.ConsumeService)
[2018-08-03 00:38:32,513] INFO single-cluster-monitor/Current index=94, nextIndex=-1 for partition=2 (com.linkedin.kmf.services.ConsumeService)
[2018-08-03 00:38:32,514] INFO single-cluster-monitor/Current index=94, nextIndex=-1 for partition=10 (com.linkedin.kmf.services.ConsumeService)
[2018-08-03 00:38:32,586] INFO single-cluster-monitor/Current index=95, nextIndex=95 for partition=7 (com.linkedin.kmf.services.ConsumeService)
[2018-08-03 00:38:32,587] INFO single-cluster-monitor/Current index=95, nextIndex=95 for partition=4 (com.linkedin.kmf.services.ConsumeService)
[2018-08-03 00:38:32,587] INFO single-cluster-monitor/Current index=95, nextIndex=95 for partition=0 (com.linkedin.kmf.services.ConsumeService)
[2018-08-03 00:38:32,588] INFO single-cluster-monitor/Current index=95, nextIndex=95 for partition=6 (com.linkedin.kmf.services.ConsumeService)
[2018-08-03 00:38:32,588] INFO single-cluster-monitor/Current index=95, nextIndex=95 for partition=15 (com.linkedin.kmf.services.ConsumeService)
from kafka-monitor.
Hey @jlisam,
Thanks for raising the issue. Here is the expected behavior of after kafka-monitor is restarted:
-
For any given partition, ProduceService will send a message with index = 0. Say the offset of this produced message is
offset1
. Then the producer will send messages with index sequence (offset1, offset1+1, offset1+2,...). So the sequence of indexes of messages from ProduceService is(0, offset1, offset1 + 1, ...)
-
For any given partition, ConsumeService starts consuming from the log end offset, which means that ConsumeService will not consume any message that was produced before it is started. Say the sequence of indexes of the messages it has consumed is (m1, m2, m3, ...). ConsumeService will discard the first offset
m2
and start from the second offset. It expects offsets starting from the second offset to be consecutive. In other words, it expect sequence of indexes to be (m1, m2, m2 + 1, m2 + 2, ..). If there is a message whose index is smaller than the previous index (not including the first message), then this is considered a message duplication.
So it looks like the logic is correct for kafka-monitor restart. I think we can prove that, if there is no message loss and message duplication in the Kafka side, ConsumeService should always see consecutively incrementing indexes, right?
Given that you actually observed something weird, there may be problem somewhere. I currently don't have an explanation for that.
from kafka-monitor.
thanks @lindong28 for the prompt response! I will try to track this. So far I can't reproduce this locally but when my instance is running in kubernetes, I almost always hit that case.
from kafka-monitor.
@jlisam Thanks for reporting this issue! Since LinkedIn Kafka team does not use Kubernetes, it is hard for us to investigate this issue if this issue only happens when Kafka Monitor is used with Kubernetes. We will try to fix the issue if you or other use can provide more specific information on what is going wrong with Kafka Monitor to help us debug it.
from kafka-monitor.
I will close this issue. Please feel free to re-open if there is more actionable information for us.
from kafka-monitor.
Related Issues (20)
- Jetty Service exception HOT 3
- GUI Monitor HOT 8
- Roadmap and project faith HOT 3
- When apps stop fully running there's no mechanism to either restart them or abort the monitor HOT 1
- StatsD reporting of availability metrics is broken
- ClassNotFoundException: com.linkedin.kmf.services.ProduceService HOT 1
- ERROR lassNotFoundException: com.linkedin.kmf.apps.SingleClusterMonitor HOT 2
- Support MM2 HOT 1
- Producer Restarting{Feature Optimisation] HOT 2
- bump avro-util to 0.2.118 do to the critical CVE-2022-42889 HOT 2
- Xinfra monitor fails with ClassCastException HOT 3
- Remove zookeeper dependency HOT 3
- Ensure proper retry and backoff for newly created monitor topics HOT 1
- Hard to detect one Bad Broker.
- Unable to list topics HOT 1
- Add validator for all required configs HOT 1
- [QUESTION] Are there plans to publish the kafka-monitor on the docker hub? HOT 1
- OffsetCommitService does not work with SSL cluster HOT 1
- CVE-2023-44981 HOT 1
- Need help to enable compression for data produced by kafka-monitor HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafka-monitor.