linkedin / kafka-monitor

Xinfra Monitor monitors the availability of Kafka clusters by producing synthetic workloads using end-to-end pipelines to obtain derived vital statistics - E2E latency, service produce/consume availability, offsets commit availability & latency, message loss rate and more.

Xinfra Monitor

Xinfra Monitor (formerly Kafka Monitor) is a framework to implement and execute long-running kafka system tests in a real cluster. It complements Kafka’s existing system tests by capturing potential bugs or regressions that are only likely to occur after prolonged period of time or with low probability. Moreover, it allows you to monitor Kafka cluster using end-to-end pipelines to obtain a number of derived vital stats such as

1. End-to-end latency
2. Service availability
3. Produce and Consume availability
4. Consumer offset commit availability
5. Consumer offset commit latency
6. Kafka message loss rate
7. And many, many more.

You can easily deploy Xinfra Monitor to test and monitor your Kafka cluster without requiring any change to your application.

Xinfra Monitor can automatically create the monitor topic with the specified config and increase partition count of the monitor topic to ensure partition# >= broker#. It can also reassign partition and trigger preferred leader election to ensure that each broker acts as leader of at least one partition of the monitor topic. This allows Xinfra Monitor to detect performance issue on every broker without requiring users to manually manage the partition assignment of the monitor topic.

Xinfra Monitor is used in conjunction with different middle-layer services such as li-apache-kafka-clients in order to monitor single clusters, pipeline desination clusters, and other types of clusters as done in Linkedin engineering for real-time cluster healthchecks.

These are some of the metrics emitted from a Xinfra Monitor instance.

kmf:type=kafka-monitor:offline-runnable-count
kmf.services:type=produce-service,name=*:produce-availability-avg
kmf.services:type=consume-service,name=*:consume-availability-avg
kmf.services:type=produce-service,name=*:records-produced-total
kmf.services:type=consume-service,name=*:records-consumed-total
kmf.services:type=produce-service,name=*:records-produced-rate
kmf.services:type=produce-service,name=*:produce-error-rate
kmf.services:type=consume-service,name=*:consume-error-rate
kmf.services:type=consume-service,name=*:records-lost-total
kmf.services:type=consume-service,name=*:records-lost-rate
kmf.services:type=consume-service,name=*:records-duplicated-total
kmf.services:type=consume-service,name=*:records-delay-ms-avg
kmf.services:type=commit-availability-service,name=*:offsets-committed-avg
kmf.services:type=commit-availability-service,name=*:offsets-committed-total
kmf.services:type=commit-availability-service,name=*:failed-commit-offsets-avg
kmf.services:type=commit-availability-service,name=*:failed-commit-offsets-total
kmf.services:type=commit-latency-service,name=*:commit-offset-latency-ms-avg
kmf.services:type=commit-latency-service,name=*:commit-offset-latency-ms-max
kmf.services:type=commit-latency-service,name=*:commit-offset-latency-ms-99th
kmf.services:type=commit-latency-service,name=*:commit-offset-latency-ms-999th
kmf.services:type=commit-latency-service,name=*:commit-offset-latency-ms-9999th

Getting Started

Prerequisites

Xinfra Monitor requires Gradle 2.0 or higher. Java 7 should be used for building in order to support both Java 7 and Java 8 at runtime.

Xinfra Monitor supports Apache Kafka 0.8 to 2.0:

• Use branch 0.8.2.2 to work with Apache Kafka 0.8
• Use branch 0.9.0.1 to work with Apache Kafka 0.9
• Use branch 0.10.2.1 to work with Apache Kafka 0.10
• Use branch 0.11.x to work with Apache Kafka 0.11
• Use branch 1.0.x to work with Apache Kafka 1.0
• Use branch 1.1.x to work with Apache Kafka 1.1
• Use master branch to work with Apache Kafka 2.0

Configuration Tips

1. We advise advanced users to run Xinfra Monitor with ./bin/xinfra-monitor-start.sh config/xinfra-monitor.properties. The default xinfra-monitor.properties in the repo provides an simple example of how to monitor a single cluster. You probably need to change the value of zookeeper.connect and bootstrap.servers to point to your cluster.


2. The full list of configs and their documentation can be found in the code of Config class for respective service, e.g. ProduceServiceConfig.java and ConsumeServiceConfig.java.


3. You can specify multiple SingleClusterMonitor in the xinfra-monitor.properties to monitor multiple Kafka clusters in one Xinfra Monitor process. As another advanced use-case, you can point ProduceService and ConsumeService to two different Kafka clusters that are connected by MirrorMaker to monitor their end-to-end latency.


4. Xinfra Monitor by default will automatically create the monitor topic based on the e.g. topic-management.replicationFactor and topic-management.partitionsToBrokersRatio specified in the config. replicationFactor is 1 by default and you probably want to change it to the same replication factor as used for your existing topics. You can disable auto topic creation by setting produce.topic.topicCreationEnabled to false.


5. Xinfra Monitor can automatically increase partition count of the monitor topic to ensure partition# >= broker#. It can also reassign partition and trigger preferred leader election to ensure that each broker acts as leader of at least one partition of the monitor topic. To use this feature, use either EndToEndTest or TopicManagementService in the properties file.


6. When using Secure Sockets Layer (SSL) or any non-plaintext security protocol for AdminClient, please configure the following entries in the single-cluster-monitor props, produce.producer.props, as well as consume.consumer.props. https://docs.confluent.io/current/installation/configuration/admin-configs.html
   1. ssl.key.password
   2. ssl.keystore.location
   3. ssl.keystore.password
   4. ssl.truststore.location
   5. ssl.truststore.password

Build Xinfra Monitor

$ git clone https://github.com/linkedin/kafka-monitor.git
$ cd kafka-monitor 
$ ./gradlew jar

Start XinfraMonitor to run tests/services specified in the config file

$ ./bin/xinfra-monitor-start.sh config/xinfra-monitor.properties

Run Xinfra Monitor with arbitrary producer/consumer configuration (e.g. SASL enabled client)

Edit config/xinfra-monitor.properties to specify custom configurations for producer in the key/value map produce.producer.props in config/xinfra-monitor.properties. Similarly specify configurations for consumer as well. The documentation for producer and consumer in the key/value maps can be found in the Apache Kafka wiki.

$ ./bin/xinfra-monitor-start.sh config/xinfra-monitor.properties

Run SingleClusterMonitor app to monitor kafka cluster

Metrics produce-availability-avg and consume-availability-avg demonstrate whether messages can be properly produced to and consumed from this cluster. See Service Overview wiki for how these metrics are derived.

$ ./bin/single-cluster-monitor.sh --topic test --broker-list localhost:9092 --zookeeper localhost:2181

Run MultiClusterMonitor app to monitor a pipeline of Kafka clusters connected by MirrorMaker

Edit config/multi-cluster-monitor.properties to specify the right broker and zookeeper url as suggested by the comment in the properties file

Metrics produce-availability-avg and consume-availability-avg demonstrate whether messages can be properly produced to the source cluster and consumed from the destination cluster. See config/multi-cluster-monitor.properties for the full jmx path for these metrics.

$ ./bin/xinfra-monitor-start.sh config/multi-cluster-monitor.properties

Run checkstyle on the java code

./gradlew checkstyleMain checkstyleTest

Build IDE project

./gradlew idea
./gradlew eclipse

Wiki

• Motivation
• Design Overview
• Service and App Overview
• Future Work
• Application Configuration

kafka-monitor's Issues

Issue in running kafkamonitor on windows

When i am running after build , i am getting the bellow error

Error: Could not find or load main class com.linkedin.kmf.core.KafkaMonitor

publish on docker hub?

Is it possible to get a working image published on docker hub? Tried building one myself, had to modify the Dockerfile at docker/Dockerfile, the logic works (consumer / producer), but the UI and jolokia are not accessible. Would be great to get something to plug and play readily available.

feature request : remove hardcoded jolokia binding from JolokiaService and webapp

Hi,
Can I suggest to remove those hardcoded port and url from the webapp to JolokiaService ? this makes impossible to use the dashboard using a kubernetes cluster, (because ingress controller doesn't have that 8778 port, and opening ports on external ip address ( where there is the dashboard) is a no-no )

also even without kubernetes, frontend and backend which communicate using semi-hardcoded address ( the same address of frontend must be exposing the backend ) and hardcoded port is a headache to manage

thank you everyone,
Francesco

Produce more than ~1000 records per second?

Hi,

I am trying to test what throughput I can expect with my currently available hardware and configured numbers of brokers / partitions / replication factor, etc.

However, using this monitor, it seems like I can only lower "produce.record.delay.ms" to 1, which obviously means the most that I am able to produce/consume is 1000 records / second.

Is there any way of improving this? I tried changing the value of "produce.record.delay.ms" to 0, and the entire thing stops working (all values 0.0). Can I spin up one more than one producer, for example?

Thank you for your help.

Duplicated messages when kafka-monitor is restarted

I had to do a redeploy of kafka-monitor and I started seeing duplicated messages through the metrics. Now I understand how kafka-monitor defines something as duplicated through the producing and consuming of the indices; however I was wondering what was the expected behavior of a restart? I do notice that it's trying to catch up with the nextIndex:

[2018-08-02 23:30:34,313] INFO single-cluster-monitor/Current index=197338, nextIndex=686555 for partition=9 (com.linkedin.kmf.services.ConsumeService)
[2018-08-02 23:30:34,414] INFO single-cluster-monitor/Current index=197339, nextIndex=686555 for partition=9 (com.linkedin.kmf.services.ConsumeService)
[2018-08-02 23:30:34,515] INFO single-cluster-monitor/Current index=197340, nextIndex=686555 for partition=9 (com.linkedin.kmf.services.ConsumeService)
[2018-08-02 23:30:34,615] INFO single-cluster-monitor/Current index=197341, nextIndex=686555 for partition=9 (com.linkedin.kmf.services.ConsumeService)
[2018-08-02 23:30:34,716] INFO single-cluster-monitor/Current index=197342, nextIndex=686555 for partition=9 (com.linkedin.kmf.services.ConsumeService)

It's been taking a while so I am wondering if this expected? I haven't dug really deep into the code but I noticed this https://github.com/linkedin/kafka-monitor/blob/master/src/main/java/com/linkedin/kmf/services/ProduceService.java#L167:L177. However, I don't really understand how that helps with the restart.

logs from the producer:

{"topic":"hydro-monitor-topic","time":1533253727144,"index":207774,"producerId":"kmf-producer","content":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}
{"topic":"hydro-monitor-topic","time":1533253727144,"index":207753,"producerId":"kmf-producer","content":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}
{"topic":"hydro-monitor-topic","time":1533253727097,"index":208174,"producerId":"kmf-producer","content":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}
{"topic":"hydro-monitor-topic","time":1533253727198,"index":208175,"producerId":"kmf-producer","content":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}
{"topic":"hydro-monitor-topic","time":1533253727097,"index":208171,"producerId":"kmf-producer","content":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}
{"topic":"hydro-monitor-topic","time":1533253727198,"index":208172,"producerId":"kmf-producer","content":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}

To me it looks like the producer needs to figure out a way to get the last index produced and start from there in a restart (correct me if I am wrong). Let me know if there is anything else you would like me to provide.

Thank you 🙇

Facing Java Heap Error

Hi All,

I am getting error when run Kafka-Monitor .

Please refer the attached

screenshot.

Consume-service not showing any consumers

This is more a question than a bug report. I have kafka-monitor running and it's passing messages, based on it's stats it's running as expected. However, both kafka-manager and Burrow show the consumer list as empty. Do the consume-service consumers not persist?

records-produced-rate not increasing on increasing number of brokers

With single broker in the cluster, I am seeing records-produced-rate to be approx ~137.
I modified kafka-monitor.properties config file and added one more broker in bootstrap.servers.
I am expecting the records-produced-rate to increase because of messages flowing on 2 brokers now, but the rate is same. Tried this with seven brokers as well.
Is this an expected behaviour or is there something wrong with my configurations.
Rest of the configuration is default only.

"zookeeper.connect": "zk1:2181,zk2:2181",
"bootstrap.servers": "broker1:9092,broker2:9092",

Monitor custom metrics

Hi

I'm using kafka 0.8.2.1 and I'm trying to add additional metrics to the index.html + config lists.
I can see that the UI show the expected metrics but in the logs I dont see the lines as well as the default metrics, does this project support that?

Thanks
D.

Adding tests in kafka-monitor.properties

Hello,

I am a little confused on how exactly to add tests in the kafka-monitor.properties file. For example, how would I add the test to see if MirrorMaker is working?

Furthermore, is there functionality to notify a user when a particular service like zookeeper is not running or crashed?

Connecting to Kafka over SSL/TLS

Does kafka-monitor support connecting over an SSL Listener in Kafka?
I'd like to not have to open a plaintext listener for just kafka-monitor.

If so is what does the configuration look like? Searching for "SSL"/"TLS" didn't bring up much in issues/prs/the wiki.

Dose it support kafka 0.10？

As title.Thx~

Is there any way to run kafka-producer with Kafka 0.8?

We are using kafka 0.8.2 version. I am getting the following error while starting kafka monitor.

:~$ tailf kafka-monitor/logs/kafka-client.log
[2016-06-06 10:18:02,222] WARN block.on.buffer.full config is deprecated and will be removed soon. Please use max.block.ms (org.apache.kafka.clients.producer.KafkaProducer)
[2016-06-06 10:18:02,764] ERROR Uncaught error in kafka producer I/O thread: (org.apache.kafka.clients.producer.internals.Sender)
org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'throttle_time_ms': java.nio.BufferUnderflowException
at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:71)
at org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:439)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:265)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128)
at java.lang.Thread.run(Thread.java:745)

kafka-monitor jdk7 compile error

According the the project README and build it with JDK7 failed, it looks like some JDK8 syntax is used.
The README is misleading, shall we update it?

Kafka Monitor requires Gradle 2.0 or higher.

Java 7 should be used for building in order to support both Java 7 and Java 8 at runtime.

/opt/Development/github_repo/kafka-monitor/services/src/main/java/com/linkedin/kmf/services/ProduceService.java:176: error: ')' expected
(config, now) -> {
^

when use proxy go to network run gradlew jar error

./gradlew jar
Downloading https://services.gradle.org/distributions/gradle-3.2.1-bin.zip

Exception in thread "main" java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
at org.gradle.wrapper.Download.downloadInternal(Download.java:58)
at org.gradle.wrapper.Download.download(Download.java:44)
at org.gradle.wrapper.Install$1.call(Install.java:61)
at org.gradle.wrapper.Install$1.call(Install.java:48)
at org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAccessManager.java:65)
at org.gradle.wrapper.Install.createDist(Install.java:48)
at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:128)
at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:61)
i can down use wget to https://services.gradle.org/distributions/gradle-3.2.1-bin.zip file.

Support Kafka 2.0

As far as I can see, this project does not support Kafka 2.0. Any eta when we can expect Kafka 2.0 support?

How to connect to kafka-monitor by jconsolejmx

On a monitor-kafka machine, I can directly select a local Process connection via jconsole, but now that I want a Remote connection, how should I fill in the Remote Process.
I tried to fill in with "service:jmx:rmi:///jndi/rmi://192.168.75.130:8778/jmxrmi",but conection failed.

Page not found 404 error

I am trying to run the monitor tool. But getting 404 on the browser by hitting the localhost:8080/index.html

Make Jolokia port/url configurable ( in JolokiaService and in webapp/index.html )

Hi,

i had a requirement to deploy kafka-monitor in kubernetes, and expose JettyService ( specifically, the small js ui that draws a couple graphs ) and JolokiaService over an ingress.

In the current master branch this is rather difficult because index.html hardcodes the jolokia port/url as http:// + location.host.split(':')[0] + :8778/jolokia , with the port also hardcoded in JolokiaService.

I will be sending a PR for your review to discuss/potentially fix these issues.

thanks!
Andras

when use proxy go to network run gradlew jar error

./gradlew jar
[root@docker kafka-monitor]# ./gradlew jar
Downloading https://services.gradle.org/distributions/gradle-3.2.1-bin.zip

Exception in thread "main" java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
at org.gradle.wrapper.Download.downloadInternal(Download.java:58)
at org.gradle.wrapper.Download.download(Download.java:44)
at org.gradle.wrapper.Install$1.call(Install.java:61)
at org.gradle.wrapper.Install$1.call(Install.java:48)
at org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAccessManager.java:65)
at org.gradle.wrapper.Install.createDist(Install.java:48)
at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:128)
at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:61)
[root@docker kafka-monitor]#

how can i do ?

Can't start kafka-monitor

While I start KafkaMonitor to run tests/services specified in the config file, it shown this info:

: command not found-start.sh: line 9:
: command not found-start.sh: line 11:
: command not found-start.sh: line 13:
(No such file or directory)ava.io.FileNotFoundException: config/kafka-monitor.properties
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.(FileInputStream.java:138)
at java.io.FileInputStream.(FileInputStream.java:93)
at java.io.FileReader.(FileReader.java:58)
at com.linkedin.kmf.core.KafkaMonitor.main(KafkaMonitor.java:118)

but the file of kafka-monitor.properties is exist, and the path is correct.
Please help me sovle the problem.
Thanks.

setup sasl_ssl authentication

Hi,

How can I setup sasl_ssl authentication in Kafka-monitor?

where can I set and how?

Typo in Project Description

Hi! Just reporting this since the project description can only be modified by a repository owner.

Current Description: "Monitor the availability of Kakfa clusters with generated messages."
Proposed Description: "Monitor the availability of Kafka clusters with generated messages."

Thank you!

why set multiple times with same partitionNum value in ProduceService

Hi,
New to this project, after viewing the ProduceService, a little puzzle stuck me that:
why _partitionNum set same value partitionNum every turn in the partition loop?
this line

`listener_security_protocol_map` not taken into account

Hello,

While trying to test kafka-monitor with one of our clusters, I encountered the following error:

[2017-07-31 13:54:23,834] ERROR single-cluster-monitor/MultiClusterTopicManagementService will stop due to error. (com.linkedin.kmf.services.MultiClusterTopicManagementService)
kafka.common.KafkaException: Failed to parse the broker info from zookeeper: {"listener_security_protocol_map":{"INT_PLAINTEXT":"PLAINTEXT","EXT_PLAINTEXT":"PLAINTEXT"},"endpoints":["INT_PLAINTEXT://192.168.133.180:9092","EXT_PLAINTEXT://192.168.130.104:9093"],"j
mx_port":9990,"host":"192.168.133.180","timestamp":"1501251189444","port":9092,"version":4}

[...]

Caused by: java.lang.IllegalArgumentException: No enum constant org.apache.kafka.common.protocol.SecurityProtocol.INT_PLAINTEXT

[...]

Testing was done with the current head at the time of writing (312abe0).

collect metrics in logs

Is it possible to collect the metrics into some csv or json format logs?
Would be great to record these metrics by some intervals to get a better sense of cluster health.
Trying to build it myself.

No branch support kafka version 1.0.1

now no branch support kafka version 1.0.1

Email notifications on certain events

Is there a possibility of having the monitor send email when certain events occur?

Example scenario would be to send an email when produce-availability-avg < 0.5

No access restrictor found, access to any MBean is allowed

hi.
When I run the tools to monitor my own topic,Why always remind me "No access restrictor found, access to any MBean is allowed"？How can i solve it?

the properties file name is misspelt, kafak-monitor.properties this should be kafka-monitor.properties

Configuration Tips

We advise advanced users to run Kafka Monitor with ./bin/kafka-monitor-start.sh config/kafka-monitor.properties. The default kafak-monitor.properties in the repo provides an simple example of how to monitor a single cluster. You probably need to change the value of zookeeper.connect and bootstrap.servers to point to your cluster.

Update Wiki Typo

com.linkedin.kmf.producer.KMBaseConsumer should be com.linkedin.kmf.consumer.KMBaseConsumer in https://github.com/linkedin/kafka-monitor/wiki/Service-Design

Kafka monitor failing due to missing topic with topic creation enabled

I'm just getting started with kafka-monitor, and my first run is failing with the following exception:

Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at com.linkedin.kmf.core.KafkaMonitor.<init>(KafkaMonitor.java:46)
        at com.linkedin.kmf.core.KafkaMonitor.main(KafkaMonitor.java:132)
Caused by: java.lang.RuntimeException: Can not find valid partition number for topic kafka-monitor-topic. Please verify that the topic has been created. Ideally the partition number should be a multiple of number of brokers in the cluster.
        at com.linkedin.kmf.services.ProduceService.<init>(ProduceService.java:75)
        at com.linkedin.kmf.tests.BasicEndToEndTest.<init>(BasicEndToEndTest.java:54)
        ... 6 more

This error is technically correct as I have not set up a topic and the target Kafka is a blank dev instance. However, I'd expect kafka-monitor to start as I have produce.topic.topicCreationEnabled set to true. For the config I'm running the example config in the repo with the zookeeper.connect and bootstrap.servers pointed at my dev Kafka setup. Rather than die with an exception I'd expect kafka-monitor to set up it's test topic.

Build issue while building kafka monitor.

While building this project i got below error
Gradle dependency :

dependencies {
compile 'net.sourceforge.argparse4j:argparse4j:0.7.0'
compile 'org.slf4j:slf4j-log4j12:1.7.6'
compile 'org.apache.avro:avro:1.8.1'
compile 'org.apache.kafka:kafka_2.11:1.0.0'
compile 'org.apache.kafka:kafka-clients:0.10.2.0'
compile 'org.testng:testng:6.8.8'
compile 'org.eclipse.jetty:jetty-server:8.1.14.v20131031'
compile 'org.json:json:20140107'
compile 'com.fasterxml.jackson.core:jackson-databind:2.7.1'
compile 'org.jolokia:jolokia-jvm:1.3.3'
compile 'net.savantly:graphite-client:1.1.0-RELEASE'
compile 'com.timgroup:java-statsd-client:3.1.0'

testCompile 'org.testng:testng:6.8.8'

}
Error :
:\BIGDATA\kafka-monitor-master\src\main\java\com\linkedin\kmf\services\GraphiteMetricsReporterService.java:20: error: package org.apache.commons.lang does not exist
import org.apache.commons.lang.StringUtils;
^
D:\BIGDATA\kafka-monitor-master\src\main\java\com\linkedin\kmf\services\StatsdMetricsReporterService.java:17: error: package org.apache.commons.lang does not exist
import org.apache.commons.lang.StringUtils;
^
D:\BIGDATA\kafka-monitor-master\src\main\java\com\linkedin\kmf\common\Utils.java:162: error: incompatible types: ByteArrayOutputStream cannot be converted to JsonGenerator
Encoder encoder = new JsonEncoder(DefaultTopicSchema.MESSAGE_V0, out);
^
D:\BIGDATA\kafka-monitor-master\src\main\java\com\linkedin\kmf\services\MultiClusterTopicManagementService.java:227: error: method addPartitions in class AdminUtils cannot be applied to given types;
AdminUtils.addPartitions(zkUtils, _topic, minPartitionNum, null, false, RackAwareMode.Enforced$.MODULE$);
^
required: ZkUtils,String,Map<Object,Seq>,Seq,int,Option<Map<Object,Seq>>,boolean
found: ZkUtils,String,int,,boolean,Enforced$
reason: actual and formal argument lists differ in length
D:\BIGDATA\kafka-monitor-master\src\main\java\com\linkedin\kmf\services\ProduceService.java:134: error: cannot find symbol
producerProps.put(ProducerConfig.BLOCK_ON_BUFFER_FULL_CONFIG, "true");
^
symbol: variable BLOCK_ON_BUFFER_FULL_CONFIG
location: class ProducerConfig
D:\BIGDATA\kafka-monitor-master\src\main\java\com\linkedin\kmf\services\ProduceService.java:244: error: constructor MetricName in class MetricName cannot be applied to given types;
double recordsProduced = _sensors.metrics.metrics().get(new MetricName("records-produced-rate-partition-" + partition, METRIC_GROUP_NAME, tags)).value();
^
required: String,String,String,Map<String,String>
found: String,String,Map<String,String>
reason: actual and formal argument lists differ in length
D:\BIGDATA\kafka-monitor-master\src\main\java\com\linkedin\kmf\services\ProduceService.java:245: error: constructor MetricName in class MetricName cannot be applied to given types;
double produceError = _sensors.metrics.metrics().get(new MetricName("produce-error-rate-partition-" + partition, METRIC_GROUP_NAME, tags)).value();
^
required: String,String,String,Map<String,String>
found: String,String,Map<String,String>
reason: actual and formal argument lists differ in length
D:\BIGDATA\kafka-monitor-master\src\main\java\com\linkedin\kmf\services\ConsumeService.java:268: error: constructor MetricName in class MetricName cannot be applied to given types;
double recordsConsumedRate = _sensors.metrics.metrics().get(new MetricName("records-consumed-rate", METRIC_GROUP_NAME, tags)).value();

Add a release tag

Please add a release tag. It will be easier to provide links to binary files, as well as release notes describing your changes :)

Thank you 🙏🙏🙏

all values 0.0

I am able to get the monitor running locally in my laptop against a Kafka cluster by changing only the "topic" and "zookeeper.connect"

but when I run the monitor with the same config on a remote box it does not produce and send any msgs, please help with this.

consumer timeout is not configurable

The NewConsumer class calls the kafka Consumer.poll() API with a timeout of Long.max so effectively trying to fetch messages from the broker will never time out. This means that in a multicluster design if the kafka-monitor topic stops being replicated from the active cluster to passive clusters kafka-monitor will stop producing useful statistics, but will not log any errors about latency growing without bound. It looks like the ConsumerService class atleast expects that a Consumer.recieve() implementation can throw an exception and logs errors. So if the consumer could be configured to time out errors could be logged in this case.

Open 192.168.89.2:8000/index.html ERROR 404

when changekafka-monitor.properties and run kafka-monitor-start.sh config/kafka-monitor.properties
I open ip:8000/index.html out put ERROR 404 thanks.
@lindong28

no data on graph

I don't see any data on graph at
myserver:8000/index.html

I have no issue to run curl for port 8778 at server or my system

curl localhost:8778/jolokia/read/kmf.services:type=produce-service,name=*/produce-availability-avg {"request":{"mbean":"kmf.services:name=*,type=produce-service","attribute":"produce-availability-avg","type":"read"},"value":{"kmf.services:name=single-cluster-monitor,type=produce-service":{"produce-availability-avg":0.0}},"timestamp":1499340973,"status":200}

All values are 0 for both the scripts and kafka-monitor.properties has correct Zk and bootstrap

executed first
./bin/kafka-monitor-start.sh config/kafka-monitor.properties

The output as below

[2017-07-06 11:53:28,744] INFO kmf:type=kafka-monitor:offline-runnable-count=0.0 kmf.services:name=single-cluster-monitor,type=produce-service:produce-availability-avg=0.0 kmf.services:name=single-cluster-monitor,type=consume-service:consume-availability-avg=0.0 kmf.services:name=single-cluster-monitor,type=produce-service:records-produced-total=0.0 kmf.services:name=single-cluster-monitor,type=consume-service:records-consumed-total=0.0 kmf.services:name=single-cluster-monitor,type=consume-service:records-lost-total=0.0 kmf.services:name=single-cluster-monitor,type=consume-service:records-duplicated-total=0.0 kmf.services:name=single-cluster-monitor,type=consume-service:records-delay-ms-avg=0.0 kmf.services:name=single-cluster-monitor,type=produce-service:records-produced-rate=0.0 kmf.services:name=single-cluster-monitor,type=produce-service:produce-error-rate=0.0 kmf.services:name=single-cluster-monitor,type=consume-service:consume-error-rate=0.0 (com.linkedin.kmf.services.DefaultMetricsReporterService)

and then
./bin/single-cluster-monitor.sh --topic test --broker-list myborker:9092 --zookeeper myZK:2181

and see one exception

Exception in thread "main" java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) [2017-07-06 12:14:49,816] DEBUG Reading reply sessionid:0x15cf2d02ad9649d, packet:: clientPath:null serverPath:null finished:false header:: 1,3 replyHeader:: 1,64424744367,0 request:: '/brokers/topics/kafka_monitor_1,F response:: s{64424741794,64424741794,1499338106498,1499338106498,0,1,0,0,76,1,64424741797} (org.apache.zookeeper.ClientCnxn) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at sun.net.httpserver.ServerImpl.<init>(ServerImpl.java:100) at sun.net.httpserver.HttpServerImpl.<init>(HttpServerImpl.java:50) at sun.net.httpserver.DefaultHttpServerProvider.createHttpServer(DefaultHttpServerProvider.java:35) at com.sun.net.httpserver.HttpServer.create(HttpServer.java:130) at org.jolokia.jvmagent.JolokiaServer.createHttpServer(JolokiaServer.java:243) at org.jolokia.jvmagent.JolokiaServer.init(JolokiaServer.java:171) at org.jolokia.jvmagent.JolokiaServer.<init>(JolokiaServer.java:81) at com.linkedin.kmf.services.JolokiaService.<init>(JolokiaService.java:32) at com.linkedin.kmf.apps.SingleClusterMonitor.main(SingleClusterMonitor.java:297)

about kafka0.10

does this tool support kafka0.10 version？

ERROR : org.apache.kafka.common.errors.InvalidPartitionsException: number of partitions must be larger than 0

Hello,

When i run single-cluster-monitor.sh, i have the following error...

What can explain this error ?

For information, i use apache 1.0 version.

thanks

[2017-12-27 09:36:49,157] INFO Session: 0x1607f0c633100a4 closed (org.apache.zookeeper.ZooKeeper)
[2017-12-27 09:36:49,158] ERROR single-cluster-monitor/MultiClusterTopicManagementService will stop due to error. (com.linkedin.kmf.services.MultiClusterTopicManagementService)
org.apache.kafka.common.errors.InvalidPartitionsException: number of partitions must be larger than 0

Graph not working

I am able to run the test but the graphs are not ploting no error in the logs.

ClassCastException when running Kafka Monitor

With the latest master, I get the following exception:

./bin/single-cluster-monitor.sh --topic kafka-monitor-test --broker-list localhost:9092 --zookeeper localhost:2181
[2017-03-26 13:35:59,032] INFO ProduceServiceConfig values:
	bootstrap.servers = localhost:9092
	produce.partitioner.class = com.linkedin.kmf.partitioner.NewKMPartitioner
	produce.producer.class = com.linkedin.kmf.producer.NewProducer
	produce.producer.id = kmf-producer
	produce.record.delay.ms = 100
	produce.record.size.byte = 100
	produce.sync = true
	produce.thread.num = 5
	topic = kafka-monitor-test
	zookeeper.connect = localhost:2181
 (com.linkedin.kmf.services.configs.ProduceServiceConfig)
Exception in thread "main" java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Class
	at org.apache.kafka.common.config.AbstractConfig.getClass(AbstractConfig.java:117)
	at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:200)
	at com.linkedin.kmf.services.ProduceService.<init>(ProduceService.java:84)
	at com.linkedin.kmf.apps.SingleClusterMonitor.<init>(SingleClusterMonitor.java:57)
	at com.linkedin.kmf.apps.SingleClusterMonitor.main(SingleClusterMonitor.java:263)

From what I can see, Kafka expects the value for the configuration key ProduceServiceConfig .PARTITIONER_CLASS_CONFIGto be of type class, when it actually is of type String.

Or am I missing something here?

Update: The issue seems to be related to 1ebe99c, the exception does not occur when using revision b13b43c.

Batch expiration logged at DEBUG level

When a message batch expires it's currently logged at the DEBUG level, and is not shown at the default INFO loglevel. I think this message should probably be WARN level at least as it shows the producer is failing to send messages.

[2016-11-08 19:34:31,787] DEBUG end-to-end failed to send message (com.linkedin.kmf.services.ProduceService)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Batch Expired
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
        at com.linkedin.kmf.producer.NewProducer.send(NewProducer.java:35)
        at com.linkedin.kmf.services.ProduceService$ProduceRunnable.run(ProduceService.java:221)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.common.errors.TimeoutException: Batch Expired

Can i monitor the latency in topics ?

Hi,
I am a newbie to this, wanted to check if there is a way if i could see the latency in a topic without producing or consuming using the Producer Service or Consumer Service i.e i want to monitor a topic with real messages flowing and not the ones generated from the Producer Service or the Consumer Service.

Is there a metric exposed for latency/availability of individual broker/partition?

I see there is latency for the whole cluster under monitoring, "The average latency of records from producer to consumer". (name=single-cluster-monitor,type=consume-service, records-delay-ms-avg". Is there a way to see latency on individual hosts/broker to figure out if any broker is slow? Thanks

Configure records produced per second

How do I configure records produced per second by the monitor? It is pretty high right now.
Also I don't want to produce every second. How do I configure this as well?

Is there a plan to support latest release of Kafka 0.10.0?

Hi,

Is there a plan to support latest release of Kafka 0.10.0?

Thanks,
Kiran

topic-level configuration for kafka-monitor

Hi! I am trying to set an override for the log retention time of the kakfa-monitor topic. I went through the code, and figured out that the way to do this may be as follows, which did not work. I was wondering if this is the right approach. Thank you very much.
In kafka-monitor.properties, I have:

"single-cluster-monitor": {
    "class.name": "com.linkedin.kmf.apps.SingleClusterMonitor",
    "topic": "kafka-monitor",
    "zookeeper.connect": "localhost:2181",
    "bootstrap.servers": "localhost:9092",
    "topic-management.topicCreationEnabled": true,
    ...
    "topic-management.topicFactory.props": {
    },
    "topic-management.topic.props": {
      "log.retention.ms": 3600000
    },
    ...
}

Gradle build failed - can't access the Maven project object model (POM) artifacts

I wonder if the URLs are hard-coded in the java programs that access

exec /opt/jdk1.8.0_131/bin/java -Dorg.gradle.appname=gradlew -classpath /opt/linkedin/kafka-monitor-master/gradle/wrapper/gradle-wrapper.jar org.gradle.wrapper.GradleWrapperMain jar
:compileJava FAILED

FAILURE: Build failed with an exception.

• What went wrong:
  Could not resolve all dependencies for configuration ':compileClasspath'.

Could not resolve net.sourceforge.argparse4j:argparse4j:0.5.0.
Required by:
project :
Could not resolve net.sourceforge.argparse4j:argparse4j:0.5.0.
> Could not get resource 'https://repo1.maven.org/maven2/net/sourceforge/argparse4j/argparse4j/0.5.0/argparse4j-0.5.0.pom'.
> Could not GET 'https://repo1.maven.org/maven2/net/sourceforge/argparse4j/argparse4j/0.5.0/argparse4j-0.5.0.pom'.
> repo1.maven.org: Name or service not known
Could not resolve org.slf4j:slf4j-log4j12:1.7.6.
Required by:
project :
Could not resolve org.slf4j:slf4j-log4j12:1.7.6.
> Could not get resource 'https://repo1.maven.org/maven2/org/slf4j/slf4j-log4j12/1.7.6/slf4j-log4j12-1.7.6.pom'.
> Could not GET 'https://repo1.maven.org/maven2/org/slf4j/slf4j-log4j12/1.7.6/slf4j-log4j12-1.7.6.pom'.
> repo1.maven.org
Could not resolve org.apache.avro:avro:1.4.0.
Required by:
project :
Could not resolve org.apache.avro:avro:1.4.0.
> Could not get resource 'https://repo1.maven.org/maven2/org/apache/avro/avro/1.4.0/avro-1.4.0.pom'.
> Could not GET 'https://repo1.maven.org/maven2/org/apache/avro/avro/1.4.0/avro-1.4.0.pom'.
> repo1.maven.org
Could not resolve org.apache.kafka:kafka_2.11:0.10.1.1.
Required by:
project :
Could not resolve org.apache.kafka:kafka_2.11:0.10.1.1.
> Could not get resource 'https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.11/0.10.1.1/kafka_2.11-0.10.1.1.pom'.
> Could not GET 'https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.11/0.10.1.1/kafka_2.11-0.10.1.1.pom'.
> repo1.maven.org
Could not resolve org.apache.kafka:kafka-clients:0.10.1.1.
Required by:
project :
Could not resolve org.apache.kafka:kafka-clients:0.10.1.1.
> Could not get resource 'https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/0.10.1.1/kafka-clients-0.10.1.1.pom'.
> Could not GET 'https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/0.10.1.1/kafka-clients-0.10.1.1.pom'.
> repo1.maven.org
Could not resolve org.testng:testng:6.8.8.
Required by:
project :
Could not resolve org.testng:testng:6.8.8.
> Could not get resource 'https://repo1.maven.org/maven2/org/testng/testng/6.8.8/testng-6.8.8.pom'.
> Could not GET 'https://repo1.maven.org/maven2/org/testng/testng/6.8.8/testng-6.8.8.pom'.
> repo1.maven.org
Could not resolve org.eclipse.jetty:jetty-server:8.1.19.v20160209.
Required by:
project :
Could not resolve org.eclipse.jetty:jetty-server:8.1.19.v20160209.
> Could not get resource 'https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-server/8.1.19.v20160209/jetty-server-8.1.19.v20160209.pom'.
> Could not GET 'https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-server/8.1.19.v20160209/jetty-server-8.1.19.v20160209.pom'.
> repo1.maven.org
Could not resolve org.json:json:20140107.
Required by:
project :
Could not resolve org.json:json:20140107.
> Could not get resource 'https://repo1.maven.org/maven2/org/json/json/20140107/json-20140107.pom'.
> Could not GET 'https://repo1.maven.org/maven2/org/json/json/20140107/json-20140107.pom'.
> repo1.maven.org
Could not resolve com.fasterxml.jackson.core:jackson-databind:2.7.1.
Required by:
project :
Could not resolve com.fasterxml.jackson.core:jackson-databind:2.7.1.
> Could not get resource 'https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.7.1/jackson-databind-2.7.1.pom'.
> Could not GET 'https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.7.1/jackson-databind-2.7.1.pom'.
> repo1.maven.org
Could not resolve org.jolokia:jolokia-jvm:1.3.3.
Required by:
project :
Could not resolve org.jolokia:jolokia-jvm:1.3.3.
> Could not get resource 'https://repo1.maven.org/maven2/org/jolokia/jolokia-jvm/1.3.3/jolokia-jvm-1.3.3.pom'.
> Could not GET 'https://repo1.maven.org/maven2/org/jolokia/jolokia-jvm/1.3.3/jolokia-jvm-1.3.3.pom'.
> repo1.maven.org
Could not resolve net.savantly:graphite-client:1.1.0-RELEASE.
Required by:
project :
Could not resolve net.savantly:graphite-client:1.1.0-RELEASE.
> Could not get resource 'https://repo1.maven.org/maven2/net/savantly/graphite-client/1.1.0-RELEASE/graphite-client-1.1.0-RELEASE.pom'.
> Could not GET 'https://repo1.maven.org/maven2/net/savantly/graphite-client/1.1.0-RELEASE/graphite-client-1.1.0-RELEASE.pom'.
> repo1.maven.org
Could not resolve com.timgroup:java-statsd-client:3.0.1.
Required by:
project :
Could not resolve com.timgroup:java-statsd-client:3.0.1.
> Could not get resource 'https://repo1.maven.org/maven2/com/timgroup/java-statsd-client/3.0.1/java-statsd-client-3.0.1.pom'.
> Could not GET 'https://repo1.maven.org/maven2/com/timgroup/java-statsd-client/3.0.1/java-statsd-client-3.0.1.pom'.
> repo1.maven.org

• Try:
  Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Does Kafka Monitor support Kerberos Cluster

I am trying to make it work with Kerberoised cluster but seeing below error, Does this support kerberos ?
] ERROR single-cluster-monitor/MultiClusterTopicManagementService will stop due to error. (com.linkedin.kmf.services.MultiClusterTopicManagementService)
kafka.common.KafkaException: Failed to parse the broker info from zookeeper: {"jmx_port":-1,"timestamp":"1513377854404","endpoints":["PLAINTEXTSASL://new-server.abc.net:6667"],"host":null,"version":3,"port":-1}
at kafka.cluster.Broker$.createBroker(Broker.scala:125)
at kafka.utils.ZkUtils.getBrokerInfo(ZkUtils.scala:843)
at kafka.utils.ZkUtils$$anonfun$getAllBrokersInCluster$2.apply(ZkUtils.scala:286)
at kafka.utils.ZkUtils$$anonfun$getAllBrokersInCluster$2.apply(ZkUtils.scala:286)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at kafka.utils.ZkUtils.getAllBrokersInCluster(ZkUtils.scala:286)
at com.linkedin.kmf.common.Utils.createMonitoringTopicIfNotExists(Utils.java:92)
at com.linkedin.kmf.topicfactory.DefaultTopicFactory.createTopicIfNotExist(DefaultTopicFactory.java:26)
at com.linkedin.kmf.services.MultiClusterTopicManagementService$TopicManagementHelper.maybeCreateTopic(MultiClusterTopicManagementService.java:211)
at com.linkedin.kmf.services.MultiClusterTopicManagementService$TopicManagementRunnable.run(MultiClusterTopicManagementService.java:143)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: No security protocol defined for listener PLAINTEXTSASL
at kafka.cluster.EndPoint$$anonfun$securityProtocol$1$1.apply(EndPoint.scala:48)
at kafka.cluster.EndPoint$$anonfun$securityProtocol$1$1.apply(EndPoint.scala:48)
at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
at scala.collection.AbstractMap.getOrElse(Map.scala:59)
at kafka.cluster.EndPoint$.securityProtocol$1(EndPoint.scala:47)
at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:56)
at kafka.cluster.Broker$$anonfun$2.apply(Broker.scala:115)
at kafka.cluster.Broker$$anonfun$2.apply(Broker.scala:115)
at scala.collection.immutable.List.map(List.scala:273)
at kafka.cluster.Broker$.createBroker(Broker.scala:115)
... 21 more