Facing Java Heap Error about kafka-monitor

[lindong28]

Thanks for reporting this error @hackeys. By default the consumer fetches up to 1 MB per partition, and Kafka Monitor allocates 256 MB JVM memory. Can you provide the kafka monitor configuration or command line argument you used to start Kafka Monitor? I would like to know how many ConsumeService you have started and what is the partition number of your monitor topic.

[hackeys]

I have Kafka running in 3 nodes with TLS protocol enabled on broker . ZK is still on non-TLS .

I am running using below command ,

Attached is the config file.
kafka-monitor.txt

bin/kafka-monitor-start.sh config/kafka-monitor.properties

[hackeys]

I see these logs all the time...
Broker , ZK everything is up and running fine .

WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetwrokClient)
WARN Bootstrap broker kafkaserver2.server.com::9092 disconnected (org.apache.kafka.clients.NetwrokClient)
WARN Bootstrap broker kafkaserver3.server.com::9092 disconnected (org.apache.kafka.clients.NetwrokClient)

WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetwrokClient)
WARN Bootstrap broker kafkaserver2.server.com::9092 disconnected (org.apache.kafka.clients.NetwrokClient)
WARN Bootstrap broker kafkaserver3.server.com::9092 disconnected (org.apache.kafka.clients.NetwrokClient)

WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetwrokClient)
WARN Bootstrap broker kafkaserver2.server.com::9092 disconnected (org.apache.kafka.clients.NetwrokClient)
WARN Bootstrap broker kafkaserver3.server.com::9092 disconnected (org.apache.kafka.clients.NetwrokClient)

WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetwrokClient)
WARN Bootstrap broker kafkaserver2.server.com::9092 disconnected (org.apache.kafka.clients.NetwrokClient)
WARN Bootstrap broker kafkaserver3.server.com::9092 disconnected (org.apache.kafka.clients.NetwrokClient)

[lindong28]

Hey @hackeys, sorry for late reply. I think your zookeeper.connect config is wrong. kafka-monitor's zookeeper.connect has the same semantics and format as the zookeeper.connect of Apache Kafka. The value of this config should have format url:port, e.g. localhost:9092, which identify a single host instead of a list of hosts. Do you know the host name and port number of the zookeeper in your cluster?

The error messages you provided above indicate that KafkaMonitor can not connect to the broker at localhost:9092. Have you started broker on your local machine with port number 9092?

I still couldn't understand why you observed OutOfMemoryException with this config. Does this happen frequently or not?

Thanks.

[hackeys]

@lindong28 thanks for your response .

I often get OutofMemoryException .

I am using the correct zookeeper host name but yet i see issue.
I am also see that kafka broker at localhost:9092 is up and running yet i see these issue all the time in my log.

[lindong28]

@hackeys Do you have three zookeeper hosts in your cluster, i.e. localhost:2181,kafkaserver2.server.com:2181,kafkaserver3.server.com:2181? I think each Kafka cluster only needs one zookeeper.

Can you try to produce/consume data using the Apache kafka tools?

To produce data:
path_to_kafka/bin/kafka-producer-perf-test.sh --topic test --num-records 1000 --record-size 100 --throughput 1 --producer-props bootstrap.servers=localhost:9092

To consume data:
path_to_kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test

Are you able to use some memory profiling tool to where those memory goes? Alternatively, can you provide a heapdump before Kafka Monitor runs out of memory so that we can analyze the problem?

[hackeys]

@lindong28 i have tried all above steps and yet see no value being populated .

It always says broker is disconnected ...

[lindong28]

@hackeys If you can not produce and consume data using tools from Apache Kafka, it means there is something wrong with you Kafka cluster setup. Can you follow Apache Kafka wiki https://kafka.apache.org/ to setup Kafka cluster and make sure you can produce and consume data?

[hackeys]

@lindong28 nope. I meant i am able to produce and consume but no values in Kafka-monitor tool.

Kafka instance is being used by people on regular basis.

[hackeys]

Do i need to follow any special instructions if kafka broker is on TLS protocol ?

[lindong28]

@hackeys Kafka Monitor uses the same producer/consumer in Apache Kafka by default. If you can produce and consume using the producer/consumer in Apache Kafka, you should be able to use Kafka Monitor to monitor the same cluster as well. You need to configure producer/consumer in Kafka Monitor in the same way that you configure your existing producer/consumer. I am not sure if this should be called "special instruction".

Let us give it another try. We know that the zookeeper.connect in your previous kafka-monitor.txt is wrong in the sense that its value should identify one zookeeper instead of a list of zookeeper. Can you provide the updated kafka-monitor.txt that still causes the error? In addition, can you provide the commands used to produce and consume messages using tools in the Apache Kafka as well as the consumer property file if any?