Consumption kafka performance problems

langke93

I use logstash kafka processing log data, now meet performance bottlenecks, kafka per second received an average of 30000 log data, and logstash processing speed can't keep up with, 1 logstash node can handle only about 5000 data per second, causing a large amount of data accumulation, kafka ranks then I deployed in eight logstash node can process at ordinary times, peak but there are still a large amount of data accumulation in kafka queue.I don't know where is the bottleneck, logstash process CPU utilization rate is only 500% (24 nuclear) I try to use the Java program to connect kafka consumption data, a process can handle over 30000 data per second

langke93

logstash_kafka.conf input { kafka { zk_connect => "zk1:2181,zk2:2181,zk3:2181" group_id => "logstash" topic_id => "global_cdnlog" #consumer_threads => "8" #queue_size => 200 #fetch_message_max_bytes => 10485760 } } output { } start script： bin/logstash -f conf/kafka_stdout.conf -w 24 Commented out parameters have done adjust, but did not significantly improve performance

joekiller

A lot of times your performance isn't limited by Kafka or Logstash but Elasticsearch. You can explore if Kafka/Logstash is really causing the bottleneck using techniques discussed here: https://github.com/logstash-plugins/logstash-output-kafka/issues/20