keedio / flume-ng-kafka-sink

storm

flume-ng-demo-kafka-sink

This project is used for flume-ng to communicate with kafka 0.8.1.1.

Aditionally to original plugin, this one implements a dynamic kafka topic selection method. It's used an ExtraData field in flume event body to build the kafka destintation topic.

This plugin depends on flume-enrichment-interceptor

Build

mvn package

Deploy

You have to copy the .jar library generated in target to the plugins.d lib folder of your flume installation
(e.g /usr/lib/flume/plugins.d/flume-ng-demo-kafka-sink/lib and the .jar libraries in target/libs in plugins.d libext folder
(e.g /plugins.d/flume-ng-demo-kafka-sink/libext)

Working mode:

To build the destination topic dynamicTopic configuration parameter is used.
The format of this parameter is key1-key2..., so it's search in the ExtraData field both keys and builds the topic with them

Example:

• "dynamicTopic = hostname-domain"
• Flume eventBody = { "extraData":{"hostname": "localhost", "domain": "localdomain"}, "message": "the original body string"}

The destination topic to be build will be "localhost-localdomain"
If this topic doesn't exists the defaultTopic will be used as destination topic.

Configuration example of Kafka Demo Sink

    # In async producer:
    agent.sinks.kafka-sink.channel = memory-channel
    agent.sinks.kafka-sink.type = org.keedio.flume.sink.KafkaSink
    agent.sinks.kafka-sink.zk.connect = hadoop-manager:2181,hadoop-node1:2181,hadoop-node2:2181
    agent.sinks.kafka-sink.defaultTopic = default
    agent.sinks.kafka-sink.dynamicTopic = hostname-item
    agent.sinks.kafka-sink.batch.num.messages = 1000
    agent.sinks.kafka-sink.queue.buffering.max.ms = 1000
    agent.sinks.kafka-sink.producer.type = async
    agent.sinks.kafka-sink.metadata.broker.list = hadoop-manager:9092,hadoop-node1:9092,hadoop-node2:9092

flume-ng-kafka-sink's Issues

On kafkaSinkCounter divided by zero

suggested workaround:
f (last_sent - start_time >= 1000) {
long secondsElapsed = (last_sent - start_time) / 1000;
sendThroughput = eventCount / secondsElapsed;

Rename DemoKafkaSink by KafkaSink.

Due to new convention name the type of agent is
agent.sinks.kafka-sink.type = org.keedio.flume.sink.KafkaSink
so must search for KafkaSink instead DemoKafkaSink.