isabella232 / stream-reactor

Streaming reference architecture for ETL with Kafka and Kafka-Connect. You can find more on http://lenses.io on how we provide a unified solution to manage your connectors, most advanced SQL engine for Kafka and Kafka Streams, cluster monitoring and alerting, and more.

Join us on slack

Lenses for Apache Kafka

Lenses offers SQL (for data browsing and Kafka Streams), Kafka Connect connector management, cluster monitoring and more.

You can find more on lenses.io

Stream Reactor

A collection of components to build a real time ingestion pipeline.

Kafka Compatibility

• Kafka 2.5+ (Confluent 5.5) - Stream reactor 2.0.0+
• Kafka 2.0 -> 2.4 (Confluent 5.4) - Stream reactor 1.2.7

DEPRECATION NOTICE

The following connectors have been deprecated and are no longer included in any release from 3.0.

• Blockchain
• Bloomberg
• Rethink
• VoltDB

Connectors

Please take a moment and read the documentation and make sure the software prerequisites are met!!

Connector	Type	Description	Docs
AWS S3	Sink	Copy data from Kafka to AWS S3.	Docs
AzureDocumentDb	Sink	Copy data from Kafka and Azure Document Db.	Docs
Cassandra	Source	Copy data from Cassandra and Kafka.	Docs
*Cassandra	Sink	Certified DSE Cassandra, copy data from Kafka to Cassandra.	Docs
Coap	Source	Copy data from IoT Coap endpoints (using Californium) to Kafka.	Docs
Coap	Sink	Copy data from Kafka to IoT Coap endpoints.	Docs
Elastic 6	Sink	Copy data from Kafka to Elastic Search 6.x w. tcp or http	Docs
FTP/HTTP	Source	Copy data from FTP/HTTP to Kafka.	Docs
Hazelcast	Sink	Copy data from Kafka to Hazelcast.	Docs
HBase	Sink	Copy data from Kafka to HBase.	Docs
Hive	Source	Copy data from Hive/HDFS to Kafka.	Docs
Hive	Sink	Copy data from Kafka to Hive/HDFS	Docs
InfluxDb	Sink	Copy data from Kafka to InfluxDb.	Docs
Kudu	Sink	Copy data from Kafka to Kudu.	Docs
JMS	Source	Copy data from JMS topics/queues to Kafka.	Docs
JMS	Sink	Copy data from Kafka to JMS.	Docs
MongoDB	Sink	Copy data from Kafka to MongoDB.	Docs
MQTT	Source	Copy data from MQTT to Kafka.	Docs
MQTT	Sink	Copy data from Kafka to MQTT.	Docs
Pulsar	Source	Copy data from Pulsar to Kafka.	Docs
Pulsar	Sink	Copy data from Kafka to Pulsar.	Docs
Redis	Sink	Copy data from Kafka to Redis.	Docs

Release Notes

Please see the Stream Reactor Release Notes at Lenses Documentation.

Building

Requires gradle 6.0 to build.

To build

gradle compile

To test

gradle test

To create a fat jar

gradle shadowJar

You can also use the gradle wrapper

./gradlew shadowJar

To view dependency trees

gradle dependencies # or
gradle :kafka-connect-cassandra:dependencies

To build a particular project

gradle :kafka-connect-elastic5:build

To create a jar of a particular project:

gradle :kafka-connect-elastic5:shadowJar

Contributing

We'd love to accept your contributions! Please use GitHub pull requests: fork the repo, develop and test your code, semantically commit and submit a pull request. Thanks!

License