.Net implementation of the Apache Kafka Protocol that provides basic functionality through Producer/Consumer classes. The project also offers balanced consumer implementation. The project is a fork from ExactTarget's Kafka-net Client.
- Clone CSharpClient-for-Kafka through
git clone https://github.com/Microsoft/CSharpClient-for-Kafka.git
- Open
src\KafkaNETLibraryAndConsole.sln
in Visual Studio - Build Solution
- Open Test Window in Visual Studio: Test>Windows>Test Explorer
- Run all
- Setup local Kafka and Zookeeper
topic Dump topics metadata, such as: earliest/latest offset, replica, ISR.
consumesimple Consume data in single thread.
consumegroup Monitor latest offset gap and speed of consumer group.
consumegroupmonitor Monitor latest offset gap and speed of consumer group.
producesimple Produce data in single thread.
produceperftest Produce data in multiple thread.
eventserverperftest Http Post data to event server in multiple thread.
producemonitor Monitor latest offset.
test Run some adhoc test cases.
The Producer can send one message or an entire batch to Kafka. When sending a batch you can send to multiple topics at once
var brokerConfig = new BrokerConfiguration()
{
BrokerId = this.brokerId,
Host = this.kafkaServerName,
Port = this.kafkaPort
};
var config = new ProducerConfiguration(new List<BrokerConfiguration> { brokerConfig });
kafkaProducer = new Producer(config);
// here you construct your batch or a single message object
var batch=ConstructBatch();
kafkaProducer.Send(batch);
The simple Consumer allows full control for retrieving data. You could instantiate a Consumer directly by providing a ConsumerConfiguration and then calling Fetch. CSharpClient-for-Kafka has a higher level wrapper around Consumer which allows consumer reuse and other benefits
// create the Consumer higher level manager
var managerConfig = new KafkaSimpleManagerConfiguration()
{
FetchSize = FetchSize,
BufferSize = BufferSize,
Zookeeper = m_zookeeper
};
m_consumerManager = new KafkaSimpleManager<int, Kafka.Client.Messages.Message>(managerConfig);
// get all available partitions for a topic through the manager
var allPartitions = m_consumerManager.GetTopicPartitionsFromZK(m_topic);
// Refresh metadata and grab a consumer for desired partitions
m_consumerManager.RefreshMetadata(0, m_consumerId, 0, m_topic, true);
var partitionConsumer = m_consumerManager.GetConsumer(m_topic, partitionId);
The balanced consumer manages partition assignment for each instance in the same consumer group. Rebalance are triggered by zookeeper changes.
// Here we create a balanced consumer on one consumer machine for consumerGroupId. All machines consuming for this group will get balanced together
ConsumerConfiguration config = new ConsumerConfiguration
{
AutoCommit = false,
GroupId = consumerGroupId
ConsumerId = uniqueConsumerId
MaxFetchBufferLength = m_BufferMaxNoOfMessages,
FetchSize = fetchSize,
AutoOffsetReset = OffsetRequest.LargestTime,
NumberOfTries = 20,
ZooKeeper = new ZooKeeperConfiguration(zookeeperString, 30000, 30000, 2000)
};
var balancedConsumer = new ZookeeperConsumerConnector(config, true, m_ConsumerRebalanceHandler, m_ZKDisconnectHandler, m_ZKExpireHandler);
// grab streams for desired topics
var streams = m_ZooKeeperConsumerConnector.CreateMessageStreams(m_TopicMap, new DefaultDecoder());
var KafkaMessageStream = streams[m_Topic][0];
// start consuming stream
foreach (Message message in m_KafkaMessageStream.GetCancellable(cancellationTokenSource.Token))
....
Contributions to CSharpClient-for-Kafka are welcome. Here is how you can contribute to CSharpClient-for-Kafka:
- Submit bugs and help us verify fixes
- Submit pull requests for bug fixes and features and discuss existing proposals
kafkanet's People
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.