wvanbergen / kazoo-go

Go library to access Kafka metadata in Zookeeper

Kazoo

Kazoo is a library to interact with the Kafka metadata that lives in Zookeeper. It provides discovery of the cluster's brokers, topic metadata, and consumer groups.

API documentation can be found on godoc.org.

kazoo-go's Issues

error when creating topic

We keep getting this error even though we have 1 kafka and 1 zookeeper running in my local machine. But once i restart kafka and zoo keeper it works again.
This happens after a while using our service for creating topics and ingesting data. Once this happens we have to restart the kafka and zoo keeper for the error to go away. Do you think this is because of the library.

error creating topic err="Replication factor must be between 1 and the number of brokers" topic=test_topic

i have this exception in zookeeper

EndOfStreamException: Unable to read additional data from client sessionid 0x15aecd5f32802d3, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)

kafka

Unable to reconnect to ZooKeeper service, session 0x15aecd5f32802ce has expired, closing socket connection (org.apache.zookeeper.ClientCnxn)
[2017-03-21 10:14:56,545] INFO zookeeper state changed (Expired) (org.I0Itec.zkclient.ZkClient)

can we create topic with one partition using this library for kafka 0.10.0.1

I am using kafka 0.10.0.1 . can i create topic. My default no of partitions is 5. can i customize and create 1 partition topic using this library. also note that the kafka version is latest and it doesnt use zoo keeper for storing the meta data. Does this still support kafka latest version.

how to get kafka logsize

i want to get kafka lag,i got offset,but i can not get kafka logsize,can anyone help me
kafka version is 0.82

Provide per-partition first and last offset

The offset shell allows to dump the oldest and newest offsets currently available in a partition:
https://cwiki.apache.org/confluence/display/KAFKA/System+Tools#SystemTools-GetOffsetShell
The same information is also available from the SimpleConsumer: https://apache.googlesource.com/kafka/+/trunk/core/src/main/scala/kafka/consumer/SimpleConsumer.scala#184

This information seems to be currently missing in the data returned by kazoo-go (you can ask for the current offset of a consumergroup, but not for the per-partition oldest/newset offsets).

ability to configure topic

Should be able to configure topic. Something like this:

func (t *Topic) SetConfig(map[string]string config) error {
  ...
}

Can I submit a patch for this?

BrokersW

I would find it extremely useful to have a BrokersW function to watch for broker changes.

If we have multiple instances in the same consumergroup cannot register for different Kafka topics

The reason is https://github.com/wvanbergen/kafka/blob/master/consumergroup/consumer_group.go#L261 fetches all instances in the current consumergroup without looking at the Kafka topics they are subscribing for.

Ignore nodes already existing properly when creating ConsumerGroup

In function kazoo.mkdirRecursive , it is trying to create the node and checking if the error is ErrNodeExists which is treated as no error.

The problem with this approach is that the intermediate nodes might exist but have restrictive ACL to not allow Create operation. In this case, we directly get unauthenticated error from zookeeper. For example, in my case:

/stream/kafka/dc is the Chroot. And this path already exists.
However, Create is not allowed for /, so trying to do create /stream directly returns unauthenticated error without any further evaluation. This fails the mkdirRecursive

Wouldn't it be better to explicitly check if the node exists and then try to create it if it doesn't? With this, if the node is there, there won't be any problem with authentication. If the node does not exist, and create is not allowed, then it will fail, like it should.

ResetOffsets doesn't delete correctly

Line 420 if consumergroup.go: You are attempting to delete version 0, which may not exist.

AFAICT (I'm no expert here), you first need to call Exists to return a stat struct for this node, extract the version number from that stat structure, and then delete that version, much as is done for the parent delete at line 425 just below that.

In addition, it would be nice if there were a path in JoinConsumerGroup (package kafka) that could optionally call ResetOffsets()

Add admin tools replacements

Tasks:

• Create topic
• Modify topic config
• Reassign partitions
• Delete topic

/cc @CAFxX IBM/sarama#507

Tag it please

With the advent of dep and vgo proper semver versioning becomes a requirement. So could you please tag the release.