linkedin / kafka-tools

A collection of tools for working with Apache Kafka.

Kafka Tools

This repository is a collection of tools for working with Apache Kafka.

The Site Reliability team for Kafka at LinkedIn has built these over time in order to make managing Kafka a little bit easier. Our intention is to add to this repository as more tools are developed.

We welcome additions and modifications that make managing Kafka better for all!

Quick Start

1. First, download and install the Kafka admin utilities from: http://kafka.apache.org/downloads.html
2. Then, install kafka-tools using the following command: pip install kafka-tools

Additional documentation may be found at https://kafka-tools.readthedocs.io.

Contributing

Please see the contribution guidelines.

Other Projects

In addition to these tools, LinkedIn has also open-sourced Burrow, a robust system for monitoring Kafka consumer clients.

License

Copyright 2016 LinkedIn Corp.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at:

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

kafka-tools's Issues

Add a --version flag to commands

All commands should have a --version flag that prints out the current version.

kafka-tools overwrites kafka-python `__init__`

Hey,

I am using poetry tool to manage my dependencies.
My package uses aiokafka for consumer/producer management with kafka-python as its dependency.
When istalling both kafka-tools and aiokafka, kafka-tools overwrites __init__.py file from kafka-python dependency, resulting in an error from aiokafka: ImportError: cannot import name 'ConsumerRebalanceListener' from 'kafka'.

How can I solve it? It obviously happens because both kafka-python and kafka-tools decided to name their modules kafka, which results in one overwriting another.

usage guide

@toddpalino
I need to do partition rebalancing in rackaware mode. Can you please help me with the command for that? I was not able to find any usage guide.

Security: SASL compatibility

It seems that this tool can't be used in a secure cluster, as we need to be authenticated to the zookeeper cluster using either a super user (using a digest) or using the principal that kafka client is using.

We need to have options (not exactly sure which) to deal with secure kafka clusters

Example of stacktrace:

root@ip-10-13-80-10:/# kafka-assigner -z zoo1:2181/kafka-xfs -e set-replication-factor --topic __consumer_offsets --replication-factor 3
[INFO] Connecting to zookeeper zoo1:2181/kafka-xfs
[INFO] Getting partition list from Zookeeper
[INFO] Closing connection to zookeeper
[INFO] Partition moves required: 50
[INFO] Number of batches: 5
[INFO] Executing partition reassignment 1/5: {"version": 1, "partitions": [{"topic": "__consumer_offsets", "partition": 0, "replicas": [2, 1, 3]}, {"topic": "__consumer_offsets", "partition": 1, "replicas": [2, 1, 3]}, {"topic": "__consumer_offsets", "partition": 2, "replicas": [2, 1, 3]}, {"topic": "__consumer_offsets", "partition": 3, "replicas": [2, 1, 3]}, {"topic": "__consumer_offsets", "partition": 4, "replicas": [2, 1, 3]}, {"topic": "__consumer_offsets", "partition": 5, "replicas": [2, 1, 3]}, {"topic": "__consumer_offsets", "partition": 6, "replicas": [2, 1, 3]}, {"topic": "__consumer_offsets", "partition": 7, "replicas": [2, 1, 3]}, {"topic": "__consumer_offsets", "partition": 8, "replicas": [2, 1, 3]}, {"topic": "__consumer_offsets", "partition": 9, "replicas": [2, 1, 3]}]}
Traceback (most recent call last):
  File "/usr/local/bin/kafka-assigner", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/kafka/tools/assigner/__main__.py", line 124, in main
    batch.execute(i + 1, len(batches), args.zookeeper, tools_path, plugins, dry_run)
  File "/usr/local/lib/python2.7/dist-packages/kafka/tools/assigner/models/reassignment.py", line 63, in execute
    remaining_partitions = check_reassignment_completion(zookeeper, tools_path, assignfile.name)
  File "/usr/local/lib/python2.7/dist-packages/kafka/tools/assigner/models/reassignment.py", line 92, in check_reassignment_completion
    raise ReassignmentFailedException("The reassignment in progress failed with the following verification output:\n{0}".format(lines))
kafka.tools.assigner.exceptions.ReassignmentFailedException: The reassignment in progress failed with the following verification output:
['Status of partition reassignment: \n', "ERROR: Assigned replicas (2) don't match the list of replicas for reassignment (2,1,3) for partition [__consumer_offsets,7]\n", "ERROR: Assigned replicas (2) don't match the list of replicas for reassignment (2,1,3) for partition [__consumer_offsets,4]\n", "ERROR: Assigned replicas (2) don't match the list of replicas for reassignment (2,1,3) for partition [__consumer_offsets,3]\n", "ERROR: Assigned replicas (2) don't match the list of replicas for reassignment (2,1,3) for partition [__consumer_offsets,0]\n", "ERROR: Assigned replicas (2) don't match the list of replicas for reassignment (2,1,3) for partition [__consumer_offsets,8]\n", "ERROR: Assigned replicas (2) don't match the list of replicas for reassignment (2,1,3) for partition [__consumer_offsets,5]\n", "ERROR: Assigned replicas (2) don't match the list of replicas for reassignment (2,1,3) for partition [__consumer_offsets,6]\n", "ERROR: Assigned replicas (2) don't match the list of replicas for reassignment (2,1,3) for partition [__consumer_offsets,2]\n", "ERROR: Assigned replicas (2) don't match the list of replicas for reassignment (2,1,3) for partition [__consumer_offsets,9]\n", "ERROR: Assigned replicas (2) don't match the list of replicas for reassignment (2,1,3) for partition [__consumer_offsets,1]\n", 'Reassignment of partition [__consumer_offsets,7] failed\n', 'Reassignment of partition [__consumer_offsets,4] failed\n', 'Reassignment of partition [__consumer_offsets,3] failed\n', 'Reassignment of partition [__consumer_offsets,0] failed\n', 'Reassignment of partition [__consumer_offsets,8] failed\n', 'Reassignment of partition [__consumer_offsets,5] failed\n', 'Reassignment of partition [__consumer_offsets,6] failed\n', 'Reassignment of partition [__consumer_offsets,2] failed\n', 'Reassignment of partition [__consumer_offsets,9] failed\n', 'Reassignment of partition [__consumer_offsets,1] failed\n']

remove module not working

[root]# kafka-assigner --tools-path /opt/kafka/bin -z 10.146.1.234:2181 -e remove -b 3
[INFO] Connecting to zookeeper 10.146.1.234:2181
Traceback (most recent call last):
File "/usr/bin/kafka-assigner", line 11, in
load_entry_point('kafka-tools==0.0.3', 'console_scripts', 'kafka-assigner')()
File "/usr/lib/python3.4/site-packages/kafka_tools-0.0.3-py3.4.egg/kafka/tools/assigner/main.py", line 109, in main
cluster = Cluster.create_from_zookeeper(args.zookeeper, getattr(args, 'default_retention', 1))
File "/usr/lib/python3.4/site-packages/kafka_tools-0.0.3-py3.4.egg/kafka/tools/assigner/models/cluster.py", line 78, in create_from_zookeeper
add_brokers_from_zk(cluster, zk)
File "/usr/lib/python3.4/site-packages/kafka_tools-0.0.3-py3.4.egg/kafka/tools/assigner/models/cluster.py", line 32, in add_brokers_from_zk
cluster.add_broker(Broker.create_from_json(int(b), broker_data))
File "/usr/lib/python3.4/site-packages/kafka_tools-0.0.3-py3.4.egg/kafka/tools/assigner/models/broker.py", line 43, in create_from_json
data = json.loads(jsondata)
File "/usr/lib64/python3.4/json/init.py", line 312, in loads
s.class.name))
TypeError: the JSON object must be str, not 'bytes'

RTD Docs are not building correctly.

The RTD docs are not being generated properly.

Metadata refresh configuration default is specified in milliseconds

Python's time is in seconds. Therefore, the config does not mean what we think it means

The getting started documentation needs instructions on how to install the Kafka admin utils

Contribution guidelines should cover how to create a branch for a pull request

Error when attempting to remove a broker

$ kafka-assigner -z kafka-zookeeper:2181 balance --execute remove --broker 1 --to-brokers 4 5 6
usage: kafka-assigner balance [-h] -t
                              [{leader,count,even,rackaware,rate,size} [{leader,count,even,rackaware,rate,size} ...]]
                              [--default-retention DEFAULT_RETENTION]
kafka-assigner balance: error: argument -t/--types is required

--types is not mentioned in the help text for the remove command, and this fails also

Documentation

Can someone please update the proper documentation for this tool. it will be more helpfull

]$ kafka-assigner -z 104:2181 --tools-path /opt/kafka/bin -e balance -t size
[INFO] Connecting to zookeeper :2181
[INFO] Getting partition list from Zookeeper
[INFO] Closing connection to zookeeper
[INFO] Getting partition sizes via SSH for
[INFO] Getting partition sizes via SSH for
[INFO] Getting partition sizes via SSH for
[INFO] Starting partition balance by size
Traceback (most recent call last):
File "/usr/bin/kafka-assigner", line 11, in
load_entry_point('kafka-tools==0.0.3', 'console_scripts', 'kafka-assigner')()
File "/usr/lib/python2.7/site-packages/kafka_tools-0.0.3-py2.7.egg/kafka/tools/assigner/main.py", line 120, in main
action_to_run.process_cluster()
File "/usr/lib/python2.7/site-packages/kafka_tools-0.0.3-py2.7.egg/kafka/tools/assigner/actions/balance.py", line 48, in process_cluster
bmodule.process_cluster()
File "/usr/lib/python2.7/site-packages/kafka_tools-0.0.3-py2.7.egg/kafka/tools/assigner/actions/balancemodules/size.py", line 66, in process_cluster
margins[pos] = (getattr(partitions[pos][sizelen // 2], self._size_attr) + getattr(partitions[pos][sizelen // 2 - 1], self._size_attr)) // 4
IndexError: list index out of range
[ ~]$ kafka-assigner --help
usage: kafka-assigner [-h] [-v] -z ZOOKEEPER [-l] [-g] [-e] [-m MOVES]
[-x EXCLUDE_TOPICS] [--sizer {jmx,ssh}] [-p PROPERTY]
[-s] [--skip-ple] [--ple-size PLE_SIZE]
[--ple-wait PLE_WAIT] [--tools-path TOOLS_PATH]
{trim,elect,set-replication-factor,clone,remove,balance,reorder}

Kafka-protocol shows only version zero for the different commands

kafka-protocol uses only version zero for the different commands. Even on entering help it shows only the v0 commands. Although kafka-protocols has got support for other api_verions too. However only v0 classes are loaded and others are ignored.

Sphinx API generation for all source files

All source files should have API generation working. As of the time writing this comment, only kafka.tools.assigner.tools is enabled.

Roadmap for create_topic and delete_topic?

I noticed that there are commented-out stubs in the Client for create_topic and delete_topic. Any idea if/when these might be implemented?

kafka tools 2中kafka2.1 consumer已经全删了 可以说工具中一直显示consumer存在只是not active 。其中再次删除有些显示not empty 有些显示not exist

Rackaware rebalancing puts replicas in the same rack

I have brokers in racks:

• 115
  • 10103
  • 10104
  • 10105
• 113
  • 10116
  • 10117
  • 10118
• 114
  • 10132
  • 10133
  • 10139

I have the following distribution of replicas:

Topic: mycooltopic	Partition: 0	Leader: 10139	Replicas: 10139,10117,10116	Isr: 10116,10117,10139
Topic: mycooltopic	Partition: 1	Leader: 10133	Replicas: 10133,10118,10104	Isr: 10104,10118,10133
Topic: mycooltopic	Partition: 2	Leader: 10133	Replicas: 10133,10117,10105	Isr: 10105,10117,10133
Topic: mycooltopic	Partition: 3	Leader: 10118	Replicas: 10118,10139,10105	Isr: 10105,10118,10139
Topic: mycooltopic	Partition: 4	Leader: 10103	Replicas: 10103,10133,10105	Isr: 10103,10105,10133
Topic: mycooltopic	Partition: 5	Leader: 10103	Replicas: 10103,10133,10116	Isr: 10103,10116,10133
Topic: mycooltopic	Partition: 6	Leader: 10139	Replicas: 10139,10105,10117	Isr: 10105,10117,10139
Topic: mycooltopic	Partition: 7	Leader: 10133	Replicas: 10133,10104,10118	Isr: 10104,10118,10133
Topic: mycooltopic	Partition: 8	Leader: 10133	Replicas: 10133,10118,10105	Isr: 10105,10118,10133
Topic: mycooltopic	Partition: 9	Leader: 10117	Replicas: 10117,10133,10105	Isr: 10105,10117,10133
Topic: mycooltopic	Partition: 10	Leader: 10116	Replicas: 10116,10105,10103	Isr: 10103,10105,10116
Topic: mycooltopic	Partition: 11	Leader: 10105	Replicas: 10105,10116,10139	Isr: 10105,10116,10139
Topic: mycooltopic	Partition: 12	Leader: 10104	Replicas: 10132,10104,10116	Isr: 10104,10116,10132
Topic: mycooltopic	Partition: 13	Leader: 10105	Replicas: 10105,10139,10117	Isr: 10105,10117,10139
Topic: mycooltopic	Partition: 14	Leader: 10132	Replicas: 10132,10105,10118	Isr: 10105,10118,10132
Topic: mycooltopic	Partition: 15	Leader: 10118	Replicas: 10118,10139,10133	Isr: 10118,10133,10139
Topic: mycooltopic	Partition: 16	Leader: 10105	Replicas: 10105,10117,10104	Isr: 10105,10104,10117
Topic: mycooltopic	Partition: 17	Leader: 10105	Replicas: 10105,10118,10132	Isr: 10105,10118,10132
Topic: mycooltopic	Partition: 18	Leader: 10133	Replicas: 10133,10116,10103	Isr: 10103,10116,10133
Topic: mycooltopic	Partition: 19	Leader: 10105	Replicas: 10105,10117,10139	Isr: 10105,10117,10139
Topic: mycooltopic	Partition: 20	Leader: 10139	Replicas: 10139,10104,10116	Isr: 10104,10116,10139
Topic: mycooltopic	Partition: 21	Leader: 10105	Replicas: 10105,10132,10117	Isr: 10105,10117,10132
Topic: mycooltopic	Partition: 22	Leader: 10116	Replicas: 10116,10139,10133	Isr: 10116,10139,10133
Topic: mycooltopic	Partition: 23	Leader: 10116	Replicas: 10116,10132,10104	Isr: 10104,10116,10132
Topic: mycooltopic	Partition: 24	Leader: 10104	Replicas: 10104,10118,10132	Isr: 10104,10118,10132
Topic: mycooltopic	Partition: 25	Leader: 10118	Replicas: 10118,10133,10105	Isr: 10105,10118,10133
Topic: mycooltopic	Partition: 26	Leader: 10118	Replicas: 10118,10132,10103	Isr: 10103,10118,10132
Topic: mycooltopic	Partition: 27	Leader: 10117	Replicas: 10117,10103,10139	Isr: 10103,10117,10139
Topic: mycooltopic	Partition: 28	Leader: 10132	Replicas: 10132,10103,10116	Isr: 10116,10132,10103
Topic: mycooltopic	Partition: 29	Leader: 10132	Replicas: 10132,10104,10117	Isr: 10104,10117,10132
Topic: mycooltopic	Partition: 30	Leader: 10132	Replicas: 10132,10104,10118	Isr: 10104,10118,10132
Topic: mycooltopic	Partition: 31	Leader: 10118	Replicas: 10118,10105,10133	Isr: 10105,10118,10133
Topic: mycooltopic	Partition: 32	Leader: 10116	Replicas: 10116,10105,10132	Isr: 10105,10116,10132
Topic: mycooltopic	Partition: 33	Leader: 10116	Replicas: 10116,10139,10105	Isr: 10105,10116,10139
Topic: mycooltopic	Partition: 34	Leader: 10133	Replicas: 10133,10116,10103	Isr: 10103,10116,10133
Topic: mycooltopic	Partition: 35	Leader: 10105	Replicas: 10105,10116,10139	Isr: 10105,10116,10139
Topic: mycooltopic	Partition: 36	Leader: 10104	Replicas: 10104,10132,10117	Isr: 10104,10117,10132
Topic: mycooltopic	Partition: 37	Leader: 10132	Replicas: 10132,10103,10118	Isr: 10103,10118,10132
Topic: mycooltopic	Partition: 38	Leader: 10105	Replicas: 10105,10117,10133	Isr: 10105,10117,10133
Topic: mycooltopic	Partition: 39	Leader: 10133	Replicas: 10133,10118,10104	Isr: 10104,10118,10133
Topic: mycooltopic	Partition: 40	Leader: 10104	Replicas: 10104,10118,10132	Isr: 10104,10118,10132
Topic: mycooltopic	Partition: 41	Leader: 10117	Replicas: 10117,10133,10103	Isr: 10103,10117,10133
Topic: mycooltopic	Partition: 42	Leader: 10103	Replicas: 10103,10117,10139	Isr: 10103,10117,10139
Topic: mycooltopic	Partition: 43	Leader: 10103	Replicas: 10103,10132,10116	Isr: 10103,10116,10132
Topic: mycooltopic	Partition: 44	Leader: 10139	Replicas: 10139,10103,10117	Isr: 10103,10117,10139
Topic: mycooltopic	Partition: 45	Leader: 10132	Replicas: 10132,10104,10118	Isr: 10104,10118,10132
Topic: mycooltopic	Partition: 46	Leader: 10103	Replicas: 10117,10103,10133	Isr: 10103,10117,10133
Topic: mycooltopic	Partition: 47	Leader: 10116	Replicas: 10116,10139,10104	Isr: 10104,10116,10139
Topic: mycooltopic	Partition: 48	Leader: 10117	Replicas: 10117,10105,10132	Isr: 10105,10117,10132
Topic: mycooltopic	Partition: 49	Leader: 10116	Replicas: 10116,10132,10103	Isr: 10103,10116,10132
Topic: mycooltopic	Partition: 50	Leader: 10104	Replicas: 10104,10117,10139	Isr: 10104,10117,10139
Topic: mycooltopic	Partition: 51	Leader: 10133	Replicas: 10133,10104,10116	Isr: 10104,10116,10133
Topic: mycooltopic	Partition: 52	Leader: 10104	Replicas: 10104,10133,10117	Isr: 10104,10117,10133
Topic: mycooltopic	Partition: 53	Leader: 10104	Replicas: 10104,10139,10118	Isr: 10104,10118,10139
Topic: mycooltopic	Partition: 54	Leader: 10103	Replicas: 10103,10116,10133	Isr: 10103,10116,10133
Topic: mycooltopic	Partition: 55	Leader: 10118	Replicas: 10118,10132,10104	Isr: 10104,10118,10132
Topic: mycooltopic	Partition: 56	Leader: 10104	Replicas: 10104,10118,10132	Isr: 10104,10118,10132
Topic: mycooltopic	Partition: 57	Leader: 10117	Replicas: 10117,10133,10103	Isr: 10103,10117,10133
Topic: mycooltopic	Partition: 58	Leader: 10118	Replicas: 10118,10103,10139	Isr: 10103,10118,10139
Topic: mycooltopic	Partition: 59	Leader: 10139	Replicas: 10139,10103,10117	Isr: 10103,10117,10139
Topic: mycooltopic	Partition: 60	Leader: 10104	Replicas: 10104,10139,10118	Isr: 10104,10118,10139
Topic: mycooltopic	Partition: 61	Leader: 10117	Replicas: 10117,10104,10133	Isr: 10104,10117,10133
Topic: mycooltopic	Partition: 62	Leader: 10139	Replicas: 10139,10116,10104	Isr: 10104,10116,10139
Topic: mycooltopic	Partition: 63	Leader: 10118	Replicas: 10118,10104,10132	Isr: 10104,10118,10132
Topic: mycooltopic	Partition: 64	Leader: 10117	Replicas: 10117,10139,10105	Isr: 10105,10117,10139
Topic: mycooltopic	Partition: 65	Leader: 10118	Replicas: 10118,10105,10139	Isr: 10105,10118,10139
Topic: mycooltopic	Partition: 66	Leader: 10139	Replicas: 10139,10105,10116	Isr: 10105,10116,10139
Topic: mycooltopic	Partition: 67	Leader: 10103	Replicas: 10103,10132,10118	Isr: 10103,10118,10132
Topic: mycooltopic	Partition: 68	Leader: 10116	Replicas: 10116,10103,10133	Isr: 10103,10116,10133
Topic: mycooltopic	Partition: 69	Leader: 10132	Replicas: 10132,10116,10104	Isr: 10104,10116,10132
Topic: mycooltopic	Partition: 70	Leader: 10105	Replicas: 10105,10117,10132	Isr: 10105,10117,10132
Topic: mycooltopic	Partition: 71	Leader: 10139	Replicas: 10139,10118,10103	Isr: 10103,10118,10139
Topic: mycooltopic	Partition: 72	Leader: 10116	Replicas: 10116,10105,10139	Isr: 10105,10116,10139
Topic: mycooltopic	Partition: 73	Leader: 10103	Replicas: 10103,10133,10116	Isr: 10103,10116,10133
Topic: mycooltopic	Partition: 74	Leader: 10103	Replicas: 10103,10132,10117	Isr: 10103,10117,10132
Topic: mycooltopic	Partition: 75	Leader: 10104	Replicas: 10104,10133,10118	Isr: 10104,10118,10133
Topic: mycooltopic	Partition: 76	Leader: 10117	Replicas: 10117,10103,10133	Isr: 10103,10117,10133
Topic: mycooltopic	Partition: 77	Leader: 10139	Replicas: 10139,10116,10104	Isr: 10104,10116,10139
Topic: mycooltopic	Partition: 78	Leader: 10103	Replicas: 10103,10116,10132	Isr: 10103,10116,10132
Topic: mycooltopic	Partition: 79	Leader: 10132	Replicas: 10132,10117,10105	Isr: 10105,10117,10132
Topic: mycooltopic	Partition: 80	Leader: 10133	Replicas: 10133,10118,10103	Isr: 10103,10118,10133

At least the following partitions violate rack awareness and put two replicas in the same rack:

• 0
• 15
• 22

kafka-tools does't want to rebalance anything with rackaware, even policies:

# kafka-tools --zookeeper myzk.mydomain:2181/kafka --generate --moves 1 balance --types rackaware --types even
[WARN] Note that sizes are not calculated by this docker image
[INFO] Connecting to zookeeper myzk.mydomain:2181/kafka
[INFO] Getting partition list from Zookeeper
[INFO] Closing connection to zookeeper
[INFO] Getting partition sizes via SSH for 10116.mydomain
[INFO] Getting partition sizes via SSH for 10117.mydomain
[INFO] Getting partition sizes via SSH for 10118.mydomain
[INFO] Getting partition sizes via SSH for 10132.mydomain
[INFO] Getting partition sizes via SSH for 10133.mydomain
[INFO] Getting partition sizes via SSH for 10103.mydomain
[INFO] Getting partition sizes via SSH for 10104.mydomain
[INFO] Getting partition sizes via SSH for 10105.mydomain
[INFO] Getting partition sizes via SSH for 10139.mydomain
[WARNING] Skipping topic __consumer_offsets as it has 50 partitions, which is not a multiple of the number of brokers (9)
[INFO] Partition moves required: 0
[INFO] Number of batches: 0
[INFO] --execute flag NOT specified. DRY RUN ONLY
[INFO] Number of replica elections: 1
[INFO] Executing preferred replica election 1/1

With just rackaware it works fine:

# kafka-tools --zookeeper myzk.mydomain:2181/kafka --generate --moves 1 balance --types rackaware
[WARN] Note that sizes are not calculated by this docker image
[INFO] Connecting to zookeeper myzk.mydomain:2181/kafka
[INFO] Getting partition list from Zookeeper
[INFO] Closing connection to zookeeper
[INFO] Getting partition sizes via SSH for 10116.mydomain
[INFO] Getting partition sizes via SSH for 10117.mydomain
[INFO] Getting partition sizes via SSH for 10118.mydomain
[INFO] Getting partition sizes via SSH for 10132.mydomain
[INFO] Getting partition sizes via SSH for 10133.mydomain
[INFO] Getting partition sizes via SSH for 10103.mydomain
[INFO] Getting partition sizes via SSH for 10104.mydomain
[INFO] Getting partition sizes via SSH for 10105.mydomain
[INFO] Getting partition sizes via SSH for 10139.mydomain
[INFO] Starting partition balance by rack
[INFO] Partition moves required: 23
[INFO] Number of batches: 23
[INFO] --execute flag NOT specified. DRY RUN ONLY
[INFO] Executing partition reassignment 1/23: {"version": 1, "partitions": [{"topic": "test", "partition": 0, "replicas": [10133, 10117, 10103]}]}
[INFO] Executing partition reassignment 2/23: {"version": 1, "partitions": [{"topic": "test", "partition": 2, "replicas": [10116, 10105, 10132]}]}
[INFO] Executing partition reassignment 3/23: {"version": 1, "partitions": [{"topic": "test", "partition": 4, "replicas": [10104, 10118, 10133]}]}
[INFO] Executing partition reassignment 4/23: {"version": 1, "partitions": [{"topic": "test", "partition": 7, "replicas": [10139, 10103, 10116]}]}
[INFO] Executing partition reassignment 5/23: {"version": 1, "partitions": [{"topic": "mycooltopic", "partition": 0, "replicas": [10103, 10139, 10116]}]}
[INFO] Executing partition reassignment 6/23: {"version": 1, "partitions": [{"topic": "mycooltopic", "partition": 4, "replicas": [10118, 10133, 10105]}]}
[INFO] Executing partition reassignment 7/23: {"version": 1, "partitions": [{"topic": "mycooltopic", "partition": 9, "replicas": [10116, 10133, 10105]}]}
[INFO] Executing partition reassignment 8/23: {"version": 1, "partitions": [{"topic": "mycooltopic", "partition": 10, "replicas": [10117, 10139, 10103]}]}
[INFO] Executing partition reassignment 9/23: {"version": 1, "partitions": [{"topic": "mycooltopic", "partition": 13, "replicas": [10103, 10139, 10117]}]}
[INFO] Executing partition reassignment 10/23: {"version": 1, "partitions": [{"topic": "mycooltopic", "partition": 15, "replicas": [10105, 10117, 10133]}]}
[INFO] Executing partition reassignment 11/23: {"version": 1, "partitions": [{"topic": "mycooltopic", "partition": 16, "replicas": [10132, 10117, 10104]}]}
[INFO] Executing partition reassignment 12/23: {"version": 1, "partitions": [{"topic": "mycooltopic", "partition": 22, "replicas": [10104, 10117, 10133]}]}
[INFO] Executing partition reassignment 13/23: {"version": 1, "partitions": [{"topic": "unbound", "partition": 2, "replicas": [10105, 10139, 10117]}]}
[INFO] Executing partition reassignment 14/23: {"version": 1, "partitions": [{"topic": "unbound", "partition": 4, "replicas": [10105, 10117, 10132]}]}
[INFO] Executing partition reassignment 15/23: {"version": 1, "partitions": [{"topic": "unbound", "partition": 5, "replicas": [10105, 10117, 10132]}]}
[INFO] Executing partition reassignment 16/23: {"version": 1, "partitions": [{"topic": "unbound", "partition": 6, "replicas": [10133, 10118, 10105]}]}
[INFO] Executing partition reassignment 17/23: {"version": 1, "partitions": [{"topic": "unbound", "partition": 7, "replicas": [10116, 10132, 10103]}]}
[INFO] Executing partition reassignment 18/23: {"version": 1, "partitions": [{"topic": "unbound", "partition": 8, "replicas": [10116, 10133, 10104]}]}
[INFO] Executing partition reassignment 19/23: {"version": 1, "partitions": [{"topic": "mycooltopic_errors", "partition": 1, "replicas": [10133, 10117, 10104]}]}
[INFO] Executing partition reassignment 20/23: {"version": 1, "partitions": [{"topic": "mycooltopic_errors", "partition": 2, "replicas": [10118, 10139, 10105]}]}
[INFO] Executing partition reassignment 21/23: {"version": 1, "partitions": [{"topic": "mycooltopic_errors", "partition": 5, "replicas": [10132, 10116, 10103]}]}
[INFO] Executing partition reassignment 22/23: {"version": 1, "partitions": [{"topic": "mycooltopic_errors", "partition": 6, "replicas": [10117, 10103, 10133]}]}
[INFO] Executing partition reassignment 23/23: {"version": 1, "partitions": [{"topic": "mycooltopic_errors", "partition": 7, "replicas": [10116, 10104, 10132]}]}
[INFO] Number of replica elections: 1
[INFO] Executing preferred replica election 1/1

kafka-tools should provide the best of rackaware and even: equal balancing between nodes and rack awareness.

I'm running kafka-tools form commit 5472a89. Note that I replaced ssh with a stub that always fails, I don't want to ssh to machines to perform rebalancing.

Pex does not build cryptography correctly

Looks like there's a problem with building the pex because of the use of paramiko (which requires the cryptography module). Running the pex with a command that actually executes the SSH code yields:

Traceback (most recent call last):
  File "/export/home/tpalino/kafka-assigner/.bootstrap/_pex/pex.py", line 328, in execute
  File "/export/home/tpalino/kafka-assigner/.bootstrap/_pex/pex.py", line 260, in _wrap_coverage
  File "/export/home/tpalino/kafka-assigner/.bootstrap/_pex/pex.py", line 292, in _wrap_profiling
  File "/export/home/tpalino/kafka-assigner/.bootstrap/_pex/pex.py", line 371, in _execute
  File "/export/home/tpalino/kafka-assigner/.bootstrap/_pex/pex.py", line 429, in execute_entry
  File "/export/home/tpalino/kafka-assigner/.bootstrap/_pex/pex.py", line 447, in execute_pkg_resources
  File "/export/home/tpalino/.pex/install/kafka_tools-0.0.1-py2.py3-none-any.whl.7652e327fcb6d363239ac7d5348fc41002280dcd/kafka_tools-0.0.1-py2.py3-none-any.whl/kafka/tools/assigner/__main__.py", line 75, in main
    check_and_get_sizes(action_map[args.action], args, cluster, sizer_map)
  File "/export/home/tpalino/.pex/install/kafka_tools-0.0.1-py2.py3-none-any.whl.7652e327fcb6d363239ac7d5348fc41002280dcd/kafka_tools-0.0.1-py2.py3-none-any.whl/kafka/tools/assigner/__main__.py", line 32, in check_and_get_sizes
    sizer_to_run.get_partition_sizes()
  File "/export/home/tpalino/.pex/install/kafka_tools-0.0.1-py2.py3-none-any.whl.7652e327fcb6d363239ac7d5348fc41002280dcd/kafka_tools-0.0.1-py2.py3-none-any.whl/kafka/tools/assigner/sizers/ssh.py", line 32, in get_partition_sizes
    self._client.connect(broker.hostname, allow_agent=True)
  File "/export/home/tpalino/.pex/install/paramiko-2.0.1-py2.py3-none-any.whl.85adb5cee11fd6f5e7678d36e1377dd8127a20da/paramiko-2.0.1-py2.py3-none-any.whl/paramiko/client.py", line 338, in connect
    t.start_client()
  File "/export/home/tpalino/.pex/install/paramiko-2.0.1-py2.py3-none-any.whl.85adb5cee11fd6f5e7678d36e1377dd8127a20da/paramiko-2.0.1-py2.py3-none-any.whl/paramiko/transport.py", line 493, in start_client
    raise e
ImportError: /export/home/tpalino/.pex/install/cryptography-1.4-cp26-none-linux_x86_64.whl.f8504802fa31ea43dc6227a796aeb2016051ad99/cryptography-1.4-cp26-none-linux_x86_64.whl/cryptography/hazmat/bindings/_openssl.so: symbol SSL_SESSION_set1_id_context, version libssl.so.10 not defined in file libssl.so.10 with link time reference

help <command> does not show any help message

help [command] shows invalid command for all commands
For example : help TopicMetadata shows invalid command although it is valid.

Enable travis-ci

travis-ci should be enabled after #1 is committed.

Support kafka executables without shell extension?

Hi,

Is it possible to have the kafka-tools support kafka-reassign-partitions instead of kafka-reassign-partitions.sh ? For example, the confluent Kafka images do an apt-get install which go and places the shell scripts directly in the /usr/bin folder, but they don't have the .sh extension.

It'd be great if kafka-tools could support either.

Thoughts on this? I can try to do a PR but it might be messy

Regards,
Stephane

Even not working optimally?

The even mode in balance is the following:

even - This is a specialized type for clusters which have topics that have a number of partitions that are a multiple of the number of brokers. The module assures that every broker has exactly the same number of partitions for each topic.

Yet the code tries to SSH to my brokers?

[ec2-user@ip-172-31-12-216 ~]$ kafka-assigner  -z $ZK_HOST -e balance --types even
[INFO] Connecting to zookeeper 172.31.9.21:2181
[INFO] Getting partition list from Zookeeper
[INFO] Closing connection to zookeeper
[INFO] Getting partition sizes via SSH for 172.31.32.31
[INFO] Getting partition sizes via SSH for 172.31.9.32
[INFO] Getting partition sizes via SSH for 172.31.16.33
[INFO] Getting partition sizes via SSH for 172.31.16.34
[INFO] Starting even partition balance

I would have guessed the even module could just connect to Zookeeper to distribute partitions evenly?

Doesn't work with SSL Secured Kafka Cluster

Trying to use this tool on a secured cluster and we found out that it failed because the broker metadata, host and port stored in the broker znode, /brokers/id/{} are always set to null and -1 respectively unless PLAINTEXT is part of the LISTENERS configuration on that broker.
Is there any workaround or a way to configure this tool to work with SSL secured cluster having listeners and advertised.listeners set for SSL protocol only?
Thanks in advance

Support a configuration file

kafka-assigner should be able to support a configuration file to provide default arguments. This would be things like:

• Setting sizer selection and options
• Setting a default balance module list
• Default tools and JRE locations

The file can be looked for in a default location (e.g. ~/.kafka-tools) or specified on the command line (in which case it should be processed before the rest of the command line is).

Package into a module

prometheus sizer breaks when hostname is an ipv6 address

The prometheus sizer renders the prometheus url with this string interpolation:

url = 'http://{}:{}{}'.format(hostname, port, path)

Unfortunately, in the case where hostname is an ipv6 address, this causes the port number to be interpreted as part of the ipv6 address, causing connections to fail. The fix requires adding square brackets around the hostname when it's an ipv6 address.

I'm still working on the most graceful way to implement the fix. Pulling in an ipv6 address validating library would probably be an unnecessary dependency. Adding brackets around the hostname unconditionally isn't expected to work in general, but it might for urlopen. Adding brackets whenever the hostname has a colon in it should suffice, but could break cases where someone uses username:password@hostname as the hostname for HTTP basic auth. Adding brackets whenever the hostname has two colons in it should work fine (all ipv6 addresses have at least two colons), and I can't think of a downside, but I need to test.

Simplify using Kafka admin commands (jar files)

The problem here is that there's a confusing step in getting kafka-tools working -- installing the Kafka admin command jar files. We should hide this from users as much as possible.

I'm not sure how this should be fixed. Here's a few options:

• If the Kafka admin jar files are not available, the user should be asked if the files should be downloaded and installed for them (perhaps to their home directory?).
• The Kafka admin jar files should be packaged at build time and included as package_resources. This seems like the best option from the users' point of view, but could harm build stability.
• kafka-tools should simply just scan a list of known locations and error out if it can't find them. It should then perhaps prompt the user that they need to run some command (e.g. yum install kafka-server).

IndexError: list index out of range

kafka-tools/kafka/tools/assigner/actions/reorder.py

Line 51 in 0d98bbe

if new_leader != partition.replicas[0]:

partition.replicas[0] throws IndexError. Should be checked for null or empty before the index is referred

balancing by size fails with IndexError: list index out of range

Sometime balancing by size fails due to an IndexError: list index out of range:

INFO] Starting partition balance by size Traceback (most recent call last): File "/usr/local/bin/kafka-assigner", line 9, in <module> load_entry_point('kafka-tools==0.0.2', 'console_scripts', 'kafka-assigner')() File "/usr/local/lib/python2.7/dist-packages/kafka/tools/assigner/__main__.py", line 100, in main action_to_run.process_cluster() File "/usr/local/lib/python2.7/dist-packages/kafka/tools/assigner/actions/balance.py", line 47, in process_cluster bmodule.process_cluster() File "/usr/local/lib/python2.7/dist-packages/kafka/tools/assigner/actions/balancemodules/size.py", line 61, in process_cluster margins[pos] = (partitions[pos][sizelen // 2].size + partitions[pos][sizelen // 2 - 1].size) // 4 IndexError: list index out of range

I have 15 Kafka brokers with the following number of Partitions on them:

Broker01: 434
Broker02: 446
Broker03: 444
Broker04: 369
Broker05: 475
Broker06: 445
Broker07: 453
Broker08: 431
Broker09: 431
Broker10: 426
Broker11: 481
Broker12: 415
Broker13: 356
Broker14: 356
Broker15: 392

I could solve the problem by balancing by count in a previous (separated) step, count and size chained together produce the same error.

I should mention that I have to separated kafka data directories (two disks) on each broker, but specifying only one data dir by using '-d' did not help.

question:The latest version of kafka-tools can be connected when using the Docker to start kafka

In the latest version of kafka-tools, when kafka cluster is used for containerization, kafka-tools cannot connect to the instance on the container. It can be seen that the IP of the container is not the system IP, and it will also cause win11 card death, which needs to be solved

JMX sizer libjvm.so default path is hardcoded

While trying out the kafka-assigner command with the JMX sizer, it tried to load /export/apps/jdk/JDK-1_8_0_72/jre/lib/amd64/server/libjvm.so, which obviously does not exist in my system. It should attempt to find the path for libjvm.so from the JAVA_HOME environment variable instead, if the libjvm property is not set.

"Cluster Leadership Balance (before)" printed twice

Looks like #33 is responsible for "Cluster Leadership Balance (before)" being printed twice, as "Cluster Leadership Balance (after)" got removed.

Kafka leader ordering

Hello , I am a little confused with kafka leader ship reordering.
if new_leader is None:
for replica in partition.replicas:
leader_ratio = leaders[replica.id] / replica.num_partitions() -> will this not be most of the times 0 hence it is not balanced. What is the logic behind this ?

Support for different Kafka versions

The Client only supports a single Kafka version (currently mapped to 0.10.2). It should have a configuration to select the version of the client to support, and use the appropriate versions of requests. For reference, here are the requests supported:

0.8.0
	TopicMetadata (v0)
	ListOffset (v0)
0.8.1.1
	TopicMetadata (v0)
	ListOffset (v0)
	OffsetCommit (v0)
	OffsetFetch (v0)
0.8.2.2
	TopicMetadata (v0)
	ListOffset (v0)
	OffsetCommit (v1)
	OffsetFetch (v1) (v0 reads from ZK, v1 from Kafka)
	GroupCoordinator (v0) (was called ConsumerMetadata)
0.9.0.1
	TopicMetadata (v0)
	ListOffset (v0)
	OffsetCommit (v2)
	OffsetFetch (v1) (v0 reads from ZK, v1 from Kafka)
	GroupCoordinator (v0)
0.10.0.0
	TopicMetadata (v1)
	ListOffset (v0)
	OffsetCommit (v2)
	OffsetFetch (v1) (v0 reads from ZK, v1 from Kafka)
	GroupCoordinator (v0)
	DescribeGroups (v0)
0.10.1.1
	TopicMetadata (v2)
	ListOffset (v1)
	OffsetCommit (v2)
	OffsetFetch (v1) (v0 reads from ZK, v1 from Kafka)
	GroupCoordinator (v0)
	DescribeGroups (v0)
0.10.2.1
	TopicMetadata (v2)
	ListOffset (v1)
	OffsetCommit (v2)
	OffsetFetch (v1) (v0 reads from ZK, v1 from Kafka)
	GroupCoordinator (v0)
	DescribeGroups (v0)
0.11.0.0
	TopicMetadata (v4)
	ListOffset (v2)
	OffsetCommit (v2)
	OffsetFetch (v2) (v0 reads from ZK, v1 from Kafka)
	FindCoordinator (v1)
	DescribeGroups (v1)
1.0.0
	TopicMetadata (v5)
	ListOffset (v2)
	OffsetCommit (v3)
	OffsetFetch (v3) (v0 reads from ZK, v1 from Kafka)
	FindCoordinator (v1)
	DescribeGroups (v1)

Client will not initialize against a cluster with a missing broker

The client will throw an error if you try to connect to a cluster with a missing broker. This can happen because a replica was badly assigned, or because a broker is down, but the client should be able to work around it. Right now, it takes the metadata response and builds a map of all the topics, partitions, and brokers and connects them all together. When it tries to assign the replica for a non-existent broker, it throws a KeyError and fails.

[INFO] Connecting to bootstrap-host.linkedin.com on port 12345 using SSL
[INFO] Disconnecting from bootstrap-host.linkedin.com
[2017-09-14 13:44:30,992] ERROR in app: Exception on /v2/kafka/queuing [GET]
Traceback (most recent call last):
  File ".../Flask-0.12-py2.py3-none-any.whl/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File ".../Flask-0.12-py2.py3-none-any.whl/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File ".../Flask-0.12-py2.py3-none-any.whl/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File ".../Flask-0.12-py2.py3-none-any.whl/flask/_compat.py", line 33, in reraise
    raise value
  File ".../Flask-0.12-py2.py3-none-any.whl/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File ".../Flask-0.12-py2.py3-none-any.whl/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File ".../menagerie-1.0.6-py3-none-any.whl/menagerie/api_v2/kafka.py", line 53, in get_cluster
    (broker_list, is_jbod) = menagerie.khelper2.get_broker_list(app, cluster)
  File ".../menagerie-1.0.6-py3-none-any.whl/menagerie/khelper2.py", line 18, in get_broker_list
    kclient = _get_kafka_client(app, cluster)
  File ".../menagerie-1.0.6-py3-none-any.whl/menagerie/khelper.py", line 46, in _get_kafka_client
    newclient.connect()
  File ".../kafka_tools-0.1.3-py2.py3-none-any.whl/kafka/tools/utilities.py", line 107, in wrapper
    return item(self, *args, **kwargs)
  File ".../kafka_tools-0.1.3-py2.py3-none-any.whl/kafka/tools/client.py", line 100, in connect
    self._connected = self._maybe_bootstrap_cluster(try_brokers.pop())
  File ".../kafka_tools-0.1.3-py2.py3-none-any.whl/kafka/tools/client.py", line 403, in _maybe_bootstrap_cluster
    self._update_from_metadata(metadata)
  File ".../kafka_tools-0.1.3-py2.py3-none-any.whl/kafka/tools/client.py", line 694, in _update_from_metadata
    self._update_topics_from_metadata(metadata, delete=delete)
  File ".../kafka_tools-0.1.3-py2.py3-none-any.whl/kafka/tools/client.py", line 679, in _update_topics_from_metadata
    partition.add_or_update_replica(i, self.cluster.brokers[replica])
KeyError: 13300

What should happen is that we create a dummy Broker object that has only an ID (no host information). We'll also need to make sure that the client gracefully does not connect to this broker. One way would be to put the broker in a special dead_brokers map on the cluster, so it exists but it isn't enumerated when looking at the brokers map.