devshawn / kafka-shell

⚡A supercharged, interactive Kafka shell built on top of the existing Kafka CLI tools.

kafka-shell

A supercharged, interactive Kafka shell built on top of the existing Kafka CLI tools.

Kafka shell allows you to configure a list of clusters, and properties such as --bootstrap-server and --zookeeper for the currently selected cluster will automatically be added when the command is run. No more remembering long server addresses or ports!

Installation

Kafka shell requires python and pip. Kafka shell is a wrapper over the existing Kafka command-line tools, so those must exist within your PATH.

You can install kafka-shell using pip:

pip install kafka-shell

Usage

Kafka shell is an interactive shell. You can run it from the terminal by:

kafka-shell

From here, you can start typing kafka and the autocomplete will kick in.

Key Commands

• Change Cluster: The selected cluster commands are run against can be cycled through by pressing F2.
• Fuzzy Search: By default, fuzzy search of commands is enabled. This can be toggled on & off by pressing F3.
• In-line Help: By default, in-line help is shown along side the drop-down suggestion list. This can be toggled on & off by pressing F9.
• Exit: The shell can be exited by pressing F10 or by typing exit.

Configuration

Kafka shell allows you to configure settings and Kafka clusters to run commands against through a configuration file.

By default, when kafka-shell is first run, a directory in your home directory is generated at ~/.kafka-shell. A configuration file called config.yaml is generated in that new folder. It is a YAML file containing details about clusters, kafka-shell configuration, and more.

See CONFIGURATION.md to add a cluster to your configuration or to set other kafka-shell settings.

Features

Kafka shell simplifies running complex Kafka CLI commands as well as provides smart auto-completion of commands, options, and more. /Users/shawn/.kafka-shell/config.yaml

• Auto-completion of commands, options, and configurations
• Configure clusters to run commands against and switch between them
• Fish-style auto suggestions
• Command history
• Contextual help
• Toolbar options

Completion of Configurations

Auto completion of Kafka configuration keys and their values.

Configure Clusters, Schema Registries, & More

Configure clusters and their properties will automatically be added to commands being run.

Supported Commands

Currently, the following commands are supported:

• kafka-topics
• kafka-configs
• kafka-console-consumer
• kafka-console-producer
• kafka-avro-console-consumer
• kafka-avro-console-producer
• kafka-verifiable-consumer
• kafka-verifiable-producer
• kafka-preferred-replica-election
• kafka-replica-verification
• kafka-reassign-partitions
• kafka-broker-api-versions
• kafka-consumer-groups
• kafka-delete-records
• kafka-log-dirs
• kafka-dump-log
• kafka-acls
• ksql

Helper Commands

Currently, kafka-shell has helper commands:

• exit: exit the shell
• clear: clear the shell
• cluster-select: select a cluster
• cluster-describe: describe a cluster config

In-line help for each command and option is shown by default. This can be toggled by F9.

Contributing

Contributions are very welcome. See CONTRIBUTING.md for details.

Acknowledgement

This project was inspired by multiple amazing shells & prompts, such as saws, kube-shell, kube-prompt, http-prompt, and wharfee. It was built using prompt-toolkit. Much ❤️ to Apache Kafka and Confluent for their helpful CLI tools.

License

Copyright (c) 2019 Shawn Seymour.

Licensed under the Apache 2.0 license.

kafka-shell's Issues

Add additional docs for working with kafka in docker

I'm new to Kafka and I had a hard time understanding how to interact with the cluster that I had setup in docker-compose running on my mac. The way the CLI tools are packaged with Kafka was not intuitive to me.

I found kafka-shell when I was looking for something to make connecting and managing my local (docker) cluster a bit easier; I think it's fantastic. It would be great for my use case if you could include some documentation on what a good config might look like for working with kafka in docker for development. I think this is probably a fairly common usage.

For instance:

I found that prefixing the command with a docker exec as descried in the CONFIGURATION.md didn't quite work because the image that I'm using does not have the Kafka bin directory exported to the PATH - is there a specific image I should use for these shell scripts? I'm using wurstmeister/kafka

Configuration file validation should evaluate environment variables

kafka-shell works fine using environment variables for bootstrap server and zookeeper, but barfs at schema_registry_url due to a validation error
To Reproduce
Use a config.yaml file like this

version: 1
enable:
  history: true
  save_on_exit: true
  auto_complete: true
  auto_suggest: true
  inline_help: true
  fuzzy_search: true
cluster: local
clusters:
  local:
    bootstrap_servers: $BOOTSTRAP_SERVER
    zookeeper_connect: $ZOO_KEEPER
    schema_registry_url: $SCHEMA_REGISTRY

Expected behavior
It should evalute the environment variables
Screenshots

[root@d89fde020d7f /]# kafka-shell
Invalid user configuration (clusters, local, schema_registry_url): '$SCHEMA_REGISTRY' does 
not match '^https?://(.*)$'

config.yml removed once sing off

the config.yml files are removed once signoff of remote server.

best regards.

Ignacio F

Add command: kafka-reassign-partitions

Add support for the command:

kafka-reassign-partitions

Add command: kafka-replica-verification

Add support for the command:

kafka-replica-verification

Connect to Remote kafka cluster

Hi

Can I connect to a remote Kafka cluster ( not local) using this?

Add support for `.sh` kafka commands in the PATH

Is your feature request related to a problem? Please describe.
See discussion in #15

Describe the solution you'd like
Using kafka-shell with a docker image that uses .sh commands (e.g. kafka-topics.sh instead of just `kafka-topics, which seems to be the default behavior provided by Apache Kafka) should not result in an error:

OCI runtime exec failed: exec failed: container_linux.go:346: starting container process caused "exec: \"kafka-topics\": executable file not found in $PATH": unknown
command terminated with exit code 126

Some known images that result in this problem:

• https://hub.docker.com/r/bitnami/kafka/
• https://github.com/wurstmeister/kafka-docker

Additional context
Note that with this fix in place, running commands against docker deployed kafka should work with the following additional configuration in ~/.kafka-shell/config: command_prefix: kubectl exec svc/kafka1 --

Does it support ACL

Hello,

first this looks great, thanks for it.

Second, is there a way to add support for Kafka's ACL on the consumer and producer commands ( all would be great too ) in the config file or somewhere we don't need to retype?

Cheers

Plzzz Add more Easter eggs

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Run kafka-shell
2. ...

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [i.e. macOS Mojave]
• Version [e.g. 0.1.0]

Additional context
Add any other context about the problem here.

How to add my own custom kafka commands into this?

Hi Shawn- First of all great work!. I'm working on adding extra kafka commands into this shell. Any pointers? I was able to add to add custom command into the menu but options execution throws error. I thought to reach out to you for the pointers so that I can add some commands to this and take it forward.
Thanks

Allow for commands to be piped

Allow commands to be piped for things such as | grep. It works for commands that have no auto-fill from config (like when you actually specify --zookeeper, but otherwise fails.

More config/properties auto-completions

First I'd like to say this is a great project. This is saving me an enormous amount of typing being able to easily switch between environments etc. Thanks 👍

When trying out the config auto-complete I noticed it's not yet supported for all commands.

It does work for kafka-configs:

But when checking this for kafka-[avro]-consoler-consumer, I see that the auto-complete doesn't work yet:

Auto fill `--command-config` for commands

Auto fill --command-config for commands, like we auto-fill --consumer.config and --producer.config.

Add command: kafka-preferred-replica-election

Add support for the command:

kafka-preferred-replica-election

cannot launch kafka-shell

Describe the bug
A clear and concise description of what the bug is.

here's my output
hadoop@pascal-asus:/kafka$ kafka-shell
Invalid user configuration (root): None is not of type 'object'
hadoop@pascal-asus:/kafka$

To Reproduce
Steps to reproduce the behavior:

1. Run kafka-shell
2. ...

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [i.e. macOS Mojave]
• Version [e.g. 0.1.0]

Additional context
Add any other context about the problem here.