Add additional docs for working with kafka in docker about kafka-shell

[devshawn]

That's super helpful, thanks @solarmosaic-kflorence!

I'll add a config for using .sh commands and push a release tomorrow. :)

Hope that'll help!

[devshawn]

Hi @blankenshipz,

I haven't used this docker image, so this is interesting. Based on that docker image alone, I don't know if you could use kafka-shell in its current state to run commands via the docker image. It does look like the files are on the path, but they end in .sh which kafka-shell currently does not support. For example, docker run -it wurstmeister/kafka kafka-topics.sh works.

My docker example is based on the Confluent docker images (confluentinc/cp-kafka), which do have the files on the path. A simple way to utilize their docker images can be found here (with ZooKeeper as well):

https://github.com/simplesteph/kafka-stack-docker-compose/blob/master/zk-single-kafka-single.yml

If you do that, then it will work properly. If you'd prefer to use the wurstmeister/kafka image, you could:

• Extend that Dockerfile to make it work
• Run another daemon container (see my confluent-tools repository as an example)
• I could extend kafka-shell with some additional configurations to add in .sh

Let me know if those options work for you. I'll probably add in an additional configuration for the .sh extension this weekend.

[devshawn]

Hi @blankenshipz, have you been able to work around this issue? Anything else you need help with?

[blankenshipz]

Hi @devshawn; I haven't had a chance to try it yet but switching to the image you're using should work - I'll reopen this issue if It doesn't.

Thanks!

[solarmosaic-kflorence]

Just wanted to mention that I also stumbled here trying to use this tool with a kafka/zookeeper deployment into docker kubernetes (using cp-kafka). I get the same error for most commands, for example: sh: kafka-topics: command not found.

I am essentially using the example you posted above, but with docker stack deploy --orchestrator kubernetes

[devshawn]

@solarmosaic-kflorence - thanks for the report.

I'd be happy to assist you, but I'd need to know a bit more about your setup and how your config is set up. Feel free to open a new issue 😄. I haven't used docker stack before but I've used this tool with the kubectl prefix to run commands on running k8s containers.

[solarmosaic-kflorence]

hey @devshawn -- here is an example you can use:

kafka-stack.yaml:

version: '3.3'
services:
  # https://hub.docker.com/r/bitnami/zookeeper
  zookeeper1:
    image: bitnami/zookeeper:3.4.13
    hostname: zookeeper1
    ports:
      - "2181:2181"
    environment:
      ALLOW_ANONYMOUS_LOGIN: "yes"
    volumes:
      - zookeeper1-data:/bitnami/zookeeper

  # https://hub.docker.com/r/bitnami/kafka
  kafka1:
    image: bitnami/kafka:2.2.1
    hostname: kafka1
    ports:
      - "9092:9092"
    environment:
      ALLOW_ANONYMOUS_LOGIN: "yes"
      ALLOW_PLAINTEXT_LISTENER: "yes"
      KAFKA_CFG_ADVERTISED_LISTENERS: INTERNAL://kafka1:19092,EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
      KAFKA_CFG_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CFG_LISTENERS: INTERNAL://0.0.0.0:19092,EXTERNAL://0.0.0.0:9092
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CFG_ZOOKEEPER_CONNECT: "zookeeper1:2181"
    volumes:
      - kafka1-data:/bitnami/kafka

deploy with docker stack deploy --orchestrator kubernetes -c kafka-stack.yaml kafka-stack

I can now add the following to my ~/.kafka-shell/config: command_prefix: kubectl exec svc/kafka1 --

But when I try to execute commands, I get:

> kafka-topics --list
OCI runtime exec failed: exec failed: container_linux.go:346: starting container process caused "exec: \"kafka-topics\": executable file not found in $PATH": unknown
command terminated with exit code 126

Looks like it is for the same reason, commands are exposed in this docker image with the .sh extension.

PS: using cp-kafka the commands were fixed by using the kubectl exec svc/kafka1 -- command_prefix option -- would be nice if it were possible to set this option on execution of kafka-shell instead of having to set it in configuration (so it's easier to provide developer documentation without having people modifying their local configuration), but it's not a big deal.

[solarmosaic-kflorence]

FWIW it looks like the .sh commands are defacto: https://kafka.apache.org/quickstart

[solarmosaic-kflorence]

@devshawn since we are slightly off-topic on this ticket, I have created #18 to track. It would probably be useful to add documentation for this use-case at the same time as well.