producer can't send message to the topic about kafka-docker

[dustinfarris]

@jeven2016 where did that IP address come from? "10.196.10.6"

[BenoitPotty]

Hi Guys,

Had the same issue as you, and disabling the Firewall worked fine for me. Did anyone went further on searching which rule we should add to Firewall and be able to leave it active?

Thanks a lot.

[sscaling]

Documentation has been added in #291.

Wiki page: https://github.com/wurstmeister/kafka-docker/wiki/Connectivity

[vvscher]

I assume it's the issue of the kafka itself.

I've fixed it by:

1. mapping "kafka" (host name of the kafka inside the docker) to 127.0.0.1 in hosts file (for some reason kafka doesn't always work properly when I call it by the host different from advertised).
2. and publishing a message to this topic from console producer (to auto create the topic) before the service tries to publish anything.

I cannot call it a decent fix, but it definitely works

[yqian1991]

Hmm, I had the same issue, if you check docker-compose logs, I guess the leader is set to docker-machine ip : 9092, which caused this problem. maybe the problem of docker-compose.yml

[geekhuyang]

@yqian1991 Thanks. your meaning is the var KAFKA_ADVERTISED_HOST_NAME in docker-compose should not be the docker host ip?

[geekhuyang]

I think the ./broker-list.sh result is no correct,but I have no idea about the reason.

@wurstmeister In this case,I should set the KAFKA_ADVERTISED_HOST_NAME value is which above?
thanks

[pluswave]

same issue on my side. I guess it should be some error with specific combination of host os / docker version / kafka-docker version. So I try to describe this and the reproduce method as detail as possible here.

Host os is Centos 7, Docker version 1.10.3, build 20f81dd, java version 1.8.0 (openjdk), kafka-docker is latest git ( f32ebdc ) . the change is

~ $ git diff docker-compose.yml
diff --git a/docker-compose.yml b/docker-compose.yml
index 73dd854..4f81af8 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -9,6 +9,6 @@ kafka:
   links: 
     - zookeeper:zk
   environment:
-    KAFKA_ADVERTISED_HOST_NAME: 192.168.59.103
+    KAFKA_ADVERTISED_HOST_NAME: 192.168.1.100
   volumes:
     - /var/run/docker.sock:/var/run/docker.sock

the ip address 192.168.1.100 is the host address.

step to reproduce:

1. docker-compose up -d

2. install kafka 0.9.0.1 locally by download and install tarball from here and unzip it.

3. cd kafka_2.11-0.9.0.1

4. docker ps to find port forwarding for kafka and zookeeper. in my case

    CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS                       PORTS                                                 NAMES
   9352fee6defe        kafkadocker_kafka        "start-kafka.sh"         17 hours ago        Up 17 hours                0.0.0.0:32785->9092/tcp                               kafkadocker_kafka_1
   2a7851f7af61        wurstmeister/zookeeper   "/bin/sh -c '/usr/sbi"   17 hours ago        Up 17 hours                22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:32784->2181/tcp   kafkadocker_zookeeper_1
   

5. create topic:

   ~$ bin/kafka-topics.sh --create --topic test --zookeeper 192.168.1.100:32784 --partitions 1 --replication-factor 1
   

6. show topic info:

   ~$ bin/kafka-topics.sh --describe  --zookeeper 192.168.1.100:32784
   Topic:test PartitionCount:1    ReplicationFactor:1 Configs:
     Topic: test Partition: 0    Leader: 1001    Replicas: 1001  Isr: 1001
   

7. produce some message:

   $ bin/kafka-console-producer.sh --broker-list="192.168.1.100:32785" --topic test
   abcd
   [2016-05-11 11:21:42,527] WARN Error while fetching metadata with correlation id 0 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
   [2016-05-11 11:21:42,625] WARN Error while fetching metadata with correlation id 1 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
   [2016-05-11 11:21:42,727] WARN Error while fetching metadata with correlation id 2 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
   [2016-05-11 11:21:42,829] WARN Error while fetching metadata with correlation id 3 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
   [2016-05-11 11:21:42,931] WARN Error while fetching metadata with correlation id 4 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
   [2016-05-11 11:21:43,033] WARN Error while fetching metadata with correlation id 5 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
   [2016-05-11 11:21:43,135] WARN Error while fetching metadata with correlation id 6 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
   [2016-05-11 11:21:43,237] WARN Error while fetching metadata with correlation id 7 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
   [2016-05-11 11:21:43,339] WARN Error while fetching metadata with correlation id 8 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
   [2016-05-11 11:21:43,441] WARN Error while fetching metadata with correlation id 9 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
   [2016-05-11 11:21:43,543] WARN Error while fetching metadata with correlation id 10 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
   [2016-05-11 11:21:43,645] WARN Error while fetching metadata with correlation id 11 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
   [2016-05-11 11:21:43,747] WARN Error while fetching metadata with correlation id 12 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
   [2016-05-11 11:21:43,849] WARN Error while fetching metadata with correlation id 13 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
   [2016-05-11 11:21:43,951] WARN Error while fetching metadata with correlation id 14 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
   [2016-05-11 11:21:44,053] WARN Error while fetching metadata with correlation id 15 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
   [2016-05-11 11:21:44,155] WARN Error while fetching metadata with correlation id 16 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
   [2016-05-11 11:21:44,257] WARN Error while fetching metadata with correlation id 17 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
   [2016-05-11 11:21:44,359] WARN Error while fetching metadata with correlation id 18 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
   [2016-05-11 11:21:44,462] WARN Error while fetching metadata with correlation id 19 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
   

more debug steps:

1. port detection

   telnet 192.168.1.100 32785
   Trying 192.168.1.100...
   Connected to 192.168.1.100.
   Escape character is '^]'.
   
   
   Connection closed by foreign host.
   

2. zookeeper client:

   ls /
   [consumers, config, controller, isr_change_notification, admin, brokers, zookeeper, controller_epoch]
   ls /brokers 
   [seqid, topics, ids]
   ls /brokers/ids
   [1001]
   get /brokers/ids/1001
   {"jmx_port":-1,"timestamp":"1462872416451","endpoints":["PLAINTEXT://192.168.1.100:32785"],"host":"192.168.1.100","version":2,"port":32785}
   cZxid = 0x18
   ctime = Tue May 10 17:26:56 CST 2016
   mZxid = 0x18
   mtime = Tue May 10 17:26:56 CST 2016
   pZxid = 0x18
   cversion = 0
   dataVersion = 0
   aclVersion = 0
   ephemeralOwner = 0x15499fd99150000
   dataLength = 139
   numChildren = 0
   ls /config 
   [topics, clients, changes]
   ls /config/topics
   [test]
   get /config/topics/test
   {"version":1,"config":{}}
   cZxid = 0x1e
   ctime = Tue May 10 17:30:00 CST 2016
   mZxid = 0x1e
   mtime = Tue May 10 17:30:00 CST 2016
   pZxid = 0x1e
   cversion = 0
   dataVersion = 0
   aclVersion = 0
   ephemeralOwner = 0x0
   dataLength = 25
   numChildren = 0
   

[pluswave]

some clue about this issue for my case:

inside the kafka server container (please note I just restart the service so the port mapping changed):

docker exec -it kafkadocker_kafka_1 bash
bash-4.3# netstat -anpt
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 :::33304                :::*                    LISTEN      103/java
tcp        0      0 :::9092                 :::*                    LISTEN      103/java
tcp        0      1 ::ffff:172.17.0.8:46190 ::ffff:192.168.1.100:32791 SYN_SENT    103/java
tcp        0      0 ::ffff:172.17.0.8:57736 ::ffff:172.17.0.7:2181  ESTABLISHED 103/java

the SYN_SENT indicate the cause (but not root cause). from the kafka server (docker instance), it can't connect to DockerHost:ForwardPort to any kafka server in cluster. Instead, it can only connect to some BridgedIp:9092 but it is NOT advertised by zookeeper. However from outside world, the bridged network is not visible but only fowarded port and docker host.

The problem now is : how to fix it ?

[pluswave]

ok should be this issue: moby/moby#16137

[wurstmeister]

Hi,

sorry for the late reply, this could indeed be a firewall issue. sounds very similar to #30

@madfrog2047 the output of broker-list.sh depends on how you started start-kafka-shell.sh, which is where it is supposed to be used (see https://github.com/wurstmeister/kafka-docker/blob/master/broker-list.sh#L4)

[bikeholik]

I've got it working disabling firewalld completely (on Fedora 23) for test purposes...

[lidel]

@madfrog2047 Just out of curiosity, did you set a (custom) low tickTime in ZooKeeper by any chance? I had the same error with tickTime=10 and it is gone with tickTime=500 (embedded Kafka+ZooKeeper in JUnits).

[junjiemars]

set advertised.host.name=<container's-ip> in Kafka's server.properties solve my problem.

[Hubbitus]

It seams related to moby/moby#26594

As workaround you may use HOSTNAME_COMMAND instead of providing IP statically.
So f.e. If I add in environment section of docker-compose.yml file line:

HOSTNAME_COMMAND: "getent hosts `hostname` | cut -d' ' -f1"

it works for me.

[ppromsiri]

Hi All

I found workaround
add KAFKA_ADVERTISED_LISTENERS:
in section environment docker-compose.yml

it works. Let's try

[jeven2016]

yes, the KAFKA_ADVERTISED_LISTENERS is very useful, and following is my yml file for reference:

version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "5804:2181"
kafka:
image: wurstmeister/kafka
ports:
- "5805:9092"
environment:
KAFKA_ADVERTISED_LISTENERS: 10.196.10.6:9092
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock

[LuisIvanMD]

Hi Guys,

Had the same issue as you, and disabling the Firewall worked fine for me. Did anyone went further on searching which rule we should add to Firewall and be able to leave it active?

Thanks a lot.

Thar works for me

[JOJO0527]

I've got it working disabling firewalld completely (on Fedora 23) for test purposes...

it works for me too! thanks