ramhiser / kafka-kubernetes Goto Github PK

Currently, our Zookeeper implementation uses 3 fixed Deployments and 3 corresponding fixed Services.

PetSets look like a promising direction to avoid hardcoding the nodes because they offer:

• a stable hostname, available in DNS
• an ordinal index
• stable storage: linked to the ordinal & hostname

Zookeeper is even mentioned in the PetSet docs. There's even a zookeeper example in kubernetes/contrib.

Currently, a DNS name must be hardcoded in kafka-cluster.yaml. Currently, I'm getting the DNS name from kafka-service after its deployed via kubectl get service kafka-service.

By removing the load balancer, there's no concern of exposing to the outside world (unless that's warranted by use case) and there's no need to hardcode the DNS host.

Hi,
I've cloned your repo and changed DNS with kubernetes-master, pods are running fine but kafka is generating the errors, that are given below
[2021-03-30 08:46:32,929] WARN [Controller id=1001, targetBrokerId=1001] Error connecting to node kubernetes-master:9092 (id: 1001 rack: null) (org.apache.kafka.clients.NetworkClient) java.net.UnknownHostException: kubernetes-master at java.net.InetAddress.getAllByName0(InetAddress.java:1281) at java.net.InetAddress.getAllByName(InetAddress.java:1193) at java.net.InetAddress.getAllByName(InetAddress.java:1127) at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110) at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:501) at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:458) at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:169) at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:1002) at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:311) at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:65) at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:292) at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:246) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Can you please help me out ??

In both Zookeeper and Kafka, I'm running replication controllers instead of deployments.

Why the update? From Kubernetes docs...

Replica Set is the next-generation Replication Controller.
...
we recommend using Deployments instead of directly using Replica Sets, unless you require custom update orchestration or don’t require updates at all.