KubeDNS consistently dies about dns

[bowei]

This sounds like an issue with flannel connectivity.

kube-dns runs a set of health checks to both dnsmasq and the kube-dns component. If these health checks fail repeatedly, then the pod will be restarted. What I would do is run tcpdump and netcat to see if your packets are being delivered properly.

Regarding the SIGKILL, the behavior is to enable a graceful termination of the DNS service. When the pod is being torn-down, the service is first removed from the cluster IP VIP and sent a SIGTERM, then after the graceful termination period, a SIGKILL is sent.

[RaasAhsan]

It does seem like there's an issue with flannel. When KubeDNS starts on a non-master node, it can't hit the apiserver, which explains my first problem. I'll try to get that fixed up first, and see if that resolves the second problem.

[RaasAhsan]

So it seems like pods without host networking can communicate fine across the overlay network, but pods on worker machines can't hit pods that run use hostNetwork on the master. I'll just force the kubedns pod to run on the master for now until I can figure something else out.

Since kubedns is a critical service, I'll eventually have to figure out why it isn't working.

[RaasAhsan]

I managed to find kubernetes-sigs/kubespray#258 and setting advertise-address on the kube-apiserver fixed the problem where a pod on the node couldn't hit the apiserver on the master.

Its still failing health checks, so I'll get to fixing that.

[RaasAhsan]

Well, I figured out the second problem. The liveness health check for the kubedns pod is executed on a port that's exposed for sidecar. I had commented out the sidecar container when I was trying to debug the first problem. So the request was timing out and that's why it was consistently being restarted. Thanks for the help!