Deployment times out: Still waiting for the Kubernetes API: Get https://api.ostest.test.metalkube.org:6443/version?timeout=32s: dial tcp 192.168.123.5:6443: connect: connection refused about dev-scripts

[mcornea]

Querying the DNS VIP it returns the records for the api, dns and routes but not the SRV records:

[root@master-0 core]# dig +short @192.168.123.6 _etcd-server-ssl._tcp.ostest.test.metalkube.org SRV
[root@master-0 core]# dig +short @192.168.123.6 api.ostest.test.metalkube.org
192.168.123.5
[root@master-0 core]# dig +short @192.168.123.6 ns1.ostest.test.metalkube.org
192.168.123.6
[root@master-0 core]# dig +short @192.168.123.6 test.apps.ostest.test.metalkube.org
192.168.123.10

[knowncitizen]

I am also seeing this. Eventually:

level=debug msg="Still waiting for the Kubernetes API: Get https://api.ostest.test.metalkube.org:6443/version?timeout=32s: dial tcp 192.168.111.5:6443: connect: connection refused"
level=fatal msg="waiting for Kubernetes API: context deadline exceeded"
make: *** [ocp_run] Error 1
[jrist@galena dev-scripts]$

[dantrainor]

More logs from node:

Apr 09 20:50:40 master-0 hyperkube[1440]: W0409 20:50:40.455130 1440 status_manager.go:485] Failed to get status for pod "etcd-member-master-0_kube-system(82da9eebb7982583107992e428ae5e94)": Get https://api.ostest.test.metalkube.org:6443/api/v1/namespaces/kube-system/pods/etcd-member-master-0: dial tcp 192.168.111.5:6443: connect: connection refused

[dantrainor]

Apr 09 20:51:38 master-0 hyperkube[1440]: E0409 20:51:38.269063 1440 kubelet.go:2199] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config uninitialized

[dantrainor]

discovery/0.log:

2019-04-09T20:50:39.514546958+00:00 stderr F F0409 20:50:39.514517 1 main.go:30] Error executing etcd-setup-environment: could not find self: timed out waiting for the condition

[russellb]

I logged in to @dantrainor 's env. Indeed, the SRV record wasn't present. Interestingly though, master-0, master-1, and master-2 all resolve, which all require coredns-mdns to be working.

[dantrainor]

I'm closing out the tmate session but will offer it up to anyone else who wants to jump in and take a look. I won't change the environment from its current broken state, for now.

[dantrainor]

Did a test on a system with more memory because there was some talk earlier about OOM but that seemed to have no effect. Failures happen at the same place.

[dantrainor]

#322 will likely fix this. Testing locally a few times.