ocp stops to work after rebooting the nodes about dev-scripts

[russellb]

@eparis is there a link to an issue tracking the remaining crio issue?

The DNS fixes merged here:

#382
openshift-metal3/kni-installer#50

[e-minguez]

Bumping this as it is very relevant for demo purposes

[honza]

Is there a way to recover from this?

[stack@dell-pe-fc630-02 dev-scripts]$ oc status
Unable to connect to the server: EOF
Unable to connect to the server: EOF
Unable to connect to the server: EOF
Unable to connect to the server: EOF
Unable to connect to the server: EOF
Unable to connect to the server: EOF
Unable to connect to the server: EOF
Unable to connect to the server: EOF
Unable to connect to the server: EOF
Unable to connect to the server: EOF
Unable to connect to the server: EOF
Unable to connect to the server: EOF
Unable to connect to the server: EOF
Unable to connect to the server: EOF
Unable to connect to the server: EOF
Unable to connect to the server: EOF
Unable to connect to the server: EOF
[stack@dell-pe-fc630-02 dev-scripts]$ 

[hardys]

Anyone have more details on the state of the cluster post-reboot? Just saying it doesn't work isn't a super helpful starting point.

[e-minguez]

I've just run some tests in my environment:

NAME                                       STATUS   ROLES    AGE     VERSION
kni1-c13u23.cloud.lab.eng.bos.redhat.com   Ready    master   4h17m   v1.13.4+1ad602308
kni1-c13u25.cloud.lab.eng.bos.redhat.com   Ready    master   4h17m   v1.13.4+1ad602308
kni1-c13u27.cloud.lab.eng.bos.redhat.com   Ready    master   4h17m   v1.13.4+1ad602308

• I've just rebooted the first master, then the oc commands complained about not reaching the api... eventually the first master came back and everything seems to be working fine.

• I've powered off the 3 masters via ssh $host sudo poweroff. The hosts were down, but someone powered on again (I'm assuming ironic). It seems to work as well.

• I've powered off badly (as in force power off via the idrac). Someone powered on them again and now the cluster doesn't seem to work:

$ oc get nodes
The connection to the server api.kni1.cloud.lab.eng.bos.redhat.com:6443 was refused - did you specify the right host or port

Log from one of the masters:

Apr 17 15:31:25 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:25.421662    4463 kubelet.go:2273] node "kni1-c13u27.cloud.lab.eng.bos.redhat.com" not found
Apr 17 15:31:25 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:25.521846    4463 kubelet.go:2273] node "kni1-c13u27.cloud.lab.eng.bos.redhat.com" not found
Apr 17 15:31:25 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:25.606649    4463 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://api.kni1.cloud.lab.eng.bos.redhat.com:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dkni1-c13u27.cloud.lab.eng.bos.redhat.com&limit=500&resourceVersion=0: dial tcp 10.19.138.14:6443: connect: connection refused
Apr 17 15:31:25 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:25.622005    4463 kubelet.go:2273] node "kni1-c13u27.cloud.lab.eng.bos.redhat.com" not found
Apr 17 15:31:25 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:25.632975    4463 event.go:212] Unable to write event: 'Patch https://api.kni1.cloud.lab.eng.bos.redhat.com:6443/api/v1/namespaces/default/events/kni1-c13u27.cloud.lab.eng.bos.redhat.com.15964c113cbe6e98: dial tcp 10.19.138.14:6443: connect: connection refused' (may retry after sleeping)
Apr 17 15:31:25 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:25.722234    4463 kubelet.go:2273] node "kni1-c13u27.cloud.lab.eng.bos.redhat.com" not found
Apr 17 15:31:25 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:25.822392    4463 kubelet.go:2273] node "kni1-c13u27.cloud.lab.eng.bos.redhat.com" not found
Apr 17 15:31:25 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:25.828130    4463 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://api.kni1.cloud.lab.eng.bos.redhat.com:6443/api/v1/nodes?fieldSelector=metadata.name%3Dkni1-c13u27.cloud.lab.eng.bos.redhat.com&limit=500&resourceVersion=0: dial tcp 10.19.138.14:6443: connect: connection refused
Apr 17 15:31:25 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:25.922566    4463 kubelet.go:2273] node "kni1-c13u27.cloud.lab.eng.bos.redhat.com" not found
Apr 17 15:31:26 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:26.022742    4463 kubelet.go:2273] node "kni1-c13u27.cloud.lab.eng.bos.redhat.com" not found
Apr 17 15:31:26 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: I0417 15:31:26.095019    4463 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Apr 17 15:31:26 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:26.098531    4463 remote_runtime.go:96] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = error creating pod sandbox with name "k8s_kube-controller-manager-kni1-c13u27.cloud.lab.eng.bos.redhat.com_openshift-kube-controller-manager_156b61a1b4b9f09c24cacfcc5a27cd01_2": error creating read-write layer with ID "5e103aec20ff53693096de2e6de5ae9ecc4212fc0744325087586e8bfc627274": symlink ../5e103aec20ff53693096de2e6de5ae9ecc4212fc0744325087586e8bfc627274/diff /var/lib/containers/storage/overlay/l/QYU2BPJYRLXBYZE4URGDJA2MKS: no such file or directory
Apr 17 15:31:26 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:26.098568    4463 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "kube-controller-manager-kni1-c13u27.cloud.lab.eng.bos.redhat.com_openshift-kube-controller-manager(156b61a1b4b9f09c24cacfcc5a27cd01)" failed: rpc error: code = Unknown desc = error creating pod sandbox with name "k8s_kube-controller-manager-kni1-c13u27.cloud.lab.eng.bos.redhat.com_openshift-kube-controller-manager_156b61a1b4b9f09c24cacfcc5a27cd01_2": error creating read-write layer with ID "5e103aec20ff53693096de2e6de5ae9ecc4212fc0744325087586e8bfc627274": symlink ../5e103aec20ff53693096de2e6de5ae9ecc4212fc0744325087586e8bfc627274/diff /var/lib/containers/storage/overlay/l/QYU2BPJYRLXBYZE4URGDJA2MKS: no such file or directory
Apr 17 15:31:26 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:26.098582    4463 kuberuntime_manager.go:661] createPodSandbox for pod "kube-controller-manager-kni1-c13u27.cloud.lab.eng.bos.redhat.com_openshift-kube-controller-manager(156b61a1b4b9f09c24cacfcc5a27cd01)" failed: rpc error: code = Unknown desc = error creating pod sandbox with name "k8s_kube-controller-manager-kni1-c13u27.cloud.lab.eng.bos.redhat.com_openshift-kube-controller-manager_156b61a1b4b9f09c24cacfcc5a27cd01_2": error creating read-write layer with ID "5e103aec20ff53693096de2e6de5ae9ecc4212fc0744325087586e8bfc627274": symlink ../5e103aec20ff53693096de2e6de5ae9ecc4212fc0744325087586e8bfc627274/diff /var/lib/containers/storage/overlay/l/QYU2BPJYRLXBYZE4URGDJA2MKS: no such file or directory
Apr 17 15:31:26 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:26.098629    4463 pod_workers.go:190] Error syncing pod 156b61a1b4b9f09c24cacfcc5a27cd01 ("kube-controller-manager-kni1-c13u27.cloud.lab.eng.bos.redhat.com_openshift-kube-controller-manager(156b61a1b4b9f09c24cacfcc5a27cd01)"), skipping: failed to "CreatePodSandbox" for "kube-controller-manager-kni1-c13u27.cloud.lab.eng.bos.redhat.com_openshift-kube-controller-manager(156b61a1b4b9f09c24cacfcc5a27cd01)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-controller-manager-kni1-c13u27.cloud.lab.eng.bos.redhat.com_openshift-kube-controller-manager(156b61a1b4b9f09c24cacfcc5a27cd01)\" failed: rpc error: code = Unknown desc = error creating pod sandbox with name \"k8s_kube-controller-manager-kni1-c13u27.cloud.lab.eng.bos.redhat.com_openshift-kube-controller-manager_156b61a1b4b9f09c24cacfcc5a27cd01_2\": error creating read-write layer with ID \"5e103aec20ff53693096de2e6de5ae9ecc4212fc0744325087586e8bfc627274\": symlink ../5e103aec20ff53693096de2e6de5ae9ecc4212fc0744325087586e8bfc627274/diff /var/lib/containers/storage/overlay/l/QYU2BPJYRLXBYZE4URGDJA2MKS: no such file or directory"
Apr 17 15:31:26 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:26.122934    4463 kubelet.go:2273] node "kni1-c13u27.cloud.lab.eng.bos.redhat.com" not found
Apr 17 15:31:26 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:26.223097    4463 kubelet.go:2273] node "kni1-c13u27.cloud.lab.eng.bos.redhat.com" not found
Apr 17 15:31:26 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:26.323295    4463 kubelet.go:2273] node "kni1-c13u27.cloud.lab.eng.bos.redhat.com" not found
Apr 17 15:31:26 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:26.396109    4463 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://api.kni1.cloud.lab.eng.bos.redhat.com:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.19.138.14:6443: connect: connection refused
Apr 17 15:31:26 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:26.423446    4463 kubelet.go:2273] node "kni1-c13u27.cloud.lab.eng.bos.redhat.com" not found
Apr 17 15:31:26 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:26.523636    4463 kubelet.go:2273] node "kni1-c13u27.cloud.lab.eng.bos.redhat.com" not found
Apr 17 15:31:26 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:26.608100    4463 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://api.kni1.cloud.lab.eng.bos.redhat.com:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dkni1-c13u27.cloud.lab.eng.bos.redhat.com&limit=500&resourceVersion=0: dial tcp 10.19.138.14:6443: connect: connection refused
Apr 17 15:31:26 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:26.623782    4463 kubelet.go:2273] node "kni1-c13u27.cloud.lab.eng.bos.redhat.com" not found
Apr 17 15:31:26 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:26.723974    4463 kubelet.go:2273] node "kni1-c13u27.cloud.lab.eng.bos.redhat.com" not found
Apr 17 15:31:26 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:26.824166    4463 kubelet.go:2273] node "kni1-c13u27.cloud.lab.eng.bos.redhat.com" not found
Apr 17 15:31:26 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:26.829598    4463 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://api.kni1.cloud.lab.eng.bos.redhat.com:6443/api/v1/nodes?fieldSelector=metadata.name%3Dkni1-c13u27.cloud.lab.eng.bos.redhat.com&limit=500&resourceVersion=0: dial tcp 10.19.138.14:6443: connect: connection refused
Apr 17 15:31:26 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[4463]: E0417 15:31:26.924351    4463 kubelet.go:2273] node "kni1-c13u27.cloud.lab.eng.bos.redhat.com" not found

[eparis]

can you get me root access to this cluster?

[e-minguez]

can you get me root access to this cluster?

I've sent you some details on how to access via slack

discovery pod logs in one of the nodes say:

[core@kni1-c13u23 ~]$ sudo crictl ps
CONTAINER ID        IMAGE                                                              CREATED             STATE               NAME                           ATTEMPT             POD ID
fcae11b0163ab       32ffea1721e507f11d5506dd2c2d012a9a6c3023a942f9e009fdd56e3f322b9f   4 minutes ago       Running             discovery                      5                   52e95bc3fe2fc
fdbd15b24140f       79a9c9b3d7973b37c81a04ebced1b26e4bd9c666e4c44448bc9e8b52d2f24bbb   15 minutes ago      Running             haproxy-monitor                3                   c658c765680f5
a6450f8989816       349cbf135b12d324dada5e31f35c0fb7ca29032478c6aa88e04965103f054181   15 minutes ago      Running             mdns-publisher                 3                   42e8b4f16150c
8778656666b78       f3abd21f393a9b51c06e56e5e5793d69195a356ef7e81786235df634f7131b8a   15 minutes ago      Running             haproxy                        3                   c658c765680f5
5c79571695fb0       d300f711c1da836443b2f6dbb1c5ad1ff536dfefd44f231a03e1a4853d6c2269   15 minutes ago      Running             coredns                        3                   cdd1f4f1d662c
f6b8c30750ef1       d85ffd381ce78d7a2b7fbd6e756ebcf4bf2c716d09ec7bd6d82a7967819878df   15 minutes ago      Running             keepalived                     3                   b685615fa0538
17b3aba0ccb2e       f6d29eaf3c98d7438ef8b3a45bbbc37e6ba3cbaa3bafffb81c7b8e2a56de1929   15 minutes ago      Running             kube-apiserver-cert-syncer-6   3                   b6314cac24553
9afa9180993cb       d29efa50d761998a3e38d2a802565fb722c9bbf4c54293fdea2ab447bdf1e713   15 minutes ago      Running             scheduler                      6                   f9995d04b2aad
[core@kni1-c13u23 ~]$ crictl logs fcae11b0163ab
E0417 15:35:02.228512   18880 remote_runtime.go:282] ContainerStatus "fcae11b0163ab" from runtime service failed: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial unix /var/run/crio/crio.sock: connect: permission denied"
FATA[0000] rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial unix /var/run/crio/crio.sock: connect: permission denied" 
[core@kni1-c13u23 ~]$ sudo crictl logs fcae11b0163ab
I0417 15:30:09.419439       1 run.go:46] Version: 4.0.0-alpha.0-172-g22679b27-dirty
E0417 15:30:09.421476       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:30:09.422738       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:30:09.423692       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:30:39.426093       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:30:39.427446       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:30:39.428404       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:31:09.425194       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:31:09.427320       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:31:09.428449       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:31:39.425121       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:31:39.426087       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:31:39.427036       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:32:09.425993       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:32:09.427033       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:32:09.427947       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:32:39.425071       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:32:39.425934       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:32:39.427566       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:33:09.425358       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:33:09.426418       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:33:09.427532       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:33:39.425883       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:33:39.426821       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:33:39.427637       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:34:09.425504       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:34:09.427276       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:34:09.428309       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:34:39.425541       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:34:39.426684       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host
E0417 15:34:39.428150       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.kni1.cloud.lab.eng.bos.redhat.com on 10.19.138.16:53: no such host

[eparis]

@e-minguez got me access to his failed cluster. We found 3 different crio bugs (we know that 1 of them is fixed in 1.13.6 or later, cluster was on 1.13.3.) 1 of them we couldn't figure out, it went away when we turned on debugging. 1 of them we were able to identify and will work through.

we also found a problem in the cluster dns configuration which is being worked.

if there are future problems they probably should be a different issue and they should include at least the output of crictl pods, crictl ps, and the logs for cio and kubelet.

[russellb]

I just rebooted one of our bare metal clusters twice and it came up successfully both times. I think we can close this for now and reopen or file new issues if further issues come up.