Kubernetes cluster upgrade fails with error " FATA[00XX] Failed to set cordonded state for node" about rke

[galal-hussein]

Cordon function needed to have sleep between retries in case the kubernetes api wasn't ready yet after deployment

[soumyalj]

Tested with rke version v0.0.4-dev.
Created a Kubernetes cluster with version rancher/k8s:v1.8.2-rancher5.
Created pods and then upgraded to version rancher/k8s:v1.8.3-rancher2
Upgrade was successful . All the kubernetes components upgraded to version rancher/k8s:v1.8.3-rancher2. Creation of new pods was ok. Dashboard launched successfully.

soumyas-MBP:rke soumya$ ./rke_darwin-amd64 cluster upgrade --cluster-file clusterdigworking.yml
INFO[0000] Upgrading Kubernetes cluster                 
INFO[0000] [state] Found local kube config file, trying to get state from cluster 
INFO[0000] [state] Fetching cluster state from Kubernetes 
INFO[0000] [state] Successfully Fetched cluster state to Kubernetes ConfigMap: cluster-state 
INFO[0000] [certificates] Getting Cluster certificates from Kubernetes 
INFO[0001] [certificates] Successfully fetched Cluster certificates from Kubernetes 
INFO[0001] [ssh] Checking private key                   
INFO[0001] [ssh] Start tunnel for host [soumyarketest1] 
INFO[0001] [ssh] Start tunnel for host [soumyarketest1] 
INFO[0001] [ssh] Start tunnel for host [soumyarketest3] 
INFO[0001] [ssh] Start tunnel for host [soumyarketes2]  
INFO[0001] [upgrade] All nodes are Ready                
INFO[0001] [upgrade] Upgrading Control Plane Services   
INFO[0001] [controlplane] Upgrading the Controller Plane.. 
INFO[0003] [controlplane] Successfully pulled kube-api image on host [soumyarketest1] 
INFO[0004] [controlplane] Successfully started kube-api container on host [soumyarketest1] 
INFO[0005] [controlplane] Successfully pulled kube-controller image on host [soumyarketest1] 
INFO[0005] [controlplane] Successfully started kube-controller container on host [soumyarketest1] 
INFO[0006] [controlplane] Successfully pulled scheduler image on host [soumyarketest1] 
INFO[0007] [controlplane] Successfully started scheduler container on host [soumyarketest1] 
INFO[0009] [controlplane] Successfully pulled kube-api image on host [soumyarketest3] 
INFO[0009] [controlplane] Successfully started kube-api container on host [soumyarketest3] 
INFO[0010] [controlplane] Successfully pulled kube-controller image on host [soumyarketest3] 
INFO[0011] [controlplane] Successfully started kube-controller container on host [soumyarketest3] 
INFO[0012] [controlplane] Successfully pulled scheduler image on host [soumyarketest3] 
INFO[0012] [controlplane] Successfully started scheduler container on host [soumyarketest3] 
INFO[0013] [controlplane] Successfully upgraded Controller Plane.. 
INFO[0013] [upgrade] Control Plane Services updgraded successfully 
INFO[0013] [upgrade] Upgrading Worker Plane Services    
INFO[0013] [worker] Upgrading Worker Plane..            
INFO[0014] [worker] Successfully pulled kubelet image on host [soumyarketest1] 
INFO[0015] [worker] Successfully started kubelet container on host [soumyarketest1] 
INFO[0016] [worker] Successfully pulled kube-proxy image on host [soumyarketest1] 
INFO[0016] [worker] Successfully started kube-proxy container on host [soumyarketest1] 
INFO[0018] [worker] Successfully pulled kubelet image on host [soumyarketest3] 
INFO[0018] [worker] Successfully started kubelet container on host [soumyarketest3] 
INFO[0020] [worker] Successfully pulled kube-proxy image on host [soumyarketest3] 
INFO[0020] [worker] Successfully started kube-proxy container on host [soumyarketest3] 
INFO[0023] [worker] Successfully pulled kubelet image on host [soumyarketes2] 
INFO[0023] [worker] Successfully started kubelet container on host [soumyarketes2] 
INFO[0024] [worker] Successfully pulled kube-proxy image on host [soumyarketes2] 
INFO[0025] [worker] Successfully started kube-proxy container on host [soumyarketes2] 
INFO[0025] [worker] Successfully upgraded Worker Plane.. 
INFO[0025] [upgrade] Worker Plane Services updgraded successfully 
INFO[0025] [certificates] Save kubernetes certificates as secrets 
INFO[0046] [certificates] Successfuly saved certificates as kubernetes secret [k8s-certs] 
INFO[0046] [state] Saving cluster state to Kubernetes   
INFO[0046] [state] Successfully Saved cluster state to Kubernetes ConfigMap: cluster-state 
INFO[0046] Cluster upgraded successfully