k8s-worker doest not initialise correctly about kubernetes-on-arm

[luxas]

About the RPi 1: just type docker ps and systemctl status k8s-master
Why can't 10.0.0.10 be your DNS server?
In some cases, docker will fail strangely (resource limit) but come up some secs later
Could you provide the status of flannel? Docker failed because it didn't find the flannel subnet file:

Oct 25 00:32:48 thanos systemd[1]: docker.service: Failed to load environment files: No such file or directory
Oct 25 00:32:48 thanos systemd[1]: docker.service: Failed to run 'start-pre' task: No such file or directory 

And I guess that flannel failed because the master is down.
Please test to restart your master and provide output from there.

[nsteinmetz]

I said 10.0.0.10 does not exist on my LAN ; so unless it has some sense at k8s level, it has no sense for me ;-)

On master:

• docker ps does not return anything
• k8s-master was stopped and disabled

On worker:

[root@thanos ~]# systemctl status flannel -l
* flannel.service - Flannel Overlay Network for Kubernetes
   Loaded: loaded (/usr/lib/systemd/system/flannel.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2015-10-25 00:32:28 CEST; 17h ago
 Main PID: 615 (docker)
   CGroup: /system.slice/flannel.service
           `-615 /usr/bin/docker -H unix:///var/run/system-docker.sock run --name=k8s-flannel --net=host --privileged -v /dev/net:/dev/net -v /var/lib/flannel:/run/flannel kubernetesonarm/flannel /flanneld --etcd-endpoints=http://192.168.8.110:4001

Oct 25 16:33:52 thanos docker[615]: E1025 15:33:52.984270 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Oct 25 16:33:53 thanos docker[615]: E1025 15:33:53.986982 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Oct 25 16:33:54 thanos docker[615]: E1025 15:33:54.989621 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Oct 25 16:33:55 thanos docker[615]: E1025 15:33:55.992184 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Oct 25 16:33:56 thanos docker[615]: E1025 15:33:56.994727 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Oct 25 16:33:57 thanos docker[615]: E1025 15:33:57.997252 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Oct 25 16:33:59 thanos docker[615]: E1025 15:33:58.999962 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Oct 25 16:34:00 thanos docker[615]: E1025 15:34:00.002492 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Oct 25 16:34:01 thanos docker[615]: E1025 15:34:01.004956 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Oct 25 16:34:02 thanos docker[615]: E1025 15:34:02.007474 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured

Back on master :

[root@kubepi ~]# systemctl start k8s-master
Job for k8s-master.service failed because the control process exited with error code. See "systemctl status k8s-master.service" and "journalctl -xe" for details.
[root@kubepi ~]# systemctl status k8s-master
* k8s-master.service - The Master Components for Kubernetes
   Loaded: loaded (/usr/lib/systemd/system/k8s-master.service; enabled; vendor preset: disabled)
   Active: activating (start-post) since Sun 2015-10-25 17:35:49 EET; 6s ago
  Process: 4780 ExecStop=/usr/bin/docker stop k8s-master k8s-worker-proxy (code=exited, status=1/FAILURE)
  Process: 4792 ExecStartPre=/usr/bin/docker rm k8s-master k8s-worker-proxy (code=exited, status=1/FAILURE)
  Process: 4786 ExecStartPre=/usr/bin/docker kill k8s-master k8s-worker-proxy (code=exited, status=1/FAILURE)
 Main PID: 4798 (docker);         : 4799 (docker)
   CGroup: /system.slice/k8s-master.service
           |-4798 docker run --name=k8s-master --net=host -v /var/run/docker.sock:/var/run/docker.sock kubernetesonarm/hyperkube /hyperkube kubelet --pod_infra_container_ima...
           `-control
             `-4799 /usr/bin/docker run -d --name=k8s-worker-proxy --net=host --privileged kubernetesonarm/hyperkube /hyperkube proxy --master=http://localhost:8080 --v=2

Oct 25 17:35:50 kubepi docker[4786]: Error response from daemon: no such id: k8s-master
Oct 25 17:35:50 kubepi docker[4786]: Error response from daemon: no such id: k8s-worker-proxy
Oct 25 17:35:50 kubepi docker[4786]: Error: failed to kill containers: [k8s-master k8s-worker-proxy]
Oct 25 17:35:51 kubepi docker[4792]: Error response from daemon: no such id: k8s-master
Oct 25 17:35:52 kubepi docker[4792]: Error response from daemon: no such id: k8s-worker-proxy
Oct 25 17:35:52 kubepi docker[4792]: Error: failed to remove containers: [k8s-master k8s-worker-proxy]
Oct 25 17:35:53 kubepi docker[4799]: Unable to find image 'kubernetesonarm/hyperkube:latest' locally
Oct 25 17:35:53 kubepi docker[4799]: Repository kubernetesonarm/hyperkube already being pulled by another client. Waiting.
Oct 25 17:35:53 kubepi bash[4798]: Unable to find image 'kubernetesonarm/hyperkube:latest' locally
Oct 25 17:35:53 kubepi bash[4798]: Repository kubernetesonarm/hyperkube already being pulled by another client. Waiting.

[nsteinmetz]

So, on master, I'm pulling right now the images. I thought I did previously ?! o_O

[nsteinmetz]

Hmm seems images were not correctly pulled, I had first:

[root@kubepi ~]# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
<none>              <none>              e2cd940b564c        11 days ago         125.5 MB
<none>              <none>              234f6772e792        11 days ago         86.22 MB

Which became afer pulling again etcd:

[root@kubepi ~]# docker images
REPOSITORY                  TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
kubernetesonarm/hyperkube   latest              fd5c7305c597        11 days ago         125.5 MB
kubernetesonarm/etcd        latest              cc678c339590        11 days ago         18.2 MB

Will go on and let you know.

[nsteinmetz]

I rebooted my RPI1 (master) and Cubie (Worker).

On master side:

[root@kubepi ~]# systemctl status k8s-master
* k8s-master.service - The Master Components for Kubernetes
   Loaded: loaded (/usr/lib/systemd/system/k8s-master.service; enabled; vendor preset: disabled)
   Active: activating (start-pre) since Sun 2015-10-25 17:53:10 EET; 1min 43s ago
  Control: 179 (docker)
   CGroup: /system.slice/k8s-master.service
           `-control
             `-179 /usr/bin/docker kill k8s-master k8s-worker-proxy

Oct 25 17:53:10 kubepi systemd[1]: Starting The Master Components for Kubernetes...
[root@kubepi ~]# systemctl status k8s-master
* k8s-master.service - The Master Components for Kubernetes
   Loaded: loaded (/usr/lib/systemd/system/k8s-master.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: timeout) since Sun 2015-10-25 17:55:21 EET; 3s ago
  Process: 300 ExecStartPre=/usr/bin/docker rm k8s-master k8s-worker-proxy (code=exited, status=1/FAILURE)
  Process: 179 ExecStartPre=/usr/bin/docker kill k8s-master k8s-worker-proxy (code=exited, status=2)

Oct 25 17:55:21 kubepi docker[300]: Error response from daemon: no such id: k8s-worker-proxy
Oct 25 17:55:21 kubepi docker[300]: Error: failed to remove containers: [k8s-master k8s-worker-proxy]
Oct 25 17:55:21 kubepi systemd[1]: Failed to start The Master Components for Kubernetes.
Oct 25 17:55:21 kubepi systemd[1]: k8s-master.service: Unit entered failed state.
Oct 25 17:55:21 kubepi systemd[1]: k8s-master.service: Failed with result 'timeout'.
[root@kubepi ~]# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[root@kubepi ~]# docker images
REPOSITORY                  TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
kubernetesonarm/flannel     latest              323b7a93d03b        11 days ago         90.25 MB
kubernetesonarm/pause       latest              22e22d3123c0        11 days ago         227 kB
kubernetesonarm/hyperkube   latest              fd5c7305c597        11 days ago         125.5 MB
kubernetesonarm/etcd        latest              cc678c339590        11 days ago         18.2 MB

On worker node side, I disabled the node and could see that etcd was not on the worker node.

root@thanos ~]# kube-config disable-node
Removed symlink /etc/systemd/system/multi-user.target.wants/k8s-worker.service.
Removed symlink /etc/systemd/system/multi-user.target.wants/flannel.service.
[root@thanos ~]# docker images
REPOSITORY                  TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
kubernetesonarm/flannel     latest              323b7a93d03b        11 days ago         90.25 MB
kubernetesonarm/pause       latest              22e22d3123c0        11 days ago         227 kB
kubernetesonarm/hyperkube   latest              fd5c7305c597        11 days ago         125.5 MB
[root@thanos ~]# docker pull kubernetesonarm/etcd
[...]
Status: Downloaded newer image for kubernetesonarm/etcd:latest
[root@thanos ~]# docker images
REPOSITORY                  TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
kubernetesonarm/flannel     latest              323b7a93d03b        11 days ago         90.25 MB
kubernetesonarm/pause       latest              22e22d3123c0        11 days ago         227 kB
kubernetesonarm/hyperkube   latest              fd5c7305c597        11 days ago         125.5 MB
kubernetesonarm/etcd        latest              cc678c339590        11 days ago         18.2 MB

[nsteinmetz]

What is funny as I try to set up a master both on rpi and cubietruck to see if any differences.

On cubietruck:

[root@thanos ~]# kube-config enable-master
Transferring images to system-docker
Copying kubernetesonarm/etcd to system-docker
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/flannel.service to /usr/lib/systemd/system/flannel.service.
Job for etcd.service failed because the control process exited with error code. See "systemctl status etcd.service" and "journalctl -xe" for details.
Job for docker.service failed because a configured resource limit was exceeded. See "systemctl status docker.service" and "journalctl -xe" for details.
Starting the master containers
Created symlink from /etc/systemd/system/multi-user.target.wants/k8s-master.service to /usr/lib/systemd/system/k8s-master.service.
Job for k8s-master.service failed because the control process exited with error code. See "systemctl status k8s-master.service" and "journalctl -xe" for details.
Master Kubernetes services enabled
[root@thanos ~]# systemctl status etcd -l
* etcd.service - Etcd Master Data Store for Kubernetes Apiserver
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sun 2015-10-25 17:03:27 CET; 4min 4s ago
 Main PID: 1897 (code=exited, status=0/SUCCESS)

Oct 25 17:03:26 thanos docker[1897]: 2015/10/25 16:03:26 raft: newRaft ce2a822cea30bfca [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
Oct 25 17:03:26 thanos docker[1897]: 2015/10/25 16:03:26 raft: ce2a822cea30bfca became follower at term 1
Oct 25 17:03:26 thanos docker[1897]: 2015/10/25 16:03:26 etcdserver: added local member ce2a822cea30bfca [http://localhost:2380 http://localhost:7001] to cluster 7e27652122e8b2ae
Oct 25 17:03:26 thanos bash[1898]: Error:  501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 25 17:03:27 thanos systemd[1]: etcd.service: Control process exited, code=exited status=4
Oct 25 17:03:27 thanos docker[1897]: 2015/10/25 16:03:27 received terminated signal, shutting down...
Oct 25 17:03:27 thanos docker[1938]: k8s-etcd
Oct 25 17:03:27 thanos systemd[1]: Failed to start Etcd Master Data Store for Kubernetes Apiserver.
Oct 25 17:03:27 thanos systemd[1]: etcd.service: Unit entered failed state.
Oct 25 17:03:27 thanos systemd[1]: etcd.service: Failed with result 'exit-code'.
[root@thanos ~]# kube-config disable-node
Removed symlink /etc/systemd/system/multi-user.target.wants/k8s-master.service.
Removed symlink /etc/systemd/system/multi-user.target.wants/etcd.service.
Removed symlink /etc/systemd/system/multi-user.target.wants/flannel.service.
[root@thanos ~]# docker images
REPOSITORY                  TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
kubernetesonarm/flannel     latest              323b7a93d03b        11 days ago         90.25 MB
kubernetesonarm/pause       latest              22e22d3123c0        11 days ago         227 kB
kubernetesonarm/hyperkube   latest              fd5c7305c597        11 days ago         125.5 MB
kubernetesonarm/etcd        latest              cc678c339590        11 days ago         18.2 MB

Wheras on RPI1:

[root@kubepi ~]# kube-config disable-node
[root@kubepi ~]# kube-config enable-master
Transferring images to system-docker
Copying kubernetesonarm/etcd to system-docker
Copying kubernetesonarm/flannel to system-docker

Can I check what is in system-docker to see if it's normal that in this case both etcd and flannel are copied whereas in first case, only etcd is ?

[nsteinmetz]

Nevermind, on master, I can't run it on armv6 due to GOARM v6 vs v7 issue:

[root@kubepi ~]# systemctl status flannel
* flannel.service - Flannel Overlay Network for Kubernetes
   Loaded: loaded (/usr/lib/systemd/system/flannel.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sun 2015-10-25 18:16:59 EET; 5min ago

Oct 25 18:16:27 kubepi docker[2603]: Error response from daemon: no such id: k8s-flannel
Oct 25 18:16:27 kubepi docker[2603]: Error: failed to kill containers: [k8s-flannel]
Oct 25 18:16:28 kubepi docker[2609]: Error response from daemon: no such id: k8s-flannel
Oct 25 18:16:28 kubepi docker[2609]: Error: failed to remove containers: [k8s-flannel]
Oct 25 18:16:59 kubepi bash[2640]: runtime: this CPU has no VFPv3 floating point hardware, so it cannot run
Oct 25 18:16:59 kubepi bash[2640]: this GOARM=7 binary. Recompile using GOARM=6.
Oct 25 18:16:59 kubepi systemd[1]: flannel.service: Control process exited, code=exited status=1
Oct 25 18:16:59 kubepi systemd[1]: Failed to start Flannel Overlay Network for Kubernetes.
Oct 25 18:16:59 kubepi systemd[1]: flannel.service: Unit entered failed state.
Oct 25 18:16:59 kubepi systemd[1]: flannel.service: Failed with result 'exit-code'.

So I'll stop using my RPI1 so far as master and will focus on Cubietruck as master.

[nsteinmetz]

As this issue is going in many directions, not sure we'll get something out of this thread. Maybe it's better to close it and re-start from scratch.

[luxas]

Yeah, it might be best. Try to make a clean SD Card of latest dev branch.