Unable to start using Hypriot OS about kubernetes-on-arm

[GrantGochnauer]

I did change the master.json config to run the securityContext as root instead of as privileged because with the default settings, I get this error on startup:

 file.go:123] Can't process config file "/etc/kubernetes/manifests-multi/master.json": invalid pod: [spec.containers[3].securityContext.privileged: forbidden '<*>(0x56e8af08)true']

Edit: I got past that error by adding the following option to launching the kublet:

--allow_privileged=true

so now I run:

/hyperkube kubelet --pod_infra_container_image="kubernetesonarm/pause" --api-servers=http://localhost:8080 --v=2 --address=0.0.0.0 --hostname-override=127.0.0.1 --enable-server --config=/etc/kubernetes/manifests-multi --allow_privileged=true

However back to original issue from top post and the errors from the api server are:

Flag --address has been deprecated, see --insecure-bind-address instead
I1203 11:27:54.520282       1 plugins.go:71] No cloud provider specified.
I1203 11:27:54.523890       1 master.go:368] Node port range unspecified. Defaulting to 30000-32767.
I1203 11:27:54.526562       1 master.go:395] Will report 192.168.1.140 as public IP address.
E1203 11:27:54.583778       1 cacher.go:149] unexpected ListAndWatch error: pkg/storage/cacher.go:115: Failed to list *api.Pod: 501: All the given peers are not reachable (failed to propose on members [http://127.0.0.1:4001] twice [last error: Get http://127.0.0.1:4001/v2/keys/registry/pods?quorum=false&recursive=true&sorted=true: dial tcp 127.0.0.1:4001: connection refused]) [0]

[GrantGochnauer]

So, etcd wasn't running :).

Edit:
Well the only remaining issue I see is the logs keep printing this:

 Failed to ensure state of "/docker-daemon": failed to find pid of Docker container: exit status 1
kubelet.go:1361] Failed creating a mirror pod "k8s-master-127.0.0.1_kube-system": namespaces "kube-system" not found

Using this command

curl http://127.0.0.1:4194/validate/

I was able to determine via this post: kubernetes/kubernetes#11733 that I needed to mount 2 additional volumes but still seeing error.

Current command is:

sudo docker run -d --name=kube --net=host -v /var/run/docker.sock:/var/run/docker.sock -v /sys:/sys:ro -v /var/lib/docker/:/var/lib/docker:ro kubernetesonarm/hyperkube /hyperkube kubelet --pod_infra_container_image="kubernetesonarm/pause" --api-servers=http://localhost:8080 --v=2 --address=0.0.0.0 --enable-server --hostname-override=127.0.0.1 --config=/etc/kubernetes/manifests-multi --allow_privileged=true --cloud-provider=""

[luxas]

See the systemd service files: https://github.com/luxas/kubernetes-on-arm/tree/master/sdcard/rootfs/kube-archlinux/usr/lib/systemd/system

There you'll find the right commands to use.

There the commands for starting the containers are. If you are running on many nodes, start etcd first in system-docker, then flannel in system-docker, then main docker with flannel settings, after that k8s-master/worker.

If you just test it on one node, just start etcd and k8s-master, as you said.

This can be quite scary with all these service and dependencies.
However, I'm probably going to make a .deb package for HypriotOS, so installing the whole system is just two commands away.

Hope it helps!

[GrantGochnauer]

Ah @luxas this is super helpful and makes a ton a sense. 👍 and I would totally second the deb package for HypriotOS!

[GrantGochnauer]

I've adapted your systemd settings but for some reason it appears that kuber isn't creating the kuber namespace properly. Did I miss another initialization step?

E1204 15:58:21.824083   25444 event.go:188] Server rejected event '&api.Event{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"k8s-master-127.0.0.1.141cc060b8cc4521", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, InvolvedObject:api.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"k8s-master-127.0.0.1", UID:"fd6de061dd407e666e5047943d7f91fc", APIVersion:"v1", ResourceVersion:"", FieldPath:"spec.containers{controller-manager}"}, Reason:"Started", Message:"Started with docker id 734d71a45bfa", Source:api.EventSource{Component:"kubelet", Host:"127.0.0.1"}, FirstTimestamp:unversioned.Time{Time:time.Time{sec:63584841501, nsec:756245281, loc:(*time.Location)(0x1caf0e8)}}, LastTimestamp:unversioned.Time{Time:time.Time{sec:63584841501, nsec:756245281, loc:(*time.Location)(0x1caf0e8)}}, Count:1}': 'namespaces "kube-system" not found' (will not retry!)
E1204 15:58:21.852363   25444 kubelet.go:1361] Failed creating a mirror pod "k8s-master-127.0.0.1_kube-system": namespaces "kube-system" not found
E1204 15:58:31.008328   25444 kubelet.go:1361] Failed creating a mirror pod "k8s-master-127.0.0.1_kube-system": namespaces "kube-system" not found
E1204 15:58:40.810850   25444 kubelet.go:1361] Failed creating a mirror pod "k8s-master-127.0.0.1_kube-system": namespaces "kube-system" not found
E1204 15:58:50.839258   25444 kubelet.go:1361] Failed creating a mirror pod "k8s-master-127.0.0.1_kube-system": namespaces "kube-system" not found
E1204 15:59:00.813521   25444 kubelet.go:1361] Failed creating a mirror pod "k8s-master-127.0.0.1_kube-system": namespaces "kube-system" not found

Which is weird since apparently this should work OOB: http://kubernetes.io/v1.0/docs/admin/namespaces.html

I manually ran this and it seemed to help:

kubectl create -f kube-system.yaml 

[luxas]

I have no idea why it says that kube-system is created automatically.
Only on GCE, I would guess. At least, it doesn't, as you noticed.
As a workaround, kube-config in this project automatically creates this.
So, you have to create it yourself.

However, I've managed to make a .deb package of this project!
I will maybe upload an experimental package of v0.6.0 and hope that I may support it in v0.6.2

[GrantGochnauer]

Awesome. You rock! Thank you.

[GrantGochnauer]

One other thing that may be related to running on Hypriot OS is that I see this in the logs every few mins:

Image garbage collection failed: non-existent label "docker-images"

[luxas]

This happens on all containerized kubelet enviroinments, even on my amd64 machine
It's a main kubernetes issue, and google folks are working on the best UX and behavior when using docker for spinning up k8s.

[GrantGochnauer]

More debugging, I'm noticing this when I spin up containers and my health checks (readiness probes) are failing and am not sure whether this is related:

hairpin.go:49] Unable to find pair interface, setting up all interfaces: exec: "ethtool"

I see that every time a new pod is created. I found this: openshift/origin#4843

[luxas]

Quite funny, I made an issue about this some hours ago: kubernetes/kubernetes#18251

It's just a line that must be changed: apt-get install ethtool
I've updated this to v0.6.2

I had noticed this one, but if you continue digging I'm sure you'll find something odd I haven't encountered 😄

[luxas]

Merry chrismas @GrantGochnauer!

Here comes a small chrismas present: v0.6.2 :)
I hope you like it.

I'm exited to hear what you will come up with.
Now HypriotOS is supported OOTB, so have fun

[GrantGochnauer]

This is amazing news!! Thank you so much for all your hard work. So you have any suggestions on migrating from the manual install to the deb installer? Can I just install the deb version on top of my already installed kuberentes? Thank you!

[luxas]

It's a little risky, but I think we can make it

I came up with an amazing scipt here for checking that:

cd /
wget https://github.com/luxas/kubernetes-on-arm/releases/download/v0.6.2/kube-systemd.deb
dpkg -c kube-systemd.deb | awk '{print $6}' | xargs ls -d 2>&1 | grep -v "No such file"

# If you want to install
sudo dpkg -i kube-systemd.deb

That script will output all files you have in your system that also the deb has => probably conficts
If you after the install want to check if you are missing some files, remove the -v in the grep command.

Try to resolve the conflicts if any, and enjoy 😄

[GrantGochnauer]

No conflicts. Worked like a charm!

[root:~]# kube-config info
Architecture: armv7l
Kernel: Linux 4.1.12
CPU: 4 cores x 900 MHz

Used RAM Memory: 613 MiB
RAM Memory: 925 MiB

Used disk space: 11GB (10735504 KB)
Free disk space: 17GB (17161224 KB)

SD Card/deb package was built: 28-12-2015 18:57

kubernetes-on-arm: 
Latest commit: 247134b
Version: 0.6.2

systemd version: v215
docker version: v1.9.1

[luxas]

I'm glad to hear that! Closing as fixed :)