CPU Usage on RPI1/Workers and Cubietruck/Master about kubernetes-on-arm

[luxas]

What program do you use for measuring?

[nsteinmetz]

htop on all machines.

[luxas]

13530 root      20   0  895.9m  14.4m   0.0  1.6   0:13.32 S  `- docker <-- system-docker service
13588 root      20   0  780.5m  23.8m   1.3  2.6   2:09.95 S      `- etcd 
13646 root      20   0  831.8m   7.6m   0.0  0.8   0:00.77 S      `- flanneld
13578 root      20   0  809.9m  11.6m   0.0  1.3   0:00.83 S  `- docker <-- etcd service
13641 root      20   0  810.9m  11.7m   0.0  1.3   0:00.85 S  `- docker <-- flannel service
13688 root      20   0 1465.2m  27.6m   0.6  3.0  14:33.08 S  `- docker <-- main docker
13850 root      20   0 1000.7m  66.4m 114.7  7.2 103:07.47 S      `- hyperkube <-- kubelet
14031 root      20   0    3.2m   1.5m   0.0  0.2   0:00.01 S          `- journalctl
14040 root      20   0  771.2m   1.4m   0.0  0.1   0:00.36 S      `- pause
14056 root      20   0  878.9m  27.8m   0.0  3.0   0:42.04 S      `- hyperkube <-- proxy
14116 root      20   0  886.2m  51.3m   0.0  5.6   3:32.45 S      `- hyperkube <-- apiserver
14131 root      20   0  873.9m  25.6m   0.0  2.8   0:07.43 S      `- hyperkube <-- scheduler
14204 root      20   0  868.9m  31.1m   0.6  3.4   2:06.10 S      `- hyperkube <-- controller-manager
14421 root      20   0  771.2m   1.4m   0.0  0.1   0:00.40 S      `- pause
14455 root      20   0  771.2m   1.4m   0.0  0.1   0:00.39 S      `- pause
14469 root      20   0  771.2m   1.4m   0.0  0.1   0:00.39 S      `- pause
15394 root      20   0   11.2m   3.1m   0.0  0.3   0:00.30 S      `- nginx
15399 100       20   0   11.5m   1.5m   0.0  0.2   0:00.01 S          `- nginx
16137 root      20   0   11.2m   3.0m   0.0  0.3   0:00.32 S      `- nginx
16167 100       20   0   11.5m   1.6m   0.0  0.2   0:00.00 S          `- nginx
16143 root      20   0   11.2m   3.1m   0.0  0.3   0:00.31 S      `- nginx
16168 100       20   0   11.5m   1.5m   0.0  0.2   0:00.00 S          `- nginx
16364 root      20   0  771.2m   1.4m   0.0  0.1   0:00.35 S      `- pause
16379 root      20   0  780.5m  13.3m   1.3  1.4   1:16.32 S      `- etcd
16399 root      20   0  782.4m  11.9m   0.0  1.3   0:03.65 S      `- kube2sky
16434 root      20   0  779.4m   7.8m   0.0  0.8   0:02.56 S      `- skydns
16458 root      20   0  773.5m   3.7m   0.0  0.4   0:15.70 S      `- exechealthz
16714 root      20   0  771.2m   1.4m   0.0  0.1   0:00.38 S      `- pause
16729 root      20   0  824.1m  11.8m   0.0  1.3   0:04.08 S      `- registry
13754 root      20   0  818.9m  11.7m   0.0  1.3   0:00.84 S  `- docker <-- k8s-master service

Here are my stats from top after running ~1 hour.
All the .services has their own PID, however they don't consume much.
As you see, I run 3 nginx containers, dns and registry addon.

The big consumer is kubelet, which eats everything it can get.
Will investigate more.

[nsteinmetz]

Kube1 - Cubietruck/Master

kube2 - Cubietruck / Worker

RPI1-1 - Worker

RPI1-2 Worker

[luxas]

From htop:
The kubelet container has 4 processes running, I think this is quite normal behavior. etcd and flannel containers has 3 processes each.

However, the kubelet binary is another question: 24 processes. For comparison, etcd has 8 and flannel has 6, apiserver 10, scheduler 9, controller-manager 8, the nginx binaries has 1-2 each.

If you switch to Tree (F5) you'll see this.
In fact, I'm not sure if it's processes that Tree is showing.

[nsteinmetz]

What I can see on master in Tree mode:

• docker run k8s-master = 4 process (parent + 3 children)
• skydns : 7 process
• kube2sky : 7 process
• exechealtz : 7 process
• docker : 80+ process ?
• etcd : 8 process
• registry : 6 process
• kubelet : 28 process
• controller-manager: 9 process
• proxy : 11 process
• scheduler : 10 process
• apiserver : 11 process
• pause : 3 process
• docker --name k6s-etcd : 4 process
• docker --name k8s-lannel : 4 process

Regarding docker process, not sure about accounting as order may have moved during the accounting process ; it's about:

/usr/bin/docker -d -H unix:///var/run/docker.sock -s overlay --bip=10.1.37.1/24 --mtu=1472 --insecure-registry=registry.kube-sys...

[luxas]

I think all processes except for kubelet are normal.
kubelet is very resource-hungry and I'll test it on a "real" amd64 machine and compare

[luxas]

The same for amd64 also: in 12 minutes kubelet used 1 min of CPU time, so exact the same result there. It seems like kubelet just does so many things at the same time, that it consumes a lot of CPU.
Even when no containers are running. kubelet just starts them and monitors them a bit.
So I don't think this is an ARM issue, but I'm thankful that you made this issue, I'll add this kind of monitoring to kube-config info, so one may see the usage easily.

[nsteinmetz]

No problem, glad to help and that issues are useful :)

[GrantGochnauer]

I'd be interested to hear what you find out as well. I'm seeing the same thing on my hypriot based install where the k8s-master is idle but taking about 50% cpu (avg across 4 cores). I know this could be totally normal too ;)

[luxas]

See some awesome stats of cAdvisor: http://$MASTER_IP:4194
I didn't know this worked, it didn't last time I tested (v1.0.1), however that's one of the best methods for measuring perf

[GrantGochnauer]

Ah awesome! It works for me too

[luxas]

Now I've added CPU time to kube-config info
I also mentioned the cAdvisor port on master in the README.

[luxas]

Thanks for this report. I've added a CPU time monitor to kube-config info and it's coming now in v0.6.2

[luxas]

This link may be useful: kubernetes/kubernetes#16943