Slow CPU performance? about picl-k3os-image-generator

[sgielen]

That's a very interesting difference. Thanks for taking the time to investigate and report this. I wouldn't know OTOH what could cause that.

I know that we take a Raspbian kernel, so it's unlikely to be the cause of the difference. (Could you check the uname -a difference between the two systems to make sure?)

One big difference, perhaps, between Raspbian and k3os, is that k3os does something weird with its "/" mount. I wouldn't expect that to cause a 2-3x difference in running your benchmark, but it's one thing that comes to mind. I would also be interested to know whether the same difference is noticeable in for example Python, because I trust Python more than Bash when it comes to re-reading files from the disk such as /bin/[, which you use for testing $x. Another thing could be that k3os might use the filesystem quite a lot in the background, which could cause slowed I/O.

I don't develop k3os, so I don't know how help you any further in actually solving this. I'd recommend upstreaming your issue at https://github.com/rancher/k3os. Perhaps they will ask you to ensure this repository doesn't cause it; I would suggest to install Raspbian, perform your benchmark, then perform k3os' own ARM takeover installation, and then perform your test again. If that reproduces your issue, then it's a true k3os issue, otherwise we'd need to investigate the difference between a system with k3os' ARM takeover installation and a system running an image built by this repository.

[biern]

I appreciate your quick response 🙂

I would also be interested to know whether the same difference is noticeable in for example Python, because I trust Python more than Bash when it comes to re-reading files from the disk such as /bin/[, which you use for testing $x. Another thing could be that k3os might use the filesystem quite a lot in the background, which could cause slowed I/O.

True, I didn't think about potential IO during bash script. Still I was getting similar results in NodeJS and I've just run a similar benchmark in Python to be sure. Results below.
Worth noting is that the k3os version runs k3s server in the background which generates a slight load with occasional spikes, however I don't think that it should account for such big discrepancy.

Running python container

# raspbian with docker
docker run --rm -it docker.io/library/python:slim
# k3os
sudo ctr images pull docker.io/library/python:slim
sudo ctr run --tty --rm docker.io/library/python:slim test

Results:

Raspbian:

>>> import time ; start = time.time() ; z = [1 for i in range(100000000)] ; end = time.time() ; print ("Time elapsed:", end - start)
Time elapsed: 11.104746341705322

K3OS:

>>> import time ; start = time.time() ; z = [1 for i in range(100000000)] ; end = time.time() ; print ("Time elapsed:", end - start)
Time elapsed: 25.98793077468872

~2.34x execution time

In NodeJS:

Raspbian:

> (function () {console.time('test'); let i = 0; while(i < 10000000000) { i++ } ; console.timeEnd('test');})();
test: 22.030s

K3OS:

> (function () {console.time('test'); let i = 0; while(i < 10000000000) { i++ } ; console.timeEnd('test');})();

test: 52.799s

~2.39x execution time

[biern]

Forgot about kernel info:

k3os:

Linux xxx 4.19.75-v8+ #1270 SMP PREEMPT Tue Sep 24 18:59:17 BST 2019 aarch64 GNU/Linux

raspbian:

Linux raspberrypi 4.19.118-v8+ #1311 SMP PREEMPT Mon Apr 27 14:32:38 BST 2020 aarch64 GNU/Linux

[biern]

@sgielen
I've tried arm overlay installation, basing on ubuntu server 20 arm64 image and k3os v0.11.1 that I had handy and after installation, k3os performance is much better, actually the same as on raspbian / ubuntu server!
NodeJS and bash benchmarks run >2x faster, and the same app has ~2x better API response times.

I've tried the takeover installation method as well but I had some problems with it, I couldn't connect to the device, I'm not sure the k3os has initialized properly.

Do you have any idea what might be going one here and how to fix it? I'm also wondering if I could somehow easily reuse the OS I've generated here and use it as a base for writing new SD cards.

[sgielen]

Closing all issues, because the project is archived. Thank you for being a contributor and/or user!