Mix of Rpi1/Rpi2+CubieTruck (or armv6 vs armv7) about kubernetes-on-arm

[luxas]

It's already possible to mix and match armv6 and armv7.
Docker and Kubernetes are built for armhf, so they run on both.

So it shouldn't be a problem.
But it could be pointed out in the docs.

I have no experience with cubietruck, but I see arch linux has a guide here, so it should be quite easy to support it.
The guide how to make a Raspberry Pi SD Card is quite similar, which I am using

You could already try to write Arch to both your Rpi1 and 2.

[nsteinmetz]

Good to know, I thought that armv6 images could not be used with armv7. Otherwise, I don't see the point on docker hub for example to prefix images with armv6 / armv7 whereas I can see the point for armhf vs armel. Thanks for the precision/clarification.I can now plan having a 4 to 6 k8s cluster now :)

Regarding cubie support, let's try to have a look at it once you have some documentation about what I should do/test. I'll focus first with Rpis.

[luxas]

@nsteinmetz armv6/7 is now pointed out in the docs
I may add the Cubie scripts some day, but I will not be able to test it, because I haven't one

[nsteinmetz]

Hi Luxas,

Thanks,

Regarding cubie scripts, if you can provide me some docs on how I can build/use them, I'll be happy to test. But I don't know from where should I start now. So if you guide me, I'll be happy to help.

Nicolas

[luxas]

@nsteinmetz Uploaded code for the Cubie now.

These are the steps that the script does:

• Write one ext4 partition that fills the whole SD Card.
• Mount the sd card to a temp directory
• Download latest Arch Linux ARM filesystem for armv7: http://archlinuxarm.org/os/ArchLinuxARM-armv7-latest.tar.gz
• Get the bootloader and write it to the SD Card

Here are the guide: http://archlinuxarm.org/platforms/armv7/allwinner/cubietruck

What you could do to test: sudo sdcard/write.sh cubietruck archlinux kube-archlinux

Then, when you have your system up, run pacman -Syu uboot-cubietruck, reboot and do as usual.

I hope it will work!

[nsteinmetz]

As you may have seen, I'll start testing your project first with a Raspberry to get familiar with it. Then, I'll test on Cubie too.

Should ease the fact to distinguish specific cubie bugs from others ;)

[luxas]

Sounds good

[nsteinmetz]

Testing this by fetching the code on master.

For the cubietruck, seems code requires two partitions whereas the format builds only one.

In sdcard/os/archlinux.sh, on line 98 & 102, I think it should be $PARTITION1 and not $PARTITION2

[nsteinmetz]

Better but not perfect : too late to dive in your code.

As there is only one partition, the boot is expected to be (un)mounted which is not the case.

[...]
Partitions mounted
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
--2015-10-24 23:27:16--  http://archlinuxarm.org/os/sunxi/boot/cubietruck/u-boot-sunxi-with-spl.bin
Résolution de archlinuxarm.org (archlinuxarm.org)… 50.116.36.110
Connexion à archlinuxarm.org (archlinuxarm.org)|50.116.36.110|:80… connecté.
requête HTTP transmise, en attente de la réponse… 301 Moved Permanently
Emplacement : http://os.archlinuxarm.org/os/sunxi/boot/cubietruck/u-boot-sunxi-with-spl.bin [suivant]
--2015-10-24 23:27:19--  http://os.archlinuxarm.org/os/sunxi/boot/cubietruck/u-boot-sunxi-with-spl.bin
Résolution de os.archlinuxarm.org (os.archlinuxarm.org)… 158.69.161.16, 150.162.66.161, 72.13.197.87, ...
Connexion à os.archlinuxarm.org (os.archlinuxarm.org)|158.69.161.16|:80… connecté.
requête HTTP transmise, en attente de la réponse… 200 OK
Taille : 372016 (363K) [application/octet-stream]
Sauvegarde en : « u-boot-sunxi-with-spl.bin »

u-boot-sunxi-with-spl.bin                   100%[=============================================================================================>] 363,30K  --.-KB/s   ds 0,09s  

2015-10-24 23:27:19 (3,78 MB/s) — « u-boot-sunxi-with-spl.bin » sauvegardé [372016/372016]

363+1 enregistrements lus
363+1 enregistrements écrits
372016 octets (372 kB) copiés, 0,101826 s, 3,7 MB/s
--2015-10-24 23:27:19--  http://archlinuxarm.org/os/sunxi/boot/cubietruck/boot.scr
Résolution de archlinuxarm.org (archlinuxarm.org)… 50.116.36.110
Connexion à archlinuxarm.org (archlinuxarm.org)|50.116.36.110|:80… connecté.
requête HTTP transmise, en attente de la réponse… 200 OK
Taille : non indiqué [text/html]
Sauvegarde en : « /tmp/writesdcard/root/boot/boot.scr »

/tmp/writesdcard/root/boot/boot.scr             [ <=>                                                                                          ]   1,62K  --.-KB/s   ds 0s     

2015-10-24 23:27:20 (173 MB/s) - « /tmp/writesdcard/root/boot/boot.scr » sauvegardé [1654]

OS written to SD Card
umount: /tmp/writesdcard/boot : non monté

[nsteinmetz]

Guess you need something like in write.sh line 188:

if [ $MACHINENAME == "cubietruck"]; then
    umount $ROOT
else
    umount $BOOT $ROOT
fi

[nsteinmetz]

So as is, cubietruck does not start ; will start from scratch with a vanillia archlinux install to check first it works well and then will try to find what's missing in our context.

[nsteinmetz]

Ok found the issue, my proxy blocked the download to the boot.scr files ; so it was an html file and not the expected file ;-)

Just need to whitelist this file :)

[nsteinmetz]

So beyond the mount issue and the whitelist issue on proxy side, works like a charm ; kubie-config install in progress :)

[nsteinmetz]

... I still have an issue with enabling k8s-worker but I'll open a different issue tomorrow.

Good night :)

[luxas]

OK, glad it (partially) worked.
I released a bugfix version yesterday, v0.5.6
I think you used the latest version as I uploaded it 22:42 EEST to master yesterday.
In that case the tar warnings didn't obey the redirect :)
Will test it. Already patched umount $BOOT on cubie.

[luxas]

FYI, I will make a new branch named dev for developing between releases.
Then when I make a release those changes are merged into master.
So if you want to download code between releases, switch to dev

[luxas]

Now I'm using curl instead of wget for getting the boot files.
So now wget isn't a dependency anymore.
I also fixed:

I think it should be $PARTITION1 and not $PARTITION2

Uploaded to dev
archlinux.sh

[nsteinmetz]

Works now as expected on dev branch => close :)

thanks !

[nsteinmetz]

maybe the only point would be to remember at the end to do the:

pacman -Syu uboot-cubietruck

I only remind now that I should do it after many sdcard initialisation.

Or to add it in the kubie-config install process ?

As french with a azerty keyboard, I also need to set it correctly. Could be also an enhancement.

I also need to add a static ip but it's more my own issue than a generic one.

[luxas]

I am planning to make it possible to add customized scripts.
Will open a different issue for it.
The uboot install belongs to this project.

Here is a guide about keyboard layouts: https://wiki.archlinux.org/index.php/Keyboard_configuration_in_console

You may add static ip via your router or via arch linux

[nsteinmetz]

Yep, I'm a loadkeys fr && localectl set-keymap fr master now :)

So we keep this bug opened only for the uboot-cubietruck package.

[luxas]

I'm working on this right now.
The proposed tree:

• os/
  • archlinux.sh - the file that writes the os to the card
  • archlinux/ - optional directory with extra commands for kube-config
    • commands.sh - specifies commands just for arch linux (like pacman commands)
    • cubietruck.sh - customized commands for cubietruck
    • rpi-2.sh - customized commands for Raspberry Pi 2

The good thing with this approach is that, long-term, kube-archlinux may evolve to systemd.
So the same rootfs could re-used on e.g. Debian 8 (Bananian or HypriotOS)
WDYT?

[nsteinmetz]

Hi,

Sounds good at a first glance ; I should test latest release this week-end rather than dev version (unless you tell me the opposite)

[luxas]

It worked well last time I tested v0.5.8 on Raspberry Pi 2

I will probably look at the Raspberry Pi 1 compilation after this.
go1.5.1 will be used then. Maybe bugs are fixed with latest version.

[luxas]

I am testing latest dev right now, but the Arch Linux ARM servers seems to be down.
I don't think dev is stable right now. I'll notify when you may test.

But, did everything work for you? The whole system with private registry and so on?

[nsteinmetz]

I was about to set up my 2nd cubietruck ; I started writing the sdcard and when I came back, I noticed archlinuxarm servers were down. So my k8s-master seems fine but he's still alone right now. So I did not play wit it more so far (but so far, so good as per #13 :)

[nsteinmetz]

You can use http://ftp.halifax.rwth-aachen.de/archlinux-arm/ as an alternative mirror it seems ;-)

[luxas]

Thanks for the link!
If the servers are down for a longer time, I maybe have to use that as a fallback.

[nsteinmetz]

Seems back online btw !

[luxas]

I think the dev branch is quite stable now. You may try to write the SD Card for cubietruck. It should install the uboot package at kube-config install.

I haven't tested everything on dev and RPi 1 isn't supported yet I think, I'll upload commits to dev regulary

[nsteinmetz]

Hi,

I'll wait for the RPI1 being supported to lead further tests ; quite busy (seems everyone for others projets I'm in just wake up these last days after months of hibernation ;-) )

[luxas]

@nsteinmetz Just now I'm working on on v0.6.0, and that release is a big one.
Some changes:

• cubietruck full installation
• Upgrade Kubernetes to v1.2.0-latest
• Upgrade flannel => v0.5.4, etcd => 2.2.1, registry => 2.2.0, go => 1.4.3
• Now it's possible to build the Kubernetes binaries with Go 1.5.1, but it's much slower so it's not default
• Fix the bug that makes this not run on armv6, e.g. Raspberry Pi 1
• Add windows downloads
• Enhance security with certs and ServiceAccounts
  • This is something that isn't working on "real" machines either, and I found the solution 😄
  • Upstream: kubernetes/kubernetes#16965 and kubernetes/kubernetes#16987
• maybe add a loadbalancer

As you see, there is much to do.
I would be glad if you tested this when I release it.
I'll keep you updated.

[nsteinmetz]

It's a little bit early for Christmas ;-)

Let me know once I can test it !

[nsteinmetz]

Hi,

I'm sure you saw this ; are you aware of this project ? Looks quite close to yours.

Btw, any update for the 0.6.0 release ? ;-)

[luxas]

No, I hadn't seen that blog post. Thanks for notifying me. 😄
The v0.6.0 is coming soon, I hope this weekend.
It has been so many things to do.
Just now I have a PR pending on mainline k8s. kubernetes/kubernetes#17213

And this release is about a week delayed because of docker v1.9.0 bug on Arch Linux ARM at least: moby/moby#18125

However, today I have got everything up and running as it should. This time it was harder than usual, because these things isn't in mainline k8s (yet).
Now I just have to package everything.

[luxas]

Finally released!
Now you may test it and (I hope not 😄) return bugs.
This issue is open for the uboot package for cubietruck and for RPi 1 support.
They should both be fixed. Test it please!
There is also a lot of other fun things I mentioned earlier.

Soon, it's Christmas anyway :)

PS. If you think this project is great, star it :)

[nsteinmetz]

Woot ! :)

I'll test over the coming week and let you know !

[luxas]

Could this be closed?
Was the uboot-cubietruck package automatically installed?

[nsteinmetz]

Yes it was ! :)

The only investigation point could be that you need an extra "Y" to flash the boot files. Not sure you can enforce it as it may be sensible.

I thnk it's acceptable so far.

Thanks a lot for making cubietruck supported so fast ; Really enjoy it !

[luxas]

That was strange, I suppose you mean that this line asked for [Y/n] when installing uboot?
https://github.com/luxas/kubernetes-on-arm/blob/master/sdcard/rootfs/kube-archlinux/etc/kubernetes/dynamic-env/cubietruck.sh#L5

[nsteinmetz]

Exactly, it's maybe within uboot-cubietruck package itself and ouf of pacman control as flashing boot record is a sensible task ? That's why I say it's not a big issue to keep this [Y|n] step.