s390x support about k3s-root

[brandond]

If you build with shared libraries, those will need to be distributed in K3s as well, and the environment (LD_LIBRARY_PATH) set up to locate them. Right now we only inject the bin dir into PATH, since all the binaries are statically linked.

Socat isn't the only thing that's used, the hardened-kubernetes image also uses the iptables and ipset binaries from k3s-root as we've found that some distros package old, broken versions of these.

https://github.com/rancher/image-build-kubernetes/blob/master/Dockerfile#L62-L64

[raulcabello]

As an alternative could we just use the OS binaries in rke2 instead of the k3s-root ones? For example, we could use socat from the OS instead of socat from k3s-root when building rke2 https://github.com/rancher/rke2/blob/master/Dockerfile#L80

[rohitsakala]

@brandond Thank you for the reply. I have another question. Are static binaries used because some distros package old, broken versions of libraries? Is this the only reason or are there any other reasons too? Thanks 😄

[brandond]

It's partially so that we're not dependent on distro binaries to be up-to-date, there were (and still are I believe) a couple distros that have iptables versions that work poorly with recent versions of Kubernetes.

K3s has also historically been advertised in talks from @ibuildthecloud as running on a minimal OS that contained nothing but /bin/sh, so we bring our own everything.

[raulcabello]

@brandond we are focusing on creating rke2 now (not k3s). From the hardened-kubernetes rke2 is just using kubectl and kubelet https://github.com/rancher/rke2/blob/master/Dockerfile#L139-L142
k3s-root binaries are also used in https://github.com/rancher/image-build-calico/ and https://github.com/rancher/image-build-kube-proxy/ but since they run inside a ubi container we could use the iptables and ipset binaries from the ubi container.
I did a test and it is working fine. This wouldn't work for k3s, but for rke2 I think it should be ok.
Since we can't generate static binaries from buildroot on s390x, would using the container binaries be ok?

[brandond]

I can't think of why that would cause any problems for RKE2.

[Oats87]

The reason we use our compiled iptables/xtables binaries in the kube-proxy and calico containers is because UBI does not have both flavors (nft, legacy) of iptables and we needed both for maximum compatibility with any distro.

If it's assumed that RKE2 running on s390x will always use one or the other (I believe we're using ubi7 everywhere which should only have iptables-legacy binaries), then that should be OK.

[raulcabello]

ok, thanks for the info!