k0sproject / k0smotron Goto Github PK

Ref #171

The K0smotronCluster needs to be perssistent, in order to do so we need to be able to define the storage configuration.

We need at least to support:

• User provided PVC
• Automatically created PVC with/without storageClass
• emptyDir (we need something that just works for quick testing)

Otherwise the join tokens etc. are scoped for pod IP address only. For example, running the controlplane with the default NodePort:

kmc-cp-test-0:/# k0s token create | base64 -d | gunzip | grep server
    server: https://10.244.0.42:30443

As it is a NodePort maybe we could default to first Node we find with External Address and if one is not found, then just the first Node?

Cluster API compatible CLI so that we can do e.g.
clusterctl init --infrastructure aws --bootstrap k0smotron
clusterctl generate cluster [name] --infrastructure aws --bootstrap k0smotron --kubernetes-version [version]

Inttests need a common/util module like k0s.

We need to allow full customization of the k0s install flags.

Enable monitorability on k0smotron controlplanes

The K0smotron controller is getting far too large and every modification is quite difficult because everything is in one file making a lot of rebases necessary when we're modifying different things.

So that we don't "pollute" telemetry data too much.

Also set the k0s telemetry "tags" to indicate k0smotron clusters

k0smotron needs to have a single command to actually run tests.
I'm currently working on deploying a footloose cluster based on k0s inttests.

k0smotron controller manager pod is failing to pull image quay.io/k0sproject/k0smotron:latest

Normal   BackOff    17s               kubelet            Back-off pulling image "quay.io/k0sproject/k0smotron:latest"
  Warning  Failed     17s               kubelet            Error: ImagePullBackOff
  Normal   Pulling    5s (x2 over 21s)  kubelet            Pulling image "quay.io/k0sproject/k0smotron:latest"
  Warning  Failed     2s (x2 over 18s)  kubelet            Failed to pull image "quay.io/k0sproject/k0smotron:latest": rpc error: code = NotFound desc = failed to pull and unpack image "quay.io/k0sproject/k0smotron:latest": failed to resolve reference "quay.io/k0sproject/k0smotron:latest": quay.io/k0sproject/k0smotron:latest: not found
  Warning  Failed     2s (x2 over 18s)  kubelet            Error: ErrImagePull

We need similar getting started guide as we already have for Hetzner

We need to expose the kubeconfig somehow. A secret seems to be a reasonable way to do it.

No one is probably really using single Machine types when creating clusters thus we should use MachineDeployments now that k0smotron supports the required K0sWorkerConfigTemplate types.

We need to allow users to fully customize k0s controller config.

I see two possible ways

ConfigMap

We can allow user to specify a configmap as the reference for the config. k0smotron will need to "overwrite" some of the fields such as spec.api.externalAddress so we might need to copy the user given CM to something else, or at least do an in-place update of it.

CRD Fields

We can copy the relevant fields to k0smotron CRD. This would be best from UX point of view as it will expose all the typed fields in the CRDs. That does have some drawbacks:

• We need to also copy the defaults
• If we copy those as code dependency we will create an unnecessary dependency on k0s version

Stuff I think we should have:

• - how to set it up
• - How to create k0smotron control planes
• - what can be configured
• - How to get join tokens
• - How to join new workers
• - How to get the kubeconfig
• - How to debug
• - List of pre-requisites
• - Known limitations
• - FAQ
• - Relation to Cluster API

We're based on kube-builder's default Makefile which works well but is fairly slow because we end up rebuilding a bunch of stuff we don't need to rebuild.

We need to support user giving k0s configuration for the controlplane via custom resource.

I see two high level options for this:

• Bundle it in the KMC object (either as embedded string-yaml or as a real object)
  • real object might get us in troubles in versioning it.
• Create separate object for it and KMC referencing it.
  • might get us into same versioning challenges

One of the challenges is also the fact that k0smotron will need to interfere with the user provided config when adding things like externalAddress etc.

We need to also think about how to reload the config if user changes it. I see two options:

• if config really changes the operator will restart the pod
• we bundle in some FS watch based restart into the k0s image

One way to mitigate this is to expose config option in Cluster CRD to use external datasource such as Postgres or MySQL (compatible) DB. This way we can "externalise" the HA DB challenge. There's operators available for both MySQL and Postgres that can handle backups etc. which we definitely do not want to need to handle in k0smotron.

Hi, I'm taking interest in this project, since I started working on a similar project, that I could leave behind if k0smotron answers my needs.
I understand you're using Konnectivity to handle control plane to node communication. But how do you handle node to control plane communication ? If they're on the same network this is an easy thing but I'd like to have the control planes completely separated from the workers. Have you found a way to get this to work ?

Currently the implementation only allows nodePort and what's worse, it has the port hardcoded.

We need an ingress solution that at minimum allows to choose:

• No ingress at all (user will solve this manually)
• NodePort Service

In the future it would be good to either use the gateway API or to have one load balancer managed by k0smotron.

https://book.kubebuilder.io/reference/watching-resources/operator-managed.html

Basically this would allow us to better "propagate" e.g. status of things. So when e.g. the StatefullSet status changes, we get a Reconcile call. Based on this we can easily set the status of k0smotron.Cluster object. This in turn can be then used in both print fields and for "downstream" users of Cluster object, for example the Cluster API provider.

Hi,

Recently I followed the GitHub README.md to install k0smotron in my local Kubernetes cluster but it failed to pull image.
I opened an issue #112 and solved by following the installation command from official doc.

Can I open a PR for the below change in README.md file.

Current
kubectl apply -f https://raw.githubusercontent.com/k0sproject/k0smotron/main/install.yaml

Change to
kubectl apply -f https://docs.k0smotron.io/stable/install.yaml

Thank you.

We want to publish this as OSS project. For that we need:

• Licence, Apache 2
• Contributor guide
• Add some status info to main README

We need to provide configurable resources for the k0smotron created control planes.

It should be patch.

We need to be able to “tunnel” the connection to child control-planes (when running in separate Machines) via mothership.

On high level, something like:

Naturally, the agent->server tunnel connection needs some per cluster auth

Open questions

• Should the mgmt cluster "proxy" component do auth on the calls?
  • If yes, how to do it WITHOUT impersonation

Some ideas how to implement

• https://github.com/fatedier/frp
• pinniped; Although I'm not entirely sure if it fits fully
• https://github.com/lensapp/bored; Lens uses this for the team work, similar use case

Allow creation of consolidated roles for smaller clusters with e.g. 3 nodes. Feature based on "controller+worker" role of k0s.

Linked issue: #127

We need to somehow be able to refer to the DB URL via a secret.

We need to also support some generic cloud-init “additions”

• Create supporting files
• Run command as pre and post k0s setup “hooks” to setup e.g. kernel tunings etc.

Obviously we need to have proper "e2e" tests for Cluster API provider.

Could we use the Docker infra provider for smoke testing? AFAIK it is capable of bootstrapping nodes in containers (like we do in Footloose).

The bootstrap provider should be able to act as a provider also for the controllers. This way users can use k0smotron to also provision "traditional" clusters with dedicated controllers running on Machines.

Looking at some of the other providers they always have a single XYZConfig style CRD that covers both the controller and worker setups.

Currently the same GTAG is used here and in the k0s docs.

There might be cases when user does not want to use both CAPI and k0smotron controllers. So we should allow disabling/enabling only wanted controllers.

We need to use proper per cluster labels for all the created child objects. Now we add just some hardcoded labels which can result in bugs too when running many controlplanes.

For example in https://github.com/k0sproject/k0smotron/blob/main/internal/controller/k0smotron.io/k0smotroncluster_statefulset.go#L55-L68

If k0smotron detects that we're installing on say AWS we could automatically configure AWS cloud provider on the child cluster

From @juanluisvaladas :

The PROJECT file isn't adding a lot of value and I noticed it's already inconsistent in the main branch (became inconsistent after renaming the Cluster type) so I'm in favour of removing it entirely.

We need to make a call whether we want to maintain it or not

The build system is quite similar and it's getting more and more similar. The problem is that this requires docker images during the build process to do the build. We need to discuss which strategy we want to follow. I can think of four possibilities:

1- Upload the k0s build images on every release and use these images in k0smotron.
This is my favourite, seems like the most efficient and if we need to override an image for whatever reason implementing that through an environment variable seems trivial.

2- Copy and paste the build images that we need. This is quite easy to implement but also means twice as much work with maintenance, we need to copy and paste it forever

3- Use a subtree, which is makes checking the history complicated. This can be mitigated by having all the image generation in a separate repo and using subtrees on both k0s and k0smotron. Anyway I'm not a big fan of it anyway.

4- Use a git submodule. This is suboptimal because it involves some extra work when it comes to clone and pull and it has to be done always. Like with subtrees, I'm not a big fan.

I think pobably option 1 is the most reasonable.

https://cluster-api.sigs.k8s.io/developer/providers/bootstrap.html#bootstraptemplate-resources

This is needed to be able to use MachinePool etc. resources in cluster api.

We probably want to copy annotations too

We need a proper way to issue join tokens