deepsquare-io / clusterfactory Goto Github PK

Ambiguity about cfctl apply and targets

Source: prometheus-community/helm-charts#1500

This was already noted inside a deploy script.

We need to find a way to update Prometheus CRDs via ArgoCD.

Liveness probes are now implemented. We need a way to implement Healthchecks on compute nodes.

Add a Github CI (with daily CRON) which check the version of the extensions/applications:

• K0s (+k0sctl)
• Bitnami MetalLB
• Traefik
• cert-manager
• csi-driver-nfs
• Kubevirt
• [ ] Multus ? It's already always master.
• Prometheus ?
• [ ] Slurm ? It's updated through renovate.
• [ ] xCAT ? It's updated through renovate.

The plan:

• License and make public everything
• Add documentation
• Add CODE OF CONDUCT
• Add Github Issues Template
• Styling and layout the documentation
• Fix discord link.
• Maintains Github discussions.
• Rename csquare to <country code>-<city>-<index>.deepsquare.run with the internal DNS
• Move OnDemand url to .deepsquare.run the external DNS + SSL
• [ ] Add YUM url to .deepsquare.run the externalDNS + SSL Won't do: We will think about a better YUM repository
• Decouple our IaC from the examples:~
  • Use git push --mirror and create a branch "our cluster" or something.
• Squash to avoid secrets leaking
• [ ] Add Algolia to documentation
• Add Google Analytics
• Replace the diagrams

The CVMFS repositories cannot be renamed. (Unless we delete and recreate.)

MetalLB 0.13.0 is now available, with its chart bitnami/metallb 4.0.0.

configInline is now deprecated and CRDs are now preferred.

This major release includes the changes and features available in MetalLB from version 0.13.0. Those changes include the deprecation of configmaps for configuring the service and using CRDs instead. If you are upgrading from a previous version, you can follow the official documentation on how to migrate the configuration from a configMap to CRDs.

This major change impact the getting started.

Example:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test-network-policy
  namespace: default
spec:
  podSelector:
    matchLabels:
      role: db
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - ipBlock:
            cidr: 172.17.0.0/16
            except:
              - 172.17.1.0/24
        - namespaceSelector:
            matchLabels:
              project: myproject
        - podSelector:
            matchLabels:
              role: frontend
      ports:
        - protocol: TCP
          port: 6379
  egress:
    - to:
        - ipBlock:
            cidr: 10.0.0.0/24
      ports:
        - protocol: TCP
          port: 5978

Documentation on the issue: https://hub.armosec.io/docs/c-0054

Documentation for implementation: https://kubernetes.io/docs/concepts/services-networking/network-policies/

New yum repo: https://yum.deepsquare.run

We have to update:

• The output slurm builder CI
• The compute nodes (cloud and bare)
• Slurm Docker
• OnDemand Docker
• And maybe other stuff...

Linked issues:

• deepsquare-io/cfctl#4
• deepsquare-io/cfctl#6
• deepsquare-io/cfctl#7

Do not hard code HostPort for obvious reasons.

Maybe use IPVLAN too.

Also about disabling systemd resolved

Right now,, when applying cfctl kubeseal, it is contacting the sealed-secrets controller. We don't want that, we want an "offline" version. Instead, fetch the certificate and use it.

Enabling HTTP3 is easy with Traefik:

experimental:
  http3:
    enabled: true

ports:
  websecure:
    port: 443
    http3: true
    expose: true
    exposedPort: 443
    protocol: TCP
    tls:
      enabled: true

Remember to add port UDP redirection on the routers!

Do not use k0s extensions because they have a lot of issue and its hard to debug. We had too many "state" issues and deploy issues.

Better redeploy everything manually or with ArgoCD.

MetalLB: move to core.
Traefik: move to ArgoCD.
cert-manager: move to core.
csi-driver-nfs: move to ArgoCD.

Because we are declaring all the values inside the application.yaml, this is not GitOps friendly. We have to use the subchart pattern.

Example:

Create a chart repo with only:

app-subchart/
├── Chart.yaml
└── values.yaml

Chart.yaml

apiVersion: v2
name: wordpress
description: A Helm chart for Kubernetes
type: application
version: 0.1.0
appVersion: "9.0.3"

dependencies:
  - name: wordpress
    version: 9.0.3
    repository: https://charts.helm.sh/stable

Not only it solves every problem with GitOps, but it is the recommended solution: https://github.com/argoproj/argocd-example-apps/blob/master/helm-dependency

Reference: https://www.youtube.com/watch?v=VyuVFtp2-2M

This is a heavy breaking change as it changes all the helm applications.

We litteraly have arranged the whole repository to support the app of apps pattern.

Basically we need to:

1. The AppProject must allow deployment to the argocd namespace (and maybe others).
2. Group the Apps, Secrets, Volume... into a Kustomize project
3. Create an ArgoCD Application which deploy that Kustomize project

For example, for monitoring:

argo/monitoring/
├─ base/
│  ├─ apps/
│  ├─ ingresses/
│  ├─ volumes/
│  ├─ secrets/
│  └─ kustomization.yaml
├─ app-of-apps.yml
├─ app-project.yml
└─ namespace.yml

With this pattern, even the secrets, volumes, ingresses... get tracked by ArgoCD and we wouldn't need to use kubectl apply -k anymore (besides for app-project.yml, app-of-apps.yml and namespace.yml).

We also need to rename app-project.yml to project.yml, similarly to the examples of ArgoCD.

Reference: https://argo-cd.readthedocs.io/en/stable/operator-manual/cluster-bootstrapping/

Let's focus on Exoscale and a single availability group first.

There are multiple solutions, but this is my proposition. The idea is based on Slurm cloud-bursting. We need to solve two problems:

1. How to spawn VM which will join a private network? The private repository with Ansible can help us.
2. How to make the VM join the cluster? cfctl could be the answer.

Therefore, we should combine these two features. Because cfctl is similar to our Ansible.

• Implement in cfctl, how to create nodes on Exoscale.
• Implement a join mechanism. It's easy: healthcheck first with SSH, then use cfctl apply (or whatever).
• Implement the gRPC API for Cluster Autoscaler. Ref : https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/externalgrpc/README.md . Do not ignore the considerations. Hetzner seems to be a good example of expected user experience.

To add more details, here is the join mechanism for k0s:

1. On a controller node, call k0s token create --role=worker, this will create a join token. For extra security, the token must have an expiration time: k0s token create --role=worker --expiry=1h.
2. On the new virtual machine, via SSH, install k0s and call sudo k0s install worker --token-file /path/to/token/file.
3. Then start: sudo k0s start.

Ejecting a node is also easy: Cordon + Drain + Kubectl delete node. Then delete the VM.

This feature will certainly take time.

Linked issue: projectcalico/calico#5199

The problem is solved by hard-coding our routes.

This issue must be documented.

When SLURM jobs are launched, where do they execute?

Are all users job processes that are running on a given node executing inside that nodes slurm-controller container from the statefulset?

i.e. https://github.com/SquareFactory/ClusterFactory/blob/main/helm/slurm-cluster/templates/slurm-controller/statefulset.yml#L110

Or is there an extra layer of indirection I'm not spotting where they get spawned independently sandboxed through k0s in their own per job + per node pod?

I suppose the former must be the case since you are using SLURM to manage cgroup for CPU and GPU affinity.

ClusterFactory looks like it checks a lot of boxes for me. Thank you for putting this together and documenting it so well.

Execute scontrol reconfig or kill the container when the configmap changes.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Repository problems

These problems occurred while renovating this repository. View logs.

• WARN: Package lookup failures

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(renovate): Update Helm release kube-prometheus-stack to v58
• chore(renovate): Update Helm release mariadb to v18
• chore(renovate): Update peaceiris/actions-gh-pages action to v4
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

We have successfully setup HA. We can now write a guide about it!

• About joining
• About uninstalling if there are errors
  • etcdctl member remove
  • k0s reset
  • multus config delete
• About ejecting a controller
  • kubectl cordon
  • kubectl drain
  • kubectl delete
  • etcdctl member remove
  • k0s reset

We need to download the plugins before the installation of k0s:

curl -fsSL https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz -o plugins.tgz
tar -xf plugins.tgz