[SURE-7060] during rancher upgrade to 2.7.6 helm operation error with fleet-cleanup-clusterregistrations about fleet

[raulcabello]

QA Template

Solution

Added Security Context to the cleanup job in #1862

Testing

Install rancher 2.7.5 in a hardened cluster rke2 1.24 (see issue for more info on the env)

Upgrade to the latest rancher should not give any error in the fleet-cleanup-clusterregistrations job

Additional info

Needs a new fleet RC

[mmartin24]

QA report

Testing considerations:

For hardening, I followed the steps detailed in this guide with few adjustments.
As psp is disabled after Kubernetes version 1.24 testing was done with psa and Kubernetes version > 1.25

Tested scenarios:

Scenario 1: Fresh installation hardened rke2 cluster on Rancher performing CIS with no errors.

Setup:

• RKE2 version: rke2 v1.28.9+rke2r1
• Hardening parameters used: cis and psa.yaml
• Configured default Service account
• Fresh installation of Rancher v2.9-6d87a11ea46b7571646d7c3d7af704584c39fd62-head
• Confirmed successful Rancher deployment
• Deployed CIS benchmark rke2-cis-1.8-profile-hardened with 71 passes, 0 errors and 48 warnings

Scenario 2: Installation hardened rke2 cluster on Rancher 2.8 performing CIS with no errors, later upgrade to 2.9 and new CIS with no errors.

Setup:

• RKE2 version: rke2 v1.26.15+rke2r1
• Hardening parameters used: cis and psa.yaml
• Configured default Service account
• Fresh installation of Rancher v2.8-ec76f714a7d22be1d4266cf5385f0aef62a9a653-head
• Confirmed successful Rancher deployment
• Deployed CIS benchmark rke2-cis-1.8-profile-hardened with 71 passes, 0 errors and 48
  warnings
• Upgraded to Rancher v2.9-6d87a11ea46b7571646d7c3d7af704584c39fd62-head
• Deployed new CIS benchmark rke2-cis-1.8-profile-hardened again with 71 passes, 0 errors and 48 warnings
• Checked fleet-cleanup-clusterregistrations job did not throw any error

[sbulage]

Successfully upgraded to the Rancher 2.7.6 from Rancher 2.7.5 without any error. Also the command:

satya@opensuse15:~> kubectl get job -A
NAMESPACE       NAME                        COMPLETIONS   DURATION   AGE
fleet-default   test-fleet-examples-2ff03   1/1           21s        11m
kube-system     helm-install-traefik        1/1           27s        45m
kube-system     helm-install-traefik-crd    1/1           24s        45m

satya@opensuse15:~> kubectl get pods -n cattle-system 
NAME                               READY   STATUS      RESTARTS   AGE
helm-operation-czs7c               0/2     Completed   0          15m
helm-operation-f5k74               0/2     Completed   0          45m
helm-operation-grfgr               0/2     Completed   0          14m
helm-operation-hpcnv               0/2     Completed   0          45m
helm-operation-nlj28               0/2     Completed   0          44m
helm-operation-qz66m               0/2     Completed   0          45m
helm-operation-vxvzc               0/2     Completed   0          45m
rancher-569b86c8f5-7s8kn           1/1     Running     0          17m
rancher-webhook-788c48b988-rlg6d   1/1     Running     0          45m

Also, fleet upgraded from 0.7.0 to 0.7.1 without any error.

Please let me know if anything else need to check.

[pdiaz-suse]

I have been able to reproduce the issue, but only in the case when the cis-1.6 profile is enabled in the underlying RKE2 Kubernetes cluster.

Helm operation pods are continuously failing:

root@ip-172-31-26-45:/etc/rancher/rke2# kubectl get pod -n cattle-system
NAME                              READY   STATUS      RESTARTS      AGE
helm-operation-2klh9              1/2     Error       0             28m
helm-operation-2n6nn              1/2     Error       0             52m
helm-operation-2vqfh              1/2     Error       0             54m
helm-operation-4xskv              1/2     Error       0             47m
helm-operation-55qhj              1/2     Error       0             7m55s
helm-operation-5mf85              1/2     Error       0             3m57s
helm-operation-5rplt              2/2     Running     0             3m52s
helm-operation-78gpf              1/2     Error       0             23m
helm-operation-8jcww              1/2     Error       0             52m
helm-operation-8ztt4              1/2     Error       0             43m
helm-operation-97l5p              1/2     Error       0             52m
helm-operation-b5sfs              1/2     Error       0             22m
helm-operation-cndwz              0/2     Completed   0             52m
helm-operation-dww9d              1/2     Error       0             52m
helm-operation-gc7cs              1/2     Error       0             8m57s
helm-operation-gg29f              1/2     Error       0             33m
helm-operation-h7wsk              1/2     Error       0             18m
helm-operation-hhvqt              1/2     Error       0             18m
helm-operation-j26v7              1/2     Error       0             37m
helm-operation-jsk4k              1/2     Error       0             51m
helm-operation-nv7zz              0/2     Completed   0             53m
helm-operation-qrqsr              1/2     Error       0             13m
helm-operation-qt7bf              1/2     Error       0             52m
helm-operation-r82vk              1/2     Error       0             46m
helm-operation-r9r8w              1/2     Error       0             28m
helm-operation-s66gv              1/2     Error       0             13m
helm-operation-v687s              1/2     Error       0             51m
helm-operation-z7q6c              1/2     Error       0             43m
helm-operation-zd8t5              1/2     Error       0             38m
helm-operation-zr559              1/2     Error       0             51m
helm-operation-zs9w2              1/2     Error       0             33m
rancher-6bf9cd485c-8d5fg          1/1     Running     1 (55m ago)   56m
rancher-6bf9cd485c-qd79p          1/1     Running     0             56m
rancher-6bf9cd485c-rjphw          1/1     Running     0             56m
rancher-webhook-998454b77-ghsgd   1/1     Running     0             52m

Due to the failure in the fleet-cleanup-clusterregistrations pod related with the

root@ip-172-31-26-45:/etc/rancher/rke2# kubectl get pod -n cattle-fleet-system
NAME                                       READY   STATUS                       RESTARTS   AGE
fleet-cleanup-clusterregistrations-vsbhq   0/1     CreateContainerConfigError   0          8m26s
fleet-controller-64f5b4585-shjjb           1/1     Running                      0          52m
gitjob-58dc7cb797-wr28d                    1/1     Running                      0          52m

Events:
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  Normal   Scheduled  9m33s                   default-scheduler  Successfully assigned cattle-fleet-system/fleet-cleanup-clusterregistrations-vsbhq to ip-172-31-26-45
  Warning  Failed     7m39s (x12 over 9m33s)  kubelet            Error: container has runAsNonRoot and image will run as root (pod: "fleet-cleanup-clusterregistrations-vsbhq_cattle-fleet-system(d32d3d68-6bbf-4783-9ecc-d6200284b411)", container: cleanup)
  Normal   Pulled     4m32s (x26 over 9m33s)  kubelet            Container image "rancher/fleet-agent:v0.8.0" already present on machine

[HoustonDad]

Also hitting this issue!

Rancher 2.7.7 and RKE2 1.24.x w/CIS Profile 1.6 enabled

[jcox10]

This appears to be a missing PSP for hardened clusters using CIS profile, running <=1.24. On 1.25+ the entire cattle-fleet-system namespace is exempted from the PSA.

A quick workaround is to just bind the unrestricted PSP to the service account:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: unrestricted-psp
  namespace: cattle-fleet-system
rules:
- apiGroups:
  - extensions
  resourceNames:
  - system-unrestricted-psp
  resources:
  - podsecuritypolicies
  verbs:
    - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: unrestricted-psp
  namespace: cattle-fleet-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: unrestricted-psp
subjects:
- kind: ServiceAccount
  name: fleet-controller

Once the rancher charts are fixed, the Role and RoleBinding can be deleted.

[manno]

PR #1862 needs a backport to v0.9