[BUG] : Requeing error messages in Logs when user create RoleTemplates without annotations about rancher

[JonCrowther]

Investigating. I see it removing my roletemplate called jono-test in the logs.

2024/05/27 19:37:57 [INFO] Starting cluster controllers for local
I0527 19:37:57.842467   35156 leaderelection.go:260] successfully acquired lease kube-system/cattle-controllers
2024/05/27 19:37:57 [INFO] Reconciling GlobalRoles
2024/05/27 19:37:57 [INFO] Reconciling RoleTemplates
2024/05/27 19:37:57 [INFO] Removing jono-test
2024/05/27 19:37:57 [INFO] Registering namespaceHandler for adding labels 

That is coming from here:

rancher/pkg/data/management/rolebuilder.go

Line 203 in 320f350

logrus.Infof("Removing %v", name)

---

As I suspected, this is not related to the annotations. The annotations get re-added immediately after deleting them by a controller. Creating the RoleTemplate and not touching the annotations yields the same results. It also happens in v2.9. It doesn't require an upgrade either, for a dev set-up stopping and starting a build triggers the reconcile and deletes the RoleTemplate

[JonCrowther]

I believe I've found the root cause. The RoleTemplate provided by @dasarinaidu has the label "authz.management.cattle.io/bootstrapping": "default-roletemplate". That label is meant to specify which RoleTemplates are default RoleTemplates that are created by Rancher. It's so we don't delete them during cleanup. However, on startup (or upgrade), Rancher creates a list of all the expected default RoleTemplates. Then it gets all the RoleTemplates with the label "authz.management.cattle.io/bootstrapping": "default-roletemplate" and makes sure that only the expected default RoleTemplates exist. Because the RoleTemplate created had the default label, Rancher deletes it since it isn't actually part of the default RoleTemplates.

If you remove that label, it does not get deleted on start/upgrade. I would consider this expected behaviour. Users should not be re-using the labels Rancher uses for synchronizing the system. That could lead to inconsistencies across the system.