Alerts failing out of the box: K8SControllerManagerDown and K8SSchedulerDown (on Kargo) about kube-prometheus

[rutsky]

Thanks! This issue is resolved.

[brancz]

Actually your setup looks good, you just need to create the services in manifests/k8s/self-hosted/. That should work, but you may also want to update to the latest versions of the manifests as the configuration is now generated by the Prometheus Operator, so no manual configuration maintenance necessary anymore 🎉 🙂 .

Just as a reminder the content of this repo has moved to the Prometheus Operator repository. We can solve this issue here, but future issues should be opened there (feel free to tag those with [kube-prometheus]).

[rutsky]

Creating services from manifests/k8s/self-hosted/ almost solves this issue, thanks @brancz !

"almost" because in Kargo Schduler has k8s-app=kube-controller label instead of excepted by manifests k8s-app=kube-controller-manager:
https://github.com/kubernetes-incubator/kargo/blob/dbf1329/roles/kubernetes/master/templates/manifests/kube-controller-manager.manifest.j2#L7

Is this Kargo bug? Or there is no "standard" on those labels?

[brancz]

As both bootkube and kops have the k8s-app=kube-controller-manager label I think this should be adapted in kargo. (as far as I can tell the manifests in kubernetes/kubernetes also use kube-controller-manager)

[brancz]

Unfortunately there is no standard on those labels yet, it's on my list to start pushing for them.

[brancz]

For the reference: kubernetes-retired/contrib#2420 (comment)

[rutsky]

@brancz thanks, I opened issue on Kargo.
Have you opened ticket on the Kubernetes repo to introduce some standard regarding labels?

[brancz]

Quite literally opened it seconds ago 🙂 . Let's see how it goes.

[brancz]

Thanks for reminding me to open that issue! 😁 Let's hope we'll get something good out of it.