Information about absent values about awesome-prometheus-alerts

[samber]

Oh! I see. 😲
I had this trouble in the past as well.

Let me think about that. I will get back to you in a few days. 🤔

[samber]

@BenjaminHerbert I just discovered a metric: prometheus_rule_evaluation_failures_total. This counter tell you when a rule fail to be executed, leading to ignored alerts.

expr: increase(prometheus_rule_evaluation_failures_total) > 0

[samber]

Using this query, you detect an absent job in prometheus.yml. Not a killed exporter.

What is your use case ?

[BenjaminHerbert]

My question concerns the absence of particular metrics and if it might be helpful for users to see an example for this. Up was just an example for a metric. My use case was creating alerts for failing or missing backup attempts with velero.
The alerts did not fire, as the metric I was using was renamed. Adding a check for absent(metric{job=myname}) prevents that such alerts break due to metrics going missing or due to alerting on metrics that do not exist.

What do you think?

[samber]

Hi @BenjaminHerbert

Sorry for the delay.

This is a chicken and egg problem: how to monitor the monitoring platform ? 😱

I think there are 3 things you can do:

• add an alert that detect killed exporters (expr: up == 0) => https://awesome-prometheus-alerts.grep.to/rules#prometheus-internals
• do not let your infrastructure update automatically exporters, in order to avoid breaking changes in metric names
• upgrade exporters manually after reading changelogs

If there is no changelog, just pray... 🙏 🤣

Adding an alert for each metric used by your alerting system (such as absent(node_memory_MemAvailable_bytes)) is probably over engineering.

Closing this issue. Feel free to open it again if necessary ;)