globocom / slo-generator

Easy setup a service level objective using prometheus

slo-generator

Easily setup a service level objective using prometheus, based on lessons from the SRE workbook.

slo-generator -slo.path=slo_example.yml -rule.output rule.yml

Generating

Look the file slo_example.yml to see how to parametrize SLOs and generate Prometheus rules by running the following command:

Alert methods currently supported

• 1. Target Error Rate ≥ SLO Threshold, using alertMethod: simple
• 2. Increased Alert Window, using alertMethod: simple
• 3. Incrementing Alert Duration, using alertMethod: simple
• 4. Alert on Burn Rate alertMethod: simple and burnRate: <rate>
• 5. Multiple Burn Rate Alerts, using alertMethod: multi-window and shortWindow: false
• 6. Multiwindow, Multi-Burn-Rate Alerts, using alertMethod: multi-window

alertMethod: simple

Look the file slo_simple_example.yml to see a full example of usage. the simple alert method require two params:

1. alertWindow: how far back in time will used to alerting. supported values: 5m, 30m, 1h, 2h, 6h, 1d and 3d.
2. alertWait: for long time will begin fire an alert.

alertMethod: multi-window

The philosofy of this alert is described on the section of book: (https://landing.google.com/sre/workbook/chapters/alerting-on-slos#6-multiwindow-multi-burn-rate-alerts)

SLOs at scale

The Workbook suggests to create classes to simplify how to set a SLO for your services, read details about concepts here

Look at slo_example_with_classes.yml and slo_classes.yml to see how to define classes and associate with your services.

After that, you can run the command specifying the classes:

slo-generator -slo.path=slo_example_with_classes.yml -classes.path slo_classes.yml -rule.output rule.yml

Kubernetes integration

We support to export SLOs as a well known PrometheusRule resources managed by prometheus-operator, just use -kubernetes flag, example:

slo-generator -kubernetes -slo.path=slo_example.yml > slo_manifest.yml
cat slo_manifest.yml
kubectl apply -f slo_manifest.yml

Grafana integration

All generated SLOs are visible by grafana:

Import dashboard using following JSON files

slo-generator's Issues

Support custom SLO types

In google book there are more types of SLO and also custom types if needed.
Lets add support for Batch and Storage SLO

Support other alerting methods

Other methods are described in SRE workbook chapter: alerting on SLO

Allow customization of service label

We are using label product for the same purpose as service used here.
It does not conflict with kubernetes service label.