roshan8 / slo-tracker

A tool to track SLA, SLO and Error budgets

SLO-Tracker

A simple but effective way to track SLO's and Error budgets. SLO-tracker can be integrated with few alerting tools via webhook integration to receive SLO violating incidents.

Motivation

SLOs are a predefined approach to defining the amount of downtime that is acceptable for your product. Tracking SLO's and error budgets can be hard without proper tooling support because of the multitude of alert sources.

Since SLO's are directly related to the user experience of the product, more and more companies are defining their SLO's and tracking the error budget. Even though most monitoring tools provide SLO tracking feature out of the box, SREs don't find it to be a wholesome solution. This is because SREs might have different tools for monitoring different services and their respective SLIs.

In the event of an incident, manually aggregating these SLIs from multiple tools and placing it in one place to track error budgets is very challenging. Sometimes self-hosted monitoring tools might have short retention policies, thus making it hard to retain SLO metrics for longer durations(>6 months). And the most challenging situation is when false positive alerts reduce your error budget and you're grappling to gain valuable minutes into your error budget.

Highlights

• Provides a unified dashboard for all the SLOs that have been set up, in turn giving insights into the SLIs being tracked
• Provides the ability to claim your falsely spent Error Budget back by marking erroneous SLO violation alerts as False Positives
• Supports manual alert creation from the web app when a violation not caught by alert sources (or not supported alert-sources)
• Displays basic Analytics for SLO violation distribution (SLI distribution graph, Error budget consumption over time)
• Easy to set up, lightweight since it only stores and computes what matters (SLO violation alerts) and not the bulk of the data (every single metric)

Installation and usage

docker-compose up --build -d      

admin:admin is the default creds for UI, Which can be changed in docker-compose file.

Supported integrations

• Prometheus
• Datadog
• Newrelic
• Pingdom
• Grafana
• SLO-Tracker API

P.S: Sending all the alerts SLO-tracker from your monitoring tool defeats the purpose. Please send only SLO violating incidents to this tool.

UI Screenshots

Demo dashboard: http://demo.slotracker.com:3000/ (creds: admin:admin)
Blog: https://dev.to/squadcast/introducing-our-open-source-slo-tracker-a-simple-tool-to-track-slos-and-error-budget-4dp

Contributing

Refer to CONTRIBUTING.md

Stargazers over time

slo-tracker's Issues

Bug report on the False Positives feature

What is currently happening:

1. An incident is created in the dashboard for an alert in one of the available monitoring tools
2. The incident is in open state
3. The incident is marked as a false positive post analysis
4. The incident still remains in the open state

What should happen:

1. An incident is created in the dashboard for an alert in one of the available monitoring tools
2. The incident is in open state
3. The incident is marked as a false positive post analysis
4. The incident is moved to closed state

Why should the behaviour change:
The incident still should be moved to closed state since it is irrelevant when marked as a false positive and should not be accounted towards the error budget calculations or analytics

MySQL connection strings hard-coded throughout

Even though environment variables are available, we still end up using strings like:

    dbConnectionString := fmt.Sprintf("root:SecretPassword@tcp(%s:3306)/", config.DBHost)

which is really only valid for MySQL. It would be nice to make this all more flexible. At least allowing for sqlite installations for easier testing.

Credentials for demo installation missing

Hey, just stumbled upon this via Reddit and wanted to take a look at the demo installation referenced at the bottom of the README. But it seems I need credentials. demo / demo didn't work, admin/admin did. Maybe you want to add this in the README

Is Elastic APM and AWS CloudWatch supported as integration for SLO tracker?

Hi,

Is Elastic APM and AWS CloudWatch supported as integration for SLO tracker?

Project status?

As mentioned below, this has been adopted by Squadcast some time ago but there haven't been any active updates to the project since then. Are there plans to keep this in line with the version implemented at Squadcast? Adoption seems a lot less attractive if it won't be receiving any or ongoing updates.

I'm planning to start contributing back to the open-source project with all the new features that we added to squadcast.com once I get time!

Originally posted by @roshan8 in #20 (comment)

Add integration with VictoriaMetrics

It would be great if slo-tracker could integrate with VictoriaMetrics + vmalert.

Failing build with the latest image version of docker nodejs

Running docker-compose up --build -d results in the following error:

Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './lib/tokenize' is not defined by "exports" in /usr/src/app/node_modules/postcss-safe-parser/node_modules/postcss/package.json
    at new NodeError (node:internal/errors:371:5)
    at throwExportsNotFound (node:internal/modules/esm/resolve:429:9)
    at packageExportsResolve (node:internal/modules/esm/resolve:705:3)
    at resolveExports (node:internal/modules/cjs/loader:482:36)
    at Function.Module._findPath (node:internal/modules/cjs/loader:522:31)
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:919:27)
    at Function.Module._load (node:internal/modules/cjs/loader:778:27)
    at Module.require (node:internal/modules/cjs/loader:999:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/usr/src/app/node_modules/postcss-safe-parser/lib/safe-parser.js:1:17) {
  code: 'ERR_PACKAGE_PATH_NOT_EXPORTED'
}

Node.js v17.5.0
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
1 error occurred:
	* Status: The command '/bin/sh -c yarn build' returned a non-zero code: 1, Code: 1

As a fix i changed the image version in ui/Dockerfile from node:latest to node:16 and it worked

Running npm install react-scripts failed

Step 6/14 : RUN npm install react-scripts --save -legacy-peer-deps
---> Running in c8d0aa420339
npm ERR! code ERR_SOCKET_TIMEOUT
npm ERR! errno ERR_SOCKET_TIMEOUT
npm ERR! network Invalid response body while trying to fetch https://registry.npmjs.org/rc-tree: Socket timeout
npm ERR! network This is a problem related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly. See: 'npm help config'

npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2022-04-01T13_40_04_573Z-debug-0.log
The command '/bin/sh -c npm install react-scripts --save -legacy-peer-deps' returned a non-zero code: 1
ERROR: Service 'slo-fe' failed to build : Build failed

I see network-related errors/timeouts but my internet connectivity is solid.