monitoring-tools

Kamila's monitoring toolkit. Revolves around Prometheus.

Quickly set up a monitoring server

Note: I made this for DataCenterLight. Check it out!

cdist is an interesting take on configuration management developed at ungleich. It is simple, versatile, and has no dependencies on the target system. In fact, it is a bit too simple compared to other, far more sophisticated configuration management solutions -- and that's precisely why I like it.

I use it for configuring all my monitoring servers (among other things). Below you can find a cdist manifest snippet that comes from our real configuration at DataCenterLight.

I have created several cdist types useful for installing a Prometheus monitoring server, as well as many exporters:

High-level types for Prometheus and friends

__prometheus_server: Installs and configures Prometheus server. Yay!
__prometheus_alertmanager: Installs and configures Prometheus Alertmanager. Yay too!
__grafana_dashboard: Installs and configures Grafana. Yay three!

Extras (use as needed)

Daemontools

Daemontools are awesome. In line with the Unix philosophy, they do one thing and do it well. In this case they make sure that services run. Without the messy parts!

__daemontools: Installs daemontools or your favorite derivative, plus an init script in case your package doesn't come with one.
__daemontools_service: Creates a service directory. I gladly use this for everything that doesn't come with a good init script, such as some exporters. It's so simple!

Installing golang and go packages

Most distros provide a reasonably new version of Prometheus and the basic exporters (worst case you'll have to use the --install-from-backports with my __prometheus_server). This is the recommended way. However, if you need something more, these types let you install a recent version of golang and go packages. Note that I am currently not using these in production. I am listing them here anyway for completeness.

__golang_from_vendor: Installs golang from https://golang.org. This makes it easy to install any version of golang, including those that your old and obsolete distro doesn't have.
__go_get: Easy installation of golang packages straight from online repos. Useful for installing a newer version of Prometheus or various exporters.

Example

Put this in you cdist manifest, edit to your needs, and rejoice!

Monitoring server:

__prometheus_server \
    --install-from-backports \
    --config "$__manifest/files/prometheus.yml" \
    --retention-days 14 \
    --storage-path /data/prometheus \
    --rule-files "$__manifest/files/*.rules"
    
__prometheus_alertmanager \
    --install-from-backports \
    --config "$__manifest/files/alertmanager.yml" \
    --storage-path /data/alertmanager
    
__grafana_dashboard

A cdist lightning-quick-start

git clone https://github.com/ungleich/cdist.git
cd cdist
$EDITOR cdist/conf/manifest/init
bin/cdist config -vv <hostname>

Exporters for monitoring software that doesn't have built-in instrumentation

ttn2prom: Trivial Python webserver that accepts webhooks (i.e. HTTP requests), processes the data in them, and exports the stuff in the Prometheus format. Very easy to adapt. This version exports data from some LoRaWan temperature sensors through webhooks from The Things Network.
opennebula exporter: Exports useful metrics for OpenNebula. For now it is an evil hack, but it will get better with time.
OpenNebula integration test: tests VM creation and connectivity, and exports the results in a Prometheus-compatible format.

Useful alerts collection

TBD

anotherkamila / monitoring-tools Goto Github PK