isabella232 / sensu-aggregate-check

The Sensu Go Aggregate Check Plugin

Sensu Go Aggregate Check Plugin

• Overview
• Files
• Usage examples
• Configuration
  • Sensu Go
    • Asset registration
    • Check definition
    • RBAC
  • Sensu Core
• Installation from source
• Additional notes
• Contributing

Overview

An aggregate makes it possible to treat the result of multiple disparate check executions executed across multiple disparate systems as a single result (Event). Aggregates are extremely useful in dynamic environments and/or environments that have a reasonable tolerance for failure. Aggregates should be used when a service can be considered healthy as long as a minimum threshold is satisfied (e.g. are at least 5 healthy web servers? are at least 70% of N processes healthy?).

This plugin allows you to query the Sensu Go Backend API for Events matching certain criteria (labels). This plugin generates a set of counters (e.g. events total, events in an OK state, etc) from the Events query and provides several CLI arguments to evaluate the computed aggregate counters (e.g. --warn-percent=75).

Files

N/A

Usage examples

Help

The Sensu Go Event Aggregates Check plugin

Usage:
  sensu-aggregate-check [flags]

Flags:
  -H, --api-host string          Sensu Go Backend API Host (e.g. 'sensu-backend.example.com') (default "127.0.0.1")
  -k, --api-key string           Sensu Go Backend API Key
  -P, --api-pass string          Sensu Go Backend API Password (default "P@ssw0rd!")
  -p, --api-port string          Sensu Go Backend API Port (e.g. 4242) (default "8080")
  -u, --api-user string          Sensu Go Backend API User (default "admin")
  -l, --check-labels string      Sensu Go Event Check Labels to filter by (e.g. 'aggregate=foo')
  -C, --crit-count int           Critical threshold - count of Events in warning state
  -c, --crit-percent int         Critical threshold - % of Events in warning state
  -e, --entity-labels string     Sensu Go Event Entity Labels to filter by (e.g. 'aggregate=foo,app=bar')
  -h, --help                     help for sensu-aggregate-check
  -i, --insecure-skip-verify     skip TLS certificate verification (not recommended!)
  -n, --namespaces string        Comma-delimited list of Sensu Go Namespaces to query for Events (e.g. 'us-east-1,us-west-2') (default "default")
  -s, --secure                   Use TLS connection to API
  -t, --trusted-ca-file string   TLS CA certificate bundle in PEM format
  -W, --warn-count int           Warning threshold - count of Events in warning state
  -w, --warn-percent int         Warning threshold - % of Events in warning state

Configuration

Sensu Go

Asset registration

Assets are the best way to make use of this plugin. If you're not using an asset, please consider doing so! If you're using sensuctl 5.13 or later, you can use the following command to add the asset:

sensuctl asset add sensu/sensu-aggregate-check

If you're using an earlier version of sensuctl, you can download the asset definition from this project's Bonsai asset index page or one of the existing releases or create an executable script from this source.

Check definition

api_version: core/v2
type: CheckConfig
metadata:
  namespace: default
  name: dummy-app-aggregate
spec:
  runtime_assets:
  - sensu-aggregate-check
  command: sensu-aggregate-check --api-user=foo --api-pass=bar --check-labels='aggregate=healthz,app=dummy' --warn-percent=75 --crit-percent=50
  subscriptions:
  - backend
  publish: true
  interval: 30
  handlers:
  - slack
  - pagerduty
  - email

RBAC

It is advised to use RBAC to create a user scoped specifically for purposes such as this check and to not re-use the admin account. For this check, in particular, the account would need access to list and retrieve events. The example below shows how to create a limited-scope user and the necessary role and role-binding resources to give it the required access.

$ sensuctl user create aggregate --password='4yva#ko!Yq'
Created

$ sensuctl role create get-events --verb list,get --resource events
Created

$ sensuctl role-binding create aggregate-get-events --role=get-events --user=aggregate
Created

Though you could use the user and password combination above with this check, the best practice would be to use an API key instead. You can create the API key with sensuctl:

$ sensuctl api-key grant aggregate
Created: /api/core/v2/apikeys/03f66dbf-6fe0-40d4-8174-95b5eab95649

The key (the text after [...]/apikeys/) above can be used with the --api-key argument in place of using api-user and api-pass.

Sensu Core

N/A

Installation from source and contributing

Sensu Go

To build from source, from the local path of the sensu-aggregate-check repository:

go build

Contributing

To contribute to this plugin, see CONTRIBUTING