sensu / sensu-aggregate-check

The Sensu Go Aggregate Check Plugin

Sensu Go Aggregate Check Plugin

• Overview
• Files
• Usage examples
• Configuration
  • Sensu Go
    • Asset registration
    • Check definition
    • RBAC
  • Sensu Core
• Installation from source
• Additional notes
• Contributing

Overview

An aggregate makes it possible to treat the result of multiple disparate check executions executed across multiple disparate systems as a single result (Event). Aggregates are extremely useful in dynamic environments and/or environments that have a reasonable tolerance for failure. Aggregates should be used when a service can be considered healthy as long as a minimum threshold is satisfied (e.g. are at least 5 healthy web servers? are at least 70% of N processes healthy?).

This plugin allows you to query the Sensu Go Backend API for Events matching certain criteria (labels). This plugin generates a set of counters (e.g. events total, events in an OK state, etc) from the Events query and provides several CLI arguments to evaluate the computed aggregate counters (e.g. --warn-percent=75).

Files

N/A

Usage examples

Help

The Sensu Go Event Aggregates Check plugin

Usage:
  sensu-aggregate-check [flags]

Flags:
  -H, --api-host string          Sensu Go Backend API Host (e.g. 'sensu-backend.example.com') (default "127.0.0.1")
  -k, --api-key string           Sensu Go Backend API Key
  -P, --api-pass string          Sensu Go Backend API Password (default "P@ssw0rd!")
  -p, --api-port string          Sensu Go Backend API Port (e.g. 4242) (default "8080")
  -u, --api-user string          Sensu Go Backend API User (default "admin")
  -l, --check-labels string      Sensu Go Event Check Labels to filter by (e.g. 'aggregate=foo')
  -C, --crit-count int           Critical threshold - count of Events in warning state
  -c, --crit-percent int         Critical threshold - % of Events in warning state
  -e, --entity-labels string     Sensu Go Event Entity Labels to filter by (e.g. 'aggregate=foo,app=bar')
  -h, --help                     help for sensu-aggregate-check
  -i, --insecure-skip-verify     skip TLS certificate verification (not recommended!)
  -n, --namespaces string        Comma-delimited list of Sensu Go Namespaces to query for Events (e.g. 'us-east-1,us-west-2') (default "default")
  -s, --secure                   Use TLS connection to API
  -t, --trusted-ca-file string   TLS CA certificate bundle in PEM format
  -W, --warn-count int           Warning threshold - count of Events in warning state
  -w, --warn-percent int         Warning threshold - % of Events in warning state

Configuration

Sensu Go

Asset registration

Assets are the best way to make use of this plugin. If you're not using an asset, please consider doing so! If you're using sensuctl 5.13 or later, you can use the following command to add the asset:

sensuctl asset add sensu/sensu-aggregate-check

If you're using an earlier version of sensuctl, you can download the asset definition from this project's Bonsai asset index page or one of the existing releases or create an executable script from this source.

Check definition

api_version: core/v2
type: CheckConfig
metadata:
  namespace: default
  name: dummy-app-aggregate
spec:
  runtime_assets:
  - sensu-aggregate-check
  command: sensu-aggregate-check --api-user=foo --api-pass=bar --check-labels='aggregate=healthz,app=dummy' --warn-percent=75 --crit-percent=50
  subscriptions:
  - backend
  publish: true
  interval: 30
  handlers:
  - slack
  - pagerduty
  - email

RBAC

It is advised to use RBAC to create a user scoped specifically for purposes such as this check and to not re-use the admin account. For this check, in particular, the account would need access to list and retrieve events. The example below shows how to create a limited-scope user and the necessary role and role-binding resources to give it the required access.

$ sensuctl user create aggregate --password='4yva#ko!Yq'
Created

$ sensuctl role create get-events --verb list,get --resource events
Created

$ sensuctl role-binding create aggregate-get-events --role=get-events --user=aggregate
Created

Though you could use the user and password combination above with this check, the best practice would be to use an API key instead. You can create the API key with sensuctl:

$ sensuctl api-key grant aggregate
Created: /api/core/v2/apikeys/03f66dbf-6fe0-40d4-8174-95b5eab95649

The key (the text after [...]/apikeys/) above can be used with the --api-key argument in place of using api-user and api-pass.

Sensu Core

N/A

Installation from source and contributing

Sensu Go

To build from source, from the local path of the sensu-aggregate-check repository:

go build

Contributing

To contribute to this plugin, see CONTRIBUTING

sensu-aggregate-check's Issues

Enhancement: Query the API instead of requesting all events

The plugin should use label and field selectors to query the API rather than requesting all events.

Readme needs updating to match updated Readme guidelines

See https://github.com/sensu-plugins/community/blob/master/PLUGIN_STYLEGUIDE.md

Enterprise API Filtering

Update the plugin to leverage the newly shipped (so fresh) Enterprise API Filtering functionality. OSS/Free users will still be able to use the plugin without API Filtering, it will simply perform poorly.

README updates to better document aggregates pattern

We've seen some confusion when users try to adopt aggregates, in part because the pattern is not obvious nor really documented in our plugin readme.

We should add some additional content to the readme to make this more clear:

• Add examples of edge node checks, e.g. application health checks which have appropriate labels
• Add a diagram showing how the pattern is implemented and which agents should run which checks

Running `sensu-aggregate-check` fails with cryptic message.

All,

When I try to run sensu-aggregate-check, against a local instance of sensu-go-backend (5.18.1-9930), I'm getting a cryptic error message:

 sudo -H /opt/sensu-plugins-ruby/embedded/bin/sensu-aggregate-check --api-port 3080 --check-labels='aggregate=stability'
Error: invalid character 'C' looking for beginning of value
Usage:
  sensu-aggregate-check [flags]

Flags:
  -H, --api-host string        Sensu Go Backend API Host (e.g. 'sensu-backend.example.com') (default "127.0.0.1")
  -P, --api-pass string        Sensu Go Backend API User (default "P@ssw0rd!")
  -p, --api-port string        Sensu Go Backend API Port (e.g. 4242) (default "8080")
  -u, --api-user string        Sensu Go Backend API User (default "admin")
  -l, --check-labels string    Sensu Go Event Check Labels to filter by (e.g. 'aggregate=foo')
  -C, --crit-count int         Critical threshold - count of Events in critical state
  -c, --crit-percent int       Critical threshold - % of Events in critical state
  -e, --entity-labels string   Sensu Go Event Entity Labels to filter by (e.g. 'aggregate=foo,app=bar')
  -h, --help                   help for sensu-aggregate-check
  -n, --namespaces string      Comma-delimited list of Sensu Go Namespaces to query for Events (e.g. 'us-east-1,us-west-2') (default "default")
  -W, --warn-count int         Warning threshold - count of Events in warning state
  -w, --warn-percent int       Warning threshold - % of Events in warning state

error: invalid character 'C' looking for beginning of value
Exit 1

I can sign into the api using curl
sudo curl -k --user admin:"...." https://127.0.0.1:3080/auth so I know my --api-port is ready but I don't know what is wrong here.

• Mike D.

Enhancement: Expand filtering capabilities via field selectors

One enhancement that would make the plugin more useful would be to implement the use of field selectors when examining the events . It could then query against check names, rather than relying on check labels.

Use a Sensu API key

Sensu Go now supports API keys, no longer have to use a JWT. API keys are the preferred means of authentication/authorization for check plugins.

Replaces https://github.com/sensu/sensu-enterprise-go/issues/438

Error handling for bad api calls

When passing bad configurations, the errors returned are not helpful. This may be due to error handling from responses coming from the api.

As an example configuring --api-pass=notthecorrectpassword will return in my use case:

Error executing sensu-aggregate-check: error executing check: invalid character 'U' looking for beginning of value

While configured with the correct password the check works correctly.